add perplexity and stub out smol-IQ1_KT

README.md

@@ -26,13 +26,125 @@ Also thanks to all the folks in the quanting and inferencing community on [Beave
 Finally, I appreciate all the support from [aifoundry.org](https://aifoundry.org) and team as well as huggingface for hosting all these big quants!
 
 ## Quant Collection
-This is an interesting one, currently only one available given the the original model design. The `Q8_0-Q4_0` is `q4_0` routed experts and `q8_0` all other tensors. It works on both ik_llama.cpp and mainline llama.cpp in limited testing. It does *not* use an imatrix!
+I may try to make a smaller one e.g. `smol-IQ1_KT` or `smol-IQ2_KS` or similar but not sure how well it will go given the original is QAT'd with `compressed-tensors` to *very similar* to q4_0 except using bf16 block scales instead of fp16 but same 32 weights per block size.
+
+Perplexity computed against *wiki.test.raw*.
+
+![Perplexity Chart](images/perplexity.png "Chart showing Perplexity improving as BPW increases.")
+
+## Q8_0-Q4_0 543.617 GiB (4.549 BPW)
+Final estimate: PPL = 2.1257 +/- 0.00934
+
+This specific quant works on both ik_llama.cpp and mainline llama.cpp. It does *not* use an imatrix and was created going from the original model to full bf16 before further quantization. The exact PR used is linked below in references.
+
+<details>
+
+<summary>👈 Secret Recipe</summary>
+
+```bash
+#!/usr/bin/env bash
+
+# Q4_0 routed experts approximating original QAT design
+# Q8_0 everything else
+
+custom="
+## Attention [0-60] (GPU)
+blk\..*\.attn_k_b\.weight=q8_0
+blk\..*\.attn_v_b\.weight=q8_0
+
+# Balance of attn tensors
+blk\..*\.attn_kv_a_mqa\.weight=q8_0
+blk\..*\.attn_q_a\.weight=q8_0
+blk\..*\.attn_q_b\.weight=q8_0
+blk\..*\.attn_output\.weight=q8_0
+
+## First Single Dense Layer [0] (GPU)
+blk\..*\.ffn_down\.weight=q8_0
+blk\..*\.ffn_(gate|up)\.weight=q8_0
+
+## Shared Expert [1-60] (GPU)
+blk\..*\.ffn_down_shexp\.weight=q8_0
+blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0
+
+## Routed Experts [1-60] (CPU)
+blk\..*\.ffn_down_exps\.weight=q4_0
+blk\..*\.ffn_(gate|up)_exps\.weight=q4_0
+
+token_embd\.weight=q8_0
+output\.weight=q8_0
+"
+
+custom=$(
+  echo "$custom" | grep -v '^#' | \
+  sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
+)
 
-* `Q8_0-Q4_0` 543.617 GiB (4.549 BPW)
+numactl -N ${SOCKET} -m ${SOCKET} \
+./build/bin/llama-quantize \
+    --custom-q "$custom" \
+    /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/-384x14B-BF16-00001-of-00046.gguf \
+    /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/Kimi-K2-Thinking-Q8_0-Q4_0.gguf \
+    Q8_0 \
+    128
+```
+
+</details>
 
+## smol-IQ1_KT TODO
 Final estimate: PPL = TODO
 
-I may try to make a smaller one e.g. `smol-IQ1_KT` or `smol-IQ2_KS` or similar but not sure how well it will go given the original is QAT'd with `compressed-tensors` to *very similar* to q4_0 except using bf16 block scales instead of fp16 but same 32 weights per block size.
+Not sure this will be any good, finishing up imatrix now and will test this before releasing. Hopefully will fit in under 256GB RAM+VRAM.
+
+<details>
+
+<summary>👈 Secret Recipe</summary>
+
+```bash
+#!/usr/bin/env bash
+
+custom="
+## Attention [0-60] (GPU)
+blk\..*\.attn_k_b\.weight=q8_0
+blk\..*\.attn_v_b\.weight=q8_0
+
+# Balance of attn tensors
+blk\..*\.attn_kv_a_mqa\.weight=q8_0
+blk\..*\.attn_q_a\.weight=q8_0
+blk\..*\.attn_q_b\.weight=q8_0
+blk\..*\.attn_output\.weight=q8_0
+
+## First Single Dense Layer [0] (GPU)
+blk\..*\.ffn_down\.weight=q8_0
+blk\..*\.ffn_(gate|up)\.weight=q8_0
+
+## Shared Expert [1-60] (GPU)
+blk\..*\.ffn_down_shexp\.weight=q8_0
+blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0
+
+## Routed Experts [1-60] (CPU)
+blk\..*\.ffn_down_exps\.weight=iq1_kt
+blk\..*\.ffn_(gate|up)_exps\.weight=iq1_kt
+
+token_embd\.weight=iq4_k
+output\.weight=iq6_k
+"
+
+custom=$(
+  echo "$custom" | grep -v '^#' | \
+  sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
+)
+
+numactl -N ${SOCKET} -m ${SOCKET} \
+./build/bin/llama-quantize \
+    --custom-q "$custom" \
+    --imatrix /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/imatrix-Kimi-K2-Thinking-BF16.dat \
+    /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/-384x14B-BF16-00001-of-00046.gguf \
+    /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/Kimi-K2-Thinking-IQ1_KT.gguf \
+    IQ1_KT \
+    128
+```
+
+</details>
 
 ## Quick Start
 You might need to override the template as needed. The original is here: https://huggingface.co/moonshotai/Kimi-K2-Thinking/blob/main/chat_template.jinja