docs: correct ik_llama verdict — ~18-20% FASTER than mainline, not a "tie" (#184)

The 2026-05-22 "ik ties mainline at matched power" verdict was a wrong-engine measurement artifact: its "ik ~50/58" is exactly mainline@370, a number ik produces at NO power setting. A 2026-05-23 set-and-readback power-cap A/B + 5 independent ik runs (all ~70 code) confirm ik IQ4_KS is ~18-20% faster than llamacpp/mtp Q4_K_M on decode TPS at matched 370 W (~60/69 wall vs ~50/58), quality-tied (8-pack 101 vs 100) and ~0.5-0.8 GB leaner — faster AND leaner. Both llama.cpp engines are power-sensitive on this rig (ik -29% / mainline -42% from 370->230 W), so the comparison is only valid at matched power; the original confusion traced to unmatched/mismeasured power + a stale-container mixup. Corrected: IK_LLAMA.md (headline + measured table + bench note), SINGLE_CARD.md (picker row + prose), iq4ks-mtp.yml (Perf header), + new BENCHMARKS ik row. discussions/184 writeup updated separately. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 20:11:43 +00:00
parent c502bf13a9
commit b7353daa52
4 changed files with 13 additions and 11 deletions
--- a/BENCHMARKS.md
+++ b/BENCHMARKS.md
@@ -84,6 +84,7 @@ Primary serving model. Hybrid Qwen3-Next architecture (DeltaNet GDN + standard a
 | `llamacpp/default` | @noonghunna (1× 3090) | Unsloth Q5_K_XL | 262K | 21 / 21 | — | ~20 GB | 2026-04-21 | bulletproof — different engine, different memory allocator, no Cliff 1 / Cliff 2. Slow decode but cliff-immune. |
 | `llamacpp/mtp` ⭐ | @noonghunna (1× 3090) | Unsloth Q4_K_M MTP (`unsloth/Qwen3.6-27B-MTP-GGUF`) | **131K** | **51.28 / 59.72** (decode, n=3, CV 1.9% / 0.5%) | **1063** | ~22.9 GB (~1.6 GB headroom) | 2026-05-19 | **Mainline llama.cpp build 9235 (d14ce3dab, PR #22673 merged)** + MTP `n=2` + q4_0 KV + `-ub 1024` + native template + `--reasoning off`. **verify-stress 7/7 PASS incl. 60K + 91K needle recall** — at this config the Cliff 2 single-prompt narrative is **config-driven, not architectural** (the prior "50–60K wall" was the `-ub 2048` activation-peak bound; `-ub 1024` halves it and walks past 91K cleanly). **Quality 8-pack 102/150 (68%)** — beats every Qwen vLLM-dual config in [#119](https://github.com/noonghunna/club-3090/discussions/119) by 6–16 pp. **Aider-polyglot-30 17/30 (56.7%)** — matches Qwen vLLM bf16 dual exactly on half the hardware (per-lang: cpp 3/5, go 4/5, java 2/5, js 4/5, python 3/5, rust 1/5). **Per-GPU code TPS (59.72) ≈ vLLM-dual configs (60–63)** — engine-side per-card rate is identical; vLLM dual's aggregate advantage is purely from the second card. Zero patches, no Genesis, no AutoRound. [PR #166](https://github.com/noonghunna/club-3090/pull/166). |
 | `llamacpp/mtp-vision` (NEW) | @noonghunna (1× 3090) | Unsloth Q4_K_M MTP + mmproj-F16 | **49K** | **56.52 / 66.17** (decode, n=5, CV 1.6% / 1.7%) | **1158** | ~20.5 GB (~3.5 GB headroom) | 2026-05-20 | **First stack profile combining MTP + vision** (sweep-verified on build 9235; the older "strip mmproj when MTP" rule was obsolete). Same Q4_K_M MTP GGUF + q4_0 KV + MTP `n=2` + `-ub 1024` + `--mmproj mmproj-F16.gguf` (vision projector loaded). **Multimodal probe ✅** (model answered "Red" on a synthetic 64×64 red PNG in 0.5s; mmproj pipeline functional end-to-end). **verify-stress 7/7 PASS** (`sapphire iguana 19` @ 9.8K, `crimson falcon 18` @ 29.3K, all 7 rungs clean at the 49K ctx). TPS being ~10% higher than the no-vision Config A above isn't strictly A/B-controlled across GPUs — Config A was on GPU 0 after ~2 hrs continuous load; this was a fresh GPU 1; plausibly KV-cache-locality (49K pool 2.6× smaller than 131K) + thermal-state. [PR #166](https://github.com/noonghunna/club-3090/pull/166). |
+| `ik-llama/iq4ks-mtp` ⭐ | @noonghunna (1× 3090, set+readback 370 W) | ubergarm MTP-IQ4_KS (`ubergarm/Qwen3.6-27B-GGUF`) | **200K** | **59.67 / 68.78** (decode 60.39 / 72.40, n=3, CV 1.5% / 0.6%) | **1109** | ~22.4 GB | 2026-05-23 | **ik_llama.cpp `cu13-server` + IQK fused kernels + q4_0 KV + MTP n=2 + `-khad`/`-vhad` + native template, thinking-off. ~18–20% FASTER decode TPS than shipped `llamacpp/mtp` Q4_K_M** (49.69 / 57.50 wall, same rig + 370 W cap) — verified by a **set-and-readback power-cap A/B**: ik leads at *both* 230 W and 370 W (both llama.cpp engines are power-sensitive on this rig — ik −29% / mainline −42% from 370→230 W; comparison only valid at matched power). 8-pack **101/150** (≈ tie, mainline 100), **soak PASS** (p50 71.5, 0 err, 0/100 silent-empty, 0 MiB growth), verify-stress 8/8 incl. ceiling-ladder fill to 183K @ 200K, aider-polyglot-30 19/30. **Corrects the [#184](https://github.com/noonghunna/club-3090/discussions/184) "tie"** — that was a wrong-engine measurement artifact: its "ik ~50/58" is exactly mainline@370, a number ik produces at *no* power setting (ik = 42/51 @ 230 W, 60/72 @ 370 W). |
 | llama.cpp PR [#22673](https://github.com/ggml-org/llama.cpp/pull/22673) MTP, custom build (`Qwen3.6-27B-MTP-Q4_K_M-GGUF` + `--spec-type mtp --spec-draft-n-max 3`) | @efschu (**2× Tesla V100-SXM2-16GB**, Xeon Gold 6154, Debian 13, custom-built llama-server docker) | Q4_K_M MTP | 100K | **49.96 / 62.46** | — | 15.6 GB/card (15,596 MiB at 100K ctx) | 2026-05-06 | **First V100 (sm_70 Volta) cross-rig data on the matrix** — only non-3090/4090/5090 GPU class tested. vLLM blocked (V100=CC 7.0, vLLM needs ≥7.5); fell back to llama.cpp via am17an's PR #22673 with a custom-built docker. **All 7 stress checks PASS including 90K NIAH** (Cliff 2 territory). 2× cards via tensor split (`-sm tensor`). MTP n=3, accept rates not in log. ~80 W/card (V100 max 300 W). [Issue #80](https://github.com/noonghunna/club-3090/issues/80). |
 | llama.cpp PR [#22673](https://github.com/ggml-org/llama.cpp/pull/22673) MTP, host build (`havenoammo/Qwen3.6-27B-MTP-UD-GGUF` + `--spec-type mtp --spec-draft-n-max 3` + q4_0 KV) | @lamentofhighborne (1× RTX 3090, PCIe x8, 350W) | UD-Q4_K_XL + Q8_0 MTP head | **131K** | **47.12 / 60.42** | — | ~23.1 GiB | 2026-05-07 | **First 1× 3090 llama.cpp MTP data point** on Qwen3.6-27B. Decode 47.60 / 61.71 TPS, TTFT 212 / 194 ms. **`verify-full-mtp.sh` PASS 8/8** (locally-adapted), **`verify-stress-mtp.sh` PASS 7/7 including 91K needle at 131K ctx** — pushes the documented llama.cpp MTP ctx ceiling from ~64-80K (q8_0 KV) to 131K (q4_0 KV). MTP acceptance 78.7%; recurrent 65-layer bug from froggeric's earlier MTP GGUF did **NOT** reproduce on havenoammo's UD GGUF. Native host build (no Docker), surfaced engine-coupling shortcomings in our verify/soak harness — see [Issue #85](https://github.com/noonghunna/club-3090/issues/85). |
 | llama.cpp PR [#22673](https://github.com/ggml-org/llama.cpp/pull/22673) MTP, host build (`froggeric/Qwen3.6-27B-MTP-GGUF` + `--spec-type mtp --spec-draft-n-max 3` + q4_0 KV) | @lamentofhighborne (1× RTX 3090, PCIe x8, 350 W) | Q4_K_M MTP | **164K** | **47.49 / 55.09** | — | ~22.2 GiB | 2026-05-07 | **Second 1× 3090 llama.cpp MTP data point on same rig** — froggeric's Q4_K_M MTP GGUF vs havenoammo's UD-Q4_K_XL above. Decode 47.91 / 55.81 TPS, TTFT 96 / 98 ms. `verify-full-mtp.sh` PASS 8/8, `verify-stress-mtp.sh` PASS 7/7 incl. 91K needle at 164K ctx. Functional MTP acceptance **86.7%**; canonical acceptance 55.3% narr / 71.2% code. **Ctx-fit ladder**: 262K OOMed MTP, 229K served without MTP, 196K initialized MTP but daemon died at 90K stress; 164K was the stable stress-passing ceiling on this rig. **Beats havenoammo on narr (47.49 vs 47.12, +0.8%) and ctx ceiling (164K vs 131K) but trails on code (55.09 vs 60.42, −9%)**. Manual long-context needles also passed at **120K** (39.39 decode TPS, 81% MTP accept) and **150K** (35.44 decode TPS, 80% MTP accept). MTP+vision incompat (per froggeric's model card); separate no-MTP+vision path passed 65K and 150K. [Issue #94](https://github.com/noonghunna/club-3090/issues/94). |
--- a/docs/SINGLE_CARD.md
+++ b/docs/SINGLE_CARD.md
@@ -42,7 +42,7 @@ For workloads that **don't** accumulate context across turns (single-shot RAG, s
 | **Bulletproof, no cliffs** (production service, unpredictable inputs) | [`llamacpp/default`](../models/qwen3.6-27b/llama-cpp/compose/single/mtp.yml) (alias of `llamacpp/mtp`) | **262K** (via `-ub 512`) | 52 / 61 | ~23 GB |
 | **llama.cpp + MTP, fast + long ctx** ⭐ (IDE agents, opencode, Hermes, long-multi-turn agentic) | [`llamacpp/mtp`](../models/qwen3.6-27b/llama-cpp/compose/single/mtp.yml) | **131K** | **51 / 60** | ~22.5 GB |
 | **llama.cpp + MTP + vision** (multimodal chat, screenshot-debugging, vision-aware review) | [`llamacpp/mtp-vision`](../models/qwen3.6-27b/llama-cpp/compose/single/mtp-vision.yml) | **49K** | **57 / 66** | ~20.5 GB |
-| **ik_llama + IQ4_KS + MTP** (best quality-per-bit GGUF + leanest VRAM; advanced-quant track) | [`iq4ks-mtp`](../models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp.yml) | **262K** | ~50 / ~58 | **~22 GB** (leanest) |
+| **ik_llama + IQ4_KS + MTP** ⭐ (fastest single-card + leanest VRAM; advanced-quant track) | [`iq4ks-mtp`](../models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp.yml) | **262K** | **~60 / ~69** | **~22 GB** (leanest) |
 | **ik_llama + IQ4_KS + MTP + vision** | [`iq4ks-mtp-vision`](../models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp-vision.yml) | **160K** | TBD | ~21 GB |
 | **ik_llama + two-stage spec-dec** 🧪 (ngram+MTP, code-optimized, experimental) | [`iq4ks-two-stage`](../models/qwen3.6-27b/ik-llama/compose/single/iq4ks-two-stage.yml) | **131K** | TBD | ~22 GB |
 | **Small-context vLLM safe path** ([@stiggy2k16](https://github.com/noonghunna/club-3090/issues/43) data point) — IDE agents capped at <60K accumulated, when you need vLLM speed but llama.cpp is too slow. ⚠️ Genesis-free, but its default pin is purged (#167) — run it with `VLLM_IMAGE=vllm/vllm-openai:latest` until the pin's bumped | [`minimal.yml`](../models/qwen3.6-27b/vllm/compose/single/minimal.yml) at `--gpu-memory-utilization 0.95 --max-model-len 65536` | **64K** | ~32 / ~33 (no MTP) | ~22.4 GB |
@@ -142,7 +142,7 @@ For the cross-card TP=2 picture, see [`DUAL_CARD.md`](DUAL_CARD.md).

 **Workload:** best quality-per-bit GGUF on a single 3090. Same cliff-immunity as llama.cpp (same ggml memory model), but with fork-exclusive IQK imatrix quants that beat mainline Q4_K_M on perplexity.

-ubergarm `Qwen3.6-27B-MTP-IQ4_KS.gguf` (IQK imatrix quant, built-in MTP head) + q4_0 KV + `-khad`/`-vhad` (Hadamard K+V cache transforms) + MTP `n=2` + `--merge-qkv` + `--parallel-tool-calls` (ik-exclusive). **262K context** on one 3090. At matched power it **ties `llamacpp/mtp`** on TPS (~50 narr / ~58 code) and quality (8-pack 103 vs 102) — the earlier "+18–20%" was a power/card artifact ([#184](https://github.com/noonghunna/club-3090/discussions/184)). Its real edge is a **~0.5–0.8 GB leaner footprint**, which makes it the pick when VRAM is tight (sub-24 GB, shared GPU, WSL). Engine: `ghcr.io/ikawrakow/ik-llama-cpp:cu13-server`. See [`docs/engines/IK_LLAMA.md`](engines/IK_LLAMA.md) for the full deep dive.
+ubergarm `Qwen3.6-27B-MTP-IQ4_KS.gguf` (IQK imatrix quant, built-in MTP head) + q4_0 KV + `-khad`/`-vhad` (Hadamard K+V cache transforms) + MTP `n=2` + `--merge-qkv` + `--parallel-tool-calls` (ik-exclusive). **262K context** on one 3090. At matched 370 W it's **~18–20% faster than `llamacpp/mtp`** on TPS (~60 narr / ~69 code vs ~50 / ~58), quality-tied (8-pack 103 vs 102), and **~0.5–0.8 GB leaner** — the faster *and* leaner single-card path. (A 2026-05-22 "tie" was a wrong-engine measurement artifact — corrected 2026-05-23 via a power-cap-controlled A/B, [#184](https://github.com/noonghunna/club-3090/discussions/184).) Engine: `ghcr.io/ikawrakow/ik-llama-cpp:cu13-server`. See [`docs/engines/IK_LLAMA.md`](engines/IK_LLAMA.md) for the full deep dive.

 ### ik_llama + IQ4_KS + MTP + vision — `iq4ks-mtp-vision.yml`

--- a/docs/engines/IK_LLAMA.md
+++ b/docs/engines/IK_LLAMA.md
@@ -2,7 +2,7 @@

 **Role on this stack:** the engine you reach for when you want **newer, higher-quality-per-bit quants** than mainline llama.cpp ships — specifically the **IQK imatrix family** (`IQ4_KS`, `IQ5_KS`, …) that exists *only* in this fork. It's a llama.cpp fork (ikawrakow), so it inherits llama.cpp's cliff-immune memory model and broad hardware support, then adds a co-designed quant + kernel stack on top.

-> **In one line:** llama.cpp's robustness + fork-exclusive IQK quants + fused CUDA kernels → **MTP, clean to 262K on one card, quality on par with `llamacpp/mtp`** (8-pack 103 vs 102), at a **~0.5–0.8 GB leaner VRAM footprint**. On TPS it's a **tie** with `llamacpp/mtp` at matched power (~50 narr / ~58 code on a 3090); the leaner footprint — not speed — is the reason to pick it (it shines when VRAM-tight: sub-24 GB, shared GPU, WSL). Full matched-power write-up: [discussions/184](https://github.com/noonghunna/club-3090/discussions/184).
+> **In one line:** llama.cpp's robustness + fork-exclusive IQK quants + fused CUDA kernels → **MTP, clean to 262K on one card, quality on par with `llamacpp/mtp`** (8-pack 103 vs 102), at a **~0.5–0.8 GB leaner VRAM footprint**. It's also **~18–20% faster** than `llamacpp/mtp` on decode TPS at matched 370 W (~60 narr / ~69 code vs ~50 / ~58 on a 3090) — so it's the faster *and* leaner single-card path. The trade is a second engine to maintain + the IQK quant. Full matched-power write-up: [discussions/184](https://github.com/noonghunna/club-3090/discussions/184).

 For *what the quants actually are* and how IQK compares to k-quants / i-quants / AWQ, see **[../QUANTIZATION.md](../QUANTIZATION.md)**. For the cross-engine overview see **[../INFERENCE_ENGINES.md](../INFERENCE_ENGINES.md)**.

@@ -153,14 +153,14 @@ ik_llama has several quality-relevant flags beyond what the default composes use

 | Metric | Value | vs shipped `llamacpp/mtp` (Q4_K_M) |
 |---|---|---|
-| Decode TPS (narr / code) | ~50 / ~58 | **tie** at matched 370 W (the earlier "+18–20%" was a power/card artifact — see [#184](https://github.com/noonghunna/club-3090/discussions/184)) |
+| Decode TPS (narr / code) | **~60 / ~72** | **~18–20% faster** than `llamacpp/mtp` (~50 / ~59) at matched 370 W — confirmed by a power-cap-controlled A/B (ik leads at 230 W *and* 370 W); the earlier "tie" was a wrong-engine measurement artifact, see [#184](https://github.com/noonghunna/club-3090/discussions/184) |
 | Max context (1× 3090) | **262K** (q4_0 KV) | tie (mainline also reaches 262K via `-ub 512`) |
-| VRAM @ 262K | ~22.5 GB | **~0.5–0.8 GB leaner** ← ik's one real edge |
+| VRAM @ 262K | ~22.5 GB | **~0.5–0.8 GB leaner** (a second edge, alongside the TPS lead) |
 | verify-stress | 7/7 (incl. 91K Cliff 2 needle) | parity |
 | Quality 8-pack | **103/150** | ≈ tie (mainline 102) |
 | toolcall-15 | 60% (native template) | tie — native won the A/B on **both** engines |

-> Bench: canonical prompt, 3 warmup + measured runs, **verified 370 W same-card matched comparison**, q4_0 KV / MTP n=2 / thinking-off. The honest finding is a TPS/quality/context tie — ik's edge is the leaner footprint, useful when VRAM-tight. See [#184](https://github.com/noonghunna/club-3090/discussions/184) + [../../BENCHMARKS.md](../../BENCHMARKS.md).
+> Bench: canonical prompt, 3 warmup + measured runs, **set-and-readback 370 W same-card matched comparison**, q4_0 KV / MTP n=2 / thinking-off. The honest finding: **ik is ~18–20% faster on decode TPS** (quality + context tied, ~0.5–0.8 GB leaner) — faster *and* leaner. The 2026-05-22 "tie" was a wrong-engine measurement artifact (its "tie" number is exactly mainline@370, which ik produces at no power setting); corrected 2026-05-23 via a power-cap-controlled A/B + 5 independent ik runs all at ~70 code. See [#184](https://github.com/noonghunna/club-3090/discussions/184) + [../../BENCHMARKS.md](../../BENCHMARKS.md).
 > 
 > **Two-stage (ngram+MTP):** not yet benched. Compose exists (`iq4ks-two-stage.yml`), PR #1789 merged 2026-05-15. Expected to outperform MTP-only on code workloads with repeated patterns; bench pending.

--- a/models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp.yml
+++ b/models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp.yml
@@ -29,11 +29,12 @@
 #              → 21.5 GB if you need the extra ~1 GB back.
 #   Status:    ✅ Shipped (advanced-quant track, #180). In the wizard +
 #              `bash scripts/switch.sh ik-llama/iq4ks-mtp`.
-#   Perf:      Ties shipped llamacpp/mtp Q4_K_M on TPS (~50 narr / ~58 code at a
-#              verified, matched 370 W on the same card) AND 8-pack quality
-#              (103 vs 102) — the earlier "+18-20%" was a power/card artifact
-#              (see discussions/184). ik's real edge is a ~0.5-0.8 GB leaner
-#              VRAM footprint.
+#   Perf:      ~18-20% FASTER decode TPS than shipped llamacpp/mtp Q4_K_M
+#              (~60 narr / ~69 code vs ~50/58, at set-and-readback matched 370 W
+#              same card), quality-tied (8-pack 103 vs 102), AND ~0.5-0.8 GB
+#              leaner VRAM — faster AND leaner. Confirmed by a power-cap A/B +
+#              5 ik runs (all ~70 code). The 2026-05-22 "tie" was a wrong-engine
+#              measurement artifact (its tie number = mainline@370). See discussions/184.
 #   Best for:  VRAM-tight single-card (sub-24 GB, shared GPU, WSL display tax)
 #              where the leaner footprint matters; or if you want IQK quants.
 # ---------------------------------------------------------------------------