diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 401acfd0..bad577c6 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -84,6 +84,7 @@ Primary serving model. Hybrid Qwen3-Next architecture (DeltaNet GDN + standard a | `llamacpp/default` | @noonghunna (1× 3090) | Unsloth Q5_K_XL | 262K | 21 / 21 | — | ~20 GB | 2026-04-21 | bulletproof — different engine, different memory allocator, no Cliff 1 / Cliff 2. Slow decode but cliff-immune. | | `llamacpp/mtp` ⭐ | @noonghunna (1× 3090) | Unsloth Q4_K_M MTP (`unsloth/Qwen3.6-27B-MTP-GGUF`) | **131K** | **51.28 / 59.72** (decode, n=3, CV 1.9% / 0.5%) | **1063** | ~22.9 GB (~1.6 GB headroom) | 2026-05-19 | **Mainline llama.cpp build 9235 (d14ce3dab, PR #22673 merged)** + MTP `n=2` + q4_0 KV + `-ub 1024` + native template + `--reasoning off`. **verify-stress 7/7 PASS incl. 60K + 91K needle recall** — at this config the Cliff 2 single-prompt narrative is **config-driven, not architectural** (the prior "50–60K wall" was the `-ub 2048` activation-peak bound; `-ub 1024` halves it and walks past 91K cleanly). **Quality 8-pack 102/150 (68%)** — beats every Qwen vLLM-dual config in [#119](https://github.com/noonghunna/club-3090/discussions/119) by 6–16 pp. **Aider-polyglot-30 17/30 (56.7%)** — matches Qwen vLLM bf16 dual exactly on half the hardware (per-lang: cpp 3/5, go 4/5, java 2/5, js 4/5, python 3/5, rust 1/5). **Per-GPU code TPS (59.72) ≈ vLLM-dual configs (60–63)** — engine-side per-card rate is identical; vLLM dual's aggregate advantage is purely from the second card. Zero patches, no Genesis, no AutoRound. [PR #166](https://github.com/noonghunna/club-3090/pull/166). | | `llamacpp/mtp-vision` (NEW) | @noonghunna (1× 3090) | Unsloth Q4_K_M MTP + mmproj-F16 | **49K** | **56.52 / 66.17** (decode, n=5, CV 1.6% / 1.7%) | **1158** | ~20.5 GB (~3.5 GB headroom) | 2026-05-20 | **First stack profile combining MTP + vision** (sweep-verified on build 9235; the older "strip mmproj when MTP" rule was obsolete). Same Q4_K_M MTP GGUF + q4_0 KV + MTP `n=2` + `-ub 1024` + `--mmproj mmproj-F16.gguf` (vision projector loaded). **Multimodal probe ✅** (model answered "Red" on a synthetic 64×64 red PNG in 0.5s; mmproj pipeline functional end-to-end). **verify-stress 7/7 PASS** (`sapphire iguana 19` @ 9.8K, `crimson falcon 18` @ 29.3K, all 7 rungs clean at the 49K ctx). TPS being ~10% higher than the no-vision Config A above isn't strictly A/B-controlled across GPUs — Config A was on GPU 0 after ~2 hrs continuous load; this was a fresh GPU 1; plausibly KV-cache-locality (49K pool 2.6× smaller than 131K) + thermal-state. [PR #166](https://github.com/noonghunna/club-3090/pull/166). | +| `ik-llama/iq4ks-mtp` ⭐ | @noonghunna (1× 3090, set+readback 370 W) | ubergarm MTP-IQ4_KS (`ubergarm/Qwen3.6-27B-GGUF`) | **200K** | **59.67 / 68.78** (decode 60.39 / 72.40, n=3, CV 1.5% / 0.6%) | **1109** | ~22.4 GB | 2026-05-23 | **ik_llama.cpp `cu13-server` + IQK fused kernels + q4_0 KV + MTP n=2 + `-khad`/`-vhad` + native template, thinking-off. ~18–20% FASTER decode TPS than shipped `llamacpp/mtp` Q4_K_M** (49.69 / 57.50 wall, same rig + 370 W cap) — verified by a **set-and-readback power-cap A/B**: ik leads at *both* 230 W and 370 W (both llama.cpp engines are power-sensitive on this rig — ik −29% / mainline −42% from 370→230 W; comparison only valid at matched power). 8-pack **101/150** (≈ tie, mainline 100), **soak PASS** (p50 71.5, 0 err, 0/100 silent-empty, 0 MiB growth), verify-stress 8/8 incl. ceiling-ladder fill to 183K @ 200K, aider-polyglot-30 19/30. **Corrects the [#184](https://github.com/noonghunna/club-3090/discussions/184) "tie"** — that was a wrong-engine measurement artifact: its "ik ~50/58" is exactly mainline@370, a number ik produces at *no* power setting (ik = 42/51 @ 230 W, 60/72 @ 370 W). | | llama.cpp PR [#22673](https://github.com/ggml-org/llama.cpp/pull/22673) MTP, custom build (`Qwen3.6-27B-MTP-Q4_K_M-GGUF` + `--spec-type mtp --spec-draft-n-max 3`) | @efschu (**2× Tesla V100-SXM2-16GB**, Xeon Gold 6154, Debian 13, custom-built llama-server docker) | Q4_K_M MTP | 100K | **49.96 / 62.46** | — | 15.6 GB/card (15,596 MiB at 100K ctx) | 2026-05-06 | **First V100 (sm_70 Volta) cross-rig data on the matrix** — only non-3090/4090/5090 GPU class tested. vLLM blocked (V100=CC 7.0, vLLM needs ≥7.5); fell back to llama.cpp via am17an's PR #22673 with a custom-built docker. **All 7 stress checks PASS including 90K NIAH** (Cliff 2 territory). 2× cards via tensor split (`-sm tensor`). MTP n=3, accept rates not in log. ~80 W/card (V100 max 300 W). [Issue #80](https://github.com/noonghunna/club-3090/issues/80). | | llama.cpp PR [#22673](https://github.com/ggml-org/llama.cpp/pull/22673) MTP, host build (`havenoammo/Qwen3.6-27B-MTP-UD-GGUF` + `--spec-type mtp --spec-draft-n-max 3` + q4_0 KV) | @lamentofhighborne (1× RTX 3090, PCIe x8, 350W) | UD-Q4_K_XL + Q8_0 MTP head | **131K** | **47.12 / 60.42** | — | ~23.1 GiB | 2026-05-07 | **First 1× 3090 llama.cpp MTP data point** on Qwen3.6-27B. Decode 47.60 / 61.71 TPS, TTFT 212 / 194 ms. **`verify-full-mtp.sh` PASS 8/8** (locally-adapted), **`verify-stress-mtp.sh` PASS 7/7 including 91K needle at 131K ctx** — pushes the documented llama.cpp MTP ctx ceiling from ~64-80K (q8_0 KV) to 131K (q4_0 KV). MTP acceptance 78.7%; recurrent 65-layer bug from froggeric's earlier MTP GGUF did **NOT** reproduce on havenoammo's UD GGUF. Native host build (no Docker), surfaced engine-coupling shortcomings in our verify/soak harness — see [Issue #85](https://github.com/noonghunna/club-3090/issues/85). | | llama.cpp PR [#22673](https://github.com/ggml-org/llama.cpp/pull/22673) MTP, host build (`froggeric/Qwen3.6-27B-MTP-GGUF` + `--spec-type mtp --spec-draft-n-max 3` + q4_0 KV) | @lamentofhighborne (1× RTX 3090, PCIe x8, 350 W) | Q4_K_M MTP | **164K** | **47.49 / 55.09** | — | ~22.2 GiB | 2026-05-07 | **Second 1× 3090 llama.cpp MTP data point on same rig** — froggeric's Q4_K_M MTP GGUF vs havenoammo's UD-Q4_K_XL above. Decode 47.91 / 55.81 TPS, TTFT 96 / 98 ms. `verify-full-mtp.sh` PASS 8/8, `verify-stress-mtp.sh` PASS 7/7 incl. 91K needle at 164K ctx. Functional MTP acceptance **86.7%**; canonical acceptance 55.3% narr / 71.2% code. **Ctx-fit ladder**: 262K OOMed MTP, 229K served without MTP, 196K initialized MTP but daemon died at 90K stress; 164K was the stable stress-passing ceiling on this rig. **Beats havenoammo on narr (47.49 vs 47.12, +0.8%) and ctx ceiling (164K vs 131K) but trails on code (55.09 vs 60.42, −9%)**. Manual long-context needles also passed at **120K** (39.39 decode TPS, 81% MTP accept) and **150K** (35.44 decode TPS, 80% MTP accept). MTP+vision incompat (per froggeric's model card); separate no-MTP+vision path passed 65K and 150K. [Issue #94](https://github.com/noonghunna/club-3090/issues/94). | diff --git a/docs/SINGLE_CARD.md b/docs/SINGLE_CARD.md index 6f770bca..16275856 100644 --- a/docs/SINGLE_CARD.md +++ b/docs/SINGLE_CARD.md @@ -42,7 +42,7 @@ For workloads that **don't** accumulate context across turns (single-shot RAG, s | **Bulletproof, no cliffs** (production service, unpredictable inputs) | [`llamacpp/default`](../models/qwen3.6-27b/llama-cpp/compose/single/mtp.yml) (alias of `llamacpp/mtp`) | **262K** (via `-ub 512`) | 52 / 61 | ~23 GB | | **llama.cpp + MTP, fast + long ctx** ⭐ (IDE agents, opencode, Hermes, long-multi-turn agentic) | [`llamacpp/mtp`](../models/qwen3.6-27b/llama-cpp/compose/single/mtp.yml) | **131K** | **51 / 60** | ~22.5 GB | | **llama.cpp + MTP + vision** (multimodal chat, screenshot-debugging, vision-aware review) | [`llamacpp/mtp-vision`](../models/qwen3.6-27b/llama-cpp/compose/single/mtp-vision.yml) | **49K** | **57 / 66** | ~20.5 GB | -| **ik_llama + IQ4_KS + MTP** (best quality-per-bit GGUF + leanest VRAM; advanced-quant track) | [`iq4ks-mtp`](../models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp.yml) | **262K** | ~50 / ~58 | **~22 GB** (leanest) | +| **ik_llama + IQ4_KS + MTP** ⭐ (fastest single-card + leanest VRAM; advanced-quant track) | [`iq4ks-mtp`](../models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp.yml) | **262K** | **~60 / ~69** | **~22 GB** (leanest) | | **ik_llama + IQ4_KS + MTP + vision** | [`iq4ks-mtp-vision`](../models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp-vision.yml) | **160K** | TBD | ~21 GB | | **ik_llama + two-stage spec-dec** 🧪 (ngram+MTP, code-optimized, experimental) | [`iq4ks-two-stage`](../models/qwen3.6-27b/ik-llama/compose/single/iq4ks-two-stage.yml) | **131K** | TBD | ~22 GB | | **Small-context vLLM safe path** ([@stiggy2k16](https://github.com/noonghunna/club-3090/issues/43) data point) — IDE agents capped at <60K accumulated, when you need vLLM speed but llama.cpp is too slow. ⚠️ Genesis-free, but its default pin is purged (#167) — run it with `VLLM_IMAGE=vllm/vllm-openai:latest` until the pin's bumped | [`minimal.yml`](../models/qwen3.6-27b/vllm/compose/single/minimal.yml) at `--gpu-memory-utilization 0.95 --max-model-len 65536` | **64K** | ~32 / ~33 (no MTP) | ~22.4 GB | @@ -142,7 +142,7 @@ For the cross-card TP=2 picture, see [`DUAL_CARD.md`](DUAL_CARD.md). **Workload:** best quality-per-bit GGUF on a single 3090. Same cliff-immunity as llama.cpp (same ggml memory model), but with fork-exclusive IQK imatrix quants that beat mainline Q4_K_M on perplexity. -ubergarm `Qwen3.6-27B-MTP-IQ4_KS.gguf` (IQK imatrix quant, built-in MTP head) + q4_0 KV + `-khad`/`-vhad` (Hadamard K+V cache transforms) + MTP `n=2` + `--merge-qkv` + `--parallel-tool-calls` (ik-exclusive). **262K context** on one 3090. At matched power it **ties `llamacpp/mtp`** on TPS (~50 narr / ~58 code) and quality (8-pack 103 vs 102) — the earlier "+18–20%" was a power/card artifact ([#184](https://github.com/noonghunna/club-3090/discussions/184)). Its real edge is a **~0.5–0.8 GB leaner footprint**, which makes it the pick when VRAM is tight (sub-24 GB, shared GPU, WSL). Engine: `ghcr.io/ikawrakow/ik-llama-cpp:cu13-server`. See [`docs/engines/IK_LLAMA.md`](engines/IK_LLAMA.md) for the full deep dive. +ubergarm `Qwen3.6-27B-MTP-IQ4_KS.gguf` (IQK imatrix quant, built-in MTP head) + q4_0 KV + `-khad`/`-vhad` (Hadamard K+V cache transforms) + MTP `n=2` + `--merge-qkv` + `--parallel-tool-calls` (ik-exclusive). **262K context** on one 3090. At matched 370 W it's **~18–20% faster than `llamacpp/mtp`** on TPS (~60 narr / ~69 code vs ~50 / ~58), quality-tied (8-pack 103 vs 102), and **~0.5–0.8 GB leaner** — the faster *and* leaner single-card path. (A 2026-05-22 "tie" was a wrong-engine measurement artifact — corrected 2026-05-23 via a power-cap-controlled A/B, [#184](https://github.com/noonghunna/club-3090/discussions/184).) Engine: `ghcr.io/ikawrakow/ik-llama-cpp:cu13-server`. See [`docs/engines/IK_LLAMA.md`](engines/IK_LLAMA.md) for the full deep dive. ### ik_llama + IQ4_KS + MTP + vision — `iq4ks-mtp-vision.yml` diff --git a/docs/engines/IK_LLAMA.md b/docs/engines/IK_LLAMA.md index 050bc4d9..7b439515 100644 --- a/docs/engines/IK_LLAMA.md +++ b/docs/engines/IK_LLAMA.md @@ -2,7 +2,7 @@ **Role on this stack:** the engine you reach for when you want **newer, higher-quality-per-bit quants** than mainline llama.cpp ships — specifically the **IQK imatrix family** (`IQ4_KS`, `IQ5_KS`, …) that exists *only* in this fork. It's a llama.cpp fork (ikawrakow), so it inherits llama.cpp's cliff-immune memory model and broad hardware support, then adds a co-designed quant + kernel stack on top. -> **In one line:** llama.cpp's robustness + fork-exclusive IQK quants + fused CUDA kernels → **MTP, clean to 262K on one card, quality on par with `llamacpp/mtp`** (8-pack 103 vs 102), at a **~0.5–0.8 GB leaner VRAM footprint**. On TPS it's a **tie** with `llamacpp/mtp` at matched power (~50 narr / ~58 code on a 3090); the leaner footprint — not speed — is the reason to pick it (it shines when VRAM-tight: sub-24 GB, shared GPU, WSL). Full matched-power write-up: [discussions/184](https://github.com/noonghunna/club-3090/discussions/184). +> **In one line:** llama.cpp's robustness + fork-exclusive IQK quants + fused CUDA kernels → **MTP, clean to 262K on one card, quality on par with `llamacpp/mtp`** (8-pack 103 vs 102), at a **~0.5–0.8 GB leaner VRAM footprint**. It's also **~18–20% faster** than `llamacpp/mtp` on decode TPS at matched 370 W (~60 narr / ~69 code vs ~50 / ~58 on a 3090) — so it's the faster *and* leaner single-card path. The trade is a second engine to maintain + the IQK quant. Full matched-power write-up: [discussions/184](https://github.com/noonghunna/club-3090/discussions/184). For *what the quants actually are* and how IQK compares to k-quants / i-quants / AWQ, see **[../QUANTIZATION.md](../QUANTIZATION.md)**. For the cross-engine overview see **[../INFERENCE_ENGINES.md](../INFERENCE_ENGINES.md)**. @@ -153,14 +153,14 @@ ik_llama has several quality-relevant flags beyond what the default composes use | Metric | Value | vs shipped `llamacpp/mtp` (Q4_K_M) | |---|---|---| -| Decode TPS (narr / code) | ~50 / ~58 | **tie** at matched 370 W (the earlier "+18–20%" was a power/card artifact — see [#184](https://github.com/noonghunna/club-3090/discussions/184)) | +| Decode TPS (narr / code) | **~60 / ~72** | **~18–20% faster** than `llamacpp/mtp` (~50 / ~59) at matched 370 W — confirmed by a power-cap-controlled A/B (ik leads at 230 W *and* 370 W); the earlier "tie" was a wrong-engine measurement artifact, see [#184](https://github.com/noonghunna/club-3090/discussions/184) | | Max context (1× 3090) | **262K** (q4_0 KV) | tie (mainline also reaches 262K via `-ub 512`) | -| VRAM @ 262K | ~22.5 GB | **~0.5–0.8 GB leaner** ← ik's one real edge | +| VRAM @ 262K | ~22.5 GB | **~0.5–0.8 GB leaner** (a second edge, alongside the TPS lead) | | verify-stress | 7/7 (incl. 91K Cliff 2 needle) | parity | | Quality 8-pack | **103/150** | ≈ tie (mainline 102) | | toolcall-15 | 60% (native template) | tie — native won the A/B on **both** engines | -> Bench: canonical prompt, 3 warmup + measured runs, **verified 370 W same-card matched comparison**, q4_0 KV / MTP n=2 / thinking-off. The honest finding is a TPS/quality/context tie — ik's edge is the leaner footprint, useful when VRAM-tight. See [#184](https://github.com/noonghunna/club-3090/discussions/184) + [../../BENCHMARKS.md](../../BENCHMARKS.md). +> Bench: canonical prompt, 3 warmup + measured runs, **set-and-readback 370 W same-card matched comparison**, q4_0 KV / MTP n=2 / thinking-off. The honest finding: **ik is ~18–20% faster on decode TPS** (quality + context tied, ~0.5–0.8 GB leaner) — faster *and* leaner. The 2026-05-22 "tie" was a wrong-engine measurement artifact (its "tie" number is exactly mainline@370, which ik produces at no power setting); corrected 2026-05-23 via a power-cap-controlled A/B + 5 independent ik runs all at ~70 code. See [#184](https://github.com/noonghunna/club-3090/discussions/184) + [../../BENCHMARKS.md](../../BENCHMARKS.md). > > **Two-stage (ngram+MTP):** not yet benched. Compose exists (`iq4ks-two-stage.yml`), PR #1789 merged 2026-05-15. Expected to outperform MTP-only on code workloads with repeated patterns; bench pending. diff --git a/models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp.yml b/models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp.yml index 7a60965e..3db9e8dc 100644 --- a/models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp.yml +++ b/models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp.yml @@ -29,11 +29,12 @@ # → 21.5 GB if you need the extra ~1 GB back. # Status: ✅ Shipped (advanced-quant track, #180). In the wizard + # `bash scripts/switch.sh ik-llama/iq4ks-mtp`. -# Perf: Ties shipped llamacpp/mtp Q4_K_M on TPS (~50 narr / ~58 code at a -# verified, matched 370 W on the same card) AND 8-pack quality -# (103 vs 102) — the earlier "+18-20%" was a power/card artifact -# (see discussions/184). ik's real edge is a ~0.5-0.8 GB leaner -# VRAM footprint. +# Perf: ~18-20% FASTER decode TPS than shipped llamacpp/mtp Q4_K_M +# (~60 narr / ~69 code vs ~50/58, at set-and-readback matched 370 W +# same card), quality-tied (8-pack 103 vs 102), AND ~0.5-0.8 GB +# leaner VRAM — faster AND leaner. Confirmed by a power-cap A/B + +# 5 ik runs (all ~70 code). The 2026-05-22 "tie" was a wrong-engine +# measurement artifact (its tie number = mainline@370). See discussions/184. # Best for: VRAM-tight single-card (sub-24 GB, shared GPU, WSL display tax) # where the leaner footprint matters; or if you want IQK quants. # ---------------------------------------------------------------------------