# Benchmarks — measured numbers, by model

This file is the consolidated cross-rig table for every compose variant we
ship, with **measured** numbers (not derived estimates). It's intentionally
**append-friendly** — every row carries an explicit `Rig` cell so multiple
contributors can publish numbers for the same compose without rewriting the
file.

Rows land here:
- when a contributor opens a PR adding a new compose variant, OR
- when a contributor supplies canonical bench output via the [Numbers from your rig](https://github.com/noonghunna/club-3090/issues/new?template=numbers-from-your-rig.yml) issue template.

Per-model qualitative findings, framework comparisons, and "why we picked
this quant" rationale live in `models/<model>/INTERNALS.md` (or the local
`learnings/` tree). This file is **just the numbers, anchored to (rig, date)**.

---

## Canonical bench

All `Narr / Code TPS` rows come from `bash scripts/bench.sh`, which runs:

> **Narrative:** "Write a detailed 800-word essay explaining transformer attention." (`max_tokens=1000`)
>
> **Code:** "Write a Python implementation of quicksort with comments explaining each step." (`max_tokens=800`)
>
> Sampling: `temperature=0.6, top_p=0.95, top_k=20, presence_penalty=0.0, enable_thinking=false`. Three warmups + five measured runs per prompt. Mean wall TPS reported.
>
> **Quality (8-pack) sampling:** behavioral packs run at **temperature 0** (greedy, reproducible) by default — the canonical bar for cross-config / cross-model ranking. To score a model at its *recommended* temp instead, the composes ship per-model sampling defaults (Qwen3.6 0.6 / Qwopus3.6 0.8 / Gemma 1.0, env-overridable) and the eval inherits them via `quality-test.sh --sampling-from-server` (tagged non-canonical). See [QUALITY_TEST.md → Sampling & temperature](docs/QUALITY_TEST.md#sampling--temperature).

`PP tok/s` is prompt-processing throughput. For vLLM rows, `bench.sh` scrapes
the most recent `Avg prompt throughput` lines from container logs. For
llama.cpp or engines without that log shape, run `PP=1 bash scripts/bench.sh`
to add a single long-prompt fallback probe that computes prompt tokens over
TTFT. Existing rows show `—` until re-benched with v0.7.1+ tooling.

Cross-rig numbers are comparable because the prompt + sampling are pinned. Variations against your rig usually trace back to power caps, PCIe lane counts, or pin (vLLM image SHA / Genesis commit) — see [`scripts/report.sh`](scripts/report.sh) which captures all three.

## How to add a row for your rig

1. Run `bash scripts/report.sh --full > my-rig.md` — captures hardware (incl. power caps + PCIe lanes), stack version (vLLM image SHA, Genesis commit), verify-full + verify-stress + **SOAK_MODE=continuous** + canonical bench numbers in one ~35-min pass. (Or `--bench` for the fast subset; soak-continuous catches Cliff 2b which the others don't.)
2. Open the [Numbers from your rig](https://github.com/noonghunna/club-3090/issues/new?template=numbers-from-your-rig.yml) issue template, paste the report, mention which compose variant you ran.
3. We'll append your numbers as a row in the appropriate table here, with `Rig` cell formatted `@your-handle (rig-shape)` — e.g. `@whamp (4× 3090 PCIe x4/x8/x16/x16, 300 W)`.

If the same compose has multiple rig rows showing different numbers, that's a feature — it tells future readers what's portable vs rig-specific.

### Notes column convention — soak verdict

Every row's Notes cell should start (or include) an explicit soak verdict so readers can grep at a glance:

- `Soak: ✓ PASS` — clean (no errors, 0 silent-empty turns, <200 MiB VRAM growth across 5×5 continuous sessions)
- `Soak: ⚠ borderline` — within thresholds but worth flagging (e.g. 240 MiB growth, slow turns >30s on x4 PCIe, etc.)
- `Soak: ✗ FAIL` — Cliff 2b suspect, silent-empty turns, mid-soak errors, or growth-threshold overshoot
- `Soak: —` — not run (acceptable but discouraged; row is "bench-only" until a follow-up brings the soak verdict)

Soak-continuous is the only signal that catches Cliff 2b. If your row says `Soak: —`, future readers will assume the worst; if your row says `Soak: ✓ PASS`, that's the 30-second answer to "is this rig+compose stable for hermes/openhands traffic." Use `bash scripts/soak-test.sh --continuous` (~25 min, auto-detects endpoint + container) to add the verdict if you initially submitted bench-only.

---

## Qwen3.6-27B

Primary serving model. Hybrid Qwen3-Next architecture (DeltaNet GDN + standard attention). Quants used: AutoRound INT4 (vLLM), Unsloth Q5_K_XL GGUF (llama.cpp).

### Single-card (1× RTX 3090) — vLLM

> ⚠️ **Cliff 2b open on `long-text*` / `long-vision` (2026-05-05)** — Genesis v7.72.2's PN59 streaming-GDN orchestrator doesn't engage on the chunked-prefill path 24 GB single-card configs are forced to take. Single-prompt prefill at >~50K may OOM. Filed at [Sandermage/genesis-vllm-patches#22](https://github.com/Sandermage/genesis-vllm-patches/issues/22). **Safe single-card paths**: `llamacpp/default` (no Cliff 2b) or single-prompt context capped at <50K. **TP=2 paths escape the cliff** entirely (see Dual-card section).

| Compose | Rig | KV | Max ctx | Narr / Code TPS | PP tok/s | Peak VRAM | Date | Notes |
| --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- |
| `minimal.yml` (`mem-util 0.95 max-model-len 65536`) | @noonghunna (1× 3090, x16, 350 W) | TQ3 | 64K | ~32 / ~33 | — | ~22.4 GB | 2026-05-03 | no MTP. [stiggy2k16](https://github.com/noonghunna/club-3090/issues/43) cross-rig data point — short-prompt vLLM-safe path when llama.cpp is too slow. |
| `long-vision.yml` | @noonghunna (1× 3090) | TQ3 | 145K | 50 / 66 | — | ~23.0 GB | 2026-04-30 | vision + tools + thinking. mem-util 0.95. |
| `long-text.yml` ⭐ | @noonghunna (1× 3090) | TQ3 | 180K | 50 / 67 | — | ~22.3 GB | 2026-04-30 | text-only (vision tower dropped). MTP n=3. mem-util 0.93. **Default for RAG / IDE agents below 25K accumulated ctx**. |
| `long-text.yml` | @laurimyllari (1× **4090**, AMD Ryzen 7 7800X3D, 230W cap) | TQ3 | **90K** (forced by KV-pool fit on 4090 — see Notes) | **102.96 / 103.09** | — | ~23.7 GB | 2026-05-05 | **First 4090 single-card vLLM bench** on club-3090. **Required `max-model-len` drop from 180K→90K** at default mem-util 0.92 (KV cache budget on his 24 GB 4090 is tighter than the 3090s the compose was calibrated against — likely 4090 driver/desktop overhead consumes more idle VRAM). MTP n=3 active, AL 3.34-3.45 narr / per-pos accept 92-95% / 79-84% / 62-67%. CV 2.2%/2.2%. Verify-stress hit Cliff 2b OOM at long-vision 50 MiB (sidesteps via long-text). [Issue #71](https://github.com/noonghunna/club-3090/issues/71) + [disc #62](https://github.com/noonghunna/club-3090/discussions/62#discussioncomment-16821619). |
| `long-text-no-mtp.yml` | @noonghunna (1× 3090) | TQ3 | 200K | TBD | — | ~21.0 GB | — | max-context single-shot, no MTP. Slow decode but biggest ctx window. |
| `bounded-thinking.yml` | @noonghunna (1× 3090) | TQ3 | 180K | 50 / 66 | — | ~21.7 GB | 2026-05-04 | structured-CoT FSM in reasoning channel; **recommended grammar: DeepSeek scratchpad** (PLAN/NOTE×0-15/VERDICT). Phase 3 final: **93.9% HE+ / 66.0% LCB v6** (87.4% combined, +1 net vs the andthattoo G/A/E baseline). Andthattoo G/A/E grammar also works (94.5% HE+ / 62.0% LCB / 86.9% combined, ~4× tighter think budget — pass via `extra_body`). See [STRUCTURED_COT.md](docs/STRUCTURED_COT.md). |
| `tools-text.yml` | @noonghunna (1× 3090) | fp8 | 75K | TBD | — | TBD | — | IDE-agent path that escapes the long-text Cliff 1 mech B leak (see [#16](https://github.com/noonghunna/club-3090/issues/16)). |
| `dual-dflash.yml`-shape forced TP=1 (DFlash N=5, fp8 KV, mem-util 0.96, custom_all_reduce disabled) | @efschu (1× **RTX 5090** 32 GB, AMD Ryzen 9 5950X, Debian trixie, PCIe x8, 575 W cap) | fp8 | 49K (KV-fit at 0.96 mem-util) | **126.53 / 200.11** (decode 127.98 / 204.80) | — | 31.5 GB | 2026-05-07 | **First single-5090 DFlash data point** on club-3090. AutoRound INT4 weights + DFlash N=5 draft. CV 3.0%/2.0%. **Code TPS 200 is the highest single-card number measured on the matrix** — beats single-3090 (50/67 long-text) by ~3× on code, single-4090 (102/103 at 90K) by ~2× code. Trade is ctx ceiling: 49K vs 90K-180K on 24 GB cards, due to KV-pool fit at fp8 + 32 GB total VRAM. vLLM `nightly-01d4d1ad3` (post-v7.72.2 uplift). [Issue #93](https://github.com/noonghunna/club-3090/issues/93). |
| `vllm/default` (single, MAX_MODEL_LEN=48000, mem-util 0.92, MTP n=3) | @ygafarov (1× 3090 via **oculink eGPU on PCIe x4**, AMD Ryzen AI MAX+ 395 / Strix Halo miniPC, CachyOS, 124 GB RAM, 290W cap) | TQ3 | 48K | **68.86 / 91.70** (decode 69.27 / 92.76) | — | 23.6 GB | 2026-05-09 | **Soak: ⚠ borderline** (VRAM grew 240 MiB > 200 MiB threshold, 3 turns >30s; 100% TPS retention + 0 errors + 0 silent-empty turns — x4-PCIe accretion + bus-latency under prefill, not a leak. Threshold may need an "eGPU bus class" allowance.) **First Strix-Halo-miniPC + oculink-eGPU bench** on club-3090. Single 3090 over PCIe x4 (oculink) instead of x8/x16 internal. CV 1.6%/2.7%. MTP AL 3.31, accept 76.9% (per-pos 0.918/0.772/0.616). `scheduler_reserve_full_isl=False`. Driver 595.71.05 (very new, CUDA 13.2). vLLM `nightly-01d4d1ad3`. [Issue #113](https://github.com/noonghunna/club-3090/issues/113). |

### Single-card (1× RTX 3090) — llama.cpp

> ℹ️ **4090 ctx-derate.** A 24 GB 4090 carries more idle desktop + driver VRAM than a headless 3090, so single-card context ceilings land ~15–20% lower. Start below the 3090 number and confirm headroom with `verify-stress.sh` (watch its ceiling VRAM-margin line). Observed across engines: ik two-stage 200K→**160K**, `long-text` 180K→90K, `dual-dflash-noviz` 200K→180K.

| Compose | Rig | Quant | Max ctx | Narr / Code TPS | PP tok/s | Peak VRAM | Date | Notes |
| --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- |
| `llamacpp/default` | @noonghunna (1× 3090) | Unsloth Q5_K_XL | 262K | 21 / 21 | — | ~20 GB | 2026-04-21 | bulletproof — different engine, different memory allocator, no Cliff 1 / Cliff 2. Slow decode but cliff-immune. |
| `llamacpp/mtp` ⭐ (alias `llamacpp/default`) | @noonghunna (1× 3090) | Unsloth Q4_K_M MTP (`unsloth/Qwen3.6-27B-MTP-GGUF`) | **200K** | **50.27 / 58.92** (decode, n=3, CV 0.6% / 1.4%; wall 49.69 / 57.50) | **1025** | ~22.4 GB (~1.6 GB headroom) | 2026-05-23 | **Mainline llama.cpp `server-cuda-b9246` (PR #22673 MTP merged)** + MTP `n=2` + q4_0 KV + native template + `--reasoning off`, at the **200K default (`-ub 512`)** — fills ~183K with ~1.1 GB margin; `CTX_SIZE=131072 UBATCH_SIZE=1024` is the faster-prefill override (~51 / 60 decode). **verify-stress 8/8 PASS** incl. the ceiling-ladder fill to 183K @ 200K — the Cliff 2 single-prompt narrative is **config-driven, not architectural**. **Quality 8-pack 100/150 (67%)** — on par with the best Qwen vLLM-dual configs in [#119](https://github.com/noonghunna/club-3090/discussions/119). **Aider-polyglot-30 22/30 (73%)** — per-lang cpp 4/5, go 4/5, java 3/5, js 5/5, python 3/5, rust 3/5. **Per-GPU code TPS (58.92) ≈ vLLM-dual per-card (60–63)** — vLLM dual's aggregate edge is purely the second card. Zero patches, no Genesis, no AutoRound. 200K thinking-off rebench; [PR #166](https://github.com/noonghunna/club-3090/pull/166) · [#201](https://github.com/noonghunna/club-3090/pull/201). |
| `llamacpp/qwen27b-pi-reasoning-single` 🧪 | @noonghunna (1× 3090, 370 W) | bytkim Q4_K_M MTP **pi-reasoning** ([`bytkim/Qwen3.6-27B-MTP-pi-reasoning-GGUF`](https://huggingface.co/bytkim/Qwen3.6-27B-MTP-pi-reasoning-GGUF)) | **~188K** (200K-alloc) | **47.9 / 55.3** (decode, n=5, CV <2%; wall 47.4 / 54.2; thinking-off) | **1030** | ~22.7 GB | 2026-06-18 | **Mainline llama.cpp `server-cuda-b9246` + MTP `n=2` + q4_0/q4_0 KV**, config per the model card (temp 1.0 / top-k 0, **reasoning-ON**). A community "Pi-style" reasoning/coding-agent fine-tune. **MTP head ≡ base `llamacpp/mtp`** — a matched-power A/B put the two **dead-even** (73% vs 72% draft accept, identical decode); 47.9/55.3 here is ~on par with base 50.27/58.92 (within ~5% on the canonical prompts). ⚠️ **Power-sensitive**: ~28.5 / 32.9 under a 230 W cap (mainline −42% 370→230 W). **8-pack 104/150 think-off · 106/150 think-on**; **verify-stress 8/8** (NIAH→183K @ 91% of the 200K pool); **soak PASS** (0 MiB growth, 0/100 silent-empty, p50 54.5, 102% retention). Context: 200K-alloc fills ~188K usable — 262K-alloc OOMs ~176K (FA scratch grows with alloc), full 262K is beellama-only; author tested only 128K (`CTX_SIZE=131072` for the strict card window). Verbose even thinking-off → give it generous `max_tokens` (a few deterministic-pack misses were `finish_reason=length`). 🧪 experimental / opt-in (`--force`). [PR #425](https://github.com/noonghunna/club-3090/pull/425). |
| `llamacpp/mtp-vision` (2026-05-20; superseded by re-tune ↓) | @noonghunna (1× 3090) | Unsloth Q4_K_M MTP + mmproj-F16 | **49K** | **56.52 / 66.17** (decode, n=5, CV 1.6% / 1.7%) | **1158** | ~20.5 GB (~3.5 GB headroom) | 2026-05-20 | **First stack profile combining MTP + vision** (sweep-verified on build 9235; the older "strip mmproj when MTP" rule was obsolete). Same Q4_K_M MTP GGUF + q4_0 KV + MTP `n=2` + `-ub 1024` + `--mmproj mmproj-F16.gguf` (vision projector loaded). **Multimodal probe ✅** (model answered "Red" on a synthetic 64×64 red PNG in 0.5s; mmproj pipeline functional end-to-end). **verify-stress 8/8 PASS** (`sapphire iguana 19` @ 9.8K, `crimson falcon 18` @ 29.3K, all 7 rungs clean at the 49K ctx). TPS being ~10% higher than the no-vision Config A above isn't strictly A/B-controlled across GPUs — Config A was on GPU 0 after ~2 hrs continuous load; this was a fresh GPU 1; plausibly KV-cache-locality (49K pool 2.6× smaller than 131K) + thermal-state. [PR #166](https://github.com/noonghunna/club-3090/pull/166). |
| `llamacpp/mtp-vision` (re-tuned ⭐) | @noonghunna (1× 3090, 370 W) | Unsloth Q4_K_M MTP + mmproj-F16 @ **1 M-px** (`IMAGE_MAX_TOKENS=1024`) | **150K** (fills ~138K) | — (decode ≈ 56 / 66, carried from the 49K row — ctx-independent, not re-benched) | — | ~22.5 GB boot · **561 MB free @ 138K ceiling** | 2026-05-25 | **Vision re-tune ([PR #227](https://github.com/noonghunna/club-3090/pull/227), #437).** The 49K row ↑ and the later 160K compose default were never validated under the current `verify-stress [8/8]` ceiling ladder. Re-laddered: **full-res 4M-px (`IMAGE_MAX_TOKENS=4096`) OOM-crashes at 160K** under the agentic battery; the old "196K @ `-ub 512`" claim **walls at ~125K fill**. New measured-safe default = **150K @ 1M-px**: completes `[8/8]`, recalls to 138K (91%), 561 MB free at ceiling, no crash. Full-res 4M-px = documented override (lower ctx). >150K *context* → text-only `llamacpp/mtp`. n=1. |
| `ik-llama/iq4ks-mtp` ⭐ | @noonghunna (1× 3090, set+readback 370 W) | ubergarm MTP-IQ4_KS (`ubergarm/Qwen3.6-27B-GGUF`) | **200K** | **59.67 / 68.78** (decode 60.39 / 72.40, n=3, CV 1.5% / 0.6%) | **1109** | ~22.4 GB | 2026-05-23 | **ik_llama.cpp `cu13-server` + IQK fused kernels + q4_0 KV + MTP n=2 + `-khad`/`-vhad` + native template, thinking-off. ~18–20% FASTER decode TPS than shipped `llamacpp/mtp` Q4_K_M** (49.69 / 57.50 wall, same rig + 370 W cap) — verified by a **set-and-readback power-cap A/B**: ik leads at *both* 230 W and 370 W (both llama.cpp engines are power-sensitive on this rig — ik −29% / mainline −42% from 370→230 W; comparison only valid at matched power). 8-pack **101/150** (≈ tie, mainline 100), **soak PASS** (p50 71.5, 0 err, 0/100 silent-empty, 0 MiB growth), verify-stress 8/8 incl. ceiling-ladder fill to 183K @ 200K, aider-polyglot-30 19/30 (per-lang: cpp 4/5, go 4/5, java 3/5, js 3/5, python 3/5, rust 2/5). **Corrects the [#184](https://github.com/noonghunna/club-3090/discussions/184) "tie"** — that was a wrong-engine measurement artifact: its "ik ~50/58" is exactly mainline@370, a number ik produces at *no* power setting (ik = 42/51 @ 230 W, 60/72 @ 370 W). **Post-[#38](https://github.com/noonghunna/benchlocal-cli/pull/38)-chain rebench (2026-05-28):** 8-pack **107/150 (71%)** (+6 vs 101) — gains from [PR #42](https://github.com/noonghunna/benchlocal-cli/pull/42) deterministic hermes sampler (0→11/20) + [PR #43](https://github.com/noonghunna/benchlocal-cli/pull/43)/[#47](https://github.com/noonghunna/benchlocal-cli/pull/47) cli-40 timeout defaults + honest tags (11→19/40); aider **18/30** (≈ tie, within ±2 single-run noise) — all 30 exercises reached the LLM ([PR #44](https://github.com/noonghunna/benchlocal-cli/pull/44) git-checkout fix). 40-min batch wall completed inside [PR #43](https://github.com/noonghunna/benchlocal-cli/pull/43)'s TPS-aware scaled budget (50 min) — first real run that would have been prematurely killed by the static 30-min default. Numbers above are pre-#38 baseline (preserved for the audit trail); these are the same compose under post-#38 verifiers + sampler. |
| `ik-llama/iq4ks-two-stage` ⭐ (code) | @noonghunna (1× 3090, 370 W) | ubergarm MTP-IQ4_KS (`ubergarm/Qwen3.6-27B-GGUF`) | **200K** | **59.4 / 97.8** (decode, n=3, CV 1.9% / 5.3%) | — | ~21.6 GB | 2026-05-24 | **ik_llama.cpp `cu13-server` TWO-STAGE spec-dec ([PR #1789](https://github.com/ikawrakow/ik_llama.cpp/pull/1789)): ngram-mod n_max=4 + MTP n_max=3**, q4_0 KV + `-khad`/`-vhad` + native template, thinking-off. **Code decode +35% vs `iq4ks-mtp` MTP-only** (97.8 vs 72.40, matched same-rig 370 W); narrative ~tie (59.4 vs 60.39). ngram n_max swept {2,3,4,5,8,16} → inverted-U peaking at **4** (16→81.5, 8→90.4, **4→97.8**, 2→78.8): too-long drafts waste verify compute, too-short stop matching repeated code spans. **Lossless** — 8-pack **98/150** (≈ MTP-only 100), aider-polyglot-30 **16/30** (≈ MTP-only 19), both within single-run noise; verify-full green. verify-stress @ 200K fills to 183K with correct needle recall (ceiling VRAM margin 833 MB is just under the 1024 MB sustained-agent guard → `CTX_SIZE=180224` for heavy agentic-near-ceiling). ~0.8 GB leaner than iq4ks-mtp at idle. Tune [#210](https://github.com/noonghunna/club-3090/pull/210) · ctx 200K + ⭐ [#212](https://github.com/noonghunna/club-3090/pull/212). |
| `ik-llama/iq4ks-mtp-vision` (NEW ⭐) | @noonghunna (1× 3090, 370 W) | ubergarm MTP-IQ4_KS (`ubergarm/Qwen3.6-27B-GGUF`) + mmproj-F16 @ **1 M-px** | **160K** (fills ~147K) | — (decode ≈ 60 / 72, carried from `ik-llama/iq4ks-mtp` ↑) | — | ~22.4 GB boot · **503 MB free @ 147K ceiling** | 2026-05-25 | **Vision re-tune ([PR #227](https://github.com/noonghunna/club-3090/pull/227), #437); graduated EVAL #402 → Production.** Full `verify-stress [8/8]` @ 160K + 1M-px: recalls to 147K (91%), 503 MB free at ceiling, no crash, vision functional (read needle text off a 1024² image). **180K OOMs at fill; 200K leaves only 891 MB after one image** — shrinking `IMAGE_MAX_TOKENS` does NOT buy ctx (vision ceiling gated by KV + fill-time runtime overhead, not the image-encode buffer). ~0.5 GB leaner than the llama vision row → why ik holds 160K where llama lands 150K. Full-res 4M-px = override. n=1 (noisy). |
| `beellama/dflash` ⭐ (single-card default 2026-05-30) | @noonghunna (1× 3090, 370 W) | Unsloth Q5_K_S + Anbeeld DFlash-IQ4_XS draft | **160K** (ships 102K; 200K OOMs on prefill) | **50.2 / 99.7** (decode 50.4 / 101.3, n=5, CV 5.6% / 7.4%) | ~1066 | ~22.1 GB @ 102K | 2026-05-30 | **Promoted to the single-card DEFAULT** (resolver `ENGINE_PREFERENCE[single]` #1 + `caveats` DEFAULTS row). beellama.cpp (Anbeeld llama.cpp fork) DFlash spec-dec + windowed KV, `q5_0`/`q4_1` KV, native template, thinking-off. **Code decode ~100 TPS — fastest single-card 3090 code path** (vs ik 72, mainline 59); narrative ~50 (ik leads at 60). **Output-lossless DFlash** → quality = the Q5_K_S target: **8-pack 107/150 (71%) think-off, 113/150 (75%) think-on** (same-session, [#239](https://github.com/noonghunna/club-3090/discussions/239)). DFlash accept 29–37% (prose/code). **Ceiling ladder (2026-05-30):** 130K comfortable (1.6 GB headroom), 160K usable (115K-tok prefill OK, 0.8 GB), **200K boots but OOMs on prefill**. Served via our **unofficial multi-arch image** `beellama-cpp:multiarch-b9459-07ac3ce` (sm_86/89/120 = 3090/4090/5090; **sm_89/120 compiled-not-validated** — only 3090 verified on-rig). Community fork chain, no official Docker yet (Anbeeld v0.3.0 WIP). |

> **v0.3.0 DFlash-vs-no-spec A/B (2026-06-03, `bench.sh` narrative n=5, measured no-spec controls):** DFlash prose is **net-positive** on v0.3.0 across the board — Qwen single Q5 no-spec **35.9** → DFlash **45.7 (+27%)**; Qwen **dual Q8** no-spec **23.4** → DFlash **35.6 (+52%)**; Gemma single Q4 no-spec **34.8** → DFlash **44.6 (+28%)**. AR ~0.29–0.41 on all three v0.3.0 images (`efe856397`/`e0663be`/`63abcd3`). ⤷ The earlier v0.3.0 "prose-acceptance regression" (~0.07 AR / net-negative) is **RETRACTED** — it was an AR over-read + a wrong no-spec baseline (the ~37 we'd used was never the real dual-Q8 no-spec). New adaptive-DM HEAD `63abcd3` is neutral. See `learnings/qwen3.6-27b.md` 2026-06-03 + [docs/UPSTREAM.md](docs/UPSTREAM.md).

| `ik-llama/iq4ks-two-stage` (1× **4090**) | @laurimyllari (1× **4090** 24 GB, AMD Ryzen 7 7800X3D, 450 W) | ubergarm MTP-IQ4_KS (`ubergarm/Qwen3.6-27B-GGUF`) | **160K** (derate from 200K) | **82.5 / 120.9** (decode, n=5, CV 4.4% / 7.9%; wall 81.0 / 114.2) | **2311** | ~22.8 GB boot (21.5 GB at bench) | 2026-05-24 | **First 4090 ik-two-stage cross-rig.** Required **ctx derate 200K→160K** — OOM at 200K (a 24 GB 4090's idle desktop+driver VRAM is tighter than a headless 3090; matches his own `long-text` 180K→90K + snoby `dual-dflash-noviz` 200K→180K). **+39% narr / +24% code vs the 3090 two-stage** (59.4 / 97.8) — newer silicon + higher PL. **verify-stress 8/8 PASS**, **soak PASS** (silent_empty 0/100, p50 112.65, 0 MiB growth, retention 100.9%). ⚠ 8-pack 101/150 + aider-polyglot 17/30 were measured **pre-[PR #31 grader fix](https://github.com/noonghunna/benchlocal-cli/pull/31)** (toolcall/reasonmath false-negatives) → under-count true capability; re-measure pending. [disc #184](https://github.com/noonghunna/club-3090/discussions/184#discussioncomment-17042752). |
| llama.cpp PR [#22673](https://github.com/ggml-org/llama.cpp/pull/22673) MTP, custom build (`Qwen3.6-27B-MTP-Q4_K_M-GGUF` + `--spec-type mtp --spec-draft-n-max 3`) | @efschu (**2× Tesla V100-SXM2-16GB**, Xeon Gold 6154, Debian 13, custom-built llama-server docker) | Q4_K_M MTP | 100K | **49.96 / 62.46** | — | 15.6 GB/card (15,596 MiB at 100K ctx) | 2026-05-06 | **First V100 (sm_70 Volta) cross-rig data on the matrix** — only non-3090/4090/5090 GPU class tested. vLLM blocked (V100=CC 7.0, vLLM needs ≥7.5); fell back to llama.cpp via am17an's PR #22673 with a custom-built docker. **All 7 stress checks PASS including 90K NIAH** (Cliff 2 territory). 2× cards via tensor split (`-sm tensor`). MTP n=3, accept rates not in log. ~80 W/card (V100 max 300 W). [Issue #80](https://github.com/noonghunna/club-3090/issues/80). |
| llama.cpp PR [#22673](https://github.com/ggml-org/llama.cpp/pull/22673) MTP, host build (`havenoammo/Qwen3.6-27B-MTP-UD-GGUF` + `--spec-type mtp --spec-draft-n-max 3` + q4_0 KV) | @lamentofhighborne (1× RTX 3090, PCIe x8, 350W) | UD-Q4_K_XL + Q8_0 MTP head | **131K** | **47.12 / 60.42** | — | ~23.1 GiB | 2026-05-07 | **First 1× 3090 llama.cpp MTP data point** on Qwen3.6-27B. Decode 47.60 / 61.71 TPS, TTFT 212 / 194 ms. **`verify-full-mtp.sh` PASS 8/8** (locally-adapted), **`verify-stress-mtp.sh` PASS 7/7 including 91K needle at 131K ctx** — pushes the documented llama.cpp MTP ctx ceiling from ~64-80K (q8_0 KV) to 131K (q4_0 KV). MTP acceptance 78.7%; recurrent 65-layer bug from froggeric's earlier MTP GGUF did **NOT** reproduce on havenoammo's UD GGUF. Native host build (no Docker), surfaced engine-coupling shortcomings in our verify/soak harness — see [Issue #85](https://github.com/noonghunna/club-3090/issues/85). |
| llama.cpp PR [#22673](https://github.com/ggml-org/llama.cpp/pull/22673) MTP, host build (`froggeric/Qwen3.6-27B-MTP-GGUF` + `--spec-type mtp --spec-draft-n-max 3` + q4_0 KV) | @lamentofhighborne (1× RTX 3090, PCIe x8, 350 W) | Q4_K_M MTP | **164K** | **47.49 / 55.09** | — | ~22.2 GiB | 2026-05-07 | **Second 1× 3090 llama.cpp MTP data point on same rig** — froggeric's Q4_K_M MTP GGUF vs havenoammo's UD-Q4_K_XL above. Decode 47.91 / 55.81 TPS, TTFT 96 / 98 ms. `verify-full-mtp.sh` PASS 8/8, `verify-stress-mtp.sh` PASS 7/7 incl. 91K needle at 164K ctx. Functional MTP acceptance **86.7%**; canonical acceptance 55.3% narr / 71.2% code. **Ctx-fit ladder**: 262K OOMed MTP, 229K served without MTP, 196K initialized MTP but daemon died at 90K stress; 164K was the stable stress-passing ceiling on this rig. **Beats havenoammo on narr (47.49 vs 47.12, +0.8%) and ctx ceiling (164K vs 131K) but trails on code (55.09 vs 60.42, −9%)**. Manual long-context needles also passed at **120K** (39.39 decode TPS, 81% MTP accept) and **150K** (35.44 decode TPS, 80% MTP accept). MTP+vision incompat (per froggeric's model card); separate no-MTP+vision path passed 65K and 150K. [Issue #94](https://github.com/noonghunna/club-3090/issues/94). |

### Thinking on vs off — code-gen (HumanEval+ / LiveCodeBench v6)

Stock Qwen3.6-27B, single-card llama.cpp (`--reasoning on`), **FREE thinking-on vs `enable_thinking=false`**, `max_tokens=10240`, temp 0.0, n=1/problem, 2026-05-24.

| Arm | HumanEval+ 164 | LiveCodeBench v6 70 | All 234 | mean out tok (HE+ / LCB) | FREE truncated (HE+ / LCB) |
| --- | ---: | ---: | ---: | ---: | ---: |
| **no-think** (shipped default) | **155 (94.5%)** | **63 (90.0%)** | **218 (93.2%)** | 319 / 2610 | — |
| FREE (thinking-on) | 151 (92.1%) | 45 (64.3%) | 196 (83.8%) | 2902 / 6662 | 0% / 49% |

**Thinking is net-negative for code on stock Qwen3.6 — and bounding it doesn't flip that.** Two failure modes: on **easy code (HE+)** thinking finishes (0% truncation) yet overthinks correct→wrong (−4, at ~13× the tokens); on **hard code (LCB v6)** it *runs away* — **49% never converge within the 10K budget** and truncate→fail (−18). Giving FREE a bigger budget or **grammar-bounding the reasoning** (the structured-CoT `bounded-thinking.yml` row above: **93.9% HE+ / 66.0% LCB v6**, *separate eval, different conditions*) caps the runaway but lands LCB at ~66% — still **~24 pp under no-think's 90%**. Bounding makes thinking *terminate*, not *out-reason just writing the code*. (The FREE arm's completion-only LCB slice looks like ~parity — 35/36 — but that's selection bias: the problems thinking finished naturally are the easy ones.) **Verdict: thinking-off is the correct default for code-gen; thinking-on — free or grammar-bounded — ties at best, at multiples of the token cost.** Thinking's payoff, if any, is on non-code reasoning (math/science) — untested here; the `quality-test.sh --reasoning` packs (GSM-Symbolic, GPQA, `default_thinking: on`) are the probe. A clean same-run 3-arm (no-think / FREE / FSM-bounded) on LCB would pin the grammar-rescue exactly.

### Dual-card (2× RTX 3090, TP=2)

> NVLink auto-detection: dual-card composes now detect NVLink presence automatically. The `dual-nvlink*.yml` files are deprecated stubs that extend the unified compose with `NVLINK_MODE=force_on`. All NVLink bench rows below were measured with NVLink enabled (either via auto-detection or the deprecated stub). PCIe rows used `NCCL_P2P_DISABLE=1`.

| Compose | Rig | KV | Max ctx | Narr / Code TPS | PP tok/s | Peak VRAM | Date | Notes |
| --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- |
| `dual/nvfp4/mtp.yml` (**Marlin W4A16 fallback**, v0.24.0) 🧪 | @noonghunna (2× 3090 PCIe, sm_86) | fp8/e4m3 | 262144 | **69.7 / 85.5** (decode, n=3, CV <2%) | — | 23.77 GB/card | 2026-07-11 | **First NVFP4 boot on Ampere** — nvidia/Qwen3.6-27B-NVFP4 via the v0.24 Marlin W4A16 weight-only fallback (capability ≥ 7.5, no Hopper gate; spotted by @superalesha on X, code+live-confirmed here). Full production shape works: 262K + fp8/e4m3 KV + **MTP n=3 (accept 97–99%, AL 3.92–3.98)**. 8-pack think-off **110/150** — statistical TIE with the fp8 tier's 109 (TC 14·IF 13·SO 14·DE 13·RM 13·BF 13·HA 9/20·CLI 21/40); the Marlin-dequant path costs zero quality (weight-identical to native-FP4 rigs; W4A16 activations, native-FP4 unmeasured). **No speed edge on Ampere** (~20% below the AutoRound tier's ~87 narr) — a capability/quality datapoint (drove the #670 unhide + ⚑ fallback badge), NOT a recommended Ampere serving path for this model; sub-sm_90 value = models where NVFP4 is the only quant. |
| `dual.yml` ⭐ | @noonghunna (2× 3090 PCIe, no NVLink) | fp8 | 262K (237K single-prompt verified) | 69 / 89 | — | ~23.6 GB | 2026-04-29 | tested 2-card baseline. fp8 KV, 2 streams, full feature set. **PASSES v2 continuous soak** (Cliff 2b clean). |
| `dual.yml` ⭐ (**vLLM v0.24.0** pin) | @noonghunna (2× 3090 PCIe, no NVLink; caps GPU0 370 W / GPU1 420 W) | fp8 | 262K | **70.0 / 91.5** (decode 70.7 / 93.5) | — | ~21.5 GB/card | 2026-06-30 | **`vllm-stable` v0.22.0 → v0.24.0 pin-bump validation** — ≈ par with the v0.22.0 baseline (69/89). TP=2 symmetric, MTP n=3 **accept 3.51**, KV pool **622K / 2.37×**. verify-full **8/8** (streaming tool-calls clean → qwen3coder sidecar droppable), verify-stress **NIAH→240K**, soak-continuous **PASS** (0 growth / 0 err / 100% retention), quality toolcall 11/15 (73%) · instructfollow 15/15 (100%) (`--quick`, thinking=mixed, ≥ baseline). **`vllm-marlin-pad` confirmed native** ([#45295](https://github.com/vllm-project/vllm/pull/45295) + [#45176](https://github.com/vllm-project/vllm/pull/45176)) — served overlay-free. v0.24.0 default Mamba-cache `align` + experimental prefix-caching → NIAH-clean, no mitigation needed. |
| `dual-max` (FP8 wts + int8-PTH KV, **v0.24.0**) 🧪 | @noonghunna (2× 3090 PCIe, caps 370/420 W) | int8-PTH | 262K | **82.0 / 104.6** (decode 83.1 / 108.2) | — | ~21.4 GB/card | 2026-06-30 | **The 8-bit _decode_ corner.** FP8 weights via **Marlin W8A16** (no native FP8 on Ampere → weight-only dequant; vLLM warns "may degrade compute-heavy workloads"). MTP n=3, KV pool 295K/1.13×, TTFT 158/167 ms. 8-pack `--full` **107/150** (thinking-off: toolcall 14 · IF 15 · structout 14 · dataextract 9 · reasonmath 12 · bugfind 14 · hermes 11/20 · cli 18/40). **Corrects the stale "~56 TPS" probe** — real v0.24.0 decode is 83/108 (fastest decoder on the rig). Head-to-head vs W8A8 ↓. |
| `dual-max` + `KV_CACHE_DTYPE=fp8` (e4m3 via **FlashInfer**, **v0.24.0**) 🧪 | @noonghunna (2× 3090 PCIe, caps 370/420 W) | fp8/e4m3 | 262K | *bench-agentic decode:* warm **114.4** → **115.0 @ 35K** (vs int8-PTH warm 130.8 → **50.7** @ 35K) | **1212–1998** (94K→240K NIAH ladder; int8-PTH 467–1021) | 449 MB free @ ladder (int8-PTH 1263) | 2026-07-06 | **KV A/B — decode-at-DEPTH (jesse [#594](https://github.com/noonghunna/club-3090/pull/594)).** `fp8` resolves to **e4m3** and boots on sm_86 via **FLASHINFER** (int8-PTH is `TRITON_ATTN`-only) at **scale=1.0** (the checkpoint is weight-only — `kv_cache_scheme: None`, no KV scales — and `calculate_kv_scales` is *disabled* on Qwen3-Next hybrid: GDN/recurrent state → unreliable warmup scales, so 1.0 is the only fp8 KV available here). NOT `fp8_e5m2` — that's hard-rejected with fp8 checkpoints. **fp8/e4m3 decode stays FLAT → 2.3× int8-PTH at 35K; ~2× prefill; TTFT growth 7.3× vs 11.4×; NIAH recall TIES to 240K (all 6 rungs + 58K/91K).** **8-pack quality 109/150 (jesse, [#594](https://github.com/noonghunna/club-3090/pull/594)) — TIES int8-PTH's 107/150** (+2, within noise; per-pack all wash) → quality-neutral *despite* the flat scale=1.0. int8-PTH wins short-ctx (warm +14%) + VRAM headroom (1263 vs 449 MB free). **soak-continuous PASS** (0 MiB growth / 0 err / 0-of-25 silent / 100% retention / p50 125.5; margin holds at 509 MB free @ deepest fill) → all #594 gates green. |
| `dual` INT8 **W8A8** (Avesed) + int8-PTH KV, **v0.24.0** 🧪 | @noonghunna (2× 3090 PCIe, caps 370/420 W) | int8-PTH | 262K | **76.2 / 96.4** (decode 76.9 / 99.0) | **2062 → 1021** (prefill t/s @10K→90K) | ~21.4 GB/card | 2026-06-30 | **The 8-bit _prefill_ corner.** INT8 weights + dynamic INT8 activations → native **CUTLASS INT8** prefill (no dequant). TTFT **122 ms** (vs FP8 158). Prefill **+17–51%** vs FP8 at matched ctx (2062 vs 1364 @10K); decode trades ~8% (96 vs 105). 8-pack `--full` **107/150 — TIED with FP8** (thinking-off: toolcall 14 · IF 15 · structout 14 · dataextract 10 · reasonmath 12 · bugfind 14 · hermes 10/20 · cli 18/40) → **activation-quant cost = zero**. NIAH clean to 240K, pool 295K/1.13×. Same compose as `dual-max`, only `--model`+`--quantization compressed-tensors` differ. **Experimental data point — the 8-bit _prefill_ corner vs FP8's _decode_ corner; NOT shipped as a slug** (pending cross-rig + thinking-on arm before any catalog decision). **⚠️ Hardware ceiling: W8A8 INT8 compute is sm 7.5–9.x only — NOT supported on Blackwell (compute capability ≥ 10.0, incl. RTX 5090 sm_120 / RTX 6000-Blackwell)** per [llm-compressor W8A8 docs](https://docs.vllm.ai/en/latest/features/quantization/llm_compressor/int8_w8a8/) ("INT8 is not supported on compute capability >= 10.0"). So this prefill-corner path **can't serve the Blackwell community** — a factor against promotion (FP8 weights stay the Blackwell-safe 8-bit path). |
| `dual-turbo.yml` | @noonghunna (2× 3090 PCIe) | TQ3 | 262K | 58 / 76 per-stream (**269 TPS aggregate at 4 streams**) | — | ~19.8 GB | 2026-04-29 | TQ3 KV — 4.67× concurrency for multi-tenant agent workloads. |
| `dual-turbo.yml` ⭐ | @noonghunna (2× 3090 PCIe) | TQ3 | 262K | **81.21 / 108.20** single-stream | — | **20.0 GB** | 2026-05-05 | **v7.72.2 uplift**: Genesis pin `7b9fd319` + vLLM `01d4d1ad3` (Sander's PROD pin). 6 redundant local sidecars dropped (PN35/PN30/PN25/P78/PN34 supersede). 5 measured runs each, CV 2.3%/0.9%. AL 3.46. **VRAM −2.1 GB/card vs v7.69 baseline** (PN35 native + PN59 fold value). All 8/8 verify-full checks pass. |
| `dual-dflash.yml` | @noonghunna (2× 3090 PCIe) | fp8 | 185K | 82 / **125** | — | ~23.6 GB | 2026-04-29 | DFlash N=5 + 1.75 GB draft / card. AL ~4.4. Fastest 2-card short-prompt code path. |
| `dual-dflash.yml` | @apriori (2× 3090 + EPYC 7302P, Arch Linux, 230 W cap, NODE topology, no NVLink) | fp8 | 185K | **78.44 / 122.71** | — | ~24.0 GB | 2026-05-05 | **First EPYC + Arch cross-rig data on `dual-dflash`** — matches @noonghunna baseline within run-to-run CV (78/127 reference, narr drift +0.4 / code −3.4%). **PASSES continuous soak** (0 MiB VRAM growth, 0 errors, 0/25 silent-empty, 100% TPS retention) — first independent confirmation `dual-dflash` is Cliff 2b clean cross-rig. 3 turns >30s TTFT warning (informational). [Discussion #18](https://github.com/noonghunna/club-3090/discussions/18#discussioncomment-16819551). |
| `dual-dflash-noviz.yml` | @noonghunna (2× 3090 PCIe) | fp8 | 200K | 78 / **127** | — | ~23.8 GB | 2026-04-29 | DFlash + no vision tower. +15K ctx vs `dual-dflash`. |
| `dual-dflash-noviz.yml` | @snoby (2× **4090** PCIe — 5-GPU rig, GPUs 2,3, no NVLink, [#46](https://github.com/noonghunna/club-3090/issues/46)) | fp8 | **180K** | 92.55 / **148.99** | — | ~21.8 GB | 2026-05-04 | First non-3090 cross-rig data. **Required `max-model-len` drop from 200K→180K** vs 3090 baseline (boot OOM at 200K) — 4090 ctx-ceiling gotcha pending investigation. +17% TPS lift vs same compose on 3090 (78→92.55 narr / 127→148.99 code). |
| `dual.yml` | @guybrush01 (2× **5090** PCIe 5.0 x8, Ryzen 9 9950X3D2, CachyOS, driver 610.43.02, 575 W cap, no NVLink) | fp8 | 262K | **153.41 / 196.91** (decode 154.62 / 200.13) | — | ~31 GB/card (934 MB free at 262K ceiling) | 2026-06-25 | **First Blackwell / dual-5090 cross-rig data.** Qwen3.6-27B dual (AutoRound-INT4 + fp8 KV + MTP n=3) on vLLM v0.22.0, TP=2, sm_120 — boots + serves the full 262K with **no changes**. **~2× the 3090 dual** (vs @noonghunna v7.72.2 ~89/115) at tiny TTFT (~51/53 ms), CV 1.5%/1.2%. MTP accept 2.71–3.03. verify-stress **8/8** (NIAH→240K, Cliff 2 clear); soak-continuous **PASS** (0 err, 0/25 silent-empty, 0 MiB growth, p50 decode 252). Blackwell symm-mem all-reduce N/A → PYNCCL fallback (custom all-reduce off anyway, as on our PCIe rigs). PCIe 5.0 x8 ≈ Gen4 x16. **8-pack quality 110/150** (`quality-test.sh --full`, thinking=mixed, 2026-06-27) — lands on our dual-3090 baseline (109/150) within n=1 noise (det 63 vs 64 · sandbox 47 vs 45), confirming 8-pack quality is weights+KV+sampler-bound, not arch-bound; the two soft packs (dataextract 9/15 numeric-typing, cli-40 20/40 agentic) are the known Qwen-family floors. [Issue #474](https://github.com/noonghunna/club-3090/issues/474) ([quality run](https://github.com/noonghunna/club-3090/issues/474#issuecomment-4821406331)). |
| `dual-nvlink.yml` | @JusefPol (2× 3090 PCIe x8 + **NVLink 4× bonded**, i7-11700K, 365 W/card) | fp8 | 262K | **108.81 / 138.55** | — | ~23.7 GB | 2026-05-04 | First NVLink cross-rig data. **+58% narr / +56% code TPS vs `dual.yml` PCIe-only baseline (69 / 89)** — NVLink reduces the per-token NCCL allreduce latency floor; compounds at multi-stream. verify-stress 8/8 PASS incl. 91K needle. **PASSES v2 continuous soak** (5 sessions × 5 turns, 0 MiB growth, 100% TPS retention). MTP n=3, 65–98% per-position accept. PR [#31](https://github.com/noonghunna/club-3090/pull/31). |
| `dual-nvlink-turbo.yml` ⭐ | @danbedford (2× 3090 NVLink, 230W cap) | TQ3 | 262K | **102.34 / 133.98** | — | ~22.3 GB | 2026-05-05 | **v7.72.2-rebench** (image `nightly-01d4d1ad3`). 4-stream TurboQuant KV + NVLink. **+11% narr / +12% code vs same-rig PCIe `dual-turbo` (#73 below)** — controlled A/B on identical hardware, only `NCCL_P2P_LEVEL` differs. Custom all-reduce ENABLED (disabled on PCIe). CV 3.1% narr / 1.8% code. PR [#56](https://github.com/noonghunna/club-3090/pull/56) + [Issue #69](https://github.com/noonghunna/club-3090/issues/69). |
| `dual.yml` | @danbedford (2× 3090 NVLink-cable-attached, run as PCIe via `NCCL_P2P_DISABLE=1`, 230W cap) | fp8 | 262K | **89.24 / 114.57** | — | ~23.7 GB | 2026-05-06 | **First controlled PCIe-vs-NVLink A/B on same rig** — pair with `dual-nvlink.yml` row immediately above. **+15% narr / +15% code lift from NVLink** (#74 102/132 vs this 89/115). CV 3.8%/2.5%. **Note: this corrects the "+58% narr / +56% code" claim from JusefPol's row** — that comparison conflated NVLink lift with v7.72.2 lift (his baseline was 2026-04-29 dual.yml at 69/89 on the older image). On a strictly v7.72.2-controlled comparison NVLink adds ~15%, not ~58%. [Issue #77](https://github.com/noonghunna/club-3090/issues/77). |
| `dual-turbo.yml` | @danbedford (2× 3090 NVLink-cable-attached, run as PCIe via `NCCL_P2P_DISABLE=1`, 230W cap) | TQ3 | 262K | 91.58 / 120.00 | — | ~22.0 GB | 2026-05-06 | Companion to `dual-nvlink-turbo` row above for the controlled A/B. NVLink lift on TQ3 path: **+11% / +12%**. CV 3.2%/1.9%. [Issue #73](https://github.com/noonghunna/club-3090/issues/73). |
| `dual-nvlink.yml` | @danbedford (2× 3090 NVLink, 230W cap) | fp8 | 262K | **102.09 / 131.59** | — | ~24.0 GB | 2026-05-06 | Second cross-rig data on `dual-nvlink.yml` (vs JusefPol's earlier 108.81/138.55). Lower than JusefPol partly explained by his lower power cap (365 W/card vs 230) — on memory-bandwidth-bound decode, 2 GB/card more thermal headroom doesn't compound much, so close-but-lower at half the wattage is consistent. CV 2.6%/1.4%. [Issue #74](https://github.com/noonghunna/club-3090/issues/74). |
| `dual-dflash.yml` | @danbedford (2× 3090 PCIe NVLink-cable-attached but `NCCL_P2P_DISABLE=1`, 230W cap) | FP16 | 185K | 86.62 / **141.02** | — | ~24.0 GB | 2026-05-06 | Third cross-rig DFlash data point (after @noonghunna 82/125 + @lolren 87/142). **Code TPS 141 ties lolren's 142** as the highest measured on club-3090. CV 2.4%/5.0%. [Issue #75](https://github.com/noonghunna/club-3090/issues/75). |
| `dual-dflash-noviz.yml` | @danbedford (2× 3090 PCIe NVLink-cable-attached but `NCCL_P2P_DISABLE=1`, 230W cap) | FP16 | 200K | 88.31 / **142.79** | — | ~23.9 GB | 2026-05-06 | DFlash + no vision tower. Beats @noonghunna baseline 78/127 (+13%/+12%). CV 2.3%/2.9%. [Issue #76](https://github.com/noonghunna/club-3090/issues/76). |
| `dual-nvlink-dflash.yml` ⭐ NEW | @danbedford (2× 3090 NVLink, 230W cap, i9-11900KF) | FP16 | 185K | **101.55 / 163.33** | — | 24.06 GB/card | 2026-05-07 | **First NVLink-enabled DFlash row.** Mirrors `dual-dflash.yml` shape but enables NCCL P2P over NVLink + custom_all_reduce. **+17% narr / +16% code over his own PCIe `dual-dflash` row above** (86.62 / 141.02 — same rig with `NCCL_P2P_DISABLE=1`). Decode 102.43 / 166.54 TPS, CV 1.8%/1.9%. **PASSES continuous soak** (0 errors, 0 silent-empty, 0 MiB growth, 100% TPS retention, p50 66.71). verify-full 8/8 + verify-stress 8/8 incl. 91K Cliff 2 needle. PR [#92](https://github.com/noonghunna/club-3090/pull/92). |
| `dual-nvlink-dflash-noviz.yml` ⭐ NEW | @danbedford (2× 3090 NVLink, 230W cap) | FP16 | **188K** | **103.24 / 167.45** | — | ~23.97 GB/card | 2026-05-07 | **NVLink + DFlash + no vision** — pushes the with-vision 185K ctx ceiling to **188K** by dropping MoonViT (~0.78 GB freed). Empirically determined: 189K had only 1/3 success rate (flaky on freshly rebooted system), 188K is the stable ceiling. **+17% narr / +17% code over his own PCIe `dual-dflash-noviz` row above** (88.31 / 142.79). Decode 104.07 / 171.01 TPS, CV 2.2%/3.6%. **PASSES continuous soak** (p50 66.75, 100% retention). verify-full 8/8 + verify-stress 8/8. PR [#96](https://github.com/noonghunna/club-3090/pull/96). |
| `dual.yml`-shape **+ patched P2P drivers** (no NVLink hardware) | @aaronlockhartdev (2× 3090 PCIe x16, EPYC 7F52, Arch Linux, custom Dockerfile via [Sam McLeod's guide](https://smcleod.net/2026/02/patching-nvidias-driver-and-vllm-to-enable-p2p-on-consumer-gpus/) — patched `aikitoria/open-gpu-kernel-modules` + vLLM `cuda.py` `return True` patch) | fp8 | 262K | **93 / 125** | — | n/a | 2026-05-07 | **First patched-driver P2P cross-rig data point** — answers the question raised in [disc #70](https://github.com/noonghunna/club-3090/discussions/70). Same-rig controlled A/B: unpatched baseline 91 narr / 114 code → patched P2P 93 / 125 = **+2% narr / +9% code**. Compared to NVLink hardware lift (+15% / +15% per @danbedford's controlled A/B): patched P2P captures **~60% of NVLink's code gain but ~13% of NVLink's narr gain** — code workloads (spec-decode K+1 verify is heavily cross-card matmul) benefit more from cross-card bandwidth than narr decode (more sequential per-token). For ~95% of dual-3090 owners without NVLink, the trade is small TPS lift vs custom kernel module + DKMS maintenance burden. [Issue #91](https://github.com/noonghunna/club-3090/issues/91). |
| `dual-dflash-noviz.yml`-shape **+ patched P2P drivers** (no NVLink hardware, custom_all_reduce ENABLED) | @aaronlockhartdev (2× 3090 PCIe x16, EPYC 7F52, Arch Linux, patched kernel module + `NCCL_P2P_LEVEL=PHB`) | fp8 | 200K | **100.47 / 160.15** (decode 101.53 / 164.44) | — | ~22.2 GB/card | 2026-05-07 | **Second patched-P2P cross-rig data point** — extends [#91 dual.yml result](https://github.com/noonghunna/club-3090/issues/91) to the DFlash + no-vision path. Same-rig controlled A/B: unpatched baseline 82.55 narr / 134.45 code → patched P2P 100.47 / 160.15 = **+22% narr / +19% code**. **Significantly larger lift than `dual.yml`-shape** (+22%/+19% here vs +2%/+9% on `dual.yml`) — DFlash's K+1 cross-card verify pattern stresses peer-bandwidth more than fp8-only `dual.yml`. **Important methodology update**: `NCCL_P2P_LEVEL=PHB` alone with the default vLLM image produced the same lift as the full vLLM `cuda.py` patch — **the in-container vLLM source patch is unnecessary**, only the kernel module patch matters. CV 4.6%/2.4%. custom_all_reduce ENABLED (vs disabled on the `dual.yml` row). [Issue #95](https://github.com/noonghunna/club-3090/issues/95) + [disc #70](https://github.com/noonghunna/club-3090/discussions/70). |
| `dual-dflash-noviz.yml` | @hlo-world (2× 3090 **PCIe x4 each**, Ryzen 5 5600, Ubuntu 24.04, driver 595.71.05; aikitoria patched driver installed host-side but run used stack-default `NCCL_P2P_DISABLE=1` — **P2P NOT engaged**) | fp8 | 200K | **76.15 / 117.31** (decode 76.94 / 121.75) | — | ~23.8 GB/card | 2026-05-18 | **First dual-3090 PCIe x4 data point.** Lanes negotiated x4/x4 (GPU max x16); report.sh auto-flagged the narrow slot. Boot log `[nvlink] NVLink DISABLED — NCCL_P2P_DISABLE=1, custom all-reduce OFF` → this is an **x4-lanes / P2P-off** number, *not* a patched-P2P anchor. vs @noonghunna x16-ish baseline 78/127: only **−2% narr / −8% code** — `dual-dflash-noviz` is far less interconnect-bound than the x4 warning implies, even without P2P. CV 5.4%/3.2%. For a true patched-P2P anchor see @aaronlockhartdev's validated recipe (#95): patched kernel module + `NCCL_P2P_LEVEL=PHB` + custom_all_reduce ENABLED (the vLLM `cuda.py` patch is unnecessary). First in-the-wild exercise of the #156 lspci report subsection (sudo-less degradation note rendered correctly). [Issue #158](https://github.com/noonghunna/club-3090/issues/158) |
| `dual.yml` + **`NVLINK_MODE=pcie_p2p`** (PR #291) | @hlo-world (2× 3090 **PCIe x4/x4**, Ryzen 5 5600, Ubuntu 24.04, aikitoria patched driver 595.71.05 — same rig as #158 above) | fp8 | 262K | **77 / 103** | — | ~22.4 GB/card | 2026-06-01 | **First validation of the shipped `NVLINK_MODE=pcie_p2p` mode ([PR #291](https://github.com/noonghunna/club-3090/pull/291)) — the one rig that can engage PCIe-P2P (patched driver, no NVLink).** Boot: `NVLINK_MODE=pcie_p2p — NCCL_P2P_LEVEL=PHB, custom all-reduce ON`. Controlled same-rig A/B: P2P **off** (`NCCL_P2P_DISABLE=1`) 70 / 90 → P2P **on** 77 / 103 = **+10% narr / +14% code**. Confirms the [#290](https://github.com/noonghunna/club-3090/issues/290) regression fix end-to-end. Larger lift than @aaronlockhartdev's x16 `dual.yml` (+2%/+9%, #91) — consistent with **x4 lanes making the link more of a bottleneck**, so P2P helps more. (Config runs MTP n=3 per the `Qwen3_5MTP` boot arch; the issue titles say "dflash".) Issues [#294](https://github.com/noonghunna/club-3090/issues/294) (off) + [#295](https://github.com/noonghunna/club-3090/issues/295) (on). |
| `dual.yml` | @oven1231231234 (2× 3090 **NVLink**, PCIe 3.0 x16) | fp8 | 262K | **67.24 / 86.14** (decode 67.89 / 87.80) | — | ~23.9 GB/card | 2026-06-26 | **NVLink arm of a same-rig NVLink-vs-PHB toggle** ([#488](https://github.com/noonghunna/club-3090/issues/488)). soak-continuous **PASS** (p50 decode 122.6, 0 err, 0 MiB growth, 0/25 silent-empty, 100% retention). TTFT 143 ms, CV 2.2%/1.6%. MTP AL 3.41–3.54, ~84% draft accept. Within-rig **+56–60% over the PHB arm below** (consistent across soak/narr/code) — **but NOT power-matched** (325/298 W here vs 228/253 W on PHB; benches ~8 h apart) and **larger than the controlled #77 A/B** (+15% at pinned 230 W). PCIe 3.0 x16 ≈ 4.0 x8 (#77) in bandwidth, so the bus doesn't explain the gap → treat the magnitude as an upper bound pending a power-pinned re-run. |
| `dual.yml` | @oven1231231234 (2× 3090 **PHB / no-P2P**, PCIe 3.0 x16) | fp8 | 262K | **42.74 / 53.53** (decode 43.15 / 54.84) | — | ~24.1 GB/card | 2026-06-26 | **PHB arm** (true no-bridge topology; pair with the NVLink row above, [#488](https://github.com/noonghunna/club-3090/issues/488)). soak-continuous **PASS** (p50 decode 78.5). TTFT 220–230 ms; lopsided util (89/39%) consistent with allreduce serialization. **Narrative 43 TPS is below a single 3090** (~45–50) → suspect power-throttle and/or a pathological no-P2P path on this board, not pure interconnect cost; see #488 for the power-matched re-run ask. **Corroborated board-specific 2026-06-29**: @chriskerley78910's PHB/no-P2P rig (PCIe 4.0 bifurcated x8/x8) benches code decode **129.75** (CV 2.8%) on the same compose — a healthy PHB number ~2.4× this row's, confirming the 43/54 arm reflects this board, not PHB topology generally. |
| `carnice-bf16mtp.yml` | @noonghunna (2× 3090 PCIe, no NVLink) | fp8 | 262K | **72** / **80** | — | ~22.25 GB | 2026-05-04 | **Carnice-V2-27B (Hermes agentic fine-tune) + BF16 MTP overlay**. Full 262K context, 2 streams. 71.75 narr / 80.35 code wall TPS (n=5 each, CV ~11%), MTP AL 3.02-3.14, TTFT 141ms. Patched chat template for Hermes JSON tool calls. verify-full 7/8 PASS. soak PASS. |
| `beellama/carnice-v2-dual-q8-mtp` 🧪 | @noonghunna (2× 3090 PCIe, no NVLink, 370 W cap) | q8_0/q8_0 | **262K** | **40.7 / 44.0** (decode, n=5, CV <1.3%) | **1197** | 21.8 / 21.2 GB/card | 2026-06-16 | **Carnice-V2-27B Q8_0 + embedded MTP head on beellama v0.3.2-preview (layer-split, -ts 0.55,0.45) — the quality-max dual Carnice ([#403](https://github.com/noonghunna/club-3090/discussions/403)).** TTFT 79/78 ms. verify-full all-pass · verify-stress **8/8** (NIAH ladder → 240K, 5.2 GB ceiling margin) · soak fresh 20×5 **PASS** (0 MiB growth, 0/100 silent-empty, p50 42.2 TPS, 100% retention). 8-pack think-OFF **103/150** / think-ON **105/150** (wash) — in-band (≈ qwopus-coder 103; Q5 single = 110). MTP accept ~81% @ n=1. **q8_0 KV chosen over the originally-requested kvarn6**: KV A/B = +17% prefill (1003 vs 860 t/s — escapes KVarN software-compression compute, q4=q8=1004 so it's the path not the bit-width), higher fidelity (q8 > kvarn6 q6-class), reference-aligned, and fits 262K on dual. `-b`/`-ub`/`--no-mmap` A/B'd FLAT — KV type was the only prefill lever. **n=2 = +13% validated opt-in** (`DRAFT_N_MAX=2`; lossless, lower 62% accept costs speed only). DFlash ruled out (only BASE-27B drafters exist → ~10% accept on the fine-tune; no Carnice-matched drafter). beellama v0.3.2 rolling pre-release → 🧪. |
| `dual.yml` ⭐ | @lolren (2× 3090 PCIe + Ryzen 9 5950X, **250W/card cap**) | fp8 | 262K | **89.78 / 117.60** | — | ~22.3 GB | 2026-05-05 | **First cross-rig data on the v7.72.2 uplift** (image `nightly-01d4d1ad3`, post-PR #59). +30% narr / +32% code over @noonghunna 2026-04-29 baseline (69/89 on older image) — confirms the v7.72.2 dividend cross-rig. CV 3.3%/2.0%. MTP AL ~3.5, per-pos accept 94/84/72%. [Disc #18](https://github.com/noonghunna/club-3090/discussions/18#discussioncomment-16820303). |
| `dual-dflash.yml` | @lolren (2× 3090 PCIe + Ryzen 9 5950X, 250W cap) | FP16 | 185K | 87.10 / **142.0** | — | ~22.1 GB | 2026-05-05 | Older image `nightly-7a1eb8ac2`. +6% narr / +14% code over @noonghunna baseline (82/125) — likely Ryzen 5950X advantage on prefill. DFlash AL ~4.5, per-pos accept 93/81/68/56/48%, avg accept 69%. [Disc #18](https://github.com/noonghunna/club-3090/discussions/18#discussioncomment-16820303). |
| `bounded-thinking.yml` | @lolren (2× 3090 PCIe + Ryzen 9 5950X, 250W cap, **MTP-disabled-suspected**) | TQ3 | 180K | 64.86 / 64.96 (CV **0.1%**) | — | ~22.3 GB | 2026-05-05 | **Anomaly:** lolren reports "no spec-decode" on this run despite `bounded-thinking.yml` shipping `--speculative-config mtp n=3` by default. Near-identical narr=code TPS + extreme CV stability (0.1%) suggests MTP was inactive — likely because his image was older `nightly-7a1eb8ac2` (pre-v7.72.2 + pre-PN35). Re-test on `nightly-01d4d1ad3` should restore MTP path → expect ~50/66 narr/code with normal CV. Tracked. [Disc #18](https://github.com/noonghunna/club-3090/discussions/18#discussioncomment-16820303). |
| `dual.yml` | @JDWarner (**Mixed RTX A5000 + RTX 3090**, both **Razer Core X eGPU enclosures over Thunderbolt 3**, Intel NUC11TNH i5-1135G7, **16 GB RAM**, headless, A5000=230W cap / 3090=290W cap, PCIe **x4 Gen 3** per card) | fp8 | 262K | **56.83 / 72.47** (soak p50 93.09) | — | ~23.6 GB/card | 2026-05-09 | **Soak: ✓ PASS** (5×5, 0 errors, 0 silent-empty, 100% TPS retention, 0 MiB growth). **First TB3 dual-eGPU + mixed-arch cross-rig data**. The setup that "shouldn't work": each card on a separate TB3 controller → ~3.94 GB/s effective per card vs ~32 GB/s on PCIe x16 Gen 4 (~8× cut), mixed Ampere SKUs (workstation A5000 + consumer 3090 with different mem bandwidth + clocks), 16 GB system RAM total. **Result: matches `dual.yml` PCIe x16 baseline within run-to-run noise** — confirms decode on Qwen3.6-27B is per-card-bandwidth bound, cross-card NCCL allreduce is small enough that even an 8× link cut doesn't dominate. Extends @aaronlockhartdev's #91/#95 finding (patched-P2P only +2%/+9% on `dual.yml`) in the opposite direction: even with 8× *less* cross-card bandwidth, decode holds. MTP AL 3.39-3.52, per-pos accept 0.93/0.83/0.70 (89% avg). verify-full + verify-stress all PASS. Genesis pin `7b9fd319` (v7.72.2). [Issue #107](https://github.com/noonghunna/club-3090/issues/107). |
| `dual/docker-compose.yml` (default) | @ygafarov (**3090 via USB4 eGPU dock + 5070 Ti via OCuLink** — heterogeneous Ampere + Blackwell consumer dual-eGPU, AMD Ryzen AI MAX+ 395 / Strix Halo miniPC, CachyOS, 123 GB RAM, 290 W cap both cards, PCIe **x4** per card — USB4 ≈ 3.94 GB/s, OCuLink ≈ 7.88 GB/s) | fp8 | 200K | **65.10 / 85.81** | — | 17.1 / 15.7 GB | 2026-05-12 | **First heterogeneous Ampere + Blackwell consumer dual-eGPU on the matrix.** TP=2 bound by the slower USB4 link in allreduce + sm_86 kernels (5070 Ti spends back-half of step waiting — 91% util but only 125 W out of 290 W cap). KV pool 200K @ 1.00× concurrency — VRAM cap from the 5070 Ti's 16 GiB (model takes 13.8 GiB/card → only ~2.2 GiB left for KV on the smaller card). verify-stress 8/8 incl. **91K needle recall** (Cliff 2 clean). Soak ⚠ borderline (360 MiB > 200 MiB threshold — same eGPU-bus accretion as ygafarov's own #113 single-card row above at 240 MiB; 100% TPS retention + 0 silent-empty + 0 errors so not a leak). MTP AL 3.50, per-pos accept 0.94/0.86/0.70. CV 4.5%/1.8%. **Slower than ygafarov's own single-3090 #113 row** (68.86/91.70 at 48K) — on this rig the single-card path is recommended; the 5070 Ti adds VRAM cap pain without TPS gain. Driver 595.71.05, vLLM `nightly-1acd67a79`, no Genesis (Blackwell consumer not on allowlist). [Issue #120](https://github.com/noonghunna/club-3090/issues/120). |
| `dual.yml` | @OVDEN13 (2× 3090 + Ryzen 7 5700X, **PCIe Gen 4 x4 + x8 asymmetric**, 300 W cap, no NVLink, Ubuntu 26.04 / driver 595.58.03 / CUDA 13.2, 92 GB RAM, bare metal) | fp8 | 262K | **75.80 / 99.04** | — | 22.3 GB/card | 2026-05-15 | **First asymmetric PCIe-lane data point on the matrix** (Gen 4 x4 to one card, x8 to the other — half the cross-card bandwidth of @danbedford's Gen 4 x16 baseline). **−15% narr / −14% code vs @danbedford's same-compose PCIe-only A/B** (89.24 / 114.57, [#77](https://github.com/noonghunna/club-3090/issues/77)) — gap consistent with the Gen 4 x4 card being the bottleneck for cross-card NCCL allreduce. MTP AL 2.71–3.51, per-pos accept ranges 0.81/0.56/0.35 → 0.95/0.85/0.72; the wide variance and lower floor vs @lolren's stable 0.94/0.84/0.72 ([disc #18](https://github.com/noonghunna/club-3090/discussions/18#discussioncomment-16820303)) suggest draft-forward sync stalls on the slow lane. Both GPUs pegged at 297 W power cap (80% / 87% util — mildly asymmetric, consistent with x4+x8 imbalance). CV 1.2% narr / 4.2% code. **Soak: ✓ PASS** (20×5 fresh mode, p50 101.41 TPS, p95 TTFT 1645 ms, 0 errors, 0 silent-empty, 100.7% retention, 0 MiB growth). Stack: master @ `b1c68b4` (v0.7.2-7-g, pre-v0.7.3 tag), vLLM `nightly-1acd67a795eb`, Genesis pin `7b9fd319`. [Issue #142](https://github.com/noonghunna/club-3090/issues/142). |
| `dual.yml` | @mgabor3141 (2× 3090, ASUS PRIME X399-A + AMD TR 1950X (Zen 1, 16c/32t, 2017) + 16 GB DDR4 (single-DIMM), GPU 0 + GPU 1 both PCIe 3.0 ×16, 280 W cap, no NVLink, CachyOS bare metal, headless, `iommu=pt`) | fp8 | 262K | **58.98 / 79.19** | — | ~22.6 GB/card | 2026-05-21 | **Validates the v0.7.3 HARDWARE.md prediction for PCIe Gen 3 + pre-Zen3 rigs** (predicted band 53.9-62.9 / 70.6-82.3, lands mid-band). Same-rig hardware-only upgrade from the Z77 + 3770K + Gen 2 ×4 row below: +8% narr / +15% code on fixing the cross-card PCIe bottleneck. **Soak: ✓ PASS clean** (p50 114.02, 0 errors, 100% retention, 0 MiB growth). verify-stress 8/8 incl. 91K needle. **Required `iommu=pt` on kernel cmdline** (AMD-Vi Translated mode crashes both GPUs with Xid 154 under sustained TP=2 DMA; diagnosis in [#178 comment](https://github.com/noonghunna/club-3090/issues/178#issuecomment-4509363017), now documented in [HARDWARE.md](https://github.com/noonghunna/club-3090/blob/master/docs/HARDWARE.md)). vLLM `nightly-bf610c2f` (v0.7.3 routing puts non-TQ3 composes here). [Issue #178]. |
| `dual.yml` | @mgabor3141 (2× 3090, ASUS P8Z77-V LX + **i7-3770K Ivy Bridge, 2012** + 16 GB DDR3, **GPU0 PCIe 3.0 ×16 / GPU1 PCIe 2.0 ×4 ≈ 2 GB/s**, 280 W cap, no NVLink, CachyOS bare metal, headless) | fp8 | 262K | **54.46 / 68.64** (decode 54.93 / 70.27) | — | ~22.6 GB/card | 2026-05-21 | **Slowest cross-card link on the matrix** (~2 GB/s — half of @JDWarner's TB3 dual-eGPU). The "slap 3090s into e-waste" baseline — and it's serviceable. Sits **just below @JDWarner's TB3 row** (56.83 / 72.47, ~3.94 GB/s), confirming the per-card-bandwidth-bound thesis holds even at ~2 GB/s cross-card — extends @aaronlockhartdev [#91](https://github.com/noonghunna/club-3090/issues/91)/[#95](https://github.com/noonghunna/club-3090/issues/95) + @JDWarner [#107](https://github.com/noonghunna/club-3090/issues/107). TTFT 158 / 171 ms, CV 1.6–2.2%. MTP AL 3.46, per-pos accept 95/86/75% (avg 81.9%). **Soak: ✓ PASS clean** (p50 99.84, 0/25 silent-empty, 100% retention, <200 MiB growth); verify-stress 8/8 + quality 101/150 all PASS. Cliff 2 doesn't fire (TP=2 splits the GDN forward). vLLM `nightly-bf610c2f`. [Issue #178](https://github.com/noonghunna/club-3090/issues/178). |
| `dual.yml`-shape (**hand-rolled, stock `vllm/vllm-openai:nightly`, NO Genesis**) — **Proxmox VFIO GPU-passthrough VM** | @duart (2× 3090 **NVLink**, Dell T630 + 2× Xeon E5-2695 v4 Broadwell, 256 GB DDR4, PCIe 3.0 ×16, models on ramdisk) | fp8 | 262K | **76.46 / 75.00** (decode 77.86 / 77.79) | — | ~21.5 GB/card | 2026-05-21 | **First Proxmox-passthrough + enterprise-server datapoint.** Self-solved **41/55 → 76/78**: root cause was NOT the report's idle `LnkSta` 2.5 GT/s (that's PCIe ASPM at idle — trains to 8 GT/s under load) but enabling **MTP FULL_AND_PIECEWISE** cudagraph (+50% vs PIECEWISE-only). Stock upstream nightly `v0.21.1rc1.dev132` **without Genesis** → ~15% below @lolren's Genesis `dual.yml` (89.78 / 117.60) and ~30% below the NVLink Genesis rows — the gap is the v7.72.2 Genesis dividend, not the rig/virtualization. NUMA-aligned (vCPUs+mem+GPU same node), 8 vCPU sufficient (vLLM uses ~3). CV ~1%. verify-stress 8/8 incl. 91K Cliff 2 needle; soak PASS clean (p50 92.90, 0/25 silent-empty, 0 MiB growth, 100% retention). [Disc #162](https://github.com/noonghunna/club-3090/discussions/162#discussioncomment-17002449). |

### Quad-card (4× RTX 3090, TP=4)

| Compose | Rig | KV | Max ctx | Narr / Code TPS | PP tok/s | Peak VRAM | Date | Notes |
| --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- |
| `multi4.yml` | @whamp (4× 3090 PCIe x4/x16/x8/x16, 300 W cap, no NVLink) | fp8 | 262K | 63 / 76 | — | ~23.5 GB | 2026-05-03 | TP=4 capacity king. **6.77× concurrency at 262K**. PASSES v2 continuous soak (20 sessions, 0 MiB growth, 90.8% TPS retention). PR [#44](https://github.com/noonghunna/club-3090/pull/44). |
| `multi4.yml` | [@alanspires #127](https://github.com/noonghunna/club-3090/issues/127) (**6× 3090 VFIO-passthrough**, AMD EPYC 7313 host, all cards 250W cap, no NVLink) | fp8 | 262K | **74.93 / 92.85** | — | ~21.7 GB | 2026-05-14 | TP=4 on a 6-card rig — GPUs 4-5 free (Qwen `num_kv_heads=4` doesn't divide 6, so TP=6 invalid). **First VFIO-passthrough + virtualized data point** — scripts (verify-full / verify-stress / soak) all PASS on the virt envelope without modification. KV pool 1,774,963 tokens at 6.77× concurrency. Soak p50 122.84 / p95 161.14 across 5 multi-turn sessions, 0 errors. |
| `multi4-dflash.yml` | @whamp (4× 3090 PCIe x4/x16/x8/x16, 300 W cap) | fp8 | 262K | 64 / **104** | — | ~22.0 GB | 2026-05-03 | TP=4 + DFlash. 2.27× concurrency at 262K. PASSES v2 continuous soak (5 sessions, 0 MiB growth, 100% TPS retention). **Bench-vs-soak inversion**: bench shows DFlash wins by 37% on short-prompt code, soak shows DFlash *loses* by 47% on multi-turn agent — DFlash AL likely collapses on mixed prompts. PR [#44](https://github.com/noonghunna/club-3090/pull/44). |
| `multi-max` | @Whamp (4× 3090 PCIe x4/x16/x8/x16, 230 W, aikitoria `595.71.05-p2p` kernel) | int8-PTH | 262K | **85.09 / 102.43** | — | ~23 GB/card | 2026-06-19 | **First real 4-card TP=4 validation of `vllm/qwen-27b-multi-max`** (shipped 🧪 with duals as proxies). fp8 weights + INT8-PTH KV. verify-full + verify-stress **7/7** (Cliff 2) + soak-continuous **PASS** (0 err, 0/25 silent-empty, 0 MiB growth, p50 80.92, 100% retention). TP=4 narr (85) ≈ our 2-card TP=2 → TP scaling flat past TP=2 on PCIe; multi4's value is the **6.77× KV pool**, not single-stream TPS. P2P kernel installed (engagement in this run TBD — see thread). [Issue #446](https://github.com/noonghunna/club-3090/issues/446). |
| `multi-fast` | @Whamp (same rig as above) | fp8 | 262K | **59.00 / 73.88** | — | — | 2026-06-22 | Same-session baseline for the int8-PTH A/B below — INT4 weights + fp8 KV. [Issue #446](https://github.com/noonghunna/club-3090/issues/446). |
| `multi-fast` + `KV_CACHE_DTYPE=int8_per_token_head` | @Whamp (same rig) | int8-PTH | 262K | **92.50 / 121.85** | — | ~22.2 GB/card | 2026-06-22 | **Standout TP=4 config — INT4 weights + INT8-PTH KV.** vs same-session `multi-fast` fp8 KV (59/74) = **+57% narr / +65% code from the KV format alone** (far bigger than anything int8-PTH buys at TP=2, where the KV format is ~a throughput wash — worth understanding). vs `multi-max` fp8-weights (85/102) = +9%/+19% from INT4-vs-fp8 weights. MTP accept 3.38–3.54. Candidate for a shipped INT4+int8-PTH multi4 config **if the KV win holds without P2P**. [Issue #446](https://github.com/noonghunna/club-3090/issues/446). |
| `multi-fast` (**v0.24.0**) | @ryanmpelletier (4× 3090 PCIe **x16**, 350 W, Proxmox VM passthrough, no NVLink) | fp8 | 262K | **74.76 / 90.83** | **1288 → 1175** (prefill t/s @10K→90K) | ~22.7 GB/card | 2026-07-05 | **Promoted `vllm/qwen-27b-multi-fast` → ✅ Production** on this report — the 3rd concordant 4-card validation (after @alanspires 6×VFIO + @Whamp) and the first on **v0.24.0**. verify-full 9/9 · verify-stress needles clean to 240K · soak-continuous **PASS** (0 err / 0 silent / <200 MiB) · bench n=5 CV 4.2%/3.0%. **All-x16 topology beats @Whamp's mixed-lane 59/74** on the same fp8-KV config → PCIe lane width matters for TP=4 all-reduce; ~matches @alanspires 74.93/92.85. KV pool **1.77M / 6.77×** — the 4-card value is concurrency; single-stream decode ~flat vs the 2-card 70.7/93.5. TPS-only (no 8-pack; quality is TP-invariant, carried from `vllm/dual` 109/150). [Issue #584](https://github.com/noonghunna/club-3090/issues/584). |
| `multi-max` (**v0.24.0**) | @ryanmpelletier (4× 3090 PCIe **x16**, 350 W, Proxmox VM passthrough, no NVLink) | fp8-e4m3 | 262K | **74.10 / 91.30** | **1275 → 1141** (prefill t/s @10K→90K) | ~23.5 GB/card | 2026-07-07 | **Promoted `vllm/qwen-27b-multi-max` → ✅ Production** on this report — the clean v0.24.0 4-card fp8/e4m3-KV validation the ⚠️-caveat required (supersedes @Whamp's stale int8-PTH #446). Attention backend **FLASHINFER** (the #595 int8-PTH→fp8/e4m3 flip live at TP=4). verify-full 9/9 · verify-stress clean to 240,634 tok · soak-continuous **PASS** (0 err / 0 silent / <200 MiB, p50 124.94) · **full 8-pack 111/150** (@ryanmpelletier built the benchlocal sandboxes) — ties the 2-card dual-max proxy 109/150 → max-tier quality **TP-invariant**. [Issue #584](https://github.com/noonghunna/club-3090/issues/584). |
| `multi-max` (**v0.24.0**) | @MoppelMat (4× 3090 PCIe **mixed x4/x4/x8/x8**, 300 W cap, bare metal AMD 5950X, no NVLink) | fp8-e4m3 | 262K | **79.23 / 101.61** | **964 → 887** (prefill t/s @10K→90K) | ~23.5 GB/card | 2026-07-07 | **2nd independent 4-card `vllm/qwen-27b-multi-max` validation** (corroborates @ryanmpelletier #584). MTP live · prefill-90K clean · soak-continuous **PASS**. **Decode beats ryan's x16 rig (79/102 vs 74/91)** — bare metal, no VM-passthrough overhead; **prefill trails ~30%** (964/887 vs 1275/1141) — the mixed x4/x8 lanes are the bandwidth-bound bottleneck, while decode is ~lane-insensitive at TP=4. Clean decode(compute) / prefill(interconnect) split. [Issue #625](https://github.com/noonghunna/club-3090/issues/625). |

### Verify-stress + soak-continuous matrix

Not TPS, but load-bearing. Every shipped variant is validated against:

- `bash scripts/verify-full.sh` — fast functional smoke (8 checks)
- `bash scripts/verify-stress.sh` — boundary tests including Cliff 2 needle recall (probe 7: 60K + 90K needles)
- `SOAK_MODE=continuous bash scripts/soak-test.sh` — multi-turn accumulating-context cliff (Cliff 2b at ~25K)

| Variant | Rig | verify-full | verify-stress 8/8 | soak-continuous | Date |
|---|---|---|---|---|---|
| `minimal.yml` (single-card vLLM) | @noonghunna | PASS | PASS at 64K | **FAIL** — Cliff 2b fires | 2026-05-03 |
| `long-text.yml` | @noonghunna | PASS | PASS at 180K | **FAIL** — Cliff 2b fires | 2026-05-03 |
| `long-vision.yml` | @noonghunna | PASS | PASS at 145K | **FAIL** — Cliff 2b fires | 2026-05-03 |
| `bounded-thinking.yml` | @noonghunna | PASS | PASS at 180K | **FAIL** — Cliff 2b fires | 2026-05-03 |
| `tools-text.yml` | @noonghunna | PASS | PASS at 75K | **FAIL** — Cliff 2b fires | 2026-05-03 |
| `llamacpp/default` | @noonghunna | PASS | PASS at 262K | **PASS** — different engine, no cliff | 2026-04-21 |
| `dual.yml` (TP=2) | @noonghunna | PASS | PASS at 262K (237K single-prompt) | **PASS** | 2026-05-03 |
| `dual-turbo.yml` (TP=2) | @noonghunna | PASS | PASS at 262K | **PASS** (assumed by activation-split argument; not yet measured cross-rig) | 2026-04-29 |
| `dual-dflash.yml` (TP=2) | @noonghunna | PASS | PASS at 185K | TBD | — |
| `dual-dflash-noviz.yml` (TP=2) | @noonghunna | PASS | PASS at 200K | TBD | — |
| `multi4.yml` (TP=4) | @whamp | PASS | PASS at 262K (incl. 58K + 91K needles) | **PASS** (20 sessions, 0 MiB growth, 90.8% retention) | 2026-05-03 |
| `multi4-dflash.yml` (TP=4) | @whamp | PASS | PASS at 262K (incl. 58K + 91K needles) | **PASS** (5 sessions, 0 MiB growth, 100% retention; ⚠ 4 turns >30s; n=5 small) | 2026-05-03 |

The single-card vLLM Cliff 2b status is canonicalized in [#41](https://github.com/noonghunna/club-3090/issues/41) — fix is gated on upstream [Sandermage genesis-vllm-patches#19](https://github.com/Sandermage/genesis-vllm-patches/issues/19). See [docs/CLIFFS.md](docs/CLIFFS.md) for the byte-level explanation.

### Concurrency sweep — dense 27B vs 35B-A3B MoE aggregate throughput (`concurrency-probe.sh`)

First first-party aggregate matrix (2026-07-10, @noonghunna 2× 3090 PCIe, vLLM v0.24.0, `SWEEP` mode — fresh boot per N, 5 rounds, steady-state last round). **Aggregate = summed completion tokens / wall**; per-stream = median decode tok/s. All arms clean (0 errors, 0 silent-empty, 0 VRAM growth, retention ≥99%).

**Agent shape (16K-token prompt / 256 gen):**

| N | 27B `dual-fast` (MTP n=3) per-stream | ×N decode-agg | 35B-A3B `dual` (MTP off) per-stream | ×N decode-agg |
|--:|--:|--:|--:|--:|
| 1 | 87.3 | 87 | — | — |
| 2 | 51.8 | **104** ⭐ | 133.0 | 266 |
| 4 | 22.1 | 88 | 68.0 | **272** ⭐ |
| 8 | 6.8 | 54 ⚠ | 32.6 | 261 |
| 16 | — | — | 15.4 | 246 |

**Generation shape (512-token prompt / 800 gen):** 27B @ N=8 → **211 tok/s aggregate** (58.4/stream) · 35B-A3B @ N=16 → **1,037 tok/s aggregate** (92.8/stream, retention 99.4%).

Takeaways: (1) the **dense 27B's batching knee is N=2** at agent contexts — decode-aggregate *halves* by N=8 (MTP-under-batching + chunked-prefill interleave; per-stream falls faster than N grows); (2) the **A3B MoE holds ~250–270 decode-agg flat to N=16** (3B active params + tiny hybrid-attention KV) — the multi-agent serving pick; (3) at 16K contexts *end-to-end generated* tok/s is prefill-bound (~20 for 27B, ~93–121 for MoE, nearly flat in N) — batching at deep context buys utilization, not generated tokens; the >1K aggregate lives at generation-heavy shapes. See FAQ "How do I serve multiple coding agents concurrently".

### Cross-engine — Luce DFlash (lucebox-hub) on Qwen3.5-27B

Not directly comparable to vLLM rows above (different engine, different bench script, different model — Qwen3.5-27B not 3.6 because the 3.6 DFlash draft is still under training as of 2026-05-04). Bench harness: `lucebox-hub/dflash/scripts/bench_he.py`, HumanEval 10 prompts, n_gen=128.

| Config | Rig | Mean tok/s | AL | Accept % | Notes |
|---|---|---:|---:|---:|---|
| Same-card, default KV | @noonghunna (1× 3090) | 73.97 | 6.39 | 41.3% | Range 52.7–108.7 across 10 HE prompts. Bench 2026-05-04. |
| Same-card, **K8V4** (`-ctk q8_0 -ctv q4_0`) | @noonghunna (1× 3090) | 74.68 | 6.38 | 41.4% | Range 54.6–109.1. **+1% over default KV** — basically identical. KV-format optimization doesn't help at HE-scale (<150-tok prompts × 128-tok gen) where KV pool isn't the bottleneck. Asymmetric quant via [PR #56/#54](https://github.com/Luce-Org/lucebox-hub/pull/56) merged 2026-04-28. |
| Dual-GPU split ([PR #80](https://github.com/Luce-Org/lucebox-hub/pull/80) `--target-gpu 0 --draft-gpu 1 --draft-feature-mirror`) | @noonghunna (2× 3090, no NVLink, **P2P "Chipset Not Supported"**) | 75.24 | 6.39 | 41.3% | Range 54.2–110.0. **+1.7% over same-card** — but **NOT a fair test of the split's value**. CUDA P2P access is disabled at the chipset level on this rig (PHB topology, consumer-board limitation). The lucebox dual-GPU code path requires P2P for direct draft-feature transfers; without it, falls back to host-staging copies (CPU↔GPU bouncing). The published 51.86 tok/s on dual 2080 Ti 22GB ([PR #80](https://github.com/Luce-Org/lucebox-hub/pull/80)) presumably ran with P2P available. **Verdict for our hardware class**: dual-GPU split needs a P2P-capable interconnect (NVLink or peer-supported chipset) to deliver its value. PHB+CNS rigs see no benefit. |

### PFlash long-context compression on 1× 3090 — measured ceiling 131K source

Bench harness: `lucebox-hub/dflash/scripts/phase_split_dual_gpu.py bench-niah` (PFlash drafter only, no target loaded — measures the prefill compression phase). Drafter: `Qwen3-0.6B-BF16.gguf`, BSA enabled, keep_ratio=0.05.

| Source ctx | Compressed | Ratio | PFlash time | tok/s | Key + answer retained |
|---:|---:|---:|---:|---:|:---:|
| 16,372 | 788 | 0.048 | 1.08 s | 15,117 | ✓ ✓ |
| 32,764 | 1,628 | 0.050 | 1.80 s | 18,205 | ✓ ✓ |
| 65,524 | 3,252 | 0.050 | 4.37 s | 15,009 | ✓ ✓ |
| **131,068** | **6,524** | **0.050** | **10.80 s** | **12,135** | **✓ ✓** |
| 199,996 | OOM at layer 25 (390 MiB ephemeral alloc) | — | — | — | ✗ ✗ |
| 259,996 | OOM at layer 18 (507 MiB ephemeral alloc) | — | — | — | ✗ ✗ |

**Compression-phase result**: PFlash drafter scoring works up to **131K source on 1× 24 GB / 3090** — compresses to 6.5K (5%) in 10.8s with NIAH key + answer retained. Vanilla llama.cpp pp131072 takes ~257s per Luce's published numbers, so the compression phase alone is **~24× faster** at this context. Adding target prefill on the compressed 6.5K would estimated ~1-2s (untested), suggesting ~12-13s end-to-end TTFT vs ~257s vanilla.

Above 131K source the drafter's **ephemeral forward-pass tensors** (`K_curr/V_curr/Q_last` per layer at full sequence length) exceed 24 GB. K-cache quantization (`--pflash-k-type q8_0`) didn't help — the failing allocs are forward-pass not cache. Bench `lucebox-pflash-niah-q8k-20260504-150600/` confirmed identical OOM at 200K and 260K with both BF16 and q8_0 K cache.

**On the @weicj 24K → 262K phase-split claim** ([PR #78](https://github.com/Luce-Org/lucebox-hub/pull/78)): not refuted but not reproduced on our hardware class either — their setup was 2× 22 GB Ti with **target also loaded** on one card; "24K single-card" was target+drafter co-resident. Our 131K is drafter-alone on 24 GB, which already passes their dual-GPU 262K-style scaling sanity-check. Reproducing 262K specifically would need investigation of their drafter config (chunk_size, lookahead, BSA window) — drafter activation footprint at 200K+ is the binding constraint regardless of how many GPUs are present.

#### What we have NOT validated — gates before "shippable"

This is **directional evidence** (TTFT compression + NIAH retention at 131K), not a complete validation. The gates we hold every other shipped compose to are still open for PFlash:

| Gate | Status | Notes |
|---|---|---|
| TTFT speedup at long context | ✅ measured (~24× compression alone) | Single test — needs reproduction across prompt shapes |
| NIAH single-needle retrieval | ✅ measured at every ctx ≤131K | Synthetic test only — single key+answer pair per prompt |
| Target prefill on compressed tokens | ❌ unmeasured | Bench harness measures PFlash phase only |
| Decode TPS after compressed prefill | ❌ unmeasured | End-to-end TTFT + decode pipeline not tested |
| **HumanEval+ / LCB v6 pass@1** | ❌ **not applicable** — those benches have <2K-token prompts; PFlash's compression path wouldn't even engage | Need long-context coding benches (repo-understanding, RULER+code) |
| **Long-context QA accuracy** (RULER, LongBench, multi-needle) | ❌ unmeasured | The actual quality gate — does compression preserve task performance, not just synthetic needle retrieval? |
| `verify-stress.sh` 7/7 | ❌ **0/7 PASS** (2026-05-04, see below) | OpenAI server gate — multiple distinct failures |
| `SOAK_MODE=continuous` | ❌ blocked | Daemon dies during stress; soak can't run on a dead daemon |
| Multi-turn compression stability | ❌ blocked by 3rd-cycle CUDA bug | Daemon hits illegal-mem-access on 3rd compress regardless of GPU layout |

#### verify-stress on PFlash-enabled lucebox server (2026-05-04, single-card and dual-GPU)

Tested via `URL=http://localhost:8004 MODEL=luce-dflash bash scripts/verify-stress.sh` against a `lucebox-hub/dflash/scripts/server.py` boot with `--prefill-compression auto --prefill-threshold 8000 --prefill-keep-ratio 0.05` + Qwen3-0.6B-BF16 drafter. Two configurations:

- **Single-GPU**: target+dflash draft+pflash drafter all on GPU 0 → OOM at 75 MiB on 2nd request, daemon exits, all subsequent probes 503. Logs: `results/lucebox-pflash-verify-stress-20260504-152713/`.
- **Dual-GPU**: local `server.py` patch reading `LUCEBOX_TARGET_GPU=0 LUCEBOX_DRAFT_GPU=1 LUCEBOX_DRAFT_FEATURE_MIRROR=1` to pin dflash draft to GPU 1. Probes 1 (10K + 30K) survive but **return wrong needle answers** because PFlash @ keep=0.05 drops the needle phrase. Probe 2 (25K tool prefill) crashes the daemon with `CUDA error: an illegal memory access was encountered` on the 3rd compress cycle. Probes 3-7 all 503. Logs: `results/lucebox-pflash-verify-stress-dualgpu-20260504-153220/`.

| Probe | Single-GPU | Dual-GPU | Failure mode |
|---|---|---|---|
| 1. 10K + 30K needle | ✗ blank reply | ✗ wrong content | PFlash drops needle phrase from top-5% kept chunks |
| 2. 25K tool prefill | ✗ daemon dead | ✗ HTTP 500 → daemon dies | CUDA illegal-mem-access on 3rd pflash compress |
| 3. IDE-agent | ✗ HTTP 503 | ✗ HTTP 500 | Cliff 1 mech B class on lucebox path; daemon already dead in single-GPU run |
| 4-6. Multi-turn / LCB / reasoning | ✗ HTTP 503 | ✗ HTTP 503 | Daemon died at probe 2 |
| 7. 60K + 90K needle | ✗ HTTP 500 | ✗ HTTP 500 | Daemon dead |

**Two distinct failure classes in the integrated PFlash + DFlash + OpenAI server path:**
1. **Compression-vs-retrieval at keep=0.05**: short factual needles (color animal num) don't survive top-15-chunks selection in 10-30K contexts. The standalone NIAH bench (#230) used a needle/filler pattern PFlash's importance scorer favors; verify-stress's pattern doesn't replicate that. Means PFlash's "key+answer retained" claim is filler-pattern-dependent at moderate contexts.
2. **3rd-cycle multi-cycle daemon stability**: lucebox-hub upstream bug — CUDA illegal-mem-access in `ggml_backend_buffer_free` after the 3rd pflash compress cycle, regardless of single- vs dual-GPU. Daemon doesn't recover; subsequent requests 503.

**Honest read**: PFlash gives us a compelling single-stat win (24× TTFT compression + NIAH-retention) at 131K source on 1× 3090 in the **standalone bench harness**, but the **integrated OpenAI server path fails the verify-stress gate**. PFlash is **not a shippable club-3090 path today**. Re-evaluate when (a) lucebox-hub fixes the 3rd-cycle CUDA stability bug, (b) `--pflash-gpu` lands in `server.py` (currently only standalone bench has it), and (c) the importance scorer / keep ratio reliably preserves arbitrary short needles. Long-context QA harness (RULER or similar) is the meaningful next investment if any of those three land.

**Setup gotcha** for anyone re-running on consumer rigs: check `nvidia-smi topo -p2p r` before configuring `--target-gpu` / `--draft-gpu`. If the matrix shows `CNS` (Chipset Not Supported), the dual-GPU split won't deliver its claimed uplift on that hardware regardless of whether you have multiple GPUs. NVLink-bonded setups would also typically expose P2P (a different cross-rig contributor would need to confirm on lucebox specifically; @JusefPol's [#31](https://github.com/noonghunna/club-3090/pull/31) NVLink win was measured on vLLM TP=2, not lucebox). PHB-only consumer boards typically lack P2P.

---

## Qwen3.6-40B-Deckard

Dense 40B uncensored community merge of Qwen3.6 (DavidAU Opus-Deckard). Q6_K GGUF with a BF16 MTP head injected by PiehSoft (`PiehSoft/Qwen3.6-40B-Deckard-MTP-Q6_K`). Arch is `qwen35-dense` (standard GQA, 97 layers) per the GGUF header — not a Qwen3-Next hybrid. Dual 3090 only (31 GB > 24 GB). llama.cpp mainline, immutable pin `server-cuda-b9570`.

### Dual-card (2× RTX 3090) — llama.cpp

| Compose | Rig | KV | Max ctx | Narr / Code TPS | PP tok/s | Peak VRAM | Date | Notes |
| --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- |
| `mtp.yml` (MTP n=2) | @noonghunna (2× 3090, PCIe) | q8_0/q8_0 | 131072 | 36 / 46 (MTP on) | — | ~16.4+17.8 GB @16K | 2026-06-09 | ✅ Production. MTP n=2 sweet spot (41.60 tok/s aggregate, 0.81 accept). MTP off: 22.7 narr (+59%/+104% with MTP). 128K ceiling @q8_0 KV (192K OOMs). 8-pack (think-off): **105/150** — MTP-off == MTP-on (spec-dec lossless); det 62/75 (toolcall 15·instructfollow 14·structoutput 13·dataextract 10·reasonmath 10) + sandbox 43/75 (bugfind 13·hermesagent 15/20·cli-40 15/40). **verify-full 8/8**, **verify-stress 8/8**, **soak-continuous PASS** (0 MiB growth, 0/25 silent-empty, 25 turns). ≈ Qwen3.6-27B band. Arch confirmed `qwen35-dense` from GGUF header. |

---

## Tess-4-27B

Migtissera's Qwen3.5-based dense 27B instruct/agentic fine-tune (`migtissera/Tess-4-27B-GGUF`). Q4_K_M GGUF with a **separate (external) MTP draft head** — the catalog's first external-MTP compose (engaged via `--spec-draft-model … --spec-type draft-mtp`, vs Deckard's *embedded* head). Arch is `qwen35-dense` (dense = non-MoE; hybrid attention — 48 linear + 16 full-attention layers, 3:1 interleave, 64 total; the GGUF `qwen35` arch carries the hybrid internally). Vision-capable base (F16 mmproj on disk) but shipped **text-only**. llama.cpp mainline, pin `server-cuda-b9967` (bumped from b9246 in #680 — think-ON quality +4 on the engine A/B, think-OFF tie; dated rows below record the pin they ran on).

### Dual-card (2× RTX 3090) — llama.cpp

| Compose | Rig | KV | Max ctx | Narr / Code TPS | PP tok/s | Peak VRAM | Date | Notes |
| --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- |
| `mtp.yml` (ext MTP n=2) | @noonghunna (2× 3090, PCIe) | q4_0/q4_0 | 262144 | 52 / 68 | ~1325 (short) / ~1017 (90K) | 12.6+17.2 GB | 2026-07-09 | ⚠️ Production w/ caveats. External MTP (separate `mtp-*.gguf` draft). **n=2 = n-sweep sweet spot** (2026-07-09: server-side decode 57.9 tok/s / 0.62 accept peak; falls to 34.9 / 0.27 @ n=6). decode CV 2.4%, TTFT 233 ms. **verify-stress 8/8** (NIAH ladder clean to 240,634 tok = 91% of 262K, ~5.9 GB free at deepest fill). **soak-continuous PASS** (0 err, 0/100 silent-empty, p50 66.4, 96.3% retention). 8-pack (benchlocal `--full`): **115/150 think-off · 118/150 think-on** — ties/edges qwen3.6-27b **dual-max (109)** and LEADS agentic (hermesagent **15/20 vs 9** · cli-40 **25/40 vs 20**); per-pack OFF: toolcall 14·instructfollow 13·structoutput 14·dataextract 12·reasonmath 11·bugfind 11. **vs dual-max:** ~½ the throughput (52 vs ~114 TPS) for a quality tie/edge + vision base + smaller footprint (12.6+17.2 vs ~22 GB/card). ⚠️ Caveat: streaming tool-calls + thinking-ON → `finish=length` (use thinking-off for tool/agent). ⤷ 2026-07-11: no longer reproduces on the shipped b9967 + 16K-budget config (3/3 clean incl. parallel 2-tool; caveat held pending the full-gate refresh). Arch confirmed `qwen35-dense` from GGUF header. |

| `mtp.yml` (ext MTP n=2) — **b9967 re-bench** | @noonghunna (2× 3090, PCIe) | q4_0/q4_0 | 262144 | **63.4 / 73.5** (decode 64.6 / 75.9, CV <3%) | 1516 (short) / 1178 (90K) | 12.3+16.4 GB | 2026-07-11 | Fresh-boot re-bench of the shipped compose on the **b9967 pin** (#680): +22% narr / +8% code vs the b9246 row above (same protocol; settle/power conditions may differ across dates). MTP aggregate accept **0.737**. Same-session **EAGLE3-external A/B** (self-converted `club-b9967` Q4_K_M draft, [#674 recipe](https://github.com/noonghunna/club-3090/pull/674#issuecomment-4948884765), n=2, only draft+spec-type changed): **47.5 / 61.3 wall, accept 0.408** → −25% narr / −17% code — **MTP stays; no EAGLE3 compose.** Mechanism note: EAGLE3 accepts ~0.40 BOTH hidden-state-fed (vLLM, #662) and token-fed (here) → head-capability-limited, feed-robust; MTP is the reverse (0% hidden-state vs 0.74 token-fed → alignment-limited, capability-strong). A retrained/stronger head flips this — artifact kept on disk for the re-test. **Shipped-config 8-pack refresh (2026-07-12, native template + 16K budget): OFF 116/150 (TC 15·IF 13·SO 13·DE 12·RM 12·BF 12·HA 14·CLI 25) · ON 117/150 single draw (TC 14·IF 15·SO 14·DE 13·RM 13·BF 13·HA 15·CLI 20)** — no pack regression vs the 2026-07-09 basis; compose promoted ⚠️→✅ (streaming+thinking caveat non-repro 3/3 on this config). |

### Dual-card (2× RTX 3090) — vLLM

| Compose | Rig | KV | Max ctx | Narr / Code TPS | PP tok/s | Peak VRAM | Date | Notes |
| --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- |
| `vllm/tess-dual-nvfp4` (Marlin W4A16 fallback, spec-off) 🧪 | @noonghunna (2× 3090 PCIe, sm_86) | fp8/e4m3 | 131072 | 62.4 decode (spec-off, #679 gate) | — | ~23 GB/card | 2026-07-12 | **A0 quality baseline (8-pack `--full`, both modes): 106/150 think-off · 113/150 think-on** (OFF: TC 13·IF 13·SO 14·DE 12·RM **14**·BF 10·HA 13·CLI **17**; ON: TC 14·IF 15·SO 14·DE 12·RM 13·BF 14·HA 13·CLI 18). **Undershoots the GGUF bar** (shipped llama.cpp dual: 116/117) — the OFF gap is **cli-40-concentrated (−8)**, deterministic packs tie or better (RM-OFF 14/15 = best Tess RM ever measured). Note: on Ampere this is already the W4A16-*optimistic* bound (Marlin fallback never quantizes activations — native-FP4 rigs run the harsher W4A4). Per the pre-registered rule the slug stays 🧪 and the **huginnfork NVFP4A16 (weights-only recipe) pin-fallback A/B is triggered**, plus the FP8 W8A16 precision arm. Grok-§3 discriminators: model floor holds (12/16 fail both legs); CLI-31 fails BOTH vLLM legs (refusal not model-intrinsic → llama b9967 flip was engine-side); 3 of the 5 llama-only fails (DE-05/06, HA-09) PASS here → llama-path-specific, not model floor. |

### Single-card (1× RTX 4090) — llama.cpp

| Compose | Rig | KV | Max ctx | Narr / Code TPS | PP tok/s | Peak VRAM | Date | Notes |
| --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- |
| Eagle3 draft (mainline `server-cuda-b9932`, no compose yet) | @seanyourhighness (1× 4090, WSL2) | q4_0/q4_0 | 98304 | — / ~76 (code, min-p-0 sweep; not a formal `bench.sh` rebench) | — | ~23 GB (~1.5 GB free @ 98K) | 2026-07-10 | **First single-4090 Tess-4 + Eagle3 data point.** Mainline llama.cpp b9932, `Tess-4-27B-Q4_K_M` + **Tess-matched Eagle3 draft** (`Tess-4-27B-EAGLE3-Q4_K_M`, 0.39 GB) via `--spec-type draft-eagle3 --spec-draft-n-max 2`, pinned vanilla Unsloth template, q4_0/q4_0 KV, ctx 98304, b4096/u1024, `--reasoning on`. Sampler `min-p 0` (temp 0.6 / top-p 0.95 / top-k 20). **8-pack (benchlocal `--full`, packs v1.0.x): 124/150 (82.7%) think-on · 115/150 (76.7%) think-off.** Per-pack ON: toolcall 15·instructfollow 15·structoutput 14·dataextract 14·reasonmath 13·bugfind 13·hermesagent 15/20·cli-40 25/40; OFF: toolcall 14·instructfollow 13·structoutput 14·dataextract 12·reasonmath 11·bugfind 12·hermesagent 15/20·cli-40 24/40. **vs same-rig DFlash baseline (2026-07-09, min-p 0.75): 125 on / 117 off — within 1–2 scenarios (single-run noise)** → Eagle3 is quality-equivalent to DFlash (both output-lossless) and the `min-p 0.75→0` change costs no measurable quality. **`--spec-draft-n-max` sweep (this rig):** n_max=2 optimal (~76 code TPS), 4→68, 6→54, 8→47 — deeper drafting strictly *worse* for Tess-4 (deep-draft acceptance decays; rejected drafts waste target verification). n=1/scenario. Beellama never involved — DFlash *draft GGUF* ran on the same mainline b9932 draft path. **Draft provenance (reproducible):** self-converted from `migtissera/Tess-4-27B-EAGLE3` (610M BF16 safetensors) at the same b9932 checkout — `convert_hf_to_gguf.py <eagle3-dir> --target-model-dir <tess-metadata-dir>` (**metadata-only target: base config.json + tokenizer, ~20 MB — no 27B weights needed**) → BF16 GGUF (1.2 GB, bakes `eagle3.target_layers` (2,32,61) + the 32K→248K `d2t` vocab map) → `llama-quantize … Q4_K_M` (403 MB). Draft-quant is quality-free (acceptance flat 27.4–28.0% at n=4 across BF16/Q8_0/Q4_K_M) → smallest draft wins on latency. Recipe: [#674 comment](https://github.com/noonghunna/club-3090/pull/674#issuecomment-4948884765). |

---

## See also

- [docs/SINGLE_CARD.md](docs/SINGLE_CARD.md) — single-card variant picker
- [docs/DUAL_CARD.md](docs/DUAL_CARD.md) — 2-card variant picker
- [docs/MULTI_CARD.md](docs/MULTI_CARD.md) — 4+ card variant picker
- [docs/STRUCTURED_COT.md](docs/STRUCTURED_COT.md) — bounded-thinking benchmark on HumanEval+ + LiveCodeBench v6
- [docs/CLIFFS.md](docs/CLIFFS.md) — known failure modes and which variants escape them
- [CONTRIBUTING.md](CONTRIBUTING.md) — how to add a row

---

<a id="gemma-4-31b-community-experimental"></a>

## Gemma 4 31B (community-experimental)

2026-05-29 cleanup: active vLLM Gemma 31B registry is now `vllm/gemma-mtp`, `vllm/gemma-int8`, and `vllm/gemma-mtp-tp1`, pinned to immutable `vllm/vllm-openai:v0.21.0`; DFlash, AWQ, TQ3, bf16-long, and the separate 262K alias are retired from active composes.

2026-05-31 update: the two dual slugs were renamed to name the real difference (both carry MTP n=4; only the KV format differs) — `vllm/gemma-mtp` → **`vllm/gemma-bf16-mtp`** (BF16 KV) and `vllm/gemma-int8` → **`vllm/gemma-int8-mtp`** (INT8 PTH KV). `gemma-bf16-mtp` moved to stock `v0.22.0` (no overlay) and its default laddered 32K → **131K** under the dual-card max-context-priority rule ([`docs/DUAL_CARD.md`](docs/DUAL_CARD.md)). Then (#287) **`gemma-int8-mtp` ALSO moved to `v0.22.0`** — #40391 rebased onto v0.22.0 + re-delivered as a lean boot-time diff-apply (was 7 full-module mounts); default ctx raised to native **262K**; #41800/#41991 dropped (in stock); **#42006 streaming-multi-tool-fix re-instated on both duals** (lean diff). Then (#289) **`--reasoning-parser gemma4` wired on both** — without it, reasoning-on left the `<|channel>` trace in `content` → structured packs broke (`invalid_json`); it's a vLLM config gap, not a Gemma limit.

**8-pack quality (current grader, `--full`, 2026-05-31):** int8 thinking-OFF **105/150**, int8 reasoning-ON **107/150**, bf16 reasoning-ON **111/150** — all within the ±5–7 8-pack noise band (reasoning-on ≈ a wash; **InstructFollow 8→14** the one clear gain; bf16 ≈ int8 KV on quality). Reasoning-on costs latency (ReasonMath/BugFind p95 ~doubled) for no aggregate gain → **thinking-OFF stays the default**, reasoning-on is a clean opt-in once the parser is wired. Full per-pack breakdown + serving (TPS / TTFT / VRAM / KV pool) for all three configs: [discussions/239](https://github.com/noonghunna/club-3090/discussions/239#discussioncomment-17127352).

Cross-rig data on Google's official Gemma 4 MTP "assistant" drafter (released 2026-05-05). PR [#41745](https://github.com/vllm-project/vllm/pull/41745) merged 2026-05-06 → today's nightly contains it natively (overlay dropped 2026-05-08). The companion compose [`dual/int8.yml`](models/gemma-4-31b/vllm/compose/dual/autoround-int4/int8.yml) (added 2026-05-08) vendors PR [#40391](https://github.com/vllm-project/vllm/pull/40391) (rebased) + PR #42006 + PR #41991 to unlock per-token-head INT8 KV → 8.2× context lift on Ampere (32K → 262K). See announcement [discussion #67](https://github.com/noonghunna/club-3090/discussions/67) for the original Gemma 4 setup story; Phase 2 INT8 PTH validation in progress 2026-05-08.

| Compose | Rig | KV | Max ctx | Narr / Code TPS | PP tok/s | AL | Per-pos accept (code) | Peak VRAM | Date | Notes |
| --- | --- | --- | ---: | ---: | ---: | ---: | --- | ---: | --- | --- |
| **`dual/qat-awq-int4/base.yml` (TP=2, max-num-seqs=2, MAX_MODEL_LEN=229376) — v0.24.0 ⭐⭐ prod default** | @noonghunna (2× 3090 PCIe) | bf16 | **224K** | **~59 decode** (code, CV 0.1%; soak p50 58.7) | ~420–716 (depth-dep) | n/a (MTP-off) | — | ~23 GB/card (47,088 MiB total) | 2026-07-02 | **`vllm/gemma-31b-dual` — cyankiwi QAT-AWQ-int4, STOCK vLLM v0.24.0, OVERLAY-FREE.** The v0.24.0 consolidation 31b — folds onto `vllm-stable`, retires the 31b's `vllm-gemma-stable` dependence. **bf16 KV (not int8-PTH):** on v0.24.0 int8-PTH allocates 262K but **SILENTLY craters recall past ~32K** (needs #40391, open/unmerged upstream — verified 2026-07-01: both cyankiwi + w4a16 crater identically; the SAME cyankiwi weights recall clean to 112K+ on v0.22.0+#40391). bf16 has NO cliff: **verify-stress ALL 5 ceiling rungs to 210K (91%), VRAM margin 1162 MB > 1024** (the 0.97/245K thin-margin flag resolved at 0.95/224K). **verify-full 9/9** (tools + streaming-tool-calls + reasoning, MTP-off). **Soak PASS** (0 err / 0 MiB growth / 0-of-100 silent / 99.6% retention). MTP DISABLED — Gemma-4 MTP×tools broken on v0.24.0 (#39043; #42006 unmerged). Trades ~224K vs int8-mtp's 262K for **zero overlays**; int8-PTH 262K returns when #40391 merges. Supersedes gemma-int8-mtp / bf16-mtp / qat-w4a16 (all now 🗑️ Deprecated). |
| `dual.yml` (TP=2) | @noonghunna (2× 3090 PCIe, no NVLink, 230W cap) | bf16 | 32K | **108.87 / 142.25** | — | **3.94-4.04** | 92 / 79 / 68 / 59 % | 22.5 GB/card | 2026-05-05 | First Ampere consumer cross-rig data on Google MTP drafters. **+1.79× narr / +2.31× code** over baseline (61 TPS no-spec-decode same TP). **PASSES continuous soak** (100 turns, 0 errors / 0 silent-empty / 0 MiB growth, 98.3% TPS retention). bf16 KV (fp8 blocked on Ampere — see TP=1 row). PR [#41745](https://github.com/vllm-project/vllm/pull/41745) overlay + transformers 5.8.0 entrypoint. |
| `dual.yml` (TP=2) re-bench post-#41745 merge | @noonghunna (2× 3090 PCIe, 230W cap) | bf16 | 32K | **105.91 / 141.11** | — | 3.94 | (warming) | 21.5 GB/card | 2026-05-08 | Re-validated on post-merge nightly `1acd67a795...` (PR #41745 overlay dropped, transformers entrypoint upgrade dropped). Within CV of 109/142 baseline → cleanup is parity-clean. KV pool 99K tokens, 3.03× concurrency at 32K. |
| **`dual/bf16-mtp.yml` (TP=2, max-num-seqs=4, MAX_MODEL_LEN=131072)** ⭐ | @noonghunna (2× 3090 PCIe, 230W cap) | bf16 | **131K** | **118.83 / 154.26** (CV 2.1% / 2.8%) | — | 3.87-3.98 | 90 / 80 / 69 / 56 % | 22.2 GB/card | 2026-05-31 | **Max-ctx-priority default (was 32K) — `vllm/gemma-bf16-mtp` on stock v0.22.0, NO overlay.** BF16 KV pool **196,527 tok @ mem-util 0.95** (measured, vLLM boot log); `max-num-seqs=4` is a CAP not a reservation, so raising the ceiling is ~free for short-request concurrency (see [`docs/DUAL_CARD.md`](docs/DUAL_CARD.md)). 4.0× context lift over the 32K bf16 ceiling at **zero TPS cost** (118/154 ≥ the 109/142 baseline, within CV). **PASSES verify-stress 8/8** incl. NIAH at 57K / 90K / 94K / **120K** (`sapphire narwhal 21` recalled at 120,329 tok = 91% of n_ctx). ⚠️ vision + near-max ctx: only ~1.4 GB/card free at 120K single-stream → drop `max-num-seqs`/`mem-util` for vision-heavy long-ctx. |
| **`dual-int8.yml` (TP=2, max-num-seqs=4)** ⭐ | @noonghunna (2× 3090 PCIe, 230W cap) | **int8_per_token_head** | **98K** | **96.16 / 127.11** | — | 3.79 | (warming) | 22.2 GB/card | 2026-05-08 | **3.07× context lift over bf16 ceiling on Ampere — INT8 PTH KV unblocks Gemma 4 long-context.** Vendors PR #40391 rebased + PR #42006 + PR #41991 stacked (see `models/gemma-4-31b/vllm/patches/`). KV pool **354K tokens, 3.6× concurrency**. ~10% TPS cost vs bf16 / 32K. **PASSES verify-stress 8/8** incl. 91K Cliff-2 needle. PR #40391's per-token-head page-size fix routes via `get_padded_attention_kv_cache_shape()`; INT8 (not fp8) is the right Ampere dtype because Triton `fp8e4nv` kernel is not supported on sm_86 (Ada/Blackwell only). |
| **`dual-int8.yml` (TP=2, max-num-seqs=1, MAX_MODEL_LEN=262144)** ⭐⭐ | @noonghunna (2× 3090 PCIe, 230W cap) | int8_per_token_head | **262K (model native max)** | **95.27 / 125.93** | — | 3.93 | (warming) | 22.1 GB/card | 2026-05-08 | **8.2× context lift vs dual.yml — full Gemma 4 native context (262144) unblocked on dual 3090 Ampere.** KV pool 455K tokens, 1.74× concurrency at full 262K. **PASSES verify-stress 8/8** + **137K NIAH PASS** (correctly recalled needle from 137,557-token prompt, 5min wall, ~458 prefill TPS). Per-token TPS preserved at full max-model-len (95/126 at 262K vs 96/127 at 98K — bench prompt size dominates, not max-model-len). Override `MAX_MODEL_LEN=262144 MAX_NUM_SEQS=1`. |
| **`int8.yml` (TP=2, max-num-seqs=4, MAX_MODEL_LEN=262144) — v0.22.0 ⭐ prod default** | @noonghunna (2× 3090 PCIe, 230W cap) | int8_per_token_head | **262K** | **106.31 / 139.51** (CV 1.8% / 2.0%) | — | 3.68-3.94 | 88 / 73 / 59 / 48 % | 22.1 GB/card | 2026-05-31 | **`vllm/gemma-int8-mtp` on stock `v0.22.0`** — #40391 rebased onto v0.22.0 + lean boot-time diff-apply (`patches/vllm-pr40391-v0.22.0/`, was 7 full-module mounts) + #42006 streaming-tool-fix + `--reasoning-parser gemma4`. KV pool **447,199 tok**, 1.72× concurrency at 262K. **Soak (continuous) PASS** (0 err / 0 silent-empty / 0 VRAM growth / 100% TPS retention / p50 87.96). **Quality (--full, thinking-off) 105/150** (det 59/75 · sandbox 46/75); reasoning-on A/B with the parser = 107/150 (wash, +2 within noise — IF+6/BF+2 vs RM-2/HA-2 at higher latency → thinking-off stays default, reasoning-on is opt-in). TPS ≈ the v0.21.0 rows (no regression from the v0.22.0 move + overlays). |
| **`qat-w4a16/int8.yml` (TP=2, n-swept default n=3, MAX_MODEL_LEN=262144) — 🧪 QAT-int4 A/B** | @noonghunna (2× 3090 PCIe, 370W) | int8_per_token_head | **262K** | **73.96 / 87.72** (CV 1.7% / 1.4%) | — | **~2.3** | 64 / 39 / 25 % | ~22.8 GB/card (KV 419K tok) | 2026-06-07 | **`vllm/gemma-31b-qat-w4a16-dual` — unsloth QAT W4A16 (compressed-tensors int4) A/B vs the autoround-int4 `int8.yml`.** Boots clean on stock `vllm-gemma-stable` (NO #44494 workaround — tower-based `Gemma4ForConditionalGeneration`, unlike the 12B unified arch). **8-pack 109/150 vs autoround's 105 (+4, within ±5-7 noise ≈ quality TIE; real instructfollow edge IF 15-vs-8, offset by hermes/cli/TC/RM).** BUT **weaker spec-decode: AL ~2.3 vs autoround's ~3.9** — the QAT-int4 logits align worse with the assistant drafter, so it's slower (74/88 @370W < autoround's 106/139 @230W *despite MORE power*; power NOT matched, but the AL gap is the clean signal). **n-swept @370W: n2 72.9/86.1 · n3 74.0/87.7 · n4 71.6/87.8** — all within ~3% (the fast acceptance decay caps the spec benefit); **n=3 set as the default** (top narr + tied code + ~20% less drafting than autoround's n=4). **Comparable quality, slower → autoround-int4 (`gemma-int8-mtp`) stays the default.** Boots/serves only; NIAH + soak not yet run. |
| `dual-dflash.yml` (TP=2, n=7) | @noonghunna (2× 3090 PCIe, no NVLink, 230W cap) | bf16 | 32K | **95.16 / 167.55** | — | **~3.0 narr / 5.23 code** | 89 / 78 / 66 / 57 / 50 / 43 / 39 % | 22.7 GB/card | 2026-05-06 | First Ampere consumer cross-rig data on **z-lab Gemma 4 DFlash** block-diffusion drafter (vLLM PR [#41703](https://github.com/vllm-project/vllm/pull/41703) — Codex-rebased onto upstream/main). **+2.74× code / +1.56× narr** over baseline. **PASSES continuous soak** (100 turns, 0 errors / 0 silent-empty / 0 MiB growth, 98.6% TPS retention, p50 55.8 TPS — 5.8% higher than n=5). DFlash dominates MTP on **code (+18%)**; MTP wins on narrative (+15%). n-sweep: n=5 109/141 (best narr) → n=6 99/161 (knee) → **n=7 95/168 (code-optimal default)** → n=8 91/167 (dominated) → n=15 82/172 (past knee). PR #41703 overlay (12 RO-mounted files) + transformers 5.8.0 + nightly `e47c98ef`. |
| `dual-dflash.yml` (TP=2, n=7) re-bench | @noonghunna (2× 3090 PCIe, 230W cap) | bf16 | 32K | **104.48 / 176.66** (CV 2.1% / 3.6%) | — | 2.85 narr / 4.11-4.94 code | (warm) | 22.3 GB/card | 2026-05-08 | Re-validated after Phase 2 INT8 PTH session. Same overlay + same `e47c98ef` pin — modest uplift over 2026-05-06 (warm-cache + ambient variance — CV ranges overlap at +1σ). KV pool 42,848 tokens, 1.31× concurrency at 32K. Ampere upper-bound for Gemma 4 + DFlash: code-optimal at 177 TPS. Combining DFlash drafter with PR #40391 INT8 PTH KV (Phase 3, dual-dflash-int8.yml) is the next structural step — would unlock long-context code-optimal. |
| **`dual-dflash-int8.yml` (TP=2, n=7, MAX_MODEL_LEN=262144, MAX_NUM_SEQS=1)** ⭐⭐⭐ | @noonghunna (2× 3090 PCIe, 230W cap) | int8_per_token_head + drafter bf16 | **262K (model native max)** | **86.86 / 145.96** (CV 0.9% / 2.2%) | — | 5.0-5.3 long-ctx code | (warm) | 22.0 GB/card | 2026-05-08 | **8.2× context lift over `dual-dflash.yml` 32K bf16 baseline — DFlash + INT8 PTH KV unblocked on Ampere via [vLLM PR #42102](https://github.com/vllm-project/vllm/pull/42102) (our patch).** Matches the 32K bf16 baseline's code TPS within CV at 8× more context (146 vs 168 = -13% perf cost for 8× ctx). KV pool 168,178 tokens, 0.64× concurrency at full 262K — effective single-stream serving ceiling ~168K. **NIAH PASS at 98,444 tokens** (`bronze octopus 17` recalled cleanly, 157s wall = ~625 effective prefill TPS). DFlash drafter uses BF16 KV in independent pool (target uses INT8 PTH); the patch partitions them at unify-time, drafter cache_dtype overridden to "auto" in qwen3_dflash.py, FA metadata scheduler reads per-spec dtype. **n-sweep at 262K config 2026-05-08**: n=5 81.91/138.87 (-6/-5%), n=7 86.86/145.96 (default), n=8 86.63/152.06 (+0/+4% but CV 5.7% — within noise). n=7 retained as default — sweet spot didn't shift meaningfully from the 32K bf16 baseline. **DFlash code-optimal advantage preserved at long ctx**: 146 code TPS vs dual-int8.yml's 126 at 262K = +16% code (offset: -10% narr). Pin nightly `e47c98ef`; needs `dual-dflash-int8.yml` compose (still ⚠️ flagged DOES NOT BOOT until PR #42102 lands — currently requires the vllm-src patch mounts). |
| `single.yml` (TP=1) | @noonghunna (1× 3090) | bf16 / fp8 | — | **boot OOM** | — | — | — | — | 2026-05-05 | **Upstream-blocked on Ampere consumer.** bf16 KV: weights+drafter+profiling at 8K ctx + mem-util 0.95 leaves zero KV pool ("No available memory for the cache blocks"). fp8 KV: Triton `fp8e4nv not supported in this architecture` on sm_86 (Ampere supports `fp8e4b15`/`fp8e5` only); but `fp8_e5m2` is rejected by `gemma4_mm.py:1336` allowlist. Compose preserved for re-test when (a) vLLM adds Ampere-aware fp8 dispatch OR (b) PR #41745 relaxes the assert. Gemma 4 26B-A4B MoE single-card is the obvious follow-up. |
| **`dual-awq.yml` (TP=2, MAX_NUM_SEQS=4, MTP n=4)** ⭐ | @noonghunna (2× 3090 PCIe, 230W cap) | bf16 | **65K** | **104.59 / 130.56** (CV 1.7% / 0.4%) | — | 3.07 narr / 3.55-3.88 code | 0.79 / 0.59 / 0.43 / 0.31 | 19.8 GB/card | 2026-05-08 | **Cross-rig reproducer of @3dluvr's [#103 bench](https://github.com/noonghunna/club-3090/issues/103) — AWQ-4bit weights instead of AutoRound INT4.** Bypasses PR #40391 (per-token-head bug) entirely because AWQ doesn't use FP8 KV — trades weight quant precision for ctx instead of trading KV precision. Vendor: `cyankiwi/gemma-4-31B-it-AWQ-4bit` (~17 GB on disk, AWQ-pack-quantized group_size=32, asymmetric, MSE observer; routed via vLLM compressed-tensors loader → Marlin kernel). KV pool 89,228 tokens, 1.36× concurrency at 65K. Numbers comparable to dual.yml (BF16 INT4 AutoRound, 105.91/141.11) on narrative; slightly slower code at this n. Default config — multi-stream agent / RAG. |
| **`dual-awq.yml` (TP=2, MAX_NUM_SEQS=1, MTP n=8, MAX_MODEL_LEN=118304)** ⭐⭐ | @noonghunna (2× 3090 PCIe, 230W cap) | bf16 | **118K** | **101.16 / 141.90** (CV 2.5% / 0.4%) | — | 3.7 narr / 5.13 code | 0.89 / 0.77 / 0.64 / 0.54 / 0.45 / 0.37 / 0.26 / 0.21 | 19.8 GB/card | 2026-05-08 | **3.7× context lift over dual.yml's BF16 32K ceiling.** Closer match to @3dluvr's #103 anchor (113/163 at 195K with `--dtype half --async-scheduling --cudagraph_capture_sizes [9]`). Our config: `--dtype bfloat16`, default cudagraph capture sizes `[1,2,4,8,16]` (auto-clamped to single-stream). vLLM auto-estimated 195K won't fit at 0.85 mem-util (10.18 GiB KV needed vs 7.25 GiB available); 118K is the achievable ceiling at this mem-util / cudagraph budget. **NIAH PASS at 88K** (recalled "bronze octopus 17" from 88,527-token prompt, 11-tok completion in 127.1s wall, ~696 effective prefill TPS). KV pool 118,304 tokens, 1.00× concurrency. Code AL 5.13 (n=8 saturates well on code, less so on narr where AL is 3.7). **n=8 vs n=4 trade**: code +9% (130 → 142), narrative -3% (104 → 101) — n=8 dominates n=4 for code. Override `MAX_MODEL_LEN=118304 MAX_NUM_SEQS=1 MTP_N=8`. **A/B finding 2026-05-08** — three tuning flags tested individually on this rig (with bench n=3 each, sync-baseline 101.16/141.90): |
| Flag | Narr Δ | Code Δ | Verdict |
|---|---:|---:|---|
| `--dtype half` (vs bfloat16) | -3% | -2% | Ampere has NO fp16 hardware accel on sm_86 — bfloat16 is canonical |
| `--async-scheduling` | ~0% | ~0% | within CV noise (CV 6.3% on narr) |
| `cudagraph_capture_sizes [9]` | -3% | +2.5% | single-shape capture optimizes n=8 hot batch (1 base + 8 spec = 9), hurts narrative-dynamic patterns |
None close the **-13% narr / -11% code gap to 3dluvr's anchor**. Remaining gap likely rig-specific (3dluvr's EPYC 7J13 / different PCIe topology / 275W cap / etc) rather than tunable via flags. Cross-rig productionizable settings: stick with default `--dtype bfloat16`, default cudagraph, default scheduling. Custom override `cudagraph_capture_sizes [9]` worth it ONLY if you serve >95% n=8-MTP-single-stream code traffic (e.g. dedicated coding-agent endpoint) where the +2.5% code lift exceeds the -3% narrative loss. |
| `dual.yml`-shape forced TP=1 | @apnar (1× **RTX 5090** 32 GB, air-cooled, 600 W) | bf16 | 32K | **159.67 / 215.10** (decode 160.71 / 217.30) | 27.5 GB | 2026-05-07 | **First single-5090 Gemma 4 MTP data point.** First non-OOM single-card Gemma 4 result on the matrix — the 32 GB Blackwell envelope clears the 24 GB Ampere boot OOM. CV 1.9%/1.8%, peak 426 W. **+46% narr / +51% code over @noonghunna's 2× 3090 TP=2 baseline (109/142)** — single-card 5090 beats dual-3090 on Gemma 4. [Disc #67](https://github.com/noonghunna/club-3090/discussions/67#discussioncomment-16832042). |
| `dual-dflash.yml`-shape forced TP=1 (mem-util 0.96, max-model-len 12000) | @apnar (1× **RTX 5090** 32 GB, air-cooled, 600 W) | bf16 | **12K** | **150.40 / 261.06** (decode 151.16 / 264.62) | 28.8 GB | 2026-05-07 | **First single-5090 Gemma 4 DFlash data point.** Trade vs MTP row above: ~6% narr loss, **+21% code lift** (215→261). 1st-warmup TTFT outlier (73 s) suggests cudagraph warmup taking longer on first request; subsequent warmups stable at <40 ms. CV 3.6%/2.8%, peak 440 W. **Required mem-util 0.96 + max-model-len 12K** to fit BF16 weights + DFlash N=5 drafter on 32 GB — DFlash drafter footprint pushes out ctx ceiling vs MTP's 32K. [Disc #67](https://github.com/noonghunna/club-3090/discussions/67#discussioncomment-16832042). |

### Single-card (1× 3090) — llama.cpp (no vLLM)

Two single-3090 llama.cpp paths, measured 2026-05-27 (`bench.sh`, n=5; full writeup [discussion #239](https://github.com/noonghunna/club-3090/discussions/239)). Both: `gemma-4-31B-it` Q4_K_S + `q5_0`/`q4_1` KV; SWA-windowed KV scales context to ~131–200K. **beellama is now the shipped single-card Gemma-4 default** (⚠️ caveats — thin VRAM margin at the 128K ceiling; served via Anbeeld's OFFICIAL image since the v0.3.2-preview re-pin — sm_120 absent from official builds, upstream ask [Anbeeld#85](https://github.com/anbeeld/beellama.cpp/issues/85); see [docs/UPSTREAM.md](docs/UPSTREAM.md)). The mainline-`FA_ALL_QUANTS` row is the no-fork reference (from-source build); re-point the default to it when [llama.cpp#23398](https://github.com/ggml-org/llama.cpp/pull/23398) Gemma-4 MTP merges.

| Config | Rig | KV | Bench ctx | Narr / Code TPS | PP tok/s | DFlash accept | 8-pack (think off / on) | Peak VRAM | Date | Notes |
| --- | --- | --- | ---: | ---: | ---: | --- | --- | ---: | --- | --- |
| **mainline llama.cpp + `FA_ALL_QUANTS`** (no spec-dec) | @noonghunna (1× 3090) | q5_0/q4_1 | 65K | **36.8 / 36.8** | ~277 | — | **109/150** / — | 19.2 GB | 2026-05-27 | **The no-fork supported path.** Built from source with `-DGGML_CUDA_FA_ALL_QUANTS=ON` — the head_dim=512 quant-KV FA kernel the stock `ggml-org/llama.cpp` image omits (→ ~12 TPS otherwise). CV 0.1%, TTFT 0.3 s. **⚠️ DOES NOT REPRODUCE (2026-05-28):** a re-bench of this exact config (same from-source binary, same q5_0/q4_1 KV, same 65K) measured **13.6 / 15.4 decode TPS** — ~2.5× below this row. Ruled out as causes via on-rig diagnosis: launch flags (`--reasoning auto --jinja` A/B = no effect), throttle/thermal/power-cap (all inactive, 320W of a 370W cap, 69°C), contention (sole process), and memory-clock starvation (ran 9501 MHz ≈ stock max). The LACT +1000-mem/+200-core OC was **off** this session (would explain only ~15–20%, not 2.5×). 43–59% GPU util during decode points to an unexplained per-token stall. **Number needs re-validation under a known/controlled rig state (OC on, fresh GPU) before it's trusted** — do not propagate. Surfaced incidentally while validating benchlocal-cli #59. |
| **beellama.cpp + DFlash** ⭐ (single-card default) | @noonghunna (1× 3090) | q5_0/q4_1 | 100K | **47 / 88** | ~855 | ~20–80% (workload) | 109/150 / **114/150** | 21.8 GB | 2026-05-27 | **Single-card Gemma-4 default (⚠️ caveats)** — the only viable fast single-card path (vLLM FA-walled at head_dim=512, ik walls ~24K, stock llama.cpp ~12 TPS). DFlash external drafter — **1.28× narr / 2.40× code** over mainline; TTFT ~0.1 s. TPS thinking-neutral (47/88 both modes). Thinking-ON nets **+5** on the 8-pack (agentic gains; reasonmath −1 noise). Served via unofficial multi-arch image `beellama-cpp:multiarch-b9459-07ac3ce` (sm_86/89/120; **sm_89/120 compiled-not-validated**). CV 3–6%. |
| **beellama.cpp + DFlash** ⭐ re-gate on Anbeeld OFFICIAL v0.3.2-preview (slug `beellama/gemma-dflash`) | @noonghunna (1× 3090, 370 W) | q5_0/q4_1 | **128K** (compose default) | **44.91 / 79.76** (decode, n=5, CV 2.1% / 5.6%) | 949 @10K → 603 @90K (client-observed probe) | — (not captured this gate) | **108/150** / **113/150** | 23.0 GB · 929 MB free @117K ceiling | 2026-07-04 | **First gate on the OFFICIAL engine pin** (digest `858e7cf…` = the current `beellama-local` pin) — replaces the 2026-05-27 row above as the slug's reference; rebench tag `gemma-dflash-regate`. verify-full ✓ · NIAH ladder **clean to 117,513 tok** (91% of 128K) · ceiling margin **929 MB** — still < the 1024 MB agent-safety bar (⚠️ caveat stays) but up from ~177 MB on the 2026-05-31 validation → v0.3.2 freed ~750 MB at ceiling · soak PASS (p50 59.8, 100.6% retention, 0 growth, 0/100 silent-empty) · TTFT 115 ms. 8-pack on the CURRENT harness (post benchlocal #79/#81 — not comparable to the 2026-05-27 numbers): think-ON +5 = in-band wash (bugfind 15/15 + hermes 14/20 lift, dataextract −1) → think-off default re-confirmed. vs the old row: −4% narr / −9% code (different image + 128K-vs-100K boot + harness era; narr within CV, code delta unresolved at CV 5.6%). **Baseline row seeded** → `baselines.yml` (`source_tag: gemma-dflash-regate`). First gate through the fixed `rebench-full` protocol n=5 (the old `RUNS:-3` default under-ran every prior orchestrated gate). |


## Gemma 4 12B (gemma4_unified — experimental)

New `gemma4_unified` arch (vLLM PR #44429 / llama.cpp #24077, merged 2026-06-03). Card advertises 256K; HF config trains to 131072 but the global layers' **Proportional RoPE extrapolates cleanly past it** — reachable on all three engines once the engine stops sizing the RoPE cache to `max_position_embeddings`. All rows 2026-06-04, @noonghunna, NIAH = needle-in-haystack exact recall. Quality 8-pack pending.

### Single-card (1× RTX 3090) — 256K via `--override-kv`
| Compose | Rig | KV | Max ctx | Narr / Code TPS | PP tok/s | Peak VRAM | Date | Notes |
|---|---|---|---|---|---|---|---|---|
| `llamacpp/gemma-12b-single-q8kxl` ⭐ | 1× 3090 | q8_0 | **256K** | **51.3 / 51.4** | ~1000 | ~16 GB | 2026-06-04 | Mainline `ggml-org/llama.cpp:server-cuda` + Unsloth Q8_K_XL. 256K via `--override-kv gemma4.context_length=int:262144` (lifts llama.cpp's `n_ctx_train` cap; native p-RoPE extrapolates — **NOT** YaRN, gemma4 ignores rope-scaling). **NIAH clean at 150K + 200K + ~246K.** No spec-dec (gemma4-assistant MTP arch unmerged, llama.cpp #23398). Fastest single-card 256K path. |
| `beellama/gemma-12b-single-q8kxl` | 1× 3090 | q5_0/q4_1 | **256K** | 47.0 / 46.9 | ~990 | ~15 GB | 2026-06-04 | beellama v0.3.0 stable + Q8_K_XL turbo KV, same override-kv 256K (NIAH 200K + 246K ✓). KV-quant is **TPS-neutral** (q8_0 = 47.7 ≈ q5_0/q4_1); the ~9% gap vs mainline is engine, not KV. No spec-dec; no DFlash drafter for the 12B. |
| `vllm/gemma-12b-single-int8-mtp` | 1× 3090 | int8 AutoRound | **256K** | **117 / 122.5** | KV 310,328 tok · 1.18× | ~20.7 GB | 2026-06-04 | Intel AutoRound INT8 (W8A16) + assistant external drafter on stock `vllm/vllm-openai:gemma4-unified` (no overlay). **Full 256K fits ONE card** (gemma4 KV ~32 KB/tok; drafter resident). NIAH exact-recall 140K/170K/200K/230K/241K. n-sweep (code-gen): n2 96.7 / n3 115.5 / **n4 117.0** / n5 122.5 TPS, accept_len 2.62/3.33/3.67/4.06, vs ~50 undrafted. Default n=4 (`SPEC_N=5` code-max). TTFT 65 ms. **Quality 8-pack 105/150 (70%) think-OFF** (toolcall 13 · instructfollow 14 · structoutput 14 · reasonmath 13 · dataextract 10 · bugfind 14 · hermesagent 11/20 · cli 16/40) — **on par with the bf16 dual's 94/150** (INT8 ≈ bf16, no degradation; spread is agentic-pack variance). bf16 KV only. **⚠️ Production w/ caveats** — ephemeral arch-preview image (pin a digest; → ✅ on a stable vLLM gemma4_unified release). |

### Dual-card (2× RTX 3090, TP=2) — vLLM `gemma4-unified` (stock; p-RoPE overlay dropped 2026-06-04, config fix)
| Compose | Rig | KV | Max ctx | Narr / Code TPS | Concurrency @256K | Peak VRAM | Date | Notes |
|---|---|---|---|---|---|---|---|---|
| `vllm/gemma-12b-dual-bf16-mtp` ⭐ | 2× 3090 PCIe | bf16 | **256K** | **76.4 / 120.9** | KV 426,382 tok · 1.63× | ~22 GB/card | 2026-06-04 | Stock `vllm/vllm-openai:gemma4-unified` (no overlay — 256K from the upstream config fix vllm#39914, overlay dropped PR #309) + assistant MTP drafter (`gemma-4-12B-it-assistant`, n=4): **+61% narr / +154% code** vs undrafted (47.5/47.5). AL 2.4 narr / 3.7–4.0 code; accept ~36% / ~70%. **The only 256K + MTP dual path.** NIAH 150K+200K exact recall; TRITON_ATTN (head dims 256/512). **Quality (rebench-full 2026-06-04):** 8-pack 94/150 (63%) think-OFF · 85/150 (57%) think-ON; **soak PASS** (0 err · 0/100 silent-empty · 0 MiB growth · p50 115.7 TPS · 94.8% retention). **⚠️ Production w/ caveats** — ephemeral arch-preview image (pin a digest; → ✅ on a stable vLLM gemma4_unified release). |


---

## DiffusionGemma 26B-A4B (dLLM — experimental)

Block-diffusion LLM (`DiffusionGemmaForBlockDiffusion`), not autoregressive — it emits **blocks** of tokens per step, so decode TPS reads much higher than AR models and is high-variance (entropy-dependent). Runs on Ampere via **stock `vllm/vllm-openai:v0.24.0`** (DiffusionGemma arch native — PR #45163 merged 2026-06-12; bumped 2026-07-02 off the former `:gemma` branch digest) + our 3 bind-mounted Ampere/TP fixes (Marlin-K-pad ×2 + diffusion_gemma TP-vocab/dtype) — the Marlin-pad is what clears the sm_86 FP8 shared-mem tile wall (`Padding FP8 MoE Marlin intermediate 352→384`), still needed because #45295's dense marlin-pad (native in v0.24.0) doesn't cover the `marlin_moe_wna16` MoE fp8 path. Slug `vllm/diffusiongemma-dual` (🧪, `--force`).

| Config | Rig | KV | Max ctx | Narr / Code TPS | KV pool | Peak VRAM | Date | Notes |
| --- | --- | --- | ---: | ---: | --- | ---: | --- | --- |
| `vllm/diffusiongemma-dual` (TP=2, fp8) — **maintainer** | @noonghunna (2× 3090 **PCIe, no NVLink**) | bf16 | 262K | ~177 / 180 (peak ~1100 low-entropy) | 413K tok · 1.58× | ~23 GB/card | 2026-06-11 | Reference numbers from the shipped compose `status_note`. dLLM block-diffusion; 8-pack 100/150 (5-pack 84%); NIAH→250K; eager-only; `max_new_tokens` 16384 (self-terminates ~1.2–1.8K words). |
| `vllm/diffusiongemma-dual` (TP=2, fp8) — **cross-rig ⭐** | @steamEngineer (2× 3090 **NVLink** NV4, X299 i9-10920X quad-channel, PCIe 3.0 ×16, Proxmox LXC, 350 W) | bf16 | 262K | **294 / 370** (decode, n=5, CV 8.5% / 21%; wall 209 / 242) | 413,470 tok · 1.58× | ~23.1 / 22.9 GB/card | 2026-06-14 | **First cross-rig confirmation of the shipped compose** ([#405](https://github.com/noonghunna/club-3090/issues/405)) — same image digest `9c719fc…` (vLLM `0.22.1rc1.dev357`), our Marlin-pad mounts visible working in the boot log. **Decode notably above the PCIe reference (294 vs ~177)** — plausibly NVLink lifting the TP=2 all-reduce, though block-diffusion variance is high (CV up to 21%), so not a clean attribution. TTFT ~1.4 s narr / ~1.1 s. `verify-stress` 8/8 (NIAH usable to **184K**, quality ceiling 214K — recall miss at 81%); `verify-full` 8/9 (the lone fail is the SSE-chunk check — a **false-fail for block-diffusion**, which emits whole blocks, not token-by-token; the text was correct). Soak not run (was blocked by the auto-detect bug, fixed in #414). Agentic: O(n)-like TTFT growth + 3/12 tool-call misses at depth. 8-pack not run. |

<a id="moe-models"></a>

## MoE models (v0.7.3 — preview track)

First-pass numbers for the two MoE models onboarded in v0.7.3. **Preview** track = `vllm-nightly-clean` engine (no Genesis patches, no TQ3 KV, no MTP yet) — exercises the upstream MoE loader [PR #42521](https://github.com/vllm-project/vllm/pull/42521) (qwen3_5_moe weight loading, merged 2026-05-14) without other overlays. Production track (Genesis-anchored, MTP, longer context) lands in v0.7.4 after Genesis v7.73.x re-anchors on a post-#42521 nightly. **Update 2026-05-30:** the **dual** path is now promoted to Production (`vllm/qwen-35b-a3b-dual`, **262K + vision**, vLLM **v0.22.0** stable) — Genesis proved *unnecessary* here (DeltaNet-hybrid KV is cheap, so 262K fits TP=2 with 7.8× concurrency) and built-in MTP is net-negative on vLLM at every depth tested. See the ⭐ PRODUCTION row below.

| Compose | Rig | KV | Max ctx | Narr / Code TPS | PP tok/s | AL | Per-pos accept | Peak VRAM | Date | Notes |
| --- | --- | --- | ---: | ---: | ---: | ---: | --- | ---: | --- | --- |
| **`qwen3.6-35b-a3b/dual/preview.yml` (TP=2)** ⭐ | @noonghunna (2× 3090 PCIe, 230 W cap) | fp8_e5m2 | 16K | **182.68 / 177.45** (decode 186.98 / 186.90) | — | n/a (no drafter) | n/a | **21.94 GB/card** | 2026-05-15 | **First v0.7.3 MoE preview row on the matrix.** Engine `vllm-nightly-clean` (nightly `bf610c2f`, post-#42521). No Genesis, no MTP — exercises the upstream qwen3_5_moe loader cleanly. CV **0.3% / 0.9%** (very stable). TTFT 126 ms. GPU 0 at 99% util / 292 W, GPU 1 at 85% / 244 W. Decode TPS basically identical narr vs code (187 / 187) — characteristic of MoE memory-bandwidth-bound decode (3 B active params per forward, weights fit cache). **~2× the Qwen 3.6 27B dense `dual.yml` baseline** (89/118 wall) on the same hardware — MoE's active-params advantage on Ampere. Next steps: TQ3 KV + MTP (built-in head) after Genesis v7.73.x lands; longer ctx after upstream MoE expert dispatch overhead is measured. Compose: `models/qwen3.6-35b-a3b/vllm/compose/dual/autoround-int4/preview.yml` (since promoted → `dual/autoround-int4/fp8.yml`, see row below). |
| **`qwen3.6-35b-a3b/dual/fp8.yml` (TP=2, vision, no-MTP) ⭐ PRODUCTION** | @noonghunna (2× 3090 PCIe) | fp8_e5m2 | **262K** | **178.5 / 173.7** (decode 182.3 / 182.3) | — | n/a (no drafter) | n/a | **22.1 GB/card** | 2026-05-30 | **Promoted preview → Production (slug `vllm/qwen-35b-a3b-dual`, engine vLLM `v0.22.0` stable, no overlays).** Full gate: max-ctx probe (**262K fits, 7.81× concurrency, 2.05M-token KV pool** — DeltaNet-hybrid KV is cheap, so the 16K preview cap was ~16× too conservative; kv-calc's MoE weight-anchor was wrong, real INT4 weights ≈10 GB/card not 20); `verify-stress` **NIAH-clean to 240K** (91%, no Cliff-2); `soak-continuous` **PASS** (0 growth / 0 err / 0 silent / 100% retention); quality `--full` det **79/90 (88%)** · hermes 11/20 · cli 14/40 · aider-polyglot **13/30 (43%)** (≥ the ik-llama `apex-fit-q8q5` ref); live **vision** smoke (read a test image correctly @ 262K). CV <0.5%. **MTP REJECTED at n=2 *and* n=3** (n2 ≈ −45% / n3 ≈ −51% wall TPS despite AL 2.4-3.4 — the MoE built-in-MTP draft forward pays inter-GPU TP=2 sync the acceptance can't amortize; ik-llama avoids this by running single-card). Served drafter-less; the built-in-MTP A/B compose (`fp8-mtp.yml`) has since been removed — net-negative on this MoE at vLLM TP=2, finding recorded in this row. Compose: `models/qwen3.6-35b-a3b/vllm/compose/dual/autoround-int4/fp8.yml`. |
| **`qwen3.6-35b-a3b/single/nvfp4/fp8.yml` (TP=1, NVFP4, no-MTP) — ✅ FIRST NVFP4 VALIDATION** | @paulp83 (1× RTX 5090 32 GB, 600 W, PCIe x4) | fp8/e4m3 @ scale=1.0 | 131K | **251.8 / 251.1** (decode **255.8 / 257.9**, n=5, CV <1%) | ~10.8K @90K (TTFT 6.3 s) | n/a (no drafter) | n/a | **30.6 GB** | 2026-07-07 | **First community boot of the whole NVFP4 tier** ([#619](https://github.com/noonghunna/club-3090/issues/619)). nvidia modelopt NVFP4 MoE (NVFP4 expert FFNs + FP8 attention) on vLLM **v0.24.0** (quant=`modelopt_mixed`), fp8 KV. `verify-full` **9/9** (tool-calls + streaming + reasoning), `verify-stress` needle-clean (9.8K + 29K), `soak-continuous` **PASS** (15 MiB growth / 100% retention / 0 err / 0 silent-empty / p50 311 TPS). TTFT 60 ms. **~2.5–3× our 2× 3090 AutoRound `dual` tier** — native FP4 GEMM + Blackwell bandwidth on a 3B-active MoE. **MTP OFF** (built-in head net-negative on this MoE; `speculative_config=None` confirmed as shipped). **fp8 KV runs scale=1.0** (checkpoint declares FP8 KV but ships no scale tensors — the #594-tied regime). VRAM **30.6/32 GB @131K** — tight but flat through soak on a 32 GB card (MAX_MODEL_LEN=98304 if a longer soak OOMs). **8-pack quality NOT run** (benchlocal sandboxes weren't built) — owed; stays 🧪 until it lands. Compose: `models/qwen3.6-35b-a3b/vllm/compose/single/nvfp4/fp8.yml`. Registry: `vllm/qwen-35b-a3b-single-nvfp4`. |
| **`unsloth/Qwen3.6-35B-A3B-NVFP4-Fast` (TP=2, no drafter) — first MoE-FP4-on-Ampere boot; no catalog compose yet** | @noonghunna (2× 3090 PCIe) | fp8 (e4m3; checkpoint's calibrated k/v scales **LOAD and apply** — in-worker verified 2026-07-11 on the 27B sibling) | **262K** | **175.5 / 172.2** (decode **179.5 / 179.4**, n=5, CV ≤0.8%) | **8289 @10K · 5155 @90K** (TTFT 1.2 s / 17.9 s) | n/a (MTP head shipped unquantized but OFF — built-in MTP net-negative on this MoE at TP=2, see `dual/fp8.yml` row) | n/a | **22.46 GB/card** | 2026-07-11 | **STATISTICAL TIE with the shipped AutoRound `dual` tier on BOTH axes** — decode 179.5 vs 182.3 (−1.5%), 8-pack think-off **103/150** (TC 14 · IF 13 · SO 14 · DE 12 · RM 11 · BF 12 · HA 7/20 · CLI **20/40**) vs the tier's 104-equivalent (det 79/90 + HA 11 + CLI 14; 2026-05-30) — det/HA within noise, **cli-40 +6 = best cli seen on this MoE**; 4 of 13 HA fails were 300 s runner-timeouts, not wrong answers. compressed-tensors **mixed** recipe (FP8 W8A8 attention/linear_attn/lm_head + **true W4A4 NVFP4 expert FFNs**), vision unquantized. On Ampere runs weight-only via **`MARLIN` NvFp4 MoE backend** (first confirmed boot of that path) + FLASHINFER attention — so this is the W4A16 execution bound: **native-FP4 rigs quantize activations too (W4A4 ≈ 2× KLD on this family — huginnfork) and need their own quality run**. Unlike the dense 27B (where Marlin-NVFP4 costs ~20%), the A3B MoE's decode is barely weight-bandwidth-bound → the fallback penalty ~vanishes. Boot: TP=2, `--kv-cache-dtype fp8`, quant auto-detect (compressed-tensors — NOT `modelopt`), froggeric template pinned, util 0.92. |
| **`unsloth/Qwen3.6-27B-NVFP4` (TP=2, MTP n=3) — provider A/B vs the nvidia export; no catalog compose** | @noonghunna (2× 3090 PCIe) | fp8 (e4m3; checkpoint's 32 calibrated k/v scales **LOAD and apply** — in-worker `apply_model` probe, all 16 full-attn layers at shipped values 0.023–0.271) | **262K** | **66.3 / 87.9** (decode **66.9 / 90.4**, n=5, CV 4.6–6.4%) | **1618 @10K · 1240 @90K** | n=3 (window artifact prevented a clean accept %) | (not cleanly captured) | **23.55 GB/card** | 2026-07-11 | **PROVIDER TIE with nvidia/Qwen3.6-27B-NVFP4 on every axis** — decode 66.9/90.4 vs 69.7/85.5 (−4%/+6%), 8-pack think-off **109/150** (TC **15** · IF 13 · SO 14 · DE 13 · RM 12 · BF 12 · HA 9/20 · CLI 21/40) vs nvidia's 110 and the fp8 tier's 109 (three-way statistical tie), **NIAH clean to 240,634 tok (91% of 262K, all 6 rungs)** with the thin-margin advisory at ceiling (573 MB free). compressed-tensors mixed recipe (FP8-dynamic attention + W4A4 NVFP4 FFNs except layers 56–63 kept FP8; unquantized bf16 mtp.*); quant auto-detects (NOT `--quantization modelopt`). **First observed MTP head loading through the compressed-tensors path** (`qwen3_5_mtp` n=3, spec=on first boot). Marlin W4A16 fallback → W4A4 activation loss not exercised (native-FP4 rigs need their own quality run). **Catalog verdict: no second 27B-NVFP4 slug** (equal-not-better; lean-catalog rule) — the nvidia-pinned `vllm/qwen-27b-dual-nvfp4` stays, unsloth's runs BYO via pull.sh; revisit if the shipped KV scales are verified active AND show a recall/quality edge. |
| **`qwen3.6-27b/single/nvfp4/mtp.yml` (TP=1, NVFP4, **MTP n=3**) — `SPEC=on` + 65K** | @guybrush01 (1× RTX 5090 32 GB) | fp8/e4m3 @ scale=1.0 | 65K | **127.9 / 153.1** (decode **129.2 / 155.3**, n=5, CV 2.0% / 1.8%) | **4437 @10K** | ~3.2 | 0.87 / 0.75 / 0.60 (74%) | **31.2 GB** | 2026-07-08 | **The MTP-on config that fits 32 GB** ([#613](https://github.com/noonghunna/club-3090/issues/613)). nvidia modelopt NVFP4 dense 27B (NVFP4 FFN + FP8 attention + unquantized MTP head) on vLLM **v0.24.0** (quant=`modelopt`), fp8 KV, **MTP n=3** @ **65K + `util 0.85`**. **MTP-on OOMed at the earlier 98K default** — the KV reservation left no room for the draft head + cudagraphs + GDN prefill scratch; dropping ctx to 65K frees it → MTP loads + spec-decodes (accept ~3.2, draft ~74%) at **~2× the `SPEC=off` throughput** (71/71). `verify-stress` **all-pass** (60K/90K needle clean; above-ctx cleanly rejected). VRAM **31.2/32 GB @65K** (~1.4 GB free, flat). `SPEC=off` is the max-ctx fallback (81K/98K @ 71 TPS) and the tight-system-RAM path (@paulp83, 28 GB host, [#617](https://github.com/noonghunna/club-3090/issues/617)). Drove the default re-derate (98K/`util 0.92` → 65K/`util 0.85`, MTP kept). **8-pack owed** → stays 🧪. Compose: `models/qwen3.6-27b/vllm/compose/single/nvfp4/mtp.yml`. Registry: `vllm/qwen-27b-single-nvfp4`. |
| `qwen3.6-35b-a3b/dual/preview-mtp.yml` (TP=2, MTP n=3, built-in head) ⚠️ | @noonghunna (2× 3090 PCIe, 230 W cap) | fp8_e5m2 | 16K | **90.36 / 115.33** (decode 91.48 / 119.65) | — | **3.44** (narr) | 0.927 / 0.810 / 0.698 | 22.72 GB/card | 2026-05-15 | **MTP MAKES THINGS SLOWER on Qwen MoE preview path.** A/B vs preview.yml above (same config, same nightly, same hardware): **−51% narr / −35% code wall TPS** despite 81.2% avg draft acceptance and AL 3.44. Per-position accept (0.927 / 0.810 / 0.698) is healthy — the draft head works correctly. Surfacing cause: asymmetric GPU util (GPU 0 at **39%** / 233 W vs GPU 1 at 79% / 191 W; non-MTP run had both at 85-99%) indicates the draft forward pass on MoE imposes inter-GPU sync overhead the acceptance gain can't amortize. Vendor warning at boot: `max_num_scheduled_tokens=4096 ... suboptimal performance ... increase max_num_batched_tokens to accommodate the additional draft token slots`. **CV 5.4% / 2.0%** (less stable than no-MTP). VRAM +0.78 GB/card vs no-MTP (MTP head workspace). **Practical implication**: for v0.7.3, route Qwen 35B-A3B users to `preview.yml` (no MTP) as the default — MTP gives worse latency despite high acceptance. Re-test after (a) Genesis v7.73.x re-anchors on a post-#42521 nightly (Cliff 2 mitigations may unblock the underlying scheduler overhead), (b) max_num_batched_tokens raised to ~12K to satisfy the vLLM warning, or (c) MTP n=2 to see if smaller spec depth changes the calculus. Compose: `models/qwen3.6-35b-a3b/vllm/compose/dual/autoround-int4/preview-mtp.yml`. |
| **`qwen3.6-35b-a3b/ik-llama/single/mudler-apex-compact/mtp.yml` (TP=1, q4_0 KV, MTP n=5, --fit)** | @noonghunna (1× 3090, 370 W cap) | q4_0 / q4_0 | 200K | **96.47 / 144.01** (decode 98.54 / 150.98, n=5, CV 5.4% / 4.5%) | ~3000 @10K → ~1283 @180K | (ik-llama MTP active, AL not in bench log) | (not reported) | 20.46 GB | 2026-05-28 | **First-ever real run of the existing EVAL-only compose** (APEX weights weren't on disk before today). Pre-`fit-mtp.yml` baseline, kept as the q4-KV / max-headroom variant. `verify-full` 8/8, `verify-stress` 8/8 incl. 180K NIAH at 91% of n_ctx. **Confirms ik built-in MTP on the 35B-A3B MoE does NOT pay the vLLM `−51%` penalty** (compare row above) — MoE×MTP is a vLLM-scheduler-specific problem, not architectural. TTFT 205 ms. Compose: `models/qwen3.6-35b-a3b/ik-llama/compose/single/mudler-apex-compact/mtp.yml`. |
| **`qwen3.6-35b-a3b/ik-llama/single/mudler-apex-compact/fit-mtp.yml` (TP=1, q8/q5 KV, MTP n=4)** ⭐⭐ | @noonghunna (1× 3090, 370 W cap) | **q8_0 (K) / q5_0 (V) — asymmetric K-high/V-low** | **196K** | **103.25 / 149.12** (decode 105.63 / 156.80, n=5, CV 3.0% / 1.5%) | ~3000 @10K → ~1283 @180K | (ik built-in MTP active, AL not in bench log) | (not reported) | **21.09 GB** | 2026-05-28 | **Captures @laurimyllari's `--fit` + asymmetric KV config from [discussion #241](https://github.com/noonghunna/club-3090/discussions/241).** A/B vs `mtp.yml` row above (same engine, same weights, same rig + 370 W): **+7% narr / +4% code wall TPS** at **tighter CV** (3.0/1.5% vs 5.4/4.5%) for +0.6 GB VRAM — the Anbeeld K-high/V-low pattern materialised. `verify-full` 8/8, `verify-stress` **8/8 incl. 180K NIAH** (91% of 196K, 26.4 GB margin, 1283 tok/s prefill at ceiling). Adds `--no-mmap` + `--cache-ram 4096` per his fit-mtp.yml. **Deterministic quality 76/90 = 84%** on PR #38-fixed verifiers (toolcall 14/15, instructfollow **15/15**, structoutput 14/15, dataextract 11/15, reasonmath 12/15, bugfind 10/15). **Sandbox-pack quality measured post-fix** (benchlocal-cli [#42](https://github.com/noonghunna/benchlocal-cli/pull/42) hermes thinking-on deterministic sampler + [#43](https://github.com/noonghunna/benchlocal-cli/pull/43) per-pack timeout defaults + [#44](https://github.com/noonghunna/benchlocal-cli/pull/44) aider git-checkout guard + club-3090 [#245](https://github.com/noonghunna/club-3090/pull/245) wrapper-omit): **hermesagent-20 11/20 (55%), cli-40 12/40 (30%), aider-polyglot-30 12/30 (40%)**. cli-40 retains 7 "timeout" failures but those are sandbox-internal agent-give-up (max latency 19 s, well under the 300 s budget) — distinct from wall-clock, tracked in a follow-up benchlocal-cli issue. `soak-continuous` PASS (0 errors, 0/25 silent_empty, 0 VRAM growth, 100% retention, p50 decode 223 TPS). vs @laurimyllari on 4090 (205/254 TPS, 105-107/150 8-pack pre-PR #38): 3090 trails on TPS as expected (newer silicon + power), deterministic quality is on par. MoE × MTP no-penalty confirmed in ik-llama (cf. `preview-mtp.yml`). Compose: `models/qwen3.6-35b-a3b/ik-llama/compose/single/mudler-apex-compact/fit-mtp.yml`. Registry tag: `ik-llama/apex-fit-q8q5`. |
| **`qwen3.6-35b-a3b/ik-llama/single/byteshape-iq4xs/mtp.yml` (TP=1, q4_0 KV, MTP n=2, --fit)** ⚠️ | @noonghunna (1× 3090, 370 W) | q4_0 / q4_0 | **262K** (--fit auto) | **112.90 / 128.61** (decode 115.60 / 137.07, n=5, CV 1.9% / 1.8%) | ~1859 @95K → ~1050 @240K | (ik built-in MTP n=2 active) | **110/150 (73%, off)** | **22.5 GB** | 2026-06-02 | **Community intake from PR #293 (@Rhonstin)** — byteshape IQ4_XS 4.19bpw MoE GGUF (embedded MTP head), single-card 35B-A3B. First-party validated on 1× 3090: `verify-full` all-pass, `verify-stress` **8/8 incl. NIAH→240K** (91% of 262K, no Cliff; KV self 1584 MiB @ 262K — MoE KV is cheap), `soak-continuous` **PASS** (0 err, 0 VRAM growth, 0/25 silent-empty, 100% retention, p50 223 TPS). **8-pack 110/150 (think-off)**: toolcall 15/15 · instructfollow 14/15 · structoutput 14/15 · dataextract 13/15 · reasonmath 13/15 · bugfind 13/15 · hermesagent-20 11/20 · cli-40 17/40 — reproduces @Rhonstin's 111/150 within noise. q4/q4 KV buys **full 262K** (vs the apex sibling's q8/q5 @196K). Intake fixes vs #293: image cu13-server, port 8058. Compose: `models/qwen3.6-35b-a3b/ik-llama/compose/single/byteshape-iq4xs/mtp.yml`. Registry tag: `ik-llama/byteshape-iq4xs-mtp`. |
| `ik-llama/byteshape-iq4xs-mtp` (1× 3090 **water-cooled**) | @syangsao (1× 3090 water-cooled, **330 W cap** (default 390), driver 595.71.05, persistence on; [#445](https://github.com/noonghunna/club-3090/issues/445)) | q4_0 / q4_0 | **262K** | **133.05 / 154.81** (decode **135.51 / 160.90**, CV 1.4% / 1.1%, TTFT 135 ms) | 1941 @94K → ~1130 @214K (NIAH-rung diagnostic) | ~22.5 GB | 2026-06-19 | **First water-cooled datapoint on the config — +17% decode over the air-cooled reference row above (115.6/137.1 @ 370 W) at 40 W LESS.** Sustained-clock effect: decode on this config is thermal/clock-bound enough that water cooling beats a higher power cap. verify-stress ladder **6/6 clean, fillable to 240,635 tok** (91% of 262K). Useful anchor for the power-sweep guidance in [docs/HARDWARE.md](docs/HARDWARE.md#power). |
| `Qwen3.6-35B-A3B-Cerebellum-v3` (1× 3090, mainline llama.cpp) — **author data point, NOT a catalog compose** | @deucebucket (1× 3090, Bazzite bare metal, host llama.cpp b9603, 420 W limit / ~300–316 W draw, desktop compositor on same GPU) | q8_0 / q8_0 | 131K (196K boots @ ~15.9 GB, fill-probed ~119K — NIAH ladder not run) | **147.9 / 146.2** (decode 150.5 / 149.9; `report.sh --full` re-run 141.8 / 141.2) | — | n/a (no drafter) | n/a | **~15.1 GB** (15,115 MiB @ 131K) | 2026-06-12 | **Author-reported; not club-validated or cataloged** — declined as a catalog compose ([PR #393](https://github.com/noonghunna/club-3090/pull/393)): headline value is the lean ~15 GB footprint (fits a **16 GB card**, where the ik-llama siblings at 21–22.5 GB don't), but we have no 16 GB card to validate/support that tier, and on 24 GB it's context/quality-dominated by `apex-fit-q8q5` / `byteshape-iq4xs`. Quant: **ablation-informed per-tensor mixed precision** (each tensor's Q2_K sensitivity measured, precision allocated under a size budget) → 11.96 GB GGUF on a stock ggml-org image. Det-5-pack **63/75** (toolcall 13 · instructfollow 15 · structoutput 14 · dataextract 9 · reasonmath 12; benchlocal `--medium`, sandboxed packs not run — no docker). TPS at `-np 4` provenance (single-card convention is `-np 1`). Drove the README `llama.cpp ❌→✅` correction (`559a3fe`). Weights + per-question evidence: [deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF](https://huggingface.co/deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF). |
| **`qwen3.6-35b-a3b/llama-cpp/dual/morikomorizz-q6kp/mtp.yml` (TP=2, q8_0 KV, MTP n=3)** 🧪 | @noonghunna (2× 3090 PCIe) | q8_0 / q8_0 | **262K** | **113.4 / ~150** (decode, n=3 shipped; wall 111.6 / ~144, CV<1% narr) | ~1820 @240K | (MTP draft-mtp n=3; acc narr 47% / code 83%) | n/a | **19.9 / 18.4 GB/card** | 2026-06-14 | **Uncensored "HauhauCS-Aggressive" 35B-A3B fine-tune** — morikomorizz Q6_K_P GGUF with embedded nextn MTP head, on **mainline llama.cpp b9570**, dual-card layer-split `-ts 0.55,0.45` (rebalances the MTP draft card; even `1,1` skews ~3 GB at 262K). **MTP loads clean** (`speculative decoding context initialized`) — the prior HauhauCS-MTP `ret=-3` was an ik-llama/older-build issue, not mainline. **n-sweep @262K (canonical bench.sh, same fresh container): n=3 113.4/~150 vs n=1 125.2/136.4 → n=3 = −9% prose / +10% code** (code-leaning default by maintainer request; n=1 is prose-best). ⚠️ An earlier rebench read 147/162 @ n=1 that **did NOT reproduce** on fresh boot (125 reproducible — GPU-state TPS variance; the 147 is not trusted). `verify-stress` **8/8 incl. NIAH→240K** (91% of 262K), `soak` fresh 20×5 **PASS** (0 MiB growth, 0/100 silent-empty, p50 162.4 TPS, 99.6% retention). **8-pack think-OFF 103/150 (69%) · think-ON 105/150 (70%)** (wash; reasoning-on costs heavy latency — hermes hit 300s — for +2): toolcall 14/14 · instructfollow 12/14 · structoutput 14/14 · dataextract 11/8 · reasonmath 11/12 · bugfind 13/13 · hermesagent 11/11 (of 20) · cli 17/19 (of 40). Mid-band for the class (byteshape 110); uncensoring buys compliance, not capability. **REASONING=on default** (vs stack thinking-off). Community GGUF, **digest-UNPINNED** + uncensored → 🧪. Compose: `models/qwen3.6-35b-a3b/llama-cpp/compose/dual/morikomorizz-q6kp/mtp.yml`. Registry tag: `llamacpp/hauhaucs-35ba3b-dual`. |
| **`gemma-4-26b-a4b/dual/awq.yml` (TP=2)** ⭐ | @noonghunna (2× 3090 PCIe, 230 W cap) | bf16 | 32K | **138.88 / 138.67** (decode 139.92 / 139.98) | — | n/a (no drafter) | n/a | **23.45 GB/card** | 2026-05-15 | **First v0.7.3 Gemma MoE production-track row.** Engine `vllm-nightly-clean` (nightly `bf610c2f`) + vendored [vLLM PR #40886](https://github.com/vllm-project/vllm/pull/40886) overlay (compressed-tensors AWQ MoE key remapping — applied at boot via anchor-based Python patcher in `models/gemma-4-26b-a4b/vllm/patches/vllm-pr40886-awq-moe-keys/install.sh`). Weights: `cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit` (17 GB). CV **0.2% / 0.0%** — extraordinarily stable, characteristic of MoE memory-bandwidth-bound decode. TTFT 53 ms. GPU 0 at 98% util / 360 W, GPU 1 at 59% / 299 W. **Identical narr/code wall TPS (139 / 139)** — same MoE signature as Qwen 3.6 35B-A3B preview above (per-token weight reads dominate, prompt content distribution doesn't matter). ~76% of Qwen 35B-A3B preview's narr TPS — consistent with 4 B vs 3 B active params per forward. Compose: `models/gemma-4-26b-a4b/vllm/compose/dual/awq/bf16.yml`. |
| **`gemma-4-26b-a4b/dual/awq-mtp.yml` (TP=2, MTP n=4)** ⭐⭐ | @noonghunna (2× 3090 PCIe, 230 W cap) | bf16 | 32K | **155.05 / 207.02** (decode 156.68 / 210.61) | — | **3.04 narr / 3.79 code** | 0.77 / 0.55 / 0.40 / 0.29 (narr); 50.9% avg (narr), 73.5% avg (code) | 23.50 GB/card | 2026-05-15 | **MTP boosts Gemma MoE: +12% narr / +49% code over awq.yml**. Engine `vllm-nightly-clean` + PR #40886 overlay + external `google/gemma-4-26B-A4B-it-assistant` drafter (832 MB BF16). Both GPUs symmetric at 98% util — **the external-drafter path doesn't pay the inter-GPU sync penalty that the Qwen 35B-A3B built-in MTP head pays** (see `preview-mtp.yml` row above where MTP made things 50% slower). Mechanism: external drafter is a small dense model (`Gemma4AssistantForCausalLM`, ~0.5 B params) — the draft forward bypasses MoE expert routing entirely. The 32-49% code lift confirms MTP is structurally compatible with Gemma's hybrid-SWA attention on the AWQ compressed-tensors path. **Recommended default for Gemma 26B-A4B going forward; awq.yml stays as the no-drafter A/B reference.** Compose: `models/gemma-4-26b-a4b/vllm/compose/dual/awq/mtp.yml`. |
| **`gemma-4-26b-a4b/single/awq/int8.yml` (TP=1, INT8-PTH KV, MTP n=4)** ⭐⭐ ⚠️ Production-w/-caveats | @noonghunna (1× 3090, 370 W) | int8_per_token_head | **176K** | **168.0 / 217.4** (decode 169.2 / 219.9, CV 1.9% / 2.0%) | ~4416 @10K → 976 @161K | **3.06–3.79** | 0.77–0.85 / 0.58–0.74 / 0.42–0.65 / 0.29–0.54 | **23.6 GB** | 2026-06-06 | **First single-3090 Gemma-4-26B-A4B — INT8-PTH KV (vendored [PR #40391](https://github.com/vllm-project/vllm/pull/40391) on stock v0.22.0, engine `vllm-gemma-stable`) lifts the single card from the prior bf16/16K boot-OOM to 176K.** Slug `vllm/gemma-26ba4b-single`; cyankiwi AWQ-4bit (Marlin WNA16 MoE) + external MTP drafter (`gemma-26b-it-assistant`, 832 MB BF16, n=4). Gate PASS (rebench `gemma-26ba4b-int8r`): `verify-full` ✓, `verify-stress` **NIAH-clean to 161K** (91% of 176K, all boundary checks), `soak` 20×5 **PASS 0-growth**. **8-pack 109/150 think-ON** (98/150 think-off) — toolcall 13/15 · instructfollow 14/15 · structoutput 14/15 · dataextract 9→11/15 · reasonmath 13→14/15 · bugfind 13/15 · hermesagent 10/20 · cli-40 12→21/40 — **on par with gemma-4-31B's think-ON 107/150**. Caveats: needs the #40391 overlay; **262K reachable only WITHOUT the MTP drafter** (drafter weights + cudagraph capture cost ~86K tok — mem_util 0.94, 0.96 OOMs the capture tail); think-OFF agentic/extraction softer (cli-40 30% / DataExtract 60%, recover to 52%/73% with thinking). Compose: `models/gemma-4-26b-a4b/vllm/compose/single/awq/int8.yml`. |
| `gemma-4-26b-a4b/dual/docker-compose.yml` (TP=2, Intel AutoRound INT4) | @noonghunna (2× 3090 PCIe) | — | — | **boot fail (SM86)** | — | — | — | — | 2026-05-15 | **Ampere-blocked on Intel AutoRound INT4.** `moe_intermediate_size=704` is not a multiple of `group_size=128` (5.5×); Marlin K-dim alignment fails. SM86 has no WNA16 kernel for unaligned K-dim — only SM90+ Cutlass W4A8 / Machete handle arbitrary shapes. Same failure on TP=1 (no split) and TP=2 (split to 352 per rank). Both Intel quant variants (`int4-mixed-AutoRound` and `int4-AutoRound`) hit the same error. **AWQ is the Ampere path** (row above). AutoRound compose preserved here as a documented-blocker for SM90+ rigs (RTX 5090 / Pro 6000 should boot it). |


---

## Ornith-1.0 (DeepReinforce agentic-coding fine-tunes — 🧪 experimental)

DeepReinforce agentic-coding RL fine-tunes of the Qwen3-Next family (`qwen35` 9B dense-hybrid · `qwen35moe` 35B-A3B). Both land **in-band with the base Qwen** on general capability — the fine-tune's value, where any, is coding (the 35B edges the base on aider; the 9B is a footprint play). ik_llama GGUF, drafter-free ngram, full 262K.

| Compose | Rig | KV | Max ctx | Narr / Code TPS | PP tok/s | Peak VRAM | Date | Notes |
| --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- |
| `ik-llama/ornith9b-single` | @noonghunna (1× 3090) | q8_0 | 262K | 100.6 / 100.6 (decode 102.0 / 102.7) | — | **13.4 GB** | 2026-06-25 | **Qwen3-Next dense-FFN HYBRID** (arch `qwen35`: 8 full-attn + 24 GDN, NON-MoE) — 262K KV only 4.25 GB. No MTP → ngram (~0.59 accept, works despite DeltaNet). verify-stress 8/8, soak PASS. **8-pack 91/150 off / 95 on**; cli-40 7/40 is partly a `<solution>`-sign-off artifact (→13 hardened). **NICHE ONLY** — gemma-4-12b beats it on quality (105) + speed (117/122). The lean 13.4 GB footprint is the only reason to pick it. Slug shipped [#477](https://github.com/noonghunna/club-3090/pull/477). |
| `ik-llama/ornith35b-dual` | @noonghunna (2× 3090) | q8_0 | 262K | 106.5 / 103.6 (decode 108.8 / 108.6) | — | ~22 GB/card | 2026-06-26 | **qwen35moe MoE hybrid** (40L, 256 experts / 8 active ~3B, NO MTP). 262K KV ~2.7 GB. ngram opt-in (Q8 dual tight; needs `n_max=32` + balanced `-ts`). verify-stress 8/8, soak PASS. **8-pack 105/150 off == on — TIES base qwen3.6-35b-a3b** (byteshape 110, within noise) but **EDGES it on aider-polyglot-30: 15/30 (off == on) vs the base's 12–13** + perfect bugfind 15/15. **Coding-leaning 35B-A3B**; the base stays the general pick. Slug shipped [#479](https://github.com/noonghunna/club-3090/pull/479). |

---

## Agents-A1 (InternScience 35B agentic MoE — ⚠️ production w/ caveats)

InternScience's own 35B agentic MoE (Qwen3-Next MoE architecture; card declares its OWN base — not a Qwen fine-tune slug), served from the official FP8-dynamic compressed-tensors checkpoint. **The agentic thinking-ON specialist**: ties the base qwen3.6-35b-a3b on general capability, beats it on cli-40. First model onboarded end-to-end through the Bring & Validate lane (T2 producer-zero).

| Compose | Rig | KV | Max ctx | Narr / Code TPS | PP tok/s | Peak VRAM | Date | Notes |
| --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- |
| `vllm/agents-a1-dual` | @noonghunna (2× 3090 PCIe, 370/420 W) | fp8_e5m2 | 262K | **150.9 / 149.6** (decode 154.0 / 153.8, n=3, CV 0.1%, TTFT 130/128 ms — n + pair-order corrected 2026-07-04 from the tag artifact) | 7.9K→4.6K t/s across 94K→240K (NIAH-rung diagnostic) | ~22.0 GB/card | 2026-07-03 | **Qwen3-Next MoE** (geometry == 35B-A3B; ~3B active), NO MTP head (safetensors-verified) → drafter-free. On sm_86 vLLM serves **Marlin FP8-MoE weight-only** (activation quant inert — sm_89+ runs the real checkpoint). verify-stress **8/8** — staggered NIAH exact-recall to **240K (91%)**, VRAM Δ0 across the ladder; soak PASS (0 growth, 0/100 silent-empty, 99.8% retention). **8-pack --full: OFF 105/150 · ON 110/150** (post benchlocal #79+#81 harness; fresh sandbox images): toolcall 15/15 OFF · IF 15/15 ON · **cli-40 thinking-ON 23/40 — the stack's highest** (base: 17/40) · hermes 12/20 OFF but **regresses to 9/20 thinking-ON** (verified model behavior). vs base 35B-A3B: general TIES (ON 110=110), decode −13% (154 vs 178) — reach for A1 on tool/CLI-agent work with thinking ENABLED. Vision retained. Dual-only (36 GB weights). |
| `vllm/agents-a1-dual` | @sumo-dandan (2× 3090 — one ×16@Gen3, one **×4 OCuLink eGPU dock** off chipset M.2; both capped 250 W; [#552](https://github.com/noonghunna/club-3090/issues/552)) | fp8_e5m2 | 262K | **150.3 / 149.6** (decode 152.2 / 152.3, n=5, CV 0.0%, TTFT 84/83 ms) | 7.4K→4.4K t/s across 94K→240K (NIAH-rung diagnostic) | ~23.4 GB/card | 2026-07-04 | **First cross-rig confirmation** — decode within ~1% of the gate rig despite the eGPU-docked card at PCIe 4.0 ×4 + 250 W caps: single-stream TP=2 decode is VRAM-bandwidth-bound, lane width ~irrelevant (SFF/eGPU-builder datapoint). Full battery mirrors the gate: NIAH **6/6 to 240K exact recall** (VRAM Δ−4 MB) · soak PASS (0 growth, 0/25 silent-empty, 100% retention, p50 182 decode) · agentic TTFT **sub-linear** (5.1× at 24.8× ctx to 35K) · kv-calc calibration 7/7 on-rig. Boot engaged **custom all-reduce (PCIe-P2P auto-detect)** — clean throughout, first signal for that path on a chipset-attached card. **Quality `--medium` (2026-07-04, current harness):** toolcall 15/15 · IF 15/15 · structout 14/15 · dataextract 13/15 · reasonmath 12/15 = **69/75 (92%)** — deterministic-5 on par with the gate rig. |
| `vllm/agents-a1-dual` | @guybrush01 (2× **RTX 5090** PCIe 5.0 x8, Ryzen 9 9950X3D2, CachyOS, driver 610.43.02, 575 W; [#567](https://github.com/noonghunna/club-3090/issues/567)) | fp8_e5m2 | 262K | **217.9 / 217.1** (decode **220.21 / 220.36**, n=5, CV 0.0%, TTFT 49/49 ms) | 13.8K–15.3K t/s at 184K–240K (NIAH-rung diagnostic) | ~30.5 GB/card (3+ GB free at 240K) | 2026-07-04 | **First Blackwell / dual-5090 A1 data — +43% decode over the 3090 gate (154) at CV 0.0%**, and ~7–10× the 3090s' deep-context prefill. ⚠️ Runs the **forced-Marlin workaround path** (`VLLM_TEST_FORCE_FP8_MARLIN=1`, [#548](https://github.com/noonghunna/club-3090/issues/548)) — vLLM v0.24.0's native Cutlass FP8 kernel selection breaks on this hybrid arch at sm_120, so these numbers are **weight-only dequant, NOT native FP8 GEMMs**: likely headroom on top once the upstream fix lands (docs/UPSTREAM.md). verify-full pass · NIAH **6/6 to 240,634 tok** · soak-continuous PASS (0 MiB growth, 100% retention) · kv-calc calibration 7/7. PCIe x8 flagged by report.sh; decode unaffected (VRAM-bandwidth-bound — matches @sumo-dandan's ×4-lane finding). |

---

## Quality benches — Aider Polyglot 30

Pass rate on a curated 30-exercise subset of [aider-polyglot-benchmark](https://github.com/Aider-AI/polyglot-benchmark) (5 per language across cpp/go/java/javascript/python/rust, mix of easy/medium/hard). Tests **edit-format reliability** AND **algorithmic correctness** — does the model emit diffs aider can apply, AND do the resulting tests pass.

Run via [`benchlocal-cli`](https://github.com/noonghunna/benchlocal-cli) `aider-polyglot-30` pack. Different from the TPS rows above — this is a quality / agentic-coding signal, not a throughput measurement.

| Model | Compose | Rig | Pass / Total | % | Wall (real) | Wall (sum-dur) | Tokens (P+C) | Date | Notes |
|---|---|---|---:|---:|---:|---:|---:|---|---|
| **Qwen 3.6 27B** (AutoRound INT4) | `dual.yml` (TP=2) | @noonghunna (2× 3090 PCIe, 230W cap) | **20 / 30** | **66.7%** | 19.0 min | 34.0 min | 436K + 111K = 547K | 2026-05-10 | **`enable_thinking=false`** (server-side `--default-chat-template-kwargs '{"enable_thinking": false}'` + per-request `extra_body` belt). With thinking ON: 0/30 (1500s timeout exceeded before any exercise completed — Qwen burns the token budget on hidden CoT). Per-language: cpp 3/5 · go 4/5 · **java 4/5** · js 4/5 · python 2/5 · rust 3/5. `threads=2`. |
| **Gemma 4 31B** (Intel AutoRound INT4) | `dual.yml` (TP=2) | @noonghunna (2× 3090 PCIe, 230W cap) | 17 / 30 | 56.7% | 19.2 min | 19.2 min | 380K + 72K = 452K | 2026-05-10 | Default thinking off (Gemma 4's chat template requires explicit `enable_thinking=true` to enable). Per-language: cpp 2/5 · **go 4/5** · java 1/5 · **js 4/5** · python 3/5 · rust 3/5. `threads=2`. |

**Notable**:
- Qwen 3.6 27B beats Gemma 4 31B by **+10 pp** despite 4 GB fewer parameters. Java is the biggest swing (4/5 vs 1/5 — `affine-cipher` specifically tripped Gemma).
- Gemma is meaningfully faster wall-clock (sum-of-exercise-durations 19 vs 34 min), suggesting Qwen produces longer per-turn answers but they convert to passes more reliably.
- Qwen with thinking ON is unusable for this kind of bench on club-3090 hardware: hits the 1500s subprocess timeout cap (now bumped to 2700s) before completing any exercises — the hidden CoT eats the per-exercise token budget. **Set `enable_thinking=false` for any agentic / multi-turn workload.**

**To re-run cross-rig**: `bash scripts/quality-test.sh --pack aider-polyglot-30 --enable-sandboxed-packs` against your endpoint. See [docs/QUALITY_TEST.md](docs/QUALITY_TEST.md) for the harness setup.

---

## Head-to-head — Qwen 3.6 27B vs Gemma 4 31B on dual 3090 (TP=2, MTP)

Single-rig comparison run 2026-05-10 on @noonghunna's 2× 3090 PCIe rig (230 W cap each). Both models served via `dual.yml` compose with MTP speculative decode and `enable_thinking=false`. Aim: agentic-workload pick guidance.

### Config & pin delta — what differs between the two runs

| | Qwen 3.6 27B | Gemma 4 31B |
|---|---|---|
| Weights quant | Lorbus AutoRound INT4 (mtp.fc preserved BF16) | Intel AutoRound INT4 |
| MTP type | Built-in head, `method: mtp` n=3 | External 0.5B BF16 drafter (`gemma-4-31b-it-assistant`) via [vllm#41745](https://github.com/vllm-project/vllm/pull/41745), n=4 |
| KV cache | `fp8_e5m2` | BF16 |
| Max ctx | 262144 (262K) | 32768 (32K — BF16 KV ceiling on 2× 24 GB; use `gemma-int8-mtp` for 262K) |
| `--gpu-memory-utilization` | 0.92 | 0.92 |
| TP | 2 | 2 |
| vLLM nightly | `01d4d1ad` (2026-05-04) | `1acd67a7` (2026-05-08) — newer, post Gemma 4 MTP merge |
| Genesis pin | v7.64 (loaded via setup.sh) | n/a (Gemma 4 doesn't use Genesis patches) |
| thinking | OFF (server-side `--default-chat-template-kwargs`) | OFF (Gemma 4 default; needs explicit `enable_thinking=true` to enable) |

### TPS (canonical `bench.sh`: 800-word essay narrative + quicksort C++ code)

| Bench | Qwen 3.6 27B | Gemma 4 31B | Gemma vs Qwen |
|---|---:|---:|---:|
| Narrative decode TPS | 68.83 | **108.46** | **+57%** |
| Code decode TPS | 87.54 | **139.88** | **+60%** |
| TTFT narrative | 151 ms | **70 ms** | ~2× faster |
| TTFT code | 125 ms | **62 ms** | ~2× faster |
| VRAM/card | 23.7 GiB | **22.5 GiB** | −0.8 GiB |
| MTP mean accept length | 3.30–3.56 | 3.05–3.96 (warming) | similar |
| MTP avg accept rate | 76.5–85.2% | 51.2–73.9% (warming) | Qwen slightly higher when warm |
| CV (narr / code) | 3.9% / 4.1% | 1.4% / 1.0% | Gemma noticeably more stable |
| GPU power (per card) | 308 W / 263 W | 351 W / 296 W | Gemma uses ~13% more power |

### Quality — 8-pack `quality-test.sh --full` (150 scenarios)

Same endpoint, same thinking-off default. **Note** — `BENCHLOCAL_HERMES_RESOLVE_LOCALHOST=1` env var required for hermesagent-20 to reach host vLLM from the sandbox container (auto-set by `scripts/quality-test.sh` for localhost URLs since [83bf73d](https://github.com/noonghunna/club-3090/commit/83bf73d)).

| Pack | Qwen 3.6 27B | Gemma 4 31B | Δ (Gemma − Qwen) | Workload type |
|---|---:|---:|---:|---|
| toolcall-15 | 10/15 (67%) | 9/15 (60%) | **−7 pp** | Multi-tool call sequencing |
| instructfollow-15 | 13/15 (87%) | 13/15 (87%) | tied | Format-constraint following |
| structoutput-15 | 13/15 (87%) | 13/15 (87%) | tied | JSON / CSV schema emission |
| dataextract-15 | 15/15 (**100%**) | 15/15 (**100%**) | tied | Information extraction |
| reasonmath-15 | 6/15 (40%) | 6/15 (40%) | tied | Math (thinking-off cost = identical) |
| bugfind-15 | 12/15 (80%) | **14/15 (93%)** | **+13 pp** | Code debugging |
| hermesagent-20 | 10/20 (50%) | **12/20 (60%)** | **+10 pp** | Multi-turn agentic (Hermes harness, v0.7.4 grader) |
| cli-40 | 21/40 (52%) | 20/40 (50%) | −2 pp | CLI task completion |
| **TOTAL** | **100/150 (67%)** | **102/150 (68%)** | **+1 pp** | |

### Quality — Aider Polyglot 30 (per-language breakdown)

Cross-reference [Quality benches — Aider Polyglot 30](#quality-benches--aider-polyglot-30) above. Same `dual.yml` configs, same compose, single-shot.

| Language | Qwen 3.6 27B | Gemma 4 31B | Δ |
|---|---:|---:|---:|
| C++ | 3/5 (60%) | 2/5 (40%) | −1 |
| Go | 4/5 (80%) | 4/5 (80%) | tied |
| Java | **4/5 (80%)** | 1/5 (20%) | **−3** (Gemma fell apart on `affine-cipher`) |
| JavaScript | 4/5 (80%) | 4/5 (80%) | tied |
| Python | 2/5 (40%) | **3/5 (60%)** | **+1** |
| Rust | 3/5 (60%) | 3/5 (60%) | tied |
| **Total** | **20/30 (66.7%)** | 17/30 (56.7%) | **−10 pp** |

### Matched-config rebench (later same day) — most of the "Gemma +60% TPS" gap was config drift

After the first run, the two legs differed on (a) vLLM nightly, (b) KV dtype, (c) max context, (d) MTP n. Re-ran both with everything matched (vLLM `1acd67a7` · `int8_per_token_head` KV · 262K ctx · MTP n=4 · TP=2 · mem-util 0.92 · max-num-seqs=2 · AutoRound INT4 W4A16 group_size 128). Compose files added: `models/qwen3.6-27b/vllm/compose/dual/autoround-int4/int8.yml` (new, this work) + `models/gemma-4-31b/vllm/compose/dual/autoround-int4/int8.yml` (existing, `SPEC_N_MAX` parameterized).

**Notable side-finding:** Qwen3-Next + INT8 PTH KV (vllm#40391) **works end-to-end on our stack** — first validation. Real chat completions, MTP acceptance ~75%, **+27%/+33% TPS** vs Qwen's prior fp8 KV row above (most of which is the newer vLLM nightly + INT8 PTH KV combo, not architectural). Qwen `int8.yml` is now a candidate shipping path.

#### Matched-config performance (2026-05-10 PM)

| Metric | Qwen 3.6 27B int8.yml | Gemma 4 31B int8.yml | Gemma vs Qwen |
|---|---:|---:|---:|
| Narrative decode TPS | 88.05 | **98.12** | **+11%** |
| Code decode TPS | 120.17 | **129.34** | **+8%** |
| TTFT narrative | 152 ms | **79 ms** | **−48%** |
| TTFT code | 137 ms | **79 ms** | **−42%** |
| CV narrative / code | 1.6% / 5.3% | 2.1% / 0.6% | both clean |
| MTP avg accept (warm) | 75.3% | 70.6% | Qwen +5 pp |
| MTP per-pos rates (n=4) | 0.94 / 0.83 / 0.69 / 0.55 | 0.89 / 0.75 / 0.65 / 0.55 | Qwen earlier-pos better |

#### Concurrency + VRAM (matched config)

| Metric | Qwen | Gemma | Δ |
|---|---:|---:|---:|
| VRAM used / card | 21.4 GiB | 22.5 GiB | +1.0 GiB Gemma |
| Available KV cache / card | 11.10 GiB | 10.82 GiB | -0.28 GiB Gemma |
| **GPU KV cache size (tokens)** | **605,495** | 466,892 | **+30% Qwen** |
| max_num_seqs (configured) | 2 | 2 | tied |
| **Max concurrency @ 262K/req** | **2.31×** | 1.78× | +30% Qwen |
| Practical concurrency @ 100K/req | ~6.0× | ~4.7× | +28% Qwen |
| Practical concurrency @ 32K/req | ~18.9× | ~14.6× | +30% Qwen |

#### Matched-config headline

- **Per-stream speed:** Gemma wins **~10% decode TPS + ~45% TTFT** — that's the model-architecture-attributable gap. Dense attention prefills faster than DeltaNet hybrid.
- **Throughput at scale:** Qwen wins **~30% more KV pool** — supports more concurrent streams at the same per-stream context budget. Lighter weights + smaller per-token KV from the hybrid attention.
- **Workload pick:** Single-agent / chat → Gemma. Multi-tenant fleet → Qwen.
- **Original mismatched table below preserved** for the "shipping defaults" comparison. The +60% gap from this morning was nearly all vLLM-nightly + KV-class drift, not model intrinsics.

### Headline takeaways (original mismatched-config bench, kept for reference)

- **TPS:** Gemma 4 31B is **~60% faster** across both narrative and code, plus ~2× faster TTFT and 0.8 GiB less VRAM per card. For throughput-bound workloads at ≤32K ctx, Gemma 4 wins decisively at this config. For long-context (>32K), Qwen 3.6 27B's 262K ceiling + fp8 KV is the only option in this matchup (use `gemma-int8-mtp` for Gemma at 262K — separate compose, not benched here).
- **Quality aggregate:** Effectively tied — **180-scenario combined total: Qwen 120/180 (66.7%), Gemma 119/180 (66.1%)**. Within noise. Neither model is generically "better"; the choice is workload-shaped.
- **Quality by workload:** Gemma wins agentic + bug-fix (`bugfind +13 pp`, `hermesagent +10 pp`). Qwen wins polyglot code editing (`aider-polyglot +10 pp`, driven mostly by Java). Tool-call ordering favors Qwen (`toolcall +7 pp`). Five packs are dead-even.
- **reasonmath 6/15 (40%) on BOTH** — the thinking-off cost is identical and model-independent. Math problems with strict-format expectations need thinking-on (and the `aider-polyglot-30` row above shows what Qwen-with-thinking does to wall time — 0/30 hit the 1500s subprocess cap, now 2700s).
- **Pin/config delta:** Gemma's vLLM nightly is 4 days newer (post Gemma 4 MTP merge), KV is BF16 vs Qwen's fp8_e5m2, and Gemma uses an external 0.5B drafter vs Qwen's built-in MTP head. Same `dual.yml` topology, same `--gpu-memory-utilization 0.92`, same TP=2.

### How to reproduce on your rig

```bash
# Qwen leg
gpu-mode 27b                                                    # bring up at :8010
RUNS=3 WARMUPS=1 bash scripts/bench.sh                          # TPS
bash scripts/quality-test.sh --full                             # 8-pack quality
benchlocal-cli run --pack aider-polyglot-30 \
  --endpoint http://localhost:8010 --model qwen3.6-27b

# Gemma leg (cycle GPU)
gpu-mode gemma                                                  # bring up at :8030
URL=http://localhost:8030 MODEL=gemma-4-31b \
  RUNS=3 WARMUPS=1 bash scripts/bench.sh
URL=http://localhost:8030 MODEL=gemma-4-31b \
  bash scripts/quality-test.sh --full
benchlocal-cli run --pack aider-polyglot-30 \
  --endpoint http://localhost:8030 --model gemma-4-31b
```

End-to-end takes ~3 hours on dual 3090. quality-test.sh auto-detects the endpoint and served-model-id, so `URL`/`MODEL` overrides are only needed when running against non-default ports.