From 2e05264c4b3a3042d133717c8d87cacdaef941f2 Mon Sep 17 00:00:00 2001 From: noonghunna <10742901+noonghunna@users.noreply.github.com> Date: Mon, 6 Jul 2026 22:31:41 +0000 Subject: [PATCH] =?UTF-8?q?docs(dual):=20de-stale=20the=20dual-max=20row?= =?UTF-8?q?=20=E2=80=94=20fp8/e4m3=20KV=20+=20Production=20post-#594?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The row still titled dual-max '(FP8 weights + int8-PTH KV)' with a ๐Ÿงช trailer even though the #594 flip note was already appended inside it โ€” exactly the mixed signal that led #602 to expect the old tier. Header now names fp8/e4m3 (#594), status matches the registry (Production), and a one-line fast-vs-max qualification (speed+headroom vs weight-fidelity, 8-pack tie, no more depth penalty) + the 2x5090 cross-arch 109/150 corroboration. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm --- docs/DUAL_CARD.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/DUAL_CARD.md b/docs/DUAL_CARD.md index d2896b0b..2502b9a3 100644 --- a/docs/DUAL_CARD.md +++ b/docs/DUAL_CARD.md @@ -19,7 +19,7 @@ You have **2ร— RTX 3090s**. This page is the front door for picking a config and | **Hermes agentic fine-tune** (Carnice tool specialization) | [`carnice-bf16mtp.yml`](../models/qwen3.6-27b/vllm/compose/dual/carnice-bf16mtp/bf16-mtp.yml) | **262K** | **72 / 80** | ~22.3 / 24 GB | BF16 MTP overlay. Hermes-style assistant. Available on HF: [wasifb/Carnice_V2_27B_INT4_BF16MTP](https://huggingface.co/wasifb/Carnice_V2_27B_INT4_BF16MTP) | | General-purpose default โ€” the **"fast" tier** (vision + tools + long ctx) | [`dual.yml`](../models/qwen3.6-27b/vllm/compose/dual/autoround-int4/fp8-mtp.yml) โญ (โ‰ก `vllm/qwen-27b-dual-fast`) | **262K** (237K single-prompt verified) | **69 / 89** | ~23.6 / 24 GB | AutoRound INT4 + fp8 KV, 2 streams, MTP n=3, full feature set. The proven path. KV pool **622K / 2.37ร—** โ€” the largest of the dual family (lightest weights). | | **"Balanced" tier** (int8-PTH KV fidelity bet) | [`dual-balanced`](../models/qwen3.6-27b/vllm/compose/dual/awq-bf16-int4/int8.yml) ๐Ÿงช (`vllm/qwen-27b-dual-balanced`) | **262K** | ~67 (probe) | KV pool 370K / 1.41ร— | cyankiwi **AWQ INT4** (int4 group-32, Marlin WNA16) + **int8-PTH** KV. 8-pack **105/150** (3-way tie โ€ ). โš ๏ธ **Dominated by fast** โ€” slower (~67 vs ~89), smaller pool (370K vs fast's 622K; the 27 GB AWQ weights leave less KV room than fast's 17.5 GB autoround), tied/below 8-pack. Only possible edge: int8-PTH KV fidelity > fast's fp8 (same size โ€” a fidelity bet, **unproven**; the 8-pack is blind to it). Keep only if NIAH (โ€ ) proves it. ๐Ÿงช Experimental. | -| **"Max accuracy" tier** (FP8 weights + int8-PTH KV) | [`dual-max`](../models/qwen3.6-27b/vllm/compose/dual/fp8/mtp.yml) ๐Ÿงช (`vllm/qwen-27b-dual-max`) | **262K** | **83 / 108** | KV pool 295K / 1.13ร— | **FP8** weights (e4m3, Marlin W8A16 on Ampere โ€” memory win, no compute speedup) + **int8-PTH** KV. 8-pack **110/150** (3-way tie โ€ ). Decode **83/108** (v0.24.0, 2026-06-30 โ€” corrects the stale ~56 probe); real tradeoffs are the smallest KV pool + slowest prefill/TTFT (158 ms; Marlin dequant is compute-heavy). **KV decode-at-DEPTH ([#594](https://github.com/noonghunna/club-3090/pull/594), 2026-07-06):** int8-PTH KV is `TRITON_ATTN`-only and its single-stream decode **craters with context** (130.8โ†’50.7 TPS @ 35K); swapping the env to **`KV_CACHE_DTYPE=fp8`** (e4m3 โ†’ FlashInfer) keeps decode **flat** (~115 @ 35K, โ‰ˆ2.3ร—) at ~2ร— prefill, **equal NIAH recall to 240K**, and **8-pack quality that ties (109 vs 107/150, [#594](https://github.com/noonghunna/club-3090/pull/594))** โ€” quality-neutral despite scale=1.0 (`calculate_kv_scales` is disabled on Qwen3-Next hybrid). **soak-continuous PASS** โ€” the tighter margin holds (509 MB free @ deepest fill, 0 MiB growth, 100% retention, p50 125.5); all #594 gates green. ๐Ÿงช Experimental. | +| **"Max accuracy" tier** (FP8 weights + fp8/e4m3 KV โ€” [#594](https://github.com/noonghunna/club-3090/pull/594)) | [`dual-max`](../models/qwen3.6-27b/vllm/compose/dual/fp8/mtp.yml) ๐Ÿงช (`vllm/qwen-27b-dual-max`) | **262K** | **83 / 108** | KV pool 295K / 1.13ร— | **FP8** weights (e4m3, Marlin W8A16 on Ampere โ€” memory win, no compute speedup) + **fp8/e4m3** KV (int8-PTH until 2026-07-06 โ€” see the #594 note below; pull if your compose predates it). 8-pack **110/150** (3-way tie โ€ ). Decode **83/108** (v0.24.0, 2026-06-30 โ€” corrects the stale ~56 probe); real tradeoffs are the smallest KV pool + slowest prefill/TTFT (158 ms; Marlin dequant is compute-heavy). **KV decode-at-DEPTH ([#594](https://github.com/noonghunna/club-3090/pull/594), 2026-07-06):** int8-PTH KV is `TRITON_ATTN`-only and its single-stream decode **craters with context** (130.8โ†’50.7 TPS @ 35K); swapping the env to **`KV_CACHE_DTYPE=fp8`** (e4m3 โ†’ FlashInfer) keeps decode **flat** (~115 @ 35K, โ‰ˆ2.3ร—) at ~2ร— prefill, **equal NIAH recall to 240K**, and **8-pack quality that ties (109 vs 107/150, [#594](https://github.com/noonghunna/club-3090/pull/594))** โ€” quality-neutral despite scale=1.0 (`calculate_kv_scales` is disabled on Qwen3-Next hybrid). **soak-continuous PASS** โ€” the tighter margin holds (509 MB free @ deepest fill, 0 MiB growth, 100% retention, p50 125.5); all #594 gates green. โœ… Production. **Fast-vs-max in one line:** fast is the *speed + headroom* tier (int4 weights โ†’ lighter memory traffic, biggest KV pool); max is the *weight-fidelity* tier โ€” the 8-pack ties (within ยฑ5โ€“7 noise) and, post-#594, max no longer pays a decode-at-depth penalty for it. Cross-arch corroboration: 2ร—5090 scores the identical 109/150 ([#571](https://github.com/noonghunna/club-3090/discussions/571#discussioncomment-17553023)). | | Multi-tenant (4 concurrent agents at full ctx) | [`dual-turbo.yml`](../models/qwen3.6-27b/vllm/compose/dual/autoround-int4/turbo.yml) | **262K** | **58 / 76** per-stream (269 TPS aggregate at 4 streams) | ~19.8 / 24 GB | TQ3 KV (3 bits/token) + full v7.69 PROD env-var stack โ€” 4.67ร— concurrency. **20 GB Ampere users:** override `--kv-cache-dtype turboquant_3bit_nc` โ†’ `fp8_e5m2`; see [HARDWARE.md](HARDWARE.md#note-for-sub-24-gb-cards) + [#47](https://github.com/noonghunna/club-3090/issues/47). | | Peak code TPS (DFlash) | **`beellama/qwen-dflash-dual`** โ€” vLLM `dual-dflash*` ~~deprecated~~ | 262K | ~145 code | โ€” | โš ๏ธ **vLLM `dual-dflash` / `dual-dflash-noviz` deprecated 2026-05-31** โ€” superseded by `dual.yml` + stranded on a now-purged nightly; DFlash on dual moved to beellama (v0.3.0 ๐Ÿงช). Rationale + original numbers: [#297](https://github.com/noonghunna/club-3090/discussions/297). |