alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions 38 Packages Projects Releases Wiki Activity

docs(dual): de-stale the dual-max row — fp8/e4m3 KV + Production post-#594

Browse Source 
The row still titled dual-max '(FP8 weights + int8-PTH KV)' with a 🧪
trailer even though the #594 flip note was already appended inside it —
exactly the mixed signal that led #602 to expect the old tier. Header
now names fp8/e4m3 (#594), status matches the registry (Production),
and a one-line fast-vs-max qualification (speed+headroom vs
weight-fidelity, 8-pack tie, no more depth penalty) + the 2x5090
cross-arch 109/150 corroboration.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
This commit is contained in:
noonghunna
2026-07-06 22:31:41 +00:00
parent 55c468d15c
commit 2e05264c4b
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/DUAL_CARD.md
View File

@@ -19,7 +19,7 @@ You have **2× RTX 3090s**. This page is the front door for picking a config and
| **Hermes agentic fine-tune** (Carnice tool specialization) | [`carnice-bf16mtp.yml`](../models/qwen3.6-27b/vllm/compose/dual/carnice-bf16mtp/bf16-mtp.yml) | **262K** | **72 / 80** | ~22.3 / 24 GB | BF16 MTP overlay. Hermes-style assistant. Available on HF: [wasifb/Carnice_V2_27B_INT4_BF16MTP](https://huggingface.co/wasifb/Carnice_V2_27B_INT4_BF16MTP) |
| General-purpose default — the **"fast" tier** (vision + tools + long ctx) | [`dual.yml`](../models/qwen3.6-27b/vllm/compose/dual/autoround-int4/fp8-mtp.yml) ⭐ (≡ `vllm/qwen-27b-dual-fast`) | **262K** (237K single-prompt verified) | **69 / 89** | ~23.6 / 24 GB | AutoRound INT4 + fp8 KV, 2 streams, MTP n=3, full feature set. The proven path. KV pool **622K / 2.37×** — the largest of the dual family (lightest weights). |
| **"Balanced" tier** (int8-PTH KV fidelity bet) | [`dual-balanced`](../models/qwen3.6-27b/vllm/compose/dual/awq-bf16-int4/int8.yml) 🧪 (`vllm/qwen-27b-dual-balanced`) | **262K** | ~67 (probe) | KV pool 370K / 1.41× | cyankiwi **AWQ INT4** (int4 group-32, Marlin WNA16) + **int8-PTH** KV. 8-pack **105/150** (3-way tie †). ⚠️ **Dominated by fast** — slower (~67 vs ~89), smaller pool (370K vs fast's 622K; the 27 GB AWQ weights leave less KV room than fast's 17.5 GB autoround), tied/below 8-pack. Only possible edge: int8-PTH KV fidelity > fast's fp8 (same size — a fidelity bet, **unproven**; the 8-pack is blind to it). Keep only if NIAH (†) proves it. 🧪 Experimental. |
| **"Max accuracy" tier** (FP8 weights + int8-PTH KV) | [`dual-max`](../models/qwen3.6-27b/vllm/compose/dual/fp8/mtp.yml) 🧪 (`vllm/qwen-27b-dual-max`) | **262K** | **83 / 108** | KV pool 295K / 1.13× | **FP8** weights (e4m3, Marlin W8A16 on Ampere — memory win, no compute speedup) + **int8-PTH** KV. 8-pack **110/150** (3-way tie †). Decode **83/108** (v0.24.0, 2026-06-30 — corrects the stale ~56 probe); real tradeoffs are the smallest KV pool + slowest prefill/TTFT (158 ms; Marlin dequant is compute-heavy). **KV decode-at-DEPTH ([#594](https://github.com/noonghunna/club-3090/pull/594), 2026-07-06):** int8-PTH KV is `TRITON_ATTN`-only and its single-stream decode **craters with context** (130.8→50.7 TPS @ 35K); swapping the env to **`KV_CACHE_DTYPE=fp8`** (e4m3 → FlashInfer) keeps decode **flat** (~115 @ 35K, ≈2.3×) at ~2× prefill, **equal NIAH recall to 240K**, and **8-pack quality that ties (109 vs 107/150, [#594](https://github.com/noonghunna/club-3090/pull/594))** — quality-neutral despite scale=1.0 (`calculate_kv_scales` is disabled on Qwen3-Next hybrid). **soak-continuous PASS** — the tighter margin holds (509 MB free @ deepest fill, 0 MiB growth, 100% retention, p50 125.5); all #594 gates green. 🧪 Experimental. |
| **"Max accuracy" tier** (FP8 weights + fp8/e4m3 KV — [#594](https://github.com/noonghunna/club-3090/pull/594)) | [`dual-max`](../models/qwen3.6-27b/vllm/compose/dual/fp8/mtp.yml) 🧪 (`vllm/qwen-27b-dual-max`) | **262K** | **83 / 108** | KV pool 295K / 1.13× | **FP8** weights (e4m3, Marlin W8A16 on Ampere — memory win, no compute speedup) + **fp8/e4m3** KV (int8-PTH until 2026-07-06 — see the #594 note below; pull if your compose predates it). 8-pack **110/150** (3-way tie †). Decode **83/108** (v0.24.0, 2026-06-30 — corrects the stale ~56 probe); real tradeoffs are the smallest KV pool + slowest prefill/TTFT (158 ms; Marlin dequant is compute-heavy). **KV decode-at-DEPTH ([#594](https://github.com/noonghunna/club-3090/pull/594), 2026-07-06):** int8-PTH KV is `TRITON_ATTN`-only and its single-stream decode **craters with context** (130.8→50.7 TPS @ 35K); swapping the env to **`KV_CACHE_DTYPE=fp8`** (e4m3 → FlashInfer) keeps decode **flat** (~115 @ 35K, ≈2.3×) at ~2× prefill, **equal NIAH recall to 240K**, and **8-pack quality that ties (109 vs 107/150, [#594](https://github.com/noonghunna/club-3090/pull/594))** — quality-neutral despite scale=1.0 (`calculate_kv_scales` is disabled on Qwen3-Next hybrid). **soak-continuous PASS** — the tighter margin holds (509 MB free @ deepest fill, 0 MiB growth, 100% retention, p50 125.5); all #594 gates green. ✅ Production. **Fast-vs-max in one line:** fast is the *speed + headroom* tier (int4 weights → lighter memory traffic, biggest KV pool); max is the *weight-fidelity* tier — the 8-pack ties (within ±5–7 noise) and, post-#594, max no longer pays a decode-at-depth penalty for it. Cross-arch corroboration: 2×5090 scores the identical 109/150 ([#571](https://github.com/noonghunna/club-3090/discussions/571#discussioncomment-17553023)). |
| Multi-tenant (4 concurrent agents at full ctx) | [`dual-turbo.yml`](../models/qwen3.6-27b/vllm/compose/dual/autoround-int4/turbo.yml) | **262K** | **58 / 76** per-stream (269 TPS aggregate at 4 streams) | ~19.8 / 24 GB | TQ3 KV (3 bits/token) + full v7.69 PROD env-var stack — 4.67× concurrency. **20 GB Ampere users:** override `--kv-cache-dtype turboquant_3bit_nc` → `fp8_e5m2`; see [HARDWARE.md](HARDWARE.md#note-for-sub-24-gb-cards) + [#47](https://github.com/noonghunna/club-3090/issues/47). |
| Peak code TPS (DFlash) | **`beellama/qwen-dflash-dual`** — vLLM `dual-dflash*` ~~deprecated~~ | 262K | ~145 code | — | ⚠️ **vLLM `dual-dflash` / `dual-dflash-noviz` deprecated 2026-05-31** — superseded by `dual.yml` + stranded on a now-purged nightly; DFlash on dual moved to beellama (v0.3.0 🧪). Rationale + original numbers: [#297](https://github.com/noonghunna/club-3090/discussions/297). |