alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions 38 Packages Projects Releases Wiki Activity

Gate nvfp4 KV to datacenter Blackwell only (sm_100/103) — found on #571

Browse Source 
Two 5090 owners hit `--kv-cache-dtype nvfp4 requires sm100f` crashing
mid-boot on the #246 A/B (disc #571). Root-caused from vLLM #43562 /
TRT-LLM #10241: nvfp4 KV forces the trtllm-gen FP4 FMHA, built ONLY for
datacenter Blackwell sm_100/sm_103. Consumer Blackwell (sm_120/121 —
RTX 5090 / PRO 6000 Blackwell) is a HIGHER cc number but a different
family with no FMHA build. NVFP4 *weights* work there; only the KV path
doesn't. Our #246 gate had used a numeric ">=10.0" floor that wrongly
passed sm_120 — a floor can't express "sm_100/103 but not the
numerically-higher sm_120".

- gates.py: new `_ARCH_KERNEL_SM_FAMILY` allowlist ({nvfp4: sm_100/103});
  dropped nvfp4 from the numeric `_ARCH_KERNEL_SM` floor; family-membership
  reject with the FMHA reason + fp8_e4m3 fallback.
- arch-ab.sh: nvfp4 arm now refuses on consumer Blackwell (not just
  <sm_10), naming the FMHA gap + the fp8_e4m3 path; dropped nvfp4 from the
  recommended arms in help.
- hardware profiles: removed nvfp4 from rtx-5090 / rtx-6000-pro-blackwell
  KV lists (both sm_120) + added a why-not note.
- docs (DTYPE_MATRIX / HARDWARE / KV_MATH / QUANTIZATION): corrected the
  "Blackwell sm >= 10.0" framing to "datacenter sm_100/103 only".
- UPSTREAM.md: #43562 / TRT-LLM #10241 row + re-test trigger.
- test-arch-ab: nvfp4 refuses on sm_86 AND sm_120, allowed on sm_100;
  the dual-5090 all-arms test drops nvfp4.

Full scripts gate 66/66.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
This commit is contained in:
noonghunna
2026-07-05 03:23:49 +00:00
parent 0f41e739ed
commit 6f674fa2ce
10 changed files with 77 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/QUANTIZATION.md
View File

@@ -163,7 +163,7 @@ Independent of the weight quant, you can quantize the **KV cache** — this is w
| `q4_0` | 4 | llama.cpp / ik | Halves KV vs q8_0 → enables **262K on one 3090** (ik IQ4_KS). Tiny quality cost. |
| `fp8_e5m2` | 8 | vLLM | Our `vllm/dual` default (AutoRound weights) — the Ampere-safe storage-only fp8. |
| `fp8_e4m3` | 8 | vLLM | Same bytes as e5m2 but **native FP8 Tensor-Core compute on sm_89+** (Ada/Hopper/Blackwell; hard-rejected on Ampere). Since [#246](https://github.com/noonghunna/club-3090/issues/246) the launchers inject it automatically on those cards for the pilot slugs — you don't hand-pick it; `KV_CACHE_DTYPE=` in your env overrides. |
| `nvfp4` | 4 | vLLM ≥ v0.24.0 | Blackwell-only (sm ≥ 10.0) FP4 KV — a valid dtype literal in our pin, **unvalidated on this stack** (candidate on the Blackwell hardware profiles; #246 A/B arm 3). Needs a NIAH-clean cross-rig gate before it's anyone's default. |
| `nvfp4` | 4 | vLLM ≥ v0.24.0 | **DATACENTER Blackwell only (sm_100/103)** FP4 KV. The trtllm-gen FP4 FMHA has no consumer-Blackwell (sm_120/121) build → **crashes on RTX 5090s** despite their FP4 hardware ([vLLM #43562](https://github.com/vllm-project/vllm/issues/43562) / [TRT-LLM #10241](https://github.com/NVIDIA/TensorRT-LLM/issues/10241); confirmed disc #571). NVFP4 *weights* work on consumer; only KV doesn't. Consumer answer = `fp8_e4m3`. |
| **`int8_per_token_head`** | 8 | vLLM | ~1 byte/tok like fp8; **native in stock v0.22.0** for standard models (Gemma-4 needs the #40391 overlay). The KV path for **compressed-tensors weights (AWQ/FP8/INT8) at long context** — those can't use fp8 KV. |
| **TQ3 (TurboQuant)** | 3 | vLLM (Genesis) | 3-bit KV — beats fp8 on long-context memory; powers our `dual-turbo`. See [TQ3_MTP_GENESIS.md](TQ3_MTP_GENESIS.md) + [CLIFFS.md](CLIFFS.md). |
| `-khad` (modifier) | — | **ik only** | Hadamard transform on the K-cache → recovers accuracy lost to KV quantization, so you keep quality at q4_0/q8_0. |