Consolidates the validated findings from the #246 A/B + #571 volunteer
data + vLLM/TRT-LLM source review. New DTYPE_MATRIX section "Having the
Tensor Cores ≠ using them" makes the two load-bearing distinctions
explicit:
- Axis 1 — weights vs KV are gated SEPARATELY. FP8/FP4 *weights* work
wherever the TCs exist (sm_89+ FP8, all Blackwell FP4). KV is
different because it rides the attention (FMHA) kernel.
- Axis 2 — KV compute vs KV storage. Native FP8/FP4 *attention* compute
needs FA3 (Hopper sm_90 only) or trtllm-gen FMHA (datacenter Blackwell
sm_100/103 only). On EVERY consumer card — Ada 4090 (sm_89), consumer
Blackwell 5090/PRO-6000 (sm_120), DGX Spark GB10 (sm_121) — FP8 KV is
storage-only and nvfp4 KV doesn't work.
Consequences documented (both empirically confirmed): e4m3 ≡ e5m2 in
speed on consumer cards (86.73 vs 86.66 on a 5090, disc #571 — a
precision choice, not a perf lever); nvfp4 KV crashes on consumer
Blackwell (#43562). DGX Spark = same sm_12x family as the 5090.
Corrected now-wrong claims: DTYPE_MATRIX "Ada/Blackwell get a real win
on FP8 KV" (false — Hopper/DC-only) + the per-arch Ada/Blackwell-consumer
rows; QUANTIZATION + HARDWARE "native FP8 compute on sm_89+" → storage-
only + precision framing. Compute-win KV on consumer cards = INT8-PTH.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Two 5090 owners hit `--kv-cache-dtype nvfp4 requires sm100f` crashing
mid-boot on the #246 A/B (disc #571). Root-caused from vLLM #43562 /
TRT-LLM #10241: nvfp4 KV forces the trtllm-gen FP4 FMHA, built ONLY for
datacenter Blackwell sm_100/sm_103. Consumer Blackwell (sm_120/121 —
RTX 5090 / PRO 6000 Blackwell) is a HIGHER cc number but a different
family with no FMHA build. NVFP4 *weights* work there; only the KV path
doesn't. Our #246 gate had used a numeric ">=10.0" floor that wrongly
passed sm_120 — a floor can't express "sm_100/103 but not the
numerically-higher sm_120".
- gates.py: new `_ARCH_KERNEL_SM_FAMILY` allowlist ({nvfp4: sm_100/103});
dropped nvfp4 from the numeric `_ARCH_KERNEL_SM` floor; family-membership
reject with the FMHA reason + fp8_e4m3 fallback.
- arch-ab.sh: nvfp4 arm now refuses on consumer Blackwell (not just
<sm_10), naming the FMHA gap + the fp8_e4m3 path; dropped nvfp4 from the
recommended arms in help.
- hardware profiles: removed nvfp4 from rtx-5090 / rtx-6000-pro-blackwell
KV lists (both sm_120) + added a why-not note.
- docs (DTYPE_MATRIX / HARDWARE / KV_MATH / QUANTIZATION): corrected the
"Blackwell sm >= 10.0" framing to "datacenter sm_100/103 only".
- UPSTREAM.md: #43562 / TRT-LLM #10241 row + re-test trigger.
- test-arch-ab: nvfp4 refuses on sm_86 AND sm_120, allowed on sm_100;
the dual-5090 all-arms test drops nvfp4.
Full scripts gate 66/66.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
launch.sh/switch.sh now detect GPU arch and export KV_CACHE_DTYPE=
fp8_e4m3 (native FP8 compute) on sm_89+ cards for the pilot slugs
(vllm/dual, vllm/minimal). The injected value is the hardware
profiles' dormant kv_format_default.balanced -- one source of truth
shared with the pull gates; the Ampere no-op is data equality
(3090-class balanced = fp8_e5m2 = compose default -> nothing emitted),
not a code branch.
Injection guards (all load-bearing):
- pilot allowlist only; expansion gated on the #246 cross-rig A/B
- per-variant: registry kv_format == fp8_e5m2 only (int8-PTH/TQ/bf16
slugs never touched -- compressed-tensors weights reject fp8 KV)
- vLLM-family variants only; explicit user KV_CACHE_DTYPE= wins;
unmapped cards / heterogeneous rigs / no nvidia-smi -> no injection
- direct `docker compose up` keeps Ampere-safe compose defaults
Rides the existing resolve-variant-pin export seam (new optional
--gpu-spec); VLLM_ATTENTION_BACKEND is whitelisted but ships no value
(vLLM auto-detect stays the default until measured). Preflight banner
names the detected arch class.
Consistency fixes the injection exposed:
- gates.py _ARCH_KERNEL_SM: fp8_e4m3 9.0 -> 8.9 (vLLM's real floor is
SM89+; we'd otherwise inject e4m3 on 4090s our own pull-gate calls
unloadable) + new nvfp4: 10.0 row (v0.24.0 literal had NO gate --
a 3090 pull of an nvfp4 config wouldn't have been rejected)
- Blackwell hardware profiles: nvfp4 declared as CANDIDATE capability
(engine list unchanged until validated -- gates take the intersection)
- kv-calc: projected nvfp4 rows (bytes/elem + activation coefs),
calibration unchanged
Docs: HARDWARE.md new section, KV_MATH/QUANTIZATION/DTYPE_MATRIX rows
(incl. retiring the Genesis-era "e4m3 undertuned per #51" advisory in
favor of the A/B).
Validated: 8-case injection matrix in test-launch-compat; live no-op
on the real 2x3090 (spec built, nothing injected); faked 4090 through
the real bash detection path emits e4m3; compose interpolation both
ways; kv-calc --calibration green; full scripts gate 64/64.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The fast/balanced/max composes aren't rungs on a single fidelity axis —
they're corners of a 3-axis trade-space (weight-fidelity / decode+context
/ prefill-TTFT). §4b names the axes, the corner map, the legitimacy rule
(a differentiator no instrument can measure isn't a tier), scheme-vs-
algorithm, and why "balanced" stays provisional. It fills the prefill
corner with the measured INT8 W8A8 vs FP8 result (v0.24.0): quality tied
107/150, W8A8 wins prefill/TTFT (native CUTLASS INT8), FP8 wins decode —
a prefill-vs-decode tradeoff, not a replacement. W8A8 qualifies as an
analyzed tier; shipping it as a slug is deferred (not committed).
- §4a "Tiering principle" now points into §4b (topology projection).
- DUAL_CARD balanced/max footnote cross-links §4b for the rationale.
- QUANTIZATION.md added to the docs/README.md index.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Capture the quant/KV findings so others don't re-hit them:
DTYPE_MATRIX.md — new "KV-quant × checkpoint compatibility — the two Ampere traps":
(1) fp8 KV is rejected for compressed-tensors checkpoints (AWQ/FP8/INT8 weights),
flag-independent; auto_round/GPTQ unaffected. (2) int8_per_token_head is NATIVE
in stock v0.22.0 for uniform-head-dim models — #40391 is the Gemma-4-only
(interleaved 256/512 head-dim → page-size unification) adapter, don't copy it.
+ an Ampere KV-dtype picker table; tag the INT8-PTH row native/overlay status.
QUANTIZATION.md — new §4a "Picking a quant by fidelity (KLD) — and where QAT fits":
Phaelon74 KLD ranking (INT8 0.009 < FP8 0.023 < AWQ-BF16-INT4 0.042 < AWQ-INT4
0.051 < AutoRound 0.063); KLD is weights-only (KV-quant adds separate error);
QAT only out-earns PTQ at <=4-bit (8-bit PTQ already near-lossless); dual=fidelity /
single=fit tiering. + int8_per_token_head row + the fp8-guard caveat in §5.
FIX the stale §4 FP8 line: FP8 *weights* DO run on Ampere via Marlin W8A16 (not
"emulated / KV-only") and are a top-fidelity option.
FAQ.md — new Q: "My AWQ/FP8 model errors on --kv-cache-dtype fp8" → use int8-PTH.
Gate 42/42.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
