diff --git a/docs/DTYPE_MATRIX.md b/docs/DTYPE_MATRIX.md index 8bf742b9..93da3e50 100644 --- a/docs/DTYPE_MATRIX.md +++ b/docs/DTYPE_MATRIX.md @@ -114,7 +114,7 @@ KV cache is a separate concern from weights — vLLM ships several KV-quant sche **Emerging KV recipes** (worth watching, not yet defaults): - **Block-scaled FP8 KV** — per-block scales like MXFP8 but applied to KV cache rather than weights. Recovers accuracy at long-context where flat FP8 can lose precision in deep layers. Lands on Hopper / Blackwell first. -- **NVFP4 KV** — Blackwell-native, ~4× smaller than FP16 KV at NVFP4 weight accuracy levels. **A valid `--kv-cache-dtype nvfp4` literal in our v0.24.0 pin** (declared as a *candidate* on the Blackwell hardware profiles; SM-gated ≥10.0). Unvalidated on this stack — it's arm 3 of the #246 cross-rig A/B; a literal existing ≠ working kernels + clean recall (see the int8-PTH-on-Gemma and mainline-TQ precedents). +- **NVFP4 KV** — **DATACENTER Blackwell only (sm_100/sm_103: B100/B200/GB200).** It forces vLLM's trtllm-gen FP4 FMHA, which has **no consumer-Blackwell (sm_120/121) build** — so it **crashes on RTX 5090s** despite their FP4 hardware ([vLLM #43562](https://github.com/vllm-project/vllm/issues/43562) / [TRT-LLM #10241](https://github.com/NVIDIA/TensorRT-LLM/issues/10241); empirically hit on two 5090s via #246, disc #571). NVFP4 *weights* run on consumer Blackwell — only the KV/attention path doesn't. Consumer-Blackwell FP4-era KV = **fp8_e4m3**. (A textbook 'literal exists ≠ kernel exists' — the exact trap the #246 A/B was built to catch.) - **TensorRT-LLM W4A8 / W4A4** — Hopper/Blackwell weight+activation quant recipes that pair INT4/NVFP4 weights with FP8/FP4 activations. Out of scope for the vLLM-first composes here but worth knowing if you cross-shop. ### KV-quant × checkpoint compatibility — the two Ampere traps @@ -322,7 +322,7 @@ What to ship as the default for each GPU class, given the matrix above: | **Ampere DC (A100)** | AutoRound INT4 or FP16 | TQ3 / fp8 | MTP n=3 | Same composes as 3090, more VRAM headroom | | **Ada (4090 / L40)** | AutoRound INT4 (preserve INT4 path) **OR** FP8 weights for full TC use | **fp8_e4m3 (HW — launcher-injected for pilot slugs, #246)** / TQ3 / INT8 PTH | MTP n=3 / DFlash | Same composes work; FP8 KV is now a real perf win | | **Hopper (H100/H200)** | FP8 weights (FBGEMM/INC) | **fp8 (HW transformer engine)** | MTP / DFlash | Not a primary target — these cards are usually already running their own optimised stacks | -| **Blackwell consumer (5090/5080/5070)** | AutoRound INT4 today; NVFP4 when vLLM kernels mature | **fp8_e4m3 (launcher-injected for pilot slugs, #246 — A/B pending)** · nvfp4 = candidate arm 3 | MTP n=3 | The old "e4m3 path undertuned per #51" advisory was **Genesis-nightly-era** — stock v0.24.0 is what the #246 A/B measures. Genesis treats Blackwell consumer as a separate regime — some Hopper-targeted patches don't apply | +| **Blackwell consumer (5090/5080/5070)** | AutoRound INT4 today; NVFP4 *weights* work | **fp8_e4m3** (launcher-injected for pilot slugs, #246 — A/B pending). **nvfp4 KV does NOT work here** — datacenter-Blackwell-only FMHA kernel (#43562) | MTP n=3 | The old "e4m3 path undertuned per #51" advisory was **Genesis-nightly-era** — stock v0.24.0 is what the #246 A/B measures. Genesis treats Blackwell consumer as a separate regime — some Hopper-targeted patches don't apply | | **Blackwell DC (B100/B200/GB200)** | NVFP4 / FP8 weights | NVFP4 / fp8 | MTP / DFlash | Out of scope for club-3090 | The starred row (Ampere consumer) is the actual target of this stack; everything else is "should work, here's the data point we have". diff --git a/docs/HARDWARE.md b/docs/HARDWARE.md index ad2c89c8..509e7ed1 100644 --- a/docs/HARDWARE.md +++ b/docs/HARDWARE.md @@ -84,7 +84,7 @@ Mechanics and boundaries: - **Direct `docker compose -f … up` bypasses all of this** and keeps the Ampere-safe compose defaults on any card. - The preflight banner names the detected class: `[preflight] arch: ada (sm_8.9) — arch-aware KV defaults active for pilot slugs (#246)`. - `VLLM_ATTENTION_BACKEND` is plumbed through the same channel but **ships no value** — vLLM's backend auto-detect is the default until someone measures a better per-arch choice. -- **Blackwell + `nvfp4` KV**: a valid dtype literal in our v0.24.0 pin and declared as a *candidate* on the Blackwell hardware profiles, but unvalidated (no NIAH/quality data on any rig we've seen) — it's arm 3 of the #246 A/B, opt-in via `KV_CACHE_DTYPE=nvfp4`, not a default. +- **`nvfp4` KV is DATACENTER-Blackwell-only** (sm_100/sm_103). It needs vLLM's trtllm-gen FP4 FMHA, which has no consumer-Blackwell (sm_120/121) build — so it **crashes on RTX 5090s** even though they run NVFP4 *weights* fine ([vLLM #43562](https://github.com/vllm-project/vllm/issues/43562) / [TRT-LLM #10241](https://github.com/NVIDIA/TensorRT-LLM/issues/10241); confirmed on two 5090s, disc #571). On consumer Blackwell use **fp8_e4m3** KV — the launchers inject it automatically for the pilot slugs. --- diff --git a/docs/KV_MATH.md b/docs/KV_MATH.md index a1f825cf..4ff8b675 100644 --- a/docs/KV_MATH.md +++ b/docs/KV_MATH.md @@ -107,7 +107,7 @@ For hybrid architectures (DeltaNet, SWA), only the **growing** attention layers | `k8v4` | 0.75 | Mixed precision | | `q4_0` | ~0.56 | Includes packed-quant overhead | | `turboquant_3bit_nc` (TQ3) | ~0.425 | Genesis-supplied; cheapest KV format on this stack | -| `nvfp4` | ~0.56 (**projected**) | 4-bit elements + fp8 block scale per 16 (9/16 B). Blackwell-only (sm ≥ 10.0); vLLM v0.24.0 literal, **no measured boot on this stack** — #246 A/B arm 3. kv-calc carries mirrored-fp8 activation coefs until a real boot calibrates them | +| `nvfp4` | ~0.56 (**projected**) | 4-bit elements + fp8 block scale per 16 (9/16 B). **DATACENTER Blackwell only (sm_100/103)** — the trtllm-gen FP4 FMHA has no consumer (sm_120/121) build, so it crashes on 5090s ([vLLM #43562](https://github.com/vllm-project/vllm/issues/43562)). No measured boot on this stack. kv-calc carries mirrored-fp8 coefs until a datacenter-Blackwell boot calibrates them | **Note on Ampere**: `fp8_e4m3` is NOT supported by the Triton kernel on sm_86 (3090/3090-Ti/A5000). Use `fp8_e5m2` (engine-level fallback) or `int8_per_token_head` (vendored via PR #42102). On sm_89+ the launchers inject `fp8_e4m3` automatically for the #246 pilot slugs. See [DTYPE_MATRIX.md](DTYPE_MATRIX.md). diff --git a/docs/QUANTIZATION.md b/docs/QUANTIZATION.md index 1ca923c1..c3243b23 100644 --- a/docs/QUANTIZATION.md +++ b/docs/QUANTIZATION.md @@ -163,7 +163,7 @@ Independent of the weight quant, you can quantize the **KV cache** — this is w | `q4_0` | 4 | llama.cpp / ik | Halves KV vs q8_0 → enables **262K on one 3090** (ik IQ4_KS). Tiny quality cost. | | `fp8_e5m2` | 8 | vLLM | Our `vllm/dual` default (AutoRound weights) — the Ampere-safe storage-only fp8. | | `fp8_e4m3` | 8 | vLLM | Same bytes as e5m2 but **native FP8 Tensor-Core compute on sm_89+** (Ada/Hopper/Blackwell; hard-rejected on Ampere). Since [#246](https://github.com/noonghunna/club-3090/issues/246) the launchers inject it automatically on those cards for the pilot slugs — you don't hand-pick it; `KV_CACHE_DTYPE=` in your env overrides. | -| `nvfp4` | 4 | vLLM ≥ v0.24.0 | Blackwell-only (sm ≥ 10.0) FP4 KV — a valid dtype literal in our pin, **unvalidated on this stack** (candidate on the Blackwell hardware profiles; #246 A/B arm 3). Needs a NIAH-clean cross-rig gate before it's anyone's default. | +| `nvfp4` | 4 | vLLM ≥ v0.24.0 | **DATACENTER Blackwell only (sm_100/103)** FP4 KV. The trtllm-gen FP4 FMHA has no consumer-Blackwell (sm_120/121) build → **crashes on RTX 5090s** despite their FP4 hardware ([vLLM #43562](https://github.com/vllm-project/vllm/issues/43562) / [TRT-LLM #10241](https://github.com/NVIDIA/TensorRT-LLM/issues/10241); confirmed disc #571). NVFP4 *weights* work on consumer; only KV doesn't. Consumer answer = `fp8_e4m3`. | | **`int8_per_token_head`** | 8 | vLLM | ~1 byte/tok like fp8; **native in stock v0.22.0** for standard models (Gemma-4 needs the #40391 overlay). The KV path for **compressed-tensors weights (AWQ/FP8/INT8) at long context** — those can't use fp8 KV. | | **TQ3 (TurboQuant)** | 3 | vLLM (Genesis) | 3-bit KV — beats fp8 on long-context memory; powers our `dual-turbo`. See [TQ3_MTP_GENESIS.md](TQ3_MTP_GENESIS.md) + [CLIFFS.md](CLIFFS.md). | | `-khad` (modifier) | — | **ik only** | Hadamard transform on the K-cache → recovers accuracy lost to KV quantization, so you keep quality at q4_0/q8_0. | diff --git a/docs/UPSTREAM.md b/docs/UPSTREAM.md index 6ae90c7a..e83cb175 100644 --- a/docs/UPSTREAM.md +++ b/docs/UPSTREAM.md @@ -86,6 +86,7 @@ purged (e.g. `VLLM_IMAGE=vllm/vllm-openai:latest`). | Issue / PR | Status | Why it matters | Workaround | |---|---|---|---| +| **[#43562](https://github.com/vllm-project/vllm/issues/43562)** (+ [TRT-LLM #10241](https://github.com/NVIDIA/TensorRT-LLM/issues/10241)) — `--kv-cache-dtype nvfp4` unsupported on consumer Blackwell (sm_120/121) | 🟡 Open upstream / **local gate added** | nvfp4 KV forces vLLM's **trtllm-gen FP4 FMHA**, built ONLY for **datacenter Blackwell sm_100/sm_103** (B100/B200/GB200). Consumer Blackwell (sm_120/121 — RTX 5090 / PRO 6000 Blackwell) is a higher cc number but a **different family with no FMHA build** → crashes at boot (`requires sm100f`). NVFP4 *weights* work on consumer; only the KV/attention path doesn't. Empirically hit on two 5090s via the #246 A/B ([disc #571](https://github.com/noonghunna/club-3090/discussions/571), 2026-07-05); our #246 gate had wrongly used a numeric `≥10.0` floor that passed sm_120. | Local: `gates.py` `_ARCH_KERNEL_SM_FAMILY` now allowlists nvfp4 to `{sm_100, sm_103}` (excludes sm_120/121); `arch-ab.sh` nvfp4 arm refuses on consumer Blackwell with the fp8_e4m3 fallback; nvfp4 dropped from the 5090 / PRO-6000-Blackwell hardware-profile KV lists. **Consumer-Blackwell FP4-era KV = `fp8_e4m3`.** Re-test trigger: TRT-LLM #10241 ships the sm_120 FP4 FMHA and a vLLM release picks it up → re-add nvfp4 to the consumer profiles + arch-ab arms. | | **(to file)** — torch_aot_compile cache key ignores GPU arch / resolved quant kernel → cross-arch cache load crashes (`AttributeError: 'QKVParallelLinear' object has no attribute 'workspace'`) | 🔴 **To file upstream** (mechanism pinned 2026-07-04; draft pending maintainer review) | **Root cause pinned by the stack trace in club-3090 [#548](https://github.com/noonghunna/club-3090/issues/548):** the crash fires inside `torch/_dynamo/aot_compile.py __call__` immediately after `Directly load AOT compilation from path …/torch_aot_compile/61bda05f…` — an AOT graph **compiled on our sm_86 rig** (Marlin-processed FP8 layers → graph bakes `layer.workspace` reads) loading onto **sm_120** (Cutlass-processed layers → no `workspace` attr). vLLM's AOT cache key (`61bda05f…`) is IDENTICAL across the two arches despite different per-layer kernel processing — cross-arch cache hits should be impossible. The trigger on user rigs was **us**: A1's promotion accidentally git-shipped the whole compiled cache (3,667 files), and the compose warm-start mount fed it to every fresh pull (fixed — cache un-tracked, house `.gitignore` pattern applied). | Two independent unblocks: **(a)** delete the mounted `torch_compile` cache contents → fresh AOT compile against the rig's own kernel selection (native Cutlass FP8 on sm_89+ — the preferred path, likely faster than #567's numbers); **(b)** `VLLM_TEST_FORCE_FP8_MARLIN=1` forces layer processing to match the sm_86-baked graph (validated in [#567](https://github.com/noonghunna/club-3090/issues/567), but costs native FP8 GEMMs). Upstream fix = include compute capability / resolved kernel config in the AOT cache key. | | **[#42609](https://github.com/vllm-project/vllm/issues/42609)** — custom_all_reduce IPC handle fails with `expandable_segments` | 🟡 Open / **local fix active** | `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` crashes vLLM custom all-reduce at graph-buffer registration — `custom_all_reduce.cuh:455 'invalid argument'` (`cudaIpcGetMemHandle` on a `cuMemMap` VA range is not a valid IPC source). Upstream's title scopes the repro to **DP>1 AND TP>1**, but the same crash class fires on **TP-only** startup paths ([#42583](https://github.com/vllm-project/vllm/issues/42583) Qwen3.5+LoRA TP=4, [#45604](https://github.com/vllm-project/vllm/issues/45604) MiniMax TP=4). On our **2-card NVLink / patched-PCIe-P2P** rigs custom AR is ON at TP=2, so the `expandable_segments:True` that dual composes inject **for the PCIe (custom-AR-OFF) path** crashes boot when P2P is enabled. Earlier signal: JusefPol's NVLink boot crash (club-3090 PR #31). | [`scripts/detect_nvlink.sh`](../scripts/detect_nvlink.sh) strips **only** the `expandable_segments` token from `PYTORCH_CUDA_ALLOC_CONF` on the P2P / custom-AR path (other knobs like `max_split_size_mb` preserved); the PCIe path keeps it. Guard: `scripts/tests/test-detect-nvlink-alloc-conf.sh`. Drop when [#43923](https://github.com/vllm-project/vllm/pull/43923) (the open in-engine fix targeting `custom_allreduce` + `expandable_segments`) merges and lands in our pin. NB: [#40812](https://github.com/vllm-project/vllm/pull/40812) (merged v0.20.1, already in our v0.22.0 pin) only auto-disables `expandable_segments` around the **cumem sleep-mode pool**, not the `custom_all_reduce` path — it does **not** cover this crash. | | **[#45163](https://github.com/vllm-project/vllm/pull/45163)** — DiffusionGemma (vLLM's first discrete-diffusion LM) | 🟢 **#45163 MERGED (in v0.24.0) — on stock v0.24.0 + 3 vendored Ampere/TP fixes** | The DiffusionGemma 26B-A4B arch (`DiffusionGemmaForBlockDiffusion`, V2 model runner) is in **no** released/nightly vLLM and only in this unmerged draft — BUT vLLM publishes an official **`vllm/vllm-openai:gemma`** image (a stock build of the dgemma branch commit `74b5964f`) that has it baked in. Needed to serve `vllm/diffusiongemma-dual` (the dLLM studio model). | Engine `vllm-diffusion-gemma` **pins `vllm/vllm-openai:gemma` BY DIGEST** (`sha256:9c719fc0…`, purge-resistant) + bind-mounts **3 fix files** that are NOT upstream (vLLM tests H100/B200 + TP=1): [`models/diffusiongemma-26b-a4b/vllm/patches/gemma-image-fixes/`](../models/diffusiongemma-26b-a4b/vllm/patches/gemma-image-fixes/README.md) — `marlin.py`+`marlin_utils_fp8.py` (sm_86 fp8 Marlin sub-tile-K pad; `:gemma` clean dies in warmup on the `K=352/1056` wall) + `diffusion_gemma.py` (TP-vocab soft-embed + dtype fix). Status 🧪 experimental (no DEFAULTS row; launch `--force`) — soak-continuous PASS (0-growth, 0/25 silent-empty) + 8-pack 100/150, but held at experimental because the arch is in *no release* (unmerged + mutable `:gemma` tag) and the dLLM SSE/streaming-NIAH checks can't cleanly gate. **Supersedes the former 123-file sideload-on-nightly** (PR #358 → simplified here: same perf, 120 fewer files). **Re-pin trigger**: vLLM re-pushes `:gemma` → re-pin the digest + rebase the 3 fixes. **Promote trigger (decision 2026-06-11)**: when #45163 merges into a pinnable release → re-pin to it, re-run the gate, flip 🧪 → ⚠️ caveats (documented caveats: long-ctx exact-recall degrades past ~30K, block-streaming SSE, dual-only) — see club-3090 task. **Drop trigger**: the K-pad lands upstream (our #40361 / an Ampere Marlin fix) AND the TP-vocab fix merges into `:gemma` → mount nothing. **(2026-06-12: the K-pad half landed on vLLM main via [#45295](https://github.com/vllm-project/vllm/pull/45295) — general `marlin_padded_nk`; our #40361 closed-superseded. Drop still gated on a stock image that includes #45295 + the TP-vocab merge.)** **UPDATE 2026-07-02:** #45163 **MERGED** (shipped in v0.24.0) → engine `vllm-diffusion-gemma` bumped off the `:gemma` branch digest to stock **`vllm/vllm-openai:v0.24.0`** (arch native). The 3 Ampere/TP fixes STAY vendored + re-validated on stock v0.24.0 (boot + coherent TP=2). #45295's marlin-pad (native in v0.24.0) is the *dense/linear* path — it does NOT cover DiffusionGemma's `marlin_moe_wna16` sm_86 fp8 K-pad (native-no-mounts test still crashed there), so `marlin.py`+`marlin_utils_fp8.py` remain required; native `diffusion_gemma.py` is still TP=1 → the TP-vocab fix remains required. Compose also gained the `--model` flag fix (v0.24.0 positional-arg change). Stays 🧪 (still 3 vendored fixes + no soak on v0.24.0). | diff --git a/scripts/arch-ab.sh b/scripts/arch-ab.sh index c7815bec..4e799dc5 100755 --- a/scripts/arch-ab.sh +++ b/scripts/arch-ab.sh @@ -10,7 +10,9 @@ # # Usage: # bash scripts/arch-ab.sh # arms e5m2,e4m3; variant by GPU count -# bash scripts/arch-ab.sh --arms e5m2,e4m3,nvfp4 # Blackwell (sm>=10) opt-in third arm +# bash scripts/arch-ab.sh --arms e5m2,e4m3 # standard A/B (any sm_89+ rig) +# (nvfp4 arm exists but refuses on consumer Blackwell — sm_120/121 5090s lack +# the nvfp4-KV FMHA kernel; it needs DATACENTER Blackwell sm_100/103.) # bash scripts/arch-ab.sh --variant vllm/dual # force the variant # bash scripts/arch-ab.sh --dry-run # print the plan, run nothing # bash scripts/arch-ab.sh --resume # skip arms/steps with artifacts @@ -20,8 +22,11 @@ # Arms: # e5m2 fp8_e5m2 KV on the pilot variant (control — today's shipped default) # e4m3 fp8_e4m3 KV on the pilot variant (native FP8 KV compute on sm_89+) -# nvfp4 nvfp4 KV on the pilot variant (Blackwell-only, UNVALIDATED — refuse -# below sm 10.0) +# nvfp4 nvfp4 KV on the pilot variant — DATACENTER Blackwell (sm_100/103) +# ONLY. Consumer Blackwell (sm_120/121, RTX 5090 / PRO 6000 Blackwell) +# runs FP4 weights but has no nvfp4-KV FMHA kernel (vLLM #43562 / +# TRT-LLM #10241); the arm refuses there. fp8_e4m3 is the +# consumer-Blackwell FP4-era KV path. # fp8w vllm/qwen-27b-dual-max STOCK (FP8 weights + int8-PTH KV, no dtype # override; dual-rig only) — measures the native-FP8-WEIGHTS lift on # sm_89+ vs Ampere's Marlin-dequant path. NOTE: fp8 KV is rejected on @@ -94,8 +99,14 @@ for arm in "${ARM_LIST[@]}"; do || die "arm 'e4m3' needs sm>=8.9 (fp8_e4m3 KV is boot-rejected on Ampere); detected min sm_${MIN_SM}. On a 3090-class rig there is no arch delta to measure — this A/B is for Ada/Blackwell rigs (run --arms e5m2 if you just want the control numbers)" fi if [[ "$arm" == "nvfp4" ]]; then - awk -v a="$MIN_SM" 'BEGIN{exit !(a>=10.0)}' \ - || die "arm 'nvfp4' needs Blackwell (sm>=10.0); detected min sm_${MIN_SM}. Drop it: --arms e5m2,e4m3" + # nvfp4 KV forces vLLM's trtllm-gen FP4 FMHA — built ONLY for DATACENTER + # Blackwell sm_100/sm_103 (B100/B200/GB200). Consumer Blackwell (sm_120/121, + # RTX 5090 / PRO 6000 Blackwell) is a HIGHER cc number but a different family + # with NO FMHA build → crashes mid-boot (vLLM #43562 / TRT-LLM #10241). A + # plain ">=10.0" would wrongly pass sm_120, which is exactly what crashed two + # 5090s on disc #571 (2026-07-05). fp8_e4m3 IS the FP4-era KV path there. + awk -v a="$MIN_SM" 'BEGIN{exit !(a==10.0 || a==10.3)}' \ + || die "arm 'nvfp4' needs DATACENTER Blackwell (sm_100/sm_103); detected sm_${MIN_SM}. Consumer Blackwell (sm_120/121, e.g. RTX 5090) runs FP4 WEIGHTS but has NO nvfp4-KV FMHA kernel (vLLM #43562 / TRT-LLM #10241) — it crashes mid-boot. Use --arms e5m2,e4m3 — e4m3 is the FP4-era KV path for your card." fi if [[ "$arm" == "fp8w" && "$GPU_COUNT" -lt 2 ]]; then die "arm 'fp8w' runs vllm/qwen-27b-dual-max (TP=2) and needs 2 GPUs (detected ${GPU_COUNT})" diff --git a/scripts/lib/profiles/gates.py b/scripts/lib/profiles/gates.py index 8cc1c061..190c8fad 100644 --- a/scripts/lib/profiles/gates.py +++ b/scripts/lib/profiles/gates.py @@ -77,12 +77,25 @@ REPO_ROOT = Path(__file__).resolve().parents[3] # turboquant_3bit_nc 9.0 — MAINLINE TQ3 kernels (Genesis-Ampere TQ3 is a # different path; mirrors arch_patches Gemma # kernel_constraints) -# nvfp4 10.0 — Blackwell FP4 tensor cores; v0.24.0 literal, -# unvalidated on this stack (#246 A/B arm 3) +# nvfp4 family-gated (see _ARCH_KERNEL_SM_FAMILY) — NOT a floor _ARCH_KERNEL_SM = { "fp8_e4m3": 8.9, "turboquant_3bit_nc": 9.0, - "nvfp4": 10.0, +} + +# KV formats whose kernel is gated to a SPECIFIC SM FAMILY, not a "≥ floor". +# A numeric floor is WRONG here: consumer Blackwell (sm_120/121, RTX 5090 / +# PRO 6000 Blackwell) is a HIGHER compute-capability number than datacenter +# Blackwell (sm_100/103) but a DIFFERENT family that lacks the kernel. +# nvfp4 KV forces vLLM's trtllm-gen FP4 FMHA (fused attention), which is +# built ONLY for datacenter Blackwell sm_100/sm_103 (B100/B200/GB200). +# No sm_120/121 build exists → nvfp4 KV crashes on consumer 5090s +# despite their FP4 hardware (nvfp4 *weights* are fine — different +# path). Re-test when TRT-LLM #10241 / vLLM #43562 add the sm_120 +# FMHA. Consumer-Blackwell KV answer is fp8_e4m3. (Empirically hit on +# two 5090s via the #246 A/B, disc #571, 2026-07-05.) +_ARCH_KERNEL_SM_FAMILY = { + "nvfp4": {10.0, 10.3}, } @@ -500,7 +513,7 @@ def c0_engine_support( f"llama.cpp / ik-llama / beellama compose, not a vLLM one" ) - # 3d. hardware SM ≥ max(engine.min_sm, registry required_sm, arch-kernel) + # 3d. hardware SM ≥ max(engine.min_sm, registry required_sm, arch-kernel floor) need_sm = _required_sm(engine, entry, kv_format) if float(hardware_sm) < need_sm: return _incompat( @@ -509,8 +522,20 @@ def c0_engine_support( f"registry.required_sm={entry.get('required_sm')}, " f"arch-kernel[{kv_format}]=" f"{_ARCH_KERNEL_SM.get(kv_format, 0.0):g}); " - f"e.g. fp8_e4m3 needs SM 8.9+, Gemma-TQ3 SM 9.0+, nvfp4 SM 10.0+ " - f"— NOT loadable on sm_86" + f"e.g. fp8_e4m3 needs SM 8.9+, Gemma-TQ3 SM 9.0+ — NOT loadable on sm_86" + ) + + # 3d-bis. family-specific kernels — a numeric floor can't express these. + # nvfp4 KV needs the trtllm-gen FP4 FMHA, built only for datacenter Blackwell + # sm_100/sm_103; consumer sm_120/121 is a HIGHER number but has no build. + allowed_family = _ARCH_KERNEL_SM_FAMILY.get(kv_format) + if allowed_family is not None and float(hardware_sm) not in allowed_family: + fam = "/".join(f"sm_{s:g}" for s in sorted(allowed_family)) + return _incompat( + f"kv_format '{kv_format}' needs a family-specific kernel available " + f"only on {fam} (datacenter Blackwell); sm_{float(hardware_sm):g} has " + f"no build. nvfp4 KV's trtllm-gen FP4 FMHA has no consumer-Blackwell " + f"(sm_120/121) kernel (vLLM #43562 / TRT-LLM #10241) — use fp8_e4m3 KV" ) # --- 4. supported ----------------------------------------------------- diff --git a/scripts/lib/profiles/hardware/rtx-5090.yml b/scripts/lib/profiles/hardware/rtx-5090.yml index f658428f..7fd3c7e2 100644 --- a/scripts/lib/profiles/hardware/rtx-5090.yml +++ b/scripts/lib/profiles/hardware/rtx-5090.yml @@ -5,14 +5,15 @@ sm: 12.0 vram_gb: 32 arch: blackwell-consumer mem_util_safe: 0.96 +# NOTE: nvfp4 KV is DELIBERATELY NOT listed — it needs the trtllm-gen FP4 FMHA +# kernel, built only for datacenter Blackwell (sm_100/103). This is CONSUMER +# Blackwell (sm_120): FP4 weights work, but nvfp4 KV has no FMHA build and +# crashes (vLLM #43562 / TRT-LLM #10241). fp8_e4m3 is the FP4-era KV here. supported_kv_formats: - bf16 - fp16 - fp8_e5m2 - fp8_e4m3 - - nvfp4 # Blackwell FP4 KV — vLLM v0.24.0 literal; CANDIDATE, unvalidated - # on this stack (#246 A/B arm 3). Promote to kv_format_default - # only after a NIAH-clean cross-rig gate. - turboquant_3bit_nc - int8_per_token_head - q4_0 diff --git a/scripts/lib/profiles/hardware/rtx-6000-pro-blackwell.yml b/scripts/lib/profiles/hardware/rtx-6000-pro-blackwell.yml index 78df36d2..d8c58984 100644 --- a/scripts/lib/profiles/hardware/rtx-6000-pro-blackwell.yml +++ b/scripts/lib/profiles/hardware/rtx-6000-pro-blackwell.yml @@ -5,14 +5,15 @@ sm: 12.0 vram_gb: 96 arch: blackwell-workstation mem_util_safe: 0.96 +# NOTE: nvfp4 KV is DELIBERATELY NOT listed — it needs the trtllm-gen FP4 FMHA +# kernel, built only for datacenter Blackwell (sm_100/103). This is CONSUMER +# Blackwell (sm_120): FP4 weights work, but nvfp4 KV has no FMHA build and +# crashes (vLLM #43562 / TRT-LLM #10241). fp8_e4m3 is the FP4-era KV here. supported_kv_formats: - bf16 - fp16 - fp8_e5m2 - fp8_e4m3 - - nvfp4 # Blackwell FP4 KV — vLLM v0.24.0 literal; CANDIDATE, unvalidated - # on this stack (#246 A/B arm 3). Promote to kv_format_default - # only after a NIAH-clean cross-rig gate. - turboquant_3bit_nc - int8_per_token_head - q4_0 diff --git a/scripts/tests/test-arch-ab.sh b/scripts/tests/test-arch-ab.sh index 04e06912..f8f40d70 100644 --- a/scripts/tests/test-arch-ab.sh +++ b/scripts/tests/test-arch-ab.sh @@ -26,10 +26,11 @@ assert_contains "$out" "arm e5m2: vllm/minimal + KV_CACHE_DTYPE=fp8_e5m2" assert_contains "$out" "arm e4m3: vllm/minimal + KV_CACHE_DTYPE=fp8_e4m3" assert_contains "$out" "dry run — nothing executed" -# --- dual auto-pick + all four arms on 2x blackwell --------------------------- -out="$(CLUB3090_FAKE_GPUS="$GPU_5090x2" bash scripts/arch-ab.sh --arms e5m2,e4m3,nvfp4,fp8w --dry-run)" +# --- dual auto-pick + the arms that RUN on 2x consumer Blackwell -------------- +# (nvfp4 is excluded here — sm_120 has no nvfp4-KV FMHA kernel; covered below.) +out="$(CLUB3090_FAKE_GPUS="$GPU_5090x2" bash scripts/arch-ab.sh --arms e5m2,e4m3,fp8w --dry-run)" assert_contains "$out" "variant=vllm/dual" -assert_contains "$out" "arm nvfp4: vllm/dual + KV_CACHE_DTYPE=nvfp4" +assert_contains "$out" "arm e4m3: vllm/dual + KV_CACHE_DTYPE=fp8_e4m3" assert_contains "$out" "arm fp8w: vllm/qwen-27b-dual-max STOCK" # --- refusals (fail-loud, each names the fix) --------------------------------- @@ -42,10 +43,21 @@ assert_contains "$out" "no arch delta to measure" # ...but the explicit control-only run stays possible on Ampere out="$(CLUB3090_FAKE_GPUS="$GPU_3090" bash scripts/arch-ab.sh --arms e5m2 --dry-run)" assert_contains "$out" "arm e5m2" +# nvfp4 needs DATACENTER Blackwell (sm_100/103) — refuse on Ampere AND on +# CONSUMER Blackwell (sm_120), which is a higher number but has no FMHA kernel +# (the disc #571 crash: a plain ">=10" wrongly passed sm_120). if out="$(CLUB3090_FAKE_GPUS="$GPU_3090" bash scripts/arch-ab.sh --arms e5m2,nvfp4 --dry-run 2>&1)"; then fail "nvfp4 on sm_8.6 must refuse" fi -assert_contains "$out" "needs Blackwell" +assert_contains "$out" "DATACENTER Blackwell" +GPU_5090="0:NVIDIA_GeForce_RTX_5090:32607:12.0" +if out="$(CLUB3090_FAKE_GPUS="$GPU_5090" bash scripts/arch-ab.sh --arms e5m2,nvfp4 --dry-run 2>&1)"; then + fail "nvfp4 on consumer sm_12.0 must refuse (no FMHA kernel)" +fi +assert_contains "$out" "DATACENTER Blackwell" +# ...but datacenter Blackwell (sm_100) is allowed +out="$(CLUB3090_FAKE_GPUS="0:NVIDIA_B200:186000:10.0" bash scripts/arch-ab.sh --arms nvfp4 --dry-run 2>&1)" +assert_contains "$out" "arm nvfp4" if out="$(CLUB3090_FAKE_GPUS="$GPU_4090" bash scripts/arch-ab.sh --arms fp8w --dry-run 2>&1)"; then fail "fp8w on 1 GPU must refuse" fi