First 8-pack on vllm/tess-dual-nvfp4 (both modes, same-night GGUF
comparator 116/117). Gap is cli-40-concentrated (−8 off); deterministic
packs tie or better (RM-off 14/15 best-ever Tess). Per the
pre-registered rule the huginnfork NVFP4A16 weights-only A/B + FP8
precision arm are the next gates. BENCHMARKS vLLM subsection + registry
note + compose Quality line; Grok-§3 discriminator results recorded in
learnings.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Shipped-config quality refresh on b9967 + 16K budget (native template,
both modes, no pack regression vs the 2026-07-09 basis; first toolcall
15/15 OFF). The streaming+thinking finish=length caveat is non-repro
3/3 on this config — Status ⚠️→✅, caveat history kept as a RESOLVED
note, registry caveats→production, Quality lines refreshed. Drift +
parity + registry-json guards green.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
b9246 was 52 days / 721 builds old. Validated on Tess dual (full quality
A/B, 2026-07-11): think-OFF total tie 115=115 (greedy-leg engine
isolation — totals neutral, though 4 scenarios flipped each way),
think-ON 122 vs the 118x3 pin — the lift is thinking-path-specific;
3 previously-100%-reproducible failures now pass (CLI-31 safety refusal,
CLI-20 archive precision, RM-04 math). Boots clean, draft-mtp works.
Also unifies the pin landscape: Deckard's b9570 and the ROLLING tags on
gemma-12b/vibethinker (a #187-policy violation — rolling crash-looped at
b9282) all move to the same validated build. Engine-profile spec now
matches the composes. Non-Tess slugs: boot re-validation is spot-checked
(smoke to follow on PR); bench numbers stay dated to their measured pin;
baselines rows go stale by design (the staleness badge is the feature).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Fable 5 <[email protected]>
migtissera NVFP4 via Marlin W4A16 fallback: 62.4 tok/s decode spec-off
TP=2 @131K (BENCHMARKS 2026-07-11) vs the llama.cpp entry's 57.9 with
MTP — first vLLM-servable Tess on consumer cards. Ships experimental:
the 8-pack (the vLLM quality program A0 baseline) + stress/soak are the
promotion gate; pin-fallback rule to huginnfork NVFP4A16 documented in
compose + weights manual_note.
Catalog plumbing: nvfp4 weights variant; qwen35-dense added to
vllm-stable supported_model_families (live-validated); froggeric
template coverage extended to the tess compose (shared vendored copy,
patches.yml); kv_calc_supported=false on the model (hybrid — KV on
16/64 layers, naive math ~4x over; wire at promotion). Also finishes
the 2026-07-11 hybrid correction in the two spots today's sweep missed
(model YAML comment, llamacpp registry status_note). Counts 63→64 /
64→65. Full suite 77/78 (submit-bench = worktree fixture artifact);
diagnose-profile GREEN.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Fable 5 <[email protected]>
In-worker apply_model probe: all 16 full-attention layers carry the
shipped calibrated values byte-exact (0.023-0.271, none at 1.0), so the
provider A/B was calibrated-scales vs scale=1.0 — and tied. Corrects the
"load-on-hybrid unverified, assume scale=1.0" claims in the nvfp4-fast
compose header, registry status_note, and both BENCHMARKS rows.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Add vllm/qwen-35b-a3b-dual-nvfp4-fast: Ampere-validated unsloth NVFP4 MoE
New weights variant nvfp4-fast (unsloth/Qwen3.6-35B-A3B-NVFP4-Fast,
compressed-tensors mixed W4A4+FP8, auto-detected quant) + dual compose
(port 8080) + registry entry. First-party validated on the reference
2x3090 2026-07-11 — first MoE-FP4 fallback boot (MARLIN NvFp4 MoE
backend): decode 179.5/179.4 and 8-pack think-off 103/150, double
statistical tie with the AutoRound tier at full 262K (22.46 GB/card).
Ships 🧪 until verify-full/stress/soak run; native-FP4 (sm_90+) quality
remains the arc's missing datapoint (W4A4 activations unexercised on
Ampere). Registry counts 62→63, disk 63→64.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Fix MAX_ARG_STRLEN cliff: pass registry emit via temp file, not env
The full registry JSON (and the intermediate tab rows) were passed to
python heredocs through single environment variables. A single env value
is capped at MAX_ARG_STRLEN (~128 KB on Linux); the emit crossed it at
63 registry entries — "Argument list too long" in test-baselines and
test-registry-json, with the runtime REGISTRY_TAB pass in
registry-emit.sh sitting a few KB from the same cliff. All three now
write to a mktemp file and pass the path; python reads it with
encoding="utf-8".
Full suite: 76/77 green (test-submit-bench needs gitignored
results/rebench fixtures absent in a fresh worktree — env artifact).
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Move nvfp4-fast slug to port 8081 — 8080 is Open WebUI on common rigs
Caught by the boot-the-actual-compose gate: the health probe on 8080 hit
Open WebUI and verify-full ran against the wrong service. 8081 is free
and unclaimed in the registry.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Record actual gate results in nvfp4-fast compose header
switch.sh --force boot 272s + verify-full 7/1/2: the single failure is
streaming-toolcall+thinking-on finish=length — the known family caveat
class, not compose-specific. Stress/soak remain the 🧪→promotion gate.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Fable 5 <[email protected]>
NVFP4 runs on sub-sm_90 cards via the Marlin W4A16 weight-only fallback
since vLLM v0.24 (floor = capability 7.5, marlin_utils_fp4). Live-confirmed
on 2x RTX 3090 sm_86 2026-07-11: the shipped dual/nvfp4/mtp.yml boots at
262K + fp8 KV + MTP n=3 (accept 97%+), 69.7/85.5 decode TPS, and scores
8-pack think-off 110/150 — a statistical tie with the fp8 production
tier's 109. c3 hiding these slugs as "incompatible-hw" was factually wrong.
Semantic split — required_sm stays the NATIVE floor; new registry field
`fallback_sm` = the weight-only-fallback floor:
- kv-calc: in the band [fallback_sm, required_sm) the fit is PRICED
normally and annotated with `hw_fallback` {required_sm, card_sm, note}
instead of returning incompatible-hw (both --fit and --fit-all).
- c3: FitVerdict carries hw_fallback; fits-* + fallback -> "⚑" glyph,
VISIBLE by default. incompatible-hw hiding stays for true incompatibles.
- gates.py/compat.py: the hard SM floor becomes fallback_sm when present
(else required_sm, unchanged).
- Registry: fallback_sm=7.5 on the 4 NVFP4 slugs + status_note honesty
pass ("REFUSES on Ampere" -> fallback reality + measured numbers);
same corrections in the 4 compose headers.
- test-kv-calc-fit section (f) rewritten to the new contract.
On Ampere these slugs still carry the honest downside in the note: no
speed edge (~20% slower than the AutoRound tier for the same model) —
their sub-sm_90 value is models where NVFP4 is the only quant.
Guards green: kv-calc-fit, registry-disk, status-drift, switch/launch
parity, profiles-compat, pullgate, pull, registry-json/emit, diagnose,
launch-compat, default resolvers, kv-calc --calibration; c3 248 fast
tests + headless incompatible-hw hide tests.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Two independent RTX 5090 validations agree within noise:
- #619 (@paulp83): first NVFP4 boot on Blackwell — verify-full 9/9,
verify-stress needle-clean, soak-continuous PASS.
- #612 + #652 (@guybrush01): verify-full 9/9, verify-stress to 120K
(91% of 131K), soak PASS, p50 ~311 TPS. (#652 re-confirms #612 within
noise — one rig confirmed twice.)
status experimental -> caveats + compose header Status/Caveats. Caveat =
8-pack quality NOT yet cross-rig-measured (sandboxes weren't built on
these runs; quality run pending, gated on #492) — reverts to experimental
if a quality run regresses. NVFP4 stays Blackwell/Hopper-only (sm_90+).
Unchanged: 27b-single-nvfp4 (shipped util OOMs a 5090 — #613/#617, needs a
config fix) and both dual-nvfp4 slugs (no reports) remain experimental.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Companion to #654 (route-G): the safetensors weight-swap path had the same
"compose in the project tree" problem the user flagged — swap_apply.emit_swap_
compose wrote _brought-<san>.yml NEXT TO the sibling compose (so its relative
../ mounts resolved), dropping throwaway files beside catalog composes.
Route-C's MOUNT was already fine (it dir-mounts the brought weights to a distinct
/brought-model), so this is location-only: absolutize every sibling volume via a
new _absolutize_volume (${VAR:-../rel} → ${VAR:-/abs}, plain ../rel → /abs; the
vLLM sibling has 5 relative mounts — HF cache, torch_compile, triton, the
froggeric chat template, detect_nvlink) so the compose is relocatable, then write
it to the runtime dir beside the pull dir (<MODEL_DIR>/.cache/huggingface/
club3090/composes/), falling back beside the sibling.
Validated at emit level: all relative sources absolutized, ${MODEL_DIR} env var
preserved, brought mount intact, compose lands in the runtime dir not the repo.
+2 assertions in test-pull-swap.sh's emit group. (That .sh has a PRE-EXISTING
env failure on this rig — identical on master, unrelated.)
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
resolve_hf_home was `--hf-home > $HF_HOME > $XDG > ~/.cache`, ignoring MODEL_DIR.
So a bare `pull.sh <repo>` (or a nohup apply-swap) run with only .env's MODEL_DIR
set and no explicit HF_HOME silently fell to ~/.cache on the ROOT disk — the
footgun that misplaced a brought model's 35 GB of weights off the models volume
(and would bite users the same way). Insert a MODEL_DIR step (env, else parsed
from the repo .env, encoding=utf-8) between $HF_HOME and $XDG, matching what
switch.sh/launch.sh/c3 already resolve. c3 is unaffected — it sets HF_HOME
explicitly, which still wins. New guard test-hf-home-resolve (PASS 1-5);
test-pull / test-pullgate-download / test-download-lock still green.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
select_weight_files built the sharded set as `"-of-" in n or startswith
"model-"`, so a grafted MTP/nextn head (e.g. `mtp_grafted.safetensors`) — a
real weight the model needs with MTP enabled — was dropped, even though
detect_mtp_head/_has_mtp_weight_file already see it. Live dogfood: Tess-4-27B-FP8
downloaded its 7 model-* shards but silently omitted `mtp_grafted.safetensors`,
which would break MTP serving (club-3090 #617). Union any non-shard mtp/nextn
*.safetensors head into the sharded download set. Guard: test-pullgate-download
gains an MTP-head fixture.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
* c3+pull: detect in-progress downloads + per-repo lock (#617)
Repeated ① Bring [D] presses spawned N concurrent `hf download` runs racing
into the same `.incomplete` staging dir (5-deep live), each rmtree-ing and
re-fetching the others' partial. Two disk-truth gaps behind that:
1. No dedup. `download_model` now acquires an atomic per-repo lock
(`mkdir <pull_dir>/.download.lock` + a `pid` file = holder PID + UTC start)
BEFORE the staging rmtree. A 2nd concurrent call for the same slug is
REFUSED with `failure="in-progress"` (not a race); swap_apply surfaces it
as `in_progress`, and `pull.sh --apply-swap` exits **rc=3** ("already
downloading", distinct from rc=1 failure). A STALE lock (dead holder — a
crashed/SIGKILL'd download that skipped `finally`) is reclaimed on the next
call, so a leaked lock self-heals — more robust than a signal trap (a
SIGKILL skips traps too). Released in `finally` on every return path.
2. No in-progress detection. Presence probes only saw the FINAL dir (empty
until hf moves files out of `.incomplete`), so a mid-flight download read as
absent → the false "download did not complete". `services.bring_download_in_
progress` now reads the lock (live PID) + `.incomplete` bytes for a
best-effort pct; c3's fit-check render + [D] no-op guard + [k] cancel are
disk-aware, so a running download is REFLECTED (and not duplicated) even
across a c3 restart or a bare `pull.sh` started outside the session — the
in-memory tracker (#643) couldn't see those.
download_model is the shared chokepoint (bring apply-swap AND catalog fetch),
so both paths get the lock; the body moved verbatim to `_download_model_impl`
behind the locking wrapper — no behaviour change beyond the added guard.
Tests: scripts/tests/test-download-lock.sh (live-refuse + stale-reclaim +
release); c3 test_services (in-progress live/stale/absent + pct) +
test_app_headless (disk-detected → downloading render, [D] no-op, [k] kill).
245 services/registry + 131 app blast-radius tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* downloader: refuse a fresh pidless lock (close acquire-window race)
The lock acquires with mkdir then writes its pid file a beat later. A 2nd
caller landing in that window saw the lock dir but no pid → read_active
returned None → the wrapper treated it as stale and rmtree'd + reclaimed it,
letting BOTH callers proceed — the exact duplicate-download race the lock
exists to prevent. Now: a pid-present-but-dead lock still reclaims (unchanged),
but a PIDLESS lock only reclaims once it's older than a 10s grace — a fresh
pidless lock is a holder mid-acquire and is refused, not stolen. Guard adds
PASS 4 (fresh pidless → refuse) + PASS 5 (old pidless → reclaim).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
* baselines: backfill dual-max quality_8pk_think_on (110/150)
The vllm/qwen-27b-dual-max row logged only the thinking-off 8-pack
(107/150); the thinking-enabled run existed since the 2026-06-07 #515
quant-A/B (fp8 dual-max full 8-pack = 110/150, pack-defaults regime,
autoround sibling 109) but was never promoted. Backfilled with
provenance so it isn't re-derived. Guards: test-baselines,
test-quality-thinking, test-quality-baseline all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* preflight: warn on single-card GPU_MEMORY_UTILIZATION override above default
Two 5090 testers (#617 paul, and the same trap earlier) OOMed the tool-prefill
step of verify-stress by setting GPU_MEMORY_UTILIZATION=0.92 on the nvfp4
single-card slug whose validated default is 0.85. On one GPU a higher util
steals the free VRAM a large tool-response prefill needs for its activation
peak, so vLLM OOMs mid-prefill even though boot succeeds — a confusing failure
(boot ok, then HTTP 500 at step 2/8) that vLLM's own error hints at ("lower
--gpu-memory-utilization").
Add preflight_single_card_util(): an advisory WARN that fires only when the
user overrode GPU_MEMORY_UTILIZATION *above* the compose's shipped default on a
TP<=1 config. Runs even under --force (the nvfp4 slug launches with --force),
and no-ops for dual/multi-card, non-vLLM engines (no util default in the
compose), a plain default run, or util at/below the default. Wired into
switch.sh alongside the other force-independent hints.
Verified: behavior matrix (single>default WARN; single=default/below/none and
dual all silent) + test-preflight-compose-deps / test-model-switch /
test-switch-registry-parity / test-launch-compat all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
* feat(c3): ② Serve spec-dec — engine-driven drafter SELECTOR (not just on/off)
The spec-dec dropdown now lists the resolved ENGINE's supported drafters + off
(vLLM: mtp / mtp_assistant; beellama: dflash / mtp / mtp_gguf; …) instead of a
bare on/off — so the drafter TYPE is selectable, and it's engine-correct (dflash
only appears when the swap engine is beellama).
- swap_apply: the SPEC entrypoint REBUILDS --speculative-config via printf from
${DRAFTER_METHOD}/${DRAFTER_N} (defaults = the sibling's), so switching the
drafter is an env override — no re-emit. Validated: default→mtp n=3,
DRAFTER_METHOD=mtp_assistant→that config, SPEC=off→dropped, compose config valid.
- services: engine_drafters() reads the profile's supported_drafters (shares the
stdlib list-parse with engine_kv_formats via _engine_yaml_list); serve_override_
defaults adds DRAFTER_OPTIONS + SPEC_METHOD + SPEC_N.
- app: the spec Select value is the drafter method (or "off"); collect maps
method→SPEC=on+DRAFTER_METHOD, off→SPEC=off.
Drafters with extra requirements (mtp_assistant needs an assistant model; dflash
needs the beellama engine) are honestly listed per-engine — picking one the swap
can't satisfy fails at boot (loose-validation stance), or use a matching ① Bring
slug. 173 byo/serve/override tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* fix(launcher): GGUF verify-glob (#634) + launch.sh .env engine-image passthrough (#632)
#634 — setup.sh re-hardcoded VERIFY_GLOB=*.safetensors at the primary
download/verify (line 625), clobbering the recipe's glob that
load_weight_recipe already set from PRIMARY_WEIGHT_KEY. Every GGUF primary
fetch (WEIGHTS=gguf/iq4ks) then failed verify ("No *.safetensors found")
despite a byte-correct download. Now defaults to ${VERIFY_GLOB} (the recipe's
*.gguf); VERIFY_GLOB_OVERRIDE still wins.
#632 — launch.sh's .env load loop gated to CLUB3090_DEFAULT_* only, so
IK_LLAMA_IMAGE / LLAMACPP_IMAGE pins in .env were silently dropped (the two
GGUF engines are not profile-injected, so .env is their only override path —
the natural persistence for the #633 cu12 workaround never reached compose).
Broadened the allowlist to the engine-image keys; switch.sh already loaded
them (no filter). Both launchers now echo an ik-llama/llama.cpp image pin so a
wrong-image boot is never silent. (DelspoN's report cited the engine-pin
whitelist, which is the vllm/beellama-only profile-injection path — the real
drop was the .env loop filter.)
Verified: the .env loop now loads IK_LLAMA_IMAGE/LLAMACPP_IMAGE and still drops
un-allowlisted keys; test-model-switch / test-launch-compat / *-registry-parity
/ test-setup-picker / test-model-weights-registry all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
* feat(byo): apply-swap — download + serve a Route-C fine-tune
A curated-arch fine-tune (e.g. josefprusa/ThinkingCap-Qwen3.6-27B AutoRound
→ qwen3.6-27b) hard-stops at the pull gate's stratum-5 no-fit-model (correct
— nothing to price), so the c3 Bring [D] download failed with "download did
not complete". Route-C was pure guidance ("clone the compose, point --model
at your weights") with no code behind it — the deferred "swap_path apply".
Build it as a DISTINCT action that never touches the locked 6-stratum gate:
- scripts/lib/profiles/swap_apply.py (new): resolve the Route-C swap
(arch→curated sibling + has_mtp_head from deriver), download the brought
weights SHA-verified via downloader.download_model (a minimal EInput), and
emit a serve-locally compose that CLONES the --profile-like sibling's REAL
compose (keeping its curated chat-template / reasoning+tool parsers / MTP
wiring — NOT the derived-vllm template, which drops all that) with --model
re-pointed at a `<weights>:/brought-model:ro` mount, --served-model-name set
to the brought basename, and --speculative-config kept iff the checkpoint
carries an MTP head (else dropped). Written alongside the sibling compose as
`_brought-<san>.yml` (gitignored) so its relative ../ mounts resolve.
- scripts/pull.sh: intercept `--apply-swap` in the wrapper (strictly additive,
like --json) → run the apply-swap action; prints `[apply-swap] compose: …`.
The gate path is byte-unchanged without the flag.
- c3: services.run_bring_download gains `apply_swap=` (appends --apply-swap +
captures the emitted compose via last_swap_compose()); the [D] worker passes
it on route==C and stashes the compose; ② Serve serves that swap compose
directly (serve_generated) instead of a reproduction of the sibling's own
catalog compose.
- pull.py: fix the stale NOTE — "the bf16 base won't fit and lacks the MTP
head" → the base HAS the head; now just the size reason.
Tests: test-pull-swap.sh section 4 (emit: --model repointed, MTP kept/dropped
by head presence, curated flags + JSON args survive the YAML round-trip,
distinct container_name); test_services apply_swap flag + compose capture. The
locked-gate hard-stop tests stay green; --json fit-check byte-identical.
Validated live: pull.sh --json still no-fit-model+route-C, and apply_swap emits
a correct ThinkingCap swap compose (MTP kept).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* c3: reframe Route-C fit-check card as ✓ Servable + a clear [D] action
The fit-check card for a curated-arch fine-tune (Route-C swap) was
self-contradicting: a RED "not eligible" + a bare "no-fit-model" token at
the top, then a GREEN "② Serve is armed with <sibling>" at the bottom —
which read as a dead-end AND named the wrong model (the sibling, not the
brought fine-tune). The engine's "no-fit-model" is the *means* (generic
fit-math can't price a curated-hybrid arch), not the user's answer; the
outcome is servable via the sibling's recipe.
Reframe the Route-C case in _byo_result_text:
- green "✓ Servable — a fine-tune of <sibling>" instead of red "not eligible",
- one plain "How it serves" line (reuses the sibling's chat-template / tools /
spec-dec with your weights) instead of the --quantization/impl bullets,
- MTP kept/dropped line from has_mtp_head,
- the next-step is an explicit "→ Press [D] to download + serve <BROUGHT model>"
(the fine-tune, not the sibling) — which #628's --apply-swap now fulfils,
- raw verdict + backend note dimmed for debugging, not the headline.
Non-swap cases (eligible / Route A / B / plain no-fit) are byte-unchanged.
Regression test added; c3 fast suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
The original OOM at 98K was caused by MTP, not the context length. Two
5090s (#613 @guybrush01, #617 @paulp83) swept the space: MTP-on at 98K
leaves no room for the draft head + cudagraphs + GDN prefill scratch and
OOMs — but at 65K + util 0.85 it PASSES, with MTP's ~2x intact:
verify-stress all-pass, decode 129/155 TPS (vs 71/71 with SPEC=off), MTP
accept ~3.2, ~1.4 GB VRAM free.
So the default keeps MTP and trades context, rather than dropping MTP:
- compose single/nvfp4/mtp.yml: MAX_MODEL_LEN 98304 -> 65536,
GPU_MEMORY_UTILIZATION 0.92 -> 0.85, header updated to the swept config.
- registry: max_ctx 98304 -> 65536, mem_util 0.92 -> 0.85, status_note.
- BENCHMARKS: guybrush's SPEC=on+65K row (129/155 decode, accept 3.2).
- baselines.yml: 27B single-nvfp4 submission (stale=false, v0.24.0).
SPEC=off stays exposed as the max-ctx fallback (81K/98K @ 71 TPS) and the
tight-system-RAM path (MTP's draft load OOM-kills a 28 GB host, #617).
80 GB+ cards raise MAX_MODEL_LEN toward 262K with MTP on. Stays 🧪 (8-pack
quality still owed). Scoped guards green (baselines / status-drift /
switch+launch parity / profiles-compat).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The BYO fit-check's Route-C weight-swap (pull.sh _swap_path) hardcoded
`drop_spec_config: True` on the premise "a generic repo carries no MTP
head." That's false for a fine-tune that PRESERVES the head — so any such
model (e.g. ThinkingCap-Qwen3.6-27B, whose AutoRound INT4 ships the full
15-tensor BF16 mtp head in model_mtp_bf16.safetensors) was silently served
MTP-off, losing spec-dec despite the head being present.
Fix: detect it. `deriver.detect_mtp_head(config, api)` returns True when the
config DECLARES the MTP layers (`mtp_num_hidden_layers` / nested
`num_nextn_predict_layers`) AND a dedicated mtp weights file is present —
ground-truth for the separate-file layout fine-tune re-quants use, using
signals the deriver already has in hand (no extra fetch). Exposed as an
additive `has_mtp_head` on der.profile; _swap_path sets
`drop_spec_config = not has_mtp_head`. An embedded-head repo (head baked
into the shards with no named file) still falls back to drop — conservative,
and the named-file layout is the norm.
Validated on real HF metadata: josefprusa/ThinkingCap AutoRound →
has_mtp_head=True (keep --speculative-config); a plain AWQ re-quant →
False (drop). test-pull-swap replaces its tautology check with a real
detect_mtp_head unit test (declares+file, plain, nested text_config,
declares-but-no-file). Pull-gate suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The ⚠️-caveat required a clean v0.24.0 4-card fp8/e4m3-KV report to
upgrade to ✅. Two landed:
- #584 (@ryanmpelletier, 4× 3090 x16, Proxmox): attention backend
FLASHINFER (the #595 int8-PTH→fp8/e4m3 flip live at TP=4), verify-full
9/9, verify-stress clean to 240,634 tok, soak-continuous PASS, bench
n=5, and the full 8-pack at 111/150 — ties the 2-card dual-max proxy
(109/150), so max-tier quality is TP-invariant (measured, not assumed).
- #625 (@MoppelMat, 4× 3090 bare metal, mixed x4/x8, 300 W): a second
independent 4-card rig — decode 79/102, prefill-90K clean, soak PASS.
Decode beats ryan's x16 rig (bare metal); prefill trails ~30% (narrow
lanes are the bandwidth-bound bottleneck; decode is lane-insensitive
at TP=4) — a clean decode(compute)/prefill(interconnect) split.
Changes:
- compose header: Status ⚠️ → ✅, drop the Caveats block, Quality now
cites the 4-card 111/150, prose updated to the promotion evidence.
- registry: status caveats → production + refreshed status_note.
- baselines.yml: replace @Whamp's doubly-stale #446 int8-PTH row (v0.22.0)
with ryan's fresh v0.24.0 fp8-KV submission (TPS + 8-pack + ctx + soak);
MoppelMat noted as the corroborating 2nd rig. Emit computes stale=false.
- BENCHMARKS.md: append both v0.24.0 4-card rows (append-only history).
- MULTI_CARD.md: drop the "no 4-card bench yet / needs TP=4 confirmation"
prose now that both reports have landed.
Guards green: test-baselines, test-compose-status-drift,
test-{switch,launch}-registry-parity.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
@guybrush01's 5090 (#612) independently reproduces @paulp83's #619 to
within noise (decode 257.3/258.0 vs 255.8/257.9) AND validates deeper —
verify-stress needle-clean to 120K (91% of 131K) vs #619's 29K, at the
full 131K (no derate; the 35B MoE's tiny KV holds on a 32 GB 5090).
Two independent 5090s agreeing is stronger evidence than one, so the
1x5090-pcie submission now carries the deeper 120K ctx_validated and
credits both submitters/sources. TPS unchanged (identical within noise).
8-pack quality still owed on both (neither built the benchlocal sandboxes).
Responded on #612 (validation) + #617 (paulp83's 27B — a DIFFERENT failure
from #613: engine never loaded / 525 MiB container, MTP + 28 GB RAM, not the
#613 post-boot VRAM OOM). learnings updated (35b + the 27b two-failure note).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
@paulp83's single-5090 run is the first time the NVFP4 tier has booted
anywhere. Boots clean (vLLM v0.24.0, quant=modelopt_mixed, fp8 KV @131K,
MTP off as shipped), verify-full 9/9, verify-stress needle-clean (9.8K +
29K), soak-continuous PASS (15 MiB growth / 100% retention / 0 err / p50
311). Decode 255.8 narr / 257.9 code TPS @ 60 ms TTFT — ~2.5-3x our 2x3090
AutoRound tier (native FP4 GEMM). VRAM 30.6/32 GB @131K — tight but flat.
- registry: de-blind the status_note (drop "never booted" → first 5090
validation with numbers); stays 🧪 (8-pack quality still owed — that run's
benchlocal sandboxes weren't built).
- baselines.yml: first NVFP4 row — a 1x5090-pcie submission (narr/code TPS,
clean@29K, soak PASS, engine v0.24.0, tier submitted); surfaces ⑂-labelled
in c3 like the multi-max submission. No primary row (we can't run it).
- compose header: AUTHORED BLIND block → FIRST COMMUNITY VALIDATION with the
gate results + the 98304 fallback note.
- BENCHMARKS.md: the first NVFP4 row in the 35B-A3B section.
Confirms the two design calls: MTP-off (speculative_config=None) and fp8 KV
at scale=1.0 (declared-not-shipped scales) both loaded correctly on the
modelopt path. learnings + memory updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Per maintainer call: "cluster" conventionally means multiple networked
machines (a non-goal here — LiteLLM fronts multi-host) and is best reserved
for a future enterprise/multi-node meaning. "Pod" is the accurate analogy
for what this is — one model on a GPU subset on ONE host (k8s/RunPod sense).
The capability is unchanged; only the name.
Scoped rename (cluster→pod, case-aware) across the pod feature ONLY:
- scripts/cluster.sh → scripts/pod.sh; test-cluster-cli.sh → test-pod-cli.sh;
docs/CLUSTERS.md → docs/PODS.md
- estate_cli.py verbs + wording; app.py (ClusterCreateScreen→PodCreateScreen,
action_new_cluster→new_pod, _populate_clusters→_populate_pods, #cluster-view
→#pod-view, the [N] help/empty-state text); services.py cluster_create_plan
→pod_create_plan; data.py kind cluster_create→pod_create; tests + doc
pointers (HARDWARE/MULTI_CARD/README/c3-README)
- UNTOUCHED (unrelated "cluster"): compat.py + test-profiles-compat.sh (the
VRAM-topology classifier), services.py:1863 / test_services.py (the scene-
table "cluster by group" verb), older docs, .venv
Also folds in the [N] discoverability fix (n was already bound to
serving_switch — moved to N; empty-estate now shows a "no pods — [N] new
pod" affordance + a help entry) and a heterogeneous-rig section in PODS.md:
one homogeneous pod per card family (2×3090 · GB10 · 6000 Pro) is the clean
pattern — mixing families in one TP pod makes NCCL wait on the slowest +
wastes VRAM; a worked 2-pod lifecycle walkthrough.
Verified: test-pod-cli + estate/gpu/profiles guards green; 25 c3 pod/binding
+ 239 fast tests green; pod.sh live create/list/D1-reject on 2×3090; zero
stray "cluster" in pod files (scene-verb preserved); no CLUSTERS.md links
left; PODS.md leak-clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The cockpit surface for multi-model clusters: the Operate/Orch pane now
groups estate instances (clusters) with their GPUs stacked and a placement
health badge fed by the Phase-A assertion.
- estate_cli.py D3 (#610 addendum 3): report-state's active_estate.instances
now carries a per-instance placement verdict {requested, actual,
placement} (only probed for RUNNING instances). ONE source feeds both
cluster.sh status and the cockpit — no second poll.
- app.py C1: _populate_clusters renders a #cluster-view block below the
serving line — per cluster a header (● name · slug · :port · badge) with
GPUs stacked beneath (└─ GPU0 GPU1) + a trailing free-GPU line. The badge
is the Phase-A verdict: ✓ placed / ⚠ PLACEMENT MISMATCH, so the view shows
where clusters ACTUALLY landed (the failure mode that opened #610). Hidden
when the estate declares no clusters — the single-model case is unaffected.
- test-cluster-view headless: grouping + GPU stacking + ✓/⚠ badges + free
line; empty-when-no-clusters. Serving-panel siblings unaffected.
LIVE-VERIFIED (2x3090): cluster.sh up chat → report-state carries
placement=ok → C1 renders ✓ placed. 8 headless + 239 fast c3 + estate
guards green.
Design: /opt/ai/docs/serve-cockpit-design.md §10 (C1 shipped; C2 wizard is
the remaining Phase-C piece — the CLI wizard covers the capability today).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The CLI seam the c3 cluster UX (Phase C) will consume, and a full headless/
SSH cluster capability on its own. A "cluster" = a named model on a chosen
GPU set + port (an estate instance).
- estate_cli.py gains create / list / status / rm verbs (D2: NEW verbs go
IN estate_cli.py, which owns the schema + validate_estate + boot/down —
ONE validation path shared with hand-written estate files and the wizard).
up/down reuse `boot`/`down --only <name>`.
- create runs the D1 fit-vs-set logic (#610 addendum 3): kv-calc --card is
single-card + registry TP, so count != compose TP is a HARD REJECT, a
heterogeneous set is estimated against its min-VRAM card (+ note), and the
whole set is re-validated (validate_estate: GPU collision, port collision,
per-instance fits) before append. GPU indices stay index-based in the
estate file; UUIDs resolve at boot (Phase A).
- status carries the {requested, actual, placement} verdict per cluster
(the shape the c3 badge reads); list/status take --json.
- scripts/cluster.sh: the ergonomic bash front (create/list/status/up/down/
rm), a thin wrapper over estate_cli.py.
- test-cluster-cli.sh (NEW): full lifecycle hardware-free via CLUB3090_FAKE_
GPUS — create/D1-reject/collision-reject/list/status/rm.
LIVE-VERIFIED (2x3090): create chat (fit-clean ~20.2 GiB) → up → estate boot
→ "✓ placement verified" → status "serving ✓ placement=ok", GPU0 idle /
GPU1 loaded → down → rm. D1 count!=TP + GPU-collision rejects confirmed.
Part of #610 (Phase A′). Next: Phase C (c3 cluster view + wizard).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Extends the #611 primitive to the estate (multi-cluster) path and adds a
post-boot placement check — the foundation Phase A′ (cluster.sh) and C
(c3 cluster UX) sit on.
- scripts/lib/gpu-select.sh (NEW, shared lib): factors the #611 inline
resolver out of launch.sh into gpu_select_indices_to_uuids /
gpu_select_export, plus gpu_select_container_uuids +
gpu_select_assert_placement. launch.sh sources it (no re-inlined drift)
and runs the placement assertion after verify-full.
- estate_cli.py: resolve_gpu_uuids() (python twin of the bash resolver) +
compose_env / compose_override_doc now UUID-pin CUDA_/NVIDIA_VISIBLE_
DEVICES (ESTATE_GPUS stays index-based — the compose device_ids read the
host view). assert_placement[_quiet]() runs after each instance is ready
(sequential → stderr, parallel → per-instance boot log), returning the
{requested, actual, placement: ok|mismatch|unknown} verdict — the shape
cluster.sh status + the c3 badge will read.
- Placement uses --query-compute-apps=gpu_uuid (NOT --query-gpu): under CDI
the container sees all cards but RUNS on the CUDA-masked set, so
compute-apps is the runtime-agnostic ground truth.
- test-gpu-select (NEW): asserts the bash + python resolvers agree on real
hardware and both fall back identically; test-compose-gpu-mask-passthrough
updated to follow the resolver into the lib.
LIVE-VERIFIED (2x3090, classic runtime): `launch.sh --gpus 1` → UUID-pinned,
verify-full 8/8, "✓ placement verified", host GPU0 = 1 MiB / GPU1 = 20.8 GB.
estate compose_env confirmed UUID-izing while ESTATE_GPUS stays index-based.
CDI leg still awaits mog. Relevant guard sweep (11) green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Verification pass on the blind-authored claims (vLLM source + safetensors
indexes, 2026-07-06):
- vLLM modelopt SUPPORTS fp8 KV with NVFP4/mixed-precision weights by
design: KV_CACHE_QUANT_ALGOS = ['FP8','NVFP4'] + ModelOptKVCacheMethod;
our --quantization modelopt + --kv-cache-dtype fp8 pairing is canonical.
- BOTH nvidia checkpoints DECLARE kv_cache_quant_algo=FP8 in
hf_quant_config.json, but NEITHER ships k_scale/v_scale tensors (indexes
enumerate only linear input_scale/weight_scale/weight_scale_2). The
shipped composes claimed '27B bakes KV scales' and 'MoE has none, unlike
the 27B' — both wrong; the two checkpoints are IDENTICAL on KV: declared
FP8, no scales, scale=1.0 at runtime (calculate_kv_scales disabled on
Qwen3-Next hybrid) — exactly the #594-quality-tied regime.
Text-only; no flag changes (the composes were already serving the right
config — the rationale strings were wrong).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
First slice-3e quality ingest: @guybrush01's on-rig benchlocal --full
(2026-07-06, #571) lands 109/150 on vllm/qwen-27b-dual-max @ 2x5090 —
exact total parity with the 2x3090 primary's 109/150 (#594): Blackwell
native FP8 GEMM and Ampere Marlin W8A16 score the same 8-pack. n=2
on-rig runs (102 -> 109) both within the ±5-7 noise band; row carries
the better-evidenced latest run with per-pack detail + source link in
quality_env.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
nvidia/Qwen3.6-35B-A3B-NVFP4 (modelopt MIXED_PRECISION MoE: NVFP4 gs16
expert FFNs + FP8-static attention + unquantized mtp.* head, 23.4 GB, 3B
active) — sibling of the 27B nvfp4 pair (#603), same authored-blind /
required_sm=9.0 / community-validated pattern. The single slug is the
GB10/DGX-Spark marquee: a 3B-active MoE is the right shape for
big-capacity, lower-bandwidth unified-memory parts — kv-calc says full
262K at 25% of a GB10's budget (MoE KV is tiny, ~1.3 GB @131K).
Two deliberate deltas vs the 27B nvfp4 composes:
- NO MTP (drafter=None): our measured finding on this MoE — the built-in
head shares the MoE forward and is net-negative (-51%); headers forbid
blind re-adding.
- fp8/e4m3 KV at scale=1.0: this checkpoint has NO baked KV scales
(kv_cache_scheme null, unlike the 27B) — same regime the production 27B
fp8 tier quality-tied at (#594); flagged as a known-unknown.
- vllm/qwen-35b-a3b-single-nvfp4 (:8078, 🧪) — 131K default for 5090
32 GB (kv-calc 27.4/29.4 @93%); GB10 raises to 262K via MAX_MODEL_LEN.
- vllm/qwen-35b-a3b-dual-nvfp4 (:8079, 🧪) — TP=2 @262K, 15.7 GB/card
(53%) on 2x 5090.
Wiring mirrors #603: weights entry + weights.py aliases + kv-calc
(weights_nvfp4_gb on the MoE spec, nvfp4 branch, cfg map, aliases) +
registry entries (kvcalc-priced, no DEFAULTS rows) + counts 61/62. Full
scripts/tests suite + kv-calc --calibration green; sm_86 → incompatible-hw
(auto-hidden in c3 per #604), rtx-5090 → fits-clean 27.4/15.7 GB.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
nvidia/Qwen3.6-27B-NVFP4 (modelopt v0.45 MIXED_PRECISION: NVFP4 gs16 FFN +
FP8-static attention + FP8 KV scales baked + unquantized mtp.* head, 21.9 GB,
full VLM) — the first catalog tier this dev rig CANNOT run: NVIDIA supports
Hopper + Blackwell only, so both slugs are AUTHORED BLIND and gated
required_sm=9.0. The first community boot is the validation (funnel /
numbers-from-your-rig); compose headers carry the exact ask + known-unknowns
(MTP-on-modelopt, Cliff 2 on Blackwell, froggeric template).
- vllm/qwen-27b-single-nvfp4 (:8076, 🧪) — 131K default sized for the
smallest target (5090 32 GB: kv-calc 28.5/29.4 GB PASS @97%); GB10 128 GB
runs the FULL 262K at 29% budget (4 streams @262K = 50%) — the single-card
home; H100 @262K = 46%. MAX_MODEL_LEN env raises it.
- vllm/qwen-27b-dual-nvfp4 (:8077, 🧪) — TP=2 @262K full ctx, mirrors the
production dual-max shape with ~11 GB/card weights (vs 14.5 FP8) → 2x 5090
primary target; kv-calc 22.1 GB/card (75% budget).
Wiring, per ADDING_MODELS + the compat model:
- weights entry (format=modelopt, hf_repo, 22 GB) + weights.py aliases
- engines/vllm-stable.yml supported_weight_formats += modelopt (stock vLLM
supports --quantization modelopt; the FP4 KERNELS need sm>=9.0 which is
encoded per-slug via required_sm, not at the engine — loader is
arch-agnostic)
- kv-calc: weights_nvfp4_gb in the qwen spec, nvfp4 branch in
_weights_per_card_gb, weights_variant pass-through for qwen3.6-27b compose
cfgs, nvfp4-single/nvfp4-dual aliases, CLI --weights-variant choice
- registry: both entries kvcalc_key-wired (NOT SKIP — kv-calc predicts fit
on hardware we don't own, which is the point), no DEFAULTS rows
- KV is fp8_e4m3 NOT nvfp4-KV (consumer Blackwell has no FP4 FMHA — see
hardware/rtx-5090.yml + vllm#43562); this checkpoint bakes FP8 KV scales
Gates: full scripts/tests suite green (registry-disk counts 59/60,
profiles-compat + diagnose-profile fit on the 5090 canonical scenarios),
kv-calc --calibration green. Honest refusal verified on this rig: C3
(sm >= 9 floor) + C5 (fp8_e4m3 KV) reject on rtx-3090; switch.sh --list
shows both as (NA: experimental).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
ryan's Proxmox VM (no python3-yaml) fails switch.sh with
`ModuleNotFoundError: No module named 'yaml'` — surfaced by #599's
error-unswallowing. The launcher table path only used PyYAML for one
thing: pulling container_name out of each compose. Three-layer fix:
1. PyYAML is now OPTIONAL on the table path: a regex container_name
fallback (first non-comment `container_name:` line + the existing
${VAR:-default} unwrap) kicks in when yaml is absent.
CLUB3090_EMIT_NO_YAML=1 forces the fallback so CI can exercise it.
2. The --json contract path (c3 / baselines join) legitimately requires
PyYAML — it now checks FIRST (before load_profiles, which also
imports yaml) and fails with an actionable Fix: line
(`sudo apt install python3-yaml`), not a bare traceback.
3. Output side of the #599 locale class: a PIPED stdout under LC_ALL=C
defaults to ASCII → UnicodeEncodeError printing the unicode in
status notes. Both emit blocks now pin
sys.stdout/stderr.reconfigure(encoding="utf-8").
New guard test-registry-emit-no-yaml.sh asserts byte-identical table
output across {baseline, no-yaml, C-locale, no-yaml+C-locale (the
literal #584 rig)} and the actionable --json refusal. Live-validated:
`CLUB3090_EMIT_NO_YAML=1 LC_ALL=C PYTHONUTF8=0 PYTHONCOERCECLOCALE=0
bash scripts/switch.sh --list` renders all 48 variants. AGENTS.md
encoding section extended with the stdlib-only + write-side rules.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
prism-pro/apex slugs showed the coarse "gguf" in the Weights column —
their tokens (ex0bit-prism-pro-dq, mudler-apex-*) are custom-named
mixed-quant packs with no quant segment to extract. Ground truth read
from the GGUF headers' general.file_type (local files; apex-quality via
an HTTP Range read of the HF artifact):
ex0bit-prism-pro-dq → Q3_K_M → q3km (5 slugs)
mudler-apex-compact → Q4_K_M → q4km (3 slugs)
mudler-apex-quality → Q6_K → q6k (1 slug)
Baked as an optional `quant_label:` on the weights entries (provenance
comments inline), emitted as weights_quant_label alongside weights_format,
threaded onto the row. Label fallback chain: token quant segment →
quant_label → format → raw token. Sweep: zero "gguf" labels remain across
all 57 slugs. Community pack authors can set quant_label in their PRs.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The Weights column hand-map covered 12 of the catalog's 30 weights_variant
tokens; the other 18 fell to a first-'-'-segment fallback that showed the
PROVIDER prefix instead of the quant — "beellama", "unsloth",
"deepreinforce", "mudler" — wrong for ~24 of 57 slugs.
Replace the map with a pattern extractor: an explicit GGUF quant segment
(q4km / q8kxl / iq4ks / q6kp …) wins; then the known safetensors formats
(nvfp4 / w4a16 / awq→awq4 / autoround→int4·AR|int8·AR / fp8 / bf16); else
fall back to the model profile's `format:` — threaded through the emit as
weights_format (models/<id>.yml weights[<variant>].format, joined once) —
so fine-tune artifact tokens with no quant segment (mudler-apex-compact)
honestly read "gguf" instead of the provider name. Verified against the
live emit: all 57 slugs label correctly, zero provider-looking labels.
Also adds the missing ⑂ legend to the catalog status line ("⑂ =
community-submitted numbers (other rig) — not a local baseline"), shown
whenever any loaded row's measurement carries submission_rig — the marker
shipped in #598 but was undecodable without opening the slug detail card.
Tests: 4 new (labeller regression ×3 + legend headless), 8 catalog headless
+ 239 fast + switch/launch parity guards green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Surfaces each slug's weight-quant and KV-cache format in the Catalog table —
useful after the fp8-KV arc (which tier uses which KV is now a real
differentiator). Adds `kv_format` to the registry-emit --json contract (from
COMPOSE_REGISTRY) + threads it onto the VariantRow (same object.__setattr__
pattern as the other facets); weights comes free from the existing
CatalogEntry.weights_variant. Two compact columns appended after `engine`:
weights: fp8 · int4·AR · awq4 · qat4 · bf16 · iq4ks …
kv: fp8/e4m3 · fp8/e5m2 · int8-PTH · tq3 · bf16 · q4_0 …
e.g. vllm/dual → int4·AR / fp8/e5m2 ; dual-max → fp8 / fp8/e4m3 ;
dual-balanced → awq4 / int8-PTH. 239 c3 tests + registry-parity guards green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
registry_variant_rows read the compose YAMLs + baselines.yml via Path.read_text()
with NO encoding -> the locale default. On a non-UTF-8 locale (e.g. a minimal
Proxmox VM's ASCII / ANSI_X3.4-1968), reading a compose header containing unicode
(the long-present "—", plus "× → ⚠ ≈" added in #594/#595) crashes with
UnicodeDecodeError, which container_name() re-raises -> the whole emit dies ->
switch/launch fail with "could not derive variant tables from compose_registry.py".
Reported by @ryanmpelletier (#584) after pulling #595.
Fix: pass encoding="utf-8" to all three reads (compose container_name, compose
reaches-scan, baselines join). Also surface the previously-swallowed python
traceback in derive_switch/launch_variant_tables (the 2>/dev/null hid the real
cause) so future emit failures are debuggable.
Verified: default read_text() crashes under LC_ALL=C ASCII on the unicode compose
("byte 0xe2"); encoding="utf-8" reads clean. Emit (69 rows) + switch/launch
registry-parity + status-drift guards green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The catalog join (services.py) skipped submission-only slugs, so the main TPS
column stayed blank for 4-card slugs (multi-fast, multi-max) — their numbers
only appeared in the detail panel. Now the join falls back to the BEST cross-rig
submission, tagged submission_rig, and Measurement.tps_label renders it
⑂-labelled (a submission is NOT this rig's own on-rig bar). Updated the slice-3
test to assert the new behaviour.
Also seed multi-max as a stale-flagged @Whamp #446 submission (int8-PTH,
pre-#595 fp8 flip) so it shows *something* with the caveat until the pending
4-card fp8 re-test (#584) lands.
Result: multi-fast -> "75/91 ⑂ · 8pk 108/150"; multi-max -> "85/102 ⑂ †".
239 c3 tests + baselines guards green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The Bring pane's slug-detail card (_funnel_slug_details) rendered only the
primary "bar", so submission-only slugs — multi-fast (4-card, no on-rig bar) —
showed nothing. Render the cross-rig ⑂ submission lines there too (design:
NEVER merged into the bar; a 4-card number isn't this rig's bar), and add the
omitted quality_8pk to the ⑂ line in both renderers.
Also fill vllm/dual + vllm/qwen-27b-dual-fast quality_8pk = 108/150, carried
from multi-fast #584 (ryan, 4x3090, same AutoRound-INT4 weights) — TP-invariant
(4-card = 2-card; TP is not a quality lever, confirmed on #584); provenance
noted in quality_env. Replace with an on-rig 2-card --full if one is run.
Result: multi-fast now shows "⑂ 4x3090-pcie 75/91 TPS · 8pk 108/150"; fast tier
shows its 8pk. multi-max stays blank (no data yet — fills from its pending
4-card fp8 re-test, #584). 239 c3 tests + baselines guards green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
guybrush01 ran the full 8-pack (thinking-off) on dual-max on his 2x 5090:
102/150 — within +-5-7 noise of our 2x3090 fp8 (107). So FP8 is near-lossless
on native Blackwell too, which closes the 'route Blackwell -> FP8 weights'
recommendation gate (fp8 quality confirmed on BOTH Ampere-Marlin and native
Blackwell fp8).
- baselines.yml: guybrush's 2x5090-pcie dual-max submission gains
quality_8pk: 102/150 + quality_env (harness fingerprint).
- One dip noted: dataextract 9/15 (vs 13 elsewhere) — mostly verifier_fail,
the known DE brittleness cluster, not an obvious fp8 regression. thinking-on
run pending.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
@ryanmpelletier rebuilt the benchlocal sandboxes and ran the full 8-pack on
his 4x 3090 (v0.24.0, thinking-off): TOTAL 108/150 — within +-1 of the 109/150
we carry from the 2-card fast tier (vllm/dual). So multi-fast's quality is now
MEASURED TP-invariant on real 4-card hardware, not assumed.
- baselines.yml: ryan's 4x3090-pcie submission gains quality_8pk: 108/150 +
quality_env (harness provenance — the first quality ingest into a submission
row, the friction-#8 / slice-3e hook: a quality number carries its harness
fingerprint so it's reproducible).
- compose header Quality: 'open follow-up' -> the measured 108/150 confirmation.
Closes the last open item on the multi-fast promotion (bench + soak + quality
all confirmed on 4-card). Does NOT touch multi-max's caveat (that needs an
fp8 4-card report, not INT4).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The 4-card FP8 max tier is byte-identical to the now-production
vllm/qwen-27b-dual-max apart from TP=4 + gpu-count. We can't self-validate
(2-card dev rig), so it rides @Whamp's cross-rig full chain — #446, 4x 3090:
verify-full + verify-stress 7/7 + soak-continuous PASS (85/102).
Status experimental -> ⚠️ Production w/ caveats (registry 'caveats' + header
Caveats line + drift guard green). The caveat, stated honestly: that
validation was on an OLDER engine (pre-v0.24.0 pin) + a non-standard rig
(aikitoria P2P kernel, mixed x4/x16/x8/x16 lanes), single report — no clean
v0.24.0 4-card datapoint yet. A fresh one (like @ryanmpelletier's for
multi-fast) upgrades it to bare ✅.
No baseline row inducted: @Whamp's number is under-specified (engine version
unstated, non-standard rig), so inducting it as THE bar would mislead — the
status_note carries the provenance + caveats instead. No DEFAULTS change.
Removed the now-contradictory '🧪 Experimental until a ≥4x host validates it'
prose from the header.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The FP8 max-accuracy tier had bench (83.1/108.2) + quality (107/150) on the
2x3090 reference rig but no soak-continuous run — the one missing gate item.
Ran the full operational gate fresh on the v0.24.0 pin:
- verify-full 9/9
- verify-stress: all 6 rungs, fillable to 240,636 tok clean (91%)
- soak-continuous PASS: 0 err / 0 growth / 100% retention, p50 decode 85
That clears the production bar.
- registry + compose header: status experimental -> production (drift guard
green). status_note also de-staled: dropped the '~56 TPS' probe / 'slowest of
the three' framing (corrected to decode 83/108, the slow axis is PREFILL/TTFT
from MarlinFP8 W8A16 on Ampere, not decode) + the full-gate results.
- baselines.yml: row comment notes the soak PASS completing the gate.
- No DEFAULTS[(qwen,vllm,dual)] change -> vllm/dual (fast) stays the dual
default; dual-max just joins the actionable list as the max-fidelity tier.
Bonus for the 5090 crowd: dual-max is now a non-experimental config, so the
launch command drops --force (bash scripts/switch.sh vllm/qwen-27b-dual-max).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
#588 fixed the fp8-weights DeepGEMM crash but the _deepgemm_env gate (and my
guard) matched weights_variant == 'fp8' EXACTLY — missing agents-a1, whose
weights are 'fp8-dynamic' (compressed-tensors FP8). Those are genuine FP8
weights that route FP8 GEMM, so a 5090 running vllm/agents-a1-dual would still
hit the 'recipe not found' crash the whole framework is meant to prevent.
- _deepgemm_env gate: weights_variant == 'fp8' -> startswith('fp8'), catching
both 'fp8' and 'fp8-dynamic' (compressed-tensors). INT4/AWQ/W8A8/bf16 never
route FP8 GEMM so they stay correctly excluded.
- agents-a1 compose: add the - VLLM_USE_DEEP_GEMM pass-through.
- test-deepgemm-fp8-parity: broadened to startswith('fp8') so it now covers
agents-a1 (5 fp8-family composes) + any future fp8-* variant.
- test-launch-compat: new case — fp8-dynamic on 5090 injects VLLM_USE_DEEP_GEMM=0.
Now ALL fp8-family vLLM slugs are 5090-safe via the launcher (dual-max,
multi-max, dual-lmcache, diffusiongemma-dual, agents-a1). Convention: fp8
weight variants must be named fp8-* for the gate/guard to catch them.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
@ryanmpelletier's flawless 4x 3090 report (#584) is the cross-rig
validation the compose header required ('until a real >=4x 3090 host
validates it'): v0.24.0, no power caps, all x16, verify-full 9/9,
verify-stress needles clean to 240K, soak-continuous PASS, bench n=5.
Third concordant 4-card validation (after @alanspires 6xVFIO + @Whamp),
first on v0.24.0.
- registry + compose header: status experimental -> production (both, drift
guard green). No DEFAULTS[(qwen,vllm,multi4)] entry -> no resolver cascade;
promotion just moves it onto the actionable list. Quality is TP-invariant,
carried from the vllm/dual proxy (109/150); 4-card 8-pack is the open
confirmation (report.sh runs none).
- baselines.yml: vllm/qwen-27b-multi-fast gains its 4x3090-pcie submission
(tier: submitted; decode 74.76/90.83, prefill 1288->1175, KV pool
1.77M/6.77x, NIAH 240K). Submission-only by construction - the 2-card
reference rig can never produce a local 4-card row.
- BENCHMARKS.md: ryan's row (all-x16 74.8/90.8 beats @Whamp's mixed-lane
59/74 on the same fp8-KV config -> PCIe lane width matters for TP=4).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The dual-max FP8 2x3090 decode bench (83.1/108.2, v0.24.0, 2026-06-30) lived
only in BENCHMARKS.md:122 + disc #515 (c17490565) — never inducted into
baselines.yml, so c3 / the registry-emit join showed only guybrush01's
2x5090 submission and no local bar. Surfaced while diagnosing #585.
- baselines.yml: vllm/qwen-27b-dual-max gains its primary tier: local row
(83.1/108.2 · TTFT 158 · prefill 1364->875 · 8-pack 107/150 · NIAH 240K ·
v0.24.0 = current pin -> FRESH); guybrush's 2x5090 submission preserved.
- compose header + DUAL_CARD.md: the stale '~56 TPS' probe (and the now-false
'slowest of the three' framing) -> real decode 83/108; the genuine
tradeoffs (smallest KV pool 295K/1.13x, slowest prefill/TTFT 158ms from
FP8's compute-heavy Marlin W8A16 dequant) kept. BENCHMARKS.md already
corrected; this closes the two spots that still read ~56.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
