Surfaces each slug's weight-quant and KV-cache format in the Catalog table —
useful after the fp8-KV arc (which tier uses which KV is now a real
differentiator). Adds `kv_format` to the registry-emit --json contract (from
COMPOSE_REGISTRY) + threads it onto the VariantRow (same object.__setattr__
pattern as the other facets); weights comes free from the existing
CatalogEntry.weights_variant. Two compact columns appended after `engine`:
weights: fp8 · int4·AR · awq4 · qat4 · bf16 · iq4ks …
kv: fp8/e4m3 · fp8/e5m2 · int8-PTH · tq3 · bf16 · q4_0 …
e.g. vllm/dual → int4·AR / fp8/e5m2 ; dual-max → fp8 / fp8/e4m3 ;
dual-balanced → awq4 / int8-PTH. 239 c3 tests + registry-parity guards green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
registry_variant_rows read the compose YAMLs + baselines.yml via Path.read_text()
with NO encoding -> the locale default. On a non-UTF-8 locale (e.g. a minimal
Proxmox VM's ASCII / ANSI_X3.4-1968), reading a compose header containing unicode
(the long-present "—", plus "× → ⚠ ≈" added in #594/#595) crashes with
UnicodeDecodeError, which container_name() re-raises -> the whole emit dies ->
switch/launch fail with "could not derive variant tables from compose_registry.py".
Reported by @ryanmpelletier (#584) after pulling #595.
Fix: pass encoding="utf-8" to all three reads (compose container_name, compose
reaches-scan, baselines join). Also surface the previously-swallowed python
traceback in derive_switch/launch_variant_tables (the 2>/dev/null hid the real
cause) so future emit failures are debuggable.
Verified: default read_text() crashes under LC_ALL=C ASCII on the unicode compose
("byte 0xe2"); encoding="utf-8" reads clean. Emit (69 rows) + switch/launch
registry-parity + status-drift guards green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The catalog join (services.py) skipped submission-only slugs, so the main TPS
column stayed blank for 4-card slugs (multi-fast, multi-max) — their numbers
only appeared in the detail panel. Now the join falls back to the BEST cross-rig
submission, tagged submission_rig, and Measurement.tps_label renders it
⑂-labelled (a submission is NOT this rig's own on-rig bar). Updated the slice-3
test to assert the new behaviour.
Also seed multi-max as a stale-flagged @Whamp #446 submission (int8-PTH,
pre-#595 fp8 flip) so it shows *something* with the caveat until the pending
4-card fp8 re-test (#584) lands.
Result: multi-fast -> "75/91 ⑂ · 8pk 108/150"; multi-max -> "85/102 ⑂ †".
239 c3 tests + baselines guards green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The Bring pane's slug-detail card (_funnel_slug_details) rendered only the
primary "bar", so submission-only slugs — multi-fast (4-card, no on-rig bar) —
showed nothing. Render the cross-rig ⑂ submission lines there too (design:
NEVER merged into the bar; a 4-card number isn't this rig's bar), and add the
omitted quality_8pk to the ⑂ line in both renderers.
Also fill vllm/dual + vllm/qwen-27b-dual-fast quality_8pk = 108/150, carried
from multi-fast #584 (ryan, 4x3090, same AutoRound-INT4 weights) — TP-invariant
(4-card = 2-card; TP is not a quality lever, confirmed on #584); provenance
noted in quality_env. Replace with an on-rig 2-card --full if one is run.
Result: multi-fast now shows "⑂ 4x3090-pcie 75/91 TPS · 8pk 108/150"; fast tier
shows its 8pk. multi-max stays blank (no data yet — fills from its pending
4-card fp8 re-test, #584). 239 c3 tests + baselines guards green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Per llm-compressor's W8A8 guide, INT8 W8A8 quant is unsupported on compute
capability >= 10.0 (RTX 5090 sm_120, RTX 6000-Blackwell). Record it where it
matters: the experimental W8A8 "prefill corner" BENCHMARKS row (a factor
against promotion — can't serve the Blackwell community; FP8 weights stay the
Blackwell-safe 8-bit path) + a "silicon != kernel" note on the DTYPE_MATRIX
Blackwell line (the INT8 TCs exist, but the W8A8 kernel doesn't build there).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The header "Dependencies" comments cited old stable pins (v0.21.0/v0.22.0) while
the actual image (vllm-stable engine profile install.spec + running container) is
v0.24.0. Point the comments at the engine profile as the source of truth (#254 —
"the version lives in install.spec") so they don't re-drift on the next bump. Also
refresh the dual-max fp8 header's stale "re-bench needed" note with the #594 fp8
numbers. Comment-only; YAML unchanged, status-drift green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The prior 4-card validation (@Whamp #446) was on the OLD int8-PTH KV. #595 flips
to fp8/e4m3; the fp8 config is validated on the 2-card dual-max proxy (all gates
green) but not yet re-confirmed at TP=4. Request a fresh 4-card report before
upgrading ⚠️ caveats -> ✅.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Comment-only polish on the KV flip (no runtime change):
- fp8 -> e4m3 runs at scale=1.0 (checkpoint is weight-only; calculate_kv_scales
is disabled on Qwen3-Next hybrid), not "loads the checkpoint's scales"
- must be `fp8` not `fp8_e5m2` (e5m2 hard-rejected with fp8 checkpoints)
- backend FlashInfer (int8-PTH is TRITON_ATTN-only) -> flat decode at depth
- quality 109 ties int8-PTH 107; soak-continuous PASS (0 growth, 509 MB margin)
- fix comparison-table column spacing
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
guybrush01 ran the full 8-pack (thinking-off) on dual-max on his 2x 5090:
102/150 — within +-5-7 noise of our 2x3090 fp8 (107). So FP8 is near-lossless
on native Blackwell too, which closes the 'route Blackwell -> FP8 weights'
recommendation gate (fp8 quality confirmed on BOTH Ampere-Marlin and native
Blackwell fp8).
- baselines.yml: guybrush's 2x5090-pcie dual-max submission gains
quality_8pk: 102/150 + quality_env (harness fingerprint).
- One dip noted: dataextract 9/15 (vs 13 elsewhere) — mostly verifier_fail,
the known DE brittleness cluster, not an obvious fp8 regression. thinking-on
run pending.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
@ryanmpelletier rebuilt the benchlocal sandboxes and ran the full 8-pack on
his 4x 3090 (v0.24.0, thinking-off): TOTAL 108/150 — within +-1 of the 109/150
we carry from the 2-card fast tier (vllm/dual). So multi-fast's quality is now
MEASURED TP-invariant on real 4-card hardware, not assumed.
- baselines.yml: ryan's 4x3090-pcie submission gains quality_8pk: 108/150 +
quality_env (harness provenance — the first quality ingest into a submission
row, the friction-#8 / slice-3e hook: a quality number carries its harness
fingerprint so it's reproducible).
- compose header Quality: 'open follow-up' -> the measured 108/150 confirmation.
Closes the last open item on the multi-fast promotion (bench + soak + quality
all confirmed on 4-card). Does NOT touch multi-max's caveat (that needs an
fp8 4-card report, not INT4).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The 4-card FP8 max tier is byte-identical to the now-production
vllm/qwen-27b-dual-max apart from TP=4 + gpu-count. We can't self-validate
(2-card dev rig), so it rides @Whamp's cross-rig full chain — #446, 4x 3090:
verify-full + verify-stress 7/7 + soak-continuous PASS (85/102).
Status experimental -> ⚠️ Production w/ caveats (registry 'caveats' + header
Caveats line + drift guard green). The caveat, stated honestly: that
validation was on an OLDER engine (pre-v0.24.0 pin) + a non-standard rig
(aikitoria P2P kernel, mixed x4/x16/x8/x16 lanes), single report — no clean
v0.24.0 4-card datapoint yet. A fresh one (like @ryanmpelletier's for
multi-fast) upgrades it to bare ✅.
No baseline row inducted: @Whamp's number is under-specified (engine version
unstated, non-standard rig), so inducting it as THE bar would mislead — the
status_note carries the provenance + caveats instead. No DEFAULTS change.
Removed the now-contradictory '🧪 Experimental until a ≥4x host validates it'
prose from the header.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The FP8 max-accuracy tier had bench (83.1/108.2) + quality (107/150) on the
2x3090 reference rig but no soak-continuous run — the one missing gate item.
Ran the full operational gate fresh on the v0.24.0 pin:
- verify-full 9/9
- verify-stress: all 6 rungs, fillable to 240,636 tok clean (91%)
- soak-continuous PASS: 0 err / 0 growth / 100% retention, p50 decode 85
That clears the production bar.
- registry + compose header: status experimental -> production (drift guard
green). status_note also de-staled: dropped the '~56 TPS' probe / 'slowest of
the three' framing (corrected to decode 83/108, the slow axis is PREFILL/TTFT
from MarlinFP8 W8A16 on Ampere, not decode) + the full-gate results.
- baselines.yml: row comment notes the soak PASS completing the gate.
- No DEFAULTS[(qwen,vllm,dual)] change -> vllm/dual (fast) stays the dual
default; dual-max just joins the actionable list as the max-fidelity tier.
Bonus for the 5090 crowd: dual-max is now a non-experimental config, so the
launch command drops --force (bash scripts/switch.sh vllm/qwen-27b-dual-max).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
#588 fixed the fp8-weights DeepGEMM crash but the _deepgemm_env gate (and my
guard) matched weights_variant == 'fp8' EXACTLY — missing agents-a1, whose
weights are 'fp8-dynamic' (compressed-tensors FP8). Those are genuine FP8
weights that route FP8 GEMM, so a 5090 running vllm/agents-a1-dual would still
hit the 'recipe not found' crash the whole framework is meant to prevent.
- _deepgemm_env gate: weights_variant == 'fp8' -> startswith('fp8'), catching
both 'fp8' and 'fp8-dynamic' (compressed-tensors). INT4/AWQ/W8A8/bf16 never
route FP8 GEMM so they stay correctly excluded.
- agents-a1 compose: add the - VLLM_USE_DEEP_GEMM pass-through.
- test-deepgemm-fp8-parity: broadened to startswith('fp8') so it now covers
agents-a1 (5 fp8-family composes) + any future fp8-* variant.
- test-launch-compat: new case — fp8-dynamic on 5090 injects VLLM_USE_DEEP_GEMM=0.
Now ALL fp8-family vLLM slugs are 5090-safe via the launcher (dual-max,
multi-max, dual-lmcache, diffusiongemma-dual, agents-a1). Convention: fp8
weight variants must be named fp8-* for the gate/guard to catch them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
#580 added the consumer-Blackwell DeepGEMM auto-disable pass-through to
dual-max, but the other fp8-weights vLLM composes — hand-maintained
parallel copies, no extends/include — had silently drifted without it:
multi-max, dual-lmcache, diffusiongemma-dual. On a 5090 / PRO 6000 / GB10
(sm_120/121) the launcher's _deepgemm_env injects VLLM_USE_DEEP_GEMM=0, but
with no receiving line docker-compose drops it → the fp8-GEMM 'recipe not
found' boot crash #580 fixed for dual-max (disc #571).
- Add the pass-through (+ the shared comment) to all
three. All 4 fp8-weights vLLM composes now carry it.
- NEW test-deepgemm-fp8-parity: asserts every fp8-weights vLLM compose has
the pass-through — REDs on the exact drift class that caused this (verified
it fails when the line is removed, passes when restored). Scope matches the
_deepgemm_env gate (weights_variant==fp8 + vllm engine); INT4/AWQ/bf16 never
invoke DeepGEMM so they're correctly excluded.
Removes ONE 5090 blocker (fp8 crash) — NOT a blanket 'runs on 5090': beellama
composes stay broken on sm_120 (Anbeeld#85, CUDA 12.4), llama.cpp/INT4 on
sm_120 remain unvalidated (no 5090 on the reference rig).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
@ryanmpelletier's flawless 4x 3090 report (#584) is the cross-rig
validation the compose header required ('until a real >=4x 3090 host
validates it'): v0.24.0, no power caps, all x16, verify-full 9/9,
verify-stress needles clean to 240K, soak-continuous PASS, bench n=5.
Third concordant 4-card validation (after @alanspires 6xVFIO + @Whamp),
first on v0.24.0.
- registry + compose header: status experimental -> production (both, drift
guard green). No DEFAULTS[(qwen,vllm,multi4)] entry -> no resolver cascade;
promotion just moves it onto the actionable list. Quality is TP-invariant,
carried from the vllm/dual proxy (109/150); 4-card 8-pack is the open
confirmation (report.sh runs none).
- baselines.yml: vllm/qwen-27b-multi-fast gains its 4x3090-pcie submission
(tier: submitted; decode 74.76/90.83, prefill 1288->1175, KV pool
1.77M/6.77x, NIAH 240K). Submission-only by construction - the 2-card
reference rig can never produce a local 4-card row.
- BENCHMARKS.md: ryan's row (all-x16 74.8/90.8 beats @Whamp's mixed-lane
59/74 on the same fp8-KV config -> PCIe lane width matters for TP=4).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The dual-max FP8 2x3090 decode bench (83.1/108.2, v0.24.0, 2026-06-30) lived
only in BENCHMARKS.md:122 + disc #515 (c17490565) — never inducted into
baselines.yml, so c3 / the registry-emit join showed only guybrush01's
2x5090 submission and no local bar. Surfaced while diagnosing #585.
- baselines.yml: vllm/qwen-27b-dual-max gains its primary tier: local row
(83.1/108.2 · TTFT 158 · prefill 1364->875 · 8-pack 107/150 · NIAH 240K ·
v0.24.0 = current pin -> FRESH); guybrush's 2x5090 submission preserved.
- compose header + DUAL_CARD.md: the stale '~56 TPS' probe (and the now-false
'slowest of the three' framing) -> real decode 83/108; the genuine
tradeoffs (smallest KV pool 295K/1.13x, slowest prefill/TTFT 158ms from
FP8's compute-heavy Marlin W8A16 dequant) kept. BENCHMARKS.md already
corrected; this closes the two spots that still read ~56.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Three maintainer frictions from live round 2:
- label = topology/engine/model-quant ONLY — the serving-stem tail
duplicated the path axes and read as a second slug (some registry rows
carry a subpath in the file field: 'dual/piehsoft-q6k/mtp'). The stem
(basename only) is appended SOLELY to disambiguate genuine collisions
(two slugs sharing all three axes, e.g. fp8-mtp vs turbo)
- every input carries a TITLE (bare dropdowns read as anonymous fields):
field = Vertical(title + widget); titles toggle WITH their widgets so
the staged reveal stays intact
- NEW selected-slug detail card beside the HF-repo verdict: status · ctx
· port · drafter/vision · the shipped bar (TPS/8pk + provenance +
staleness dagger) · caveat note — follows the selection, hides on the
custom-slug sentinel
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Live-dogfood round 1: the slug default came from the rig-topology rule
(2 cards → dual), recommending a DUAL slug for a 5 GiB gguf — reads as
'the UI wants this on two cards'. New funnel_recommended: the smallest
fitting topology (options are size-floored + topology-sorted, so its
first group = the cheapest config that holds the artifact), preferring
the registry's curated default for the (family, topology) within it,
else the first functional-status option. The pick is starred (⭐) AND
pre-selected; the full filtered list stays reachable below it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Live-dogfood round 1 (Qwythos-9B-Claude-Mythos-5-1M-GGUF): the repo ships
base + MTP builds per quant (…-Q4_K_M.gguf / …-MTP-Q4_K_M.gguf); token-
keyed grouping merged them into ONE "2-part" variant with a summed, wrong
size (10.7 GiB shown for a 5.2 GiB pick) and no way to select just one.
Group by STEM instead (basename minus the -NNNNN-of-NNNNN part suffix) —
true multi-part shards share a stem so 'parts' still counts them; distinct
artifacts don't. Label = stem minus the repo-wide common prefix (standard
repos keep their plain quant token; multi-artifact repos keep the
distinguishing part: Q4_K_M vs MTP-Q4_K_M). Guard case added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The Bring page reveals nothing template-side until the artifact is known:
- [Inspect] runs the deriver's artifact inventory (HF metadata only, never
downloads) — a GGUF-only repo is a first-class bring, no longer
unsupported-format; lineage (base_model) rides the verdict (friction #11)
- GGUF repos: ALL discovered quants presented (sizes, multi-part counts;
mmproj split out) — the pick comes BEFORE any slug appears; the pick
Select starts BLANK so only a genuine user pick reveals stage 3
- slug Select: EVERY catalog option passing the ABSOLUTE artifact→engine
compat filter (GGUF never sees vLLM; safetensors never sees the llama.cpp
family), labeled topology-first (topology/engine/model-quant · serving),
sorted so models group under topology/engine; custom-slug escape kept
- topology floor (maintainer rule): hide topologies whose total VRAM can't
hold the weights — rig-relative (cards × min-card-VRAM × 0.90), never
fixed thresholds; ONE-directional (larger topologies never hidden —
small-quant-on-more-GPUs is legitimate); unhostable card-counts hidden;
unknown size/rig → no floor (never guess-hide)
- fit-check success surfaces the weights state: on disk → explicit ② Serve
handoff; absent → [D] downloads via the REAL pull.sh (SHA-verified,
streamed into the pane; disk write, no GPU claim), re-probed on completion
- presence probe mirrors downloader.py pull_dir/sanitize_slug with a
parity drift-guard test; HF_HOME pinned identically for probe + download
c3 suite 779/779 (11 new: staged reveal, gguf pick flow, compat filter,
label/sort, size floor, presence probe, weights line + handoff).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
artifact_inventory(api) enumerates a repo's servable artifacts WITHOUT
gating on format — a GGUF-only repo is a first-class bring (design §2b-1/2;
select_weight_files stays the vLLM/safetensors gate). GGUF variants are
enumerated at any depth, grouped by quant token with multi-part files
summed, mmproj projectors split out (never a variant); safetensors sets
reuse the adapter-excluding filter; cardData.base_model rides along
(friction #11 — lineage for ⑤'s taxonomy + credits). inspect_repo() wraps
the API fetch with structured errors; CLI: deriver.py --inventory <repo>
--json (the c3 Bring pane's INSPECT subprocess). Offline guard test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
catalog-baseline.sh extracts runner_version from the benchlocal results
JSON and stamps quality_env: { harness: "benchlocal-cli X.Y.Z" } next to
quality_8pk — the half-deployed-sandbox lesson (§2.1.4): a quality number
without its harness fingerprint is unreproducible. sandbox_digest rides
along once benchlocal emits it (upstream candidate). Arms on DIFFERENT
harness versions → loud warn + field omitted (the arm delta isn't
comparable). Guard validates the shape; fixtures updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
A NEW model has no same-model bar BY DEFINITION — the lane's primary case.
measure_vs_bar now accepts class_hint (①'s swap_path sibling, wired from
the session's fit-check) and falls back to the labeled CLASS bar:
bar_is_class + class_model ride the struct, a caveat says the deltas are
class-relative, engine-matching still applies. Without a hint the no-bar
caveat points at ① fit-check instead of dead-ending.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Bundle mode inducts a volunteer's rebench bundle into the slug's
submissions: map with provenance FROM THE BUNDLE — rig/power from rig.txt,
engine pin from container-config.json Config.Image — never from this rig
(nvidia-smi / resolve_variant_pin would stamp our fingerprint onto foreign
numbers). --source and --submitted-by are required, no $USER default.
- splice safety both directions: a primary re-induction preserves an
existing submissions map; bundle mode never touches the primary row
- one row per rig_class (newest replaces; history stays in git)
- multi-tag bundles refuse without --from-tag selection
- test-catalog-baseline: synthetic-bundle fixture covering refusals,
bundle-derived provenance, add/replace, submission-only entries,
splice preservation
Dogfood: first external row — guybrush01's #571 fp8w bundle lands as
vllm/qwen-27b-dual-max submissions[2x5090-pcie] (134.52/165.11 decode,
NIAH-clean 240,635 tok, v0.24.0, tier: submitted, TPS-only pending his
quality run).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Add the slice-3 trust-boundary schema to baselines.yml: every primary row
gains tier: local; an optional submissions: map (keyed by rig_class, e.g.
2x5090-pcie) carries cross-rig rows with required source + tier
submitted|reproduced. A slug may be submission-only (hardware we don't have).
- test-baselines: shared field validator across both row shapes; rig_class
key format + rig-field parity; tier enums; submission-only entries legal
- registry-emit _baseline_for: submissions ride the join with per-submission
staleness (pin comparison is rig-independent)
- c3: a submission-only baseline is NOT the bar (TPS column stays em-dash);
detail panel renders rig-labeled, tier-badged cross-rig lines, never merged
into the local bar
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Two fixes from guybrush01's fp8-weights 5090 run (disc #571), which needed both
to boot the fp8w arm on consumer Blackwell.
1) VLLM_USE_DEEP_GEMM: vLLM's DeepGEMM fp8-GEMM path is built for Hopper (sm_90)
+ datacenter Blackwell (sm_100/103). Consumer Blackwell (5090 / PRO 6000 /
GB10, sm_120/121) has no recipe and hard-fails "recipe not found" at boot.
Add _deepgemm_env: for fp8-weights slugs, inject VLLM_USE_DEEP_GEMM=0 on the
consumer SMs (sm_120/121 confirmed-broken; sm_89 Ada added proactively — it
routes fp8 via Marlin/CUTLASS so disabling is a harmless no-op that pre-empts
the same wall for 4090 owners). Hopper/datacenter untouched. dual/fp8/mtp.yml
gains a pass-through env; both launchers whitelist the export.
2) arch-ab.sh fp8w arm: add --force. vllm/qwen-27b-dual-max is status=experimental
so switch.sh gates it without --force.
test-launch-compat locks 5090/Ada-down / Hopper-keep / non-fp8-skip / user-pin.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The concurrency envelope spends the KV pool, but nothing sized the pool per
card: the composes default to --gpu-memory-utilization 0.92 and the launcher
never adjusted it. For discrete cards that's fine-to-conservative (their
mem_util_safe is 0.95-0.96, above the default). For DGX Spark it's a real
safety hole: its 128 GB is unified LPDDR5X shared with the Grace CPU/OS, so
mem_util_safe is 0.85 — booting at 0.92 would take ~118 GB and starve the OS.
The seeded Spark envelope row already assumes 0.85; the launch would wrongly
use 0.92.
Add _mem_util_env (same seam as _envelope_env): inject GPU_MEMORY_UTILIZATION
DOWNWARD only — when a detected card's mem_util_safe is below the compose's
registry mem_util. Heterogeneous rigs clamp to the lowest ceiling (one GMU
applies across all ranks; a unified-memory card forces the rig down). Today
this fires for exactly one card: Spark -> 0.85.
It deliberately NEVER raises above the tested default. A 3090/5090 could give
0.95-0.96, but that changes the validated Cliff-2b margin and risks boot-OOM,
so the upward move stays a validated opt-in on the soak protocol, not an
automatic bump (and the big-card concurrency ceiling is bandwidth-bound anyway,
so the extra pool mostly buys context headroom).
Both launchers whitelist the new export; test-launch-compat locks Spark-down /
discrete-no-raise / het-min / user-pin-wins.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Enhances concurrency-probe.sh into the tool that upgrades a `computed` envelope
row to `validated` (see /opt/ai/docs/phase2-soak-validation-protocol.md). All
additive — the plain fit-check stays the default behaviour.
The design's core is two row classes, two bars, and this probe serves both:
• pool-ceiling rows (5090): VALIDATE=1 fills each stream to the served
--max-model-len (or TARGET_CTX), runs 6 rounds, gates fit + >=98% TPS
retention. The value IS the kv-calc ceiling, so this is a fit+stability test.
• bandwidth-cap rows (PRO 6000 / Spark): SWEEP="4 8 12" SLUG=... TPS_FLOOR=15
reboots per N (vLLM can't hot-change max-num-seqs), probes decode-dominated,
and prints the throughput KNEE — the largest clean N whose per-stream decode
TPS clears the floor. A fit test is useless here (N=8 trivially fits 96 GB).
Key addition: streamed per-stream DECODE tok/s (TTFT-separated) — the only
honest throughput number at deep context, where prefill dominates wall time.
Retention drops the round-1 cudagraph warmup once >=4 rounds so it isn't
flattered. Machine-readable RESULT line drives the sweep's knee-finding.
SWEEP needs SLUG (refuses with exit 2 otherwise); SWEEP_DRY=1 plans the reboots
without booting. New test-concurrency-probe.sh guards syntax + refusal + dry-plan
(offline / CI-safe). Live-validated on the dev-rig Qwen: streaming TPS, floor
gate (FAIL on high floor), and VALIDATE fill all behaved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
