@paulp83's single-5090 run is the first time the NVFP4 tier has booted
anywhere. Boots clean (vLLM v0.24.0, quant=modelopt_mixed, fp8 KV @131K,
MTP off as shipped), verify-full 9/9, verify-stress needle-clean (9.8K +
29K), soak-continuous PASS (15 MiB growth / 100% retention / 0 err / p50
311). Decode 255.8 narr / 257.9 code TPS @ 60 ms TTFT — ~2.5-3x our 2x3090
AutoRound tier (native FP4 GEMM). VRAM 30.6/32 GB @131K — tight but flat.
- registry: de-blind the status_note (drop "never booted" → first 5090
validation with numbers); stays 🧪 (8-pack quality still owed — that run's
benchlocal sandboxes weren't built).
- baselines.yml: first NVFP4 row — a 1x5090-pcie submission (narr/code TPS,
clean@29K, soak PASS, engine v0.24.0, tier submitted); surfaces ⑂-labelled
in c3 like the multi-max submission. No primary row (we can't run it).
- compose header: AUTHORED BLIND block → FIRST COMMUNITY VALIDATION with the
gate results + the 98304 fallback note.
- BENCHMARKS.md: the first NVFP4 row in the 35B-A3B section.
Confirms the two design calls: MTP-off (speculative_config=None) and fp8 KV
at scale=1.0 (declared-not-shipped scales) both loaded correctly on the
modelopt path. learnings + memory updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Per maintainer call: "cluster" conventionally means multiple networked
machines (a non-goal here — LiteLLM fronts multi-host) and is best reserved
for a future enterprise/multi-node meaning. "Pod" is the accurate analogy
for what this is — one model on a GPU subset on ONE host (k8s/RunPod sense).
The capability is unchanged; only the name.
Scoped rename (cluster→pod, case-aware) across the pod feature ONLY:
- scripts/cluster.sh → scripts/pod.sh; test-cluster-cli.sh → test-pod-cli.sh;
docs/CLUSTERS.md → docs/PODS.md
- estate_cli.py verbs + wording; app.py (ClusterCreateScreen→PodCreateScreen,
action_new_cluster→new_pod, _populate_clusters→_populate_pods, #cluster-view
→#pod-view, the [N] help/empty-state text); services.py cluster_create_plan
→pod_create_plan; data.py kind cluster_create→pod_create; tests + doc
pointers (HARDWARE/MULTI_CARD/README/c3-README)
- UNTOUCHED (unrelated "cluster"): compat.py + test-profiles-compat.sh (the
VRAM-topology classifier), services.py:1863 / test_services.py (the scene-
table "cluster by group" verb), older docs, .venv
Also folds in the [N] discoverability fix (n was already bound to
serving_switch — moved to N; empty-estate now shows a "no pods — [N] new
pod" affordance + a help entry) and a heterogeneous-rig section in PODS.md:
one homogeneous pod per card family (2×3090 · GB10 · 6000 Pro) is the clean
pattern — mixing families in one TP pod makes NCCL wait on the slowest +
wastes VRAM; a worked 2-pod lifecycle walkthrough.
Verified: test-pod-cli + estate/gpu/profiles guards green; 25 c3 pod/binding
+ 239 fast tests green; pod.sh live create/list/D1-reject on 2×3090; zero
stray "cluster" in pod files (scene-verb preserved); no CLUSTERS.md links
left; PODS.md leak-clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Clusters (cluster.sh + the c3 view/wizard + the estate file + CDI pinning)
had grown into a subsystem scattered across HARDWARE.md and the c3 README.
Consolidate into a dedicated docs/CLUSTERS.md, sitting alongside
SINGLE/DUAL/MULTI_CARD as the multi-model-per-host workload axis:
- concept (cluster = model + GPU set + port = estate instance) + cluster-vs-TP
- a Defaults section answering "what happens before you make any cluster":
no default cluster / empty estate; a plain launch boots ONE model whose
GPU count = the compose's TP (prompts both/all on multi-GPU), NOT the
cluster machinery — clusters are opt-in
- CLI quickstart (create/list/status/up/down/rm) + c3 quickstart ([n] wizard,
cluster view)
- D1 fit-vs-set rules, placement verification, the estate file schema, the
command reference
HARDWARE.md keeps the GPU-pinning mechanics (UUID / CDI-NixOS — a hardware
topic) and points to CLUSTERS for management; c3 README, MULTI_CARD, and the
docs index now link CLUSTERS. All cross-links verified; public-safe.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The in-cockpit cluster-create surface — Phase C's remaining piece.
[n] in Operate · Orchestration opens ClusterCreateScreen: name (Input) ·
slug (Select of catalog slugs) · GPU set (Input, prefilled with the free
GPUs from the estate poll). The modal dismisses {name, slug, gpus}; the app
builds a cluster_create ActionPlan (requires_reconcile=False — a file write,
no GPU claim) and routes it through the standard confirm gate → cluster.sh
create, which runs the D1 fit-vs-set + validate_estate gates and refuses a
bad set (count!=TP, GPU/port collision). The modal never touches the rig
(codebase convention — same as PowerCapMenuScreen).
- app.py: ClusterCreateScreen (mirrors SettingsScreen) + action_new_cluster
/ _on_cluster_create / _cluster_free_gpus + the [n] binding, action
descriptor, and Operate/Orchestration action-context gate.
- services.py: cluster_create_plan builder (mirrors set_default: a
requires_reconcile=False file write).
- data.py: cluster_create ActionPlan kind.
- test: modal collects+dismisses the payload, a blank field stays open, and
the plan builder produces the right cmd with requires_reconcile=False.
Verified: 9 headless (C1+C2+siblings) + 51 binding/action headless + 239
fast c3 green. Plan cmd integration-checked (creates a fit-clean TP=2
cluster). Design: serve-cockpit-design.md §10 — C2 SHIPPED; #610 Phase C
complete.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The cockpit surface for multi-model clusters: the Operate/Orch pane now
groups estate instances (clusters) with their GPUs stacked and a placement
health badge fed by the Phase-A assertion.
- estate_cli.py D3 (#610 addendum 3): report-state's active_estate.instances
now carries a per-instance placement verdict {requested, actual,
placement} (only probed for RUNNING instances). ONE source feeds both
cluster.sh status and the cockpit — no second poll.
- app.py C1: _populate_clusters renders a #cluster-view block below the
serving line — per cluster a header (● name · slug · :port · badge) with
GPUs stacked beneath (└─ GPU0 GPU1) + a trailing free-GPU line. The badge
is the Phase-A verdict: ✓ placed / ⚠ PLACEMENT MISMATCH, so the view shows
where clusters ACTUALLY landed (the failure mode that opened #610). Hidden
when the estate declares no clusters — the single-model case is unaffected.
- test-cluster-view headless: grouping + GPU stacking + ✓/⚠ badges + free
line; empty-when-no-clusters. Serving-panel siblings unaffected.
LIVE-VERIFIED (2x3090): cluster.sh up chat → report-state carries
placement=ok → C1 renders ✓ placed. 8 headless + 239 fast c3 + estate
guards green.
Design: /opt/ai/docs/serve-cockpit-design.md §10 (C1 shipped; C2 wizard is
the remaining Phase-C piece — the CLI wizard covers the capability today).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The CLI seam the c3 cluster UX (Phase C) will consume, and a full headless/
SSH cluster capability on its own. A "cluster" = a named model on a chosen
GPU set + port (an estate instance).
- estate_cli.py gains create / list / status / rm verbs (D2: NEW verbs go
IN estate_cli.py, which owns the schema + validate_estate + boot/down —
ONE validation path shared with hand-written estate files and the wizard).
up/down reuse `boot`/`down --only <name>`.
- create runs the D1 fit-vs-set logic (#610 addendum 3): kv-calc --card is
single-card + registry TP, so count != compose TP is a HARD REJECT, a
heterogeneous set is estimated against its min-VRAM card (+ note), and the
whole set is re-validated (validate_estate: GPU collision, port collision,
per-instance fits) before append. GPU indices stay index-based in the
estate file; UUIDs resolve at boot (Phase A).
- status carries the {requested, actual, placement} verdict per cluster
(the shape the c3 badge reads); list/status take --json.
- scripts/cluster.sh: the ergonomic bash front (create/list/status/up/down/
rm), a thin wrapper over estate_cli.py.
- test-cluster-cli.sh (NEW): full lifecycle hardware-free via CLUB3090_FAKE_
GPUS — create/D1-reject/collision-reject/list/status/rm.
LIVE-VERIFIED (2x3090): create chat (fit-clean ~20.2 GiB) → up → estate boot
→ "✓ placement verified" → status "serving ✓ placement=ok", GPU0 idle /
GPU1 loaded → down → rm. D1 count!=TP + GPU-collision rejects confirmed.
Part of #610 (Phase A′). Next: Phase C (c3 cluster view + wizard).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Extends the #611 primitive to the estate (multi-cluster) path and adds a
post-boot placement check — the foundation Phase A′ (cluster.sh) and C
(c3 cluster UX) sit on.
- scripts/lib/gpu-select.sh (NEW, shared lib): factors the #611 inline
resolver out of launch.sh into gpu_select_indices_to_uuids /
gpu_select_export, plus gpu_select_container_uuids +
gpu_select_assert_placement. launch.sh sources it (no re-inlined drift)
and runs the placement assertion after verify-full.
- estate_cli.py: resolve_gpu_uuids() (python twin of the bash resolver) +
compose_env / compose_override_doc now UUID-pin CUDA_/NVIDIA_VISIBLE_
DEVICES (ESTATE_GPUS stays index-based — the compose device_ids read the
host view). assert_placement[_quiet]() runs after each instance is ready
(sequential → stderr, parallel → per-instance boot log), returning the
{requested, actual, placement: ok|mismatch|unknown} verdict — the shape
cluster.sh status + the c3 badge will read.
- Placement uses --query-compute-apps=gpu_uuid (NOT --query-gpu): under CDI
the container sees all cards but RUNS on the CUDA-masked set, so
compute-apps is the runtime-agnostic ground truth.
- test-gpu-select (NEW): asserts the bash + python resolvers agree on real
hardware and both fall back identically; test-compose-gpu-mask-passthrough
updated to follow the resolver into the lib.
LIVE-VERIFIED (2x3090, classic runtime): `launch.sh --gpus 1` → UUID-pinned,
verify-full 8/8, "✓ placement verified", host GPU0 = 1 MiB / GPU1 = 20.8 GB.
estate compose_env confirmed UUID-izing while ESTATE_GPUS stays index-based.
CDI leg still awaits mog. Relevant guard sweep (11) green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
--gpus selection was NVIDIA_VISIBLE_DEVICES-only, which CDI runtimes
(NixOS hardware.nvidia-container-toolkit, nvidia-ctk cdi) IGNORE — mog's
3x3090 rig got all cards exposed and vLLM took the wrong two. Index-based
CUDA_VISIBLE_DEVICES can't fix it either: the classic runtime renumbers
the exposed set in-container.
UUIDs solve both at once:
- launch.sh --gpus N,M resolves indices -> GPU UUIDs (nvidia-smi -i) and
exports BOTH NVIDIA_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES as UUIDs.
Classic runtime: exposes-by-UUID; CUDA mask agrees (renumbering-proof).
CDI: exposure comes from device_ids (nvidia.com/gpu=all) and the CUDA
UUID mask is what pins the cards. Index fallback if the query fails.
- 26 composes gain a bare `- CUDA_VISIBLE_DEVICES` passthrough (unset ->
absent -> zero change for existing users); stale commented hints removed.
- preflight: selector consumers are host-index-based -> new
_preflight_selector_normalize maps GPU-xxxx tokens back to indices at
the single choke point (the first live boot caught this: preflight
rejected the UUID selector).
- new guard test-compose-gpu-mask-passthrough.sh: every compose with the
NVIDIA env line must pass CUDA_VISIBLE_DEVICES through + launch.sh must
keep the UUID resolution.
- docs/HARDWARE.md: "Pinning specific GPUs (and CDI / NixOS runtimes)" —
the runtime table, the CDI deploy-block swap, the in-container
renumbering gotcha, manual recipes.
LIVE-VERIFIED on this rig (classic runtime, 2x3090):
launch.sh --variant vllm/minimal --gpus 1
-> host GPU0 = 1 MiB (empty), GPU1 = 20.7 GB (model), both env vars
UUID-set in-container. CDI leg validation goes to mog (NixOS reporter).
Follow-up (tracked on #610): estate ESTATE_GPUS UUID-ization for CDI
multi-instance rigs; post-boot visible-GPU-count sanity check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Verification pass on the blind-authored claims (vLLM source + safetensors
indexes, 2026-07-06):
- vLLM modelopt SUPPORTS fp8 KV with NVFP4/mixed-precision weights by
design: KV_CACHE_QUANT_ALGOS = ['FP8','NVFP4'] + ModelOptKVCacheMethod;
our --quantization modelopt + --kv-cache-dtype fp8 pairing is canonical.
- BOTH nvidia checkpoints DECLARE kv_cache_quant_algo=FP8 in
hf_quant_config.json, but NEITHER ships k_scale/v_scale tensors (indexes
enumerate only linear input_scale/weight_scale/weight_scale_2). The
shipped composes claimed '27B bakes KV scales' and 'MoE has none, unlike
the 27B' — both wrong; the two checkpoints are IDENTICAL on KV: declared
FP8, no scales, scale=1.0 at runtime (calculate_kv_scales disabled on
Qwen3-Next hybrid) — exactly the #594-quality-tied regime.
Text-only; no flag changes (the composes were already serving the right
config — the rationale strings were wrong).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The row still titled dual-max '(FP8 weights + int8-PTH KV)' with a 🧪
trailer even though the #594 flip note was already appended inside it —
exactly the mixed signal that led #602 to expect the old tier. Header
now names fp8/e4m3 (#594), status matches the registry (Production),
and a one-line fast-vs-max qualification (speed+headroom vs
weight-fidelity, 8-pack tie, no more depth penalty) + the 2x5090
cross-arch 109/150 corroboration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
First slice-3e quality ingest: @guybrush01's on-rig benchlocal --full
(2026-07-06, #571) lands 109/150 on vllm/qwen-27b-dual-max @ 2x5090 —
exact total parity with the 2x3090 primary's 109/150 (#594): Blackwell
native FP8 GEMM and Ampere Marlin W8A16 score the same 8-pack. n=2
on-rig runs (102 -> 109) both within the ±5-7 noise band; row carries
the better-evidenced latest run with per-pack detail + source link in
quality_env.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
nvidia/Qwen3.6-35B-A3B-NVFP4 (modelopt MIXED_PRECISION MoE: NVFP4 gs16
expert FFNs + FP8-static attention + unquantized mtp.* head, 23.4 GB, 3B
active) — sibling of the 27B nvfp4 pair (#603), same authored-blind /
required_sm=9.0 / community-validated pattern. The single slug is the
GB10/DGX-Spark marquee: a 3B-active MoE is the right shape for
big-capacity, lower-bandwidth unified-memory parts — kv-calc says full
262K at 25% of a GB10's budget (MoE KV is tiny, ~1.3 GB @131K).
Two deliberate deltas vs the 27B nvfp4 composes:
- NO MTP (drafter=None): our measured finding on this MoE — the built-in
head shares the MoE forward and is net-negative (-51%); headers forbid
blind re-adding.
- fp8/e4m3 KV at scale=1.0: this checkpoint has NO baked KV scales
(kv_cache_scheme null, unlike the 27B) — same regime the production 27B
fp8 tier quality-tied at (#594); flagged as a known-unknown.
- vllm/qwen-35b-a3b-single-nvfp4 (:8078, 🧪) — 131K default for 5090
32 GB (kv-calc 27.4/29.4 @93%); GB10 raises to 262K via MAX_MODEL_LEN.
- vllm/qwen-35b-a3b-dual-nvfp4 (:8079, 🧪) — TP=2 @262K, 15.7 GB/card
(53%) on 2x 5090.
Wiring mirrors #603: weights entry + weights.py aliases + kv-calc
(weights_nvfp4_gb on the MoE spec, nvfp4 branch, cfg map, aliases) +
registry entries (kvcalc-priced, no DEFAULTS rows) + counts 61/62. Full
scripts/tests suite + kv-calc --calibration green; sm_86 → incompatible-hw
(auto-hidden in c3 per #604), rtx-5090 → fits-clean 27.4/15.7 GB.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The metadata version is frozen at install time and benchlocal fixes are
pushed without bumping it, so "0.9.4" cannot identify the running code
(live proof on the dev rig: metadata 0.9.4, checkout actually at
v0.9.7-6-gc541550). pip records the truth in direct_url.json:
- report.sh: the Quality-tooling line now appends `source:` —
`git@<sha9>` for pip-from-git installs (vcs_info.commit_id), or
`git describe --tags --always --dirty` for editable checkouts (the
checkout PATH is withheld from the public report).
- quality-test.sh staleness reference: editable installs update via
`git pull` WITHOUT rewriting the console script, so the mtime
under-reports "CLI last updated" — now max(script mtime, editable
checkout last-commit time). Caught a real case on the dev rig: the
cli/hermes sandboxes were built ~1 h before the same day's last
benchlocal commit. Stale-warning timestamps now carry %H:%M so
same-day boundaries read sensibly.
test-quality-baseline / test-quality-thinking / test-report-calib green;
report section verified live + leak-clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Community rigs hit two silent quality-test failure classes: (1) runs
"skipping" the sandboxed packs because the Docker images were never built
(the #492 warning exists but --sandboxed-only still warn-and-ran a
guaranteed-useless run), and (2) rigs that pulled a benchlocal-cli update
kept scoring on OLD sandbox images until told to rebuild manually — nothing
detected the drift. And report.sh collected none of this, so triage had to
ask.
quality-test.sh:
- --sandboxed-only + missing images/Docker → HARD FAIL up front with the
build instructions (a run with zero runnable packs is never intended).
--full keeps warn-and-continue (deterministic packs are still useful).
- NEW staleness heuristic: each sandbox image's Created is compared against
the benchlocal-cli console-script mtime (rewritten on every (re)install —
portable across pip-from-git AND editable-checkout installs). Image older
than CLI → warning naming the image + both dates + the rebuild command.
Explicitly labelled heuristic (an unrelated reinstall trips it) → WARN,
never abort.
report.sh:
- New "Quality tooling (benchlocal-cli + sandboxes)" section: CLI path /
version (via the console-script's own interpreter) / install date, the 4
sandbox images with build dates + per-image OLDER-than-CLI flags + rebuild
hint, and the newest results/quality/*.json. All best-effort; piped
through redact (verified leak-clean live).
Exact-version staleness (image labels + a `sandbox-status` command) needs
upstream benchlocal-cli changes — tracked in the todo.
Verified live: hard-fail branch (docker stub), warn-continue branch,
staleness branch (fresh CLI vs real images), report section rendering.
test-quality-baseline / test-quality-thinking / test-report-calib green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
On rigs whose card can't run a slug's kernels (registry required_sm above
the local SM — e.g. the NVFP4 slugs on this sm_86 rig), the catalog showed
the slugs normally and the Download card pitched a 20 GB fetch that could
never boot. Now:
- kv-calc fit verdict gains the arch floor: fit_verdict()/fit_all_verdicts()
return {"verdict": "incompatible-hw", required_sm, card_sm, error} when
the card's SM (derived from the hardware profiles — same source as compat
C3) is below the slug's required_sm. Bare-number --card carries no arch
info → gate skipped (permissive). Applies to kvcalc-SKIP slugs too.
- Catalog: incompatible-hw rows share the DEPRECATED [h] bucket — hidden by
default, revealed by [h]; status line counts them separately
("+N incompatible-hw hidden — h"). The verdict lands with async fit
enrichment, so rows fold away on the enrichment refresh.
- Download confirm card leads with "⊘ no compatible hardware detected —
requires sm ≥ X (Hopper/Blackwell); this rig's card is sm_Y. It will NOT
boot on this machine." BEFORE the size/disk pitch — download still
proceeds (staging for another rig is legitimate). Serve card gets the
same warning line under fit.
- FitVerdict: incompatible-hw verdict + ⊘ glyph + required_sm/card_sm.
Also updates test-registry-json's VARIANT_KEYS for the #600 emit fields
(kv_format / weights_format / weights_quant_label) — the contract gate
only surfaced post-merge.
Tests: kv-calc-fit section (f) — 3090→incompatible-hw w/ sm fields,
5090→fits-clean, bare-number permissive, fit-all parity; 2 new headless
(hide+reveal+count, download-card warning). Full scripts/tests suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
nvidia/Qwen3.6-27B-NVFP4 (modelopt v0.45 MIXED_PRECISION: NVFP4 gs16 FFN +
FP8-static attention + FP8 KV scales baked + unquantized mtp.* head, 21.9 GB,
full VLM) — the first catalog tier this dev rig CANNOT run: NVIDIA supports
Hopper + Blackwell only, so both slugs are AUTHORED BLIND and gated
required_sm=9.0. The first community boot is the validation (funnel /
numbers-from-your-rig); compose headers carry the exact ask + known-unknowns
(MTP-on-modelopt, Cliff 2 on Blackwell, froggeric template).
- vllm/qwen-27b-single-nvfp4 (:8076, 🧪) — 131K default sized for the
smallest target (5090 32 GB: kv-calc 28.5/29.4 GB PASS @97%); GB10 128 GB
runs the FULL 262K at 29% budget (4 streams @262K = 50%) — the single-card
home; H100 @262K = 46%. MAX_MODEL_LEN env raises it.
- vllm/qwen-27b-dual-nvfp4 (:8077, 🧪) — TP=2 @262K full ctx, mirrors the
production dual-max shape with ~11 GB/card weights (vs 14.5 FP8) → 2x 5090
primary target; kv-calc 22.1 GB/card (75% budget).
Wiring, per ADDING_MODELS + the compat model:
- weights entry (format=modelopt, hf_repo, 22 GB) + weights.py aliases
- engines/vllm-stable.yml supported_weight_formats += modelopt (stock vLLM
supports --quantization modelopt; the FP4 KERNELS need sm>=9.0 which is
encoded per-slug via required_sm, not at the engine — loader is
arch-agnostic)
- kv-calc: weights_nvfp4_gb in the qwen spec, nvfp4 branch in
_weights_per_card_gb, weights_variant pass-through for qwen3.6-27b compose
cfgs, nvfp4-single/nvfp4-dual aliases, CLI --weights-variant choice
- registry: both entries kvcalc_key-wired (NOT SKIP — kv-calc predicts fit
on hardware we don't own, which is the point), no DEFAULTS rows
- KV is fp8_e4m3 NOT nvfp4-KV (consumer Blackwell has no FP4 FMHA — see
hardware/rtx-5090.yml + vllm#43562); this checkpoint bakes FP8 KV scales
Gates: full scripts/tests suite green (registry-disk counts 59/60,
profiles-compat + diagnose-profile fit on the 5090 canonical scenarios),
kv-calc --calibration green. Honest refusal verified on this rig: C3
(sm >= 9 floor) + C5 (fp8_e4m3 KV) reject on rtx-3090; switch.sh --list
shows both as (NA: experimental).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
ryan's Proxmox VM (no python3-yaml) fails switch.sh with
`ModuleNotFoundError: No module named 'yaml'` — surfaced by #599's
error-unswallowing. The launcher table path only used PyYAML for one
thing: pulling container_name out of each compose. Three-layer fix:
1. PyYAML is now OPTIONAL on the table path: a regex container_name
fallback (first non-comment `container_name:` line + the existing
${VAR:-default} unwrap) kicks in when yaml is absent.
CLUB3090_EMIT_NO_YAML=1 forces the fallback so CI can exercise it.
2. The --json contract path (c3 / baselines join) legitimately requires
PyYAML — it now checks FIRST (before load_profiles, which also
imports yaml) and fails with an actionable Fix: line
(`sudo apt install python3-yaml`), not a bare traceback.
3. Output side of the #599 locale class: a PIPED stdout under LC_ALL=C
defaults to ASCII → UnicodeEncodeError printing the unicode in
status notes. Both emit blocks now pin
sys.stdout/stderr.reconfigure(encoding="utf-8").
New guard test-registry-emit-no-yaml.sh asserts byte-identical table
output across {baseline, no-yaml, C-locale, no-yaml+C-locale (the
literal #584 rig)} and the actionable --json refusal. Live-validated:
`CLUB3090_EMIT_NO_YAML=1 LC_ALL=C PYTHONUTF8=0 PYTHONCOERCECLOCALE=0
bash scripts/switch.sh --list` renders all 48 variants. AGENTS.md
encoding section extended with the stdlib-only + write-side rules.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
prism-pro/apex slugs showed the coarse "gguf" in the Weights column —
their tokens (ex0bit-prism-pro-dq, mudler-apex-*) are custom-named
mixed-quant packs with no quant segment to extract. Ground truth read
from the GGUF headers' general.file_type (local files; apex-quality via
an HTTP Range read of the HF artifact):
ex0bit-prism-pro-dq → Q3_K_M → q3km (5 slugs)
mudler-apex-compact → Q4_K_M → q4km (3 slugs)
mudler-apex-quality → Q6_K → q6k (1 slug)
Baked as an optional `quant_label:` on the weights entries (provenance
comments inline), emitted as weights_quant_label alongside weights_format,
threaded onto the row. Label fallback chain: token quant segment →
quant_label → format → raw token. Sweep: zero "gguf" labels remain across
all 57 slugs. Community pack authors can set quant_label in their PRs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The Weights column hand-map covered 12 of the catalog's 30 weights_variant
tokens; the other 18 fell to a first-'-'-segment fallback that showed the
PROVIDER prefix instead of the quant — "beellama", "unsloth",
"deepreinforce", "mudler" — wrong for ~24 of 57 slugs.
Replace the map with a pattern extractor: an explicit GGUF quant segment
(q4km / q8kxl / iq4ks / q6kp …) wins; then the known safetensors formats
(nvfp4 / w4a16 / awq→awq4 / autoround→int4·AR|int8·AR / fp8 / bf16); else
fall back to the model profile's `format:` — threaded through the emit as
weights_format (models/<id>.yml weights[<variant>].format, joined once) —
so fine-tune artifact tokens with no quant segment (mudler-apex-compact)
honestly read "gguf" instead of the provider name. Verified against the
live emit: all 57 slugs label correctly, zero provider-looking labels.
Also adds the missing ⑂ legend to the catalog status line ("⑂ =
community-submitted numbers (other rig) — not a local baseline"), shown
whenever any loaded row's measurement carries submission_rig — the marker
shipped in #598 but was undecodable without opening the slug detail card.
Tests: 4 new (labeller regression ×3 + legend headless), 8 catalog headless
+ 239 fast + switch/launch parity guards green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Audit of the agent guide against current repo state; ten fixes:
- Hardware truths: reframe as "reference rig" — repo supports 10 hardware
classes (profiles/hardware/*.yml); Ampere-only constraints (no native FP8
compute) no longer stated as universal; fp8 composes are 5090-safe via the
launcher's arch-conditional VLLM_USE_DEEP_GEMM pass-through; MTP +
beellama-DFlash noted as the working spec-dec paths.
- beellama default-resolver text: it HAS live DEFAULTS entries and IS the
single-card default for qwen3.6-27b + gemma-4-31b (text said "no entries
yet, blocked on upstream image").
- Engine-pin table: vLLM pins are release tags (never nightly-* — upstream
purges them); beellama row added (digest-pinned); patches.yml named as
patch source of truth over a hand-listed set.
- New convention: encoding="utf-8" on every Python read of repo files —
non-UTF-8-locale community rigs crash on unicode compose headers (#599);
incl. the repro recipe + the no-blind-2>/dev/null corollary.
- Tests: rebench-full.sh documented as the canonical eval orchestrator;
MODEL=<served-name> 404-gotcha called out.
- New subsection: tools/serve-cockpit (c3) — own venv/pytest suite, fast vs
slow test split, registry-emit --json contract, VS16-emoji width rule.
- Full-suite scoping: catalog-shape changes only; targeted guards otherwise.
- Orphan-compose list (all 3 files gone) replaced with the compose/_archive/
convention (archive vs deprecate distinction).
- Genesis section: status line — no shipped compose enables Genesis;
guidance retained for reintroduction.
- Symlink self-description fixed: AGENTS.md is canonical, CLAUDE.md links it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The status column misaligned every column after it on rows with ⚠️/👁️/⏸️/🗑️:
those four glyphs carry a U+FE0F variation selector, so Rich's cell_len reserves
2 cols but many terminals render them 1-wide → the row's trailing cells slide
left by one. Padding can't fix it (it inflates cell_len and the column
auto-sizes to the max, breaking the other rows).
Two-part fix:
1. Swap the 4 VS16 glyphs in _STATUS_GLYPH for Emoji_Presentation=Yes
equivalents (no VS16, cell_len == terminal width == 2 everywhere):
⚠️→❗👁️→👀⏸️→🚧🗑️→🚫. Legend synced to match.
2. Move status to the LAST catalog column so its glyph — the one
variable-width cell — has nothing after it to misalign (belt + braces).
Also folds the config columns up: new order is
model · slug · weights · kv · ctx · TPS · 8pk · topo · engine · status
so the quant/KV a user picks by sits next to the identity, ahead of the
numbers. test_catalog_datatable_has_columns updated for the new layout.
9 catalog headless tests + 239 fast c3 tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Surfaces each slug's weight-quant and KV-cache format in the Catalog table —
useful after the fp8-KV arc (which tier uses which KV is now a real
differentiator). Adds `kv_format` to the registry-emit --json contract (from
COMPOSE_REGISTRY) + threads it onto the VariantRow (same object.__setattr__
pattern as the other facets); weights comes free from the existing
CatalogEntry.weights_variant. Two compact columns appended after `engine`:
weights: fp8 · int4·AR · awq4 · qat4 · bf16 · iq4ks …
kv: fp8/e4m3 · fp8/e5m2 · int8-PTH · tq3 · bf16 · q4_0 …
e.g. vllm/dual → int4·AR / fp8/e5m2 ; dual-max → fp8 / fp8/e4m3 ;
dual-balanced → awq4 / int8-PTH. 239 c3 tests + registry-parity guards green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
registry_variant_rows read the compose YAMLs + baselines.yml via Path.read_text()
with NO encoding -> the locale default. On a non-UTF-8 locale (e.g. a minimal
Proxmox VM's ASCII / ANSI_X3.4-1968), reading a compose header containing unicode
(the long-present "—", plus "× → ⚠ ≈" added in #594/#595) crashes with
UnicodeDecodeError, which container_name() re-raises -> the whole emit dies ->
switch/launch fail with "could not derive variant tables from compose_registry.py".
Reported by @ryanmpelletier (#584) after pulling #595.
Fix: pass encoding="utf-8" to all three reads (compose container_name, compose
reaches-scan, baselines join). Also surface the previously-swallowed python
traceback in derive_switch/launch_variant_tables (the 2>/dev/null hid the real
cause) so future emit failures are debuggable.
Verified: default read_text() crashes under LC_ALL=C ASCII on the unicode compose
("byte 0xe2"); encoding="utf-8" reads clean. Emit (69 rows) + switch/launch
registry-parity + status-drift guards green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The catalog join (services.py) skipped submission-only slugs, so the main TPS
column stayed blank for 4-card slugs (multi-fast, multi-max) — their numbers
only appeared in the detail panel. Now the join falls back to the BEST cross-rig
submission, tagged submission_rig, and Measurement.tps_label renders it
⑂-labelled (a submission is NOT this rig's own on-rig bar). Updated the slice-3
test to assert the new behaviour.
Also seed multi-max as a stale-flagged @Whamp #446 submission (int8-PTH,
pre-#595 fp8 flip) so it shows *something* with the caveat until the pending
4-card fp8 re-test (#584) lands.
Result: multi-fast -> "75/91 ⑂ · 8pk 108/150"; multi-max -> "85/102 ⑂ †".
239 c3 tests + baselines guards green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The Bring pane's slug-detail card (_funnel_slug_details) rendered only the
primary "bar", so submission-only slugs — multi-fast (4-card, no on-rig bar) —
showed nothing. Render the cross-rig ⑂ submission lines there too (design:
NEVER merged into the bar; a 4-card number isn't this rig's bar), and add the
omitted quality_8pk to the ⑂ line in both renderers.
Also fill vllm/dual + vllm/qwen-27b-dual-fast quality_8pk = 108/150, carried
from multi-fast #584 (ryan, 4x3090, same AutoRound-INT4 weights) — TP-invariant
(4-card = 2-card; TP is not a quality lever, confirmed on #584); provenance
noted in quality_env. Replace with an on-rig 2-card --full if one is run.
Result: multi-fast now shows "⑂ 4x3090-pcie 75/91 TPS · 8pk 108/150"; fast tier
shows its 8pk. multi-max stays blank (no data yet — fills from its pending
4-card fp8 re-test, #584). 239 c3 tests + baselines guards green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Per llm-compressor's W8A8 guide, INT8 W8A8 quant is unsupported on compute
capability >= 10.0 (RTX 5090 sm_120, RTX 6000-Blackwell). Record it where it
matters: the experimental W8A8 "prefill corner" BENCHMARKS row (a factor
against promotion — can't serve the Blackwell community; FP8 weights stay the
Blackwell-safe 8-bit path) + a "silicon != kernel" note on the DTYPE_MATRIX
Blackwell line (the INT8 TCs exist, but the W8A8 kernel doesn't build there).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The header "Dependencies" comments cited old stable pins (v0.21.0/v0.22.0) while
the actual image (vllm-stable engine profile install.spec + running container) is
v0.24.0. Point the comments at the engine profile as the source of truth (#254 —
"the version lives in install.spec") so they don't re-drift on the next bump. Also
refresh the dual-max fp8 header's stale "re-bench needed" note with the #594 fp8
numbers. Comment-only; YAML unchanged, status-drift green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The prior 4-card validation (@Whamp #446) was on the OLD int8-PTH KV. #595 flips
to fp8/e4m3; the fp8 config is validated on the 2-card dual-max proxy (all gates
green) but not yet re-confirmed at TP=4. Request a fresh 4-card report before
upgrading ⚠️ caveats -> ✅.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Comment-only polish on the KV flip (no runtime change):
- fp8 -> e4m3 runs at scale=1.0 (checkpoint is weight-only; calculate_kv_scales
is disabled on Qwen3-Next hybrid), not "loads the checkpoint's scales"
- must be `fp8` not `fp8_e5m2` (e5m2 hard-rejected with fp8 checkpoints)
- backend FlashInfer (int8-PTH is TRITON_ATTN-only) -> flat decode at depth
- quality 109 ties int8-PTH 107; soak-continuous PASS (0 growth, 509 MB margin)
- fix comparison-table column spacing
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
