The pinned cu13 ik-llama digest has a CUDA 13.2 runtime. On a driver whose
supported CUDA < 13.2 (e.g. 580.159 = CUDA 13.0, which satisfies the README's
"580.x+" but NOT this pin) the forward-compat path fails on GeForce (CUDA error
804) -> silent CPU fallback -> segfault crash-loop, with launch.sh just timing
out after 600 s and no actionable hint.
- preflight.sh: preflight_ik_llama_image() detects the driver's supported CUDA
(_driver_cuda_version) and, for ik-llama variants on a driver < 13.2 with
IK_LLAMA_IMAGE unset, auto-selects the cu12 sibling build (same build 4574,
CUDA 12.6, backward-compatible, DelspoN-validated ~68 TPS on 580.159) + prints
a ⚠ hint. >=13.2 keeps cu13; a user IK_LLAMA_IMAGE pin always wins (echoed).
_cuda_ge() does the major.minor compare. Verified: 13.3/13.2 GE, 13.0/12.6 LT.
- switch.sh: call it right after export_variant_engine_pin, before compose up
(the single up path — launch.sh delegates here). preflight.sh is already
sourced in that block.
- docs: qualify the driver requirement in the ik-llama compose header + HARDWARE.md
("580.x+" is insufficient; needs CUDA >= 13.2; auto-fallback documented).
Note: this rig is CUDA 13.3, so the <13.2 boot path is verified by logic +
DelspoN's cu12-server-4574 validation, not a local boot. Rolling cu12 tags are
NOT safe (cu12-server-100 rejects the compose's --spec-type flag, #633) — pinned
to the build-number tag matching the cu13 digest's build.
Closes#633.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(c3): ② Serve spec-dec — engine-driven drafter SELECTOR (not just on/off)
The spec-dec dropdown now lists the resolved ENGINE's supported drafters + off
(vLLM: mtp / mtp_assistant; beellama: dflash / mtp / mtp_gguf; …) instead of a
bare on/off — so the drafter TYPE is selectable, and it's engine-correct (dflash
only appears when the swap engine is beellama).
- swap_apply: the SPEC entrypoint REBUILDS --speculative-config via printf from
${DRAFTER_METHOD}/${DRAFTER_N} (defaults = the sibling's), so switching the
drafter is an env override — no re-emit. Validated: default→mtp n=3,
DRAFTER_METHOD=mtp_assistant→that config, SPEC=off→dropped, compose config valid.
- services: engine_drafters() reads the profile's supported_drafters (shares the
stdlib list-parse with engine_kv_formats via _engine_yaml_list); serve_override_
defaults adds DRAFTER_OPTIONS + SPEC_METHOD + SPEC_N.
- app: the spec Select value is the drafter method (or "off"); collect maps
method→SPEC=on+DRAFTER_METHOD, off→SPEC=off.
Drafters with extra requirements (mtp_assistant needs an assistant model; dflash
needs the beellama engine) are honestly listed per-engine — picking one the swap
can't satisfy fails at boot (loose-validation stance), or use a matching ① Bring
slug. 173 byo/serve/override tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* fix(launcher): GGUF verify-glob (#634) + launch.sh .env engine-image passthrough (#632)
#634 — setup.sh re-hardcoded VERIFY_GLOB=*.safetensors at the primary
download/verify (line 625), clobbering the recipe's glob that
load_weight_recipe already set from PRIMARY_WEIGHT_KEY. Every GGUF primary
fetch (WEIGHTS=gguf/iq4ks) then failed verify ("No *.safetensors found")
despite a byte-correct download. Now defaults to ${VERIFY_GLOB} (the recipe's
*.gguf); VERIFY_GLOB_OVERRIDE still wins.
#632 — launch.sh's .env load loop gated to CLUB3090_DEFAULT_* only, so
IK_LLAMA_IMAGE / LLAMACPP_IMAGE pins in .env were silently dropped (the two
GGUF engines are not profile-injected, so .env is their only override path —
the natural persistence for the #633 cu12 workaround never reached compose).
Broadened the allowlist to the engine-image keys; switch.sh already loaded
them (no filter). Both launchers now echo an ik-llama/llama.cpp image pin so a
wrong-image boot is never silent. (DelspoN's report cited the engine-pin
whitelist, which is the vllm/beellama-only profile-injection path — the real
drop was the .env loop filter.)
Verified: the .env loop now loads IK_LLAMA_IMAGE/LLAMACPP_IMAGE and still drops
un-allowlisted keys; test-model-switch / test-launch-compat / *-registry-parity
/ test-setup-picker / test-model-weights-registry all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Four dogfood refinements to the #636 override editor:
1. Custom values in the dropdowns. ctx / KV / util each gain a trailing
"✎ custom…" sentinel that reveals a companion free-text Input
(on_select_changed, same idiom as the ① Bring profile hatch) — so any value
the presets omit is reachable (e.g. ctx=81920, KV=turboquant_4bit_nc).
collect_overrides reads the companion when the sentinel is selected.
2. spec-dec shows the real drafter. The label is now e.g. "MTP n=3 (on)" /
"off — no spec-dec" instead of a bare "on" (value still on/off — the ${SPEC}
gate). Parsed from the sibling's --speculative-config.
3. KV types are ENGINE-driven. The KV dropdown reads the resolved engine's
supported_kv_formats (vLLM: fp8 family + int8-PTH; llama.cpp: q4_0/q5_0/…) via
the new engine_kv_formats() — not a generic list. turboquant_4bit_nc (native
in v0.24.0 but not yet in the profile's list) stays reachable via ✎ custom.
4. Slug preview. A read-only line above the fields shows what you're overriding:
engine · ctx · KV · spec · util (from serve_override_defaults, now also
parsing ENGINE + SPEC_DRAFTER; sys.path pinned so the import is cwd-safe).
171 byo/serve/override tests pass.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 2 of the ② Serve rework (Phase 1 = #635 dead-end fix): a Route-C brought
model can be re-tuned before serve, without hand-editing a compose. Fields
(pre-filled from the resolved slug's defaults, dropdowns to avoid typos):
• served name → SERVED_NAME (Input)
• ctx → MAX_MODEL_LEN (Select — presets + the slug's own default)
• KV cache → KV_CACHE_DTYPE (Select — fp8_e5m2/e4m3/turboquant_4bit_nc/…)
• spec-decode → SPEC on/off (Select)
• VRAM util → GPU_MEMORY_UTILIZATION (Select)
Mechanism (all five ride env — no per-serve compose rewrite):
- data.py: ActionPlan gains `env`; dispatch merges {**os.environ, **plan.env}.
- services.py: serve_generated(compose_path, overrides) → plan.env (+ pins
MODEL_DIR); serve_override_defaults() parses the sibling compose's
${VAR:-default} for pre-fill (stdlib regex, no PyYAML).
- swap_apply.py: emit parameterizes --served-model-name ${SERVED_NAME:-…} and
gates --speculative-config behind ${SPEC:-on} via the SAME entrypoint the
shipped nvfp4 compose uses — so SPEC=off drops the MTP drafter at up-time.
- app.py: LaneServePane fields, revealed + pre-filled only for a Route-C armed
model (hidden otherwise); collect_overrides() → serve_generated. Loose
validation (dropdowns constrain; the reconcile gate + boot are the real check).
Validated: emitted compose `docker compose config` resolves SERVED_NAME=X and
SPEC=off (entrypoint gate) cleanly; 169 byo/serve/override tests pass.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two live-dogfood issues on the Bring & Validate ② Serve stage:
1. Dead end / stale text. The ② Serve armed card said it serves "the resolved
catalog compose … NOT your brought model's weights" — the OPPOSITE of what
#628/#630 actually do for a Route-C fine-tune (they serve YOUR weights via
the sibling's recipe). And when unarmed it fell back to a dim placeholder
that reads as empty. bring-funnel-design §2b item 7 explicitly forbids this
("no dead end"). set_armed is now route-aware: Route-C → "serves <brought>
(your brought weights) via <sibling>'s recipe → [⏎] to serve"; non-swap
routes keep the honest "catalog reproduction, untested" wording.
2. ⏎ overload. #631 made ⏎ on ① Bring advance-when-present, but ⏎ there also
runs the fit-check, so it re-fit instead of proceeding. Reverted: ⏎ ALWAYS
fit-checks; a DEDICATED key [s] advances ① Bring → ② Serve (guarded on
servable + weights-on-disk; absent → notify to press [D]). [s] joins the
existing context-routed s_key (Containers restart · ④ Measure submit).
Card + weights-line now say "press [s] to continue to ② Serve".
Tests: test_bring_s_key_advances_to_serve_when_present, _no_advance_when_
weights_absent, test_bring_enter_always_fitchecks_never_advances, test_serve_
armed_route_c_serves_your_weights; updated the s_key-gating test to include
tab-bring. 195 touched-surface tests pass.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #630 card told the user to "press [⏎] ② Serve" when the brought weights
were on disk, but ⏎ on the ① Bring tab re-ran the fit-check (its _validate_
primary → _trigger_lane_bring), so pressing it just re-fit instead of
proceeding — the reported confusion.
Make ⏎ DWIM on ① Bring (_bring_primary):
- an already fit-checked, servable target whose weights are ON DISK (and the
repo input is unchanged) → ADVANCE to the pre-armed ② Serve tab (⏎ there
serves via the #630 emit-and-serve path);
- otherwise (fresh/changed target, or weights absent) → run the fit-check as
before. Absent stays on fit-check because the card there points at [D]
(download is the real next step). The [Fit] button still forces a re-check.
Also fixes the stale "([2/]] next stage)" hint on the weights-line → "(press
[⏎] to continue)", and the present-card next-step wording.
Tests: test_bring_enter_advances_to_serve_when_present (present → active tab
flips to tab-serve) + test_bring_enter_refits_when_weights_absent (absent →
stays on tab-bring). 156 touched-surface tests pass.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A Route-C fit-check whose weights were already downloaded showed a
self-contradiction: the verdict card hardcoded "→ Press [D] to download +
serve" while the weights-line below it correctly said "✓ weights on disk →
② Serve". [D] is not only download — for a Route-C brought model it emits the
serve-locally swap compose that ② Serve then serves — so it can't simply be
hidden.
Fix (presence-aware, "② Serve owns the emission"):
- _byo_result_text takes weights_present (probed once in the fit-check
handler, feeding BOTH the card and the weights-line so they can't disagree).
On disk → the card points at ② Serve, no [D]; absent → the [D] download
prompt as before.
- pull.sh gains --emit-only (with --apply-swap): passes do_download=False so
apply_swap emits the serve compose WITHOUT downloading — the present-weights
path.
- action_serve_untested: Route-C + weights present + no swap compose yet →
run_bring_emit_and_serve emits via pull.sh --apply-swap --emit-only, then
serves it. So a present-weights brought model serves straight from ② Serve
with no [D] step.
Tests: fixed test_bring_result_points_forward_to_serve (stale since #628 — it
asserted "→ ② Serve" on an absent-weights card that now honestly shows [D]);
added test_bring_result_present_points_to_serve (present → ② Serve, no [D]);
extended the run_bring_download test for --emit-only. 139 touched-surface
tests pass; pull.sh --emit-only validated end-to-end.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(byo): apply-swap — download + serve a Route-C fine-tune
A curated-arch fine-tune (e.g. josefprusa/ThinkingCap-Qwen3.6-27B AutoRound
→ qwen3.6-27b) hard-stops at the pull gate's stratum-5 no-fit-model (correct
— nothing to price), so the c3 Bring [D] download failed with "download did
not complete". Route-C was pure guidance ("clone the compose, point --model
at your weights") with no code behind it — the deferred "swap_path apply".
Build it as a DISTINCT action that never touches the locked 6-stratum gate:
- scripts/lib/profiles/swap_apply.py (new): resolve the Route-C swap
(arch→curated sibling + has_mtp_head from deriver), download the brought
weights SHA-verified via downloader.download_model (a minimal EInput), and
emit a serve-locally compose that CLONES the --profile-like sibling's REAL
compose (keeping its curated chat-template / reasoning+tool parsers / MTP
wiring — NOT the derived-vllm template, which drops all that) with --model
re-pointed at a `<weights>:/brought-model:ro` mount, --served-model-name set
to the brought basename, and --speculative-config kept iff the checkpoint
carries an MTP head (else dropped). Written alongside the sibling compose as
`_brought-<san>.yml` (gitignored) so its relative ../ mounts resolve.
- scripts/pull.sh: intercept `--apply-swap` in the wrapper (strictly additive,
like --json) → run the apply-swap action; prints `[apply-swap] compose: …`.
The gate path is byte-unchanged without the flag.
- c3: services.run_bring_download gains `apply_swap=` (appends --apply-swap +
captures the emitted compose via last_swap_compose()); the [D] worker passes
it on route==C and stashes the compose; ② Serve serves that swap compose
directly (serve_generated) instead of a reproduction of the sibling's own
catalog compose.
- pull.py: fix the stale NOTE — "the bf16 base won't fit and lacks the MTP
head" → the base HAS the head; now just the size reason.
Tests: test-pull-swap.sh section 4 (emit: --model repointed, MTP kept/dropped
by head presence, curated flags + JSON args survive the YAML round-trip,
distinct container_name); test_services apply_swap flag + compose capture. The
locked-gate hard-stop tests stay green; --json fit-check byte-identical.
Validated live: pull.sh --json still no-fit-model+route-C, and apply_swap emits
a correct ThinkingCap swap compose (MTP kept).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* c3: reframe Route-C fit-check card as ✓ Servable + a clear [D] action
The fit-check card for a curated-arch fine-tune (Route-C swap) was
self-contradicting: a RED "not eligible" + a bare "no-fit-model" token at
the top, then a GREEN "② Serve is armed with <sibling>" at the bottom —
which read as a dead-end AND named the wrong model (the sibling, not the
brought fine-tune). The engine's "no-fit-model" is the *means* (generic
fit-math can't price a curated-hybrid arch), not the user's answer; the
outcome is servable via the sibling's recipe.
Reframe the Route-C case in _byo_result_text:
- green "✓ Servable — a fine-tune of <sibling>" instead of red "not eligible",
- one plain "How it serves" line (reuses the sibling's chat-template / tools /
spec-dec with your weights) instead of the --quantization/impl bullets,
- MTP kept/dropped line from has_mtp_head,
- the next-step is an explicit "→ Press [D] to download + serve <BROUGHT model>"
(the fine-tune, not the sibling) — which #628's --apply-swap now fulfils,
- raw verdict + backend note dimmed for debugging, not the headline.
Non-swap cases (eligible / Route A / B / plain no-fit) are byte-unchanged.
Regression test added; c3 fast suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The original OOM at 98K was caused by MTP, not the context length. Two
5090s (#613 @guybrush01, #617 @paulp83) swept the space: MTP-on at 98K
leaves no room for the draft head + cudagraphs + GDN prefill scratch and
OOMs — but at 65K + util 0.85 it PASSES, with MTP's ~2x intact:
verify-stress all-pass, decode 129/155 TPS (vs 71/71 with SPEC=off), MTP
accept ~3.2, ~1.4 GB VRAM free.
So the default keeps MTP and trades context, rather than dropping MTP:
- compose single/nvfp4/mtp.yml: MAX_MODEL_LEN 98304 -> 65536,
GPU_MEMORY_UTILIZATION 0.92 -> 0.85, header updated to the swept config.
- registry: max_ctx 98304 -> 65536, mem_util 0.92 -> 0.85, status_note.
- BENCHMARKS: guybrush's SPEC=on+65K row (129/155 decode, accept 3.2).
- baselines.yml: 27B single-nvfp4 submission (stale=false, v0.24.0).
SPEC=off stays exposed as the max-ctx fallback (81K/98K @ 71 TPS) and the
tight-system-RAM path (MTP's draft load OOM-kills a 28 GB host, #617).
80 GB+ cards raise MAX_MODEL_LEN toward 262K with MTP on. Stays 🧪 (8-pack
quality still owed). Scoped guards green (baselines / status-drift /
switch+launch parity / profiles-compat).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The BYO fit-check's Route-C weight-swap (pull.sh _swap_path) hardcoded
`drop_spec_config: True` on the premise "a generic repo carries no MTP
head." That's false for a fine-tune that PRESERVES the head — so any such
model (e.g. ThinkingCap-Qwen3.6-27B, whose AutoRound INT4 ships the full
15-tensor BF16 mtp head in model_mtp_bf16.safetensors) was silently served
MTP-off, losing spec-dec despite the head being present.
Fix: detect it. `deriver.detect_mtp_head(config, api)` returns True when the
config DECLARES the MTP layers (`mtp_num_hidden_layers` / nested
`num_nextn_predict_layers`) AND a dedicated mtp weights file is present —
ground-truth for the separate-file layout fine-tune re-quants use, using
signals the deriver already has in hand (no extra fetch). Exposed as an
additive `has_mtp_head` on der.profile; _swap_path sets
`drop_spec_config = not has_mtp_head`. An embedded-head repo (head baked
into the shards with no named file) still falls back to drop — conservative,
and the named-file layout is the norm.
Validated on real HF metadata: josefprusa/ThinkingCap AutoRound →
has_mtp_head=True (keep --speculative-config); a plain AWQ re-quant →
False (drop). test-pull-swap replaces its tautology check with a real
detect_mtp_head unit test (declares+file, plain, nested text_config,
declares-but-no-file). Pull-gate suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The ⚠️-caveat required a clean v0.24.0 4-card fp8/e4m3-KV report to
upgrade to ✅. Two landed:
- #584 (@ryanmpelletier, 4× 3090 x16, Proxmox): attention backend
FLASHINFER (the #595 int8-PTH→fp8/e4m3 flip live at TP=4), verify-full
9/9, verify-stress clean to 240,634 tok, soak-continuous PASS, bench
n=5, and the full 8-pack at 111/150 — ties the 2-card dual-max proxy
(109/150), so max-tier quality is TP-invariant (measured, not assumed).
- #625 (@MoppelMat, 4× 3090 bare metal, mixed x4/x8, 300 W): a second
independent 4-card rig — decode 79/102, prefill-90K clean, soak PASS.
Decode beats ryan's x16 rig (bare metal); prefill trails ~30% (narrow
lanes are the bandwidth-bound bottleneck; decode is lane-insensitive
at TP=4) — a clean decode(compute)/prefill(interconnect) split.
Changes:
- compose header: Status ⚠️ → ✅, drop the Caveats block, Quality now
cites the 4-card 111/150, prose updated to the promotion evidence.
- registry: status caveats → production + refreshed status_note.
- baselines.yml: replace @Whamp's doubly-stale #446 int8-PTH row (v0.22.0)
with ryan's fresh v0.24.0 fp8-KV submission (TPS + 8-pack + ctx + soak);
MoppelMat noted as the corroborating 2nd rig. Emit computes stale=false.
- BENCHMARKS.md: append both v0.24.0 4-card rows (append-only history).
- MULTI_CARD.md: drop the "no 4-card bench yet / needs TP=4 confirmation"
prose now that both reports have landed.
Guards green: test-baselines, test-compose-status-drift,
test-{switch,launch}-registry-parity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Surface which speculative-decoding drafter each slug enables by default,
so the Catalog answers "does this compose do MTP / DFlash / none?" at a
glance — a serving-config facet alongside weights and kv.
Derived entirely from the registry `drafter` id already emitted by
registry-emit.sh and threaded into the cockpit (row.drafter) — no emit /
registry / guard changes. `_spec_token()` maps the id to a compact
method·mechanism token: DFlash is always an external drafter (no suffix),
so the built-in-vs-external split lives only within MTP, where the suffix
names the actual mechanism — MTP (built-in head) · MTP·gguf (external GGUF
drafter) · MTP·asst (external assistant model) · DFlash · ngram (reserved)
· — (none). The full form (n= count + source) stays in the slug detail
card.
The Bring & Validate funnel mirrors the facet: the spec token is FOLDED
into the option label so two candidates identical on topology/engine/quant
but differing only by drafter disambiguate on `· MTP` instead of falling
to the serving-stem tail; no-drafter slugs stay bare.
Tests: _spec_token/_spec_label over every real drafter id; funnel fold +
disambiguation; column-set + ordering (weights·kv·spec before ctx).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The 27B nvfp4 ships MTP n=3 (the head is unquantized). On tight-system-RAM
rigs the extra draft-model load OOMs during boot — @paulp83's 28 GB laptop
5090 timed out on the 27B while the MTP-off 35B booted fine on the same
rig (#617). Previously the only workaround was hand-deleting the two
--speculative-config lines.
Both 27B nvfp4 composes (single + dual) now take `SPEC=off`:
SPEC=off bash scripts/switch.sh vllm/qwen-27b-single-nvfp4
- bare `- SPEC` env passthrough + a bash entrypoint that appends
--speculative-config only when SPEC != off (default: MTP on, unchanged).
- single: gained an entrypoint (was plain command); dual: folded into the
existing NVLink-detect entrypoint. --speculative-config moved out of the
static command into the conditional.
- headers advertise the toggle (Drafter line + the MTP comment).
Registry drafter stays qwen-mtp-builtin (MTP is still the default; SPEC=off
is a runtime override). Verified: entrypoint simulated with compose
$$→$ de-escaping — SPEC unset emits --speculative-config, SPEC=off omits it
+ prints the notice. Compose guards (mounts / status-drift / image-drift /
registry-parity / generate-compose) green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
@guybrush01's 5090 (#612) independently reproduces @paulp83's #619 to
within noise (decode 257.3/258.0 vs 255.8/257.9) AND validates deeper —
verify-stress needle-clean to 120K (91% of 131K) vs #619's 29K, at the
full 131K (no derate; the 35B MoE's tiny KV holds on a 32 GB 5090).
Two independent 5090s agreeing is stronger evidence than one, so the
1x5090-pcie submission now carries the deeper 120K ctx_validated and
credits both submitters/sources. TPS unchanged (identical within noise).
8-pack quality still owed on both (neither built the benchlocal sandboxes).
Responded on #612 (validation) + #617 (paulp83's 27B — a DIFFERENT failure
from #613: engine never loaded / 525 MiB container, MTP + 28 GB RAM, not the
#613 post-boot VRAM OOM). learnings updated (35b + the 27b two-failure note).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
@paulp83's single-5090 run is the first time the NVFP4 tier has booted
anywhere. Boots clean (vLLM v0.24.0, quant=modelopt_mixed, fp8 KV @131K,
MTP off as shipped), verify-full 9/9, verify-stress needle-clean (9.8K +
29K), soak-continuous PASS (15 MiB growth / 100% retention / 0 err / p50
311). Decode 255.8 narr / 257.9 code TPS @ 60 ms TTFT — ~2.5-3x our 2x3090
AutoRound tier (native FP4 GEMM). VRAM 30.6/32 GB @131K — tight but flat.
- registry: de-blind the status_note (drop "never booted" → first 5090
validation with numbers); stays 🧪 (8-pack quality still owed — that run's
benchlocal sandboxes weren't built).
- baselines.yml: first NVFP4 row — a 1x5090-pcie submission (narr/code TPS,
clean@29K, soak PASS, engine v0.24.0, tier submitted); surfaces ⑂-labelled
in c3 like the multi-max submission. No primary row (we can't run it).
- compose header: AUTHORED BLIND block → FIRST COMMUNITY VALIDATION with the
gate results + the 98304 fallback note.
- BENCHMARKS.md: the first NVFP4 row in the 35B-A3B section.
Confirms the two design calls: MTP-off (speculative_config=None) and fp8 KV
at scale=1.0 (declared-not-shipped scales) both loaded correctly on the
modelopt path. learnings + memory updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Per maintainer call: "cluster" conventionally means multiple networked
machines (a non-goal here — LiteLLM fronts multi-host) and is best reserved
for a future enterprise/multi-node meaning. "Pod" is the accurate analogy
for what this is — one model on a GPU subset on ONE host (k8s/RunPod sense).
The capability is unchanged; only the name.
Scoped rename (cluster→pod, case-aware) across the pod feature ONLY:
- scripts/cluster.sh → scripts/pod.sh; test-cluster-cli.sh → test-pod-cli.sh;
docs/CLUSTERS.md → docs/PODS.md
- estate_cli.py verbs + wording; app.py (ClusterCreateScreen→PodCreateScreen,
action_new_cluster→new_pod, _populate_clusters→_populate_pods, #cluster-view
→#pod-view, the [N] help/empty-state text); services.py cluster_create_plan
→pod_create_plan; data.py kind cluster_create→pod_create; tests + doc
pointers (HARDWARE/MULTI_CARD/README/c3-README)
- UNTOUCHED (unrelated "cluster"): compat.py + test-profiles-compat.sh (the
VRAM-topology classifier), services.py:1863 / test_services.py (the scene-
table "cluster by group" verb), older docs, .venv
Also folds in the [N] discoverability fix (n was already bound to
serving_switch — moved to N; empty-estate now shows a "no pods — [N] new
pod" affordance + a help entry) and a heterogeneous-rig section in PODS.md:
one homogeneous pod per card family (2×3090 · GB10 · 6000 Pro) is the clean
pattern — mixing families in one TP pod makes NCCL wait on the slowest +
wastes VRAM; a worked 2-pod lifecycle walkthrough.
Verified: test-pod-cli + estate/gpu/profiles guards green; 25 c3 pod/binding
+ 239 fast tests green; pod.sh live create/list/D1-reject on 2×3090; zero
stray "cluster" in pod files (scene-verb preserved); no CLUSTERS.md links
left; PODS.md leak-clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Clusters (cluster.sh + the c3 view/wizard + the estate file + CDI pinning)
had grown into a subsystem scattered across HARDWARE.md and the c3 README.
Consolidate into a dedicated docs/CLUSTERS.md, sitting alongside
SINGLE/DUAL/MULTI_CARD as the multi-model-per-host workload axis:
- concept (cluster = model + GPU set + port = estate instance) + cluster-vs-TP
- a Defaults section answering "what happens before you make any cluster":
no default cluster / empty estate; a plain launch boots ONE model whose
GPU count = the compose's TP (prompts both/all on multi-GPU), NOT the
cluster machinery — clusters are opt-in
- CLI quickstart (create/list/status/up/down/rm) + c3 quickstart ([n] wizard,
cluster view)
- D1 fit-vs-set rules, placement verification, the estate file schema, the
command reference
HARDWARE.md keeps the GPU-pinning mechanics (UUID / CDI-NixOS — a hardware
topic) and points to CLUSTERS for management; c3 README, MULTI_CARD, and the
docs index now link CLUSTERS. All cross-links verified; public-safe.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The in-cockpit cluster-create surface — Phase C's remaining piece.
[n] in Operate · Orchestration opens ClusterCreateScreen: name (Input) ·
slug (Select of catalog slugs) · GPU set (Input, prefilled with the free
GPUs from the estate poll). The modal dismisses {name, slug, gpus}; the app
builds a cluster_create ActionPlan (requires_reconcile=False — a file write,
no GPU claim) and routes it through the standard confirm gate → cluster.sh
create, which runs the D1 fit-vs-set + validate_estate gates and refuses a
bad set (count!=TP, GPU/port collision). The modal never touches the rig
(codebase convention — same as PowerCapMenuScreen).
- app.py: ClusterCreateScreen (mirrors SettingsScreen) + action_new_cluster
/ _on_cluster_create / _cluster_free_gpus + the [n] binding, action
descriptor, and Operate/Orchestration action-context gate.
- services.py: cluster_create_plan builder (mirrors set_default: a
requires_reconcile=False file write).
- data.py: cluster_create ActionPlan kind.
- test: modal collects+dismisses the payload, a blank field stays open, and
the plan builder produces the right cmd with requires_reconcile=False.
Verified: 9 headless (C1+C2+siblings) + 51 binding/action headless + 239
fast c3 green. Plan cmd integration-checked (creates a fit-clean TP=2
cluster). Design: serve-cockpit-design.md §10 — C2 SHIPPED; #610 Phase C
complete.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The cockpit surface for multi-model clusters: the Operate/Orch pane now
groups estate instances (clusters) with their GPUs stacked and a placement
health badge fed by the Phase-A assertion.
- estate_cli.py D3 (#610 addendum 3): report-state's active_estate.instances
now carries a per-instance placement verdict {requested, actual,
placement} (only probed for RUNNING instances). ONE source feeds both
cluster.sh status and the cockpit — no second poll.
- app.py C1: _populate_clusters renders a #cluster-view block below the
serving line — per cluster a header (● name · slug · :port · badge) with
GPUs stacked beneath (└─ GPU0 GPU1) + a trailing free-GPU line. The badge
is the Phase-A verdict: ✓ placed / ⚠ PLACEMENT MISMATCH, so the view shows
where clusters ACTUALLY landed (the failure mode that opened #610). Hidden
when the estate declares no clusters — the single-model case is unaffected.
- test-cluster-view headless: grouping + GPU stacking + ✓/⚠ badges + free
line; empty-when-no-clusters. Serving-panel siblings unaffected.
LIVE-VERIFIED (2x3090): cluster.sh up chat → report-state carries
placement=ok → C1 renders ✓ placed. 8 headless + 239 fast c3 + estate
guards green.
Design: /opt/ai/docs/serve-cockpit-design.md §10 (C1 shipped; C2 wizard is
the remaining Phase-C piece — the CLI wizard covers the capability today).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The CLI seam the c3 cluster UX (Phase C) will consume, and a full headless/
SSH cluster capability on its own. A "cluster" = a named model on a chosen
GPU set + port (an estate instance).
- estate_cli.py gains create / list / status / rm verbs (D2: NEW verbs go
IN estate_cli.py, which owns the schema + validate_estate + boot/down —
ONE validation path shared with hand-written estate files and the wizard).
up/down reuse `boot`/`down --only <name>`.
- create runs the D1 fit-vs-set logic (#610 addendum 3): kv-calc --card is
single-card + registry TP, so count != compose TP is a HARD REJECT, a
heterogeneous set is estimated against its min-VRAM card (+ note), and the
whole set is re-validated (validate_estate: GPU collision, port collision,
per-instance fits) before append. GPU indices stay index-based in the
estate file; UUIDs resolve at boot (Phase A).
- status carries the {requested, actual, placement} verdict per cluster
(the shape the c3 badge reads); list/status take --json.
- scripts/cluster.sh: the ergonomic bash front (create/list/status/up/down/
rm), a thin wrapper over estate_cli.py.
- test-cluster-cli.sh (NEW): full lifecycle hardware-free via CLUB3090_FAKE_
GPUS — create/D1-reject/collision-reject/list/status/rm.
LIVE-VERIFIED (2x3090): create chat (fit-clean ~20.2 GiB) → up → estate boot
→ "✓ placement verified" → status "serving ✓ placement=ok", GPU0 idle /
GPU1 loaded → down → rm. D1 count!=TP + GPU-collision rejects confirmed.
Part of #610 (Phase A′). Next: Phase C (c3 cluster view + wizard).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Extends the #611 primitive to the estate (multi-cluster) path and adds a
post-boot placement check — the foundation Phase A′ (cluster.sh) and C
(c3 cluster UX) sit on.
- scripts/lib/gpu-select.sh (NEW, shared lib): factors the #611 inline
resolver out of launch.sh into gpu_select_indices_to_uuids /
gpu_select_export, plus gpu_select_container_uuids +
gpu_select_assert_placement. launch.sh sources it (no re-inlined drift)
and runs the placement assertion after verify-full.
- estate_cli.py: resolve_gpu_uuids() (python twin of the bash resolver) +
compose_env / compose_override_doc now UUID-pin CUDA_/NVIDIA_VISIBLE_
DEVICES (ESTATE_GPUS stays index-based — the compose device_ids read the
host view). assert_placement[_quiet]() runs after each instance is ready
(sequential → stderr, parallel → per-instance boot log), returning the
{requested, actual, placement: ok|mismatch|unknown} verdict — the shape
cluster.sh status + the c3 badge will read.
- Placement uses --query-compute-apps=gpu_uuid (NOT --query-gpu): under CDI
the container sees all cards but RUNS on the CUDA-masked set, so
compute-apps is the runtime-agnostic ground truth.
- test-gpu-select (NEW): asserts the bash + python resolvers agree on real
hardware and both fall back identically; test-compose-gpu-mask-passthrough
updated to follow the resolver into the lib.
LIVE-VERIFIED (2x3090, classic runtime): `launch.sh --gpus 1` → UUID-pinned,
verify-full 8/8, "✓ placement verified", host GPU0 = 1 MiB / GPU1 = 20.8 GB.
estate compose_env confirmed UUID-izing while ESTATE_GPUS stays index-based.
CDI leg still awaits mog. Relevant guard sweep (11) green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
--gpus selection was NVIDIA_VISIBLE_DEVICES-only, which CDI runtimes
(NixOS hardware.nvidia-container-toolkit, nvidia-ctk cdi) IGNORE — mog's
3x3090 rig got all cards exposed and vLLM took the wrong two. Index-based
CUDA_VISIBLE_DEVICES can't fix it either: the classic runtime renumbers
the exposed set in-container.
UUIDs solve both at once:
- launch.sh --gpus N,M resolves indices -> GPU UUIDs (nvidia-smi -i) and
exports BOTH NVIDIA_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES as UUIDs.
Classic runtime: exposes-by-UUID; CUDA mask agrees (renumbering-proof).
CDI: exposure comes from device_ids (nvidia.com/gpu=all) and the CUDA
UUID mask is what pins the cards. Index fallback if the query fails.
- 26 composes gain a bare `- CUDA_VISIBLE_DEVICES` passthrough (unset ->
absent -> zero change for existing users); stale commented hints removed.
- preflight: selector consumers are host-index-based -> new
_preflight_selector_normalize maps GPU-xxxx tokens back to indices at
the single choke point (the first live boot caught this: preflight
rejected the UUID selector).
- new guard test-compose-gpu-mask-passthrough.sh: every compose with the
NVIDIA env line must pass CUDA_VISIBLE_DEVICES through + launch.sh must
keep the UUID resolution.
- docs/HARDWARE.md: "Pinning specific GPUs (and CDI / NixOS runtimes)" —
the runtime table, the CDI deploy-block swap, the in-container
renumbering gotcha, manual recipes.
LIVE-VERIFIED on this rig (classic runtime, 2x3090):
launch.sh --variant vllm/minimal --gpus 1
-> host GPU0 = 1 MiB (empty), GPU1 = 20.7 GB (model), both env vars
UUID-set in-container. CDI leg validation goes to mog (NixOS reporter).
Follow-up (tracked on #610): estate ESTATE_GPUS UUID-ization for CDI
multi-instance rigs; post-boot visible-GPU-count sanity check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Verification pass on the blind-authored claims (vLLM source + safetensors
indexes, 2026-07-06):
- vLLM modelopt SUPPORTS fp8 KV with NVFP4/mixed-precision weights by
design: KV_CACHE_QUANT_ALGOS = ['FP8','NVFP4'] + ModelOptKVCacheMethod;
our --quantization modelopt + --kv-cache-dtype fp8 pairing is canonical.
- BOTH nvidia checkpoints DECLARE kv_cache_quant_algo=FP8 in
hf_quant_config.json, but NEITHER ships k_scale/v_scale tensors (indexes
enumerate only linear input_scale/weight_scale/weight_scale_2). The
shipped composes claimed '27B bakes KV scales' and 'MoE has none, unlike
the 27B' — both wrong; the two checkpoints are IDENTICAL on KV: declared
FP8, no scales, scale=1.0 at runtime (calculate_kv_scales disabled on
Qwen3-Next hybrid) — exactly the #594-quality-tied regime.
Text-only; no flag changes (the composes were already serving the right
config — the rationale strings were wrong).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The row still titled dual-max '(FP8 weights + int8-PTH KV)' with a 🧪
trailer even though the #594 flip note was already appended inside it —
exactly the mixed signal that led #602 to expect the old tier. Header
now names fp8/e4m3 (#594), status matches the registry (Production),
and a one-line fast-vs-max qualification (speed+headroom vs
weight-fidelity, 8-pack tie, no more depth penalty) + the 2x5090
cross-arch 109/150 corroboration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
First slice-3e quality ingest: @guybrush01's on-rig benchlocal --full
(2026-07-06, #571) lands 109/150 on vllm/qwen-27b-dual-max @ 2x5090 —
exact total parity with the 2x3090 primary's 109/150 (#594): Blackwell
native FP8 GEMM and Ampere Marlin W8A16 score the same 8-pack. n=2
on-rig runs (102 -> 109) both within the ±5-7 noise band; row carries
the better-evidenced latest run with per-pack detail + source link in
quality_env.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
nvidia/Qwen3.6-35B-A3B-NVFP4 (modelopt MIXED_PRECISION MoE: NVFP4 gs16
expert FFNs + FP8-static attention + unquantized mtp.* head, 23.4 GB, 3B
active) — sibling of the 27B nvfp4 pair (#603), same authored-blind /
required_sm=9.0 / community-validated pattern. The single slug is the
GB10/DGX-Spark marquee: a 3B-active MoE is the right shape for
big-capacity, lower-bandwidth unified-memory parts — kv-calc says full
262K at 25% of a GB10's budget (MoE KV is tiny, ~1.3 GB @131K).
Two deliberate deltas vs the 27B nvfp4 composes:
- NO MTP (drafter=None): our measured finding on this MoE — the built-in
head shares the MoE forward and is net-negative (-51%); headers forbid
blind re-adding.
- fp8/e4m3 KV at scale=1.0: this checkpoint has NO baked KV scales
(kv_cache_scheme null, unlike the 27B) — same regime the production 27B
fp8 tier quality-tied at (#594); flagged as a known-unknown.
- vllm/qwen-35b-a3b-single-nvfp4 (:8078, 🧪) — 131K default for 5090
32 GB (kv-calc 27.4/29.4 @93%); GB10 raises to 262K via MAX_MODEL_LEN.
- vllm/qwen-35b-a3b-dual-nvfp4 (:8079, 🧪) — TP=2 @262K, 15.7 GB/card
(53%) on 2x 5090.
Wiring mirrors #603: weights entry + weights.py aliases + kv-calc
(weights_nvfp4_gb on the MoE spec, nvfp4 branch, cfg map, aliases) +
registry entries (kvcalc-priced, no DEFAULTS rows) + counts 61/62. Full
scripts/tests suite + kv-calc --calibration green; sm_86 → incompatible-hw
(auto-hidden in c3 per #604), rtx-5090 → fits-clean 27.4/15.7 GB.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The metadata version is frozen at install time and benchlocal fixes are
pushed without bumping it, so "0.9.4" cannot identify the running code
(live proof on the dev rig: metadata 0.9.4, checkout actually at
v0.9.7-6-gc541550). pip records the truth in direct_url.json:
- report.sh: the Quality-tooling line now appends `source:` —
`git@<sha9>` for pip-from-git installs (vcs_info.commit_id), or
`git describe --tags --always --dirty` for editable checkouts (the
checkout PATH is withheld from the public report).
- quality-test.sh staleness reference: editable installs update via
`git pull` WITHOUT rewriting the console script, so the mtime
under-reports "CLI last updated" — now max(script mtime, editable
checkout last-commit time). Caught a real case on the dev rig: the
cli/hermes sandboxes were built ~1 h before the same day's last
benchlocal commit. Stale-warning timestamps now carry %H:%M so
same-day boundaries read sensibly.
test-quality-baseline / test-quality-thinking / test-report-calib green;
report section verified live + leak-clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Community rigs hit two silent quality-test failure classes: (1) runs
"skipping" the sandboxed packs because the Docker images were never built
(the #492 warning exists but --sandboxed-only still warn-and-ran a
guaranteed-useless run), and (2) rigs that pulled a benchlocal-cli update
kept scoring on OLD sandbox images until told to rebuild manually — nothing
detected the drift. And report.sh collected none of this, so triage had to
ask.
quality-test.sh:
- --sandboxed-only + missing images/Docker → HARD FAIL up front with the
build instructions (a run with zero runnable packs is never intended).
--full keeps warn-and-continue (deterministic packs are still useful).
- NEW staleness heuristic: each sandbox image's Created is compared against
the benchlocal-cli console-script mtime (rewritten on every (re)install —
portable across pip-from-git AND editable-checkout installs). Image older
than CLI → warning naming the image + both dates + the rebuild command.
Explicitly labelled heuristic (an unrelated reinstall trips it) → WARN,
never abort.
report.sh:
- New "Quality tooling (benchlocal-cli + sandboxes)" section: CLI path /
version (via the console-script's own interpreter) / install date, the 4
sandbox images with build dates + per-image OLDER-than-CLI flags + rebuild
hint, and the newest results/quality/*.json. All best-effort; piped
through redact (verified leak-clean live).
Exact-version staleness (image labels + a `sandbox-status` command) needs
upstream benchlocal-cli changes — tracked in the todo.
Verified live: hard-fail branch (docker stub), warn-continue branch,
staleness branch (fresh CLI vs real images), report section rendering.
test-quality-baseline / test-quality-thinking / test-report-calib green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
On rigs whose card can't run a slug's kernels (registry required_sm above
the local SM — e.g. the NVFP4 slugs on this sm_86 rig), the catalog showed
the slugs normally and the Download card pitched a 20 GB fetch that could
never boot. Now:
- kv-calc fit verdict gains the arch floor: fit_verdict()/fit_all_verdicts()
return {"verdict": "incompatible-hw", required_sm, card_sm, error} when
the card's SM (derived from the hardware profiles — same source as compat
C3) is below the slug's required_sm. Bare-number --card carries no arch
info → gate skipped (permissive). Applies to kvcalc-SKIP slugs too.
- Catalog: incompatible-hw rows share the DEPRECATED [h] bucket — hidden by
default, revealed by [h]; status line counts them separately
("+N incompatible-hw hidden — h"). The verdict lands with async fit
enrichment, so rows fold away on the enrichment refresh.
- Download confirm card leads with "⊘ no compatible hardware detected —
requires sm ≥ X (Hopper/Blackwell); this rig's card is sm_Y. It will NOT
boot on this machine." BEFORE the size/disk pitch — download still
proceeds (staging for another rig is legitimate). Serve card gets the
same warning line under fit.
- FitVerdict: incompatible-hw verdict + ⊘ glyph + required_sm/card_sm.
Also updates test-registry-json's VARIANT_KEYS for the #600 emit fields
(kv_format / weights_format / weights_quant_label) — the contract gate
only surfaced post-merge.
Tests: kv-calc-fit section (f) — 3090→incompatible-hw w/ sm fields,
5090→fits-clean, bare-number permissive, fit-all parity; 2 new headless
(hide+reveal+count, download-card warning). Full scripts/tests suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
