Add a 'Where weights land' section: MODEL_DIR is the single root; curated
catalog models live top-level (compose-referenced), BYO pulls derive to
.cache/huggingface/club3090/pulls/<repo-slug>/ — the SAME path pull.sh and the
c3 Bring [D] button compute, so a pull by either is discovered by both. Makes
the two intentional tiers explicit for users bringing their own weights.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Live dogfood 2026-07-09 (migtissera/Tess-4-27B, 54G bf16 on a 48G rig): the §2b
topology floor correctly hid every slug (single ≤21.6G, dual ≤43.2G, no
multi-card slug), but the funnel showed only the ✎ custom-slug sentinel with no
explanation — and the custom Input never appeared. Two cascading bugs:
- Bug A: when the size floor empties the recommended list, surface an honest
verdict — "won't fit" (compat slugs exist but all exceed VRAM; bring a smaller
quant) vs "no catalog recipe matches this artifact" (no compatible engine at
all) — instead of a silent empty dropdown.
- Bug B: with only the sentinel option, _set_select_options pre-selects it under
prevent(Select.Changed), so the Changed-gated custom-Input reveal never fires
and re-picking the sole sentinel is a no-op. reveal_slug_stage now reveals the
Input eagerly iff the sentinel is the sole option (kept hidden when real slugs
exist).
Neither is a regression — the floor (#582) + reveal are byte-identical to
pre-Grok; a 55G repo is the first to expose the all-hidden edge. Tests:
test_uiux_funnel_oversized.py (custom-hatch reachable + won't-fit verdict + no
false verdict at normal size). 254 c3 tests green.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three review nits from the phase 1/3 merges, all copy/label only — no logic
change (245 c3 tests green):
- action_serve_untested: the user-facing dead-end toast said "the bring-your-own
weight-swap is a pending follow-up" — false since #644 wired apply-swap. Now
states the real reason (fit-check resolved no sibling recipe or catalog profile
to serve) and points back to ① Bring.
- Its docstring still claimed "this does NOT serve the brought model's weights ...
DEFERRED follow-up," contradicting its own now-wired Route-C body. Rewritten to
the two actual modes (brought-weights serve vs untested reproduction).
- Disk-fit preflight cards labeled sizes "GiB" while the math is decimal GB
(size_gb * 1e9 / free / 1e9). Relabeled the disk-fit size/free/need to "GB"
(both the catalog and bring preflight cards). VRAM "GiB" labels left as-is —
those are genuinely binary.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
resolve_hf_home was `--hf-home > $HF_HOME > $XDG > ~/.cache`, ignoring MODEL_DIR.
So a bare `pull.sh <repo>` (or a nohup apply-swap) run with only .env's MODEL_DIR
set and no explicit HF_HOME silently fell to ~/.cache on the ROOT disk — the
footgun that misplaced a brought model's 35 GB of weights off the models volume
(and would bite users the same way). Insert a MODEL_DIR step (env, else parsed
from the repo .env, encoding=utf-8) between $HF_HOME and $XDG, matching what
switch.sh/launch.sh/c3 already resolve. c3 is unaffected — it sets HF_HOME
explicitly, which still wins. New guard test-hf-home-resolve (PASS 1-5);
test-pull / test-pullgate-download / test-download-lock still green.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
select_weight_files built the sharded set as `"-of-" in n or startswith
"model-"`, so a grafted MTP/nextn head (e.g. `mtp_grafted.safetensors`) — a
real weight the model needs with MTP enabled — was dropped, even though
detect_mtp_head/_has_mtp_weight_file already see it. Live dogfood: Tess-4-27B-FP8
downloaded its 7 model-* shards but silently omitted `mtp_grafted.safetensors`,
which would break MTP serving (club-3090 #617). Union any non-shard mtp/nextn
*.safetensors head into the sharded download set. Guard: test-pullgate-download
gains an MTP-head fixture.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* c3+pull: detect in-progress downloads + per-repo lock (#617)
Repeated ① Bring [D] presses spawned N concurrent `hf download` runs racing
into the same `.incomplete` staging dir (5-deep live), each rmtree-ing and
re-fetching the others' partial. Two disk-truth gaps behind that:
1. No dedup. `download_model` now acquires an atomic per-repo lock
(`mkdir <pull_dir>/.download.lock` + a `pid` file = holder PID + UTC start)
BEFORE the staging rmtree. A 2nd concurrent call for the same slug is
REFUSED with `failure="in-progress"` (not a race); swap_apply surfaces it
as `in_progress`, and `pull.sh --apply-swap` exits **rc=3** ("already
downloading", distinct from rc=1 failure). A STALE lock (dead holder — a
crashed/SIGKILL'd download that skipped `finally`) is reclaimed on the next
call, so a leaked lock self-heals — more robust than a signal trap (a
SIGKILL skips traps too). Released in `finally` on every return path.
2. No in-progress detection. Presence probes only saw the FINAL dir (empty
until hf moves files out of `.incomplete`), so a mid-flight download read as
absent → the false "download did not complete". `services.bring_download_in_
progress` now reads the lock (live PID) + `.incomplete` bytes for a
best-effort pct; c3's fit-check render + [D] no-op guard + [k] cancel are
disk-aware, so a running download is REFLECTED (and not duplicated) even
across a c3 restart or a bare `pull.sh` started outside the session — the
in-memory tracker (#643) couldn't see those.
download_model is the shared chokepoint (bring apply-swap AND catalog fetch),
so both paths get the lock; the body moved verbatim to `_download_model_impl`
behind the locking wrapper — no behaviour change beyond the added guard.
Tests: scripts/tests/test-download-lock.sh (live-refuse + stale-reclaim +
release); c3 test_services (in-progress live/stale/absent + pct) +
test_app_headless (disk-detected → downloading render, [D] no-op, [k] kill).
245 services/registry + 131 app blast-radius tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* downloader: refuse a fresh pidless lock (close acquire-window race)
The lock acquires with mkdir then writes its pid file a beat later. A 2nd
caller landing in that window saw the lock dir but no pid → read_active
returned None → the wrapper treated it as stale and rmtree'd + reclaimed it,
letting BOTH callers proceed — the exact duplicate-download race the lock
exists to prevent. Now: a pid-present-but-dead lock still reclaims (unchanged),
but a PIDLESS lock only reclaims once it's older than a 10s grace — a fresh
pidless lock is a holder mid-acquire and is refused, not stolen. Guard adds
PASS 4 (fresh pidless → refuse) + PASS 5 (old pidless → reclaim).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ① Bring [D] download was a stripped-down sibling of the catalog
Download: a single await + trusting the captured `[apply-swap] compose:`
marker, with no in-flight tracker and no cancel. Two dogfood bugs (#617):
1. "apply-swap did not complete" fired while the download was still
running — the verdict raced the marker instead of re-stating disk.
2. A re-run fit-check was blind to an in-flight download, re-offering [D]
and inviting a second 20+ GB fetch of the same repo.
Mirror the catalog run_download machinery for the bring path:
- repo-keyed `_active_bring_download` tracker, registered synchronously
in action_bring_download (no-op guard: a second [D] for a repo already
downloading just notifies).
- run_bring_download_worker polls to TRUE completion, then derives the
verdict from an AUTHORITATIVE `bring_weights_present` re-stat. Route-C
weights that landed but whose marker was missed now read as ✓ → ② Serve
(apply-swap --emit-only, no re-download), NOT a false failure.
- run_byo_check + _byo_result_text are downloading-aware: an in-flight
repo renders "⏳ downloading… [k] cancels" and suppresses the [D]
re-offer.
- [k] bring_cancel_download kills the pull.sh/hf process group via the
runner's SIGINT→TERM→KILL (the bring path had no cancel at all); shares
"k" with serving_stop, gated disjointly in check_action.
Scope: bring path only — catalog Download, the runner, and scripts/
untouched. c3 pytest: 808 passed (2 new: the downloading card render +
the tracker/guard/cancel round-trip).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* baselines: backfill dual-max quality_8pk_think_on (110/150)
The vllm/qwen-27b-dual-max row logged only the thinking-off 8-pack
(107/150); the thinking-enabled run existed since the 2026-06-07 #515
quant-A/B (fp8 dual-max full 8-pack = 110/150, pack-defaults regime,
autoround sibling 109) but was never promoted. Backfilled with
provenance so it isn't re-derived. Guards: test-baselines,
test-quality-thinking, test-quality-baseline all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* preflight: warn on single-card GPU_MEMORY_UTILIZATION override above default
Two 5090 testers (#617 paul, and the same trap earlier) OOMed the tool-prefill
step of verify-stress by setting GPU_MEMORY_UTILIZATION=0.92 on the nvfp4
single-card slug whose validated default is 0.85. On one GPU a higher util
steals the free VRAM a large tool-response prefill needs for its activation
peak, so vLLM OOMs mid-prefill even though boot succeeds — a confusing failure
(boot ok, then HTTP 500 at step 2/8) that vLLM's own error hints at ("lower
--gpu-memory-utilization").
Add preflight_single_card_util(): an advisory WARN that fires only when the
user overrode GPU_MEMORY_UTILIZATION *above* the compose's shipped default on a
TP<=1 config. Runs even under --force (the nvfp4 slug launches with --force),
and no-ops for dual/multi-card, non-vLLM engines (no util default in the
compose), a plain default run, or util at/below the default. Wired into
switch.sh alongside the other force-independent hints.
Verified: behavior matrix (single>default WARN; single=default/below/none and
dual all silent) + test-preflight-compose-deps / test-model-switch /
test-switch-registry-parity / test-launch-compat all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The spec-dec dropdown now lists the resolved ENGINE's supported drafters + off
(vLLM: mtp / mtp_assistant; beellama: dflash / mtp / mtp_gguf; …) instead of a
bare on/off — so the drafter TYPE is selectable, and it's engine-correct (dflash
only appears when the swap engine is beellama).
- swap_apply: the SPEC entrypoint REBUILDS --speculative-config via printf from
${DRAFTER_METHOD}/${DRAFTER_N} (defaults = the sibling's), so switching the
drafter is an env override — no re-emit. Validated: default→mtp n=3,
DRAFTER_METHOD=mtp_assistant→that config, SPEC=off→dropped, compose config valid.
- services: engine_drafters() reads the profile's supported_drafters (shares the
stdlib list-parse with engine_kv_formats via _engine_yaml_list); serve_override_
defaults adds DRAFTER_OPTIONS + SPEC_METHOD + SPEC_N.
- app: the spec Select value is the drafter method (or "off"); collect maps
method→SPEC=on+DRAFTER_METHOD, off→SPEC=off.
Drafters with extra requirements (mtp_assistant needs an assistant model; dflash
needs the beellama engine) are honestly listed per-engine — picking one the swap
can't satisfy fails at boot (loose-validation stance), or use a matching ① Bring
slug. 173 byo/serve/override tests pass.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pinned cu13 ik-llama digest has a CUDA 13.2 runtime. On a driver whose
supported CUDA < 13.2 (e.g. 580.159 = CUDA 13.0, which satisfies the README's
"580.x+" but NOT this pin) the forward-compat path fails on GeForce (CUDA error
804) -> silent CPU fallback -> segfault crash-loop, with launch.sh just timing
out after 600 s and no actionable hint.
- preflight.sh: preflight_ik_llama_image() detects the driver's supported CUDA
(_driver_cuda_version) and, for ik-llama variants on a driver < 13.2 with
IK_LLAMA_IMAGE unset, auto-selects the cu12 sibling build (same build 4574,
CUDA 12.6, backward-compatible, DelspoN-validated ~68 TPS on 580.159) + prints
a ⚠ hint. >=13.2 keeps cu13; a user IK_LLAMA_IMAGE pin always wins (echoed).
_cuda_ge() does the major.minor compare. Verified: 13.3/13.2 GE, 13.0/12.6 LT.
- switch.sh: call it right after export_variant_engine_pin, before compose up
(the single up path — launch.sh delegates here). preflight.sh is already
sourced in that block.
- docs: qualify the driver requirement in the ik-llama compose header + HARDWARE.md
("580.x+" is insufficient; needs CUDA >= 13.2; auto-fallback documented).
Note: this rig is CUDA 13.3, so the <13.2 boot path is verified by logic +
DelspoN's cu12-server-4574 validation, not a local boot. Rolling cu12 tags are
NOT safe (cu12-server-100 rejects the compose's --spec-type flag, #633) — pinned
to the build-number tag matching the cu13 digest's build.
Closes#633.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(c3): ② Serve spec-dec — engine-driven drafter SELECTOR (not just on/off)
The spec-dec dropdown now lists the resolved ENGINE's supported drafters + off
(vLLM: mtp / mtp_assistant; beellama: dflash / mtp / mtp_gguf; …) instead of a
bare on/off — so the drafter TYPE is selectable, and it's engine-correct (dflash
only appears when the swap engine is beellama).
- swap_apply: the SPEC entrypoint REBUILDS --speculative-config via printf from
${DRAFTER_METHOD}/${DRAFTER_N} (defaults = the sibling's), so switching the
drafter is an env override — no re-emit. Validated: default→mtp n=3,
DRAFTER_METHOD=mtp_assistant→that config, SPEC=off→dropped, compose config valid.
- services: engine_drafters() reads the profile's supported_drafters (shares the
stdlib list-parse with engine_kv_formats via _engine_yaml_list); serve_override_
defaults adds DRAFTER_OPTIONS + SPEC_METHOD + SPEC_N.
- app: the spec Select value is the drafter method (or "off"); collect maps
method→SPEC=on+DRAFTER_METHOD, off→SPEC=off.
Drafters with extra requirements (mtp_assistant needs an assistant model; dflash
needs the beellama engine) are honestly listed per-engine — picking one the swap
can't satisfy fails at boot (loose-validation stance), or use a matching ① Bring
slug. 173 byo/serve/override tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* fix(launcher): GGUF verify-glob (#634) + launch.sh .env engine-image passthrough (#632)
#634 — setup.sh re-hardcoded VERIFY_GLOB=*.safetensors at the primary
download/verify (line 625), clobbering the recipe's glob that
load_weight_recipe already set from PRIMARY_WEIGHT_KEY. Every GGUF primary
fetch (WEIGHTS=gguf/iq4ks) then failed verify ("No *.safetensors found")
despite a byte-correct download. Now defaults to ${VERIFY_GLOB} (the recipe's
*.gguf); VERIFY_GLOB_OVERRIDE still wins.
#632 — launch.sh's .env load loop gated to CLUB3090_DEFAULT_* only, so
IK_LLAMA_IMAGE / LLAMACPP_IMAGE pins in .env were silently dropped (the two
GGUF engines are not profile-injected, so .env is their only override path —
the natural persistence for the #633 cu12 workaround never reached compose).
Broadened the allowlist to the engine-image keys; switch.sh already loaded
them (no filter). Both launchers now echo an ik-llama/llama.cpp image pin so a
wrong-image boot is never silent. (DelspoN's report cited the engine-pin
whitelist, which is the vllm/beellama-only profile-injection path — the real
drop was the .env loop filter.)
Verified: the .env loop now loads IK_LLAMA_IMAGE/LLAMACPP_IMAGE and still drops
un-allowlisted keys; test-model-switch / test-launch-compat / *-registry-parity
/ test-setup-picker / test-model-weights-registry all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Four dogfood refinements to the #636 override editor:
1. Custom values in the dropdowns. ctx / KV / util each gain a trailing
"✎ custom…" sentinel that reveals a companion free-text Input
(on_select_changed, same idiom as the ① Bring profile hatch) — so any value
the presets omit is reachable (e.g. ctx=81920, KV=turboquant_4bit_nc).
collect_overrides reads the companion when the sentinel is selected.
2. spec-dec shows the real drafter. The label is now e.g. "MTP n=3 (on)" /
"off — no spec-dec" instead of a bare "on" (value still on/off — the ${SPEC}
gate). Parsed from the sibling's --speculative-config.
3. KV types are ENGINE-driven. The KV dropdown reads the resolved engine's
supported_kv_formats (vLLM: fp8 family + int8-PTH; llama.cpp: q4_0/q5_0/…) via
the new engine_kv_formats() — not a generic list. turboquant_4bit_nc (native
in v0.24.0 but not yet in the profile's list) stays reachable via ✎ custom.
4. Slug preview. A read-only line above the fields shows what you're overriding:
engine · ctx · KV · spec · util (from serve_override_defaults, now also
parsing ENGINE + SPEC_DRAFTER; sys.path pinned so the import is cwd-safe).
171 byo/serve/override tests pass.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 2 of the ② Serve rework (Phase 1 = #635 dead-end fix): a Route-C brought
model can be re-tuned before serve, without hand-editing a compose. Fields
(pre-filled from the resolved slug's defaults, dropdowns to avoid typos):
• served name → SERVED_NAME (Input)
• ctx → MAX_MODEL_LEN (Select — presets + the slug's own default)
• KV cache → KV_CACHE_DTYPE (Select — fp8_e5m2/e4m3/turboquant_4bit_nc/…)
• spec-decode → SPEC on/off (Select)
• VRAM util → GPU_MEMORY_UTILIZATION (Select)
Mechanism (all five ride env — no per-serve compose rewrite):
- data.py: ActionPlan gains `env`; dispatch merges {**os.environ, **plan.env}.
- services.py: serve_generated(compose_path, overrides) → plan.env (+ pins
MODEL_DIR); serve_override_defaults() parses the sibling compose's
${VAR:-default} for pre-fill (stdlib regex, no PyYAML).
- swap_apply.py: emit parameterizes --served-model-name ${SERVED_NAME:-…} and
gates --speculative-config behind ${SPEC:-on} via the SAME entrypoint the
shipped nvfp4 compose uses — so SPEC=off drops the MTP drafter at up-time.
- app.py: LaneServePane fields, revealed + pre-filled only for a Route-C armed
model (hidden otherwise); collect_overrides() → serve_generated. Loose
validation (dropdowns constrain; the reconcile gate + boot are the real check).
Validated: emitted compose `docker compose config` resolves SERVED_NAME=X and
SPEC=off (entrypoint gate) cleanly; 169 byo/serve/override tests pass.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two live-dogfood issues on the Bring & Validate ② Serve stage:
1. Dead end / stale text. The ② Serve armed card said it serves "the resolved
catalog compose … NOT your brought model's weights" — the OPPOSITE of what
#628/#630 actually do for a Route-C fine-tune (they serve YOUR weights via
the sibling's recipe). And when unarmed it fell back to a dim placeholder
that reads as empty. bring-funnel-design §2b item 7 explicitly forbids this
("no dead end"). set_armed is now route-aware: Route-C → "serves <brought>
(your brought weights) via <sibling>'s recipe → [⏎] to serve"; non-swap
routes keep the honest "catalog reproduction, untested" wording.
2. ⏎ overload. #631 made ⏎ on ① Bring advance-when-present, but ⏎ there also
runs the fit-check, so it re-fit instead of proceeding. Reverted: ⏎ ALWAYS
fit-checks; a DEDICATED key [s] advances ① Bring → ② Serve (guarded on
servable + weights-on-disk; absent → notify to press [D]). [s] joins the
existing context-routed s_key (Containers restart · ④ Measure submit).
Card + weights-line now say "press [s] to continue to ② Serve".
Tests: test_bring_s_key_advances_to_serve_when_present, _no_advance_when_
weights_absent, test_bring_enter_always_fitchecks_never_advances, test_serve_
armed_route_c_serves_your_weights; updated the s_key-gating test to include
tab-bring. 195 touched-surface tests pass.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #630 card told the user to "press [⏎] ② Serve" when the brought weights
were on disk, but ⏎ on the ① Bring tab re-ran the fit-check (its _validate_
primary → _trigger_lane_bring), so pressing it just re-fit instead of
proceeding — the reported confusion.
Make ⏎ DWIM on ① Bring (_bring_primary):
- an already fit-checked, servable target whose weights are ON DISK (and the
repo input is unchanged) → ADVANCE to the pre-armed ② Serve tab (⏎ there
serves via the #630 emit-and-serve path);
- otherwise (fresh/changed target, or weights absent) → run the fit-check as
before. Absent stays on fit-check because the card there points at [D]
(download is the real next step). The [Fit] button still forces a re-check.
Also fixes the stale "([2/]] next stage)" hint on the weights-line → "(press
[⏎] to continue)", and the present-card next-step wording.
Tests: test_bring_enter_advances_to_serve_when_present (present → active tab
flips to tab-serve) + test_bring_enter_refits_when_weights_absent (absent →
stays on tab-bring). 156 touched-surface tests pass.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A Route-C fit-check whose weights were already downloaded showed a
self-contradiction: the verdict card hardcoded "→ Press [D] to download +
serve" while the weights-line below it correctly said "✓ weights on disk →
② Serve". [D] is not only download — for a Route-C brought model it emits the
serve-locally swap compose that ② Serve then serves — so it can't simply be
hidden.
Fix (presence-aware, "② Serve owns the emission"):
- _byo_result_text takes weights_present (probed once in the fit-check
handler, feeding BOTH the card and the weights-line so they can't disagree).
On disk → the card points at ② Serve, no [D]; absent → the [D] download
prompt as before.
- pull.sh gains --emit-only (with --apply-swap): passes do_download=False so
apply_swap emits the serve compose WITHOUT downloading — the present-weights
path.
- action_serve_untested: Route-C + weights present + no swap compose yet →
run_bring_emit_and_serve emits via pull.sh --apply-swap --emit-only, then
serves it. So a present-weights brought model serves straight from ② Serve
with no [D] step.
Tests: fixed test_bring_result_points_forward_to_serve (stale since #628 — it
asserted "→ ② Serve" on an absent-weights card that now honestly shows [D]);
added test_bring_result_present_points_to_serve (present → ② Serve, no [D]);
extended the run_bring_download test for --emit-only. 139 touched-surface
tests pass; pull.sh --emit-only validated end-to-end.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(byo): apply-swap — download + serve a Route-C fine-tune
A curated-arch fine-tune (e.g. josefprusa/ThinkingCap-Qwen3.6-27B AutoRound
→ qwen3.6-27b) hard-stops at the pull gate's stratum-5 no-fit-model (correct
— nothing to price), so the c3 Bring [D] download failed with "download did
not complete". Route-C was pure guidance ("clone the compose, point --model
at your weights") with no code behind it — the deferred "swap_path apply".
Build it as a DISTINCT action that never touches the locked 6-stratum gate:
- scripts/lib/profiles/swap_apply.py (new): resolve the Route-C swap
(arch→curated sibling + has_mtp_head from deriver), download the brought
weights SHA-verified via downloader.download_model (a minimal EInput), and
emit a serve-locally compose that CLONES the --profile-like sibling's REAL
compose (keeping its curated chat-template / reasoning+tool parsers / MTP
wiring — NOT the derived-vllm template, which drops all that) with --model
re-pointed at a `<weights>:/brought-model:ro` mount, --served-model-name set
to the brought basename, and --speculative-config kept iff the checkpoint
carries an MTP head (else dropped). Written alongside the sibling compose as
`_brought-<san>.yml` (gitignored) so its relative ../ mounts resolve.
- scripts/pull.sh: intercept `--apply-swap` in the wrapper (strictly additive,
like --json) → run the apply-swap action; prints `[apply-swap] compose: …`.
The gate path is byte-unchanged without the flag.
- c3: services.run_bring_download gains `apply_swap=` (appends --apply-swap +
captures the emitted compose via last_swap_compose()); the [D] worker passes
it on route==C and stashes the compose; ② Serve serves that swap compose
directly (serve_generated) instead of a reproduction of the sibling's own
catalog compose.
- pull.py: fix the stale NOTE — "the bf16 base won't fit and lacks the MTP
head" → the base HAS the head; now just the size reason.
Tests: test-pull-swap.sh section 4 (emit: --model repointed, MTP kept/dropped
by head presence, curated flags + JSON args survive the YAML round-trip,
distinct container_name); test_services apply_swap flag + compose capture. The
locked-gate hard-stop tests stay green; --json fit-check byte-identical.
Validated live: pull.sh --json still no-fit-model+route-C, and apply_swap emits
a correct ThinkingCap swap compose (MTP kept).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* c3: reframe Route-C fit-check card as ✓ Servable + a clear [D] action
The fit-check card for a curated-arch fine-tune (Route-C swap) was
self-contradicting: a RED "not eligible" + a bare "no-fit-model" token at
the top, then a GREEN "② Serve is armed with <sibling>" at the bottom —
which read as a dead-end AND named the wrong model (the sibling, not the
brought fine-tune). The engine's "no-fit-model" is the *means* (generic
fit-math can't price a curated-hybrid arch), not the user's answer; the
outcome is servable via the sibling's recipe.
Reframe the Route-C case in _byo_result_text:
- green "✓ Servable — a fine-tune of <sibling>" instead of red "not eligible",
- one plain "How it serves" line (reuses the sibling's chat-template / tools /
spec-dec with your weights) instead of the --quantization/impl bullets,
- MTP kept/dropped line from has_mtp_head,
- the next-step is an explicit "→ Press [D] to download + serve <BROUGHT model>"
(the fine-tune, not the sibling) — which #628's --apply-swap now fulfils,
- raw verdict + backend note dimmed for debugging, not the headline.
Non-swap cases (eligible / Route A / B / plain no-fit) are byte-unchanged.
Regression test added; c3 fast suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The original OOM at 98K was caused by MTP, not the context length. Two
5090s (#613 @guybrush01, #617 @paulp83) swept the space: MTP-on at 98K
leaves no room for the draft head + cudagraphs + GDN prefill scratch and
OOMs — but at 65K + util 0.85 it PASSES, with MTP's ~2x intact:
verify-stress all-pass, decode 129/155 TPS (vs 71/71 with SPEC=off), MTP
accept ~3.2, ~1.4 GB VRAM free.
So the default keeps MTP and trades context, rather than dropping MTP:
- compose single/nvfp4/mtp.yml: MAX_MODEL_LEN 98304 -> 65536,
GPU_MEMORY_UTILIZATION 0.92 -> 0.85, header updated to the swept config.
- registry: max_ctx 98304 -> 65536, mem_util 0.92 -> 0.85, status_note.
- BENCHMARKS: guybrush's SPEC=on+65K row (129/155 decode, accept 3.2).
- baselines.yml: 27B single-nvfp4 submission (stale=false, v0.24.0).
SPEC=off stays exposed as the max-ctx fallback (81K/98K @ 71 TPS) and the
tight-system-RAM path (MTP's draft load OOM-kills a 28 GB host, #617).
80 GB+ cards raise MAX_MODEL_LEN toward 262K with MTP on. Stays 🧪 (8-pack
quality still owed). Scoped guards green (baselines / status-drift /
switch+launch parity / profiles-compat).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The BYO fit-check's Route-C weight-swap (pull.sh _swap_path) hardcoded
`drop_spec_config: True` on the premise "a generic repo carries no MTP
head." That's false for a fine-tune that PRESERVES the head — so any such
model (e.g. ThinkingCap-Qwen3.6-27B, whose AutoRound INT4 ships the full
15-tensor BF16 mtp head in model_mtp_bf16.safetensors) was silently served
MTP-off, losing spec-dec despite the head being present.
Fix: detect it. `deriver.detect_mtp_head(config, api)` returns True when the
config DECLARES the MTP layers (`mtp_num_hidden_layers` / nested
`num_nextn_predict_layers`) AND a dedicated mtp weights file is present —
ground-truth for the separate-file layout fine-tune re-quants use, using
signals the deriver already has in hand (no extra fetch). Exposed as an
additive `has_mtp_head` on der.profile; _swap_path sets
`drop_spec_config = not has_mtp_head`. An embedded-head repo (head baked
into the shards with no named file) still falls back to drop — conservative,
and the named-file layout is the norm.
Validated on real HF metadata: josefprusa/ThinkingCap AutoRound →
has_mtp_head=True (keep --speculative-config); a plain AWQ re-quant →
False (drop). test-pull-swap replaces its tautology check with a real
detect_mtp_head unit test (declares+file, plain, nested text_config,
declares-but-no-file). Pull-gate suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The ⚠️-caveat required a clean v0.24.0 4-card fp8/e4m3-KV report to
upgrade to ✅. Two landed:
- #584 (@ryanmpelletier, 4× 3090 x16, Proxmox): attention backend
FLASHINFER (the #595 int8-PTH→fp8/e4m3 flip live at TP=4), verify-full
9/9, verify-stress clean to 240,634 tok, soak-continuous PASS, bench
n=5, and the full 8-pack at 111/150 — ties the 2-card dual-max proxy
(109/150), so max-tier quality is TP-invariant (measured, not assumed).
- #625 (@MoppelMat, 4× 3090 bare metal, mixed x4/x8, 300 W): a second
independent 4-card rig — decode 79/102, prefill-90K clean, soak PASS.
Decode beats ryan's x16 rig (bare metal); prefill trails ~30% (narrow
lanes are the bandwidth-bound bottleneck; decode is lane-insensitive
at TP=4) — a clean decode(compute)/prefill(interconnect) split.
Changes:
- compose header: Status ⚠️ → ✅, drop the Caveats block, Quality now
cites the 4-card 111/150, prose updated to the promotion evidence.
- registry: status caveats → production + refreshed status_note.
- baselines.yml: replace @Whamp's doubly-stale #446 int8-PTH row (v0.22.0)
with ryan's fresh v0.24.0 fp8-KV submission (TPS + 8-pack + ctx + soak);
MoppelMat noted as the corroborating 2nd rig. Emit computes stale=false.
- BENCHMARKS.md: append both v0.24.0 4-card rows (append-only history).
- MULTI_CARD.md: drop the "no 4-card bench yet / needs TP=4 confirmation"
prose now that both reports have landed.
Guards green: test-baselines, test-compose-status-drift,
test-{switch,launch}-registry-parity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Surface which speculative-decoding drafter each slug enables by default,
so the Catalog answers "does this compose do MTP / DFlash / none?" at a
glance — a serving-config facet alongside weights and kv.
Derived entirely from the registry `drafter` id already emitted by
registry-emit.sh and threaded into the cockpit (row.drafter) — no emit /
registry / guard changes. `_spec_token()` maps the id to a compact
method·mechanism token: DFlash is always an external drafter (no suffix),
so the built-in-vs-external split lives only within MTP, where the suffix
names the actual mechanism — MTP (built-in head) · MTP·gguf (external GGUF
drafter) · MTP·asst (external assistant model) · DFlash · ngram (reserved)
· — (none). The full form (n= count + source) stays in the slug detail
card.
The Bring & Validate funnel mirrors the facet: the spec token is FOLDED
into the option label so two candidates identical on topology/engine/quant
but differing only by drafter disambiguate on `· MTP` instead of falling
to the serving-stem tail; no-drafter slugs stay bare.
Tests: _spec_token/_spec_label over every real drafter id; funnel fold +
disambiguation; column-set + ordering (weights·kv·spec before ctx).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The 27B nvfp4 ships MTP n=3 (the head is unquantized). On tight-system-RAM
rigs the extra draft-model load OOMs during boot — @paulp83's 28 GB laptop
5090 timed out on the 27B while the MTP-off 35B booted fine on the same
rig (#617). Previously the only workaround was hand-deleting the two
--speculative-config lines.
Both 27B nvfp4 composes (single + dual) now take `SPEC=off`:
SPEC=off bash scripts/switch.sh vllm/qwen-27b-single-nvfp4
- bare `- SPEC` env passthrough + a bash entrypoint that appends
--speculative-config only when SPEC != off (default: MTP on, unchanged).
- single: gained an entrypoint (was plain command); dual: folded into the
existing NVLink-detect entrypoint. --speculative-config moved out of the
static command into the conditional.
- headers advertise the toggle (Drafter line + the MTP comment).
Registry drafter stays qwen-mtp-builtin (MTP is still the default; SPEC=off
is a runtime override). Verified: entrypoint simulated with compose
$$→$ de-escaping — SPEC unset emits --speculative-config, SPEC=off omits it
+ prints the notice. Compose guards (mounts / status-drift / image-drift /
registry-parity / generate-compose) green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
@guybrush01's 5090 (#612) independently reproduces @paulp83's #619 to
within noise (decode 257.3/258.0 vs 255.8/257.9) AND validates deeper —
verify-stress needle-clean to 120K (91% of 131K) vs #619's 29K, at the
full 131K (no derate; the 35B MoE's tiny KV holds on a 32 GB 5090).
Two independent 5090s agreeing is stronger evidence than one, so the
1x5090-pcie submission now carries the deeper 120K ctx_validated and
credits both submitters/sources. TPS unchanged (identical within noise).
8-pack quality still owed on both (neither built the benchlocal sandboxes).
Responded on #612 (validation) + #617 (paulp83's 27B — a DIFFERENT failure
from #613: engine never loaded / 525 MiB container, MTP + 28 GB RAM, not the
#613 post-boot VRAM OOM). learnings updated (35b + the 27b two-failure note).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
@paulp83's single-5090 run is the first time the NVFP4 tier has booted
anywhere. Boots clean (vLLM v0.24.0, quant=modelopt_mixed, fp8 KV @131K,
MTP off as shipped), verify-full 9/9, verify-stress needle-clean (9.8K +
29K), soak-continuous PASS (15 MiB growth / 100% retention / 0 err / p50
311). Decode 255.8 narr / 257.9 code TPS @ 60 ms TTFT — ~2.5-3x our 2x3090
AutoRound tier (native FP4 GEMM). VRAM 30.6/32 GB @131K — tight but flat.
- registry: de-blind the status_note (drop "never booted" → first 5090
validation with numbers); stays 🧪 (8-pack quality still owed — that run's
benchlocal sandboxes weren't built).
- baselines.yml: first NVFP4 row — a 1x5090-pcie submission (narr/code TPS,
clean@29K, soak PASS, engine v0.24.0, tier submitted); surfaces ⑂-labelled
in c3 like the multi-max submission. No primary row (we can't run it).
- compose header: AUTHORED BLIND block → FIRST COMMUNITY VALIDATION with the
gate results + the 98304 fallback note.
- BENCHMARKS.md: the first NVFP4 row in the 35B-A3B section.
Confirms the two design calls: MTP-off (speculative_config=None) and fp8 KV
at scale=1.0 (declared-not-shipped scales) both loaded correctly on the
modelopt path. learnings + memory updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Per maintainer call: "cluster" conventionally means multiple networked
machines (a non-goal here — LiteLLM fronts multi-host) and is best reserved
for a future enterprise/multi-node meaning. "Pod" is the accurate analogy
for what this is — one model on a GPU subset on ONE host (k8s/RunPod sense).
The capability is unchanged; only the name.
Scoped rename (cluster→pod, case-aware) across the pod feature ONLY:
- scripts/cluster.sh → scripts/pod.sh; test-cluster-cli.sh → test-pod-cli.sh;
docs/CLUSTERS.md → docs/PODS.md
- estate_cli.py verbs + wording; app.py (ClusterCreateScreen→PodCreateScreen,
action_new_cluster→new_pod, _populate_clusters→_populate_pods, #cluster-view
→#pod-view, the [N] help/empty-state text); services.py cluster_create_plan
→pod_create_plan; data.py kind cluster_create→pod_create; tests + doc
pointers (HARDWARE/MULTI_CARD/README/c3-README)
- UNTOUCHED (unrelated "cluster"): compat.py + test-profiles-compat.sh (the
VRAM-topology classifier), services.py:1863 / test_services.py (the scene-
table "cluster by group" verb), older docs, .venv
Also folds in the [N] discoverability fix (n was already bound to
serving_switch — moved to N; empty-estate now shows a "no pods — [N] new
pod" affordance + a help entry) and a heterogeneous-rig section in PODS.md:
one homogeneous pod per card family (2×3090 · GB10 · 6000 Pro) is the clean
pattern — mixing families in one TP pod makes NCCL wait on the slowest +
wastes VRAM; a worked 2-pod lifecycle walkthrough.
Verified: test-pod-cli + estate/gpu/profiles guards green; 25 c3 pod/binding
+ 239 fast tests green; pod.sh live create/list/D1-reject on 2×3090; zero
stray "cluster" in pod files (scene-verb preserved); no CLUSTERS.md links
left; PODS.md leak-clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Clusters (cluster.sh + the c3 view/wizard + the estate file + CDI pinning)
had grown into a subsystem scattered across HARDWARE.md and the c3 README.
Consolidate into a dedicated docs/CLUSTERS.md, sitting alongside
SINGLE/DUAL/MULTI_CARD as the multi-model-per-host workload axis:
- concept (cluster = model + GPU set + port = estate instance) + cluster-vs-TP
- a Defaults section answering "what happens before you make any cluster":
no default cluster / empty estate; a plain launch boots ONE model whose
GPU count = the compose's TP (prompts both/all on multi-GPU), NOT the
cluster machinery — clusters are opt-in
- CLI quickstart (create/list/status/up/down/rm) + c3 quickstart ([n] wizard,
cluster view)
- D1 fit-vs-set rules, placement verification, the estate file schema, the
command reference
HARDWARE.md keeps the GPU-pinning mechanics (UUID / CDI-NixOS — a hardware
topic) and points to CLUSTERS for management; c3 README, MULTI_CARD, and the
docs index now link CLUSTERS. All cross-links verified; public-safe.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The in-cockpit cluster-create surface — Phase C's remaining piece.
[n] in Operate · Orchestration opens ClusterCreateScreen: name (Input) ·
slug (Select of catalog slugs) · GPU set (Input, prefilled with the free
GPUs from the estate poll). The modal dismisses {name, slug, gpus}; the app
builds a cluster_create ActionPlan (requires_reconcile=False — a file write,
no GPU claim) and routes it through the standard confirm gate → cluster.sh
create, which runs the D1 fit-vs-set + validate_estate gates and refuses a
bad set (count!=TP, GPU/port collision). The modal never touches the rig
(codebase convention — same as PowerCapMenuScreen).
- app.py: ClusterCreateScreen (mirrors SettingsScreen) + action_new_cluster
/ _on_cluster_create / _cluster_free_gpus + the [n] binding, action
descriptor, and Operate/Orchestration action-context gate.
- services.py: cluster_create_plan builder (mirrors set_default: a
requires_reconcile=False file write).
- data.py: cluster_create ActionPlan kind.
- test: modal collects+dismisses the payload, a blank field stays open, and
the plan builder produces the right cmd with requires_reconcile=False.
Verified: 9 headless (C1+C2+siblings) + 51 binding/action headless + 239
fast c3 green. Plan cmd integration-checked (creates a fit-clean TP=2
cluster). Design: serve-cockpit-design.md §10 — C2 SHIPPED; #610 Phase C
complete.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The cockpit surface for multi-model clusters: the Operate/Orch pane now
groups estate instances (clusters) with their GPUs stacked and a placement
health badge fed by the Phase-A assertion.
- estate_cli.py D3 (#610 addendum 3): report-state's active_estate.instances
now carries a per-instance placement verdict {requested, actual,
placement} (only probed for RUNNING instances). ONE source feeds both
cluster.sh status and the cockpit — no second poll.
- app.py C1: _populate_clusters renders a #cluster-view block below the
serving line — per cluster a header (● name · slug · :port · badge) with
GPUs stacked beneath (└─ GPU0 GPU1) + a trailing free-GPU line. The badge
is the Phase-A verdict: ✓ placed / ⚠ PLACEMENT MISMATCH, so the view shows
where clusters ACTUALLY landed (the failure mode that opened #610). Hidden
when the estate declares no clusters — the single-model case is unaffected.
- test-cluster-view headless: grouping + GPU stacking + ✓/⚠ badges + free
line; empty-when-no-clusters. Serving-panel siblings unaffected.
LIVE-VERIFIED (2x3090): cluster.sh up chat → report-state carries
placement=ok → C1 renders ✓ placed. 8 headless + 239 fast c3 + estate
guards green.
Design: /opt/ai/docs/serve-cockpit-design.md §10 (C1 shipped; C2 wizard is
the remaining Phase-C piece — the CLI wizard covers the capability today).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The CLI seam the c3 cluster UX (Phase C) will consume, and a full headless/
SSH cluster capability on its own. A "cluster" = a named model on a chosen
GPU set + port (an estate instance).
- estate_cli.py gains create / list / status / rm verbs (D2: NEW verbs go
IN estate_cli.py, which owns the schema + validate_estate + boot/down —
ONE validation path shared with hand-written estate files and the wizard).
up/down reuse `boot`/`down --only <name>`.
- create runs the D1 fit-vs-set logic (#610 addendum 3): kv-calc --card is
single-card + registry TP, so count != compose TP is a HARD REJECT, a
heterogeneous set is estimated against its min-VRAM card (+ note), and the
whole set is re-validated (validate_estate: GPU collision, port collision,
per-instance fits) before append. GPU indices stay index-based in the
estate file; UUIDs resolve at boot (Phase A).
- status carries the {requested, actual, placement} verdict per cluster
(the shape the c3 badge reads); list/status take --json.
- scripts/cluster.sh: the ergonomic bash front (create/list/status/up/down/
rm), a thin wrapper over estate_cli.py.
- test-cluster-cli.sh (NEW): full lifecycle hardware-free via CLUB3090_FAKE_
GPUS — create/D1-reject/collision-reject/list/status/rm.
LIVE-VERIFIED (2x3090): create chat (fit-clean ~20.2 GiB) → up → estate boot
→ "✓ placement verified" → status "serving ✓ placement=ok", GPU0 idle /
GPU1 loaded → down → rm. D1 count!=TP + GPU-collision rejects confirmed.
Part of #610 (Phase A′). Next: Phase C (c3 cluster view + wizard).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Extends the #611 primitive to the estate (multi-cluster) path and adds a
post-boot placement check — the foundation Phase A′ (cluster.sh) and C
(c3 cluster UX) sit on.
- scripts/lib/gpu-select.sh (NEW, shared lib): factors the #611 inline
resolver out of launch.sh into gpu_select_indices_to_uuids /
gpu_select_export, plus gpu_select_container_uuids +
gpu_select_assert_placement. launch.sh sources it (no re-inlined drift)
and runs the placement assertion after verify-full.
- estate_cli.py: resolve_gpu_uuids() (python twin of the bash resolver) +
compose_env / compose_override_doc now UUID-pin CUDA_/NVIDIA_VISIBLE_
DEVICES (ESTATE_GPUS stays index-based — the compose device_ids read the
host view). assert_placement[_quiet]() runs after each instance is ready
(sequential → stderr, parallel → per-instance boot log), returning the
{requested, actual, placement: ok|mismatch|unknown} verdict — the shape
cluster.sh status + the c3 badge will read.
- Placement uses --query-compute-apps=gpu_uuid (NOT --query-gpu): under CDI
the container sees all cards but RUNS on the CUDA-masked set, so
compute-apps is the runtime-agnostic ground truth.
- test-gpu-select (NEW): asserts the bash + python resolvers agree on real
hardware and both fall back identically; test-compose-gpu-mask-passthrough
updated to follow the resolver into the lib.
LIVE-VERIFIED (2x3090, classic runtime): `launch.sh --gpus 1` → UUID-pinned,
verify-full 8/8, "✓ placement verified", host GPU0 = 1 MiB / GPU1 = 20.8 GB.
estate compose_env confirmed UUID-izing while ESTATE_GPUS stays index-based.
CDI leg still awaits mog. Relevant guard sweep (11) green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
--gpus selection was NVIDIA_VISIBLE_DEVICES-only, which CDI runtimes
(NixOS hardware.nvidia-container-toolkit, nvidia-ctk cdi) IGNORE — mog's
3x3090 rig got all cards exposed and vLLM took the wrong two. Index-based
CUDA_VISIBLE_DEVICES can't fix it either: the classic runtime renumbers
the exposed set in-container.
UUIDs solve both at once:
- launch.sh --gpus N,M resolves indices -> GPU UUIDs (nvidia-smi -i) and
exports BOTH NVIDIA_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES as UUIDs.
Classic runtime: exposes-by-UUID; CUDA mask agrees (renumbering-proof).
CDI: exposure comes from device_ids (nvidia.com/gpu=all) and the CUDA
UUID mask is what pins the cards. Index fallback if the query fails.
- 26 composes gain a bare `- CUDA_VISIBLE_DEVICES` passthrough (unset ->
absent -> zero change for existing users); stale commented hints removed.
- preflight: selector consumers are host-index-based -> new
_preflight_selector_normalize maps GPU-xxxx tokens back to indices at
the single choke point (the first live boot caught this: preflight
rejected the UUID selector).
- new guard test-compose-gpu-mask-passthrough.sh: every compose with the
NVIDIA env line must pass CUDA_VISIBLE_DEVICES through + launch.sh must
keep the UUID resolution.
- docs/HARDWARE.md: "Pinning specific GPUs (and CDI / NixOS runtimes)" —
the runtime table, the CDI deploy-block swap, the in-container
renumbering gotcha, manual recipes.
LIVE-VERIFIED on this rig (classic runtime, 2x3090):
launch.sh --variant vllm/minimal --gpus 1
-> host GPU0 = 1 MiB (empty), GPU1 = 20.7 GB (model), both env vars
UUID-set in-container. CDI leg validation goes to mog (NixOS reporter).
Follow-up (tracked on #610): estate ESTATE_GPUS UUID-ization for CDI
multi-instance rigs; post-boot visible-GPU-count sanity check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
