Reconciles the earlier L2=0 finding. With L2 on (production shape), a 60 GB
L1 serves warm reads ~1.6x faster on average (~2x on the evicted half) than
a 30 GB L1 for a ~50 GB working set: older sessions that spill past a 30 GB
L1 fall to L2 disk (~3-5s) but stay in RAM at L1=60 (~1.3-2.4s).
Two-part model: L2 disk decides *whether* a read is warm (retention); L1 RAM
decides *how fast* (speed). L1 is a fast write-through front for L2, not a
standalone store — hence L2=0 gave 0/8 yet L2=1+L1=60 is fastest. Decision:
keep L1=60, it is not reclaimable headroom.
- Add SALT env to lmcache-retention-test.sh (fresh keys per A/B leg, no L2 collision)
- Add _local-lmcache-tuning/l1-ab-run.sh (two-leg orchestrator + comparison)
- Record result + reconciliation in HANDOFF.md
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bench.sh's PREFILL_PROBE salts every request fresh, so it measures cold
prefill + the immediate warm hit but never fills past L1 capacity — it
can't show retention, which is the whole point of raising LMCACHE_L1_GB.
This wrapper inserts N distinct large prefixes (sum sized between the
30 GB and 60 GB capacity lines), then re-reads them: session 1 is
LRU-oldest, so Round-2 TTFT per session is the retention curve. All-warm
= L1 held the working set (would have evicted earliest at L1=30).
Reuses bench.sh's streaming-TTFT method; sizes haystacks via /tokenize.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two tiers per #665: Tier 1 = never passed anywhere (10 — capability
gap), Tier 2 = reachable but rare (6 — probability gap; per-scenario
evidence inline). RM-13 moved out of never-set on the vLLM A0 OFF pass,
shrinking Sean's proposed 11 to 10. Same 16 selections, guard green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
First 8-pack on vllm/tess-dual-nvfp4 (both modes, same-night GGUF
comparator 116/117). Gap is cli-40-concentrated (−8 off); deterministic
packs tie or better (RM-off 14/15 best-ever Tess). Per the
pre-registered rule the huginnfork NVFP4A16 weights-only A/B + FP8
precision arm are the next gates. BENCHMARKS vLLM subsection + registry
note + compose Quality line; Grok-§3 discriminator results recorded in
learnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Shipped-config quality refresh on b9967 + 16K budget (native template,
both modes, no pack regression vs the 2026-07-09 basis; first toolcall
15/15 OFF). The streaming+thinking finish=length caveat is non-repro
3/3 on this config — Status ⚠️→✅, caveat history kept as a RESOLVED
note, registry caveats→production, Quality lines refreshed. Drift +
parity + registry-json guards green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
quality-test.sh passes through --scenario / --scenarios-file /
--incremental / --resume / --allow-partial (benchlocal #84/#85): bare
selections derive their pack set (custom mode), sandbox preflight fires
when a selection touches Docker packs, and --resume refuses
mode/selection/thinking/sampling/timeout flags rather than fork the
restored config. Selection runs print the partial-result warning.
scripts/scenario-sets/ ships the two curated Tess probe sets with
provenance headers: tess4-model-floor.txt (14 fails-everywhere + 2
thinking-only across 2 rigs / 2 drafters / 2 engine builds — the
retrain-target list) and tess4-engine-window.txt (CLI-25/31/32, the
b9932→b9967 discriminators for cheap engine-arm checks).
rerun-failed-packs.sh now re-runs a prior run's failures as ONE
selection run with --incremental durability (was: whole-pack loops).
Guards: test-scenario-sets.sh (format/provenance/passthrough) new,
test-rerun-failed-packs.sh updated; live-validated against the running
Tess serve (Set B: 3/3 pass, partial-labeled, sandbox auto-enabled).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Generalizes what beellama-pin-bump.sh does for the one digest-pinned
engine to every engine profile: rewrites install.spec + the display_name
tag echo + every registered non-deprecated compose's image: default, and
REPORTS (never edits) the pin strings that encode validation claims —
pin-asserting fixtures, arch_patches.yml loads:true rows, UPSTREAM.md
rows, hand-written launcher suggestion strings, and prose/history —
bucketed so the six lines that need hands aren't buried under 100+
baselines/compose-header mentions. --check dry-runs with a unified diff.
The judgment half (patch re-validation, live boot + warm-up, trackers)
stays with the CLAUDE.md checklist, which the script prints. Paid for
by hand three times: the v0.22→v0.24 vLLM bump, the b9967 llama.cpp
bump (#680), and the upcoming v0.25 evaluation.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
b9246 was 52 days / 721 builds old. Validated on Tess dual (full quality
A/B, 2026-07-11): think-OFF total tie 115=115 (greedy-leg engine
isolation — totals neutral, though 4 scenarios flipped each way),
think-ON 122 vs the 118x3 pin — the lift is thinking-path-specific;
3 previously-100%-reproducible failures now pass (CLI-31 safety refusal,
CLI-20 archive precision, RM-04 math). Boots clean, draft-mtp works.
Also unifies the pin landscape: Deckard's b9570 and the ROLLING tags on
gemma-12b/vibethinker (a #187-policy violation — rolling crash-looped at
b9282) all move to the same validated build. Engine-profile spec now
matches the composes. Non-Tess slugs: boot re-validation is spot-checked
(smoke to follow on PR); bench numbers stay dated to their measured pin;
baselines rows go stale by design (the staleness badge is the feature).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
migtissera NVFP4 via Marlin W4A16 fallback: 62.4 tok/s decode spec-off
TP=2 @131K (BENCHMARKS 2026-07-11) vs the llama.cpp entry's 57.9 with
MTP — first vLLM-servable Tess on consumer cards. Ships experimental:
the 8-pack (the vLLM quality program A0 baseline) + stress/soak are the
promotion gate; pin-fallback rule to huginnfork NVFP4A16 documented in
compose + weights manual_note.
Catalog plumbing: nvfp4 weights variant; qwen35-dense added to
vllm-stable supported_model_families (live-validated); froggeric
template coverage extended to the tess compose (shared vendored copy,
patches.yml); kv_calc_supported=false on the model (hybrid — KV on
16/64 layers, naive math ~4x over; wire at promotion). Also finishes
the 2026-07-11 hybrid correction in the two spots today's sweep missed
(model YAML comment, llamacpp registry status_note). Counts 63→64 /
64→65. Full suite 77/78 (submit-bench = worktree fixture artifact);
diagnose-profile GREEN.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
benchlocal-cli's finest run granularity is --pack (no per-scenario run
filter — benchlocal-cli#82), so after a --full run the "are these
failures real?" question cost another 1-2 h full re-run. This script
parses a saved RunResult, re-runs ONLY the packs containing failures
(through quality-test.sh, keeping its hermes-env/timeout guards),
matches the original run's thinking mode from the JSON, passes
--previous-result for benchlocal's own per-scenario delta, and prints a
consolidated REPRODUCED / FIXED(flake) / NEW-regression verdict.
Supports --repeat N passthrough for flakiness rates and RERUN_DRY=1.
Guard test included.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Full gate complete on the reference 2x3090: verify-stress 8/8 (NIAH
clean to 240,635 tok = 91% of 262K, ceiling margin 1,801 MB, zero drift
across the ladder) + soak-continuous PASS (0 errors, 0/25 silent-empty,
0 VRAM growth, p50 215.9, 100% retention), joining the existing bench
(179.5/179.4 decode) and 8-pack (103/150 tie) results.
Caveats (why not full Production): streaming tool-calls + thinking-on
hit finish=length (known family class, verify-full check 6); native-FP4
quality remains unvalidated (all numbers are the Ampere W4A16 bound).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
In-worker apply_model probe: all 16 full-attention layers carry the
shipped calibrated values byte-exact (0.023-0.271, none at 1.0), so the
provider A/B was calibrated-scales vs scale=1.0 — and tied. Corrects the
"load-on-hybrid unverified, assume scale=1.0" claims in the nvfp4-fast
compose header, registry status_note, and both BENCHMARKS rows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Add vllm/qwen-35b-a3b-dual-nvfp4-fast: Ampere-validated unsloth NVFP4 MoE
New weights variant nvfp4-fast (unsloth/Qwen3.6-35B-A3B-NVFP4-Fast,
compressed-tensors mixed W4A4+FP8, auto-detected quant) + dual compose
(port 8080) + registry entry. First-party validated on the reference
2x3090 2026-07-11 — first MoE-FP4 fallback boot (MARLIN NvFp4 MoE
backend): decode 179.5/179.4 and 8-pack think-off 103/150, double
statistical tie with the AutoRound tier at full 262K (22.46 GB/card).
Ships 🧪 until verify-full/stress/soak run; native-FP4 (sm_90+) quality
remains the arc's missing datapoint (W4A4 activations unexercised on
Ampere). Registry counts 62→63, disk 63→64.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Fix MAX_ARG_STRLEN cliff: pass registry emit via temp file, not env
The full registry JSON (and the intermediate tab rows) were passed to
python heredocs through single environment variables. A single env value
is capped at MAX_ARG_STRLEN (~128 KB on Linux); the emit crossed it at
63 registry entries — "Argument list too long" in test-baselines and
test-registry-json, with the runtime REGISTRY_TAB pass in
registry-emit.sh sitting a few KB from the same cliff. All three now
write to a mktemp file and pass the path; python reads it with
encoding="utf-8".
Full suite: 76/77 green (test-submit-bench needs gitignored
results/rebench fixtures absent in a fresh worktree — env artifact).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Move nvfp4-fast slug to port 8081 — 8080 is Open WebUI on common rigs
Caught by the boot-the-actual-compose gate: the health probe on 8080 hit
Open WebUI and verify-full ran against the wrong service. 8081 is free
and unclaimed in the registry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Record actual gate results in nvfp4-fast compose header
switch.sh --force boot 272s + verify-full 7/1/2: the single failure is
streaming-toolcall+thinking-on finish=length — the known family caveat
class, not compose-specific. Stress/soak remain the 🧪→promotion gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add spec-sweep.sh: draft-depth n-sweep for spec-decode drafters
The n-sweep has been hand-rolled repeatedly (Deckard MTP, Tess MTP
2026-07-09, EAGLE3 2026-07-11, gemma n=4-vs-8, DFlash n=5/7/8) — same
shape every time. This standardizes it as the tuning sibling of
concurrency-probe.sh (streams) and power-cap-sweep.sh (watts).
Two engine paths:
- llama.cpp FAST PATH (no reboots): per-request speculative.n_max +
timings.draft_n/draft_n_accepted/predicted_per_second — full curve
against one live server in ~2 min. Capability probe REFUSES (exit 3)
when the server doesn't report draft_n, so an ignored field can't
fabricate a flat curve.
- vLLM: reboot per arm via switch.sh (SPEC_N_MAX=<n>; SPEC=off for the
n=0 baseline arm) — no per-request draft-depth knob exists.
Output: curve table + sweet-spot line (flags a net-negative drafter
when n=0 wins) + machine-readable RESULT rows. SWEEP_DRY plans without
booting. Offline guards: scripts/tests/test-spec-sweep.sh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* spec-sweep: enforce the probe differ-check + llama.cpp reboot fallback
Live validation caught the gap: llama-server b9246 silently IGNORES
per-request speculative.n_max (probe draft_n identical across n) and the
first validation produced exactly the fake flat curve the probe was
designed to prevent — the differ-check was documented but not enforced.
- probe now requires draft_n(n1) != draft_n(n4); ignored field ->
reboot-per-arm fallback via MTP_DRAFT_N_MAX=<n> switch.sh (llama.cpp
boots ~15s; slug default config restored after the sweep)
- n=0 baseline guard: a compose with a hardcoded drafter that ignores
SPEC=off gets marked SPEC-OFF-IGNORED instead of lying
- validated against ground truth: reproduces the hand-rolled 2026-07-09
Tess MTP curve (n1 55.67/0.74 · n2 55.86/0.58 · n3 48.8/0.47 vs
55.6/0.75 · 57.9/0.62 · 49.3/0.46) in 3m27s
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
NVFP4 runs on sub-sm_90 cards via the Marlin W4A16 weight-only fallback
since vLLM v0.24 (floor = capability 7.5, marlin_utils_fp4). Live-confirmed
on 2x RTX 3090 sm_86 2026-07-11: the shipped dual/nvfp4/mtp.yml boots at
262K + fp8 KV + MTP n=3 (accept 97%+), 69.7/85.5 decode TPS, and scores
8-pack think-off 110/150 — a statistical tie with the fp8 production
tier's 109. c3 hiding these slugs as "incompatible-hw" was factually wrong.
Semantic split — required_sm stays the NATIVE floor; new registry field
`fallback_sm` = the weight-only-fallback floor:
- kv-calc: in the band [fallback_sm, required_sm) the fit is PRICED
normally and annotated with `hw_fallback` {required_sm, card_sm, note}
instead of returning incompatible-hw (both --fit and --fit-all).
- c3: FitVerdict carries hw_fallback; fits-* + fallback -> "⚑" glyph,
VISIBLE by default. incompatible-hw hiding stays for true incompatibles.
- gates.py/compat.py: the hard SM floor becomes fallback_sm when present
(else required_sm, unchanged).
- Registry: fallback_sm=7.5 on the 4 NVFP4 slugs + status_note honesty
pass ("REFUSES on Ampere" -> fallback reality + measured numbers);
same corrections in the 4 compose headers.
- test-kv-calc-fit section (f) rewritten to the new contract.
On Ampere these slugs still carry the honest downside in the note: no
speed edge (~20% slower than the AutoRound tier for the same model) —
their sub-sm_90 value is models where NVFP4 is the only quant.
Guards green: kv-calc-fit, registry-disk, status-drift, switch/launch
parity, profiles-compat, pullgate, pull, registry-json/emit, diagnose,
launch-compat, default resolvers, kv-calc --calibration; c3 248 fast
tests + headless incompatible-hw hide tests.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* concurrency-probe: surface aggregate tok/s in verdict, RESULT, and sweep
The probe already computed aggregate throughput (summed completion
tokens / wall) per round but only printed it in the round table — the
verdict, RESULT line, and sweep knee reported per-stream only. Users
asking "what total throughput at N agents?" (the Bruno question) had
no first-party number to point at.
- verdict: aggregate tok/s (N streams) next to per-stream
- RESULT: agg_tps= field (machine-readable, sweep-parseable)
- SWEEP: per-N summary table (per-stream vs aggregate) + aggregate in
the knee line
Knee logic unchanged (still per-stream-floor based). Offline guard
test-concurrency-probe passes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* concurrency-probe: re-resolve served model after each SWEEP boot
The top-of-script MODEL autodetect runs before SWEEP boots anything on
$URL, so it silently falls back to the default (qwen3.6-27b) — and every
request against any other model 404s, reading as errors=N per round (hit
live sweeping vllm/qwen-35b-a3b-dual on :8051; the 27B sweep only worked
because the fallback equals its served name). Re-resolve from /v1/models
after each boot; an explicit caller MODEL= pin is respected (MODEL_PINNED).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* FAQ + BENCHMARKS: first-party concurrency/aggregate-throughput matrix
Measured via the extended concurrency-probe (this PR) on 2x3090:
- dense 27B dual-fast: batching knee N=2 (~104 decode-agg @16K ctx),
collapses to ~54 by N=8; generation shape 211 agg @ N=8
- 35B-A3B MoE dual: ~250-270 decode-agg FLAT N=2..16 @16K ctx;
generation shape 1,037 tok/s aggregate @ N=16 (92.8/stream, clean)
New FAQ entry: how to serve multiple coding agents (MAX_NUM_SEQS,
ctx-vs-concurrency, model pick by architecture, honest aggregate-vs-
per-stream caveats, prefill-bound note, probe invocation).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@seanyourhighness diagnosed a hard HTTP 500 at turn 2 benching Tess-4-27B:
tool_choice=required + max_tokens=150 truncates the tool-call JSON on a
reasoning model (finish=length), and the harness replays that unterminated
`arguments` string in `messages` — llama-server then throws in
common/chat.cpp func_args_not_string() re-parsing the client history, so a
turn-1 clip poisons every subsequent turn.
Two-part fix:
- max_tokens 150 -> 600 (600 completes the call per the report).
- _safe_args guard: validate each accumulated tool_call `arguments` parses as
JSON before replaying it in `messages`; substitute "{}" if it doesn't, so a
single clip can't kill the whole ramp even if a slower model still truncates.
The upstream half (malformed client-supplied arguments -> HTTP 500 instead of
4xx, template-independent) is being filed against llama.cpp separately.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two independent RTX 5090 validations agree within noise:
- #619 (@paulp83): first NVFP4 boot on Blackwell — verify-full 9/9,
verify-stress needle-clean, soak-continuous PASS.
- #612 + #652 (@guybrush01): verify-full 9/9, verify-stress to 120K
(91% of 131K), soak PASS, p50 ~311 TPS. (#652 re-confirms #612 within
noise — one rig confirmed twice.)
status experimental -> caveats + compose header Status/Caveats. Caveat =
8-pack quality NOT yet cross-rig-measured (sandboxes weren't built on
these runs; quality run pending, gated on #492) — reverts to experimental
if a quality run regresses. NVFP4 stays Blackwell/Hopper-only (sm_90+).
Unchanged: 27b-single-nvfp4 (shipped util OOMs a 5090 — #613/#617, needs a
config fix) and both dual-nvfp4 slugs (no reports) remain experimental.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `-h|--help` case ran `sed -n '2,55p' "$0"`, but the script cd's to the
repo root before arg-parsing — so a relative-path invocation from another
cwd left `$0` unresolvable and (under set -e) `--help` printed nothing. It
also dumped raw #-comment lines and truncated at line 55, omitting the
endpoint-first + env-override sections.
Replace with a `usage()` heredoc: cwd-independent, complete (all flags + env
overrides + examples), cleanly formatted. `-h` and `--help` both print it.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The emit-swap group asserted PRE-P2b behavior and crashed on any rig (not env-
specific): P2b (swap_apply served-name + SPEC env-gating) now emits
--served-model-name ${SERVED_NAME:-<name>} and LIFTS --speculative-config out of
the command into a ${SPEC:-on}-gated entrypoint (${DRAFTER_METHOD:-mtp}). The
test still checked the plain served-name + spec-config-in-command, so _val()
returned None on the moved flag -> 'x' in None -> TypeError crash. Updated 3
assertions to P2b's shape. test-pull-swap.sh now green (also unblocks the route-C
location asserts added in #655).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Companion to #654 (route-G): the safetensors weight-swap path had the same
"compose in the project tree" problem the user flagged — swap_apply.emit_swap_
compose wrote _brought-<san>.yml NEXT TO the sibling compose (so its relative
../ mounts resolved), dropping throwaway files beside catalog composes.
Route-C's MOUNT was already fine (it dir-mounts the brought weights to a distinct
/brought-model), so this is location-only: absolutize every sibling volume via a
new _absolutize_volume (${VAR:-../rel} → ${VAR:-/abs}, plain ../rel → /abs; the
vLLM sibling has 5 relative mounts — HF cache, torch_compile, triton, the
froggeric chat template, detect_nvlink) so the compose is relocatable, then write
it to the runtime dir beside the pull dir (<MODEL_DIR>/.cache/huggingface/
club3090/composes/), falling back beside the sibling.
Validated at emit level: all relative sources absolutized, ${MODEL_DIR} env var
preserved, brought mount intact, compose lands in the runtime dir not the repo.
+2 assertions in test-pull-swap.sh's emit group. (That .sh has a PRE-EXISTING
env failure on this rig — identical on master, unrelated.)
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
resolve_hf_home was `--hf-home > $HF_HOME > $XDG > ~/.cache`, ignoring MODEL_DIR.
So a bare `pull.sh <repo>` (or a nohup apply-swap) run with only .env's MODEL_DIR
set and no explicit HF_HOME silently fell to ~/.cache on the ROOT disk — the
footgun that misplaced a brought model's 35 GB of weights off the models volume
(and would bite users the same way). Insert a MODEL_DIR step (env, else parsed
from the repo .env, encoding=utf-8) between $HF_HOME and $XDG, matching what
switch.sh/launch.sh/c3 already resolve. c3 is unaffected — it sets HF_HOME
explicitly, which still wins. New guard test-hf-home-resolve (PASS 1-5);
test-pull / test-pullgate-download / test-download-lock still green.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
select_weight_files built the sharded set as `"-of-" in n or startswith
"model-"`, so a grafted MTP/nextn head (e.g. `mtp_grafted.safetensors`) — a
real weight the model needs with MTP enabled — was dropped, even though
detect_mtp_head/_has_mtp_weight_file already see it. Live dogfood: Tess-4-27B-FP8
downloaded its 7 model-* shards but silently omitted `mtp_grafted.safetensors`,
which would break MTP serving (club-3090 #617). Union any non-shard mtp/nextn
*.safetensors head into the sharded download set. Guard: test-pullgate-download
gains an MTP-head fixture.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* c3+pull: detect in-progress downloads + per-repo lock (#617)
Repeated ① Bring [D] presses spawned N concurrent `hf download` runs racing
into the same `.incomplete` staging dir (5-deep live), each rmtree-ing and
re-fetching the others' partial. Two disk-truth gaps behind that:
1. No dedup. `download_model` now acquires an atomic per-repo lock
(`mkdir <pull_dir>/.download.lock` + a `pid` file = holder PID + UTC start)
BEFORE the staging rmtree. A 2nd concurrent call for the same slug is
REFUSED with `failure="in-progress"` (not a race); swap_apply surfaces it
as `in_progress`, and `pull.sh --apply-swap` exits **rc=3** ("already
downloading", distinct from rc=1 failure). A STALE lock (dead holder — a
crashed/SIGKILL'd download that skipped `finally`) is reclaimed on the next
call, so a leaked lock self-heals — more robust than a signal trap (a
SIGKILL skips traps too). Released in `finally` on every return path.
2. No in-progress detection. Presence probes only saw the FINAL dir (empty
until hf moves files out of `.incomplete`), so a mid-flight download read as
absent → the false "download did not complete". `services.bring_download_in_
progress` now reads the lock (live PID) + `.incomplete` bytes for a
best-effort pct; c3's fit-check render + [D] no-op guard + [k] cancel are
disk-aware, so a running download is REFLECTED (and not duplicated) even
across a c3 restart or a bare `pull.sh` started outside the session — the
in-memory tracker (#643) couldn't see those.
download_model is the shared chokepoint (bring apply-swap AND catalog fetch),
so both paths get the lock; the body moved verbatim to `_download_model_impl`
behind the locking wrapper — no behaviour change beyond the added guard.
Tests: scripts/tests/test-download-lock.sh (live-refuse + stale-reclaim +
release); c3 test_services (in-progress live/stale/absent + pct) +
test_app_headless (disk-detected → downloading render, [D] no-op, [k] kill).
245 services/registry + 131 app blast-radius tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* downloader: refuse a fresh pidless lock (close acquire-window race)
The lock acquires with mkdir then writes its pid file a beat later. A 2nd
caller landing in that window saw the lock dir but no pid → read_active
returned None → the wrapper treated it as stale and rmtree'd + reclaimed it,
letting BOTH callers proceed — the exact duplicate-download race the lock
exists to prevent. Now: a pid-present-but-dead lock still reclaims (unchanged),
but a PIDLESS lock only reclaims once it's older than a 10s grace — a fresh
pidless lock is a holder mid-acquire and is refused, not stolen. Guard adds
PASS 4 (fresh pidless → refuse) + PASS 5 (old pidless → reclaim).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* baselines: backfill dual-max quality_8pk_think_on (110/150)
The vllm/qwen-27b-dual-max row logged only the thinking-off 8-pack
(107/150); the thinking-enabled run existed since the 2026-06-07 #515
quant-A/B (fp8 dual-max full 8-pack = 110/150, pack-defaults regime,
autoround sibling 109) but was never promoted. Backfilled with
provenance so it isn't re-derived. Guards: test-baselines,
test-quality-thinking, test-quality-baseline all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* preflight: warn on single-card GPU_MEMORY_UTILIZATION override above default
Two 5090 testers (#617 paul, and the same trap earlier) OOMed the tool-prefill
step of verify-stress by setting GPU_MEMORY_UTILIZATION=0.92 on the nvfp4
single-card slug whose validated default is 0.85. On one GPU a higher util
steals the free VRAM a large tool-response prefill needs for its activation
peak, so vLLM OOMs mid-prefill even though boot succeeds — a confusing failure
(boot ok, then HTTP 500 at step 2/8) that vLLM's own error hints at ("lower
--gpu-memory-utilization").
Add preflight_single_card_util(): an advisory WARN that fires only when the
user overrode GPU_MEMORY_UTILIZATION *above* the compose's shipped default on a
TP<=1 config. Runs even under --force (the nvfp4 slug launches with --force),
and no-ops for dual/multi-card, non-vLLM engines (no util default in the
compose), a plain default run, or util at/below the default. Wired into
switch.sh alongside the other force-independent hints.
Verified: behavior matrix (single>default WARN; single=default/below/none and
dual all silent) + test-preflight-compose-deps / test-model-switch /
test-switch-registry-parity / test-launch-compat all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pinned cu13 ik-llama digest has a CUDA 13.2 runtime. On a driver whose
supported CUDA < 13.2 (e.g. 580.159 = CUDA 13.0, which satisfies the README's
"580.x+" but NOT this pin) the forward-compat path fails on GeForce (CUDA error
804) -> silent CPU fallback -> segfault crash-loop, with launch.sh just timing
out after 600 s and no actionable hint.
- preflight.sh: preflight_ik_llama_image() detects the driver's supported CUDA
(_driver_cuda_version) and, for ik-llama variants on a driver < 13.2 with
IK_LLAMA_IMAGE unset, auto-selects the cu12 sibling build (same build 4574,
CUDA 12.6, backward-compatible, DelspoN-validated ~68 TPS on 580.159) + prints
a ⚠ hint. >=13.2 keeps cu13; a user IK_LLAMA_IMAGE pin always wins (echoed).
_cuda_ge() does the major.minor compare. Verified: 13.3/13.2 GE, 13.0/12.6 LT.
- switch.sh: call it right after export_variant_engine_pin, before compose up
(the single up path — launch.sh delegates here). preflight.sh is already
sourced in that block.
- docs: qualify the driver requirement in the ik-llama compose header + HARDWARE.md
("580.x+" is insufficient; needs CUDA >= 13.2; auto-fallback documented).
Note: this rig is CUDA 13.3, so the <13.2 boot path is verified by logic +
DelspoN's cu12-server-4574 validation, not a local boot. Rolling cu12 tags are
NOT safe (cu12-server-100 rejects the compose's --spec-type flag, #633) — pinned
to the build-number tag matching the cu13 digest's build.
Closes#633.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(c3): ② Serve spec-dec — engine-driven drafter SELECTOR (not just on/off)
The spec-dec dropdown now lists the resolved ENGINE's supported drafters + off
(vLLM: mtp / mtp_assistant; beellama: dflash / mtp / mtp_gguf; …) instead of a
bare on/off — so the drafter TYPE is selectable, and it's engine-correct (dflash
only appears when the swap engine is beellama).
- swap_apply: the SPEC entrypoint REBUILDS --speculative-config via printf from
${DRAFTER_METHOD}/${DRAFTER_N} (defaults = the sibling's), so switching the
drafter is an env override — no re-emit. Validated: default→mtp n=3,
DRAFTER_METHOD=mtp_assistant→that config, SPEC=off→dropped, compose config valid.
- services: engine_drafters() reads the profile's supported_drafters (shares the
stdlib list-parse with engine_kv_formats via _engine_yaml_list); serve_override_
defaults adds DRAFTER_OPTIONS + SPEC_METHOD + SPEC_N.
- app: the spec Select value is the drafter method (or "off"); collect maps
method→SPEC=on+DRAFTER_METHOD, off→SPEC=off.
Drafters with extra requirements (mtp_assistant needs an assistant model; dflash
needs the beellama engine) are honestly listed per-engine — picking one the swap
can't satisfy fails at boot (loose-validation stance), or use a matching ① Bring
slug. 173 byo/serve/override tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* fix(launcher): GGUF verify-glob (#634) + launch.sh .env engine-image passthrough (#632)
#634 — setup.sh re-hardcoded VERIFY_GLOB=*.safetensors at the primary
download/verify (line 625), clobbering the recipe's glob that
load_weight_recipe already set from PRIMARY_WEIGHT_KEY. Every GGUF primary
fetch (WEIGHTS=gguf/iq4ks) then failed verify ("No *.safetensors found")
despite a byte-correct download. Now defaults to ${VERIFY_GLOB} (the recipe's
*.gguf); VERIFY_GLOB_OVERRIDE still wins.
#632 — launch.sh's .env load loop gated to CLUB3090_DEFAULT_* only, so
IK_LLAMA_IMAGE / LLAMACPP_IMAGE pins in .env were silently dropped (the two
GGUF engines are not profile-injected, so .env is their only override path —
the natural persistence for the #633 cu12 workaround never reached compose).
Broadened the allowlist to the engine-image keys; switch.sh already loaded
them (no filter). Both launchers now echo an ik-llama/llama.cpp image pin so a
wrong-image boot is never silent. (DelspoN's report cited the engine-pin
whitelist, which is the vllm/beellama-only profile-injection path — the real
drop was the .env loop filter.)
Verified: the .env loop now loads IK_LLAMA_IMAGE/LLAMACPP_IMAGE and still drops
un-allowlisted keys; test-model-switch / test-launch-compat / *-registry-parity
/ test-setup-picker / test-model-weights-registry all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 2 of the ② Serve rework (Phase 1 = #635 dead-end fix): a Route-C brought
model can be re-tuned before serve, without hand-editing a compose. Fields
(pre-filled from the resolved slug's defaults, dropdowns to avoid typos):
• served name → SERVED_NAME (Input)
• ctx → MAX_MODEL_LEN (Select — presets + the slug's own default)
• KV cache → KV_CACHE_DTYPE (Select — fp8_e5m2/e4m3/turboquant_4bit_nc/…)
• spec-decode → SPEC on/off (Select)
• VRAM util → GPU_MEMORY_UTILIZATION (Select)
Mechanism (all five ride env — no per-serve compose rewrite):
- data.py: ActionPlan gains `env`; dispatch merges {**os.environ, **plan.env}.
- services.py: serve_generated(compose_path, overrides) → plan.env (+ pins
MODEL_DIR); serve_override_defaults() parses the sibling compose's
${VAR:-default} for pre-fill (stdlib regex, no PyYAML).
- swap_apply.py: emit parameterizes --served-model-name ${SERVED_NAME:-…} and
gates --speculative-config behind ${SPEC:-on} via the SAME entrypoint the
shipped nvfp4 compose uses — so SPEC=off drops the MTP drafter at up-time.
- app.py: LaneServePane fields, revealed + pre-filled only for a Route-C armed
model (hidden otherwise); collect_overrides() → serve_generated. Loose
validation (dropdowns constrain; the reconcile gate + boot are the real check).
Validated: emitted compose `docker compose config` resolves SERVED_NAME=X and
SPEC=off (entrypoint gate) cleanly; 169 byo/serve/override tests pass.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A Route-C fit-check whose weights were already downloaded showed a
self-contradiction: the verdict card hardcoded "→ Press [D] to download +
serve" while the weights-line below it correctly said "✓ weights on disk →
② Serve". [D] is not only download — for a Route-C brought model it emits the
serve-locally swap compose that ② Serve then serves — so it can't simply be
hidden.
Fix (presence-aware, "② Serve owns the emission"):
- _byo_result_text takes weights_present (probed once in the fit-check
handler, feeding BOTH the card and the weights-line so they can't disagree).
On disk → the card points at ② Serve, no [D]; absent → the [D] download
prompt as before.
- pull.sh gains --emit-only (with --apply-swap): passes do_download=False so
apply_swap emits the serve compose WITHOUT downloading — the present-weights
path.
- action_serve_untested: Route-C + weights present + no swap compose yet →
run_bring_emit_and_serve emits via pull.sh --apply-swap --emit-only, then
serves it. So a present-weights brought model serves straight from ② Serve
with no [D] step.
Tests: fixed test_bring_result_points_forward_to_serve (stale since #628 — it
asserted "→ ② Serve" on an absent-weights card that now honestly shows [D]);
added test_bring_result_present_points_to_serve (present → ② Serve, no [D]);
extended the run_bring_download test for --emit-only. 139 touched-surface
tests pass; pull.sh --emit-only validated end-to-end.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(byo): apply-swap — download + serve a Route-C fine-tune
A curated-arch fine-tune (e.g. josefprusa/ThinkingCap-Qwen3.6-27B AutoRound
→ qwen3.6-27b) hard-stops at the pull gate's stratum-5 no-fit-model (correct
— nothing to price), so the c3 Bring [D] download failed with "download did
not complete". Route-C was pure guidance ("clone the compose, point --model
at your weights") with no code behind it — the deferred "swap_path apply".
Build it as a DISTINCT action that never touches the locked 6-stratum gate:
- scripts/lib/profiles/swap_apply.py (new): resolve the Route-C swap
(arch→curated sibling + has_mtp_head from deriver), download the brought
weights SHA-verified via downloader.download_model (a minimal EInput), and
emit a serve-locally compose that CLONES the --profile-like sibling's REAL
compose (keeping its curated chat-template / reasoning+tool parsers / MTP
wiring — NOT the derived-vllm template, which drops all that) with --model
re-pointed at a `<weights>:/brought-model:ro` mount, --served-model-name set
to the brought basename, and --speculative-config kept iff the checkpoint
carries an MTP head (else dropped). Written alongside the sibling compose as
`_brought-<san>.yml` (gitignored) so its relative ../ mounts resolve.
- scripts/pull.sh: intercept `--apply-swap` in the wrapper (strictly additive,
like --json) → run the apply-swap action; prints `[apply-swap] compose: …`.
The gate path is byte-unchanged without the flag.
- c3: services.run_bring_download gains `apply_swap=` (appends --apply-swap +
captures the emitted compose via last_swap_compose()); the [D] worker passes
it on route==C and stashes the compose; ② Serve serves that swap compose
directly (serve_generated) instead of a reproduction of the sibling's own
catalog compose.
- pull.py: fix the stale NOTE — "the bf16 base won't fit and lacks the MTP
head" → the base HAS the head; now just the size reason.
Tests: test-pull-swap.sh section 4 (emit: --model repointed, MTP kept/dropped
by head presence, curated flags + JSON args survive the YAML round-trip,
distinct container_name); test_services apply_swap flag + compose capture. The
locked-gate hard-stop tests stay green; --json fit-check byte-identical.
Validated live: pull.sh --json still no-fit-model+route-C, and apply_swap emits
a correct ThinkingCap swap compose (MTP kept).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* c3: reframe Route-C fit-check card as ✓ Servable + a clear [D] action
The fit-check card for a curated-arch fine-tune (Route-C swap) was
self-contradicting: a RED "not eligible" + a bare "no-fit-model" token at
the top, then a GREEN "② Serve is armed with <sibling>" at the bottom —
which read as a dead-end AND named the wrong model (the sibling, not the
brought fine-tune). The engine's "no-fit-model" is the *means* (generic
fit-math can't price a curated-hybrid arch), not the user's answer; the
outcome is servable via the sibling's recipe.
Reframe the Route-C case in _byo_result_text:
- green "✓ Servable — a fine-tune of <sibling>" instead of red "not eligible",
- one plain "How it serves" line (reuses the sibling's chat-template / tools /
spec-dec with your weights) instead of the --quantization/impl bullets,
- MTP kept/dropped line from has_mtp_head,
- the next-step is an explicit "→ Press [D] to download + serve <BROUGHT model>"
(the fine-tune, not the sibling) — which #628's --apply-swap now fulfils,
- raw verdict + backend note dimmed for debugging, not the headline.
Non-swap cases (eligible / Route A / B / plain no-fit) are byte-unchanged.
Regression test added; c3 fast suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The original OOM at 98K was caused by MTP, not the context length. Two
5090s (#613 @guybrush01, #617 @paulp83) swept the space: MTP-on at 98K
leaves no room for the draft head + cudagraphs + GDN prefill scratch and
OOMs — but at 65K + util 0.85 it PASSES, with MTP's ~2x intact:
verify-stress all-pass, decode 129/155 TPS (vs 71/71 with SPEC=off), MTP
accept ~3.2, ~1.4 GB VRAM free.
So the default keeps MTP and trades context, rather than dropping MTP:
- compose single/nvfp4/mtp.yml: MAX_MODEL_LEN 98304 -> 65536,
GPU_MEMORY_UTILIZATION 0.92 -> 0.85, header updated to the swept config.
- registry: max_ctx 98304 -> 65536, mem_util 0.92 -> 0.85, status_note.
- BENCHMARKS: guybrush's SPEC=on+65K row (129/155 decode, accept 3.2).
- baselines.yml: 27B single-nvfp4 submission (stale=false, v0.24.0).
SPEC=off stays exposed as the max-ctx fallback (81K/98K @ 71 TPS) and the
tight-system-RAM path (MTP's draft load OOM-kills a 28 GB host, #617).
80 GB+ cards raise MAX_MODEL_LEN toward 262K with MTP on. Stays 🧪 (8-pack
quality still owed). Scoped guards green (baselines / status-drift /
switch+launch parity / profiles-compat).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The BYO fit-check's Route-C weight-swap (pull.sh _swap_path) hardcoded
`drop_spec_config: True` on the premise "a generic repo carries no MTP
head." That's false for a fine-tune that PRESERVES the head — so any such
model (e.g. ThinkingCap-Qwen3.6-27B, whose AutoRound INT4 ships the full
15-tensor BF16 mtp head in model_mtp_bf16.safetensors) was silently served
MTP-off, losing spec-dec despite the head being present.
Fix: detect it. `deriver.detect_mtp_head(config, api)` returns True when the
config DECLARES the MTP layers (`mtp_num_hidden_layers` / nested
`num_nextn_predict_layers`) AND a dedicated mtp weights file is present —
ground-truth for the separate-file layout fine-tune re-quants use, using
signals the deriver already has in hand (no extra fetch). Exposed as an
additive `has_mtp_head` on der.profile; _swap_path sets
`drop_spec_config = not has_mtp_head`. An embedded-head repo (head baked
into the shards with no named file) still falls back to drop — conservative,
and the named-file layout is the norm.
Validated on real HF metadata: josefprusa/ThinkingCap AutoRound →
has_mtp_head=True (keep --speculative-config); a plain AWQ re-quant →
False (drop). test-pull-swap replaces its tautology check with a real
detect_mtp_head unit test (declares+file, plain, nested text_config,
declares-but-no-file). Pull-gate suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The ⚠️-caveat required a clean v0.24.0 4-card fp8/e4m3-KV report to
upgrade to ✅. Two landed:
- #584 (@ryanmpelletier, 4× 3090 x16, Proxmox): attention backend
FLASHINFER (the #595 int8-PTH→fp8/e4m3 flip live at TP=4), verify-full
9/9, verify-stress clean to 240,634 tok, soak-continuous PASS, bench
n=5, and the full 8-pack at 111/150 — ties the 2-card dual-max proxy
(109/150), so max-tier quality is TP-invariant (measured, not assumed).
- #625 (@MoppelMat, 4× 3090 bare metal, mixed x4/x8, 300 W): a second
independent 4-card rig — decode 79/102, prefill-90K clean, soak PASS.
Decode beats ryan's x16 rig (bare metal); prefill trails ~30% (narrow
lanes are the bandwidth-bound bottleneck; decode is lane-insensitive
at TP=4) — a clean decode(compute)/prefill(interconnect) split.
Changes:
- compose header: Status ⚠️ → ✅, drop the Caveats block, Quality now
cites the 4-card 111/150, prose updated to the promotion evidence.
- registry: status caveats → production + refreshed status_note.
- baselines.yml: replace @Whamp's doubly-stale #446 int8-PTH row (v0.22.0)
with ryan's fresh v0.24.0 fp8-KV submission (TPS + 8-pack + ctx + soak);
MoppelMat noted as the corroborating 2nd rig. Emit computes stale=false.
- BENCHMARKS.md: append both v0.24.0 4-card rows (append-only history).
- MULTI_CARD.md: drop the "no 4-card bench yet / needs TP=4 confirmation"
prose now that both reports have landed.
Guards green: test-baselines, test-compose-status-drift,
test-{switch,launch}-registry-parity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
@guybrush01's 5090 (#612) independently reproduces @paulp83's #619 to
within noise (decode 257.3/258.0 vs 255.8/257.9) AND validates deeper —
verify-stress needle-clean to 120K (91% of 131K) vs #619's 29K, at the
full 131K (no derate; the 35B MoE's tiny KV holds on a 32 GB 5090).
Two independent 5090s agreeing is stronger evidence than one, so the
1x5090-pcie submission now carries the deeper 120K ctx_validated and
credits both submitters/sources. TPS unchanged (identical within noise).
8-pack quality still owed on both (neither built the benchlocal sandboxes).
Responded on #612 (validation) + #617 (paulp83's 27B — a DIFFERENT failure
from #613: engine never loaded / 525 MiB container, MTP + 28 GB RAM, not the
#613 post-boot VRAM OOM). learnings updated (35b + the 27b two-failure note).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
@paulp83's single-5090 run is the first time the NVFP4 tier has booted
anywhere. Boots clean (vLLM v0.24.0, quant=modelopt_mixed, fp8 KV @131K,
MTP off as shipped), verify-full 9/9, verify-stress needle-clean (9.8K +
29K), soak-continuous PASS (15 MiB growth / 100% retention / 0 err / p50
311). Decode 255.8 narr / 257.9 code TPS @ 60 ms TTFT — ~2.5-3x our 2x3090
AutoRound tier (native FP4 GEMM). VRAM 30.6/32 GB @131K — tight but flat.
- registry: de-blind the status_note (drop "never booted" → first 5090
validation with numbers); stays 🧪 (8-pack quality still owed — that run's
benchlocal sandboxes weren't built).
- baselines.yml: first NVFP4 row — a 1x5090-pcie submission (narr/code TPS,
clean@29K, soak PASS, engine v0.24.0, tier submitted); surfaces ⑂-labelled
in c3 like the multi-max submission. No primary row (we can't run it).
- compose header: AUTHORED BLIND block → FIRST COMMUNITY VALIDATION with the
gate results + the 98304 fallback note.
- BENCHMARKS.md: the first NVFP4 row in the 35B-A3B section.
Confirms the two design calls: MTP-off (speculative_config=None) and fp8 KV
at scale=1.0 (declared-not-shipped scales) both loaded correctly on the
modelopt path. learnings + memory updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Per maintainer call: "cluster" conventionally means multiple networked
machines (a non-goal here — LiteLLM fronts multi-host) and is best reserved
for a future enterprise/multi-node meaning. "Pod" is the accurate analogy
for what this is — one model on a GPU subset on ONE host (k8s/RunPod sense).
The capability is unchanged; only the name.
Scoped rename (cluster→pod, case-aware) across the pod feature ONLY:
- scripts/cluster.sh → scripts/pod.sh; test-cluster-cli.sh → test-pod-cli.sh;
docs/CLUSTERS.md → docs/PODS.md
- estate_cli.py verbs + wording; app.py (ClusterCreateScreen→PodCreateScreen,
action_new_cluster→new_pod, _populate_clusters→_populate_pods, #cluster-view
→#pod-view, the [N] help/empty-state text); services.py cluster_create_plan
→pod_create_plan; data.py kind cluster_create→pod_create; tests + doc
pointers (HARDWARE/MULTI_CARD/README/c3-README)
- UNTOUCHED (unrelated "cluster"): compat.py + test-profiles-compat.sh (the
VRAM-topology classifier), services.py:1863 / test_services.py (the scene-
table "cluster by group" verb), older docs, .venv
Also folds in the [N] discoverability fix (n was already bound to
serving_switch — moved to N; empty-estate now shows a "no pods — [N] new
pod" affordance + a help entry) and a heterogeneous-rig section in PODS.md:
one homogeneous pod per card family (2×3090 · GB10 · 6000 Pro) is the clean
pattern — mixing families in one TP pod makes NCCL wait on the slowest +
wastes VRAM; a worked 2-pod lifecycle walkthrough.
Verified: test-pod-cli + estate/gpu/profiles guards green; 25 c3 pod/binding
+ 239 fast tests green; pod.sh live create/list/D1-reject on 2×3090; zero
stray "cluster" in pod files (scene-verb preserved); no CLUSTERS.md links
left; PODS.md leak-clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The cockpit surface for multi-model clusters: the Operate/Orch pane now
groups estate instances (clusters) with their GPUs stacked and a placement
health badge fed by the Phase-A assertion.
- estate_cli.py D3 (#610 addendum 3): report-state's active_estate.instances
now carries a per-instance placement verdict {requested, actual,
placement} (only probed for RUNNING instances). ONE source feeds both
cluster.sh status and the cockpit — no second poll.
- app.py C1: _populate_clusters renders a #cluster-view block below the
serving line — per cluster a header (● name · slug · :port · badge) with
GPUs stacked beneath (└─ GPU0 GPU1) + a trailing free-GPU line. The badge
is the Phase-A verdict: ✓ placed / ⚠ PLACEMENT MISMATCH, so the view shows
where clusters ACTUALLY landed (the failure mode that opened #610). Hidden
when the estate declares no clusters — the single-model case is unaffected.
- test-cluster-view headless: grouping + GPU stacking + ✓/⚠ badges + free
line; empty-when-no-clusters. Serving-panel siblings unaffected.
LIVE-VERIFIED (2x3090): cluster.sh up chat → report-state carries
placement=ok → C1 renders ✓ placed. 8 headless + 239 fast c3 + estate
guards green.
Design: /opt/ai/docs/serve-cockpit-design.md §10 (C1 shipped; C2 wizard is
the remaining Phase-C piece — the CLI wizard covers the capability today).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The CLI seam the c3 cluster UX (Phase C) will consume, and a full headless/
SSH cluster capability on its own. A "cluster" = a named model on a chosen
GPU set + port (an estate instance).
- estate_cli.py gains create / list / status / rm verbs (D2: NEW verbs go
IN estate_cli.py, which owns the schema + validate_estate + boot/down —
ONE validation path shared with hand-written estate files and the wizard).
up/down reuse `boot`/`down --only <name>`.
- create runs the D1 fit-vs-set logic (#610 addendum 3): kv-calc --card is
single-card + registry TP, so count != compose TP is a HARD REJECT, a
heterogeneous set is estimated against its min-VRAM card (+ note), and the
whole set is re-validated (validate_estate: GPU collision, port collision,
per-instance fits) before append. GPU indices stay index-based in the
estate file; UUIDs resolve at boot (Phase A).
- status carries the {requested, actual, placement} verdict per cluster
(the shape the c3 badge reads); list/status take --json.
- scripts/cluster.sh: the ergonomic bash front (create/list/status/up/down/
rm), a thin wrapper over estate_cli.py.
- test-cluster-cli.sh (NEW): full lifecycle hardware-free via CLUB3090_FAKE_
GPUS — create/D1-reject/collision-reject/list/status/rm.
LIVE-VERIFIED (2x3090): create chat (fit-clean ~20.2 GiB) → up → estate boot
→ "✓ placement verified" → status "serving ✓ placement=ok", GPU0 idle /
GPU1 loaded → down → rm. D1 count!=TP + GPU-collision rejects confirmed.
Part of #610 (Phase A′). Next: Phase C (c3 cluster view + wizard).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Extends the #611 primitive to the estate (multi-cluster) path and adds a
post-boot placement check — the foundation Phase A′ (cluster.sh) and C
(c3 cluster UX) sit on.
- scripts/lib/gpu-select.sh (NEW, shared lib): factors the #611 inline
resolver out of launch.sh into gpu_select_indices_to_uuids /
gpu_select_export, plus gpu_select_container_uuids +
gpu_select_assert_placement. launch.sh sources it (no re-inlined drift)
and runs the placement assertion after verify-full.
- estate_cli.py: resolve_gpu_uuids() (python twin of the bash resolver) +
compose_env / compose_override_doc now UUID-pin CUDA_/NVIDIA_VISIBLE_
DEVICES (ESTATE_GPUS stays index-based — the compose device_ids read the
host view). assert_placement[_quiet]() runs after each instance is ready
(sequential → stderr, parallel → per-instance boot log), returning the
{requested, actual, placement: ok|mismatch|unknown} verdict — the shape
cluster.sh status + the c3 badge will read.
- Placement uses --query-compute-apps=gpu_uuid (NOT --query-gpu): under CDI
the container sees all cards but RUNS on the CUDA-masked set, so
compute-apps is the runtime-agnostic ground truth.
- test-gpu-select (NEW): asserts the bash + python resolvers agree on real
hardware and both fall back identically; test-compose-gpu-mask-passthrough
updated to follow the resolver into the lib.
LIVE-VERIFIED (2x3090, classic runtime): `launch.sh --gpus 1` → UUID-pinned,
verify-full 8/8, "✓ placement verified", host GPU0 = 1 MiB / GPU1 = 20.8 GB.
estate compose_env confirmed UUID-izing while ESTATE_GPUS stays index-based.
CDI leg still awaits mog. Relevant guard sweep (11) green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
--gpus selection was NVIDIA_VISIBLE_DEVICES-only, which CDI runtimes
(NixOS hardware.nvidia-container-toolkit, nvidia-ctk cdi) IGNORE — mog's
3x3090 rig got all cards exposed and vLLM took the wrong two. Index-based
CUDA_VISIBLE_DEVICES can't fix it either: the classic runtime renumbers
the exposed set in-container.
UUIDs solve both at once:
- launch.sh --gpus N,M resolves indices -> GPU UUIDs (nvidia-smi -i) and
exports BOTH NVIDIA_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES as UUIDs.
Classic runtime: exposes-by-UUID; CUDA mask agrees (renumbering-proof).
CDI: exposure comes from device_ids (nvidia.com/gpu=all) and the CUDA
UUID mask is what pins the cards. Index fallback if the query fails.
- 26 composes gain a bare `- CUDA_VISIBLE_DEVICES` passthrough (unset ->
absent -> zero change for existing users); stale commented hints removed.
- preflight: selector consumers are host-index-based -> new
_preflight_selector_normalize maps GPU-xxxx tokens back to indices at
the single choke point (the first live boot caught this: preflight
rejected the UUID selector).
- new guard test-compose-gpu-mask-passthrough.sh: every compose with the
NVIDIA env line must pass CUDA_VISIBLE_DEVICES through + launch.sh must
keep the UUID resolution.
- docs/HARDWARE.md: "Pinning specific GPUs (and CDI / NixOS runtimes)" —
the runtime table, the CDI deploy-block swap, the in-container
renumbering gotcha, manual recipes.
LIVE-VERIFIED on this rig (classic runtime, 2x3090):
launch.sh --variant vllm/minimal --gpus 1
-> host GPU0 = 1 MiB (empty), GPU1 = 20.7 GB (model), both env vars
UUID-set in-container. CDI leg validation goes to mog (NixOS reporter).
Follow-up (tracked on #610): estate ESTATE_GPUS UUID-ization for CDI
multi-instance rigs; post-boot visible-GPU-count sanity check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Verification pass on the blind-authored claims (vLLM source + safetensors
indexes, 2026-07-06):
- vLLM modelopt SUPPORTS fp8 KV with NVFP4/mixed-precision weights by
design: KV_CACHE_QUANT_ALGOS = ['FP8','NVFP4'] + ModelOptKVCacheMethod;
our --quantization modelopt + --kv-cache-dtype fp8 pairing is canonical.
- BOTH nvidia checkpoints DECLARE kv_cache_quant_algo=FP8 in
hf_quant_config.json, but NEITHER ships k_scale/v_scale tensors (indexes
enumerate only linear input_scale/weight_scale/weight_scale_2). The
shipped composes claimed '27B bakes KV scales' and 'MoE has none, unlike
the 27B' — both wrong; the two checkpoints are IDENTICAL on KV: declared
FP8, no scales, scale=1.0 at runtime (calculate_kv_scales disabled on
Qwen3-Next hybrid) — exactly the #594-quality-tied regime.
Text-only; no flag changes (the composes were already serving the right
config — the rationale strings were wrong).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
First slice-3e quality ingest: @guybrush01's on-rig benchlocal --full
(2026-07-06, #571) lands 109/150 on vllm/qwen-27b-dual-max @ 2x5090 —
exact total parity with the 2x3090 primary's 109/150 (#594): Blackwell
native FP8 GEMM and Ampere Marlin W8A16 score the same 8-pack. n=2
on-rig runs (102 -> 109) both within the ±5-7 noise band; row carries
the better-evidenced latest run with per-pack detail + source link in
quality_env.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
nvidia/Qwen3.6-35B-A3B-NVFP4 (modelopt MIXED_PRECISION MoE: NVFP4 gs16
expert FFNs + FP8-static attention + unquantized mtp.* head, 23.4 GB, 3B
active) — sibling of the 27B nvfp4 pair (#603), same authored-blind /
required_sm=9.0 / community-validated pattern. The single slug is the
GB10/DGX-Spark marquee: a 3B-active MoE is the right shape for
big-capacity, lower-bandwidth unified-memory parts — kv-calc says full
262K at 25% of a GB10's budget (MoE KV is tiny, ~1.3 GB @131K).
Two deliberate deltas vs the 27B nvfp4 composes:
- NO MTP (drafter=None): our measured finding on this MoE — the built-in
head shares the MoE forward and is net-negative (-51%); headers forbid
blind re-adding.
- fp8/e4m3 KV at scale=1.0: this checkpoint has NO baked KV scales
(kv_cache_scheme null, unlike the 27B) — same regime the production 27B
fp8 tier quality-tied at (#594); flagged as a known-unknown.
- vllm/qwen-35b-a3b-single-nvfp4 (:8078, 🧪) — 131K default for 5090
32 GB (kv-calc 27.4/29.4 @93%); GB10 raises to 262K via MAX_MODEL_LEN.
- vllm/qwen-35b-a3b-dual-nvfp4 (:8079, 🧪) — TP=2 @262K, 15.7 GB/card
(53%) on 2x 5090.
Wiring mirrors #603: weights entry + weights.py aliases + kv-calc
(weights_nvfp4_gb on the MoE spec, nvfp4 branch, cfg map, aliases) +
registry entries (kvcalc-priced, no DEFAULTS rows) + counts 61/62. Full
scripts/tests suite + kv-calc --calibration green; sm_86 → incompatible-hw
(auto-hidden in c3 per #604), rtx-5090 → fits-clean 27.4/15.7 GB.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The metadata version is frozen at install time and benchlocal fixes are
pushed without bumping it, so "0.9.4" cannot identify the running code
(live proof on the dev rig: metadata 0.9.4, checkout actually at
v0.9.7-6-gc541550). pip records the truth in direct_url.json:
- report.sh: the Quality-tooling line now appends `source:` —
`git@<sha9>` for pip-from-git installs (vcs_info.commit_id), or
`git describe --tags --always --dirty` for editable checkouts (the
checkout PATH is withheld from the public report).
- quality-test.sh staleness reference: editable installs update via
`git pull` WITHOUT rewriting the console script, so the mtime
under-reports "CLI last updated" — now max(script mtime, editable
checkout last-commit time). Caught a real case on the dev rig: the
cli/hermes sandboxes were built ~1 h before the same day's last
benchlocal commit. Stale-warning timestamps now carry %H:%M so
same-day boundaries read sensibly.
test-quality-baseline / test-quality-thinking / test-report-calib green;
report section verified live + leak-clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Community rigs hit two silent quality-test failure classes: (1) runs
"skipping" the sandboxed packs because the Docker images were never built
(the #492 warning exists but --sandboxed-only still warn-and-ran a
guaranteed-useless run), and (2) rigs that pulled a benchlocal-cli update
kept scoring on OLD sandbox images until told to rebuild manually — nothing
detected the drift. And report.sh collected none of this, so triage had to
ask.
quality-test.sh:
- --sandboxed-only + missing images/Docker → HARD FAIL up front with the
build instructions (a run with zero runnable packs is never intended).
--full keeps warn-and-continue (deterministic packs are still useful).
- NEW staleness heuristic: each sandbox image's Created is compared against
the benchlocal-cli console-script mtime (rewritten on every (re)install —
portable across pip-from-git AND editable-checkout installs). Image older
than CLI → warning naming the image + both dates + the rebuild command.
Explicitly labelled heuristic (an unrelated reinstall trips it) → WARN,
never abort.
report.sh:
- New "Quality tooling (benchlocal-cli + sandboxes)" section: CLI path /
version (via the console-script's own interpreter) / install date, the 4
sandbox images with build dates + per-image OLDER-than-CLI flags + rebuild
hint, and the newest results/quality/*.json. All best-effort; piped
through redact (verified leak-clean live).
Exact-version staleness (image labels + a `sandbox-status` command) needs
upstream benchlocal-cli changes — tracked in the todo.
Verified live: hard-fail branch (docker stub), warn-continue branch,
staleness branch (fresh CLI vs real images), report section rendering.
test-quality-baseline / test-quality-thinking / test-report-calib green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
On rigs whose card can't run a slug's kernels (registry required_sm above
the local SM — e.g. the NVFP4 slugs on this sm_86 rig), the catalog showed
the slugs normally and the Download card pitched a 20 GB fetch that could
never boot. Now:
- kv-calc fit verdict gains the arch floor: fit_verdict()/fit_all_verdicts()
return {"verdict": "incompatible-hw", required_sm, card_sm, error} when
the card's SM (derived from the hardware profiles — same source as compat
C3) is below the slug's required_sm. Bare-number --card carries no arch
info → gate skipped (permissive). Applies to kvcalc-SKIP slugs too.
- Catalog: incompatible-hw rows share the DEPRECATED [h] bucket — hidden by
default, revealed by [h]; status line counts them separately
("+N incompatible-hw hidden — h"). The verdict lands with async fit
enrichment, so rows fold away on the enrichment refresh.
- Download confirm card leads with "⊘ no compatible hardware detected —
requires sm ≥ X (Hopper/Blackwell); this rig's card is sm_Y. It will NOT
boot on this machine." BEFORE the size/disk pitch — download still
proceeds (staging for another rig is legitimate). Serve card gets the
same warning line under fit.
- FitVerdict: incompatible-hw verdict + ⊘ glyph + required_sm/card_sm.
Also updates test-registry-json's VARIANT_KEYS for the #600 emit fields
(kv_format / weights_format / weights_quant_label) — the contract gate
only surfaced post-merge.
Tests: kv-calc-fit section (f) — 3090→incompatible-hw w/ sm fields,
5090→fits-clean, bare-number permissive, fit-all parity; 2 new headless
(hide+reveal+count, download-card warning). Full scripts/tests suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
