Agent-facing version of the eval guidance shipped user-facing in
ANNOUNCEMENT_TEMPLATE.md §7 (da063d7): behavioral quality (8-pack, both
reasoning modes, benchlocal + sandbox one-time setup, REASONING=on for
the thinking-ON leg) + operational health (report.sh --full), the
non-overlap guarantee, and the don't-pair-rebench-full-with-report-full
rule with the pick-by-goal split.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* concurrency-probe: surface aggregate tok/s in verdict, RESULT, and sweep
The probe already computed aggregate throughput (summed completion
tokens / wall) per round but only printed it in the round table — the
verdict, RESULT line, and sweep knee reported per-stream only. Users
asking "what total throughput at N agents?" (the Bruno question) had
no first-party number to point at.
- verdict: aggregate tok/s (N streams) next to per-stream
- RESULT: agg_tps= field (machine-readable, sweep-parseable)
- SWEEP: per-N summary table (per-stream vs aggregate) + aggregate in
the knee line
Knee logic unchanged (still per-stream-floor based). Offline guard
test-concurrency-probe passes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* concurrency-probe: re-resolve served model after each SWEEP boot
The top-of-script MODEL autodetect runs before SWEEP boots anything on
$URL, so it silently falls back to the default (qwen3.6-27b) — and every
request against any other model 404s, reading as errors=N per round (hit
live sweeping vllm/qwen-35b-a3b-dual on :8051; the 27B sweep only worked
because the fallback equals its served name). Re-resolve from /v1/models
after each boot; an explicit caller MODEL= pin is respected (MODEL_PINNED).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* FAQ + BENCHMARKS: first-party concurrency/aggregate-throughput matrix
Measured via the extended concurrency-probe (this PR) on 2x3090:
- dense 27B dual-fast: batching knee N=2 (~104 decode-agg @16K ctx),
collapses to ~54 by N=8; generation shape 211 agg @ N=8
- 35B-A3B MoE dual: ~250-270 decode-agg FLAT N=2..16 @16K ctx;
generation shape 1,037 tok/s aggregate @ N=16 (92.8/stream, clean)
New FAQ entry: how to serve multiple coding agents (MAX_NUM_SEQS,
ctx-vs-concurrency, model pick by architecture, honest aggregate-vs-
per-stream caveats, prefill-bound note, probe invocation).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@seanyourhighness diagnosed a hard HTTP 500 at turn 2 benching Tess-4-27B:
tool_choice=required + max_tokens=150 truncates the tool-call JSON on a
reasoning model (finish=length), and the harness replays that unterminated
`arguments` string in `messages` — llama-server then throws in
common/chat.cpp func_args_not_string() re-parsing the client history, so a
turn-1 clip poisons every subsequent turn.
Two-part fix:
- max_tokens 150 -> 600 (600 completes the call per the report).
- _safe_args guard: validate each accumulated tool_call `arguments` parses as
JSON before replaying it in `messages`; substitute "{}" if it doesn't, so a
single clip can't kill the whole ramp even if a slower model still truncates.
The upstream half (malformed client-supplied arguments -> HTTP 500 instead of
4xx, template-independent) is being filed against llama.cpp separately.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two independent RTX 5090 validations agree within noise:
- #619 (@paulp83): first NVFP4 boot on Blackwell — verify-full 9/9,
verify-stress needle-clean, soak-continuous PASS.
- #612 + #652 (@guybrush01): verify-full 9/9, verify-stress to 120K
(91% of 131K), soak PASS, p50 ~311 TPS. (#652 re-confirms #612 within
noise — one rig confirmed twice.)
status experimental -> caveats + compose header Status/Caveats. Caveat =
8-pack quality NOT yet cross-rig-measured (sandboxes weren't built on
these runs; quality run pending, gated on #492) — reverts to experimental
if a quality run regresses. NVFP4 stays Blackwell/Hopper-only (sm_90+).
Unchanged: 27b-single-nvfp4 (shipped util OOMs a 5090 — #613/#617, needs a
config fix) and both dual-nvfp4 slugs (no reports) remain experimental.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Standardize a reproduce/contribute section on model announcements:
build the benchlocal sandboxes, run quality-test.sh (8-pack, both
thinking) + report.sh --full (operational) — the two non-overlapping
passes. Bumps What'd help → §8, Credits → §9. First used on the
Tess-4-27B announcement (Discussions #662).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The `-h|--help` case ran `sed -n '2,55p' "$0"`, but the script cd's to the
repo root before arg-parsing — so a relative-path invocation from another
cwd left `$0` unresolvable and (under set -e) `--help` printed nothing. It
also dumped raw #-comment lines and truncated at line 55, omitting the
endpoint-first + env-override sections.
Replace with a `usage()` heredoc: cwd-independent, complete (all flags + env
overrides + examples), cleanly formatted. `-h` and `--help` both print it.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
n-sweep 2026-07-09 (dual 262K, server-side decode tok/s + draft
acceptance, 3 runs/arm after warm): n=2 peaks at 57.9 tok/s / 0.62
accept; throughput falls monotonically past it (34.9 / 0.27 @ n=6) as
Q4 draft acceptance craters. Confirms the shipped MTP_DRAFT_N_MAX=2
default (inherited from Deckard, now Tess-validated). Comment/doc note
only — no config change; full table in the per-model learnings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Pressing [D] on a brought GGUF repo ran pull.sh, the SAFETENSORS path — it aborts unsupported-format (no config.json) on a GGUF repo, so nothing downloaded ("download did not complete"). Route-G already handled fit + serve; download was the last gap.
- run_bring_download(gguf_includes=[...]): fetch those files directly into the pull dir via hf download --local-dir <pull> --include <pat>... with the hf-download.sh guards (HF_HUB_DISABLE_XET=1 resumable, HF_TOKEN from token file). Else pull.sh unchanged.
- app._gguf_download_includes(): picked quant file(s) + vision mmproj + quant-matched external MTP drafter glob. Wired into run_bring_download_worker.
Validated: mock-runner confirms the hf-download cmd vs pull.sh unchanged; a real hf download --include of the bartowski mmproj completed exit 0. +2 services tests, 232 green.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Route-G only wired spec-decode when a SEPARATE mtp-*.gguf drafter was present
(migtissera-style). A GGUF with an EMBEDDED MTP head (nextn baked into the main
quant — bartowski/unsloth-style) got the sibling's --spec-type stripped and
nothing re-added, so it served as a base model with no spec-decode.
- gguf_has_embedded_mtp(path): a minimal stdlib GGUF-metadata parser (the gguf
package isn't in the c3 venv) reads <arch>.nextn_predict_layers; >=1 = embedded
MTP. Early-returns before the big tokenizer arrays; never raises.
- _rewrite_gguf_command(embedded_mtp=...): external drafter → --spec-draft-model
+ --spec-type draft-mtp (unchanged, wins if both); else embedded → --spec-type
draft-mtp with NO draft model (activates the nextn head).
- run_gguf_emit_and_serve computes embedded_mtp on the main gguf when no external
drafter was found.
Live-validated on the embedded-MTP unsloth GGUF: llama.cpp logs "creating MTP
draft context against the target model" (self-speculation, no separate file),
"adding speculative implementation 'draft-mtp'", serves ("2+2" -> "Four").
Distinct from #654's external case ("loading draft model .../mtp-Tess..."). +5
tests (parser via synthetic mini-GGUF + both wiring branches). 230 services +
17 phase-4 green.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Clicking Serve for a brought GGUF (route-G) died with 'No module named
scripts': c3 runs from tools/serve-cockpit/, so the repo root isn't on sys.path,
yet emit_gguf_compose (and swap_apply for route-C) do
`from scripts.lib.profiles.compose_registry import …`. Only ONE of the three
call sites guarded this with a sys.path.insert; the emit site surfaced the
ImportError to the user, and fit-check's topology detection silently degraded.
Fix once in CockpitData.__init__ (add repo_root to sys.path) so every call site
works. Validated by simulating c3's runtime (repo root NOT pre-added): scripts
is unimportable before init, importable after, emit returns no error. This is
the exact condition #654's validation harness masked by inserting the path.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The emit-swap group asserted PRE-P2b behavior and crashed on any rig (not env-
specific): P2b (swap_apply served-name + SPEC env-gating) now emits
--served-model-name ${SERVED_NAME:-<name>} and LIFTS --speculative-config out of
the command into a ${SPEC:-on}-gated entrypoint (${DRAFTER_METHOD:-mtp}). The
test still checked the plain served-name + spec-config-in-command, so _val()
returned None on the moved flag -> 'x' in None -> TypeError crash. Updated 3
assertions to P2b's shape. test-pull-swap.sh now green (also unblocks the route-C
location asserts added in #655).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Companion to #654 (route-G): the safetensors weight-swap path had the same
"compose in the project tree" problem the user flagged — swap_apply.emit_swap_
compose wrote _brought-<san>.yml NEXT TO the sibling compose (so its relative
../ mounts resolved), dropping throwaway files beside catalog composes.
Route-C's MOUNT was already fine (it dir-mounts the brought weights to a distinct
/brought-model), so this is location-only: absolutize every sibling volume via a
new _absolutize_volume (${VAR:-../rel} → ${VAR:-/abs}, plain ../rel → /abs; the
vLLM sibling has 5 relative mounts — HF cache, torch_compile, triton, the
froggeric chat template, detect_nvlink) so the compose is relocatable, then write
it to the runtime dir beside the pull dir (<MODEL_DIR>/.cache/huggingface/
club3090/composes/), falling back beside the sibling.
Validated at emit level: all relative sources absolutized, ${MODEL_DIR} env var
preserved, brought mount intact, compose lands in the runtime dir not the repo.
+2 assertions in test-pull-swap.sh's emit group. (That .sh has a PRE-EXISTING
env failure on this rig — identical on master, unrelated.)
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First live boot of route-G (#650, maintainer-gated) surfaced two bugs the
compile-only tests couldn't:
- BOOT FAILURE (ExitCode 128): emit_gguf_compose bind-mounted the 3 brought
GGUFs as individual files INTO /models/brought.* — but the sibling already
mounts MODEL_DIR → /models, so OCI can't create those mountpoints inside an
already-mounted /models ("read-only file system"). Fix: the brought files live
UNDER MODEL_DIR (the pull dir), so address each at /models/<realpath-relative-
to-MODEL_DIR> (realpath follows the pull-dir symlink) and mount MODEL_DIR ONCE.
Files outside MODEL_DIR get their own /brought/<tag>-<name> mount (never /models).
A /models mount is guaranteed even if a sibling lacks one.
- WRONG LOCATION: the compose was written next to the sibling in the project
tree (to inherit relative mounts). Now the volume is absolute (MODEL_DIR →
/models, dropping the ${MODEL_DIR:-../rel} fallback) so the compose is
relocatable — written to a runtime dir on the model disk
(MODEL_DIR/.cache/huggingface/club3090/composes/), not the repo.
Live-validated end-to-end on the c3-EMITTED compose: Tess-4-27B-GGUF loads
(main + mmproj vision + the brought MTP drafter — spec-decode engages) and
serves ("capital of France" -> "Paris"). Tests: 3 new (single-mount, runtime-
dir, outside-MODEL_DIR) + 5 existing emit tests updated to /models/<rel>. 13
phase-4 + 229 services green.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a 'Where weights land' section: MODEL_DIR is the single root; curated
catalog models live top-level (compose-referenced), BYO pulls derive to
.cache/huggingface/club3090/pulls/<repo-slug>/ — the SAME path pull.sh and the
c3 Bring [D] button compute, so a pull by either is discovered by both. Makes
the two intentional tiers explicit for users bringing their own weights.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Live dogfood 2026-07-09 (migtissera/Tess-4-27B, 54G bf16 on a 48G rig): the §2b
topology floor correctly hid every slug (single ≤21.6G, dual ≤43.2G, no
multi-card slug), but the funnel showed only the ✎ custom-slug sentinel with no
explanation — and the custom Input never appeared. Two cascading bugs:
- Bug A: when the size floor empties the recommended list, surface an honest
verdict — "won't fit" (compat slugs exist but all exceed VRAM; bring a smaller
quant) vs "no catalog recipe matches this artifact" (no compatible engine at
all) — instead of a silent empty dropdown.
- Bug B: with only the sentinel option, _set_select_options pre-selects it under
prevent(Select.Changed), so the Changed-gated custom-Input reveal never fires
and re-picking the sole sentinel is a no-op. reveal_slug_stage now reveals the
Input eagerly iff the sentinel is the sole option (kept hidden when real slugs
exist).
Neither is a regression — the floor (#582) + reveal are byte-identical to
pre-Grok; a 55G repo is the first to expose the all-hidden edge. Tests:
test_uiux_funnel_oversized.py (custom-hatch reachable + won't-fit verdict + no
false verdict at normal size). 254 c3 tests green.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three review nits from the phase 1/3 merges, all copy/label only — no logic
change (245 c3 tests green):
- action_serve_untested: the user-facing dead-end toast said "the bring-your-own
weight-swap is a pending follow-up" — false since #644 wired apply-swap. Now
states the real reason (fit-check resolved no sibling recipe or catalog profile
to serve) and points back to ① Bring.
- Its docstring still claimed "this does NOT serve the brought model's weights ...
DEFERRED follow-up," contradicting its own now-wired Route-C body. Rewritten to
the two actual modes (brought-weights serve vs untested reproduction).
- Disk-fit preflight cards labeled sizes "GiB" while the math is decimal GB
(size_gb * 1e9 / free / 1e9). Relabeled the disk-fit size/free/need to "GB"
(both the catalog and bring preflight cards). VRAM "GiB" labels left as-is —
those are genuinely binary.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
resolve_hf_home was `--hf-home > $HF_HOME > $XDG > ~/.cache`, ignoring MODEL_DIR.
So a bare `pull.sh <repo>` (or a nohup apply-swap) run with only .env's MODEL_DIR
set and no explicit HF_HOME silently fell to ~/.cache on the ROOT disk — the
footgun that misplaced a brought model's 35 GB of weights off the models volume
(and would bite users the same way). Insert a MODEL_DIR step (env, else parsed
from the repo .env, encoding=utf-8) between $HF_HOME and $XDG, matching what
switch.sh/launch.sh/c3 already resolve. c3 is unaffected — it sets HF_HOME
explicitly, which still wins. New guard test-hf-home-resolve (PASS 1-5);
test-pull / test-pullgate-download / test-download-lock still green.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
select_weight_files built the sharded set as `"-of-" in n or startswith
"model-"`, so a grafted MTP/nextn head (e.g. `mtp_grafted.safetensors`) — a
real weight the model needs with MTP enabled — was dropped, even though
detect_mtp_head/_has_mtp_weight_file already see it. Live dogfood: Tess-4-27B-FP8
downloaded its 7 model-* shards but silently omitted `mtp_grafted.safetensors`,
which would break MTP serving (club-3090 #617). Union any non-shard mtp/nextn
*.safetensors head into the sharded download set. Guard: test-pullgate-download
gains an MTP-head fixture.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* c3+pull: detect in-progress downloads + per-repo lock (#617)
Repeated ① Bring [D] presses spawned N concurrent `hf download` runs racing
into the same `.incomplete` staging dir (5-deep live), each rmtree-ing and
re-fetching the others' partial. Two disk-truth gaps behind that:
1. No dedup. `download_model` now acquires an atomic per-repo lock
(`mkdir <pull_dir>/.download.lock` + a `pid` file = holder PID + UTC start)
BEFORE the staging rmtree. A 2nd concurrent call for the same slug is
REFUSED with `failure="in-progress"` (not a race); swap_apply surfaces it
as `in_progress`, and `pull.sh --apply-swap` exits **rc=3** ("already
downloading", distinct from rc=1 failure). A STALE lock (dead holder — a
crashed/SIGKILL'd download that skipped `finally`) is reclaimed on the next
call, so a leaked lock self-heals — more robust than a signal trap (a
SIGKILL skips traps too). Released in `finally` on every return path.
2. No in-progress detection. Presence probes only saw the FINAL dir (empty
until hf moves files out of `.incomplete`), so a mid-flight download read as
absent → the false "download did not complete". `services.bring_download_in_
progress` now reads the lock (live PID) + `.incomplete` bytes for a
best-effort pct; c3's fit-check render + [D] no-op guard + [k] cancel are
disk-aware, so a running download is REFLECTED (and not duplicated) even
across a c3 restart or a bare `pull.sh` started outside the session — the
in-memory tracker (#643) couldn't see those.
download_model is the shared chokepoint (bring apply-swap AND catalog fetch),
so both paths get the lock; the body moved verbatim to `_download_model_impl`
behind the locking wrapper — no behaviour change beyond the added guard.
Tests: scripts/tests/test-download-lock.sh (live-refuse + stale-reclaim +
release); c3 test_services (in-progress live/stale/absent + pct) +
test_app_headless (disk-detected → downloading render, [D] no-op, [k] kill).
245 services/registry + 131 app blast-radius tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* downloader: refuse a fresh pidless lock (close acquire-window race)
The lock acquires with mkdir then writes its pid file a beat later. A 2nd
caller landing in that window saw the lock dir but no pid → read_active
returned None → the wrapper treated it as stale and rmtree'd + reclaimed it,
letting BOTH callers proceed — the exact duplicate-download race the lock
exists to prevent. Now: a pid-present-but-dead lock still reclaims (unchanged),
but a PIDLESS lock only reclaims once it's older than a 10s grace — a fresh
pidless lock is a holder mid-acquire and is refused, not stolen. Guard adds
PASS 4 (fresh pidless → refuse) + PASS 5 (old pidless → reclaim).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ① Bring [D] download was a stripped-down sibling of the catalog
Download: a single await + trusting the captured `[apply-swap] compose:`
marker, with no in-flight tracker and no cancel. Two dogfood bugs (#617):
1. "apply-swap did not complete" fired while the download was still
running — the verdict raced the marker instead of re-stating disk.
2. A re-run fit-check was blind to an in-flight download, re-offering [D]
and inviting a second 20+ GB fetch of the same repo.
Mirror the catalog run_download machinery for the bring path:
- repo-keyed `_active_bring_download` tracker, registered synchronously
in action_bring_download (no-op guard: a second [D] for a repo already
downloading just notifies).
- run_bring_download_worker polls to TRUE completion, then derives the
verdict from an AUTHORITATIVE `bring_weights_present` re-stat. Route-C
weights that landed but whose marker was missed now read as ✓ → ② Serve
(apply-swap --emit-only, no re-download), NOT a false failure.
- run_byo_check + _byo_result_text are downloading-aware: an in-flight
repo renders "⏳ downloading… [k] cancels" and suppresses the [D]
re-offer.
- [k] bring_cancel_download kills the pull.sh/hf process group via the
runner's SIGINT→TERM→KILL (the bring path had no cancel at all); shares
"k" with serving_stop, gated disjointly in check_action.
Scope: bring path only — catalog Download, the runner, and scripts/
untouched. c3 pytest: 808 passed (2 new: the downloading card render +
the tracker/guard/cancel round-trip).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* baselines: backfill dual-max quality_8pk_think_on (110/150)
The vllm/qwen-27b-dual-max row logged only the thinking-off 8-pack
(107/150); the thinking-enabled run existed since the 2026-06-07 #515
quant-A/B (fp8 dual-max full 8-pack = 110/150, pack-defaults regime,
autoround sibling 109) but was never promoted. Backfilled with
provenance so it isn't re-derived. Guards: test-baselines,
test-quality-thinking, test-quality-baseline all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* preflight: warn on single-card GPU_MEMORY_UTILIZATION override above default
Two 5090 testers (#617 paul, and the same trap earlier) OOMed the tool-prefill
step of verify-stress by setting GPU_MEMORY_UTILIZATION=0.92 on the nvfp4
single-card slug whose validated default is 0.85. On one GPU a higher util
steals the free VRAM a large tool-response prefill needs for its activation
peak, so vLLM OOMs mid-prefill even though boot succeeds — a confusing failure
(boot ok, then HTTP 500 at step 2/8) that vLLM's own error hints at ("lower
--gpu-memory-utilization").
Add preflight_single_card_util(): an advisory WARN that fires only when the
user overrode GPU_MEMORY_UTILIZATION *above* the compose's shipped default on a
TP<=1 config. Runs even under --force (the nvfp4 slug launches with --force),
and no-ops for dual/multi-card, non-vLLM engines (no util default in the
compose), a plain default run, or util at/below the default. Wired into
switch.sh alongside the other force-independent hints.
Verified: behavior matrix (single>default WARN; single=default/below/none and
dual all silent) + test-preflight-compose-deps / test-model-switch /
test-switch-registry-parity / test-launch-compat all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The spec-dec dropdown now lists the resolved ENGINE's supported drafters + off
(vLLM: mtp / mtp_assistant; beellama: dflash / mtp / mtp_gguf; …) instead of a
bare on/off — so the drafter TYPE is selectable, and it's engine-correct (dflash
only appears when the swap engine is beellama).
- swap_apply: the SPEC entrypoint REBUILDS --speculative-config via printf from
${DRAFTER_METHOD}/${DRAFTER_N} (defaults = the sibling's), so switching the
drafter is an env override — no re-emit. Validated: default→mtp n=3,
DRAFTER_METHOD=mtp_assistant→that config, SPEC=off→dropped, compose config valid.
- services: engine_drafters() reads the profile's supported_drafters (shares the
stdlib list-parse with engine_kv_formats via _engine_yaml_list); serve_override_
defaults adds DRAFTER_OPTIONS + SPEC_METHOD + SPEC_N.
- app: the spec Select value is the drafter method (or "off"); collect maps
method→SPEC=on+DRAFTER_METHOD, off→SPEC=off.
Drafters with extra requirements (mtp_assistant needs an assistant model; dflash
needs the beellama engine) are honestly listed per-engine — picking one the swap
can't satisfy fails at boot (loose-validation stance), or use a matching ① Bring
slug. 173 byo/serve/override tests pass.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pinned cu13 ik-llama digest has a CUDA 13.2 runtime. On a driver whose
supported CUDA < 13.2 (e.g. 580.159 = CUDA 13.0, which satisfies the README's
"580.x+" but NOT this pin) the forward-compat path fails on GeForce (CUDA error
804) -> silent CPU fallback -> segfault crash-loop, with launch.sh just timing
out after 600 s and no actionable hint.
- preflight.sh: preflight_ik_llama_image() detects the driver's supported CUDA
(_driver_cuda_version) and, for ik-llama variants on a driver < 13.2 with
IK_LLAMA_IMAGE unset, auto-selects the cu12 sibling build (same build 4574,
CUDA 12.6, backward-compatible, DelspoN-validated ~68 TPS on 580.159) + prints
a ⚠ hint. >=13.2 keeps cu13; a user IK_LLAMA_IMAGE pin always wins (echoed).
_cuda_ge() does the major.minor compare. Verified: 13.3/13.2 GE, 13.0/12.6 LT.
- switch.sh: call it right after export_variant_engine_pin, before compose up
(the single up path — launch.sh delegates here). preflight.sh is already
sourced in that block.
- docs: qualify the driver requirement in the ik-llama compose header + HARDWARE.md
("580.x+" is insufficient; needs CUDA >= 13.2; auto-fallback documented).
Note: this rig is CUDA 13.3, so the <13.2 boot path is verified by logic +
DelspoN's cu12-server-4574 validation, not a local boot. Rolling cu12 tags are
NOT safe (cu12-server-100 rejects the compose's --spec-type flag, #633) — pinned
to the build-number tag matching the cu13 digest's build.
Closes#633.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(c3): ② Serve spec-dec — engine-driven drafter SELECTOR (not just on/off)
The spec-dec dropdown now lists the resolved ENGINE's supported drafters + off
(vLLM: mtp / mtp_assistant; beellama: dflash / mtp / mtp_gguf; …) instead of a
bare on/off — so the drafter TYPE is selectable, and it's engine-correct (dflash
only appears when the swap engine is beellama).
- swap_apply: the SPEC entrypoint REBUILDS --speculative-config via printf from
${DRAFTER_METHOD}/${DRAFTER_N} (defaults = the sibling's), so switching the
drafter is an env override — no re-emit. Validated: default→mtp n=3,
DRAFTER_METHOD=mtp_assistant→that config, SPEC=off→dropped, compose config valid.
- services: engine_drafters() reads the profile's supported_drafters (shares the
stdlib list-parse with engine_kv_formats via _engine_yaml_list); serve_override_
defaults adds DRAFTER_OPTIONS + SPEC_METHOD + SPEC_N.
- app: the spec Select value is the drafter method (or "off"); collect maps
method→SPEC=on+DRAFTER_METHOD, off→SPEC=off.
Drafters with extra requirements (mtp_assistant needs an assistant model; dflash
needs the beellama engine) are honestly listed per-engine — picking one the swap
can't satisfy fails at boot (loose-validation stance), or use a matching ① Bring
slug. 173 byo/serve/override tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* fix(launcher): GGUF verify-glob (#634) + launch.sh .env engine-image passthrough (#632)
#634 — setup.sh re-hardcoded VERIFY_GLOB=*.safetensors at the primary
download/verify (line 625), clobbering the recipe's glob that
load_weight_recipe already set from PRIMARY_WEIGHT_KEY. Every GGUF primary
fetch (WEIGHTS=gguf/iq4ks) then failed verify ("No *.safetensors found")
despite a byte-correct download. Now defaults to ${VERIFY_GLOB} (the recipe's
*.gguf); VERIFY_GLOB_OVERRIDE still wins.
#632 — launch.sh's .env load loop gated to CLUB3090_DEFAULT_* only, so
IK_LLAMA_IMAGE / LLAMACPP_IMAGE pins in .env were silently dropped (the two
GGUF engines are not profile-injected, so .env is their only override path —
the natural persistence for the #633 cu12 workaround never reached compose).
Broadened the allowlist to the engine-image keys; switch.sh already loaded
them (no filter). Both launchers now echo an ik-llama/llama.cpp image pin so a
wrong-image boot is never silent. (DelspoN's report cited the engine-pin
whitelist, which is the vllm/beellama-only profile-injection path — the real
drop was the .env loop filter.)
Verified: the .env loop now loads IK_LLAMA_IMAGE/LLAMACPP_IMAGE and still drops
un-allowlisted keys; test-model-switch / test-launch-compat / *-registry-parity
/ test-setup-picker / test-model-weights-registry all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Four dogfood refinements to the #636 override editor:
1. Custom values in the dropdowns. ctx / KV / util each gain a trailing
"✎ custom…" sentinel that reveals a companion free-text Input
(on_select_changed, same idiom as the ① Bring profile hatch) — so any value
the presets omit is reachable (e.g. ctx=81920, KV=turboquant_4bit_nc).
collect_overrides reads the companion when the sentinel is selected.
2. spec-dec shows the real drafter. The label is now e.g. "MTP n=3 (on)" /
"off — no spec-dec" instead of a bare "on" (value still on/off — the ${SPEC}
gate). Parsed from the sibling's --speculative-config.
3. KV types are ENGINE-driven. The KV dropdown reads the resolved engine's
supported_kv_formats (vLLM: fp8 family + int8-PTH; llama.cpp: q4_0/q5_0/…) via
the new engine_kv_formats() — not a generic list. turboquant_4bit_nc (native
in v0.24.0 but not yet in the profile's list) stays reachable via ✎ custom.
4. Slug preview. A read-only line above the fields shows what you're overriding:
engine · ctx · KV · spec · util (from serve_override_defaults, now also
parsing ENGINE + SPEC_DRAFTER; sys.path pinned so the import is cwd-safe).
171 byo/serve/override tests pass.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 2 of the ② Serve rework (Phase 1 = #635 dead-end fix): a Route-C brought
model can be re-tuned before serve, without hand-editing a compose. Fields
(pre-filled from the resolved slug's defaults, dropdowns to avoid typos):
• served name → SERVED_NAME (Input)
• ctx → MAX_MODEL_LEN (Select — presets + the slug's own default)
• KV cache → KV_CACHE_DTYPE (Select — fp8_e5m2/e4m3/turboquant_4bit_nc/…)
• spec-decode → SPEC on/off (Select)
• VRAM util → GPU_MEMORY_UTILIZATION (Select)
Mechanism (all five ride env — no per-serve compose rewrite):
- data.py: ActionPlan gains `env`; dispatch merges {**os.environ, **plan.env}.
- services.py: serve_generated(compose_path, overrides) → plan.env (+ pins
MODEL_DIR); serve_override_defaults() parses the sibling compose's
${VAR:-default} for pre-fill (stdlib regex, no PyYAML).
- swap_apply.py: emit parameterizes --served-model-name ${SERVED_NAME:-…} and
gates --speculative-config behind ${SPEC:-on} via the SAME entrypoint the
shipped nvfp4 compose uses — so SPEC=off drops the MTP drafter at up-time.
- app.py: LaneServePane fields, revealed + pre-filled only for a Route-C armed
model (hidden otherwise); collect_overrides() → serve_generated. Loose
validation (dropdowns constrain; the reconcile gate + boot are the real check).
Validated: emitted compose `docker compose config` resolves SERVED_NAME=X and
SPEC=off (entrypoint gate) cleanly; 169 byo/serve/override tests pass.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two live-dogfood issues on the Bring & Validate ② Serve stage:
1. Dead end / stale text. The ② Serve armed card said it serves "the resolved
catalog compose … NOT your brought model's weights" — the OPPOSITE of what
#628/#630 actually do for a Route-C fine-tune (they serve YOUR weights via
the sibling's recipe). And when unarmed it fell back to a dim placeholder
that reads as empty. bring-funnel-design §2b item 7 explicitly forbids this
("no dead end"). set_armed is now route-aware: Route-C → "serves <brought>
(your brought weights) via <sibling>'s recipe → [⏎] to serve"; non-swap
routes keep the honest "catalog reproduction, untested" wording.
2. ⏎ overload. #631 made ⏎ on ① Bring advance-when-present, but ⏎ there also
runs the fit-check, so it re-fit instead of proceeding. Reverted: ⏎ ALWAYS
fit-checks; a DEDICATED key [s] advances ① Bring → ② Serve (guarded on
servable + weights-on-disk; absent → notify to press [D]). [s] joins the
existing context-routed s_key (Containers restart · ④ Measure submit).
Card + weights-line now say "press [s] to continue to ② Serve".
Tests: test_bring_s_key_advances_to_serve_when_present, _no_advance_when_
weights_absent, test_bring_enter_always_fitchecks_never_advances, test_serve_
armed_route_c_serves_your_weights; updated the s_key-gating test to include
tab-bring. 195 touched-surface tests pass.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #630 card told the user to "press [⏎] ② Serve" when the brought weights
were on disk, but ⏎ on the ① Bring tab re-ran the fit-check (its _validate_
primary → _trigger_lane_bring), so pressing it just re-fit instead of
proceeding — the reported confusion.
Make ⏎ DWIM on ① Bring (_bring_primary):
- an already fit-checked, servable target whose weights are ON DISK (and the
repo input is unchanged) → ADVANCE to the pre-armed ② Serve tab (⏎ there
serves via the #630 emit-and-serve path);
- otherwise (fresh/changed target, or weights absent) → run the fit-check as
before. Absent stays on fit-check because the card there points at [D]
(download is the real next step). The [Fit] button still forces a re-check.
Also fixes the stale "([2/]] next stage)" hint on the weights-line → "(press
[⏎] to continue)", and the present-card next-step wording.
Tests: test_bring_enter_advances_to_serve_when_present (present → active tab
flips to tab-serve) + test_bring_enter_refits_when_weights_absent (absent →
stays on tab-bring). 156 touched-surface tests pass.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A Route-C fit-check whose weights were already downloaded showed a
self-contradiction: the verdict card hardcoded "→ Press [D] to download +
serve" while the weights-line below it correctly said "✓ weights on disk →
② Serve". [D] is not only download — for a Route-C brought model it emits the
serve-locally swap compose that ② Serve then serves — so it can't simply be
hidden.
Fix (presence-aware, "② Serve owns the emission"):
- _byo_result_text takes weights_present (probed once in the fit-check
handler, feeding BOTH the card and the weights-line so they can't disagree).
On disk → the card points at ② Serve, no [D]; absent → the [D] download
prompt as before.
- pull.sh gains --emit-only (with --apply-swap): passes do_download=False so
apply_swap emits the serve compose WITHOUT downloading — the present-weights
path.
- action_serve_untested: Route-C + weights present + no swap compose yet →
run_bring_emit_and_serve emits via pull.sh --apply-swap --emit-only, then
serves it. So a present-weights brought model serves straight from ② Serve
with no [D] step.
Tests: fixed test_bring_result_points_forward_to_serve (stale since #628 — it
asserted "→ ② Serve" on an absent-weights card that now honestly shows [D]);
added test_bring_result_present_points_to_serve (present → ② Serve, no [D]);
extended the run_bring_download test for --emit-only. 139 touched-surface
tests pass; pull.sh --emit-only validated end-to-end.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(byo): apply-swap — download + serve a Route-C fine-tune
A curated-arch fine-tune (e.g. josefprusa/ThinkingCap-Qwen3.6-27B AutoRound
→ qwen3.6-27b) hard-stops at the pull gate's stratum-5 no-fit-model (correct
— nothing to price), so the c3 Bring [D] download failed with "download did
not complete". Route-C was pure guidance ("clone the compose, point --model
at your weights") with no code behind it — the deferred "swap_path apply".
Build it as a DISTINCT action that never touches the locked 6-stratum gate:
- scripts/lib/profiles/swap_apply.py (new): resolve the Route-C swap
(arch→curated sibling + has_mtp_head from deriver), download the brought
weights SHA-verified via downloader.download_model (a minimal EInput), and
emit a serve-locally compose that CLONES the --profile-like sibling's REAL
compose (keeping its curated chat-template / reasoning+tool parsers / MTP
wiring — NOT the derived-vllm template, which drops all that) with --model
re-pointed at a `<weights>:/brought-model:ro` mount, --served-model-name set
to the brought basename, and --speculative-config kept iff the checkpoint
carries an MTP head (else dropped). Written alongside the sibling compose as
`_brought-<san>.yml` (gitignored) so its relative ../ mounts resolve.
- scripts/pull.sh: intercept `--apply-swap` in the wrapper (strictly additive,
like --json) → run the apply-swap action; prints `[apply-swap] compose: …`.
The gate path is byte-unchanged without the flag.
- c3: services.run_bring_download gains `apply_swap=` (appends --apply-swap +
captures the emitted compose via last_swap_compose()); the [D] worker passes
it on route==C and stashes the compose; ② Serve serves that swap compose
directly (serve_generated) instead of a reproduction of the sibling's own
catalog compose.
- pull.py: fix the stale NOTE — "the bf16 base won't fit and lacks the MTP
head" → the base HAS the head; now just the size reason.
Tests: test-pull-swap.sh section 4 (emit: --model repointed, MTP kept/dropped
by head presence, curated flags + JSON args survive the YAML round-trip,
distinct container_name); test_services apply_swap flag + compose capture. The
locked-gate hard-stop tests stay green; --json fit-check byte-identical.
Validated live: pull.sh --json still no-fit-model+route-C, and apply_swap emits
a correct ThinkingCap swap compose (MTP kept).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* c3: reframe Route-C fit-check card as ✓ Servable + a clear [D] action
The fit-check card for a curated-arch fine-tune (Route-C swap) was
self-contradicting: a RED "not eligible" + a bare "no-fit-model" token at
the top, then a GREEN "② Serve is armed with <sibling>" at the bottom —
which read as a dead-end AND named the wrong model (the sibling, not the
brought fine-tune). The engine's "no-fit-model" is the *means* (generic
fit-math can't price a curated-hybrid arch), not the user's answer; the
outcome is servable via the sibling's recipe.
Reframe the Route-C case in _byo_result_text:
- green "✓ Servable — a fine-tune of <sibling>" instead of red "not eligible",
- one plain "How it serves" line (reuses the sibling's chat-template / tools /
spec-dec with your weights) instead of the --quantization/impl bullets,
- MTP kept/dropped line from has_mtp_head,
- the next-step is an explicit "→ Press [D] to download + serve <BROUGHT model>"
(the fine-tune, not the sibling) — which #628's --apply-swap now fulfils,
- raw verdict + backend note dimmed for debugging, not the headline.
Non-swap cases (eligible / Route A / B / plain no-fit) are byte-unchanged.
Regression test added; c3 fast suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The original OOM at 98K was caused by MTP, not the context length. Two
5090s (#613 @guybrush01, #617 @paulp83) swept the space: MTP-on at 98K
leaves no room for the draft head + cudagraphs + GDN prefill scratch and
OOMs — but at 65K + util 0.85 it PASSES, with MTP's ~2x intact:
verify-stress all-pass, decode 129/155 TPS (vs 71/71 with SPEC=off), MTP
accept ~3.2, ~1.4 GB VRAM free.
So the default keeps MTP and trades context, rather than dropping MTP:
- compose single/nvfp4/mtp.yml: MAX_MODEL_LEN 98304 -> 65536,
GPU_MEMORY_UTILIZATION 0.92 -> 0.85, header updated to the swept config.
- registry: max_ctx 98304 -> 65536, mem_util 0.92 -> 0.85, status_note.
- BENCHMARKS: guybrush's SPEC=on+65K row (129/155 decode, accept 3.2).
- baselines.yml: 27B single-nvfp4 submission (stale=false, v0.24.0).
SPEC=off stays exposed as the max-ctx fallback (81K/98K @ 71 TPS) and the
tight-system-RAM path (MTP's draft load OOM-kills a 28 GB host, #617).
80 GB+ cards raise MAX_MODEL_LEN toward 262K with MTP on. Stays 🧪 (8-pack
quality still owed). Scoped guards green (baselines / status-drift /
switch+launch parity / profiles-compat).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The BYO fit-check's Route-C weight-swap (pull.sh _swap_path) hardcoded
`drop_spec_config: True` on the premise "a generic repo carries no MTP
head." That's false for a fine-tune that PRESERVES the head — so any such
model (e.g. ThinkingCap-Qwen3.6-27B, whose AutoRound INT4 ships the full
15-tensor BF16 mtp head in model_mtp_bf16.safetensors) was silently served
MTP-off, losing spec-dec despite the head being present.
Fix: detect it. `deriver.detect_mtp_head(config, api)` returns True when the
config DECLARES the MTP layers (`mtp_num_hidden_layers` / nested
`num_nextn_predict_layers`) AND a dedicated mtp weights file is present —
ground-truth for the separate-file layout fine-tune re-quants use, using
signals the deriver already has in hand (no extra fetch). Exposed as an
additive `has_mtp_head` on der.profile; _swap_path sets
`drop_spec_config = not has_mtp_head`. An embedded-head repo (head baked
into the shards with no named file) still falls back to drop — conservative,
and the named-file layout is the norm.
Validated on real HF metadata: josefprusa/ThinkingCap AutoRound →
has_mtp_head=True (keep --speculative-config); a plain AWQ re-quant →
False (drop). test-pull-swap replaces its tautology check with a real
detect_mtp_head unit test (declares+file, plain, nested text_config,
declares-but-no-file). Pull-gate suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The ⚠️-caveat required a clean v0.24.0 4-card fp8/e4m3-KV report to
upgrade to ✅. Two landed:
- #584 (@ryanmpelletier, 4× 3090 x16, Proxmox): attention backend
FLASHINFER (the #595 int8-PTH→fp8/e4m3 flip live at TP=4), verify-full
9/9, verify-stress clean to 240,634 tok, soak-continuous PASS, bench
n=5, and the full 8-pack at 111/150 — ties the 2-card dual-max proxy
(109/150), so max-tier quality is TP-invariant (measured, not assumed).
- #625 (@MoppelMat, 4× 3090 bare metal, mixed x4/x8, 300 W): a second
independent 4-card rig — decode 79/102, prefill-90K clean, soak PASS.
Decode beats ryan's x16 rig (bare metal); prefill trails ~30% (narrow
lanes are the bandwidth-bound bottleneck; decode is lane-insensitive
at TP=4) — a clean decode(compute)/prefill(interconnect) split.
Changes:
- compose header: Status ⚠️ → ✅, drop the Caveats block, Quality now
cites the 4-card 111/150, prose updated to the promotion evidence.
- registry: status caveats → production + refreshed status_note.
- baselines.yml: replace @Whamp's doubly-stale #446 int8-PTH row (v0.22.0)
with ryan's fresh v0.24.0 fp8-KV submission (TPS + 8-pack + ctx + soak);
MoppelMat noted as the corroborating 2nd rig. Emit computes stale=false.
- BENCHMARKS.md: append both v0.24.0 4-card rows (append-only history).
- MULTI_CARD.md: drop the "no 4-card bench yet / needs TP=4 confirmation"
prose now that both reports have landed.
Guards green: test-baselines, test-compose-status-drift,
test-{switch,launch}-registry-parity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Surface which speculative-decoding drafter each slug enables by default,
so the Catalog answers "does this compose do MTP / DFlash / none?" at a
glance — a serving-config facet alongside weights and kv.
Derived entirely from the registry `drafter` id already emitted by
registry-emit.sh and threaded into the cockpit (row.drafter) — no emit /
registry / guard changes. `_spec_token()` maps the id to a compact
method·mechanism token: DFlash is always an external drafter (no suffix),
so the built-in-vs-external split lives only within MTP, where the suffix
names the actual mechanism — MTP (built-in head) · MTP·gguf (external GGUF
drafter) · MTP·asst (external assistant model) · DFlash · ngram (reserved)
· — (none). The full form (n= count + source) stays in the slug detail
card.
The Bring & Validate funnel mirrors the facet: the spec token is FOLDED
into the option label so two candidates identical on topology/engine/quant
but differing only by drafter disambiguate on `· MTP` instead of falling
to the serving-stem tail; no-drafter slugs stay bare.
Tests: _spec_token/_spec_label over every real drafter id; funnel fold +
disambiguation; column-set + ordering (weights·kv·spec before ctx).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The 27B nvfp4 ships MTP n=3 (the head is unquantized). On tight-system-RAM
rigs the extra draft-model load OOMs during boot — @paulp83's 28 GB laptop
5090 timed out on the 27B while the MTP-off 35B booted fine on the same
rig (#617). Previously the only workaround was hand-deleting the two
--speculative-config lines.
Both 27B nvfp4 composes (single + dual) now take `SPEC=off`:
SPEC=off bash scripts/switch.sh vllm/qwen-27b-single-nvfp4
- bare `- SPEC` env passthrough + a bash entrypoint that appends
--speculative-config only when SPEC != off (default: MTP on, unchanged).
- single: gained an entrypoint (was plain command); dual: folded into the
existing NVLink-detect entrypoint. --speculative-config moved out of the
static command into the conditional.
- headers advertise the toggle (Drafter line + the MTP comment).
Registry drafter stays qwen-mtp-builtin (MTP is still the default; SPEC=off
is a runtime override). Verified: entrypoint simulated with compose
$$→$ de-escaping — SPEC unset emits --speculative-config, SPEC=off omits it
+ prints the notice. Compose guards (mounts / status-drift / image-drift /
registry-parity / generate-compose) green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
@guybrush01's 5090 (#612) independently reproduces @paulp83's #619 to
within noise (decode 257.3/258.0 vs 255.8/257.9) AND validates deeper —
verify-stress needle-clean to 120K (91% of 131K) vs #619's 29K, at the
full 131K (no derate; the 35B MoE's tiny KV holds on a 32 GB 5090).
Two independent 5090s agreeing is stronger evidence than one, so the
1x5090-pcie submission now carries the deeper 120K ctx_validated and
credits both submitters/sources. TPS unchanged (identical within noise).
8-pack quality still owed on both (neither built the benchlocal sandboxes).
Responded on #612 (validation) + #617 (paulp83's 27B — a DIFFERENT failure
from #613: engine never loaded / 525 MiB container, MTP + 28 GB RAM, not the
#613 post-boot VRAM OOM). learnings updated (35b + the 27b two-failure note).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
