When the Genesis/nightly single-card vLLM family was deprecated 2026-05-31
(vllm/default, long-text, long-vision, long-text-no-mtp, bounded-thinking,
tools-text), the hand-written user-facing hints weren't repointed (the
deprecation-checklist step 4 was missed) — surfaced by a #35 commenter who
read them as live. The DEFAULTS resolver was fine (vllm/default → vllm/minimal),
but the docs still advertised the dead composes.
- switch.sh header: the "Single-card vLLM" subset listed 6 deprecated composes
as recommended options AND omitted beellama/dflash (the actual single-card
default). Rewrote to lead with beellama/dflash + the functional vllm/minimal,
with a dated deprecation note (mirrors the dual-section note).
- preflight.sh: the "start with: switch.sh vllm/default" hint now points at
beellama/dflash (the single-card default).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
A careful tok/s re-test (bench.sh narrative n=5, MEASURED no-spec controls,
three v0.3.0 images: efe856397 / e0663be / 63abcd3) shows DFlash prose is
net-POSITIVE everywhere — Qwen single Q5 +27% (45.7 vs 35.9 no-spec), Qwen
dual Q8 +52% (35.6 vs 23.4), Gemma single Q4 +28-31% (44.6 vs 34.8). The
earlier "v0.3.0-wide prose-acceptance regression (~0.07 AR / net-negative)"
was a DOUBLE error: (1) over-reading the noisy/prompt-dependent acceptance-rate
diagnostic (the same efe856397 image we logged at ~0.07 now reads ~0.32 AR at
the same tok/s — Anbeeld's #288 AR caution was right), and (2) a wrong no-spec
baseline (we'd used ~37; the real dual-Q8 no-spec is 23.4). The new adaptive-DM
HEAD 63abcd3 is neutral (tok/s flat). Build-arch ruled out (a 3090 runs identical
sm_86 SASS from a fat or single-arch binary).
- compose_registry.py: 5 beellama status_notes (measured slugs assert net-positive;
un-rebenched gemma-duals retract the claim without overclaiming).
- 5 beellama compose Caveats: same correction.
- docs/UPSTREAM.md row 38: regression clause retracted; prose-recovery half of
the promotion gate dropped.
- BENCHMARKS.md: added the v0.3.0 DFlash-vs-no-spec A/B note (no-spec baselines).
- (learnings/qwen3.6-27b.md + gemma-4-31b.md got dated append-only retractions;
reported to Anbeeld at discussion #288.)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The 3 beellama composes still defaulting to the old self-hosted
multiarch-b9459-07ac3ce (the 2 singles + the gemma-q4ks dual) and the two
single-card registry status_notes predate the #296 BEELLAMA_IMAGE
centralization. b9459 is the pre-v0.3.0 build — the gemma-q4ks dual even
documented that its default "STILL has the [multi-GPU DFlash] bug". A direct
`docker compose up` (no launcher) would pull that stale image.
- image: defaults → ghcr.io/noonghunna/beellama-cpp:multiarch-v0.3.0-efe856397
(matches the q8kxl duals; broad sm_86/89/120 compat for direct-compose).
- Caveats / Pull-the-image / status_note prose → launchers inject Anbeeld's
official server-cuda-v0.3.0 (sm_86/89); 5090/sm_120 uses the multiarch build.
- Drop the obsolete "no official Docker yet (v0.3.0 WIP)" claim; note the
v0.3.0 prose-DFlash acceptance regression (code unaffected) instead.
- Preserve the two historical b9459 references (the 2026-05-30 benchmark
attribution + the old-build prose-accept comparison).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
beellama.cpp composes (ghcr.io/{anbeeld/beellama.cpp,noonghunna/beellama-cpp})
were not matched by the is_llamacpp image regex, so they fell into the vLLM
HF-cache branch and skipped the model-presence check entirely. And even on the
llama.cpp branch, only `-m`/`--model` (the target) was checked — never the
`--spec-draft-model` drafter. Result: a missing target OR drafter GGUF passed
preflight and surfaced as a cryptic in-container "failed to open GGUF file"
crash instead of preflight's "download this:" hint (#288 eddie + George).
- Add `beellama` to the is_llamacpp image regex.
- Collect drafter paths from --spec-draft-model/--model-draft/-md into their own
presence check, with a DRAFT_FILE env override mirroring GGUF_FILE/MMPROJ_FILE.
- New test case: beellama target-present + drafter-missing must refuse with the
drafter path + its `hf download Anbeeld/...` hint; both present must pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
report.sh's container globs (lightweight VRAM-holder check, kv-calc detection,
and the CONTAINER cascade) matched vllm-/llama-cpp-/club3090-/ik-llama- but not
beellama-, so a beellama dual run needed an explicit --container override to
capture docker logs + container state. Add 'name=beellama-' to all three.
Same container-recognition family as the soak-test #403 fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Rename vllm/gemma-int8→gemma-int8-mtp + vllm/gemma-mtp→gemma-bf16-mtp (both carry MTP n=4; only KV format differs); kv-calc alias fix. Ladder gemma-bf16-mtp default 32K→131K (measured BF16 KV pool 196,527 tok @ 0.95; verify-stress 8/8, NIAH→120K, bench 118.8/154.3, zero TPS regression). Codify the dual-card max-context-priority rule in DUAL_CARD.md. Suite 41/41.
Single beellama/gemma-dflash 128K (caveats; 140K OOMs under load, CTX_SIZE=102400 agent-safe). New dual beellama/gemma-dflash-dual parked upstream-gated (262K works but DFlash multi-GPU broken in 07ac3ce; fixes on v0.3.0 dev branch, no release). switch.sh --list hides upstream-gated (parked) like deprecated with its own note. UPSTREAM row tracks Anbeeld #39. Suite 38/38.
Adds the max context size to switch.sh --list. registry-emit threads a ctx label through the variant TSV (before status_note); switch.sh renders it rightmost. Production -> bare ctx; caveats/NA -> folded into the health paren (comma-separated). Rounded to nearest K (163840->164K; 32768 reads 33K). Shows registry vs compose default as a single value when they match, 'validated/compose' (e.g. 164K/200K) when they drift -- the only 3 drifts are experimental lanes; all production/caveats match. Suite 38/38.
Match gpu-mode.sh to the #279 dual prune: remove the 27b-turbo/27b-dflash/27b-dflash-noviz modes (they launched deprecated composes; defensive stop_ helpers kept), and repoint Gemma so gemma -> vllm/gemma-int8 default (:8032, 262K+vision), new gemma-mtp -> bf16 32K fallback (:8030), gemma-int8 kept as alias. bash -n clean; suite 38/38.
Bump vllm/gemma-mtp (dual, bf16+MTP) to stable v0.22.0 (validated 5/5 coherent). Deprecate vllm/gemma-mtp-tp1: fp8 KV is hardware-impossible for Gemma 4 on Ampere sm_86 (fp8e4nv kernel unsupported; fp8_e5m2 rejected by gemma4 attention allowlist; nvfp4 Blackwell-only), live-confirmed on v0.22.0. Remove the dead DEFAULTS[(gemma-4-31b,vllm,single)] row; repoint launch/setup/preflight single-card Gemma -> beellama/gemma-dflash. Also fixes a pre-existing unterminated NEXT_STEPS_NOTE string in setup.sh's gemma-4-31b arm. Suite 38/38.
Move the two default qwen3.6-27b vLLM composes off the purge-prone nightly
(#167) onto immutable v0.22.0 — the engine the 35B-A3B already runs on:
- vllm/minimal: nightly -> v0.22.0; drop the PR-35936 required-tool-fallback
overlay + its install (its serving.py does `import vllm.beam_search`, removed
in newer vLLM -> crashes on v0.22.0; the bug it patched doesn't trip on
current stable). Keep the froggeric chat template.
- vllm/dual: v0.21.0 -> v0.22.0 (already overlay-free since cf1f14f).
Validated live on 2x3090, stock v0.22.0 (no Genesis, no source overlays):
- vllm/minimal TP=1: engine v0.22.0, 5/5 coherence probes clean, 21.1 GB.
- vllm/dual TP=2 + MTP n=3: engine v0.22.0, 5/5 clean (built-in MTP works on
stock v0.22.0 without Genesis), 22.3 GB. No first-call warmup garble.
Confirms qwen3.6-27b AutoRound INT4 + fp8 KV (+ MTP) runs clean on stable
v0.22.0 — the path off the nightly treadmill (#167). Also updates the
test-launch-compat hardcoded v0.21.0 expectation for fp8-mtp.yml -> v0.22.0.
Suite 38/38.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
* Deprecate Genesis vLLM composes; repoint vLLM single default to vllm/minimal
Mark all 8 Genesis-loading qwen3.6-27b vLLM composes deprecated (registry
status + compose-header Status): vllm/default, vllm/long-text, vllm/long-text-no-mtp,
vllm/long-vision, vllm/bounded-thinking, vllm/tools-text, vllm/dual-turbo,
vllm/dual-tq3-mtp-genesis. Genesis is on hold pending Sandermage's next stable
release; the stack is moving to stable vLLM + beellama single-card.
Repoint DEFAULTS[(qwen3.6-27b, vllm, single)]: vllm/default -> vllm/minimal (the
Genesis-free fp8-KV config). The `vllm/default` token now resolves single ->
vllm/minimal, dual -> vllm/dual. The model single-card default stays beellama/dflash.
Resolver guard tests updated (vllm/default single -> vllm/minimal); SINGLE_CARD.md
example commands redirected off the deprecated slugs. Suite 38/38.
Follow-ups: (1) bump vllm/minimal + vllm/dual to v0.22.0 (Genesis-free -> stable);
(2) SINGLE_CARD.md narrative reframe (single-card vLLM deprecated, beellama default).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* switch.sh --list: hide deprecated by default, reveal with --all
With 9 Genesis composes now deprecated, --list was cluttered. Hide deprecated
variants from the default --list (tally + display loops); --all reveals them
alongside other-topology variants. Footer shows "(+N deprecated hidden --all)"
so they stay discoverable, never silently dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
setup.sh reads only the first positional (the model) and DOWNLOADS WEIGHTS;
it does not select or boot a serving config. A stray second arg — commonly a
launch slug like `vllm/int8` — was silently dropped.
In issue #250 a user ran `setup.sh gemma-4-31b vllm/int8`; the slug was
ignored, and when the real failure surfaced (a purged-nightly compose pin,
since fixed in the v0.21.0 repin) they mis-attributed it to the setup command.
Silently swallowing the arg reinforced the wrong mental model.
Now any positional beyond the model errors (exit 64) with a pointer to the
launch path (`launch.sh --variant <slug>` / `switch.sh <slug>` / `--list`).
`both` recurses with a single arg, so it's unaffected. switch.sh and launch.sh
already guard this (second positional → "multiple variants supplied"); setup.sh
was the only gap.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Gemma single-card had NO functional default (vLLM single is upstream-gated at
head_dim=512; ik-llama walls ~24K; stock llama.cpp ~12 TPS). beellama is the
only viable fast single-card Gemma-4 path, so promote it to fill the gap.
Registry: beellama/gemma-dflash experimental -> caveats + new
DEFAULTS[(gemma-4-31b, beellama, single)]. Resolver now returns it for Gemma
single (beellama is #1 in ENGINE_PREFERENCE[single]); Gemma dual unchanged.
Validated #441: 131-200K ctx, 47/88 TPS, 8-pack 109/150 think-off /
114/150 think-on (reasoning net-positive on Gemma). Reuses the multi-arch
image from #271 (sm_86/89/120; sm_89/120 compiled-not-validated).
Docs: compose header (caveats + Quality + ctx ceiling), README Gemma row,
BENCHMARKS row marker + intro, UPSTREAM.md row. Resolver guard test flipped
from a degradation assertion to beellama/gemma-dflash; full suite 38/38.
Re-test trigger: re-point the default to the no-fork mainline path when
llama.cpp#23398 (Gemma-4 MTP) merges (docs/UPSTREAM.md).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Make `beellama/dflash` the single-GPU default for Qwen3.6-27B, served via
an unofficial multi-arch image, and demote ik-llama to the balanced alt.
Decision basis (single-3090, 370 W; bench.sh n=5 + quality-test --full):
- Code TPS ~100 vs ik 69 — fastest single-card 3090 code path.
- 8-pack 107/150 (71%) think-off vs ik 99/150 (66%); 113/150 think-on
(same-session). DFlash is output-lossless, so quality == the Q5_K_S target.
- Context ceiling ladder (measured): 130K comfortable, 160K usable
(115K-tok prefill OK, 0.8 GB headroom), 200K boots but OOMs on prefill.
Ships 102K default; raise to 160K via CTX_SIZE.
- ik still wins narrative TPS (63), VRAM, and shipped vision — kept as the
balanced alt (`--variant ik-llama/iq4ks-mtp`).
Registry: `beellama/dflash` experimental -> caveats + a new
DEFAULTS[(qwen3.6-27b, beellama, single)] row. beellama is already #1 in
ENGINE_PREFERENCE[single], so the resolver now returns it for Qwen single.
Image: unofficial multi-arch ghcr build
`beellama-cpp:multiarch-b9459-07ac3ce` (sm_86/89/120 = RTX 3090/4090/5090),
built from Anbeeld/beellama.cpp .devops/cuda.Dockerfile with
CUDA_DOCKER_ARCH="86;89;120" + GGML_CUDA_FA_ALL_QUANTS=ON. Engine reports
`ARCHS = 860,890,1200`; cuobjdump confirms all 3 SASS. CAVEAT: sm_89/sm_120
are compiled but UNVALIDATED — only sm_86/3090 is verified on-rig.
Also bumps the gemma beellama compose to the same multi-arch image (stays
experimental — not promoted). Docs: README single-card framing,
INFERENCE_ENGINES.md, UPSTREAM.md row, BENCHMARKS.md Qwen beellama row.
Resolver guard tests flipped to the new default; full suite 38/38.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Every vLLM compose + engine-pin now defaults to stock vllm/vllm-openai
(nightly-SHA / v0.21.0 / v0.22.0); nothing builds or pulls the baked
vllm-club3090 image. Repoint the test-preflight-compose-deps fixture off the
retired club image to a stock tag (the image is incidental — the test asserts
on missing model weights). Document the vLLM delivery model in AGENTS.md:
patches are volume-mounted into the pinned stock image, not baked; the
vllm-club3090 GHCR package is retired-by-disuse (kept as historical release
artifacts, not deleted). Leaves the legacy dockerfile_bake delivery block +
its patch_attribution handler (test-covered, marked read-only) untouched.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Onboards Anbeeld's llama.cpp fork (beellama.cpp) as a registry engine
with two single-card DFlash composes, both experimental pending a
published Docker image:
- beellama/dflash Qwen3.6-27B Q5_K_S + Anbeeld DFlash-IQ4_XS,
q5_0/q4_1 KV, 102K ctx, port 8060
- beellama/gemma-dflash Gemma-4-31B Q4_K_S + Anbeeld DFlash-IQ4_XS,
q5_0/q4_1 KV, 102K ctx, port 8061
Composes reference the locally-built ${BEELLAMA_IMAGE:-beellama-cpp:local}
(entrypoint /app/llama-server) — upstream ships no pullable image, so
they ship as Status: Experimental and surface as (NA: experimental) in
--list (launch is --force-gated). Configs follow the fork's quickstart
docs (Qwen "Precision combo", Gemma "Balanced combo"), deviating only on
the stack-wide --reasoning off default.
Catalog wiring: engine profile beellama-local.yml; two DFlash GGUF
drafters (anbeeld-qwen-dflash, anbeeld-gemma-dflash); compose_registry.py
entries with kvcalc_key="SKIP" (llama.cpp-family, like ik-llama);
weights-map entries (target + draft GGUFs) on both models; q4_1 added to
rtx-3090 supported KV; INFERENCE_ENGINES + UPSTREAM notes; test counts
bumped (engines 8->9, drafters 6->8, registry 45->47, disk 46->48).
Default-resolver invariant preserved: beellama is #1 in
ENGINE_PREFERENCE[single], but its (NA) status + no DEFAULTS row means
the resolver skips it, so qwen3.6-27b single still resolves to
ik-llama/iq4ks-mtp (asserted by test-model-default-resolver.sh). It
auto-promotes only when a published image lands and the composes flip to
production. Full catalog suite green (only the pre-existing
test-submit-bench.sh fixture failure remains).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
`switch.sh --list` showed every slug regardless of topology, so a
single-GPU box listed dual/multi configs it can't actually run — noise.
Filter `--list` to the topologies the detected GPU count supports:
1 GPU -> single only, 2 -> single+dual, 4+ -> everything. Detection
reuses switch_topology_from_gpus (CUDA/NVIDIA_VISIBLE_DEVICES, then
nvidia-smi) mapped to a topology rank.
- `--all` (and the `--list-all` alias) bypass the filter for
discoverability; --list is deferred until args are parsed so
`--list --all` works in either order.
- Don't silently hide: print a one-line note naming the detected GPU
count and exactly which topologies were hidden, plus a `(+N … hidden
— --all)` tally in the header. No note under --all / when nothing is
hidden.
- Fail-open: with no detection signal (no selector, no nvidia-smi) show
ALL — never hide off a guessed "single" fallback.
- Counts (header + per-model) reflect the visible set; PR-A health
markers/grouping and PR-B Defaults view unchanged.
New test scripts/tests/test-list-topology-filter.sh covers 1/2-GPU
filtering, --all, --list-all, order independence, the --all-without-list
guard, and fail-open. Docs: --help usage, FAQ model-switch entry, README.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
* feat(switch): model-default resolver + user-pinnable defaults
Add a two-layer default scheme on top of the existing <engine>/default map.
<engine>/default stays the maintainer's recommendation (read-only to users);
the new <model>/default is the user's preference — their .env pin if set,
else a curated pick for the detected topology.
compose_registry.py: two maintainer knobs next to DEFAULTS —
RECOMMENDED_DEFAULT_MODELS (short opt-in shortlist, not an exhaustive
ranking; new models are NOT auto-added) and ENGINE_PREFERENCE (per-topology
engine order). Plus resolver helpers (curated_default_target,
community_default_target stub, model_default_pin_key, engine_set/model_set,
slug_topology, model_of_slug).
registry-emit.sh: shared resolver model_default_target(root, model, topology)
— the single injection point for both launchers. Precedence ladder: --variant
(caller) -> .env pin -> community seam (None today) -> curated ENGINE_PREFERENCE
walk (skips non-functional (NA) slugs) -> degradation (notice + nearest-lower
topology, else a clear "pick explicitly" message; never crashes). Plus
x_default_dispatch: X/default with X in engine-set -> engine rec; X in
model-set -> model default; else error (engines + model-ids are disjoint).
switch.sh: <model>/default token; --set-default <slug> / --clear-default
<model> (round-trip the .env pin CLUB3090_DEFAULT_<MODELID>); a Defaults view
appended to --list (also standalone via --defaults) marking user-pin vs
curated. PR-A's grouping/markers/counts and the --force gate preserved.
launch.sh: bare invocation -> first installed shortlist model -> its
<model>/default (no full wizard); a pinned fast-path ("Launch your default
<slug>? [Y/n]"); a post-boot offer ("Make <slug> your default? [y/N]"). Any
narrowing flag keeps the explicit wizard path. Also load CLUB3090_DEFAULT_*
pin keys from .env even when MODEL_DIR is exported in the shell (the existing
.env loader is MODEL_DIR-gated, which would otherwise hide the pin).
Pin validation is warn + fall back, never blocking: unknown slug / wrong
model / topology-mismatch / (NA) status -> notice + curated default.
Tests: new test-model-default-resolver.sh (curated walk, (NA) skip,
degradation, X/default dispatch, pin override + all validation paths,
community seam skipped, .env round-trip); test-default-resolver.sh extended
with <model>/default launch.sh dispatch. Full suite green (only the
pre-existing test-submit-bench.sh fixture failure remains); 45 entries
unchanged; kv-calc calibration 17/17.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* docs: document the model-default resolver + user pins
Ship the user/contributor docs alongside the resolver code (repo convention —
update existing docs, no new top-level doc).
- README: single-card realign (ik-llama = fastest blessed single default,
llama.cpp = cliff-immune alternative — matches ENGINE_PREFERENCE single
order); add "pin your default" + <model>/default to Quick start.
- CLAUDE.md (= AGENTS.md): document RECOMMENDED_DEFAULT_MODELS +
ENGINE_PREFERENCE + the shared resolver as maintainer knobs next to DEFAULTS.
- FAQ: extend "switch to a different model" with <model>/default; new "How do
I set my own default config?" Q (two-layer model, --set-default/--clear-
default/--defaults, .env key, warn+fallback validation).
- SINGLE_CARD / DUAL_CARD: the resolver + the per-topology engine order; pin
hint.
- GETTING_STARTED: first-run uses <model>/default + the in-flow "set as
default?" prompt.
- ADDING_MODELS: a new model resolves via ENGINE_PREFERENCE; add a DEFAULTS row
per engine; RECOMMENDED_DEFAULT_MODELS is not auto-grown.
- UPSTREAM: beellama Docker-image row (gates beellama onboarding; the resolver
skips it today and it auto-promotes to single default on catalog).
- models/qwen3.6-27b/CHANGELOG: dated PR-B entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
* feat(registry): add slug health/availability flag
Add a lifecycle `status` to every registry slug so `switch.sh --list`,
launch, and switch are no longer blind to a compose's health. Previously
status lived only in compose-header comments, which drifted: the Genesis
dual compose declared "Working (with Genesis)" while its pin is parked and
it won't boot clean — a user could boot a broken slug unknowingly.
- compose_registry.py: `_entry()` gains keyword-only `status`
(default "production") + `status_note`, validated against the enum
(production/caveats/experimental/preview/upstream-gated/deprecated).
Add `compose_header_status()` mapping a compose's profile-schema
`Status:` emoji to that enum.
- Sweep every compose `Status:` header to a canonical enum value and
re-flag the non-functional slugs: all *genesis* + gemma-4-31b single
fp8 -> upstream-gated; carnice -> caveats; qwopus + qwen-a3b-preview
-> preview; bf16/int8 A/Bs, llamacpp bounded-thinking, PRISM/APEX eval
lanes, gemma-4-26b-a4b onboarding -> experimental; tq3-mtp -> deprecated.
- registry-emit.sh emits `status` + `status_note` as the last two VARIANT
fields; both loaders + the parity tests read the extended field list.
- switch.sh --list: status marker (caveats -> "(caveats)", the NA set ->
"(NA: <word>)"); model/topology grouping preserved. Launch/switch gate:
production launches, caveats launches with a notice, NA warns + requires
--force. launch.sh surfaces the flag before delegating to switch.sh.
- New drift-guard test test-compose-status-drift.sh: registry status in
enum, compose header maps to enum, and the two agree.
45 entries unchanged; kv-calc calibration 17/17; full suite green (only
the pre-existing test-submit-bench fixture failure remains).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* feat(switch): add model/variant counts to --list
Header line shows supported-model count + total variants with the health
split (N production · N caveats · N NA); each model group shows its variant
count. Widen the marker column so the (NA: …)/(caveats) markers align.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
The four dual/autoround-int4/nvlink-*.yml composes (nvlink-fp8-mtp,
nvlink-turbo, nvlink-dflash, nvlink-dflash-noviz) and their
vllm/dual-nvlink* launch slugs are removed. They were thin Docker Compose
`extends:` stubs whose ONLY override was NVLINK_MODE=force_on — fully
redundant since every base dual compose now auto-detects NVLink at boot
via detect_nvlink.sh (NVLINK_MODE=auto): it probes nvidia-smi and flips on
NCCL_P2P_LEVEL=NVL + custom-all-reduce when a bridge is present, else
NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce. NVLink rigs get the
identical fast path from the base dual compose with no separate slug;
force it explicitly with `NVLINK_MODE=force_on scripts/switch.sh vllm/dual`
if auto-detect ever misses.
Scope (prune only; no engine-image change — the v0.22.0 stable-engine
consolidation is a separate follow-up):
- delete 4 nvlink-*.yml composes
- compose_registry.py: drop the 4 _entry blocks (50 -> 46 entries)
- test-compose-registry-disk.sh: count guards 50/51 -> 46/47
- profile_runtime.yml + patches.yml: drop the 4 nvlink blocks/slug refs
- test-profiles-compat.sh: retire the C13/E3 NVLink-required-compose
scenarios (the estate NVLink-gating code is now dormant; cleanup tracked
as a follow-up, code retained)
- docs: switch.sh help, AGENTS.md, HARDWARE.md, UPSTREAM.md, patches/README.md,
fp8-mtp/dflash/dflash-noviz compose headers, per-model CHANGELOG removal
entry; add a docs/README.md pointer to `switch.sh --list` as the
authoritative registry-derived compose x slug matrix
Historical NVLink bench rows (JusefPol #31, danbedford #74/#92/#96) are
preserved in BENCHMARKS.md. Full test suite green (the lone
test-submit-bench failure is a worktree-isolation artifact — it needs
gitignored results/rebench/ fixtures absent from a fresh worktree; passes
on the working tree).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
`switch.sh --list` printed a flat `slug → full/path` list, so a reader
couldn't tell which model/topology a slug like `vllm/default` or
`vllm/minimal` was bound to without parsing the path. Group the output by
model, then topology, with the serving artifact (<quant>/<serving>.yml) in
the right column — so every slug's binding is visible at a glance.
Display-only: no slug renames, no registry/emit-contract change. model is
parsed from the variant's compose dir (field 2); topology/quant/serving from
the file field. Backward-compatible — slugs and their behavior are untouched.
Verified: switch/launch registry-parity + compose-disk/mounts tests pass;
full suite green except the pre-existing fixture-only test-submit-bench. No
test parses --list output.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
fp8-mtp.yml was only ever an A/B reference -- MTP is net-negative on this MoE
at vLLM TP=2 (n2 ~= -45% / n3 ~= -51% wall TPS; the built-in-MTP draft forward
pays inter-GPU sync the acceptance can't amortize). It served no production
purpose, so remove it and its wiring.
Removed:
- models/qwen3.6-35b-a3b/vllm/compose/dual/autoround-int4/fp8-mtp.yml
- compose_registry.py entry `vllm/qwen-a3b-preview-mtp` (port 8052)
- its profile_runtime.yml capture block + calibration anchor (-> 17 anchors)
- the kv-calc COMPOSE_ALIAS_TEXT token
Test/doc updates for the new counts:
- test-kv-generic-dense + test-submit-pull: calibration contract 18/18 -> 17/17
- test-compose-registry-disk: registry 50->49, disk 51->50 composes
- test-report-calib: container ref -> live vllm-qwen36-35b-a3b-dual
- BENCHMARKS.md + KV_MATH.md notes
The MTP-net-negative finding stays recorded in the BENCHMARKS production row
and learnings/qwen3.6-35b-a3b.md. Full tests/ suite green (sole red is the
pre-existing fixture-dependent test-submit-bench, identical on clean master).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Promote the 35B-A3B MoE dual compose out of preview after full live validation
on 2× RTX 3090 (2026-05-30): max-ctx probe (262K fits, 7.81× concurrency, 2.05M-
token KV pool), bench (178/174 wall · 182 decode TPS, CV <0.5%), verify-stress
(NIAH-clean to 240K), soak-continuous (0 growth / 0 err / 0 silent / 100%
retention), quality --full (det 79/90 = 88%, aider 13/30 — ≥ ik-llama ref), and a
live vision smoke (read a test image correctly @ 262K).
- compose: preview.yml → fp8.yml (serving-stack filename per layout convention),
preview-mtp.yml → fp8-mtp.yml; fp8.yml = 262K + vision-on + no-MTP, vLLM v0.22.0
stable (no overlays), ✅ Production header.
- registry: slug vllm/qwen-a3b-preview → vllm/qwen-35b-a3b-dual, max_ctx
16384 → 262144, compose_path, DEFAULTS, kvcalc alias (kv-calc.py).
- profile_runtime capture re-synced to fp8.yml; test-pull slug updated.
- BENCHMARKS row + ADDING_MODELS pointer.
- MTP intentionally OFF: built-in MTP is net-negative on this MoE under vLLM TP=2
at BOTH n=2 (≈−45%) and n=3 (≈−51%) — the draft forward pays inter-GPU sync the
acceptance can't amortize (ik-llama avoids this single-card). fp8-mtp.yml kept as
the A/B reference only.
GATED: the registry-wide test-diagnose-profile fails for this entry because
kv-calc's qwen-MoE branch does NOT divide weights by TP (a deliberate hack:
inflated weights ≈ the vLLM-filled peak, which coincidentally passes at 16K but
starves the 262K KV in the fit-check). The config is empirically validated; this
is a kv-calc MoE-VRAM-model limitation. Lands AFTER the kv-calc fix PR (separate)
that distinguishes diagnose fit-need from the calibration filled-peak for cheap-KV
MoE. Calibration anchor left untouched here (the re-baseline belongs to that PR).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Turn 1's TTFT includes engine cold-start (compile / cudagraph capture /
first-token warmup) and is not a steady-state datapoint. Anchoring the growth
ratio to turn 1 understated growth: a live single-card Qwen3.6-27B run printed
"TTFT grew 0.4x -> stable" because it divided the warm turns by a ~30s
cold-start, when the warm turns (2->10) actually grew ~7.4x (1.5s -> 11.3s) as
context grew ~14.7x. Anchor the growth ratio + per-turn notes to the first WARM
turn (turn 2 when >=3 turns ran; fall back to turn 1 otherwise) and surface
turn 1 separately as excluded cold-start. Matches the repo's
warm-up-then-measure bench protocol.
Also document the ramp-depth caveat: the context ramp is driven by
tool_choice='required', so reachable depth is bounded by tool-call reliability
at depth on the target config (decoupling tracked as a separate enhancement).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Rewire report.sh for --agentic on current master and tighten bench-agentic framing around curve-shape output, DeltaNet/SSM O(n) TTFT growth, and opaque fixture payloads.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Every gemma-4-31b vLLM compose pinned a purged Docker Hub nightly
(e47c98ef / bf610c2f), un-bootable for fresh users (#250, #167 class).
Prune the 9-variant set to 3 and repoint survivors onto the immutable
stable release tag vllm/vllm-openai:v0.21.0 (never :latest).
Survivors: vllm/gemma-mtp (bf16 dual, default), vllm/gemma-int8 (int8-PTH
dual), vllm/gemma-mtp-tp1 (fp8 single). Dropped: gemma-dflash,
gemma-dflash-int8, gemma-int8-tq3, gemma-bf16, gemma-awq; the 262K path
folds into gemma-int8 via a CTX env override. Gemma is decoupled from the
shared Qwen nightly profiles via a new vllm-gemma-stable engine profile;
Qwen nightly profiles are untouched.
Diagnose decoupling: the pruned gemma dflash patch dirs doubled as the
diagnose tool's cross-model disk-source proxies for two Qwen overlays
(vllm-pr41703-dflash, vllm-pr42102-dflash-kv-quant). Those capabilities
are image-baked in the pinned nightly, not mountable files, so mark them
image_baked and have diagnose skip the disk-source check for image-baked
overlays. Runtime-neutral (no Qwen compose mounts them).
switch.sh launcher: handle the new VLLM_IMAGE engine-pin export (the twin
loop in launch.sh had it; switch.sh did not -> "unexpected engine pin
export"), and guard the VLLM_NIGHTLY_SHA echo for the image-only stable
profile (was unbound under set -u).
Validated on 2x RTX 3090 (Ampere, v0.21.0): full scripts/tests/*.sh suite
35/0; live boots of gemma-mtp (bf16) and gemma-int8 serve coherent output,
qwen vllm/dual regression-clean. gemma-mtp-tp1 (fp8_e4m3) is correctly
SM-gated (required_sm=9.0) -- confirmed Ampere-incompatible (Triton
"fp8e4nv not supported on sm_86"), a documented 32 GB+/Hopper variant;
beellama remains the single-card gemma path on Ampere.
Refs #451, #250, #167.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The 'Failure breakdown:' block printed at the end of every run already shows
failure_mode + full detail per failed scenario — the quickest read. Add a
'Diagnosing failures' section (breakdown first, then JSON / benchlocal-cli
inspect for full trace / older runs / filter / diff) and a quality-test.sh
footer pointing at it. Fix the QUALITY_TEST.md sample, which showed a grouped
format + invented failure modes (missing_field/wrong_value) that the tool
never emits, to the real flat per-line format + real failure_mode vocabulary.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The producer could write a `boot-fit-measured` record with null decode
TPS when bench.sh output drifted (summary block absent/unparseable) or a
metric was missing. A measured record with null TPS is worse than no
record for optimizer calibration: it looks like real data.
For a measured result_class:
- raise MeasuredRecordError if no parseable bench summary block was found
(decode_TPS mean= absent => output drift), or if the parse produced no
decode TPS. This matches the module's existing fail-loud posture
(KeyError on an unknown registry tag).
- a malformed/absent `=== GPU state ===` line is a SOFT gap (VRAM is a
fingerprint extension, not the core measured TPS): surface it in a new
top-level `parse_warnings` list instead of raising, so the gap is
explicit and never a silent null.
Non-measured classes (predicted/derived) impose no decode-TPS
requirement; genuinely-optional optimizer fields stay null. The CLI
catches MeasuredRecordError (exit 2, clean message) and echoes warnings
to stderr. Extends the test with: measured + no decode summary => fail
loud; measured + bad/absent GPU line => parse_warnings; non-measured +
no decode => no raise; and a happy-path regression asserting empty
parse_warnings.
Addresses the Codex review Medium finding on #249.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Pure additive capture: parse a completed scripts/bench.sh stdout (no GPU,
no live model) + a compose_registry tag into ONE measurement-record JSON,
written to a per-rig gitignored corpus (results/measurement-records/).
Conforms to the optimizer design's FROZEN measurement-record field names
verbatim; optimizer-only fields (objective/confidence_tier/margin_applied)
emitted null, never fabricated. Producer-proposed additions (a context-depth
TPS ladder + power_cap_w fingerprint) namespaced under measured_extensions
and flagged as Lock-criteria #6 candidates. No consumer, no lookup, no
decision logic — that half is gated behind a separate design unlock.
bench.sh left untouched (load-bearing); emitter runs standalone on saved
bench output. Test runs GPU-free via bench.sh BENCH_MOCK output.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
The benchlocal-cli `--progress` flag emits per-scenario `[N/M]` lines
to stderr — critical for the long-running modes (--full ~30-40 min,
--reasoning ~similar, --pack aider-polyglot-30 ~25-30 min) which
otherwise go dark for the whole duration. The wrapper never forwarded
the flag, so users (including agents) ran blind unless they bypassed
the wrapper.
Make --progress on by default, --no-progress as the opt-out for CI /
log-volume-sensitive contexts. Mirror PROGRESS=0/1 env var. Document
the convention in AGENTS.md so agents see it as policy, not just
private/personal memory.
Convention surfaced after wasting 25 min on a blind --full run during
autonomous validation: the wrapper logged its own progress line and
nothing else for the entire 30+ min window.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
When a user launches with a 4090 or 5090, preflight now prints a single
hint line pointing at the matching FAQ entry. Caught one class of UX
gap where users land on launch.sh without reading the README + run into
unexpected ctx ceilings (4090) or miss the 32 GB-headroom story (5090).
Detection: greps `nvidia-smi -L` output for "RTX 4090" / "RTX 5090".
Quiet on 3090-only rigs. Pure-additive — no behavior change beyond the
extra echo lines.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
The auto_container() grep hardcoded `qwen36-27b|gemma-4-31b` engine ×
model combinations, so `ik-llama-qwen36-35b-a3b-apex-fit` (and any
future 35b-a3b sibling on vllm / llama-cpp / ik-llama) was not
discovered → "no running club-3090 container found" → soak exited
in seconds.
Workaround was `CONTAINER=...` explicit; this is the regex fix.
Updated: the grep itself, the error message, the env doc, and the
--help text in scripts/soak-test.sh.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
* Add ik-llama apex-fit-q8q5 variant for Qwen3.6-35B-A3B (#242)
Captures @laurimyllari's `--fit` + asymmetric q8_0(K)/q5_0(V) KV config
from discussion #241 as a single-card ik-llama variant on the APEX
I-Compact GGUF (registry tag `ik-llama/apex-fit-q8q5`, port 8057).
Measured 1× 3090 + 370 W, n=5:
q4/q4 mtp.yml baseline (first real run — APEX weights weren't on disk
before today): 96.47 / 144.01 wall TPS narr/code (CV 5.4% / 4.5%)
20.46 GB VRAM
q8/q5 fit-mtp.yml (this variant): 103.25 / 149.12 wall TPS
(CV 3.0% / 1.5%) — +7% narr / +4% code at tighter CV, +0.6 GB VRAM
The Anbeeld K-high/V-low asymmetric-KV pattern materialised here.
Gates:
verify-full 8/8 PASS
verify-stress 8/8 PASS incl. 180K NIAH (91% of n_ctx 196608)
bench above
soak-continuous PASS (0 errors, 0/25 silent_empty, 0 VRAM growth,
100% TPS retention, p50 decode 223 TPS)
deterministic q 76/90 = 84% on PR #38 verifiers
toolcall 14/15 · instructfollow 15/15 ·
structoutput 14/15 · dataextract 11/15 ·
reasonmath 12/15 · bugfind 10/15
MoE × MTP sub-question (from #242 body) — answered: ik-llama built-in
MTP on the 35B-A3B MoE does NOT pay the vLLM −51% / −35% penalty
(cf. `qwen3.6-35b-a3b/dual/preview-mtp.yml` BENCHMARKS row). MTP context
ready at n_ctx=196608, decode bursts 270+ TPS in soak. The MoE×MTP
penalty is vLLM-scheduler-specific, not architectural.
Status set to ⚠️ Production w/ caveats because sandbox-pack quality
(hermesagent-20 0/20, aider-polyglot-30 0/30, cli-40 timeouts) hit
benchlocal-cli sandbox infrastructure issues 2026-05-28 — hermes
`tool_events=0` after the agent runs (suspected PR #38 thinking-on
sampler interaction); aider fails at git checkout
`fatal: path 'aider/__init__.py' does not exist in 'f46766c'` before
any LLM call. Neither is attributable to the model or the compose.
Pre-PR-#38 cross-rig reference from @laurimyllari's 4090 in #241:
hermes 9-12/20, aider 14-18/30 — the model class is capable; the
sandbox state on this rig needs separate work.
Catalog gates: test-compose-registry-disk count bumped 55→56 per the
documented model-add workflow. test-profiles-compat, test-model-
weights-registry, test-switch-registry-parity, test-launch-registry-
parity all PASS. Two inherited reds (test-compose-mounts-resolve and
test-patch-attribution) unchanged from master baseline — they affect
qwen3.6-27b vLLM composes, not this addition.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fit-mtp.yml: tighten --fit/--no-mmap/--cache-ram rationale (PR #243 review)
@laurimyllari clarified on PR #243 that for the I-Compact GGUF
(~17 GB on 24 GB card) both `--fit` and `--no-mmap` are largely inert
since the model fits in VRAM — the reason to keep them as defaults
is forward-compat: swapping GGUF_FILE for a bigger quant
(UD-Q8_K_XL, APEX Quality, etc.) "just works" with reasonable
partial-offload performance, without re-tuning the compose.
Replaced the two "Why" blocks in the compose docstring to lead with
his framing. Also flagged `--cache-ram 4096` (lower than ik-llama's
8192 default) with the same suspected forward-compat rationale,
pending his confirmation on the PR.
No flag changes; docstring-only.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* Promote fit-mtp.yml ✅ Production + update sandbox-pack quality
Updates the compose Status + Quality line + BENCHMARKS row after live
validation of the sandbox-pack fixes (benchlocal-cli #42/#43/#44 +
club-3090 #245, all merged today):
hermesagent-20: 0/20 → 11/20 (55%) via #42 deterministic sampler
aider-polyglot-30: 0/30 → 12/30 (40%) via #44 git-checkout from AIDER_DIR
cli-40: 11/40 → 12/40 (30%, ±1 noise) via #43 budget fix
(helps aider/hermes wall-clock; cli-40 "timeouts"
turn out to be sandbox-internal agent-give-up, not
wall-clock budget — see follow-up benchlocal-cli issue)
All gates clean: verify-full 8/8, verify-stress 8/8 (incl. 180K NIAH),
bench n=5 (+7% narr / +4% code vs q4/q4 mtp.yml), soak-continuous PASS
(0 errors, 0 silent_empty, 0 VRAM growth, 100% retention). Status:
🧪/⚠️ → ✅ Production. Caveats trimmed to the 3090 power-sensitivity
note.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
The wrapper was always passing `--timeout-per-case 60` to benchlocal-cli,
which under benchlocal-cli #41 (per-pack `timeout_per_case_default`
metadata) would override the new agentic-pack defaults back to 60s and
defeat the fix — cli-40 / hermesagent-20 / aider-polyglot-30 would
still hit budget timeouts.
Now: only pass `--timeout-per-case` to benchlocal-cli when the user
explicitly sets `TIMEOUT_PER_CASE=...` env or `--timeout-per-case N`
CLI flag. When unset, benchlocal-cli applies its per-pack metadata
defaults (60s deterministic / 300s cli-40+hermes / 1800s aider).
Backward compat: against an OLD benchlocal-cli (pre-#41), the CLI flag's
own default is 60.0 — so unsetting the wrapper override gives the same
behavior as before. Against the NEW benchlocal-cli, the per-pack default
applies. Either way the wrapper does the right thing.
Pairs with benchlocal-cli PR https://github.com/noonghunna/benchlocal-cli/pull/43.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Follow-up to #235 — the PRISM/APEX presets exercised (model, engine, KV)
combos the profile catalog didn't know about, reddening test-profiles-compat
and test-diagnose-profile (caught by the pre-tag FULL-suite run, which the
narrower #235 gate subset had missed).
- engines/llama-cpp-mainline.yml: supported_model_families += qwen3-next-moe
(the 35b-a3b APEX MoE; llama.cpp serves the MoE GGUF — the family list was
just incomplete). Fixes constraint C10 for the 3 APEX entries.
- hardware/*.yml (all 9): supported_kv_formats += q5_0, q8_0 — the llama.cpp
KV quants the engine already lists but no hardware profile did, so the q8_0
presets (prism-pro-dq-dual-vision, apex-mtp-compact-long, apex-mtp-quality-dual)
failed C5 "kv not supported by hardware". q5_0/q8_0 are software KV quants
that work on any CUDA card (laurimyllari runs q8_0 on a 4090).
- patches.yml: register the APEX chat-template (apex-qwen-chat-template) with a
symmetric-protocol drift_guard — resolves the orphan #235 introduced into
test-patch-attribution.
Validation: test-profiles-compat + test-diagnose-profile PASS. APEX patch-
attribution orphan resolved (only the 2 PRE-EXISTING sglang orphans remain =
the v0.8.5 baseline). Full suite 29/34; the 5 remaining fails are all
pre-existing-at-v0.8.5 or test-isolation, NONE from the v0.8.5..master range:
generate-compose (pre-existing), setup-picker (mock 5090 rig), submit-bench
(0 fixtures), loop-input (test-hwdetect .pull-captures cross-contamination —
passes in isolation). Leak-clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
PR-B of the compose-quant-hierarchy refactor (follows PR-A #231). launch.sh no
longer hardcodes its variant/port/container/kvcalc maps — it derives them from
compose_registry.py via a shared emitter, the same single source of truth
switch.sh already uses. Adds a topology-aware <engine>/default resolver.
- scripts/lib/registry-emit.sh (new): shared emitter — emits switch/launch
tables from COMPOSE_REGISTRY, parses container_name from each compose, and
exposes registry_default_target() for <engine>[/<topology>]/default.
- launch.sh: drop the hardcoded `declare -A LAUNCH_*` maps; populate from the
emitter. PRIMARY_MODEL constant (default qwen3.6-27b) backs bare <engine>/default.
- switch.sh: drop inline derivation; resolve vllm/default, vllm/dual/default,
vllm/multi4/default via the shared emitter.
- compose_registry.py: add kvcalc_key to entries; tools/kv-calc.py aliases.
- bench-row-formatter.sh: DEFAULTS-aware compose_display (drops the now-dead
docker-compose.yml branch).
- tests: test-launch-registry-parity.sh + test-default-resolver.sh (new);
test-switch-registry-parity refreshed onto the shared emitter.
Default resolution is an alias layer over existing registry keys — no keys
renamed, no compose content changed. Side effect: the launcher<->registry drift
that left vllm/gemma-mtp pointing at a non-existent dual/.../fp8-mtp.yml is gone
(now derives the registry's bf16-mtp.yml).
Implemented via Codex per the PR-B brief; independently re-validated before
commit: 7/7 gates PASS (switch+launch parity, default-resolver, launch-compat,
registry-disk, mounts-resolve, kv-calc 22/22); live vllm/default->dual +
vllm/dual/default->dual both verify-full 8/8; leak-clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
PR-A (#231) inserted the <quant>/ layer; two compose-path references
outside the doc sweep were left stale:
- scripts/gpu-mode.sh (the rig mode-switcher, symlinked into
/usr/local/bin) cd'd into the dual/ topology dir and -f'd bare
filenames (turbo.yml, docker-compose.yml, ...) that PR-A moved under
<quant>/. Repointed DUAL_27B_DIR / GEMMA_DUAL_DIR at the autoround-int4
quant dir (mount-safe — same cd-into-compose-dir invocation switch.sh
uses), added GEMMA_DUAL_AWQ_DIR for the awq mode, and renamed the two
ex-docker-compose.yml defaults (qwen -> fp8-mtp.yml, gemma -> bf16-mtp.yml).
All 13 mode targets verified to resolve; bash -n clean.
- vllm-pr40798 / vllm-pr40914 patch READMEs cited dual/int8-tq3.yml (the
pre-PR-A name). Commit 6182922 split int8-tq3 into the tq3-* family; the
READMEs describe "TQ3 KV + MTP n=3" = tq3-mtp.yml (whose tombstone header
cites #40914) -> dual/autoround-int4/tq3-mtp.yml.
Surfaced (not fixed here — PR-B's registry-derivation eliminates it):
launch.sh LAUNCH_VARIANT_COMPOSE[vllm/gemma-mtp] points at a non-existent
dual/autoround-int4/fp8-mtp.yml; gemma dual default is bf16-mtp.yml (the
registry is correct). test-launch-compat doesn't sweep launch.sh paths.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The help line showed the 262K native max while the registry max_ctx, the compose
default, and BENCHMARKS all use the #201 safe default of 200K. Sync the comment.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Follow-up to PR #227 (#437). The vision re-tune changed the compose defaults but
left the registry / switch.sh / BENCHMARKS describing stale values.
- compose_registry: llamacpp/mtp-vision max_ctx 49152 -> 150000 (matches the
re-tuned compose default; ik-llama/iq4ks-mtp-vision already correct at 163840).
switch.sh derives variants from the registry, so this also fixes its dynamic list.
- switch.sh usage comments: both vision variants now show the 1M-px default +
"full-res = override, lower ctx" note (were "49K" / bare "160K").
- BENCHMARKS: marked the superseded 2026-05-20 49K row; added two re-tuned vision
rows (llama 150K@1M-px fills 138K/561MB; ik 160K@1M-px fills 147K/503MB, graduated
EVAL #402 -> Production). Decode TPS carried from siblings (ctx-independent, not
re-benched); rows focus on the measured ctx-ceiling + VRAM + verify-stress [8/8].
ik vision was ALREADY registry-wired (parity test green); this is the data sync.
Refs #438.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
verify-stress.sh already hard-fails a ceiling free-VRAM margin below the
sustained-agent guard (default 1024 MB), but a margin that *passes* the guard
yet sits close to it can still OOM on a second run or server restart — the
marginal-but-passing case @laurimyllari hit on his 4090 at 160K (8/8 PASS, then
a second rebench OOMed). The report dropped the margin number entirely, so a
PASS read as comfortable headroom.
parse_verify_stress now extracts the deepest-rung free VRAM + guard from the
existing verify-stress output (both the thin-margin and ok lines), classifies
it (thin < guard / marginal < 2× guard / comfortable), and the verify-stress
report section prints e.g. "Ceiling VRAM margin: 1500 MB free (guard 1024 MB)
— ⚠ marginal: ... a second run or server restart may OOM. Lower CTX_SIZE ...".
rebench-report.py only — no change to verify-stress.sh's pass/fail logic.
Validated parse on comfortable / marginal / thin / no-margin logs.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
