New opt-in flag: after a catalog model is up + ready, switch.sh --owui upserts
an OpenAI connection in Open WebUI (host.docker.internal:<port>) so it appears
in the chat picker — no manual Admin->Connections step. scripts/lib/owui-register.sh
is conditional (no-op if OWUI not running), idempotent (skips if already present),
and drives OWUI's admin config API via a token forged from the container secret.
Validated end-to-end on Deckard-40B (:8199).
Guard suite 41/42; the 1 failure (test-measurement-record) is pre-existing
bench-output-format drift, unrelated to this change (no switch.sh/owui reference).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
When the Genesis/nightly single-card vLLM family was deprecated 2026-05-31
(vllm/default, long-text, long-vision, long-text-no-mtp, bounded-thinking,
tools-text), the hand-written user-facing hints weren't repointed (the
deprecation-checklist step 4 was missed) — surfaced by a #35 commenter who
read them as live. The DEFAULTS resolver was fine (vllm/default → vllm/minimal),
but the docs still advertised the dead composes.
- switch.sh header: the "Single-card vLLM" subset listed 6 deprecated composes
as recommended options AND omitted beellama/dflash (the actual single-card
default). Rewrote to lead with beellama/dflash + the functional vllm/minimal,
with a dated deprecation note (mirrors the dual-section note).
- preflight.sh: the "start with: switch.sh vllm/default" hint now points at
beellama/dflash (the single-card default).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Rename vllm/gemma-int8→gemma-int8-mtp + vllm/gemma-mtp→gemma-bf16-mtp (both carry MTP n=4; only KV format differs); kv-calc alias fix. Ladder gemma-bf16-mtp default 32K→131K (measured BF16 KV pool 196,527 tok @ 0.95; verify-stress 8/8, NIAH→120K, bench 118.8/154.3, zero TPS regression). Codify the dual-card max-context-priority rule in DUAL_CARD.md. Suite 41/41.
Single beellama/gemma-dflash 128K (caveats; 140K OOMs under load, CTX_SIZE=102400 agent-safe). New dual beellama/gemma-dflash-dual parked upstream-gated (262K works but DFlash multi-GPU broken in 07ac3ce; fixes on v0.3.0 dev branch, no release). switch.sh --list hides upstream-gated (parked) like deprecated with its own note. UPSTREAM row tracks Anbeeld #39. Suite 38/38.
Adds the max context size to switch.sh --list. registry-emit threads a ctx label through the variant TSV (before status_note); switch.sh renders it rightmost. Production -> bare ctx; caveats/NA -> folded into the health paren (comma-separated). Rounded to nearest K (163840->164K; 32768 reads 33K). Shows registry vs compose default as a single value when they match, 'validated/compose' (e.g. 164K/200K) when they drift -- the only 3 drifts are experimental lanes; all production/caveats match. Suite 38/38.
* Deprecate Genesis vLLM composes; repoint vLLM single default to vllm/minimal
Mark all 8 Genesis-loading qwen3.6-27b vLLM composes deprecated (registry
status + compose-header Status): vllm/default, vllm/long-text, vllm/long-text-no-mtp,
vllm/long-vision, vllm/bounded-thinking, vllm/tools-text, vllm/dual-turbo,
vllm/dual-tq3-mtp-genesis. Genesis is on hold pending Sandermage's next stable
release; the stack is moving to stable vLLM + beellama single-card.
Repoint DEFAULTS[(qwen3.6-27b, vllm, single)]: vllm/default -> vllm/minimal (the
Genesis-free fp8-KV config). The `vllm/default` token now resolves single ->
vllm/minimal, dual -> vllm/dual. The model single-card default stays beellama/dflash.
Resolver guard tests updated (vllm/default single -> vllm/minimal); SINGLE_CARD.md
example commands redirected off the deprecated slugs. Suite 38/38.
Follow-ups: (1) bump vllm/minimal + vllm/dual to v0.22.0 (Genesis-free -> stable);
(2) SINGLE_CARD.md narrative reframe (single-card vLLM deprecated, beellama default).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* switch.sh --list: hide deprecated by default, reveal with --all
With 9 Genesis composes now deprecated, --list was cluttered. Hide deprecated
variants from the default --list (tally + display loops); --all reveals them
alongside other-topology variants. Footer shows "(+N deprecated hidden --all)"
so they stay discoverable, never silently dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
`switch.sh --list` showed every slug regardless of topology, so a
single-GPU box listed dual/multi configs it can't actually run — noise.
Filter `--list` to the topologies the detected GPU count supports:
1 GPU -> single only, 2 -> single+dual, 4+ -> everything. Detection
reuses switch_topology_from_gpus (CUDA/NVIDIA_VISIBLE_DEVICES, then
nvidia-smi) mapped to a topology rank.
- `--all` (and the `--list-all` alias) bypass the filter for
discoverability; --list is deferred until args are parsed so
`--list --all` works in either order.
- Don't silently hide: print a one-line note naming the detected GPU
count and exactly which topologies were hidden, plus a `(+N … hidden
— --all)` tally in the header. No note under --all / when nothing is
hidden.
- Fail-open: with no detection signal (no selector, no nvidia-smi) show
ALL — never hide off a guessed "single" fallback.
- Counts (header + per-model) reflect the visible set; PR-A health
markers/grouping and PR-B Defaults view unchanged.
New test scripts/tests/test-list-topology-filter.sh covers 1/2-GPU
filtering, --all, --list-all, order independence, the --all-without-list
guard, and fail-open. Docs: --help usage, FAQ model-switch entry, README.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
* feat(switch): model-default resolver + user-pinnable defaults
Add a two-layer default scheme on top of the existing <engine>/default map.
<engine>/default stays the maintainer's recommendation (read-only to users);
the new <model>/default is the user's preference — their .env pin if set,
else a curated pick for the detected topology.
compose_registry.py: two maintainer knobs next to DEFAULTS —
RECOMMENDED_DEFAULT_MODELS (short opt-in shortlist, not an exhaustive
ranking; new models are NOT auto-added) and ENGINE_PREFERENCE (per-topology
engine order). Plus resolver helpers (curated_default_target,
community_default_target stub, model_default_pin_key, engine_set/model_set,
slug_topology, model_of_slug).
registry-emit.sh: shared resolver model_default_target(root, model, topology)
— the single injection point for both launchers. Precedence ladder: --variant
(caller) -> .env pin -> community seam (None today) -> curated ENGINE_PREFERENCE
walk (skips non-functional (NA) slugs) -> degradation (notice + nearest-lower
topology, else a clear "pick explicitly" message; never crashes). Plus
x_default_dispatch: X/default with X in engine-set -> engine rec; X in
model-set -> model default; else error (engines + model-ids are disjoint).
switch.sh: <model>/default token; --set-default <slug> / --clear-default
<model> (round-trip the .env pin CLUB3090_DEFAULT_<MODELID>); a Defaults view
appended to --list (also standalone via --defaults) marking user-pin vs
curated. PR-A's grouping/markers/counts and the --force gate preserved.
launch.sh: bare invocation -> first installed shortlist model -> its
<model>/default (no full wizard); a pinned fast-path ("Launch your default
<slug>? [Y/n]"); a post-boot offer ("Make <slug> your default? [y/N]"). Any
narrowing flag keeps the explicit wizard path. Also load CLUB3090_DEFAULT_*
pin keys from .env even when MODEL_DIR is exported in the shell (the existing
.env loader is MODEL_DIR-gated, which would otherwise hide the pin).
Pin validation is warn + fall back, never blocking: unknown slug / wrong
model / topology-mismatch / (NA) status -> notice + curated default.
Tests: new test-model-default-resolver.sh (curated walk, (NA) skip,
degradation, X/default dispatch, pin override + all validation paths,
community seam skipped, .env round-trip); test-default-resolver.sh extended
with <model>/default launch.sh dispatch. Full suite green (only the
pre-existing test-submit-bench.sh fixture failure remains); 45 entries
unchanged; kv-calc calibration 17/17.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* docs: document the model-default resolver + user pins
Ship the user/contributor docs alongside the resolver code (repo convention —
update existing docs, no new top-level doc).
- README: single-card realign (ik-llama = fastest blessed single default,
llama.cpp = cliff-immune alternative — matches ENGINE_PREFERENCE single
order); add "pin your default" + <model>/default to Quick start.
- CLAUDE.md (= AGENTS.md): document RECOMMENDED_DEFAULT_MODELS +
ENGINE_PREFERENCE + the shared resolver as maintainer knobs next to DEFAULTS.
- FAQ: extend "switch to a different model" with <model>/default; new "How do
I set my own default config?" Q (two-layer model, --set-default/--clear-
default/--defaults, .env key, warn+fallback validation).
- SINGLE_CARD / DUAL_CARD: the resolver + the per-topology engine order; pin
hint.
- GETTING_STARTED: first-run uses <model>/default + the in-flow "set as
default?" prompt.
- ADDING_MODELS: a new model resolves via ENGINE_PREFERENCE; add a DEFAULTS row
per engine; RECOMMENDED_DEFAULT_MODELS is not auto-grown.
- UPSTREAM: beellama Docker-image row (gates beellama onboarding; the resolver
skips it today and it auto-promotes to single default on catalog).
- models/qwen3.6-27b/CHANGELOG: dated PR-B entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
* feat(registry): add slug health/availability flag
Add a lifecycle `status` to every registry slug so `switch.sh --list`,
launch, and switch are no longer blind to a compose's health. Previously
status lived only in compose-header comments, which drifted: the Genesis
dual compose declared "Working (with Genesis)" while its pin is parked and
it won't boot clean — a user could boot a broken slug unknowingly.
- compose_registry.py: `_entry()` gains keyword-only `status`
(default "production") + `status_note`, validated against the enum
(production/caveats/experimental/preview/upstream-gated/deprecated).
Add `compose_header_status()` mapping a compose's profile-schema
`Status:` emoji to that enum.
- Sweep every compose `Status:` header to a canonical enum value and
re-flag the non-functional slugs: all *genesis* + gemma-4-31b single
fp8 -> upstream-gated; carnice -> caveats; qwopus + qwen-a3b-preview
-> preview; bf16/int8 A/Bs, llamacpp bounded-thinking, PRISM/APEX eval
lanes, gemma-4-26b-a4b onboarding -> experimental; tq3-mtp -> deprecated.
- registry-emit.sh emits `status` + `status_note` as the last two VARIANT
fields; both loaders + the parity tests read the extended field list.
- switch.sh --list: status marker (caveats -> "(caveats)", the NA set ->
"(NA: <word>)"); model/topology grouping preserved. Launch/switch gate:
production launches, caveats launches with a notice, NA warns + requires
--force. launch.sh surfaces the flag before delegating to switch.sh.
- New drift-guard test test-compose-status-drift.sh: registry status in
enum, compose header maps to enum, and the two agree.
45 entries unchanged; kv-calc calibration 17/17; full suite green (only
the pre-existing test-submit-bench fixture failure remains).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* feat(switch): add model/variant counts to --list
Header line shows supported-model count + total variants with the health
split (N production · N caveats · N NA); each model group shows its variant
count. Widen the marker column so the (NA: …)/(caveats) markers align.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
The four dual/autoround-int4/nvlink-*.yml composes (nvlink-fp8-mtp,
nvlink-turbo, nvlink-dflash, nvlink-dflash-noviz) and their
vllm/dual-nvlink* launch slugs are removed. They were thin Docker Compose
`extends:` stubs whose ONLY override was NVLINK_MODE=force_on — fully
redundant since every base dual compose now auto-detects NVLink at boot
via detect_nvlink.sh (NVLINK_MODE=auto): it probes nvidia-smi and flips on
NCCL_P2P_LEVEL=NVL + custom-all-reduce when a bridge is present, else
NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce. NVLink rigs get the
identical fast path from the base dual compose with no separate slug;
force it explicitly with `NVLINK_MODE=force_on scripts/switch.sh vllm/dual`
if auto-detect ever misses.
Scope (prune only; no engine-image change — the v0.22.0 stable-engine
consolidation is a separate follow-up):
- delete 4 nvlink-*.yml composes
- compose_registry.py: drop the 4 _entry blocks (50 -> 46 entries)
- test-compose-registry-disk.sh: count guards 50/51 -> 46/47
- profile_runtime.yml + patches.yml: drop the 4 nvlink blocks/slug refs
- test-profiles-compat.sh: retire the C13/E3 NVLink-required-compose
scenarios (the estate NVLink-gating code is now dormant; cleanup tracked
as a follow-up, code retained)
- docs: switch.sh help, AGENTS.md, HARDWARE.md, UPSTREAM.md, patches/README.md,
fp8-mtp/dflash/dflash-noviz compose headers, per-model CHANGELOG removal
entry; add a docs/README.md pointer to `switch.sh --list` as the
authoritative registry-derived compose x slug matrix
Historical NVLink bench rows (JusefPol #31, danbedford #74/#92/#96) are
preserved in BENCHMARKS.md. Full test suite green (the lone
test-submit-bench failure is a worktree-isolation artifact — it needs
gitignored results/rebench/ fixtures absent from a fresh worktree; passes
on the working tree).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
`switch.sh --list` printed a flat `slug → full/path` list, so a reader
couldn't tell which model/topology a slug like `vllm/default` or
`vllm/minimal` was bound to without parsing the path. Group the output by
model, then topology, with the serving artifact (<quant>/<serving>.yml) in
the right column — so every slug's binding is visible at a glance.
Display-only: no slug renames, no registry/emit-contract change. model is
parsed from the variant's compose dir (field 2); topology/quant/serving from
the file field. Backward-compatible — slugs and their behavior are untouched.
Verified: switch/launch registry-parity + compose-disk/mounts tests pass;
full suite green except the pre-existing fixture-only test-submit-bench. No
test parses --list output.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Every gemma-4-31b vLLM compose pinned a purged Docker Hub nightly
(e47c98ef / bf610c2f), un-bootable for fresh users (#250, #167 class).
Prune the 9-variant set to 3 and repoint survivors onto the immutable
stable release tag vllm/vllm-openai:v0.21.0 (never :latest).
Survivors: vllm/gemma-mtp (bf16 dual, default), vllm/gemma-int8 (int8-PTH
dual), vllm/gemma-mtp-tp1 (fp8 single). Dropped: gemma-dflash,
gemma-dflash-int8, gemma-int8-tq3, gemma-bf16, gemma-awq; the 262K path
folds into gemma-int8 via a CTX env override. Gemma is decoupled from the
shared Qwen nightly profiles via a new vllm-gemma-stable engine profile;
Qwen nightly profiles are untouched.
Diagnose decoupling: the pruned gemma dflash patch dirs doubled as the
diagnose tool's cross-model disk-source proxies for two Qwen overlays
(vllm-pr41703-dflash, vllm-pr42102-dflash-kv-quant). Those capabilities
are image-baked in the pinned nightly, not mountable files, so mark them
image_baked and have diagnose skip the disk-source check for image-baked
overlays. Runtime-neutral (no Qwen compose mounts them).
switch.sh launcher: handle the new VLLM_IMAGE engine-pin export (the twin
loop in launch.sh had it; switch.sh did not -> "unexpected engine pin
export"), and guard the VLLM_NIGHTLY_SHA echo for the image-only stable
profile (was unbound under set -u).
Validated on 2x RTX 3090 (Ampere, v0.21.0): full scripts/tests/*.sh suite
35/0; live boots of gemma-mtp (bf16) and gemma-int8 serve coherent output,
qwen vllm/dual regression-clean. gemma-mtp-tp1 (fp8_e4m3) is correctly
SM-gated (required_sm=9.0) -- confirmed Ampere-incompatible (Triton
"fp8e4nv not supported on sm_86"), a documented 32 GB+/Hopper variant;
beellama remains the single-card gemma path on Ampere.
Refs #451, #250, #167.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
PR-B of the compose-quant-hierarchy refactor (follows PR-A #231). launch.sh no
longer hardcodes its variant/port/container/kvcalc maps — it derives them from
compose_registry.py via a shared emitter, the same single source of truth
switch.sh already uses. Adds a topology-aware <engine>/default resolver.
- scripts/lib/registry-emit.sh (new): shared emitter — emits switch/launch
tables from COMPOSE_REGISTRY, parses container_name from each compose, and
exposes registry_default_target() for <engine>[/<topology>]/default.
- launch.sh: drop the hardcoded `declare -A LAUNCH_*` maps; populate from the
emitter. PRIMARY_MODEL constant (default qwen3.6-27b) backs bare <engine>/default.
- switch.sh: drop inline derivation; resolve vllm/default, vllm/dual/default,
vllm/multi4/default via the shared emitter.
- compose_registry.py: add kvcalc_key to entries; tools/kv-calc.py aliases.
- bench-row-formatter.sh: DEFAULTS-aware compose_display (drops the now-dead
docker-compose.yml branch).
- tests: test-launch-registry-parity.sh + test-default-resolver.sh (new);
test-switch-registry-parity refreshed onto the shared emitter.
Default resolution is an alias layer over existing registry keys — no keys
renamed, no compose content changed. Side effect: the launcher<->registry drift
that left vllm/gemma-mtp pointing at a non-existent dual/.../fp8-mtp.yml is gone
(now derives the registry's bf16-mtp.yml).
Implemented via Codex per the PR-B brief; independently re-validated before
commit: 7/7 gates PASS (switch+launch parity, default-resolver, launch-compat,
registry-disk, mounts-resolve, kv-calc 22/22); live vllm/default->dual +
vllm/dual/default->dual both verify-full 8/8; leak-clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The help line showed the 262K native max while the registry max_ctx, the compose
default, and BENCHMARKS all use the #201 safe default of 200K. Sync the comment.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Follow-up to PR #227 (#437). The vision re-tune changed the compose defaults but
left the registry / switch.sh / BENCHMARKS describing stale values.
- compose_registry: llamacpp/mtp-vision max_ctx 49152 -> 150000 (matches the
re-tuned compose default; ik-llama/iq4ks-mtp-vision already correct at 163840).
switch.sh derives variants from the registry, so this also fixes its dynamic list.
- switch.sh usage comments: both vision variants now show the 1M-px default +
"full-res = override, lower ctx" note (were "49K" / bare "160K").
- BENCHMARKS: marked the superseded 2026-05-20 49K row; added two re-tuned vision
rows (llama 150K@1M-px fills 138K/561MB; ik 160K@1M-px fills 147K/503MB, graduated
EVAL #402 -> Production). Decode TPS carried from siblings (ctx-independent, not
re-benched); rows focus on the measured ctx-ceiling + VRAM + verify-stress [8/8].
ik vision was ALREADY registry-wired (parity test green); this is the data sync.
Refs #438.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
switch.sh sourced .env unconditionally (`set -a; source`), so a stale .env
`MODEL_DIR` clobbered an exported one — the opposite precedence to launch.sh
(which guards on `-z MODEL_DIR`) and to docker compose (shell env > .env file).
A user who `export MODEL_DIR=…` then ran switch/launch got the .env value
instead (reported on #187 by rhossack + camelCase).
Parse .env line-by-line: a variable already set in the environment wins;
values are taken literally (matches docker compose's own .env semantics, no
shell expansion); a trailing CR is stripped so Windows-editor CRLF .env files
work (#187). Also print `[switch] MODEL_DIR=…` so the resolved value + its
source is unambiguous.
Closes#425.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* Add llama.cpp bounded-thinking compose
* fix(structured-cot/llamacpp): ship no-underscore grammar variant + record on-rig validation
On-rig validation found the xgrammar deepseek-scratchpad.gbnf does NOT parse under
llama.cpp's GBNF — underscored rule names (line_char/line_chars) trip
"parse: error parsing grammar: expecting newline or end at _char", and llama-server
silently falls back to UNCONSTRAINED generation (FREE==FSM token counts — silent no-fire).
Fix: deepseek-scratchpad.llamacpp.gbnf — identical language, rule names with no
underscores (llama.cpp GBNF is [a-zA-Z0-9-] only). Validated 2026-05-24 on 1x 3090:
parses, fires (reasoning_content = PLAN/NOTE/VERDICT), coexists with --spec-type
draft-mtp (draft accept 0.66-0.88). FREE->FSM 2113->1009 tok (2.1x) and 934->331 (2.8x) @ temp 0.6.
Repointed the llama.cpp compose + READMEs + STRUCTURED_COT.md llama.cpp examples to the
variant; left the vLLM grammar + refs untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* docs(structured-cot): redact pre-existing /home/wasif absolute paths
Pre-commit leak scan caught host-local /home/wasif/structured-cot/runs/... paths in
the 'What's saved on disk' section — pre-existing (2026-04-30 doc), not introduced by
this branch. Redacted to host-local relative form per no-internal-paths-in-public-docs.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
The single-card MTP defaults were bumped 262144 -> 200000 (PR #201 + the ik
commit) after the "boots != fills" finding: 262K boots and passes verify-stress
but only fills ~125K before the FA-scratch-at-fill OOM; 200K fills ~183K with
~1.1 GB margin (see docs/CLIFFS.md). The docs lagged.
Corrected to 200K (the shipped, fillable default) across the Q4_K_M-MTP +
ik IQ4_KS single-card paths:
- README.md (quick-start + TL;DR robustness pitch)
- docs/SINGLE_CARD.md (cliff-warning row, config table, llamacpp/default + ik
narratives, fallback table, Cliff 2 explainer)
- docs/engines/IK_LLAMA.md (defaults + max-context override)
- models/qwen3.6-27b/llama-cpp/README.md (single/mtp.yml heading + workhorse line)
- scripts/switch.sh (--list/--help comment)
Also reconciled an older drift: llamacpp/default = llamacpp/mtp alias (collapsed
2026-05-22), Q4_K_M MTP, no vision, ~52/61 TPS -- several spots still described
it as the retired Q3_K_XL @ ~21 TPS + vision.
Left for a separate fill-test (smaller-quant / no-MTP headroom may genuinely
exceed 200K): the "going-to-262K" max-ctx sections in docs/engines/LLAMA_CPP.md
+ VLLM.md, and the Q3_K_XL legacy rows in models/qwen3.6-27b/llama-cpp/README.md.
Dual-card vLLM 262K, the model's 262144 max_position_embeddings, and the CLIFFS
"boots!=fills" explainer are correct as-is and untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The ik_llama composes shipped (#180) but were never registered, so they were
raw-`docker compose`-only — awkward now that ik IQ4_KS is the path we point
VRAM-tight / WSL single-card users to (its ~0.5-0.8 GB leaner footprint is the
one place ik's edge pays rent). Register the two stable variants in
compose_registry.py (→ switch.sh derives them) and add them to launch.sh's
variant maps (compose/model/engine/kvcalc/order/port/container) so the wizard
offers them and `switch.sh ik-llama/iq4ks-mtp` works like the others.
The experimental two-stage compose stays raw-compose-only until benched.
Verified: registry imports (45 entries), ik present in all 7 launch maps,
bash -n clean, composes config-clean.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
* refactor(llamacpp): collapse single-card composes 3→2 (default = mtp alias)
Mainline Q4_K_M reaches the full 262K with -ub 512 (validated: boot + serve +
verify-stress 7/7 + soak PASS), so the old vanilla Q3_K_XL docker-compose.yml no
longer earns a separate profile. Collapse the llama.cpp single-card tree to two
compose files:
- `llamacpp/default` is now an ALIAS for `llamacpp/mtp` (mtp.yml) — launch.sh +
compose_registry repointed. All references (estate-CLI default, tests, docs)
keep resolving; the variant name survives.
- Retired models/qwen3.6-27b/llama-cpp/compose/single/docker-compose.yml.
- mtp.yml: document the -ub 512 -> 262K recipe in the header.
- mtp-vision.yml: align with the ik vision profile — default 160K (was 49K) via
-ub 512, and pin --image-min-tokens 1024 / --image-max-tokens 4096 (mainline
previously left image bounds at the model default). Validated: full-res 2048^2
image @ ~22.2 GB / 24 (~2.4 GB headroom).
- switch.sh / README / SINGLE_CARD.md: descriptions + the dead docker-compose.yml
link repointed.
Note: on mainline llama.cpp the VRAM lever is -ub (not -b — mainline honors -ub
independently, unlike ik_llama which forces n_ubatch=n_batch).
Docs follow-up (deferred): SINGLE_CARD.md prose still describes `default` as the
old Q3_K_XL path in places; mtp.yml's "froggeric suppresses --reasoning off" note
is now disproven (froggeric v19 + --reasoning off works on mainline b9246).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* docs(engines): ik_llama coverage + MTP-merged-on-mainline corrections
- Correct stale "llama.cpp MTP via community PR / not merged" across
INFERENCE_ENGINES.md, LLAMA_CPP.md, engines/README.md — PR #22673 merged
on mainline 2026-05-16 (the b9246 image we bench is post-merge).
- Add ik_llama.cpp coverage that was missing: engine list / repo tree /
supported-models table (README), workload picker (SINGLE_CARD), choose
table (engines/README), model engines line (qwen3.6-27b/README).
- Document new ik_llama features: two-stage ngram+MTP (PR #1789), -vhad,
--recurrent-ckpt-mode, -ctk-first (IK_LLAMA.md, INFERENCE_ENGINES.md).
- IK_LLAMA gotcha: native template won the 8-pack A/B (103 vs 99, toolcall
tied) so the ik composes default native; froggeric stays vLLM-only.
- New compose iq4ks-two-stage.yml (ngram-mod + MTP via --spec-stage). Boots
+ two-stage spec-dec engages on cu13-server (validated 2026-05-22); perf
bench pending.
Review fixes on top of the drafted changes: repo-tree └→├, a 131K↔262K
self-contradiction in llama-cpp/README, a now-stale "mainline MTP still
open PR" bullet, and softened/attributed the unverified IQ5_KS PPL figures.
External PR links verified resolving; no internal paths leaked.
Docs drafted via the Qwen coding agent; reviewed, corrected, and the new
compose boot-validated by Claude.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* refactor(composes): promote native template default + bump llama-cpp context ceilings
llama-cpp/mtp.yml: default ctx 131K → 262K (full native), -ub 512
llama-cpp/mtp-vision.yml: default ctx 49K → 160K, -b/-ub 1024
ik-llama/mtp{,-vision}.yml: native template (froggeric v19 A/B'd, native
won 8-pack 103 vs 99), +'-vhad' (V-cache Hadamard), +'--recurrent-ckpt-
mode auto' (hybrid-aware MTP rejection), removed ipc:host (not needed)
Retire stale artifacts:
ik-llama/patches/froggeric-chat-template/ — native won, no longer mounted
llama-cpp/recipes/single-card-*.sh — host-binary scripts superseded by
compose files (canonical path since v0.8.0)
8-pack ship-gate: 99/150 (vs prior native 103/150) — within noise,
6/8 packs within ±1. Delta is cli-40 variance, not a flag effect
(-vhad and --recurrent-ckpt-mode are not output-quality levers).
* docs(composes): guard -np 1 with hardware-conditional rationale
Add inline ⚠ comment to all 5 single-card composes explaining why
-np 1 is intentional on a single 24 GB card (compute-bound, not
memory-bound — extra slots divide throughput, don't multiply it).
Also expose NP as an env var on the two mainline composes (was
hardcoded; ik composes already had ${NP:-1}).
The 5090/multi-GPU carve-out is explicit: 'on a higher-throughput
card or multi-GPU the trade may flip — re-validate before raising.'
Prevents future agents from blindly parallelizing slots on Ampere.
* fix(composes): move -np guard comment out of folded scalar
The ⚠ -np 1 rationale comment was inside 'command: >-' in all 5
single-card composes. YAML folded scalars treat # lines as content,
not comments — docker compose config failed on all 5 files.
Move the comment block to YAML-level (between volumes: and command:).
All 5 now pass 'docker compose -f <file> config'.
Lesson learned: always validate compose edits with
'docker compose -f <file> config >/dev/null' before committing.
* docs(FAQ): expand WSL2 section with GPU overhead guidance
The existing FAQ entry only mentioned TDR and expandable_segments
gotchas. Expanded to explain the ~1.3 GiB invisible WSL2 overhead,
how it affects each engine path differently (dual-card: noise,
single-card vLLM: one env var, single-card llama.cpp: lower ctx),
and provide a concrete VRAM budget table showing which composes
OOM on WSL2 vs which fit at defaults.
Key finding: mainline llama.cpp mtp.yml at 262K leaves only ~1.5 GB
headroom on Linux — WSL2 eats that to ~0.2 GB (instant OOM). Fix is
CTX_SIZE=131072 (drops to 20 GB, ~2.7 GB WSL2 headroom). ik_llama
composes (IQ4_KS, smaller weights) fit at defaults on WSL2.
No compose file changes — this is FAQ docs only. The llama.cpp
composes already expose CTX_SIZE as an env var override.
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
switch.sh now DERIVES its VARIANTS + VARIANT_DEFAULT_PORT tables from
compose_registry.py (the single source of truth) instead of a hardcoded
`declare -A` map that had drifted: 20 registered composes (incl.
vllm/dual-int8 — shipped as dual/int8.yml but unlaunchable, which cost a
real A/B a config pivot) were not launchable. All 42 registered composes
are now launchable; zero launcher-only ghosts.
New deterministic test-switch-registry-parity.sh (no docker/GPU/network)
fails CI on ANY registry↔launcher mismatch in EITHER direction: registry ⊆
launcher (zero registered-but-unlaunchable), launcher ⊆ registry (zero
ghosts) — driven through the FULL shipped `switch.sh --list` path so a
manual post-derivation ghost is caught too — plus spec parity, port parity,
and every resolved compose file exists on disk. Negative-case verified: a
synthetic registry/launcher mismatch makes the test exit 1.
Additive: no [C0]/decision-logic change; no shipped compose touched.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
As discussed in #98, collapse 8 Qwen dual-card compose files into 4.
NVLink presence is now detected at startup instead of requiring
a separate compose per interconnect. The 4 nvlink-*.yml files become
deprecated stubs that extend the unified compose with NVLINK_MODE=force_on.
Add entrypoint conditionals and env-var substitution so the same compose
selects the correct NCCL settings and --disable-custom-all-reduce flag based
on detected topology. Extend the same pattern to all 7 Gemma dual composes.
Catches the failure mode reported by alexpolo1 on Discord 2026-05-11:
switch.sh reports "no club-3090 container running" but the GPU is still
pinned at ~22 GiB from a non-managed process, and the new container
OOMs at boot with a cryptic vLLM ValueError.
Two changes:
1. Widen RUNNING_PATTERN from a hard-coded variant list to `^(vllm-|llama-cpp-)`
so down_running() also catches locally-built and one-off `docker run`
instances under the same image families.
2. Add gpu_preflight() between down_running() and up_variant():
- Queries nvidia-smi for free memory per GPU.
- If any card has <80% free (insufficient for the typical 0.92
gpu-memory-utilization), abort with a diagnostic listing the
holding PIDs from nvidia-smi --query-compute-apps and suggesting
specific cleanup commands.
- FORCE=1 env bypasses the check for users who know what they're doing.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Compose files now live under `<model>/<engine>/compose/<topology>/<file>.yml`,
with topology as a folder rather than a filename prefix. Solves all 7
inconsistencies surfaced in the post-rename audit (single-card composes
without `single-` prefix, unsuffixed `docker-compose.yml` ambiguity,
fine-tunes encoding model name in filename, etc.) by making the directory
hierarchy enforce the convention.
Layout:
models/<model>/<engine>/compose/<topology>/<feature>.yml
Where:
- <model>: qwen3.6-27b, gemma-4-31b
- <engine>: vllm, llama-cpp, sglang
- <topology>: single, dual, multi3, multi4, multi8
- <feature>: docker-compose.yml (default) | turbo.yml | dflash.yml | etc.
Each topology subdir has a `docker-compose.yml` for the recommended
starter — bare `cd <topology> && docker compose up` works because
docker compose finds that filename automatically. Variants drop the
`docker-compose.` prefix since they're invoked via `-f` flag.
27 compose file moves total:
- 18 Qwen vLLM composes redistributed across single/dual/multi4
- 2 Qwen llama-cpp composes into single/
- 6 Gemma vLLM composes redistributed across single/dual
- 1 untracked qwopus-bf16mtp moved to dual/
Inside each compose: relative paths to `../patches/` and `../cache/`
bumped to `../../patches/` / `../../cache/`, and `../../../../models-cache`
to `../../../../../models-cache` (one extra `..` for the new depth).
Reference updates across 148 files (BENCHMARKS, all docs, CHANGELOGs,
sibling-table cross-references in compose headers, scripts, patch
READMEs, .github issue templates, tools/residency-instrument).
scripts/switch.sh VARIANTS map updated; tags themselves unchanged
(`vllm/dual` → `dual/docker-compose.yml`, `vllm/dual4` → `multi4/docker-compose.yml`,
`vllm/gemma-mtp` → `gemma-4-31b/.../dual/docker-compose.yml`, etc.).
AGENTS.md "Compose layout" section rewritten to describe the new
hierarchy, with concrete examples and the fine-tune exception
(`dual/carnice-bf16mtp.yml` carries the fine-tune name as a filename
prefix until the fine-tune graduates to its own model directory).
All switch.sh paths verified to resolve to actual files post-move.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Documentation distinguishes 1 / 2 / 3+ GPUs as `single` / `dual` /
`multi`, with separate doc pages (SINGLE_CARD.md, DUAL_CARD.md,
MULTI_CARD.md). Compose filename topology prefix should match.
`dual4.yml` was an awkward outlier — it had `dual` in the prefix
but actually meant TP=4 (4-card config). The clean shape: GPU
count is implicit when there's no ambiguity (`single` always 1,
`dual` always 2), and explicit when there is (`multi3` / `multi4`
/ `multi8`).
Renames:
- docker-compose.dual4.yml → docker-compose.multi4.yml
- docker-compose.dual4-dflash.yml → docker-compose.multi4-dflash.yml
Registry tags `vllm/dual4` and `vllm/dual4-dflash` in scripts/switch.sh
keep their existing names (backward compat for users running
`bash scripts/switch.sh vllm/dual4`); only the file paths in the
VARIANTS map are updated.
References updated: BENCHMARKS.md, docs/MULTI_CARD.md, docs/UPSTREAM.md,
models/qwen3.6-27b/CHANGELOG.md, models/qwen3.6-27b/vllm/patches/README.md,
sibling-table cross-references in 7 other compose headers.
AGENTS.md "Topology prefix" row tightened: examples now read
`single · dual · multi3 · multi4 · multi8` (dropped `dual-nvlink`
which is actually an interconnect feature suffix; dropped `quad`
which doesn't exist as a name in our convention).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Rename the 4 shipped Gemma composes to drop the redundant `gemma-` prefix
(model is already named by the parent directory `models/gemma-4-31b/`):
- gemma-mtp.yml → dual.yml (recommended default, MTP implicit
— matches Qwen's `dual.yml` shape)
- gemma-mtp-int8.yml → dual-int8.yml (MTP + INT8 PTH KV variant)
- gemma-mtp-tp1.yml → single.yml (TP=1 single-card variant — boot
OOMs on Ampere 24 GB; for 32 GB+ only)
- gemma-dflash.yml → dual-dflash.yml (DFlash drafter variant — same shape
as Qwen's `dual-dflash.yml`)
Convention going forward (matches what Qwen has done since launch):
- File name: `<topology>-<feature>.yml`, model implied by parent directory
- Topology: `single` | `dual` | `dual-nvlink` | `dual4` | etc.
- Feature suffix: `-turbo` | `-dflash` | `-int8` | `-awq` | etc.
Registry tags in `scripts/switch.sh` keep their existing names
(`vllm/gemma-mtp`, `vllm/gemma-mtp-tp1`, `vllm/gemma-dflash`) — only the
file paths in the VARIANTS map are updated. This preserves backward
compat for users running `bash scripts/switch.sh vllm/gemma-mtp` etc.
Container names (`vllm-gemma-4-31b-mtp` etc.) are unchanged — they already
include the model name in the `vllm-<model>-<feature>` form, so no rename
needed there.
Plus continuing the at-a-glance profile schema rollout from 4d7356a:
profile blocks added to 9 more Qwen composes (bounded-thinking,
carnice-bf16mtp, dual4, dual4-dflash, dual-nvlink, dual-nvlink-turbo,
dual-nvlink-dflash, dual-nvlink-dflash-noviz, long-text). Each declares
Model / Topology / Drafter / KV / Vision / Max ctx / Genesis / Best-for.
References updated across BENCHMARKS.md, README.md, DUAL_CARD.md,
UPSTREAM.md, scripts/setup.sh, scripts/switch.sh, all 6 Gemma compose
cross-references, model patch READMEs, and codex-brief-dflash-int8.md.
Refs: noonghunna/club-3090#67
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Cross-rig data on z-lab/gemma-4-31B-it-DFlash block-diffusion drafter — first
Ampere consumer benchmark of DFlash on Gemma 4. PR #41703 was needs-rebase
against pre-SpecDecodeBaseProposer-refactor main; ChatGPT/Codex cherry-picked
the 6 PR commits onto upstream/main 5d0fd87038b cleanly with one manual fix
on top (_warn_if_multimodal → _raise_if_multimodal rename, otherwise
multimodal inputs throw NotImplementedError).
Bench at shipped n=7 (TP=2, 2× 3090 PCIe, 230W cap):
narrative: 95 wall TPS (1.56× over no-spec-decode baseline)
code: 168 wall TPS (2.74× over baseline)
Avg accept code: ~60%, AL 5.23
n-sweep summary (n=4..15): code TPS saturates at n=7; n=8 strictly dominated
by n=7 (worse on both narr+code); n=15 past the knee. Narrative monotonically
degrades with bigger n — n=5 is best for prose at 109/141, override hint
documented in compose comment for chat workloads.
Soak PASS: 100 turns, 0 errors, 0 silent-empty, 0 MiB growth, 98.6% TPS
retention, p50 decode 55.78 TPS (vs 52.71 at n=5 — n=7 is strictly better
under soak conditions too, with 2.2 GB lower peak VRAM).
DFlash vs MTP on Gemma 4: DFlash wins code (+18%), MTP wins narrative (+15%).
Different operating regimes — block-diffusion's larger draft horizon helps
deterministic code more than prose.
Adds:
- models/gemma-4-31b/vllm/compose/docker-compose.gemma-dflash.yml
- models/gemma-4-31b/vllm/patches/vllm-gemma4-dflash/ (12 RO-mounted Python
files + README documenting provenance + drop conditions)
- scripts/switch.sh entry: vllm/gemma-dflash → port 8032
- BENCHMARKS.md row under Gemma 4 31B section
Drop the entire patches dir + overlay block when PR #41703 merges and a
vLLM :nightly tag rebuilds against it.
Co-authored-by: noonghunna <[email protected]>
* add Gemma 4 31B + Google MTP drafter — first Ampere consumer cross-rig data
Adds models/gemma-4-31b/ tree with two compose variants (TP=2 + TP=1),
vendored vLLM PR #41745 overlay, switch.sh / launch.sh wiring, plus the
script-level extensions needed to make `bash scripts/{bench,verify-*,
soak-test}.sh` auto-detect Gemma containers.
Validated 2026-05-05 on 2× RTX 3090 (Ampere sm_86, PCIe-only):
- canonical bench (3 warm + 5 measured per prompt):
narrative wall TPS = 108.87 (CV 3.2%)
code wall TPS = 142.25 (CV 2.3%)
- soak-continuous (5 sessions × 5 turns × 4 prompts = 100 turns):
verdict PASS, 0 errors, 0 silent-empty, 0 MiB VRAM growth,
98.3% TPS retention, p50 decode 111.79
- VRAM 22.5 GB/card. Speedup vs no-spec-decode baseline: 1.79× narr / 2.31× code.
First published Ampere consumer numbers on Google's Gemma 4 MTP "assistant"
drafters (released 2026-05-05). Discussion #67 has the announcement +
upstream context. BENCHMARKS.md gets a new "Gemma 4 31B" section with
both TP=2 (working) and TP=1 (upstream-blocked on Ampere) rows.
Components landed:
- models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp.yml (TP=2)
- models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp-tp1.yml
(TP=1 — preserved for re-test when upstream Ampere fp8 path lands)
- models/gemma-4-31b/vllm/patches/vllm-gemma4-mtp/ — vendored overlay
of vllm-project/vllm#41745 (lucianommartins/gemma4-mtp). 7 modified
Python files RO-mounted over the stock nightly image. Same shape as
vllm-marlin-pad. Drop the entire tree when PR merges + propagates.
- scripts/switch.sh + launch.sh: vllm/gemma-mtp + vllm/gemma-mtp-tp1
variants registered. RUNNING_PATTERN extended for vllm-gemma-4-31b*.
- scripts/preflight.sh: autodetect_endpoint extended for gemma containers.
- scripts/soak-test.sh: container grep extended.
- scripts/soak-helper.py: SOAK_NO_CHAT_TEMPLATE_KWARGS=1 env knob to
skip the Qwen3-specific `chat_template_kwargs.enable_thinking` body
field for non-Qwen models that reject it (Gemma 4, etc.).
- BENCHMARKS.md: new Gemma 4 31B section with both TP rows.
- docs/UPSTREAM.md: PR #41745 row + the Ampere fp8-blocked finding +
transformers 5.8.0 dependency entry.
Pre-merge dependencies (drop when both land):
1. vllm-project/vllm#41745 → drop the patches/ tree + the volume block
2. transformers ≥ 5.8.0 → drop the entrypoint pip install line
Out of scope:
- TP=1 single-card. Tested + upstream-blocked on Ampere consumer:
fp8 KV needed for fit; fp8_e4m3 hits Triton "fp8e4nv not supported"
on sm_86; fp8_e5m2 rejected by gemma4_mm.py:1336 allowlist. Compose
is preserved with the failing config baked in for future re-test.
- Gemma 4 26B-A4B MoE single-card. Active params ~4B → should fit
cleanly without fp8. Queued as the obvious follow-up.
- vllm/vllm-openai:gemma4-0505-cu129 image swap. The recipe-image is
Hopper-tagged and may not include sm_86 kernels; testing it would
collapse this whole 4-layer wrangle to a one-line image bump but
needs explicit Ampere validation first.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* models/gemma-4-31b/vllm/cache: gitignore + README (mirror qwen3.6 pattern)
Replaces the .gitkeep stubs with the standard .gitignore (ignore everything
except .gitignore + README.md) + README documenting the cache lifecycle.
Mirrors models/qwen3.6-27b/vllm/cache/ exactly.
Boot times this enables (validated tonight):
- cold first boot: ~3-7 min (TP=1 / TP=2)
- warm subsequent boot: ~2-3 min (cache hit)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Adds docker-compose.dual-nvlink-turbo.yml: NVLink + TurboQuant KV (TQ3) +
MTP n=3 + 4-stream + 262K. Mirrors current dual-turbo.yml (post-#59) with
the three NVLink-specific deltas applied:
- NCCL_P2P_LEVEL=NVL (vs NCCL_P2P_DISABLE=1 on PCIe)
- PYTORCH_CUDA_ALLOC_CONF without expandable_segments (JusefPol crash repro)
- --disable-custom-all-reduce removed (NVLink P2P → custom kernel wins)
Image pin matches master (nightly-01d4d1ad3); retired sidecars
(patch_workspace_lock_disable, patch_tolist_cudagraph) excluded — superseded
by Genesis v7.72.2 PN34 + P78 natives.
Bench (danbedford rig): 101.49 narr / 133.20 code wall TPS, 20.4 GB/card.
+12.6% narr / +10.7% code over own PCIe-only dual-turbo baseline.
A/B tested on the same rig. Bench was on v7.69 — re-bench welcomed on
the v7.72.2 pin this compose ships with.
Wired into scripts/switch.sh + scripts/launch.sh as
`vllm/dual-nvlink-turbo` on port 8017. Added BENCHMARKS row + updated
UPSTREAM tracker (Marlin pad PR row) to list the new compose.
Sibling-table standardization across the 7 other compose files dropped
per the original author's comment — that's better filed as a separate
issue scoped to the table format.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Dan <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Closes the UX gap raised by @snoby on #37 — previously, launching a
compose that needs the DFlash draft model without first running
WITH_DFLASH_DRAFT=1 setup.sh would fail with vLLM's generic
"Invalid repository ID or local directory" pydantic error.
Adds three new preflight functions in scripts/preflight.sh:
1. preflight_hf_token (soft warn) — wired into setup.sh before the model
download. Catches missing HF_TOKEN early and tells the user exactly
what to do (visit hf.co/settings/tokens, accept T&C, export the token).
Skip via PREFLIGHT_NO_HF_TOKEN=1.
2. preflight_compose_deps (hard error) — wired into switch.sh just before
docker compose up. Parses the target compose YAML for known model-path
patterns (qwen3.6-27b-dflash, qwen3.6-27b-mtp-head, qwen3.6-27b-autoround-int4)
and verifies each exists on host. If missing, emits a clear "Fix:" line
with the exact setup.sh invocation needed (e.g. WITH_DFLASH_DRAFT=1).
Refuses to proceed with exit 1. Skip via PREFLIGHT_NO_COMPOSE_DEPS=1.
3. preflight_kv_format_hint (soft warn) — wired into switch.sh as the last
pre-up check. Detects smallest VRAM via nvidia-smi; if <24 GB and the
target compose uses turboquant_3bit_nc KV, emits the @efschu finding
from #47 (TQ3 → fp8_e5m2 swap rule) with cross-link to docs/HARDWARE.md
+ a tools/kv-calc.py one-liner to predict the user's specific config.
Skip via PREFLIGHT_NO_KV_HINT=1.
Tested on this rig:
- HF_TOKEN unset → warns
- Missing DFlash dir → hard errors with WITH_DFLASH_DRAFT=1 hint
- Missing main model → hard errors with generic setup.sh hint
- 24 GB rig + TQ3 compose → silent (correct, TQ3 is right pick)
- Simulated 20 GB rig + TQ3 compose → fires the fp8_e5m2 hint
All preflights are individually skippable via PREFLIGHT_NO_<NAME>=1
env vars (matches existing PREFLIGHT_NO_FETCH=1 / PREFLIGHT_NO_GENESIS_PIN=1
patterns).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
After the full Phase 3 5-grammar A/B (HE+ 164 + LCB v6 50, n=214 problems
× 5 conditions = 1070 generations), bounded-thinking.yml is updated to
recommend the DeepSeek scratchpad grammar (PLAN/NOTE×0-15/VERDICT FSM at
tools/grammar-eval/deepseek-scratchpad.gbnf) as the default.
Phase 3 results (combined HE+ + LCB v6, all FSM-enforced):
- DeepSeek scratchpad: 87.4% Pass@1 (+1 net vs andthattoo, +4pp on LCB)
- andthattoo G/A/E: 86.9% (the originally-published technique)
- Holiday tagline: 86.4%
- PROMPT_TERSE (no FSM): 82.2% (Phase 2's n=30 win was subset-selection bias)
- FREE (no constraint): 78.0% (baseline)
Phase 1 reproducibility is exact: HE+ FSM Δ +4.3pp / LCB v6 Δ +24.0pp,
both match Phase 1's published numbers — validates the bench harness.
Compose ships unchanged engine-side (same vLLM image, same Genesis stack,
same TQ3 KV, same MTP n=3, same enable_in_reasoning flag). Only the
docstring's recommended-grammar pointer + the on-disk grammar files in
tools/grammar-eval/ change. Three grammars are now validated and available
client-side via extra_body={"structured_outputs": {"grammar": ...}}:
- DeepSeek scratchpad (default, best LCB)
- andthattoo G/A/E (originally-published, ~4× tighter think budget)
- Holiday tagline (extreme 24-token compression, wins LCB by 4pp too)
Combined-accuracy spread is within noise (0.5pp at n=214), so we ship one
compose rather than three siblings — choice is at the client, not at the
compose level.
Bug fix bundled: tools/grammar-eval/subset-bench.py --full --include-lcb
mode now correctly threads dataset kind through run_condition so LCB
problems use mod.run_tests_livecodebench instead of HE+ assertion-based
testing. Without this, the Phase 3 LCB shard crashed with KeyError:
'prompt' on the HE→LCB transition.
This wraps active research on bounded-thinking. Reopen if upstream FSM-
regress cluster behavior changes (Genesis pin bump, vLLM grammar engine
swap, model-family change), or if user demand surfaces for a sibling
compose pinning a non-default grammar.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Three calibration fixes to soak-test + two boot-progress improvements to
switch.sh, all from the first cross-rig soak runs on this rig today.
soak-test.sh — baseline timing
Brief said "capture warm baseline after first turn" but turn 3 ships a
12K-char tool-result paste that fills prefix cache by ~1000 MiB on the
first hit. With baseline at turn 1, every healthy config false-positives
with "growth > 200 MiB threshold" on session 1 alone. Fixed: capture
baseline at end of session 1 (after all 5 turn shapes run once). Sessions
2-N then measure real accretion from steady state.
soak-helper.py — decode_tps guard
Thinking-mode requests where vLLM bundles all reasoning into the terminal
streaming chunk produce wall ≈ ttft (no separate content delta visible).
Old code computed decode_tps = completion_tokens / (wall - ttft) with
wall - ttft ≈ 0, yielding ~2 billion TPS. Fixed: when ttft is None OR
wall - ttft < 100ms, report decode_tps = 0 (caller filterable).
soak-helper.py — TPS outlier filter in cmd_summary
Defensive — even if a future helper bug regresses, we filter unrealistic
decode_tps values (>500 t/s) from all summary computations. tps_retention
metric was previously being inflated by 2e9 outliers in early sessions
before the helper-bug landed mid-run.
Cross-rig validation results:
Config Baseline Max VRAM Growth Verdict
vllm/long-text 180K + 0.93 23316 23316 0 MiB PASS
vllm/long-vision 145K + 0.95 22778 22778 0 MiB PASS
Both configs are soak-clean under v1 synthetic shapes. Long-vision was
expected to FAIL based on issue #41 — but our 5-turn fixtures reset the
conversation each turn while hermes accumulates context across turns.
v1 catches a class of issue (raw VRAM accretion across requests); v2
fixtures will need session = continuous conversation to catch the
hermes-shaped class. Methodology limitation documented in CHANGELOG.
switch.sh — wait_ready crash detection + boot-stage progress
Closes the "container crashed silently in 2s, you wait 600s for nothing"
UX gap that surfaced today when long-text's MODEL_DIR was misconfigured.
- Crash detection: docker inspect -f '{{.State.Running}}' between polls;
if false, dump last 30 log lines + exit 1 immediately (5s vs 600s)
- Boot-stage markers: grep docker logs for "Resolved architecture",
"Loading weights", "Compilation finished", "Capturing CUDA graphs",
"Application startup complete" — surface one line per phase transition
so the wait isn't silent.
Validated end-to-end on long-vision boot today: visible progress at
60s / 68s / 80s / 120s / 176s / 196s = ready.
Two-part addition for the most common stale-setup pattern: user cloned
weeks ago, master has moved (Genesis pin bumps, compose changes, vendored
patch updates), they re-run their compose, hit a stale config, and file
an issue we already solved on master. Wispborne's _register_op_once and
GuiPerPT's pre-pull boot OOM both surfaced through this loop.
scripts/preflight.sh — preflight_repo_drift:
- Skips silently if not a git repo, on a non-master branch, or if
PREFLIGHT_NO_FETCH=1 (offline rigs / CI / forks tracking elsewhere).
- Verifies origin remote is noonghunna/club-3090 (avoids false positives
on forks pointing elsewhere).
- timeout 5 git fetch --quiet origin master — bounded so flaky networks
don't block boot.
- On behind > 0: WARN with commit count, last-fetch age (h/d), and the
one-line fix command. Soft-warning, never blocks. Tells user about
PREFLIGHT_NO_FETCH=1 for opting out.
Wired into both launch.sh and switch.sh, alongside the existing
preflight_genesis_pin so users get one consolidated stale-setup signal.
scripts/update.sh — the easy upgrade path:
- Refuses on dirty tree (git status --porcelain) — surfaces the local
edits and tells the user to commit or stash first. We don't clobber
the rare user who's been editing a compose locally.
- Refuses on non-master branch — feature branches and fork-trackers
should pull manually; this script is the master-from-origin path.
- git pull --ff-only — no merge commits, no rebase ambiguity. Diverged
branches get an explicit error pointing at git pull --rebase.
- Re-runs setup.sh (idempotent — re-pins Genesis, re-vendors Marlin).
- Tells the user to restart their variant via switch.sh — doesn't auto-
restart, so they can A/B old-vs-new if they want.
- --dry-run shows the plan without changing anything.
- --force re-runs setup.sh even when up-to-date (for "I edited Genesis
by hand and want it re-pinned" cases).
Why detection-then-explicit-command instead of "press y to auto-update":
the user will rarely have local commits (they're consumers of the recipes,
not vLLM contributors), but we still want consent — they should see what
they're committing to. The dirty-tree guard handles the rare custom-edit
case without nagging the common path.
Wires JusefPol's NVLink compose into the same machinery the rest of the
variants use:
- switch.sh — adds vllm/dual-nvlink to the usage doc, VARIANT_DEFAULT_PORT
(8014, matches the compose's PORT fallback), and the VARIANTS map. Now
'bash scripts/switch.sh vllm/dual-nvlink' works the same way 'vllm/dual'
does.
- launch.sh — adds the variant to LAUNCH_DEFAULT_PORT + LAUNCH_DEFAULT_CONTAINER
so 'bash scripts/launch.sh --variant vllm/dual-nvlink' resolves the
endpoint URL + container name correctly for post-launch verify.
Intentionally NOT added to the launch.sh interactive wizard's dual-card menu
(line 169). The wizard runs on every fresh setup, and offering an NVLink
option to a PCIe-only user would silently boot a config that crashes on
their topology. Users with NVLink can invoke explicitly via --variant.
Catches the failure mode @wispborne hit in #32: user pulled latest
club-3090 (which bumped GENESIS_PIN in scripts/setup.sh from old →
new), but didn't re-run setup.sh. The on-disk Genesis tree at
models/qwen3.6-27b/vllm/patches/genesis/ stays at the OLD pin while
setup.sh's declared pin advances. vLLM boots against the outdated
tree and hits mysterious patch failures (e.g. v7.66 PN25's
infer_schema crash that v7.69 PN25 worker-spawn registration fixed).
Adds new `preflight_genesis_pin` function to scripts/preflight.sh:
- Parses `GENESIS_PIN="${GENESIS_PIN:-<default>}"` from setup.sh to
extract the declared default value
- Reads on-disk HEAD via `git -C <genesis-dir> rev-parse --short HEAD`
- Compares (declared short-form vs on-disk short-form match)
- On mismatch: emits a [preflight] WARN block with both pins + a
"Fix: bash scripts/setup.sh qwen3.6-27b" hint
- Soft-warning only — does not block boot
Wired into both entry points:
- scripts/launch.sh (the wizard-driven path) — adds the check after
preflight_running, before the variant pick
- scripts/switch.sh (the direct stateless switcher) — adds the check
inside up_variant() right before the docker compose up call
Skips silently if the Genesis tree hasn't been cloned yet (caller
should run setup.sh first; preflight isn't the place to handle that
case). Also skips if setup.sh isn't present at the expected path
(weird state — silent skip rather than false warnings).
Tested locally:
- Current state (declared 2db18df = on-disk 2db18df): silent ✓
- bash -n passes on all three modified scripts
Coverage gap: users who bypass switch.sh + launch.sh and run
`docker compose -f docker-compose.X.yml up -d` directly won't get
the warning. That's the canonical-path-vs-bypass tradeoff; we lead
with switch.sh in the docs.
Closes the procedural follow-up flagged in
[#32 comment-4364895764](https://github.com/noonghunna/club-3090/issues/32#issuecomment-4364895764)
("setup.sh should be re-run after every git pull that bumps
GENESIS_PIN. We could add a check for this — compare setup.sh's
declared pin vs the on-disk tree's HEAD, warn if they differ.").
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
[@wispborne](https://github.com/wispborne) caught: docker-compose.long-text-no-mtp.yml
ships in the v7.69 cutover but its corresponding variant key wasn't added
to scripts/switch.sh's VARIANTS map or scripts/launch.sh's wizard list.
Result: \`bash scripts/switch.sh vllm/long-text-no-mtp\` failed and the
wizard never offered it.
Adds:
- scripts/switch.sh:
* VARIANT_DEFAULT_PORT[vllm/long-text-no-mtp]=8021 (matches compose)
* VARIANTS[vllm/long-text-no-mtp]= compose path
* RUNNING_PATTERN extended to match its container name
* Header docs updated: long-text 218K → 180K (Balanced MTP), new
long-text-no-mtp 200K (Max-context) line
- scripts/launch.sh:
* VLLM_OPTS wizard list — both Balanced MTP + Max-context surfaced
explicitly, with correct ctx ceilings (180K + 200K, not stale 218K)
* LAUNCH_DEFAULT_PORT + LAUNCH_DEFAULT_CONTAINER updated to match
Stale 218K references in the wizard description corrected to current
shipped values: long-text is 180K (after the 0.93 mem-util backoff +
v7.69 cutover), long-text-no-mtp is 200K + 0.95.
Reported in [club-3090 #32 comment-4364904521](https://github.com/noonghunna/club-3090/issues/32#issuecomment-4364904521).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Port andthattoo/structured-cot to our stack — Qwen3.6-27B AutoRound INT4
dense / 1× RTX 3090 / vLLM nightly + MTP n=3 + TQ3 KV. Re-benched on
full HumanEval+ 164 + LiveCodeBench v6 50.
Headline (max_tokens=4096, greedy):
- HumanEval+ 164: FSM 92.7% vs FREE 88.4% (+4.3pp), 30.7× compression
- LiveCodeBench v6 50: FSM 66.0% vs FREE 42.0% (+24pp), 26.2× compression
The +Δpp partly reflects FSM dodging the max_tokens=4096 truncation trap
rather than pure reasoning gain — see docs/STRUCTURED_COT.md "Honest
caveats" for the full picture.
Three port surprises worth keeping (all in docs):
1. vLLM dev205+ defaults StructuredOutputsConfig.enable_in_reasoning=False;
grammar mask only fires post-</think> unless overridden.
2. Legacy extra_body={"guided_grammar": ...} is silently dropped on
dev205+ (tip-off: identical FREE/FSM token counts). Use the new
structured_outputs.grammar field.
3. Qwen3.6 chat template auto-prefixes <think>\n; drop the leading
literal from upstream grammars when porting.
Files added:
- models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml
- docs/STRUCTURED_COT.md (public writeup)
- models/qwen3.6-27b/vllm/diagnostics/structured-cot-bench.md (internal)
Files updated:
- scripts/launch.sh wizard + scripts/switch.sh variant map
- models/qwen3.6-27b/README.md (recommended single-card list, patch surface)
- models/qwen3.6-27b/vllm/README.md (compose menu)
- docs/SINGLE_CARD.md (TL;DR table now four rows)
- CHANGELOG.md (new top entry)
Also reverts the long-text.yml experimental flag added during smoke
testing (the flag now lives only in bounded-thinking.yml) and adds the
Genesis pre-flight check we previously skipped on long-text.
Credit: andthattoo for the technique, the grammar files, and the eval
harness.
Closes / addresses 3 reported issues + adds requested feature:
#7 vid (PORT not honored, MODEL_DIR vs MODELS_DIR confusion):
- All 8 vLLM compose files now use "${PORT:-XXXX}:8000" so .env PORT
flows through. Defaults preserved per-variant (8020 single, 8010-8013
dual). llama.cpp composes already had this pattern.
- scripts/switch.sh: load .env early; per-variant default-port table;
new resolve_ready_url() picks PORT > variant default for the readiness
probe.
- scripts/launch.sh: same default-port table; final endpoint URL printed
to user reflects actual mapped port.
- .env.example: ⚠ box callout that variable names are CASE-SENSITIVE
(MODEL_DIR singular, NOT MODELS_DIR plural — silently ignored).
New PORT section documenting per-variant defaults.
#4 timxx (tools-text.yml fails "Free memory ... less than desired"):
- docs/FAQ.md: new entry "Container fails to start: Free memory..."
explaining the vLLM startup check, the two workarounds (free VRAM /
lower mem-util), and which configs hit it most often (0.97+ mem-util).
- Compose defaults unchanged (0.97 stays the right design target on
headless rigs); the FAQ documents the workaround for users with X11.
#1 fabriciomalta (per-config VRAM column):
- docs/SINGLE_CARD.md: TL;DR table now has VRAM column with mem-util.
- docs/DUAL_CARD.md: TL;DR table same + footnote explaining per-card
semantics and which dual configs would/wouldn't fit on 2× 20 GB cards
(relevant to fabriciomalta's 2× 3080-20GB use case).
#2 tenitram (empty responses) — fixed in master via aab8ff4
(P68/P69 disabled). Closed with reply pointing at the fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This was a control variant used internally to A/B-test whether MTP
worked without Genesis (it does, on fp8+MTP paths). No reason for
end users to pick it over tools-text.yml (fp8+MTP+Genesis fixes+75K,
strictly better) or minimal.yml (no Genesis at all, simplest).
Wizard already didn't surface it. switch.sh map, sibling compose
"see also" tables, patches/README, engines/VLLM.md all updated.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fast-chat (20K, fp8, vision) and default docker-compose.yml (48K, TQ3,
vision) had effectively the same TPS post-PN8. fast-chat's only
remaining differentiator was "smaller context = ~3s faster boot," and
20K is actively bad for IDE-agent users (Copilot tool-schema preamble
alone hits 20K). Net negative — removed.
Default compose was missed in the previous P68/P69 fix — it had the
same env vars enabled and the same silent-stop bug above 8000 chars.
Both now disabled with the same explanatory comment.
Updated:
- scripts/switch.sh, scripts/launch.sh — drop the variant
- docs/SINGLE_CARD.md, FAQ.md, engines/VLLM.md, model + vllm + patches
READMEs — references removed or pointed to default/tools-text
- All sibling compose YAML "see also" tables — fast-chat row removed,
tools-text row repurposed for IDE-agent guidance
- CHANGELOG entry; old historical entries kept as-is (append-only)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
