Dense 40B uncensored Qwen3.6 community merge, dual-3090 llama.cpp with
embedded MTP head (Q6_K GGUF, 31 GB). First dual llama.cpp compose in
the catalog; first 'category' field in the registry (uncensored).
Changes:
- Compose: models/qwen3.6-40b-deckard/llama-cpp/compose/dual/mtp-q6k/mtp.yml
Status: 🧪 Unverified (quality /150 + soak pending)
Config: -ngl 99 -ts 1,1 -fa on --spec-type draft-mtp --spec-draft-n-max 2
--cache-type-k q8_0 --cache-type-v q8_0 -c 131072
- Registry: llamacpp/deckard40B-dual-mtp (category=uncensored, experimental)
- DEFAULTS: (qwen3.6-40b-deckard, llamacpp, dual) → slug
- Model profile: scripts/lib/profiles/models/qwen3.6-40b-deckard.yml
- Drafter: qwen-mtp-builtin model_compat extended with qwen3.6-40b-deckard
- LiteLLM route: deckard-40b → :8199
- launch.sh: suggest_default_variant case for Deckard
- BENCHMARKS.md: new section with validated MTP n=2 numbers
- Test fixtures: registry count 43→44, disk 44→49, models 5→6
Wrinkle decisions (see PR description):
1. Dual llama.cpp: uses 'count: all' + '-ts 1,1' (tensor-split layer-split),
matching the validated serving config. No launcher changes needed — the
compose's deploy section handles GPU reservation directly.
2. Engine pin: uses rolling server-cuda tag (matching engine profile spec),
not the b9246 pin other composes use. The validated build (2026-06-09
digest 1c4ff61a) is newer than b9246 and has draft-mtp working. No engine
pin bump for other models.
NOT DONE (live validation — GPUs busy with quality eval):
- Boot/bench/soak/quality NOT run — maintainer to validate post-eval
- Status stays 🧪 until full gate passes
Rename vllm/gemma-int8→gemma-int8-mtp + vllm/gemma-mtp→gemma-bf16-mtp (both carry MTP n=4; only KV format differs); kv-calc alias fix. Ladder gemma-bf16-mtp default 32K→131K (measured BF16 KV pool 196,527 tok @ 0.95; verify-stress 8/8, NIAH→120K, bench 118.8/154.3, zero TPS regression). Codify the dual-card max-context-priority rule in DUAL_CARD.md. Suite 41/41.
Bump vllm/gemma-mtp (dual, bf16+MTP) to stable v0.22.0 (validated 5/5 coherent). Deprecate vllm/gemma-mtp-tp1: fp8 KV is hardware-impossible for Gemma 4 on Ampere sm_86 (fp8e4nv kernel unsupported; fp8_e5m2 rejected by gemma4 attention allowlist; nvfp4 Blackwell-only), live-confirmed on v0.22.0. Remove the dead DEFAULTS[(gemma-4-31b,vllm,single)] row; repoint launch/setup/preflight single-card Gemma -> beellama/gemma-dflash. Also fixes a pre-existing unterminated NEXT_STEPS_NOTE string in setup.sh's gemma-4-31b arm. Suite 38/38.
* feat(switch): model-default resolver + user-pinnable defaults
Add a two-layer default scheme on top of the existing <engine>/default map.
<engine>/default stays the maintainer's recommendation (read-only to users);
the new <model>/default is the user's preference — their .env pin if set,
else a curated pick for the detected topology.
compose_registry.py: two maintainer knobs next to DEFAULTS —
RECOMMENDED_DEFAULT_MODELS (short opt-in shortlist, not an exhaustive
ranking; new models are NOT auto-added) and ENGINE_PREFERENCE (per-topology
engine order). Plus resolver helpers (curated_default_target,
community_default_target stub, model_default_pin_key, engine_set/model_set,
slug_topology, model_of_slug).
registry-emit.sh: shared resolver model_default_target(root, model, topology)
— the single injection point for both launchers. Precedence ladder: --variant
(caller) -> .env pin -> community seam (None today) -> curated ENGINE_PREFERENCE
walk (skips non-functional (NA) slugs) -> degradation (notice + nearest-lower
topology, else a clear "pick explicitly" message; never crashes). Plus
x_default_dispatch: X/default with X in engine-set -> engine rec; X in
model-set -> model default; else error (engines + model-ids are disjoint).
switch.sh: <model>/default token; --set-default <slug> / --clear-default
<model> (round-trip the .env pin CLUB3090_DEFAULT_<MODELID>); a Defaults view
appended to --list (also standalone via --defaults) marking user-pin vs
curated. PR-A's grouping/markers/counts and the --force gate preserved.
launch.sh: bare invocation -> first installed shortlist model -> its
<model>/default (no full wizard); a pinned fast-path ("Launch your default
<slug>? [Y/n]"); a post-boot offer ("Make <slug> your default? [y/N]"). Any
narrowing flag keeps the explicit wizard path. Also load CLUB3090_DEFAULT_*
pin keys from .env even when MODEL_DIR is exported in the shell (the existing
.env loader is MODEL_DIR-gated, which would otherwise hide the pin).
Pin validation is warn + fall back, never blocking: unknown slug / wrong
model / topology-mismatch / (NA) status -> notice + curated default.
Tests: new test-model-default-resolver.sh (curated walk, (NA) skip,
degradation, X/default dispatch, pin override + all validation paths,
community seam skipped, .env round-trip); test-default-resolver.sh extended
with <model>/default launch.sh dispatch. Full suite green (only the
pre-existing test-submit-bench.sh fixture failure remains); 45 entries
unchanged; kv-calc calibration 17/17.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* docs: document the model-default resolver + user pins
Ship the user/contributor docs alongside the resolver code (repo convention —
update existing docs, no new top-level doc).
- README: single-card realign (ik-llama = fastest blessed single default,
llama.cpp = cliff-immune alternative — matches ENGINE_PREFERENCE single
order); add "pin your default" + <model>/default to Quick start.
- CLAUDE.md (= AGENTS.md): document RECOMMENDED_DEFAULT_MODELS +
ENGINE_PREFERENCE + the shared resolver as maintainer knobs next to DEFAULTS.
- FAQ: extend "switch to a different model" with <model>/default; new "How do
I set my own default config?" Q (two-layer model, --set-default/--clear-
default/--defaults, .env key, warn+fallback validation).
- SINGLE_CARD / DUAL_CARD: the resolver + the per-topology engine order; pin
hint.
- GETTING_STARTED: first-run uses <model>/default + the in-flow "set as
default?" prompt.
- ADDING_MODELS: a new model resolves via ENGINE_PREFERENCE; add a DEFAULTS row
per engine; RECOMMENDED_DEFAULT_MODELS is not auto-grown.
- UPSTREAM: beellama Docker-image row (gates beellama onboarding; the resolver
skips it today and it auto-promotes to single default on catalog).
- models/qwen3.6-27b/CHANGELOG: dated PR-B entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
* feat(registry): add slug health/availability flag
Add a lifecycle `status` to every registry slug so `switch.sh --list`,
launch, and switch are no longer blind to a compose's health. Previously
status lived only in compose-header comments, which drifted: the Genesis
dual compose declared "Working (with Genesis)" while its pin is parked and
it won't boot clean — a user could boot a broken slug unknowingly.
- compose_registry.py: `_entry()` gains keyword-only `status`
(default "production") + `status_note`, validated against the enum
(production/caveats/experimental/preview/upstream-gated/deprecated).
Add `compose_header_status()` mapping a compose's profile-schema
`Status:` emoji to that enum.
- Sweep every compose `Status:` header to a canonical enum value and
re-flag the non-functional slugs: all *genesis* + gemma-4-31b single
fp8 -> upstream-gated; carnice -> caveats; qwopus + qwen-a3b-preview
-> preview; bf16/int8 A/Bs, llamacpp bounded-thinking, PRISM/APEX eval
lanes, gemma-4-26b-a4b onboarding -> experimental; tq3-mtp -> deprecated.
- registry-emit.sh emits `status` + `status_note` as the last two VARIANT
fields; both loaders + the parity tests read the extended field list.
- switch.sh --list: status marker (caveats -> "(caveats)", the NA set ->
"(NA: <word>)"); model/topology grouping preserved. Launch/switch gate:
production launches, caveats launches with a notice, NA warns + requires
--force. launch.sh surfaces the flag before delegating to switch.sh.
- New drift-guard test test-compose-status-drift.sh: registry status in
enum, compose header maps to enum, and the two agree.
45 entries unchanged; kv-calc calibration 17/17; full suite green (only
the pre-existing test-submit-bench fixture failure remains).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* feat(switch): add model/variant counts to --list
Header line shows supported-model count + total variants with the health
split (N production · N caveats · N NA); each model group shows its variant
count. Widen the marker column so the (NA: …)/(caveats) markers align.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Every gemma-4-31b vLLM compose pinned a purged Docker Hub nightly
(e47c98ef / bf610c2f), un-bootable for fresh users (#250, #167 class).
Prune the 9-variant set to 3 and repoint survivors onto the immutable
stable release tag vllm/vllm-openai:v0.21.0 (never :latest).
Survivors: vllm/gemma-mtp (bf16 dual, default), vllm/gemma-int8 (int8-PTH
dual), vllm/gemma-mtp-tp1 (fp8 single). Dropped: gemma-dflash,
gemma-dflash-int8, gemma-int8-tq3, gemma-bf16, gemma-awq; the 262K path
folds into gemma-int8 via a CTX env override. Gemma is decoupled from the
shared Qwen nightly profiles via a new vllm-gemma-stable engine profile;
Qwen nightly profiles are untouched.
Diagnose decoupling: the pruned gemma dflash patch dirs doubled as the
diagnose tool's cross-model disk-source proxies for two Qwen overlays
(vllm-pr41703-dflash, vllm-pr42102-dflash-kv-quant). Those capabilities
are image-baked in the pinned nightly, not mountable files, so mark them
image_baked and have diagnose skip the disk-source check for image-baked
overlays. Runtime-neutral (no Qwen compose mounts them).
switch.sh launcher: handle the new VLLM_IMAGE engine-pin export (the twin
loop in launch.sh had it; switch.sh did not -> "unexpected engine pin
export"), and guard the VLLM_NIGHTLY_SHA echo for the image-only stable
profile (was unbound under set -u).
Validated on 2x RTX 3090 (Ampere, v0.21.0): full scripts/tests/*.sh suite
35/0; live boots of gemma-mtp (bf16) and gemma-int8 serve coherent output,
qwen vllm/dual regression-clean. gemma-mtp-tp1 (fp8_e4m3) is correctly
SM-gated (required_sm=9.0) -- confirmed Ampere-incompatible (Triton
"fp8e4nv not supported on sm_86"), a documented 32 GB+/Hopper variant;
beellama remains the single-card gemma path on Ampere.
Refs #451, #250, #167.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
PR-B of the compose-quant-hierarchy refactor (follows PR-A #231). launch.sh no
longer hardcodes its variant/port/container/kvcalc maps — it derives them from
compose_registry.py via a shared emitter, the same single source of truth
switch.sh already uses. Adds a topology-aware <engine>/default resolver.
- scripts/lib/registry-emit.sh (new): shared emitter — emits switch/launch
tables from COMPOSE_REGISTRY, parses container_name from each compose, and
exposes registry_default_target() for <engine>[/<topology>]/default.
- launch.sh: drop the hardcoded `declare -A LAUNCH_*` maps; populate from the
emitter. PRIMARY_MODEL constant (default qwen3.6-27b) backs bare <engine>/default.
- switch.sh: drop inline derivation; resolve vllm/default, vllm/dual/default,
vllm/multi4/default via the shared emitter.
- compose_registry.py: add kvcalc_key to entries; tools/kv-calc.py aliases.
- bench-row-formatter.sh: DEFAULTS-aware compose_display (drops the now-dead
docker-compose.yml branch).
- tests: test-launch-registry-parity.sh + test-default-resolver.sh (new);
test-switch-registry-parity refreshed onto the shared emitter.
Default resolution is an alias layer over existing registry keys — no keys
renamed, no compose content changed. Side effect: the launcher<->registry drift
that left vllm/gemma-mtp pointing at a non-existent dual/.../fp8-mtp.yml is gone
(now derives the registry's bf16-mtp.yml).
Implemented via Codex per the PR-B brief; independently re-validated before
commit: 7/7 gates PASS (switch+launch parity, default-resolver, launch-compat,
registry-disk, mounts-resolve, kv-calc 22/22); live vllm/default->dual +
vllm/dual/default->dual both verify-full 8/8; leak-clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* Add llama.cpp bounded-thinking compose
* fix(structured-cot/llamacpp): ship no-underscore grammar variant + record on-rig validation
On-rig validation found the xgrammar deepseek-scratchpad.gbnf does NOT parse under
llama.cpp's GBNF — underscored rule names (line_char/line_chars) trip
"parse: error parsing grammar: expecting newline or end at _char", and llama-server
silently falls back to UNCONSTRAINED generation (FREE==FSM token counts — silent no-fire).
Fix: deepseek-scratchpad.llamacpp.gbnf — identical language, rule names with no
underscores (llama.cpp GBNF is [a-zA-Z0-9-] only). Validated 2026-05-24 on 1x 3090:
parses, fires (reasoning_content = PLAN/NOTE/VERDICT), coexists with --spec-type
draft-mtp (draft accept 0.66-0.88). FREE->FSM 2113->1009 tok (2.1x) and 934->331 (2.8x) @ temp 0.6.
Repointed the llama.cpp compose + READMEs + STRUCTURED_COT.md llama.cpp examples to the
variant; left the vLLM grammar + refs untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* docs(structured-cot): redact pre-existing /home/wasif absolute paths
Pre-commit leak scan caught host-local /home/wasif/structured-cot/runs/... paths in
the 'What's saved on disk' section — pre-existing (2026-04-30 doc), not introduced by
this branch. Redacted to host-local relative form per no-internal-paths-in-public-docs.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
All llama.cpp + ik single composes shared one default container_name, so a
second variant hit a docker "name in use" conflict (ESTATE_CONTAINER override
existed but was undocumented). Give each non-base variant a distinct default:
- llamacpp/mtp → llama-cpp-qwen36-27b (base, unchanged)
- llamacpp/mtp-vision → llama-cpp-qwen36-27b-vision
- ik/iq4ks-mtp → ik-llama-qwen36-27b (base, unchanged)
- ik/iq4ks-mtp-vision → ik-llama-qwen36-27b-vision
- ik/iq4ks-two-stage → ik-llama-qwen36-27b-two-stage
The base names (mtp / iq4ks-mtp = the default workhorses) are kept so existing
hardcoded refs and benchlocal `docker logs <base>` keep working.
launch.sh: track the two renamed variants in LAUNCH_DEFAULT_CONTAINER, and add
the missing ik-llama/iq4ks-two-stage entry (it fell back to the wrong default
container before). gpu-mode.sh: comment-only update for the new naming.
Audit (all clear): the autodetect globs in preflight/soak/switch/health/update
are prefix-anchored (`^llama-cpp-qwen36-27b`, no end anchor) and gpu-mode uses
`== llama-cpp-*`, so the suffixed names still match — verified.
Ports still default to 8020 (concurrency uses ESTATE_PORT, per the issue's own
example) — documented with a "run two variants at once" snippet in SINGLE_CARD.
Validated: `docker compose config` resolves each distinct name; all names still
match the autodetect glob; bash -n + test-launch-compat pass.
Closes#169.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The #1 first-run stumble (e.g. #187) is having models outside the repo tree
(D: drive, ~/models, NAS) while MODEL_DIR points at the in-repo default — the
wizard then says "not found under <dir>" with no hint that MODEL_DIR is the fix.
Both not-found errors now suggest pointing MODEL_DIR at the existing weights via
a one-line .env (which launch.sh, switch.sh, and docker compose all read).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
The ik_llama composes shipped (#180) but were never registered, so they were
raw-`docker compose`-only — awkward now that ik IQ4_KS is the path we point
VRAM-tight / WSL single-card users to (its ~0.5-0.8 GB leaner footprint is the
one place ik's edge pays rent). Register the two stable variants in
compose_registry.py (→ switch.sh derives them) and add them to launch.sh's
variant maps (compose/model/engine/kvcalc/order/port/container) so the wizard
offers them and `switch.sh ik-llama/iq4ks-mtp` works like the others.
The experimental two-stage compose stays raw-compose-only until benched.
Verified: registry imports (45 entries), ik present in all 7 launch maps,
bash -n clean, composes config-clean.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
The wizard's suggest_default_variant() pointed single-card qwen3.6-27b at
vllm/long-text, which is doubly dead: its pinned vLLM nightly was purged from
Docker Hub (#167) and it hits single-card Cliff 2b. Repoint to llamacpp/default
(full 262K, cliff-immune, no nightly dependency) — matching what README,
docs/SINGLE_CARD.md, and docs/INFERENCE_ENGINES.md already recommend. vLLM
single-card users can still select vllm/tools-text explicitly.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
* refactor(llamacpp): collapse single-card composes 3→2 (default = mtp alias)
Mainline Q4_K_M reaches the full 262K with -ub 512 (validated: boot + serve +
verify-stress 7/7 + soak PASS), so the old vanilla Q3_K_XL docker-compose.yml no
longer earns a separate profile. Collapse the llama.cpp single-card tree to two
compose files:
- `llamacpp/default` is now an ALIAS for `llamacpp/mtp` (mtp.yml) — launch.sh +
compose_registry repointed. All references (estate-CLI default, tests, docs)
keep resolving; the variant name survives.
- Retired models/qwen3.6-27b/llama-cpp/compose/single/docker-compose.yml.
- mtp.yml: document the -ub 512 -> 262K recipe in the header.
- mtp-vision.yml: align with the ik vision profile — default 160K (was 49K) via
-ub 512, and pin --image-min-tokens 1024 / --image-max-tokens 4096 (mainline
previously left image bounds at the model default). Validated: full-res 2048^2
image @ ~22.2 GB / 24 (~2.4 GB headroom).
- switch.sh / README / SINGLE_CARD.md: descriptions + the dead docker-compose.yml
link repointed.
Note: on mainline llama.cpp the VRAM lever is -ub (not -b — mainline honors -ub
independently, unlike ik_llama which forces n_ubatch=n_batch).
Docs follow-up (deferred): SINGLE_CARD.md prose still describes `default` as the
old Q3_K_XL path in places; mtp.yml's "froggeric suppresses --reasoning off" note
is now disproven (froggeric v19 + --reasoning off works on mainline b9246).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* docs(engines): ik_llama coverage + MTP-merged-on-mainline corrections
- Correct stale "llama.cpp MTP via community PR / not merged" across
INFERENCE_ENGINES.md, LLAMA_CPP.md, engines/README.md — PR #22673 merged
on mainline 2026-05-16 (the b9246 image we bench is post-merge).
- Add ik_llama.cpp coverage that was missing: engine list / repo tree /
supported-models table (README), workload picker (SINGLE_CARD), choose
table (engines/README), model engines line (qwen3.6-27b/README).
- Document new ik_llama features: two-stage ngram+MTP (PR #1789), -vhad,
--recurrent-ckpt-mode, -ctk-first (IK_LLAMA.md, INFERENCE_ENGINES.md).
- IK_LLAMA gotcha: native template won the 8-pack A/B (103 vs 99, toolcall
tied) so the ik composes default native; froggeric stays vLLM-only.
- New compose iq4ks-two-stage.yml (ngram-mod + MTP via --spec-stage). Boots
+ two-stage spec-dec engages on cu13-server (validated 2026-05-22); perf
bench pending.
Review fixes on top of the drafted changes: repo-tree └→├, a 131K↔262K
self-contradiction in llama-cpp/README, a now-stale "mainline MTP still
open PR" bullet, and softened/attributed the unverified IQ5_KS PPL figures.
External PR links verified resolving; no internal paths leaked.
Docs drafted via the Qwen coding agent; reviewed, corrected, and the new
compose boot-validated by Claude.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* refactor(composes): promote native template default + bump llama-cpp context ceilings
llama-cpp/mtp.yml: default ctx 131K → 262K (full native), -ub 512
llama-cpp/mtp-vision.yml: default ctx 49K → 160K, -b/-ub 1024
ik-llama/mtp{,-vision}.yml: native template (froggeric v19 A/B'd, native
won 8-pack 103 vs 99), +'-vhad' (V-cache Hadamard), +'--recurrent-ckpt-
mode auto' (hybrid-aware MTP rejection), removed ipc:host (not needed)
Retire stale artifacts:
ik-llama/patches/froggeric-chat-template/ — native won, no longer mounted
llama-cpp/recipes/single-card-*.sh — host-binary scripts superseded by
compose files (canonical path since v0.8.0)
8-pack ship-gate: 99/150 (vs prior native 103/150) — within noise,
6/8 packs within ±1. Delta is cli-40 variance, not a flag effect
(-vhad and --recurrent-ckpt-mode are not output-quality levers).
* docs(composes): guard -np 1 with hardware-conditional rationale
Add inline ⚠ comment to all 5 single-card composes explaining why
-np 1 is intentional on a single 24 GB card (compute-bound, not
memory-bound — extra slots divide throughput, don't multiply it).
Also expose NP as an env var on the two mainline composes (was
hardcoded; ik composes already had ${NP:-1}).
The 5090/multi-GPU carve-out is explicit: 'on a higher-throughput
card or multi-GPU the trade may flip — re-validate before raising.'
Prevents future agents from blindly parallelizing slots on Ampere.
* fix(composes): move -np guard comment out of folded scalar
The ⚠ -np 1 rationale comment was inside 'command: >-' in all 5
single-card composes. YAML folded scalars treat # lines as content,
not comments — docker compose config failed on all 5 files.
Move the comment block to YAML-level (between volumes: and command:).
All 5 now pass 'docker compose -f <file> config'.
Lesson learned: always validate compose edits with
'docker compose -f <file> config >/dev/null' before committing.
* docs(FAQ): expand WSL2 section with GPU overhead guidance
The existing FAQ entry only mentioned TDR and expandable_segments
gotchas. Expanded to explain the ~1.3 GiB invisible WSL2 overhead,
how it affects each engine path differently (dual-card: noise,
single-card vLLM: one env var, single-card llama.cpp: lower ctx),
and provide a concrete VRAM budget table showing which composes
OOM on WSL2 vs which fit at defaults.
Key finding: mainline llama.cpp mtp.yml at 262K leaves only ~1.5 GB
headroom on Linux — WSL2 eats that to ~0.2 GB (instant OOM). Fix is
CTX_SIZE=131072 (drops to 20 GB, ~2.7 GB WSL2 headroom). ik_llama
composes (IQ4_KS, smaller weights) fit at defaults on WSL2.
No compose file changes — this is FAQ docs only. The llama.cpp
composes already expose CTX_SIZE as an env var override.
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
launch.sh formats kv-calc.py JSON dot-decimals ("9.0") with printf
"%.2f". Under a comma-decimal LC_NUMERIC locale (de_DE etc.) bash
printf rejects the dot — `printf: 9.0: Ungültige Zahl` — and the
launcher aborts at the VRAM-budget print. Export LC_NUMERIC=C early
(LC_CTYPE/encoding untouched, UTF-8 UI glyphs still render). Reported
by @chrischd83 (German locale) in discussion #159.
bash -n clean; export precedes all executable printf; LC_NUMERIC=C
printf "%.2f" 9.0 -> 9.00 verified. Exact de_DE failure not
reproducible on our rig (no comma-decimal locale data installed) —
mechanism is textbook; reporter on de_DE is the definitive validator.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
As discussed in #98, collapse 8 Qwen dual-card compose files into 4.
NVLink presence is now detected at startup instead of requiring
a separate compose per interconnect. The 4 nvlink-*.yml files become
deprecated stubs that extend the unified compose with NVLINK_MODE=force_on.
Add entrypoint conditionals and env-var substitution so the same compose
selects the correct NCCL settings and --disable-custom-all-reduce flag based
on detected topology. Extend the same pattern to all 7 Gemma dual composes.
The two `read -rp` loops in `choose()` and the variant-selection step
spun infinitely when stdin closed mid-prompt (piped input shorter than
the wizard asks, or test/CI invocations). Now both loops check `read`'s
exit code; on EOF, print a clear message and SIGINT the parent so the
whole process exits cleanly (exit 130).
Doesn't affect interactive users — they type real input. Affects only
non-TTY piped invocations of the wizard, which should use --variant
<name> to skip the wizard entirely.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* add Gemma 4 31B + Google MTP drafter — first Ampere consumer cross-rig data
Adds models/gemma-4-31b/ tree with two compose variants (TP=2 + TP=1),
vendored vLLM PR #41745 overlay, switch.sh / launch.sh wiring, plus the
script-level extensions needed to make `bash scripts/{bench,verify-*,
soak-test}.sh` auto-detect Gemma containers.
Validated 2026-05-05 on 2× RTX 3090 (Ampere sm_86, PCIe-only):
- canonical bench (3 warm + 5 measured per prompt):
narrative wall TPS = 108.87 (CV 3.2%)
code wall TPS = 142.25 (CV 2.3%)
- soak-continuous (5 sessions × 5 turns × 4 prompts = 100 turns):
verdict PASS, 0 errors, 0 silent-empty, 0 MiB VRAM growth,
98.3% TPS retention, p50 decode 111.79
- VRAM 22.5 GB/card. Speedup vs no-spec-decode baseline: 1.79× narr / 2.31× code.
First published Ampere consumer numbers on Google's Gemma 4 MTP "assistant"
drafters (released 2026-05-05). Discussion #67 has the announcement +
upstream context. BENCHMARKS.md gets a new "Gemma 4 31B" section with
both TP=2 (working) and TP=1 (upstream-blocked on Ampere) rows.
Components landed:
- models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp.yml (TP=2)
- models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp-tp1.yml
(TP=1 — preserved for re-test when upstream Ampere fp8 path lands)
- models/gemma-4-31b/vllm/patches/vllm-gemma4-mtp/ — vendored overlay
of vllm-project/vllm#41745 (lucianommartins/gemma4-mtp). 7 modified
Python files RO-mounted over the stock nightly image. Same shape as
vllm-marlin-pad. Drop the entire tree when PR merges + propagates.
- scripts/switch.sh + launch.sh: vllm/gemma-mtp + vllm/gemma-mtp-tp1
variants registered. RUNNING_PATTERN extended for vllm-gemma-4-31b*.
- scripts/preflight.sh: autodetect_endpoint extended for gemma containers.
- scripts/soak-test.sh: container grep extended.
- scripts/soak-helper.py: SOAK_NO_CHAT_TEMPLATE_KWARGS=1 env knob to
skip the Qwen3-specific `chat_template_kwargs.enable_thinking` body
field for non-Qwen models that reject it (Gemma 4, etc.).
- BENCHMARKS.md: new Gemma 4 31B section with both TP rows.
- docs/UPSTREAM.md: PR #41745 row + the Ampere fp8-blocked finding +
transformers 5.8.0 dependency entry.
Pre-merge dependencies (drop when both land):
1. vllm-project/vllm#41745 → drop the patches/ tree + the volume block
2. transformers ≥ 5.8.0 → drop the entrypoint pip install line
Out of scope:
- TP=1 single-card. Tested + upstream-blocked on Ampere consumer:
fp8 KV needed for fit; fp8_e4m3 hits Triton "fp8e4nv not supported"
on sm_86; fp8_e5m2 rejected by gemma4_mm.py:1336 allowlist. Compose
is preserved with the failing config baked in for future re-test.
- Gemma 4 26B-A4B MoE single-card. Active params ~4B → should fit
cleanly without fp8. Queued as the obvious follow-up.
- vllm/vllm-openai:gemma4-0505-cu129 image swap. The recipe-image is
Hopper-tagged and may not include sm_86 kernels; testing it would
collapse this whole 4-layer wrangle to a one-line image bump but
needs explicit Ampere validation first.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* models/gemma-4-31b/vllm/cache: gitignore + README (mirror qwen3.6 pattern)
Replaces the .gitkeep stubs with the standard .gitignore (ignore everything
except .gitignore + README.md) + README documenting the cache lifecycle.
Mirrors models/qwen3.6-27b/vllm/cache/ exactly.
Boot times this enables (validated tonight):
- cold first boot: ~3-7 min (TP=1 / TP=2)
- warm subsequent boot: ~2-3 min (cache hit)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Adds docker-compose.dual-nvlink-turbo.yml: NVLink + TurboQuant KV (TQ3) +
MTP n=3 + 4-stream + 262K. Mirrors current dual-turbo.yml (post-#59) with
the three NVLink-specific deltas applied:
- NCCL_P2P_LEVEL=NVL (vs NCCL_P2P_DISABLE=1 on PCIe)
- PYTORCH_CUDA_ALLOC_CONF without expandable_segments (JusefPol crash repro)
- --disable-custom-all-reduce removed (NVLink P2P → custom kernel wins)
Image pin matches master (nightly-01d4d1ad3); retired sidecars
(patch_workspace_lock_disable, patch_tolist_cudagraph) excluded — superseded
by Genesis v7.72.2 PN34 + P78 natives.
Bench (danbedford rig): 101.49 narr / 133.20 code wall TPS, 20.4 GB/card.
+12.6% narr / +10.7% code over own PCIe-only dual-turbo baseline.
A/B tested on the same rig. Bench was on v7.69 — re-bench welcomed on
the v7.72.2 pin this compose ships with.
Wired into scripts/switch.sh + scripts/launch.sh as
`vllm/dual-nvlink-turbo` on port 8017. Added BENCHMARKS row + updated
UPSTREAM tracker (Marlin pad PR row) to list the new compose.
Sibling-table standardization across the 7 other compose files dropped
per the original author's comment — that's better filed as a separate
issue scoped to the table format.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Dan <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
After the full Phase 3 5-grammar A/B (HE+ 164 + LCB v6 50, n=214 problems
× 5 conditions = 1070 generations), bounded-thinking.yml is updated to
recommend the DeepSeek scratchpad grammar (PLAN/NOTE×0-15/VERDICT FSM at
tools/grammar-eval/deepseek-scratchpad.gbnf) as the default.
Phase 3 results (combined HE+ + LCB v6, all FSM-enforced):
- DeepSeek scratchpad: 87.4% Pass@1 (+1 net vs andthattoo, +4pp on LCB)
- andthattoo G/A/E: 86.9% (the originally-published technique)
- Holiday tagline: 86.4%
- PROMPT_TERSE (no FSM): 82.2% (Phase 2's n=30 win was subset-selection bias)
- FREE (no constraint): 78.0% (baseline)
Phase 1 reproducibility is exact: HE+ FSM Δ +4.3pp / LCB v6 Δ +24.0pp,
both match Phase 1's published numbers — validates the bench harness.
Compose ships unchanged engine-side (same vLLM image, same Genesis stack,
same TQ3 KV, same MTP n=3, same enable_in_reasoning flag). Only the
docstring's recommended-grammar pointer + the on-disk grammar files in
tools/grammar-eval/ change. Three grammars are now validated and available
client-side via extra_body={"structured_outputs": {"grammar": ...}}:
- DeepSeek scratchpad (default, best LCB)
- andthattoo G/A/E (originally-published, ~4× tighter think budget)
- Holiday tagline (extreme 24-token compression, wins LCB by 4pp too)
Combined-accuracy spread is within noise (0.5pp at n=214), so we ship one
compose rather than three siblings — choice is at the client, not at the
compose level.
Bug fix bundled: tools/grammar-eval/subset-bench.py --full --include-lcb
mode now correctly threads dataset kind through run_condition so LCB
problems use mod.run_tests_livecodebench instead of HE+ assertion-based
testing. Without this, the Phase 3 LCB shard crashed with KeyError:
'prompt' on the HE→LCB transition.
This wraps active research on bounded-thinking. Reopen if upstream FSM-
regress cluster behavior changes (Genesis pin bump, vLLM grammar engine
swap, model-family change), or if user demand surfaces for a sibling
compose pinning a non-default grammar.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Two-part addition for the most common stale-setup pattern: user cloned
weeks ago, master has moved (Genesis pin bumps, compose changes, vendored
patch updates), they re-run their compose, hit a stale config, and file
an issue we already solved on master. Wispborne's _register_op_once and
GuiPerPT's pre-pull boot OOM both surfaced through this loop.
scripts/preflight.sh — preflight_repo_drift:
- Skips silently if not a git repo, on a non-master branch, or if
PREFLIGHT_NO_FETCH=1 (offline rigs / CI / forks tracking elsewhere).
- Verifies origin remote is noonghunna/club-3090 (avoids false positives
on forks pointing elsewhere).
- timeout 5 git fetch --quiet origin master — bounded so flaky networks
don't block boot.
- On behind > 0: WARN with commit count, last-fetch age (h/d), and the
one-line fix command. Soft-warning, never blocks. Tells user about
PREFLIGHT_NO_FETCH=1 for opting out.
Wired into both launch.sh and switch.sh, alongside the existing
preflight_genesis_pin so users get one consolidated stale-setup signal.
scripts/update.sh — the easy upgrade path:
- Refuses on dirty tree (git status --porcelain) — surfaces the local
edits and tells the user to commit or stash first. We don't clobber
the rare user who's been editing a compose locally.
- Refuses on non-master branch — feature branches and fork-trackers
should pull manually; this script is the master-from-origin path.
- git pull --ff-only — no merge commits, no rebase ambiguity. Diverged
branches get an explicit error pointing at git pull --rebase.
- Re-runs setup.sh (idempotent — re-pins Genesis, re-vendors Marlin).
- Tells the user to restart their variant via switch.sh — doesn't auto-
restart, so they can A/B old-vs-new if they want.
- --dry-run shows the plan without changing anything.
- --force re-runs setup.sh even when up-to-date (for "I edited Genesis
by hand and want it re-pinned" cases).
Why detection-then-explicit-command instead of "press y to auto-update":
the user will rarely have local commits (they're consumers of the recipes,
not vLLM contributors), but we still want consent — they should see what
they're committing to. The dirty-tree guard handles the rare custom-edit
case without nagging the common path.
Wires JusefPol's NVLink compose into the same machinery the rest of the
variants use:
- switch.sh — adds vllm/dual-nvlink to the usage doc, VARIANT_DEFAULT_PORT
(8014, matches the compose's PORT fallback), and the VARIANTS map. Now
'bash scripts/switch.sh vllm/dual-nvlink' works the same way 'vllm/dual'
does.
- launch.sh — adds the variant to LAUNCH_DEFAULT_PORT + LAUNCH_DEFAULT_CONTAINER
so 'bash scripts/launch.sh --variant vllm/dual-nvlink' resolves the
endpoint URL + container name correctly for post-launch verify.
Intentionally NOT added to the launch.sh interactive wizard's dual-card menu
(line 169). The wizard runs on every fresh setup, and offering an NVLink
option to a PCIe-only user would silently boot a config that crashes on
their topology. Users with NVLink can invoke explicitly via --variant.
Catches the failure mode @wispborne hit in #32: user pulled latest
club-3090 (which bumped GENESIS_PIN in scripts/setup.sh from old →
new), but didn't re-run setup.sh. The on-disk Genesis tree at
models/qwen3.6-27b/vllm/patches/genesis/ stays at the OLD pin while
setup.sh's declared pin advances. vLLM boots against the outdated
tree and hits mysterious patch failures (e.g. v7.66 PN25's
infer_schema crash that v7.69 PN25 worker-spawn registration fixed).
Adds new `preflight_genesis_pin` function to scripts/preflight.sh:
- Parses `GENESIS_PIN="${GENESIS_PIN:-<default>}"` from setup.sh to
extract the declared default value
- Reads on-disk HEAD via `git -C <genesis-dir> rev-parse --short HEAD`
- Compares (declared short-form vs on-disk short-form match)
- On mismatch: emits a [preflight] WARN block with both pins + a
"Fix: bash scripts/setup.sh qwen3.6-27b" hint
- Soft-warning only — does not block boot
Wired into both entry points:
- scripts/launch.sh (the wizard-driven path) — adds the check after
preflight_running, before the variant pick
- scripts/switch.sh (the direct stateless switcher) — adds the check
inside up_variant() right before the docker compose up call
Skips silently if the Genesis tree hasn't been cloned yet (caller
should run setup.sh first; preflight isn't the place to handle that
case). Also skips if setup.sh isn't present at the expected path
(weird state — silent skip rather than false warnings).
Tested locally:
- Current state (declared 2db18df = on-disk 2db18df): silent ✓
- bash -n passes on all three modified scripts
Coverage gap: users who bypass switch.sh + launch.sh and run
`docker compose -f docker-compose.X.yml up -d` directly won't get
the warning. That's the canonical-path-vs-bypass tradeoff; we lead
with switch.sh in the docs.
Closes the procedural follow-up flagged in
[#32 comment-4364895764](https://github.com/noonghunna/club-3090/issues/32#issuecomment-4364895764)
("setup.sh should be re-run after every git pull that bumps
GENESIS_PIN. We could add a check for this — compare setup.sh's
declared pin vs the on-disk tree's HEAD, warn if they differ.").
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
[@wispborne](https://github.com/wispborne) caught: docker-compose.long-text-no-mtp.yml
ships in the v7.69 cutover but its corresponding variant key wasn't added
to scripts/switch.sh's VARIANTS map or scripts/launch.sh's wizard list.
Result: \`bash scripts/switch.sh vllm/long-text-no-mtp\` failed and the
wizard never offered it.
Adds:
- scripts/switch.sh:
* VARIANT_DEFAULT_PORT[vllm/long-text-no-mtp]=8021 (matches compose)
* VARIANTS[vllm/long-text-no-mtp]= compose path
* RUNNING_PATTERN extended to match its container name
* Header docs updated: long-text 218K → 180K (Balanced MTP), new
long-text-no-mtp 200K (Max-context) line
- scripts/launch.sh:
* VLLM_OPTS wizard list — both Balanced MTP + Max-context surfaced
explicitly, with correct ctx ceilings (180K + 200K, not stale 218K)
* LAUNCH_DEFAULT_PORT + LAUNCH_DEFAULT_CONTAINER updated to match
Stale 218K references in the wizard description corrected to current
shipped values: long-text is 180K (after the 0.93 mem-util backoff +
v7.69 cutover), long-text-no-mtp is 200K + 0.95.
Reported in [club-3090 #32 comment-4364904521](https://github.com/noonghunna/club-3090/issues/32#issuecomment-4364904521).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
AlexCPU's report: launch.sh selected the dual variant correctly,
switch.sh booted the stack on port 8010 with container
'vllm-qwen36-27b-dual', then ran verify-full.sh without passing URL
or CONTAINER. verify-full fell back to defaults (URL=
http://localhost:8020, CONTAINER=vllm-qwen36-27b) and reported 6/8
checks failed because it was hitting the wrong endpoint.
Fix: move the port-resolution block above the verify call, add a
per-variant container-name mapping mirroring the port mapping, and
pass URL + CONTAINER through to bash $VERIFY. Also surfaces the
resolved values in the launcher output so users can see what the
verify run is actually targeting.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Qwen3.6 thinks before answering by default, so a "Capital of France?"
smoke with max_tokens=30 returns truncated mid-`<think>` content. apnar
hit this on a working stack (verify-full.sh all green) and wasted time
debugging a non-bug.
Bump all 7 user-facing curl examples to max_tokens=200 (covers a typical
think block + the one-sentence answer with headroom).
verify-full.sh / verify.sh / verify-stress.sh stay at max_tokens=30
because they already pass chat_template_kwargs.enable_thinking=false,
which skips the think block entirely.
EXAMPLES.md gets an inline note explaining the headroom + the alternative
(disable thinking via chat_template_kwargs) for users who want a tighter
smoke.
Port andthattoo/structured-cot to our stack — Qwen3.6-27B AutoRound INT4
dense / 1× RTX 3090 / vLLM nightly + MTP n=3 + TQ3 KV. Re-benched on
full HumanEval+ 164 + LiveCodeBench v6 50.
Headline (max_tokens=4096, greedy):
- HumanEval+ 164: FSM 92.7% vs FREE 88.4% (+4.3pp), 30.7× compression
- LiveCodeBench v6 50: FSM 66.0% vs FREE 42.0% (+24pp), 26.2× compression
The +Δpp partly reflects FSM dodging the max_tokens=4096 truncation trap
rather than pure reasoning gain — see docs/STRUCTURED_COT.md "Honest
caveats" for the full picture.
Three port surprises worth keeping (all in docs):
1. vLLM dev205+ defaults StructuredOutputsConfig.enable_in_reasoning=False;
grammar mask only fires post-</think> unless overridden.
2. Legacy extra_body={"guided_grammar": ...} is silently dropped on
dev205+ (tip-off: identical FREE/FSM token counts). Use the new
structured_outputs.grammar field.
3. Qwen3.6 chat template auto-prefixes <think>\n; drop the leading
literal from upstream grammars when porting.
Files added:
- models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml
- docs/STRUCTURED_COT.md (public writeup)
- models/qwen3.6-27b/vllm/diagnostics/structured-cot-bench.md (internal)
Files updated:
- scripts/launch.sh wizard + scripts/switch.sh variant map
- models/qwen3.6-27b/README.md (recommended single-card list, patch surface)
- models/qwen3.6-27b/vllm/README.md (compose menu)
- docs/SINGLE_CARD.md (TL;DR table now four rows)
- CHANGELOG.md (new top entry)
Also reverts the long-text.yml experimental flag added during smoke
testing (the flag now lives only in bounded-thinking.yml) and adds the
Genesis pre-flight check we previously skipped on long-text.
Credit: andthattoo for the technique, the grammar files, and the eval
harness.
Closes / addresses 3 reported issues + adds requested feature:
#7 vid (PORT not honored, MODEL_DIR vs MODELS_DIR confusion):
- All 8 vLLM compose files now use "${PORT:-XXXX}:8000" so .env PORT
flows through. Defaults preserved per-variant (8020 single, 8010-8013
dual). llama.cpp composes already had this pattern.
- scripts/switch.sh: load .env early; per-variant default-port table;
new resolve_ready_url() picks PORT > variant default for the readiness
probe.
- scripts/launch.sh: same default-port table; final endpoint URL printed
to user reflects actual mapped port.
- .env.example: ⚠ box callout that variable names are CASE-SENSITIVE
(MODEL_DIR singular, NOT MODELS_DIR plural — silently ignored).
New PORT section documenting per-variant defaults.
#4 timxx (tools-text.yml fails "Free memory ... less than desired"):
- docs/FAQ.md: new entry "Container fails to start: Free memory..."
explaining the vLLM startup check, the two workarounds (free VRAM /
lower mem-util), and which configs hit it most often (0.97+ mem-util).
- Compose defaults unchanged (0.97 stays the right design target on
headless rigs); the FAQ documents the workaround for users with X11.
#1 fabriciomalta (per-config VRAM column):
- docs/SINGLE_CARD.md: TL;DR table now has VRAM column with mem-util.
- docs/DUAL_CARD.md: TL;DR table same + footnote explaining per-card
semantics and which dual configs would/wouldn't fit on 2× 20 GB cards
(relevant to fabriciomalta's 2× 3080-20GB use case).
#2 tenitram (empty responses) — fixed in master via aab8ff4
(P68/P69 disabled). Closed with reply pointing at the fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
User feedback: the small-ctx variants (48K default, tools-text 75K, minimal
32K) "offer very little context and not many people will find that as
viable options." Now that Cliff 1 is closed on the long-* variants via the
PN12 anchor sidecar, they're strictly more useful than the 48K/75K
alternatives for the workloads most users come for. The single residual
limitation is Cliff 2 on single-prompt >50K, addressed by llama.cpp.
SINGLE_CARD.md:
- TL;DR table reduced to 3 recommended options (long-vision · long-text ·
llamacpp/default).
- Cliff 2 caveat promoted to a prominent ⚠️ callout right under the table —
the one limitation users need to know.
- Old per-variant sections folded; small-ctx variants moved to an
"Other variants in the repo" section as fallback / diagnostic.
scripts/launch.sh:
- Wizard leads with the 3 primary options (long-vision · long-text ·
llamacpp/default). Diagnostic / niche options bundled at the end with a
"[fallback]" prefix so they don't dominate the menu.
models/qwen3.6-27b/README.md:
- Single-card recommended-options bullet list now leads with the 3 primary
variants and explicitly names Cliff 2 as the single shipped limitation.
docs/EXAMPLES.md:
- Cline section: stop pointing at tools-text; long-* now handle Cline's
tool returns. Cliff 2 is the only remaining caveat to flag.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fast-chat (20K, fp8, vision) and default docker-compose.yml (48K, TQ3,
vision) had effectively the same TPS post-PN8. fast-chat's only
remaining differentiator was "smaller context = ~3s faster boot," and
20K is actively bad for IDE-agent users (Copilot tool-schema preamble
alone hits 20K). Net negative — removed.
Default compose was missed in the previous P68/P69 fix — it had the
same env vars enabled and the same silent-stop bug above 8000 chars.
Both now disabled with the same explanatory comment.
Updated:
- scripts/switch.sh, scripts/launch.sh — drop the variant
- docs/SINGLE_CARD.md, FAQ.md, engines/VLLM.md, model + vllm + patches
READMEs — references removed or pointed to default/tools-text
- All sibling compose YAML "see also" tables — fast-chat row removed,
tools-text row repurposed for IDE-agent guidance
- CHANGELOG entry; old historical entries kept as-is (append-only)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
- scripts/preflight.sh (new) — sourceable library: docker, GPU >= N,
disk free, GPU-idle warning, running-container note. Each error has
an actionable Fix: hint instead of a cryptic mid-run crash.
- scripts/setup.sh + scripts/launch.sh wire pre-flight in early.
launch.sh adds --no-preflight escape hatch.
- launch.sh wizard inverted: cards → workload → auto-pick engine.
Newcomers can answer "how many GPUs" and "what do I want to do" but
rarely "vLLM or llama.cpp" — engine falls out of the pick with a
one-paragraph why. --engine override still works (filters the
workload list to that engine).
- Embedded charts swapped SVG → PNG in README + SINGLE_CARD +
DUAL_CARD + qwen3.6-27b/README. Clicking a PNG on GitHub opens a
viewable image; SVGs open as raw XML. SVG remains the editable
source — re-export PNG when SVG changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
scripts/switch.sh — stateless engine/variant switcher. Brings down
whatever's running (any vllm-qwen36-27b* or llama-cpp-qwen36-27b*
container — discovers compose file via docker labels), brings up
the new variant, waits for /v1/models to respond. Supports --list
(show all 13 variants) and --down (stop without booting). Variant
names are <engine>/<file-stem>: vllm/default, vllm/dual, vllm/
dual-turbo, vllm/dual-dflash, vllm/dual-dflash-noviz, vllm/long-
vision, vllm/long-text, vllm/fast-chat, vllm/tools-text, vllm/
no-genesis-mtp, vllm/minimal, llamacpp/default, llamacpp/concurrent.
scripts/launch.sh — interactive wizard for first-run users. Asks
engine → cards → workload, maps to variant, calls switch.sh, then
runs verify-full.sh to confirm clean serving. Also accepts flags
for non-interactive use:
bash scripts/launch.sh --variant vllm/default
bash scripts/launch.sh --engine vllm --cards 1 (asks the rest)
README.md — quick-start replaces "cd into compose dir + docker
compose up" with `bash scripts/launch.sh`. Click-throughs from the
launch tweet land in a guided flow instead of having to find the
right compose file by hand.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
noonghunna