switch.sh now DERIVES its VARIANTS + VARIANT_DEFAULT_PORT tables from
compose_registry.py (the single source of truth) instead of a hardcoded
`declare -A` map that had drifted: 20 registered composes (incl.
vllm/dual-int8 — shipped as dual/int8.yml but unlaunchable, which cost a
real A/B a config pivot) were not launchable. All 42 registered composes
are now launchable; zero launcher-only ghosts.
New deterministic test-switch-registry-parity.sh (no docker/GPU/network)
fails CI on ANY registry↔launcher mismatch in EITHER direction: registry ⊆
launcher (zero registered-but-unlaunchable), launcher ⊆ registry (zero
ghosts) — driven through the FULL shipped `switch.sh --list` path so a
manual post-derivation ghost is caught too — plus spec parity, port parity,
and every resolved compose file exists on disk. Negative-case verified: a
synthetic registry/launcher mismatch makes the test exit 1.
Additive: no [C0]/decision-logic change; no shipped compose touched.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
As discussed in #98, collapse 8 Qwen dual-card compose files into 4.
NVLink presence is now detected at startup instead of requiring
a separate compose per interconnect. The 4 nvlink-*.yml files become
deprecated stubs that extend the unified compose with NVLINK_MODE=force_on.
Add entrypoint conditionals and env-var substitution so the same compose
selects the correct NCCL settings and --disable-custom-all-reduce flag based
on detected topology. Extend the same pattern to all 7 Gemma dual composes.
Catches the failure mode reported by alexpolo1 on Discord 2026-05-11:
switch.sh reports "no club-3090 container running" but the GPU is still
pinned at ~22 GiB from a non-managed process, and the new container
OOMs at boot with a cryptic vLLM ValueError.
Two changes:
1. Widen RUNNING_PATTERN from a hard-coded variant list to `^(vllm-|llama-cpp-)`
so down_running() also catches locally-built and one-off `docker run`
instances under the same image families.
2. Add gpu_preflight() between down_running() and up_variant():
- Queries nvidia-smi for free memory per GPU.
- If any card has <80% free (insufficient for the typical 0.92
gpu-memory-utilization), abort with a diagnostic listing the
holding PIDs from nvidia-smi --query-compute-apps and suggesting
specific cleanup commands.
- FORCE=1 env bypasses the check for users who know what they're doing.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Compose files now live under `<model>/<engine>/compose/<topology>/<file>.yml`,
with topology as a folder rather than a filename prefix. Solves all 7
inconsistencies surfaced in the post-rename audit (single-card composes
without `single-` prefix, unsuffixed `docker-compose.yml` ambiguity,
fine-tunes encoding model name in filename, etc.) by making the directory
hierarchy enforce the convention.
Layout:
models/<model>/<engine>/compose/<topology>/<feature>.yml
Where:
- <model>: qwen3.6-27b, gemma-4-31b
- <engine>: vllm, llama-cpp, sglang
- <topology>: single, dual, multi3, multi4, multi8
- <feature>: docker-compose.yml (default) | turbo.yml | dflash.yml | etc.
Each topology subdir has a `docker-compose.yml` for the recommended
starter — bare `cd <topology> && docker compose up` works because
docker compose finds that filename automatically. Variants drop the
`docker-compose.` prefix since they're invoked via `-f` flag.
27 compose file moves total:
- 18 Qwen vLLM composes redistributed across single/dual/multi4
- 2 Qwen llama-cpp composes into single/
- 6 Gemma vLLM composes redistributed across single/dual
- 1 untracked qwopus-bf16mtp moved to dual/
Inside each compose: relative paths to `../patches/` and `../cache/`
bumped to `../../patches/` / `../../cache/`, and `../../../../models-cache`
to `../../../../../models-cache` (one extra `..` for the new depth).
Reference updates across 148 files (BENCHMARKS, all docs, CHANGELOGs,
sibling-table cross-references in compose headers, scripts, patch
READMEs, .github issue templates, tools/residency-instrument).
scripts/switch.sh VARIANTS map updated; tags themselves unchanged
(`vllm/dual` → `dual/docker-compose.yml`, `vllm/dual4` → `multi4/docker-compose.yml`,
`vllm/gemma-mtp` → `gemma-4-31b/.../dual/docker-compose.yml`, etc.).
AGENTS.md "Compose layout" section rewritten to describe the new
hierarchy, with concrete examples and the fine-tune exception
(`dual/carnice-bf16mtp.yml` carries the fine-tune name as a filename
prefix until the fine-tune graduates to its own model directory).
All switch.sh paths verified to resolve to actual files post-move.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Documentation distinguishes 1 / 2 / 3+ GPUs as `single` / `dual` /
`multi`, with separate doc pages (SINGLE_CARD.md, DUAL_CARD.md,
MULTI_CARD.md). Compose filename topology prefix should match.
`dual4.yml` was an awkward outlier — it had `dual` in the prefix
but actually meant TP=4 (4-card config). The clean shape: GPU
count is implicit when there's no ambiguity (`single` always 1,
`dual` always 2), and explicit when there is (`multi3` / `multi4`
/ `multi8`).
Renames:
- docker-compose.dual4.yml → docker-compose.multi4.yml
- docker-compose.dual4-dflash.yml → docker-compose.multi4-dflash.yml
Registry tags `vllm/dual4` and `vllm/dual4-dflash` in scripts/switch.sh
keep their existing names (backward compat for users running
`bash scripts/switch.sh vllm/dual4`); only the file paths in the
VARIANTS map are updated.
References updated: BENCHMARKS.md, docs/MULTI_CARD.md, docs/UPSTREAM.md,
models/qwen3.6-27b/CHANGELOG.md, models/qwen3.6-27b/vllm/patches/README.md,
sibling-table cross-references in 7 other compose headers.
AGENTS.md "Topology prefix" row tightened: examples now read
`single · dual · multi3 · multi4 · multi8` (dropped `dual-nvlink`
which is actually an interconnect feature suffix; dropped `quad`
which doesn't exist as a name in our convention).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Rename the 4 shipped Gemma composes to drop the redundant `gemma-` prefix
(model is already named by the parent directory `models/gemma-4-31b/`):
- gemma-mtp.yml → dual.yml (recommended default, MTP implicit
— matches Qwen's `dual.yml` shape)
- gemma-mtp-int8.yml → dual-int8.yml (MTP + INT8 PTH KV variant)
- gemma-mtp-tp1.yml → single.yml (TP=1 single-card variant — boot
OOMs on Ampere 24 GB; for 32 GB+ only)
- gemma-dflash.yml → dual-dflash.yml (DFlash drafter variant — same shape
as Qwen's `dual-dflash.yml`)
Convention going forward (matches what Qwen has done since launch):
- File name: `<topology>-<feature>.yml`, model implied by parent directory
- Topology: `single` | `dual` | `dual-nvlink` | `dual4` | etc.
- Feature suffix: `-turbo` | `-dflash` | `-int8` | `-awq` | etc.
Registry tags in `scripts/switch.sh` keep their existing names
(`vllm/gemma-mtp`, `vllm/gemma-mtp-tp1`, `vllm/gemma-dflash`) — only the
file paths in the VARIANTS map are updated. This preserves backward
compat for users running `bash scripts/switch.sh vllm/gemma-mtp` etc.
Container names (`vllm-gemma-4-31b-mtp` etc.) are unchanged — they already
include the model name in the `vllm-<model>-<feature>` form, so no rename
needed there.
Plus continuing the at-a-glance profile schema rollout from 4d7356a:
profile blocks added to 9 more Qwen composes (bounded-thinking,
carnice-bf16mtp, dual4, dual4-dflash, dual-nvlink, dual-nvlink-turbo,
dual-nvlink-dflash, dual-nvlink-dflash-noviz, long-text). Each declares
Model / Topology / Drafter / KV / Vision / Max ctx / Genesis / Best-for.
References updated across BENCHMARKS.md, README.md, DUAL_CARD.md,
UPSTREAM.md, scripts/setup.sh, scripts/switch.sh, all 6 Gemma compose
cross-references, model patch READMEs, and codex-brief-dflash-int8.md.
Refs: noonghunna/club-3090#67
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Cross-rig data on z-lab/gemma-4-31B-it-DFlash block-diffusion drafter — first
Ampere consumer benchmark of DFlash on Gemma 4. PR #41703 was needs-rebase
against pre-SpecDecodeBaseProposer-refactor main; ChatGPT/Codex cherry-picked
the 6 PR commits onto upstream/main 5d0fd87038b cleanly with one manual fix
on top (_warn_if_multimodal → _raise_if_multimodal rename, otherwise
multimodal inputs throw NotImplementedError).
Bench at shipped n=7 (TP=2, 2× 3090 PCIe, 230W cap):
narrative: 95 wall TPS (1.56× over no-spec-decode baseline)
code: 168 wall TPS (2.74× over baseline)
Avg accept code: ~60%, AL 5.23
n-sweep summary (n=4..15): code TPS saturates at n=7; n=8 strictly dominated
by n=7 (worse on both narr+code); n=15 past the knee. Narrative monotonically
degrades with bigger n — n=5 is best for prose at 109/141, override hint
documented in compose comment for chat workloads.
Soak PASS: 100 turns, 0 errors, 0 silent-empty, 0 MiB growth, 98.6% TPS
retention, p50 decode 55.78 TPS (vs 52.71 at n=5 — n=7 is strictly better
under soak conditions too, with 2.2 GB lower peak VRAM).
DFlash vs MTP on Gemma 4: DFlash wins code (+18%), MTP wins narrative (+15%).
Different operating regimes — block-diffusion's larger draft horizon helps
deterministic code more than prose.
Adds:
- models/gemma-4-31b/vllm/compose/docker-compose.gemma-dflash.yml
- models/gemma-4-31b/vllm/patches/vllm-gemma4-dflash/ (12 RO-mounted Python
files + README documenting provenance + drop conditions)
- scripts/switch.sh entry: vllm/gemma-dflash → port 8032
- BENCHMARKS.md row under Gemma 4 31B section
Drop the entire patches dir + overlay block when PR #41703 merges and a
vLLM :nightly tag rebuilds against it.
Co-authored-by: noonghunna <[email protected]>
* add Gemma 4 31B + Google MTP drafter — first Ampere consumer cross-rig data
Adds models/gemma-4-31b/ tree with two compose variants (TP=2 + TP=1),
vendored vLLM PR #41745 overlay, switch.sh / launch.sh wiring, plus the
script-level extensions needed to make `bash scripts/{bench,verify-*,
soak-test}.sh` auto-detect Gemma containers.
Validated 2026-05-05 on 2× RTX 3090 (Ampere sm_86, PCIe-only):
- canonical bench (3 warm + 5 measured per prompt):
narrative wall TPS = 108.87 (CV 3.2%)
code wall TPS = 142.25 (CV 2.3%)
- soak-continuous (5 sessions × 5 turns × 4 prompts = 100 turns):
verdict PASS, 0 errors, 0 silent-empty, 0 MiB VRAM growth,
98.3% TPS retention, p50 decode 111.79
- VRAM 22.5 GB/card. Speedup vs no-spec-decode baseline: 1.79× narr / 2.31× code.
First published Ampere consumer numbers on Google's Gemma 4 MTP "assistant"
drafters (released 2026-05-05). Discussion #67 has the announcement +
upstream context. BENCHMARKS.md gets a new "Gemma 4 31B" section with
both TP=2 (working) and TP=1 (upstream-blocked on Ampere) rows.
Components landed:
- models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp.yml (TP=2)
- models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp-tp1.yml
(TP=1 — preserved for re-test when upstream Ampere fp8 path lands)
- models/gemma-4-31b/vllm/patches/vllm-gemma4-mtp/ — vendored overlay
of vllm-project/vllm#41745 (lucianommartins/gemma4-mtp). 7 modified
Python files RO-mounted over the stock nightly image. Same shape as
vllm-marlin-pad. Drop the entire tree when PR merges + propagates.
- scripts/switch.sh + launch.sh: vllm/gemma-mtp + vllm/gemma-mtp-tp1
variants registered. RUNNING_PATTERN extended for vllm-gemma-4-31b*.
- scripts/preflight.sh: autodetect_endpoint extended for gemma containers.
- scripts/soak-test.sh: container grep extended.
- scripts/soak-helper.py: SOAK_NO_CHAT_TEMPLATE_KWARGS=1 env knob to
skip the Qwen3-specific `chat_template_kwargs.enable_thinking` body
field for non-Qwen models that reject it (Gemma 4, etc.).
- BENCHMARKS.md: new Gemma 4 31B section with both TP rows.
- docs/UPSTREAM.md: PR #41745 row + the Ampere fp8-blocked finding +
transformers 5.8.0 dependency entry.
Pre-merge dependencies (drop when both land):
1. vllm-project/vllm#41745 → drop the patches/ tree + the volume block
2. transformers ≥ 5.8.0 → drop the entrypoint pip install line
Out of scope:
- TP=1 single-card. Tested + upstream-blocked on Ampere consumer:
fp8 KV needed for fit; fp8_e4m3 hits Triton "fp8e4nv not supported"
on sm_86; fp8_e5m2 rejected by gemma4_mm.py:1336 allowlist. Compose
is preserved with the failing config baked in for future re-test.
- Gemma 4 26B-A4B MoE single-card. Active params ~4B → should fit
cleanly without fp8. Queued as the obvious follow-up.
- vllm/vllm-openai:gemma4-0505-cu129 image swap. The recipe-image is
Hopper-tagged and may not include sm_86 kernels; testing it would
collapse this whole 4-layer wrangle to a one-line image bump but
needs explicit Ampere validation first.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* models/gemma-4-31b/vllm/cache: gitignore + README (mirror qwen3.6 pattern)
Replaces the .gitkeep stubs with the standard .gitignore (ignore everything
except .gitignore + README.md) + README documenting the cache lifecycle.
Mirrors models/qwen3.6-27b/vllm/cache/ exactly.
Boot times this enables (validated tonight):
- cold first boot: ~3-7 min (TP=1 / TP=2)
- warm subsequent boot: ~2-3 min (cache hit)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Adds docker-compose.dual-nvlink-turbo.yml: NVLink + TurboQuant KV (TQ3) +
MTP n=3 + 4-stream + 262K. Mirrors current dual-turbo.yml (post-#59) with
the three NVLink-specific deltas applied:
- NCCL_P2P_LEVEL=NVL (vs NCCL_P2P_DISABLE=1 on PCIe)
- PYTORCH_CUDA_ALLOC_CONF without expandable_segments (JusefPol crash repro)
- --disable-custom-all-reduce removed (NVLink P2P → custom kernel wins)
Image pin matches master (nightly-01d4d1ad3); retired sidecars
(patch_workspace_lock_disable, patch_tolist_cudagraph) excluded — superseded
by Genesis v7.72.2 PN34 + P78 natives.
Bench (danbedford rig): 101.49 narr / 133.20 code wall TPS, 20.4 GB/card.
+12.6% narr / +10.7% code over own PCIe-only dual-turbo baseline.
A/B tested on the same rig. Bench was on v7.69 — re-bench welcomed on
the v7.72.2 pin this compose ships with.
Wired into scripts/switch.sh + scripts/launch.sh as
`vllm/dual-nvlink-turbo` on port 8017. Added BENCHMARKS row + updated
UPSTREAM tracker (Marlin pad PR row) to list the new compose.
Sibling-table standardization across the 7 other compose files dropped
per the original author's comment — that's better filed as a separate
issue scoped to the table format.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Dan <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Closes the UX gap raised by @snoby on #37 — previously, launching a
compose that needs the DFlash draft model without first running
WITH_DFLASH_DRAFT=1 setup.sh would fail with vLLM's generic
"Invalid repository ID or local directory" pydantic error.
Adds three new preflight functions in scripts/preflight.sh:
1. preflight_hf_token (soft warn) — wired into setup.sh before the model
download. Catches missing HF_TOKEN early and tells the user exactly
what to do (visit hf.co/settings/tokens, accept T&C, export the token).
Skip via PREFLIGHT_NO_HF_TOKEN=1.
2. preflight_compose_deps (hard error) — wired into switch.sh just before
docker compose up. Parses the target compose YAML for known model-path
patterns (qwen3.6-27b-dflash, qwen3.6-27b-mtp-head, qwen3.6-27b-autoround-int4)
and verifies each exists on host. If missing, emits a clear "Fix:" line
with the exact setup.sh invocation needed (e.g. WITH_DFLASH_DRAFT=1).
Refuses to proceed with exit 1. Skip via PREFLIGHT_NO_COMPOSE_DEPS=1.
3. preflight_kv_format_hint (soft warn) — wired into switch.sh as the last
pre-up check. Detects smallest VRAM via nvidia-smi; if <24 GB and the
target compose uses turboquant_3bit_nc KV, emits the @efschu finding
from #47 (TQ3 → fp8_e5m2 swap rule) with cross-link to docs/HARDWARE.md
+ a tools/kv-calc.py one-liner to predict the user's specific config.
Skip via PREFLIGHT_NO_KV_HINT=1.
Tested on this rig:
- HF_TOKEN unset → warns
- Missing DFlash dir → hard errors with WITH_DFLASH_DRAFT=1 hint
- Missing main model → hard errors with generic setup.sh hint
- 24 GB rig + TQ3 compose → silent (correct, TQ3 is right pick)
- Simulated 20 GB rig + TQ3 compose → fires the fp8_e5m2 hint
All preflights are individually skippable via PREFLIGHT_NO_<NAME>=1
env vars (matches existing PREFLIGHT_NO_FETCH=1 / PREFLIGHT_NO_GENESIS_PIN=1
patterns).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
After the full Phase 3 5-grammar A/B (HE+ 164 + LCB v6 50, n=214 problems
× 5 conditions = 1070 generations), bounded-thinking.yml is updated to
recommend the DeepSeek scratchpad grammar (PLAN/NOTE×0-15/VERDICT FSM at
tools/grammar-eval/deepseek-scratchpad.gbnf) as the default.
Phase 3 results (combined HE+ + LCB v6, all FSM-enforced):
- DeepSeek scratchpad: 87.4% Pass@1 (+1 net vs andthattoo, +4pp on LCB)
- andthattoo G/A/E: 86.9% (the originally-published technique)
- Holiday tagline: 86.4%
- PROMPT_TERSE (no FSM): 82.2% (Phase 2's n=30 win was subset-selection bias)
- FREE (no constraint): 78.0% (baseline)
Phase 1 reproducibility is exact: HE+ FSM Δ +4.3pp / LCB v6 Δ +24.0pp,
both match Phase 1's published numbers — validates the bench harness.
Compose ships unchanged engine-side (same vLLM image, same Genesis stack,
same TQ3 KV, same MTP n=3, same enable_in_reasoning flag). Only the
docstring's recommended-grammar pointer + the on-disk grammar files in
tools/grammar-eval/ change. Three grammars are now validated and available
client-side via extra_body={"structured_outputs": {"grammar": ...}}:
- DeepSeek scratchpad (default, best LCB)
- andthattoo G/A/E (originally-published, ~4× tighter think budget)
- Holiday tagline (extreme 24-token compression, wins LCB by 4pp too)
Combined-accuracy spread is within noise (0.5pp at n=214), so we ship one
compose rather than three siblings — choice is at the client, not at the
compose level.
Bug fix bundled: tools/grammar-eval/subset-bench.py --full --include-lcb
mode now correctly threads dataset kind through run_condition so LCB
problems use mod.run_tests_livecodebench instead of HE+ assertion-based
testing. Without this, the Phase 3 LCB shard crashed with KeyError:
'prompt' on the HE→LCB transition.
This wraps active research on bounded-thinking. Reopen if upstream FSM-
regress cluster behavior changes (Genesis pin bump, vLLM grammar engine
swap, model-family change), or if user demand surfaces for a sibling
compose pinning a non-default grammar.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Three calibration fixes to soak-test + two boot-progress improvements to
switch.sh, all from the first cross-rig soak runs on this rig today.
soak-test.sh — baseline timing
Brief said "capture warm baseline after first turn" but turn 3 ships a
12K-char tool-result paste that fills prefix cache by ~1000 MiB on the
first hit. With baseline at turn 1, every healthy config false-positives
with "growth > 200 MiB threshold" on session 1 alone. Fixed: capture
baseline at end of session 1 (after all 5 turn shapes run once). Sessions
2-N then measure real accretion from steady state.
soak-helper.py — decode_tps guard
Thinking-mode requests where vLLM bundles all reasoning into the terminal
streaming chunk produce wall ≈ ttft (no separate content delta visible).
Old code computed decode_tps = completion_tokens / (wall - ttft) with
wall - ttft ≈ 0, yielding ~2 billion TPS. Fixed: when ttft is None OR
wall - ttft < 100ms, report decode_tps = 0 (caller filterable).
soak-helper.py — TPS outlier filter in cmd_summary
Defensive — even if a future helper bug regresses, we filter unrealistic
decode_tps values (>500 t/s) from all summary computations. tps_retention
metric was previously being inflated by 2e9 outliers in early sessions
before the helper-bug landed mid-run.
Cross-rig validation results:
Config Baseline Max VRAM Growth Verdict
vllm/long-text 180K + 0.93 23316 23316 0 MiB PASS
vllm/long-vision 145K + 0.95 22778 22778 0 MiB PASS
Both configs are soak-clean under v1 synthetic shapes. Long-vision was
expected to FAIL based on issue #41 — but our 5-turn fixtures reset the
conversation each turn while hermes accumulates context across turns.
v1 catches a class of issue (raw VRAM accretion across requests); v2
fixtures will need session = continuous conversation to catch the
hermes-shaped class. Methodology limitation documented in CHANGELOG.
switch.sh — wait_ready crash detection + boot-stage progress
Closes the "container crashed silently in 2s, you wait 600s for nothing"
UX gap that surfaced today when long-text's MODEL_DIR was misconfigured.
- Crash detection: docker inspect -f '{{.State.Running}}' between polls;
if false, dump last 30 log lines + exit 1 immediately (5s vs 600s)
- Boot-stage markers: grep docker logs for "Resolved architecture",
"Loading weights", "Compilation finished", "Capturing CUDA graphs",
"Application startup complete" — surface one line per phase transition
so the wait isn't silent.
Validated end-to-end on long-vision boot today: visible progress at
60s / 68s / 80s / 120s / 176s / 196s = ready.
Two-part addition for the most common stale-setup pattern: user cloned
weeks ago, master has moved (Genesis pin bumps, compose changes, vendored
patch updates), they re-run their compose, hit a stale config, and file
an issue we already solved on master. Wispborne's _register_op_once and
GuiPerPT's pre-pull boot OOM both surfaced through this loop.
scripts/preflight.sh — preflight_repo_drift:
- Skips silently if not a git repo, on a non-master branch, or if
PREFLIGHT_NO_FETCH=1 (offline rigs / CI / forks tracking elsewhere).
- Verifies origin remote is noonghunna/club-3090 (avoids false positives
on forks pointing elsewhere).
- timeout 5 git fetch --quiet origin master — bounded so flaky networks
don't block boot.
- On behind > 0: WARN with commit count, last-fetch age (h/d), and the
one-line fix command. Soft-warning, never blocks. Tells user about
PREFLIGHT_NO_FETCH=1 for opting out.
Wired into both launch.sh and switch.sh, alongside the existing
preflight_genesis_pin so users get one consolidated stale-setup signal.
scripts/update.sh — the easy upgrade path:
- Refuses on dirty tree (git status --porcelain) — surfaces the local
edits and tells the user to commit or stash first. We don't clobber
the rare user who's been editing a compose locally.
- Refuses on non-master branch — feature branches and fork-trackers
should pull manually; this script is the master-from-origin path.
- git pull --ff-only — no merge commits, no rebase ambiguity. Diverged
branches get an explicit error pointing at git pull --rebase.
- Re-runs setup.sh (idempotent — re-pins Genesis, re-vendors Marlin).
- Tells the user to restart their variant via switch.sh — doesn't auto-
restart, so they can A/B old-vs-new if they want.
- --dry-run shows the plan without changing anything.
- --force re-runs setup.sh even when up-to-date (for "I edited Genesis
by hand and want it re-pinned" cases).
Why detection-then-explicit-command instead of "press y to auto-update":
the user will rarely have local commits (they're consumers of the recipes,
not vLLM contributors), but we still want consent — they should see what
they're committing to. The dirty-tree guard handles the rare custom-edit
case without nagging the common path.
Wires JusefPol's NVLink compose into the same machinery the rest of the
variants use:
- switch.sh — adds vllm/dual-nvlink to the usage doc, VARIANT_DEFAULT_PORT
(8014, matches the compose's PORT fallback), and the VARIANTS map. Now
'bash scripts/switch.sh vllm/dual-nvlink' works the same way 'vllm/dual'
does.
- launch.sh — adds the variant to LAUNCH_DEFAULT_PORT + LAUNCH_DEFAULT_CONTAINER
so 'bash scripts/launch.sh --variant vllm/dual-nvlink' resolves the
endpoint URL + container name correctly for post-launch verify.
Intentionally NOT added to the launch.sh interactive wizard's dual-card menu
(line 169). The wizard runs on every fresh setup, and offering an NVLink
option to a PCIe-only user would silently boot a config that crashes on
their topology. Users with NVLink can invoke explicitly via --variant.
Catches the failure mode @wispborne hit in #32: user pulled latest
club-3090 (which bumped GENESIS_PIN in scripts/setup.sh from old →
new), but didn't re-run setup.sh. The on-disk Genesis tree at
models/qwen3.6-27b/vllm/patches/genesis/ stays at the OLD pin while
setup.sh's declared pin advances. vLLM boots against the outdated
tree and hits mysterious patch failures (e.g. v7.66 PN25's
infer_schema crash that v7.69 PN25 worker-spawn registration fixed).
Adds new `preflight_genesis_pin` function to scripts/preflight.sh:
- Parses `GENESIS_PIN="${GENESIS_PIN:-<default>}"` from setup.sh to
extract the declared default value
- Reads on-disk HEAD via `git -C <genesis-dir> rev-parse --short HEAD`
- Compares (declared short-form vs on-disk short-form match)
- On mismatch: emits a [preflight] WARN block with both pins + a
"Fix: bash scripts/setup.sh qwen3.6-27b" hint
- Soft-warning only — does not block boot
Wired into both entry points:
- scripts/launch.sh (the wizard-driven path) — adds the check after
preflight_running, before the variant pick
- scripts/switch.sh (the direct stateless switcher) — adds the check
inside up_variant() right before the docker compose up call
Skips silently if the Genesis tree hasn't been cloned yet (caller
should run setup.sh first; preflight isn't the place to handle that
case). Also skips if setup.sh isn't present at the expected path
(weird state — silent skip rather than false warnings).
Tested locally:
- Current state (declared 2db18df = on-disk 2db18df): silent ✓
- bash -n passes on all three modified scripts
Coverage gap: users who bypass switch.sh + launch.sh and run
`docker compose -f docker-compose.X.yml up -d` directly won't get
the warning. That's the canonical-path-vs-bypass tradeoff; we lead
with switch.sh in the docs.
Closes the procedural follow-up flagged in
[#32 comment-4364895764](https://github.com/noonghunna/club-3090/issues/32#issuecomment-4364895764)
("setup.sh should be re-run after every git pull that bumps
GENESIS_PIN. We could add a check for this — compare setup.sh's
declared pin vs the on-disk tree's HEAD, warn if they differ.").
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
[@wispborne](https://github.com/wispborne) caught: docker-compose.long-text-no-mtp.yml
ships in the v7.69 cutover but its corresponding variant key wasn't added
to scripts/switch.sh's VARIANTS map or scripts/launch.sh's wizard list.
Result: \`bash scripts/switch.sh vllm/long-text-no-mtp\` failed and the
wizard never offered it.
Adds:
- scripts/switch.sh:
* VARIANT_DEFAULT_PORT[vllm/long-text-no-mtp]=8021 (matches compose)
* VARIANTS[vllm/long-text-no-mtp]= compose path
* RUNNING_PATTERN extended to match its container name
* Header docs updated: long-text 218K → 180K (Balanced MTP), new
long-text-no-mtp 200K (Max-context) line
- scripts/launch.sh:
* VLLM_OPTS wizard list — both Balanced MTP + Max-context surfaced
explicitly, with correct ctx ceilings (180K + 200K, not stale 218K)
* LAUNCH_DEFAULT_PORT + LAUNCH_DEFAULT_CONTAINER updated to match
Stale 218K references in the wizard description corrected to current
shipped values: long-text is 180K (after the 0.93 mem-util backoff +
v7.69 cutover), long-text-no-mtp is 200K + 0.95.
Reported in [club-3090 #32 comment-4364904521](https://github.com/noonghunna/club-3090/issues/32#issuecomment-4364904521).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Port andthattoo/structured-cot to our stack — Qwen3.6-27B AutoRound INT4
dense / 1× RTX 3090 / vLLM nightly + MTP n=3 + TQ3 KV. Re-benched on
full HumanEval+ 164 + LiveCodeBench v6 50.
Headline (max_tokens=4096, greedy):
- HumanEval+ 164: FSM 92.7% vs FREE 88.4% (+4.3pp), 30.7× compression
- LiveCodeBench v6 50: FSM 66.0% vs FREE 42.0% (+24pp), 26.2× compression
The +Δpp partly reflects FSM dodging the max_tokens=4096 truncation trap
rather than pure reasoning gain — see docs/STRUCTURED_COT.md "Honest
caveats" for the full picture.
Three port surprises worth keeping (all in docs):
1. vLLM dev205+ defaults StructuredOutputsConfig.enable_in_reasoning=False;
grammar mask only fires post-</think> unless overridden.
2. Legacy extra_body={"guided_grammar": ...} is silently dropped on
dev205+ (tip-off: identical FREE/FSM token counts). Use the new
structured_outputs.grammar field.
3. Qwen3.6 chat template auto-prefixes <think>\n; drop the leading
literal from upstream grammars when porting.
Files added:
- models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml
- docs/STRUCTURED_COT.md (public writeup)
- models/qwen3.6-27b/vllm/diagnostics/structured-cot-bench.md (internal)
Files updated:
- scripts/launch.sh wizard + scripts/switch.sh variant map
- models/qwen3.6-27b/README.md (recommended single-card list, patch surface)
- models/qwen3.6-27b/vllm/README.md (compose menu)
- docs/SINGLE_CARD.md (TL;DR table now four rows)
- CHANGELOG.md (new top entry)
Also reverts the long-text.yml experimental flag added during smoke
testing (the flag now lives only in bounded-thinking.yml) and adds the
Genesis pre-flight check we previously skipped on long-text.
Credit: andthattoo for the technique, the grammar files, and the eval
harness.
Closes / addresses 3 reported issues + adds requested feature:
#7 vid (PORT not honored, MODEL_DIR vs MODELS_DIR confusion):
- All 8 vLLM compose files now use "${PORT:-XXXX}:8000" so .env PORT
flows through. Defaults preserved per-variant (8020 single, 8010-8013
dual). llama.cpp composes already had this pattern.
- scripts/switch.sh: load .env early; per-variant default-port table;
new resolve_ready_url() picks PORT > variant default for the readiness
probe.
- scripts/launch.sh: same default-port table; final endpoint URL printed
to user reflects actual mapped port.
- .env.example: ⚠ box callout that variable names are CASE-SENSITIVE
(MODEL_DIR singular, NOT MODELS_DIR plural — silently ignored).
New PORT section documenting per-variant defaults.
#4 timxx (tools-text.yml fails "Free memory ... less than desired"):
- docs/FAQ.md: new entry "Container fails to start: Free memory..."
explaining the vLLM startup check, the two workarounds (free VRAM /
lower mem-util), and which configs hit it most often (0.97+ mem-util).
- Compose defaults unchanged (0.97 stays the right design target on
headless rigs); the FAQ documents the workaround for users with X11.
#1 fabriciomalta (per-config VRAM column):
- docs/SINGLE_CARD.md: TL;DR table now has VRAM column with mem-util.
- docs/DUAL_CARD.md: TL;DR table same + footnote explaining per-card
semantics and which dual configs would/wouldn't fit on 2× 20 GB cards
(relevant to fabriciomalta's 2× 3080-20GB use case).
#2 tenitram (empty responses) — fixed in master via aab8ff4
(P68/P69 disabled). Closed with reply pointing at the fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This was a control variant used internally to A/B-test whether MTP
worked without Genesis (it does, on fp8+MTP paths). No reason for
end users to pick it over tools-text.yml (fp8+MTP+Genesis fixes+75K,
strictly better) or minimal.yml (no Genesis at all, simplest).
Wizard already didn't surface it. switch.sh map, sibling compose
"see also" tables, patches/README, engines/VLLM.md all updated.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fast-chat (20K, fp8, vision) and default docker-compose.yml (48K, TQ3,
vision) had effectively the same TPS post-PN8. fast-chat's only
remaining differentiator was "smaller context = ~3s faster boot," and
20K is actively bad for IDE-agent users (Copilot tool-schema preamble
alone hits 20K). Net negative — removed.
Default compose was missed in the previous P68/P69 fix — it had the
same env vars enabled and the same silent-stop bug above 8000 chars.
Both now disabled with the same explanatory comment.
Updated:
- scripts/switch.sh, scripts/launch.sh — drop the variant
- docs/SINGLE_CARD.md, FAQ.md, engines/VLLM.md, model + vllm + patches
READMEs — references removed or pointed to default/tools-text
- All sibling compose YAML "see also" tables — fast-chat row removed,
tools-text row repurposed for IDE-agent guidance
- CHANGELOG entry; old historical entries kept as-is (append-only)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
scripts/switch.sh — stateless engine/variant switcher. Brings down
whatever's running (any vllm-qwen36-27b* or llama-cpp-qwen36-27b*
container — discovers compose file via docker labels), brings up
the new variant, waits for /v1/models to respond. Supports --list
(show all 13 variants) and --down (stop without booting). Variant
names are <engine>/<file-stem>: vllm/default, vllm/dual, vllm/
dual-turbo, vllm/dual-dflash, vllm/dual-dflash-noviz, vllm/long-
vision, vllm/long-text, vllm/fast-chat, vllm/tools-text, vllm/
no-genesis-mtp, vllm/minimal, llamacpp/default, llamacpp/concurrent.
scripts/launch.sh — interactive wizard for first-run users. Asks
engine → cards → workload, maps to variant, calls switch.sh, then
runs verify-full.sh to confirm clean serving. Also accepts flags
for non-interactive use:
bash scripts/launch.sh --variant vllm/default
bash scripts/launch.sh --engine vllm --cards 1 (asks the rest)
README.md — quick-start replaces "cd into compose dir + docker
compose up" with `bash scripts/launch.sh`. Click-throughs from the
launch tweet land in a guided flow instead of having to find the
right compose file by hand.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
