Decision 2026-06-11: hold DiffusionGemma at 🧪 experimental; promote 🧪 → ⚠️
caveats when #45163 merges into a pinnable release (arch is in no release today
+ mutable :gemma tag = the blocker for caveats). soak-continuous PASS, bench +
8-pack done; the dLLM SSE/streaming-NIAH checks can't cleanly gate. Tracked in
club-3090 task + learnings re-test row.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Make `vllm/diffusiongemma-dual` usable through setup/pull/switch/launch, and
pivot the engine delivery from a baked local image to a SIDELOADED overlay on a
stock, pullable nightly (so non-rig users can actually run it).
Delivery (no baked image):
- Engine `vllm-diffusion-gemma` pins a STOCK `vllm/vllm-openai:nightly-2c9c07c8…`
and sideloads the vendored overlay at boot via install_script (entrypoint).
- `models/diffusiongemma-26b-a4b/vllm/patches/dgemma-overlay/` is the vendored
payload: the FULL stock-vs-`dgemma`-branch `vllm/` delta (123 .py) — a lean
PR-#45163-only overlay version-skews (`build_attn_metadata() got an unexpected
keyword argument 'causal'`: load-bearing dgemma changes live outside the PR
diff) — plus Codex's 3 fixes (marlin K-pad ×2 + diffusion_gemma TP-vocab/dtype).
install.sh cp's it over the installed vllm pkg, fail-loud arch assert.
- base.yml rewired: stock image + overlay mount + install_script entrypoint;
drops the 3 standalone marlin-k-pad mounts (folded into the overlay) and the
baked-image dependency. The dgemma-pr45163 Dockerfile is now the overlay-
regeneration helper, not the runtime engine.
Catalog wiring:
- Model profile `diffusiongemma-26b-a4b` (gemma4-swa-moe backbone + block
diffusion; text; valid_tp [1,2]; kv_calc_supported=false → kvcalc_key SKIP).
- Engine profile `vllm-diffusion-gemma` (stock install + vendored_overlays).
- compose_registry `vllm/diffusiongemma-dual` (port 8042, status upstream-gated).
NO DEFAULTS row — upstream-gated is non-functional, reachable only by slug.
- patches.yml `dgemma-vllm45163-sideload` (install_script + drift_guard) and the
diagnose_profile_cli overlay-path hint.
- docs/UPSTREAM.md row for vllm#45163 (re-pin + drop triggers).
Output-length fix carried in base.yml: lift the model's 256-tok max_new_tokens
default to 16384 via --override-generation-config (keeps the diffusion denoising
params; fixes OWUI truncation + next-turn echo).
Tests: bump the count fixtures for +1 model / +1 engine / +1 compose / +1 registry
entry (test-profiles-compat 6→7 models, 11→12 engines; test-compose-registry-disk
44→45 registry, 45→46 disk). Full gate green except test-compose-registry-disk
local-only redness from untracked nex-n2-mini WIP in the shared tree (CI-clean
validated green: tracked-only disk=46, registry=45, disk-only set all-allowed).
Validated live on 2x RTX 3090: stock nightly + sideloaded overlay boots, serves
coherent output, 262K, via `docker compose -f base.yml up` (the switch/launch path).
Upstream-gated until #45163 merges into a pinnable engine.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Post-#324 the Gemma duals run on vllm-gemma-stable (overlay-carrying), not
vllm-stable. Repoint that engine-pin row and add the missing vllm-stable
(overlay-free) row for the Qwen3-Next + dense + derived-base path. Pure
doc-fidelity; flagged during the #254 Genesis-cleanup.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replace the purged `vllm-nightly-clean` pin (nightly-bf610c2f, now 404 on
Docker Hub — so the launcher injected a dead image for every slug on it)
with a v0.22.0 two-engine design:
- `vllm-stable`: OVERLAY-FREE v0.22.0 — the broad successor to
vllm-nightly-clean. Serves Qwen3-Next (hybrid 27b + MoE 35B-A3B), generic
`dense` transformers, and uncurated derived-emission bases. Migrates
vllm/qwen-35b-a3b-dual, vllm/dual, vllm/minimal (live-validated on stock
v0.22.0: TP=2 + MTP + tool-call clean; marlin-pad confirmed unnecessary).
Added as the first loads:true arch pin on the qwen + dense rows.
- `vllm-gemma-stable`: KEPT as the OVERLAY-CARRYING v0.22.0 engine (#40391
per-head INT8 KV + #42006 tool-parser) — the Gemma 4 path. Unchanged.
- `vllm-pip-baseline`: the renamed pip `dense` lower-bound (frees the
`vllm-stable` name for the docker engine above).
Why two engines and not one: `vendored_overlays` is LOAD-BEARING, not
documentation. `derived_emittable` (CONTRACT-5) refuses any engine whose
`vendored_overlays != []` as a derived-emission base, and `diagnose-profile`
expects the Gemma overlay declared on its engine. A single engine cannot be
both overlay-free (for derived/Qwen bases) and overlay-carrying (for Gemma
provenance) — so the two stay split. Compose owns patch APPLICATION; engine
owns provenance/compatibility. (An attempt to fold both into one engine broke
4 tests via exactly these two consumers; the split is the validated shape.)
Also adds scripts/tests/test-compose-image-drift.sh: asserts every fixed
`${VLLM_IMAGE:-<tag>}` compose default equals its engine's `install.spec`
(catches bump-the-engine-forget-the-literal drift), and a docs/ADDING_MODELS.md
"Coherence rules" section codifying the compose/engine/patch separation.
42/42 gate green; resolver emits v0.22.0 for all migrated Qwen + Gemma slugs.
The remaining experimental/preview slugs still on vllm-nightly-clean migrate
in follow-up as each is validated, then that engine is deprecated.
Refs #254.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Self-descriptive <engine>/gemma-12b-<topology>-<quant>[-mtp] slugs:
vllm/gemma-12b-mtp -> vllm/gemma-12b-dual-bf16-mtp
vllm/gemma-12b-int8-mtp -> vllm/gemma-12b-single-int8-mtp
beellama/gemma-12b -> beellama/gemma-12b-single-q8kxl
llamacpp/gemma-12b -> llamacpp/gemma-12b-single-q8kxl
Pruned the two no-MTP base composes (MTP is output-lossless and fits the full
262144, so the bases bought nothing): removed vllm/gemma-12b (dual bf16 base) and
vllm/gemma-12b-int8 (single int8 base) + their compose files.
Cross-file: registry keys/entries, kv-calc COMPOSE_ALIAS_TEXT, calibration anchors
(dropped the removed dual-base anchor → kv-calc --calibration now 18/18, test
expectations updated), BENCHMARKS rows (rename 4 / collapse 2 pairs into the MTP
rows), UPSTREAM #39914 row, ADDING_MODELS slug example, disk-count 58/59 -> 56/57.
Hard rename, no back-compat aliases (slugs were day-old 🧪, no users). All stay 🧪
pending per-compose soaks + the ephemeral arch-preview image caveat.
Guard suite 41/41 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
lucianommartins (gemma upstream) confirmed on vllm#39914 that the 12B 256K OOB
was a config.json bug, now fixed: google/gemma-4-12B-it (+ the -assistant draft)
ship max_position_embeddings=262144. With the corrected config the stock
vllm/vllm-openai:gemma4-unified image sizes the Gemma4 RoPE cos/sin cache to
262144 and serves the full 256K — the vendored vllm-gemma4-prope-longctx overlay
(which sized the cache from runtime max_model_len) is redundant.
Re-validated OVERLAY-FREE on 2x 3090 sm_86 TP=2 bf16 (2026-06-04): base + MTP both
boot 262144 on the stock image, NIAH exact-recall PASS at 140K/170K/200K/230K/241K,
MTP spec-decode active. No device-side asserts.
Changes:
- models/gemma-4-12b.yml: bf16 hf_repo unsloth/gemma-4-12b-it -> google/gemma-4-12B-it
(the unsloth mirror still ships 131072 as of 2026-06-04; google carries the fix).
NOTE: re-introduces the Google license gate on base.yml — the MTP draft was
already google-sourced, so the vLLM dual path is google-gated either way.
- base.yml + mtp.yml: remove overlay mount + entrypoint install.sh invoke; rewrite
Max-ctx / Status / weights headers (256K is stock now, no overlay).
- patches.yml: drop the gemma-vllm-gemma4-prope-longctx entry.
- engines/vllm-gemma4-unified.yml: vendored_overlays -> []; notes updated.
- compose_registry.py: comment + both status_notes (no overlay).
- diagnose_profile_cli.py: drop the patch->path map entries.
- delete models/gemma-4-12b/vllm/patches/vllm-gemma4-prope-longctx/.
- UPSTREAM.md: #39914 -> Resolved (config fix supersedes; overlay dropped).
Guard suite 41/41 green (incl. test-patch-attribution).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Posted the runtime-sizing fix + club-3090 patch link as a follow-up on the
issue thread (issuecomment-4622760819). Tracker now reflects the engagement.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Implemented by Codex (mcp handoff); validated per the checkpoint report (Claude
re-validates live before the PR). Fixes the gemma4_unified prefill crash past 131072:
stock vllm/vllm-openai:gemma4-unified builds the RoPE cos/sin cache from
config.max_position_embeddings (131072), so positions above that index past the
AOT-compiled cache → CUDA device-side assert. Overlay sizes the cache from runtime
max_model_len and purges the stale TorchInductor cache (marker -cache-v2) so a fresh
262144 graph compiles.
- New overlay models/gemma-4-12b/vllm/patches/vllm-gemma4-prope-longctx/ (install.sh +
guarded diff + README), mounted + install-invoked by both dual composes.
- vllm/gemma-12b + vllm/gemma-12b-mtp default to 262144; registry max_ctx=262144;
models gemma-4-12b max_ctx_supported=262144; engine vendored_overlays + patches.yml
(gemma-vllm-gemma4-prope-longctx, verified) + diagnose path hints; UPSTREAM #39914 row.
- Codex live-validated: base NIAH 155K + 200K exact recall @ 262144 (KV 495,924 tok);
MTP spec-decode survives >131K. Cache shows rand_strided((262144,256/512)).
NOTE: test-compose-registry-disk still red until the single-card slugs are registered
(next commit). Re-test #39914 row: drop overlay when vLLM fixes it upstream.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The new gemma4_unified arch (vllm PR #44429, merged 2026-06-03) advertises 256K
via p-RoPE but config.json caps max_position_embeddings=131072 (official google ==
unsloth mirror). On the gemma4-unified preview image, NIAH is clean to ~124K but a
single-shot prefill past 131072 CUDA-OOB-crashes (vllm#39914 p-RoPE large-prefill);
a YaRN rope_scaling override is ignored (arch reads rope_parameters). Ship 131072 as
the dual default; re-test 256K on #39914 fix / image maturity. Data point added to
the #39914 thread.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
A careful tok/s re-test (bench.sh narrative n=5, MEASURED no-spec controls,
three v0.3.0 images: efe856397 / e0663be / 63abcd3) shows DFlash prose is
net-POSITIVE everywhere — Qwen single Q5 +27% (45.7 vs 35.9 no-spec), Qwen
dual Q8 +52% (35.6 vs 23.4), Gemma single Q4 +28-31% (44.6 vs 34.8). The
earlier "v0.3.0-wide prose-acceptance regression (~0.07 AR / net-negative)"
was a DOUBLE error: (1) over-reading the noisy/prompt-dependent acceptance-rate
diagnostic (the same efe856397 image we logged at ~0.07 now reads ~0.32 AR at
the same tok/s — Anbeeld's #288 AR caution was right), and (2) a wrong no-spec
baseline (we'd used ~37; the real dual-Q8 no-spec is 23.4). The new adaptive-DM
HEAD 63abcd3 is neutral (tok/s flat). Build-arch ruled out (a 3090 runs identical
sm_86 SASS from a fat or single-arch binary).
- compose_registry.py: 5 beellama status_notes (measured slugs assert net-positive;
un-rebenched gemma-duals retract the claim without overclaiming).
- 5 beellama compose Caveats: same correction.
- docs/UPSTREAM.md row 38: regression clause retracted; prose-recovery half of
the promotion gate dropped.
- BENCHMARKS.md: added the v0.3.0 DFlash-vs-no-spec A/B note (no-spec baselines).
- (learnings/qwen3.6-27b.md + gemma-4-31b.md got dated append-only retractions;
reported to Anbeeld at discussion #288.)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
DUAL_CARD.md still presented vllm/dual-dflash + dual-dflash-noviz as live
'peak code TPS' recommendations (pick-by-workload table + two sections with
setup steps), and the Gemma dflash row was a dead link (compose removed in
#451). They were pruned 2026-05-31 (superseded by vllm/dual; stranded on a
now-purged nightly). Add deprecation banners + redirect to the maintained
beellama DFlash duals; fix the stale UPSTREAM.md 'recommended DFlash path
today' line. Surfaces the rationale @hlo-world asked for in #297.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Qwen3.6-27B dual+single v0.3.0 control: prose acceptance collapses identically
(single 0.085 == dual 0.090), and Qwen is DeltaNet (NOT SWA-windowed) — so the
SWA-mismatch suspect floated for Gemma is a red herring. It's a cross-model
v0.3.0 DFlash-on-prose regression (code unaffected; Qwen dual code 145 TPS).
Multi-GPU DFlash fix confirmed on both model families.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Rename vllm/gemma-int8→gemma-int8-mtp + vllm/gemma-mtp→gemma-bf16-mtp (both carry MTP n=4; only KV format differs); kv-calc alias fix. Ladder gemma-bf16-mtp default 32K→131K (measured BF16 KV pool 196,527 tok @ 0.95; verify-stress 8/8, NIAH→120K, bench 118.8/154.3, zero TPS regression). Codify the dual-card max-context-priority rule in DUAL_CARD.md. Suite 41/41.
Single beellama/gemma-dflash 128K (caveats; 140K OOMs under load, CTX_SIZE=102400 agent-safe). New dual beellama/gemma-dflash-dual parked upstream-gated (262K works but DFlash multi-GPU broken in 07ac3ce; fixes on v0.3.0 dev branch, no release). switch.sh --list hides upstream-gated (parked) like deprecated with its own note. UPSTREAM row tracks Anbeeld #39. Suite 38/38.
Bump vllm/gemma-mtp (dual, bf16+MTP) to stable v0.22.0 (validated 5/5 coherent). Deprecate vllm/gemma-mtp-tp1: fp8 KV is hardware-impossible for Gemma 4 on Ampere sm_86 (fp8e4nv kernel unsupported; fp8_e5m2 rejected by gemma4 attention allowlist; nvfp4 Blackwell-only), live-confirmed on v0.22.0. Remove the dead DEFAULTS[(gemma-4-31b,vllm,single)] row; repoint launch/setup/preflight single-card Gemma -> beellama/gemma-dflash. Also fixes a pre-existing unterminated NEXT_STEPS_NOTE string in setup.sh's gemma-4-31b arm. Suite 38/38.
Gemma single-card had NO functional default (vLLM single is upstream-gated at
head_dim=512; ik-llama walls ~24K; stock llama.cpp ~12 TPS). beellama is the
only viable fast single-card Gemma-4 path, so promote it to fill the gap.
Registry: beellama/gemma-dflash experimental -> caveats + new
DEFAULTS[(gemma-4-31b, beellama, single)]. Resolver now returns it for Gemma
single (beellama is #1 in ENGINE_PREFERENCE[single]); Gemma dual unchanged.
Validated #441: 131-200K ctx, 47/88 TPS, 8-pack 109/150 think-off /
114/150 think-on (reasoning net-positive on Gemma). Reuses the multi-arch
image from #271 (sm_86/89/120; sm_89/120 compiled-not-validated).
Docs: compose header (caveats + Quality + ctx ceiling), README Gemma row,
BENCHMARKS row marker + intro, UPSTREAM.md row. Resolver guard test flipped
from a degradation assertion to beellama/gemma-dflash; full suite 38/38.
Re-test trigger: re-point the default to the no-fork mainline path when
llama.cpp#23398 (Gemma-4 MTP) merges (docs/UPSTREAM.md).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Make `beellama/dflash` the single-GPU default for Qwen3.6-27B, served via
an unofficial multi-arch image, and demote ik-llama to the balanced alt.
Decision basis (single-3090, 370 W; bench.sh n=5 + quality-test --full):
- Code TPS ~100 vs ik 69 — fastest single-card 3090 code path.
- 8-pack 107/150 (71%) think-off vs ik 99/150 (66%); 113/150 think-on
(same-session). DFlash is output-lossless, so quality == the Q5_K_S target.
- Context ceiling ladder (measured): 130K comfortable, 160K usable
(115K-tok prefill OK, 0.8 GB headroom), 200K boots but OOMs on prefill.
Ships 102K default; raise to 160K via CTX_SIZE.
- ik still wins narrative TPS (63), VRAM, and shipped vision — kept as the
balanced alt (`--variant ik-llama/iq4ks-mtp`).
Registry: `beellama/dflash` experimental -> caveats + a new
DEFAULTS[(qwen3.6-27b, beellama, single)] row. beellama is already #1 in
ENGINE_PREFERENCE[single], so the resolver now returns it for Qwen single.
Image: unofficial multi-arch ghcr build
`beellama-cpp:multiarch-b9459-07ac3ce` (sm_86/89/120 = RTX 3090/4090/5090),
built from Anbeeld/beellama.cpp .devops/cuda.Dockerfile with
CUDA_DOCKER_ARCH="86;89;120" + GGML_CUDA_FA_ALL_QUANTS=ON. Engine reports
`ARCHS = 860,890,1200`; cuobjdump confirms all 3 SASS. CAVEAT: sm_89/sm_120
are compiled but UNVALIDATED — only sm_86/3090 is verified on-rig.
Also bumps the gemma beellama compose to the same multi-arch image (stays
experimental — not promoted). Docs: README single-card framing,
INFERENCE_ENGINES.md, UPSTREAM.md row, BENCHMARKS.md Qwen beellama row.
Resolver guard tests flipped to the new default; full suite 38/38.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Now that the unofficial sm_86 image is published + public on GHCR, flip the two
beellama composes' BEELLAMA_IMAGE default from the local-only beellama-cpp:local
to ghcr.io/noonghunna/beellama-cpp:sm86-b9459-07ac3ce (immutable commit-pinned
tag) — so 3090 users pull-and-run instead of building. Update caveats/quickstart
+ INFERENCE_ENGINES/UPSTREAM docs (pull for sm_86, build for other arches via
BEELLAMA_IMAGE override). Status stays experimental: the resolver still skips
(NA), so ik-llama remains the single-card default — no behavior change — pending
cross-rig validation before promotion.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Onboards Anbeeld's llama.cpp fork (beellama.cpp) as a registry engine
with two single-card DFlash composes, both experimental pending a
published Docker image:
- beellama/dflash Qwen3.6-27B Q5_K_S + Anbeeld DFlash-IQ4_XS,
q5_0/q4_1 KV, 102K ctx, port 8060
- beellama/gemma-dflash Gemma-4-31B Q4_K_S + Anbeeld DFlash-IQ4_XS,
q5_0/q4_1 KV, 102K ctx, port 8061
Composes reference the locally-built ${BEELLAMA_IMAGE:-beellama-cpp:local}
(entrypoint /app/llama-server) — upstream ships no pullable image, so
they ship as Status: Experimental and surface as (NA: experimental) in
--list (launch is --force-gated). Configs follow the fork's quickstart
docs (Qwen "Precision combo", Gemma "Balanced combo"), deviating only on
the stack-wide --reasoning off default.
Catalog wiring: engine profile beellama-local.yml; two DFlash GGUF
drafters (anbeeld-qwen-dflash, anbeeld-gemma-dflash); compose_registry.py
entries with kvcalc_key="SKIP" (llama.cpp-family, like ik-llama);
weights-map entries (target + draft GGUFs) on both models; q4_1 added to
rtx-3090 supported KV; INFERENCE_ENGINES + UPSTREAM notes; test counts
bumped (engines 8->9, drafters 6->8, registry 45->47, disk 46->48).
Default-resolver invariant preserved: beellama is #1 in
ENGINE_PREFERENCE[single], but its (NA) status + no DEFAULTS row means
the resolver skips it, so qwen3.6-27b single still resolves to
ik-llama/iq4ks-mtp (asserted by test-model-default-resolver.sh). It
auto-promotes only when a published image lands and the composes flip to
production. Full catalog suite green (only the pre-existing
test-submit-bench.sh fixture failure remains).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
* feat(switch): model-default resolver + user-pinnable defaults
Add a two-layer default scheme on top of the existing <engine>/default map.
<engine>/default stays the maintainer's recommendation (read-only to users);
the new <model>/default is the user's preference — their .env pin if set,
else a curated pick for the detected topology.
compose_registry.py: two maintainer knobs next to DEFAULTS —
RECOMMENDED_DEFAULT_MODELS (short opt-in shortlist, not an exhaustive
ranking; new models are NOT auto-added) and ENGINE_PREFERENCE (per-topology
engine order). Plus resolver helpers (curated_default_target,
community_default_target stub, model_default_pin_key, engine_set/model_set,
slug_topology, model_of_slug).
registry-emit.sh: shared resolver model_default_target(root, model, topology)
— the single injection point for both launchers. Precedence ladder: --variant
(caller) -> .env pin -> community seam (None today) -> curated ENGINE_PREFERENCE
walk (skips non-functional (NA) slugs) -> degradation (notice + nearest-lower
topology, else a clear "pick explicitly" message; never crashes). Plus
x_default_dispatch: X/default with X in engine-set -> engine rec; X in
model-set -> model default; else error (engines + model-ids are disjoint).
switch.sh: <model>/default token; --set-default <slug> / --clear-default
<model> (round-trip the .env pin CLUB3090_DEFAULT_<MODELID>); a Defaults view
appended to --list (also standalone via --defaults) marking user-pin vs
curated. PR-A's grouping/markers/counts and the --force gate preserved.
launch.sh: bare invocation -> first installed shortlist model -> its
<model>/default (no full wizard); a pinned fast-path ("Launch your default
<slug>? [Y/n]"); a post-boot offer ("Make <slug> your default? [y/N]"). Any
narrowing flag keeps the explicit wizard path. Also load CLUB3090_DEFAULT_*
pin keys from .env even when MODEL_DIR is exported in the shell (the existing
.env loader is MODEL_DIR-gated, which would otherwise hide the pin).
Pin validation is warn + fall back, never blocking: unknown slug / wrong
model / topology-mismatch / (NA) status -> notice + curated default.
Tests: new test-model-default-resolver.sh (curated walk, (NA) skip,
degradation, X/default dispatch, pin override + all validation paths,
community seam skipped, .env round-trip); test-default-resolver.sh extended
with <model>/default launch.sh dispatch. Full suite green (only the
pre-existing test-submit-bench.sh fixture failure remains); 45 entries
unchanged; kv-calc calibration 17/17.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* docs: document the model-default resolver + user pins
Ship the user/contributor docs alongside the resolver code (repo convention —
update existing docs, no new top-level doc).
- README: single-card realign (ik-llama = fastest blessed single default,
llama.cpp = cliff-immune alternative — matches ENGINE_PREFERENCE single
order); add "pin your default" + <model>/default to Quick start.
- CLAUDE.md (= AGENTS.md): document RECOMMENDED_DEFAULT_MODELS +
ENGINE_PREFERENCE + the shared resolver as maintainer knobs next to DEFAULTS.
- FAQ: extend "switch to a different model" with <model>/default; new "How do
I set my own default config?" Q (two-layer model, --set-default/--clear-
default/--defaults, .env key, warn+fallback validation).
- SINGLE_CARD / DUAL_CARD: the resolver + the per-topology engine order; pin
hint.
- GETTING_STARTED: first-run uses <model>/default + the in-flow "set as
default?" prompt.
- ADDING_MODELS: a new model resolves via ENGINE_PREFERENCE; add a DEFAULTS row
per engine; RECOMMENDED_DEFAULT_MODELS is not auto-grown.
- UPSTREAM: beellama Docker-image row (gates beellama onboarding; the resolver
skips it today and it auto-promotes to single default on catalog).
- models/qwen3.6-27b/CHANGELOG: dated PR-B entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
The four dual/autoround-int4/nvlink-*.yml composes (nvlink-fp8-mtp,
nvlink-turbo, nvlink-dflash, nvlink-dflash-noviz) and their
vllm/dual-nvlink* launch slugs are removed. They were thin Docker Compose
`extends:` stubs whose ONLY override was NVLINK_MODE=force_on — fully
redundant since every base dual compose now auto-detects NVLink at boot
via detect_nvlink.sh (NVLINK_MODE=auto): it probes nvidia-smi and flips on
NCCL_P2P_LEVEL=NVL + custom-all-reduce when a bridge is present, else
NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce. NVLink rigs get the
identical fast path from the base dual compose with no separate slug;
force it explicitly with `NVLINK_MODE=force_on scripts/switch.sh vllm/dual`
if auto-detect ever misses.
Scope (prune only; no engine-image change — the v0.22.0 stable-engine
consolidation is a separate follow-up):
- delete 4 nvlink-*.yml composes
- compose_registry.py: drop the 4 _entry blocks (50 -> 46 entries)
- test-compose-registry-disk.sh: count guards 50/51 -> 46/47
- profile_runtime.yml + patches.yml: drop the 4 nvlink blocks/slug refs
- test-profiles-compat.sh: retire the C13/E3 NVLink-required-compose
scenarios (the estate NVLink-gating code is now dormant; cleanup tracked
as a follow-up, code retained)
- docs: switch.sh help, AGENTS.md, HARDWARE.md, UPSTREAM.md, patches/README.md,
fp8-mtp/dflash/dflash-noviz compose headers, per-model CHANGELOG removal
entry; add a docs/README.md pointer to `switch.sh --list` as the
authoritative registry-derived compose x slug matrix
Historical NVLink bench rows (JusefPol #31, danbedford #74/#92/#96) are
preserved in BENCHMARKS.md. Full test suite green (the lone
test-submit-bench failure is a worktree-isolation artifact — it needs
gitignored results/rebench/ fixtures absent from a fresh worktree; passes
on the working tree).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Every gemma-4-31b vLLM compose pinned a purged Docker Hub nightly
(e47c98ef / bf610c2f), un-bootable for fresh users (#250, #167 class).
Prune the 9-variant set to 3 and repoint survivors onto the immutable
stable release tag vllm/vllm-openai:v0.21.0 (never :latest).
Survivors: vllm/gemma-mtp (bf16 dual, default), vllm/gemma-int8 (int8-PTH
dual), vllm/gemma-mtp-tp1 (fp8 single). Dropped: gemma-dflash,
gemma-dflash-int8, gemma-int8-tq3, gemma-bf16, gemma-awq; the 262K path
folds into gemma-int8 via a CTX env override. Gemma is decoupled from the
shared Qwen nightly profiles via a new vllm-gemma-stable engine profile;
Qwen nightly profiles are untouched.
Diagnose decoupling: the pruned gemma dflash patch dirs doubled as the
diagnose tool's cross-model disk-source proxies for two Qwen overlays
(vllm-pr41703-dflash, vllm-pr42102-dflash-kv-quant). Those capabilities
are image-baked in the pinned nightly, not mountable files, so mark them
image_baked and have diagnose skip the disk-source check for image-baked
overlays. Runtime-neutral (no Qwen compose mounts them).
switch.sh launcher: handle the new VLLM_IMAGE engine-pin export (the twin
loop in launch.sh had it; switch.sh did not -> "unexpected engine pin
export"), and guard the VLLM_NIGHTLY_SHA echo for the image-only stable
profile (was unbound under set -u).
Validated on 2x RTX 3090 (Ampere, v0.21.0): full scripts/tests/*.sh suite
35/0; live boots of gemma-mtp (bf16) and gemma-int8 serve coherent output,
qwen vllm/dual regression-clean. gemma-mtp-tp1 (fp8_e4m3) is correctly
SM-gated (required_sm=9.0) -- confirmed Ampere-incompatible (Triton
"fp8e4nv not supported on sm_86"), a documented 32 GB+/Hopper variant;
beellama remains the single-card gemma path on Ampere.
Refs #451, #250, #167.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
#39056 is the broader tool-calls-inside-`<think>`-region-lost issue (non-streaming);
#39598 is its streaming subset, already covered by Genesis P64 (default-on, validated).
Cross-references added in both directions + workaround note for the non-streaming case.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
We don't distribute a club-3090 vLLM image to users, so the
build-vllm-image.yml workflow + its surrounding docs are dead weight. The
workflow had also been failing on every tag push and the weekly cron because
its base nightly (nightly-1acd67a7) was purged from Docker Hub (#167/#407) —
that red CI on the v0.8.4 tag is what surfaced this.
Removed:
- .github/workflows/build-vllm-image.yml (the GHCR image builder)
- docker/vllm-club3090/Dockerfile (its build recipe)
- docs/CI_RUNNER_SETUP.md (build/distribution doc)
- README + UPSTREAM references to ghcr.io/noonghunna/vllm-club3090
- docs/README index link to the deleted CI doc
Kept: the generic VLLM_IMAGE override (now documented against
vllm/vllm-openai:latest — also the #167 workaround). The launch-compat test
still exercises that override, just with an upstream image.
Recoverable from history if we ever want to ship an image again.
NOT touched: patch_attribution.py still references the dockerfile_bake
delivery mode — left for a separate decision (internal, harmless).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Maintainer rig A/B 2026-05-18 (today-anchored, vllm/dual, identical
engine, template-only delta): hermesagent-20 +10pp (50→60%, +2 scen),
7 other packs flat, streaming tool-calls OK both arms (#145 clear),
soak-continuous PASS 0/25 both arms. TPS-neutral: symmetric
restart+settle 3 bench/arm, CODE 89.06→87.62 (−1.6%, within noise),
NARRATIVE flat — the first-pass −7% was an asymmetric-protocol
artifact, not reproduced. Reproduces the prior froggeric finding on a
clean today-anchored baseline; #35936 control confound resolved.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Re-vendors the froggeric Qwen3.6 chat template from the latest upstream
snapshot for maintainer re-evaluation only.
Provenance:
- Previous upstream revision: unrecoverable. The existing local snapshot was
introduced by local commit 84498d47aa with
SHA256 94e944287ffaf8c3ed8b5840a0c92fd4ca3caefa721f4f5e31e92605e63f1ad4,
but no exact upstream commit/release match was found in available froggeric
qwen3.5/qwen3.6 archive v8-v19 or main history.
- New upstream revision: c31fd393e531dbacd92b6deb99a2037cc949f950,
timestamp 2026-05-16T13:44:07Z, release label v19,
SHA256 4649b3fa3db3fda4d51173ed4ff0175fde7ece8bbceb9d595d04d862020c9746.
Structured diff summary:
- System/tools setup: system/developer content is now extracted once and
reused for tool and non-tool paths; tool definitions render function payloads
rather than raw tool wrappers when present.
- enable_thinking gate: ns_flags now tracks has_tools, last_tool_failed, and
consecutive_failures in addition to enable_thinking. The template still
honors enable_thinking plus think_on/think_off markers, but preserve_thinking
now defaults toward preserving non-empty reasoning unless explicitly false.
- Tool-call and streaming-sensitive formatting: tool instructions now require
reasoning inside a think block before a tool call, final answers immediately
after closing think when no tool is needed, and one fully closed tool_call per
function. Tool-call emission is normalized to newline-started XML blocks.
No explicit streaming branch was added, but these boundary/whitespace changes
should be covered by the #145 streaming/tool-call smoke.
- Reasoning delimiters: the template now handles </think>, </thinking>,
malformed </ think>, and </think > endings, removes embedded tool_call
content from assistant text when structured tool calls are present, and avoids
emitting empty historical think blocks.
- Qwen 4: no Qwen 4-specific branch was found in the upstream template.
- Whitespace/Jinja control: multiple spacing and newline changes affect
system/tool prompts, think/content boundaries, tool_call boundaries, and the
thinking-disabled generation prompt. Treat tokenization and prefix-cache
behavior as validation targets.
Validation:
- bash scripts/tests/test-patch-attribution.sh
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Replace the imprecise "#41403 Gate 2" attribution with the empirically
captured root cause: vLLM config.py:101 unconditionally force-selects
TRITON_ATTN for Gemma-4's heterogeneous head_dim (256 sliding / 512 global)
as a numerical-divergence safeguard, before backend selection — so no
attention-backend/knob lever can override it (closing probe exhausted them).
Correct framing: no PR enables kv_cache_dtype *in* TRITON_ATTN; the upstream
fixes (#42637 freshest/most-targeted, #40108 ready/verified, #41123) operate
at the backend-selection / TQ-spec layer. Watch commands updated.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Records the vendored overlay landed in 1872880 in the canonical
pin-bump tracker:
- Merged upstream 2026-05-06 at commit d5b31c95
- Wired into 18 affected composes (vllm-nightly-mtp / -dflash / -full
pin targets)
- vllm-nightly-clean (bf610c2f) is post-fix and doesn't need the
overlay (install.sh has upstream-fix detection that no-ops on it)
- Drop trigger per engine spelled out:
* vllm-nightly-mtp: requires Genesis v7.73.x re-anchor
* vllm-nightly-dflash: requires PR #41703 overlay re-validation on
newer base
* vllm-nightly-full: requires PR #42102 overlay re-validation on
newer base
- Tracking issue #139 + triggered-by #138 (SEVENID) referenced
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
vLLM maintainer @benchislett closed PR #42102 on 2026-05-15 with
closed-as-slop label and no comment. Upstream issue #41559 (DFlash +
quant-KV coexistence bug) remains open.
The local fix (4 files, +333/-35) continues to ship as a vendored
overlay at models/gemma-4-31b/vllm/patches/vllm-gemma4-dflash-int8/,
baked into vllm-nightly-full + vllm-nightly-dflash EngineProfiles. No
production impact — Gemma 4 + DFlash + INT8 PTH still serves at 65K.
Re-engagement with upstream NOT recommended given vLLM's hardened
anti-AI-PR policy. Watch #41559 for any maintainer-blessed alternative;
drop our overlay if and when that lands.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The smoke-gating mechanism required a self-hosted GPU runner that we don't
maintain + administration:read permission on GITHUB_TOKEN that the default
scope lacks. Result: :latest never got promoted on the v0.7.0 release run.
Simplification:
- Remove detect-smoke-runner job (no permission dependency)
- Remove smoke job (no runner dependency)
- Add promote-aliases job: unconditionally promote :latest + :nightly-stable
to the just-built dated tag using docker buildx imagetools create
- Build job continues to produce dated + :club-vX.Y.Z (on tag pushes)
- retention job unchanged (still gated on schedule/dispatch)
- Workflow permissions reduced: actions:read no longer needed
The Docker Hub :latest convention is "most recent, no guarantees". Users who
want verified images pin to :club-vX.Y.Z. If smoke-gating becomes useful when
we get a registered runner, it can layer on top as a separate post-build job
without re-introducing the failure modes.
docs/CI_RUNNER_SETUP.md rewritten to reflect the simplified flow + manual
:latest bootstrap recipe. docs/UPSTREAM.md updated to point at :club-vX.Y.Z
as the recommended pin instead of :latest.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The UPSTREAM.md row for vllm#40807 (.tolist cudagraph crash) and the
P78 comment in tq3-mtp-genesis.yml both claimed we ship a local
patch_tolist_cudagraph.py sidecar. That sidecar was retired on
2026-05-05 when we bumped Genesis to v7.72.2 (which provides P78 as
the equivalent guard) — but the docs never caught up. Two stale lines
fixed:
- docs/UPSTREAM.md vllm#40807 row: was "⚫ Local workaround", now
"✅ Retired locally" with the actual current state (P78=0 on
tq3-mtp-genesis.yml because rebench-full leg 6 passed clean
without it; non-Genesis composes on 1acd67a79 pin run without
any guard, unvalidated at long-context TurboQuant chunked-prefill).
- tq3-mtp-genesis.yml line 197 comment: was "superseded by our
patch_tolist_cudagraph.py" (the file we no longer ship), now
explains the actual reason P78=0 (rebench evidence) plus the
flip-to-1 trigger.
Also adds a new UPSTREAM.md row for vllm#40798 + vllm#42215 — the
upstream pair that closes the workspace-locked assertion at
turboquant_attn.py:_continuation_prefill for ≥48K-token chunked-
prefill with TurboQuant KV. Independently validated on 2× 3090
sm_86 by @cferra (vllm#41403 comment, 2026-05-12). On our stack,
Genesis PN34 addresses the same symptom via a different mechanism
(relax-lock vs reserve-before-capture); when the upstream pair
propagates to a nightly we pin to, we'd A/B PN34 vs the pair.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Vendored local overlay for vllm-project/vllm#35936 — fixes empty
tool_calls=[] response when tool_choice="required" is used with the
qwen3_coder parser. Under "required", vLLM forces structured JSON
output regardless of which parser is configured; the qwen3_coder
parser then scans the JSON for its XML <tool_call> sentinel, finds
none, and returns tools_called=False.
Overlay reshapes _parse_tool_calls_from_content() in
vllm/entrypoints/openai/engine/serving.py to try JSON-validate first
when tool_choice="required", and fall back to the configured parser
only when validation fails — covering both the supports_required_and_named
True and False parser paths.
End-to-end validated 2026-05-12:
- curl tool_choice="required" + get_weather tool: tool_calls populated
- curl tool_choice="auto" regression: still works
- MLS-Bench ml-ensemble-boosting with thinking.enabled=false: agent
reaches Step 1 (edit) with no "No action returned" stall
Mounted into the 23 Qwen 3.6-27B composes pinning the post-#41434
nightly-1acd67a79 image. The single compose on the older 01d4d1ad3
pin (dual/tq3-mtp-genesis.yml) is intentionally excluded — its source
tree diverges from the rebase base.
Added UPSTREAM.md row tracking the upstream PR + drop trigger.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Five-format matrix across TQ3/TQ4/k8v4 + MTP all fail long-context needle
with first-word repetition, regardless of cudagraph state or PR #40914 K+1
dispatch. TQ3 with MTP disabled passes 7/7 verify-stress, so this is an
MTP × TurboQuant multi-query interaction (Sander's P67/P67b territory),
not TQ precision.
PR #40914 actively makes things worse on our stack: dropping it lifts
verify-stress from 3/7 to 5/7. Per vllm.ai/blog/turboquant the upstream
position is that hybrid-attention models (which Qwen3-Next is) are not yet
supported by TurboQuant — matches our matrix.
Updates anchor the conclusion in tree:
- docs/UPSTREAM.md: #40914 reframed as OPEN/negative on this stack;
P67/P67b called out as the only known working multi-query fix.
- docs/FAQ.md: #40914 answer updated with the 2026-05-11 local result.
- models/qwen3.6-27b/INTERNALS.md: P67 → P67/P67b, #40914 status flipped.
- vllm/patches/vllm-pr40914-k1-only/README.md: prepended "negative result"
status section; round-2/3 MTP-skip experiment annotated as ineffective.
- vllm/patches/vllm-pr40914-k1-only/.../turboquant_attn.py: epilogue now
writes through the caller's output buffer (Codex fix; left for the
re-test artifact, not deployed).
- vllm/compose/dual/tq3-mtp.yml: re-tombstoned to point at the upstream
feature-compat gap rather than a 5-PR landing list.
Deployable Genesis-free path remains dual/tq3-nomtp.yml; TQ + MTP path
remains dual/tq3-mtp-genesis.yml.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The dev rig had grown across three home dirs (`/opt/ai/compose/`, `/opt/ai/github/`,
`/home/wasif/`) and three repos (single-3090, dual-3090, club-3090). Disk-out
on `/` (97% used) forced a cleanup; rather than just prune, we consolidated
the layout across the whole stack while we were at it. This commit captures
what landed inside this repo.
Services consolidation:
- services/{ollama,openwebui,litellm,qdrant,searxng}/ migrated in from
/opt/ai/compose/<svc>/ (zero functional change — same docker-compose.yml).
- services/litellm/config.yaml rewritten: explicit routes for current
primaries (qwen3.6-27b-autoround → :8010, gemma-4-31b-autoround → :8030).
Removed `* → ollama/*` wildcard.
- services/comfyui/ migrated in (was /opt/ai/compose/comfyui/) — wired into
gpu-mode with full mutex against vLLM/SGLang.
scripts/gpu-mode.sh under git:
- Was a loose /opt/ai/gpu-mode.sh outside any repo. Symlinked at
/usr/local/bin/gpu-mode.
- Five Gemma 4 31B modes added: gemma, gemma-dflash, gemma-int8,
gemma-dflash-int8, gemma-awq.
- One ComfyUI mode (mutex with all LLM serving).
- prune / prune-all subcommands (safe image prune; aggressive variant adds
build cache --keep-storage 5GB + dangling networks).
- gpu-mode status now shows Docker disk + /var/lib/docker + /tmp sizes.
- compose_at() passes --env-file <repo>/.env so MODEL_DIR resolves
regardless of which compose dir gpu-mode cd's into. Fixes the recurring
"MODEL_DIR not set, defaulting to ../../../../../models-cache" warning.
- stderr no longer swallowed by compose_at() (real errors surface).
- Cross-model VRAM mutex: every Qwen mode stop_all_gemma + stop_comfyui
and vice-versa.
scripts/maintenance/ — new hygiene-tools subdir:
- list-image-pins.sh: engine-agnostic pin auditor. Scans every compose's
`image:` line, groups by `<repo>:<tag>`, flags pin-drift (multiple tags
per repo), ranks composes by patch surface.
Pin tracking:
- docs/UPSTREAM.md gains a "Pinned images" section: table of every pinned
image, why each pin exists, retirement candidate criteria.
- docs/NIGHTLY_BUMP_RUNBOOK.md (new): 7-step procedure for bumping pinned
engine images (scope → branch → patch survival → boot → verify-full +
verify-stress → bench delta → land → retire). Engine-specific notes
for vLLM nightly hashes, llama.cpp digest pinning, SGLang variants.
Path updates from the engine + model dir consolidation:
- /opt/ai/vllm-src/ → /opt/ai/engines/vllm/primary/
(in setup.sh, INTERNALS.md, several patch READMEs, docs/HARDWARE.md,
docs/FAQ.md, docs/DUAL_CARD.md, docs/UPSTREAM.md, models/qwen3.6-27b/
CHANGELOG.md)
- /mnt/models/gguf/qwen3.6-27b/ → /mnt/models/huggingface/qwen3.6-27b-gguf/
(in models/qwen3.6-27b/llama-cpp/{compose/single/*.yml, recipes/*.sh,
README.md}, docs/engines/LLAMA_CPP.md)
CHANGELOG.md narrative gap fill:
- 2026-05-10 entry for this reorg.
- 2026-05-09 entry for compose convention formalization (topology
promoted to dir level, profile schema, Status enum + Caveats, cliff
CI swap, Discord launch).
- 2026-05-08 entry for Gemma 4 INT8 PTH unblock + 262K validation.
- 2026-05-07 entry for power-cap-sweep campaign + HARDWARE.md cross-rig
charts + cross-rig benchmark rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Compose files now live under `<model>/<engine>/compose/<topology>/<file>.yml`,
with topology as a folder rather than a filename prefix. Solves all 7
inconsistencies surfaced in the post-rename audit (single-card composes
without `single-` prefix, unsuffixed `docker-compose.yml` ambiguity,
fine-tunes encoding model name in filename, etc.) by making the directory
hierarchy enforce the convention.
Layout:
models/<model>/<engine>/compose/<topology>/<feature>.yml
Where:
- <model>: qwen3.6-27b, gemma-4-31b
- <engine>: vllm, llama-cpp, sglang
- <topology>: single, dual, multi3, multi4, multi8
- <feature>: docker-compose.yml (default) | turbo.yml | dflash.yml | etc.
Each topology subdir has a `docker-compose.yml` for the recommended
starter — bare `cd <topology> && docker compose up` works because
docker compose finds that filename automatically. Variants drop the
`docker-compose.` prefix since they're invoked via `-f` flag.
27 compose file moves total:
- 18 Qwen vLLM composes redistributed across single/dual/multi4
- 2 Qwen llama-cpp composes into single/
- 6 Gemma vLLM composes redistributed across single/dual
- 1 untracked qwopus-bf16mtp moved to dual/
Inside each compose: relative paths to `../patches/` and `../cache/`
bumped to `../../patches/` / `../../cache/`, and `../../../../models-cache`
to `../../../../../models-cache` (one extra `..` for the new depth).
Reference updates across 148 files (BENCHMARKS, all docs, CHANGELOGs,
sibling-table cross-references in compose headers, scripts, patch
READMEs, .github issue templates, tools/residency-instrument).
scripts/switch.sh VARIANTS map updated; tags themselves unchanged
(`vllm/dual` → `dual/docker-compose.yml`, `vllm/dual4` → `multi4/docker-compose.yml`,
`vllm/gemma-mtp` → `gemma-4-31b/.../dual/docker-compose.yml`, etc.).
AGENTS.md "Compose layout" section rewritten to describe the new
hierarchy, with concrete examples and the fine-tune exception
(`dual/carnice-bf16mtp.yml` carries the fine-tune name as a filename
prefix until the fine-tune graduates to its own model directory).
All switch.sh paths verified to resolve to actual files post-move.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Documentation distinguishes 1 / 2 / 3+ GPUs as `single` / `dual` /
`multi`, with separate doc pages (SINGLE_CARD.md, DUAL_CARD.md,
MULTI_CARD.md). Compose filename topology prefix should match.
`dual4.yml` was an awkward outlier — it had `dual` in the prefix
but actually meant TP=4 (4-card config). The clean shape: GPU
count is implicit when there's no ambiguity (`single` always 1,
`dual` always 2), and explicit when there is (`multi3` / `multi4`
/ `multi8`).
Renames:
- docker-compose.dual4.yml → docker-compose.multi4.yml
- docker-compose.dual4-dflash.yml → docker-compose.multi4-dflash.yml
Registry tags `vllm/dual4` and `vllm/dual4-dflash` in scripts/switch.sh
keep their existing names (backward compat for users running
`bash scripts/switch.sh vllm/dual4`); only the file paths in the
VARIANTS map are updated.
References updated: BENCHMARKS.md, docs/MULTI_CARD.md, docs/UPSTREAM.md,
models/qwen3.6-27b/CHANGELOG.md, models/qwen3.6-27b/vllm/patches/README.md,
sibling-table cross-references in 7 other compose headers.
AGENTS.md "Topology prefix" row tightened: examples now read
`single · dual · multi3 · multi4 · multi8` (dropped `dual-nvlink`
which is actually an interconnect feature suffix; dropped `quad`
which doesn't exist as a name in our convention).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Rename the 4 shipped Gemma composes to drop the redundant `gemma-` prefix
(model is already named by the parent directory `models/gemma-4-31b/`):
- gemma-mtp.yml → dual.yml (recommended default, MTP implicit
— matches Qwen's `dual.yml` shape)
- gemma-mtp-int8.yml → dual-int8.yml (MTP + INT8 PTH KV variant)
- gemma-mtp-tp1.yml → single.yml (TP=1 single-card variant — boot
OOMs on Ampere 24 GB; for 32 GB+ only)
- gemma-dflash.yml → dual-dflash.yml (DFlash drafter variant — same shape
as Qwen's `dual-dflash.yml`)
Convention going forward (matches what Qwen has done since launch):
- File name: `<topology>-<feature>.yml`, model implied by parent directory
- Topology: `single` | `dual` | `dual-nvlink` | `dual4` | etc.
- Feature suffix: `-turbo` | `-dflash` | `-int8` | `-awq` | etc.
Registry tags in `scripts/switch.sh` keep their existing names
(`vllm/gemma-mtp`, `vllm/gemma-mtp-tp1`, `vllm/gemma-dflash`) — only the
file paths in the VARIANTS map are updated. This preserves backward
compat for users running `bash scripts/switch.sh vllm/gemma-mtp` etc.
Container names (`vllm-gemma-4-31b-mtp` etc.) are unchanged — they already
include the model name in the `vllm-<model>-<feature>` form, so no rename
needed there.
Plus continuing the at-a-glance profile schema rollout from 4d7356a:
profile blocks added to 9 more Qwen composes (bounded-thinking,
carnice-bf16mtp, dual4, dual4-dflash, dual-nvlink, dual-nvlink-turbo,
dual-nvlink-dflash, dual-nvlink-dflash-noviz, long-text). Each declares
Model / Topology / Drafter / KV / Vision / Max ctx / Genesis / Best-for.
References updated across BENCHMARKS.md, README.md, DUAL_CARD.md,
UPSTREAM.md, scripts/setup.sh, scripts/switch.sh, all 6 Gemma compose
cross-references, model patch READMEs, and codex-brief-dflash-int8.md.
Refs: noonghunna/club-3090#67
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
