Make `vllm/diffusiongemma-dual` usable through setup/pull/switch/launch, and
pivot the engine delivery from a baked local image to a SIDELOADED overlay on a
stock, pullable nightly (so non-rig users can actually run it).
Delivery (no baked image):
- Engine `vllm-diffusion-gemma` pins a STOCK `vllm/vllm-openai:nightly-2c9c07c8…`
and sideloads the vendored overlay at boot via install_script (entrypoint).
- `models/diffusiongemma-26b-a4b/vllm/patches/dgemma-overlay/` is the vendored
payload: the FULL stock-vs-`dgemma`-branch `vllm/` delta (123 .py) — a lean
PR-#45163-only overlay version-skews (`build_attn_metadata() got an unexpected
keyword argument 'causal'`: load-bearing dgemma changes live outside the PR
diff) — plus Codex's 3 fixes (marlin K-pad ×2 + diffusion_gemma TP-vocab/dtype).
install.sh cp's it over the installed vllm pkg, fail-loud arch assert.
- base.yml rewired: stock image + overlay mount + install_script entrypoint;
drops the 3 standalone marlin-k-pad mounts (folded into the overlay) and the
baked-image dependency. The dgemma-pr45163 Dockerfile is now the overlay-
regeneration helper, not the runtime engine.
Catalog wiring:
- Model profile `diffusiongemma-26b-a4b` (gemma4-swa-moe backbone + block
diffusion; text; valid_tp [1,2]; kv_calc_supported=false → kvcalc_key SKIP).
- Engine profile `vllm-diffusion-gemma` (stock install + vendored_overlays).
- compose_registry `vllm/diffusiongemma-dual` (port 8042, status upstream-gated).
NO DEFAULTS row — upstream-gated is non-functional, reachable only by slug.
- patches.yml `dgemma-vllm45163-sideload` (install_script + drift_guard) and the
diagnose_profile_cli overlay-path hint.
- docs/UPSTREAM.md row for vllm#45163 (re-pin + drop triggers).
Output-length fix carried in base.yml: lift the model's 256-tok max_new_tokens
default to 16384 via --override-generation-config (keeps the diffusion denoising
params; fixes OWUI truncation + next-turn echo).
Tests: bump the count fixtures for +1 model / +1 engine / +1 compose / +1 registry
entry (test-profiles-compat 6→7 models, 11→12 engines; test-compose-registry-disk
44→45 registry, 45→46 disk). Full gate green except test-compose-registry-disk
local-only redness from untracked nex-n2-mini WIP in the shared tree (CI-clean
validated green: tracked-only disk=46, registry=45, disk-only set all-allowed).
Validated live on 2x RTX 3090: stock nightly + sideloaded overlay boots, serves
coherent output, 262K, via `docker compose -f base.yml up` (the switch/launch path).
Upstream-gated until #45163 merges into a pinnable engine.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The arch-confirm blocker named in the 🧪 status note is resolved: the GGUF
header reports general.architecture=qwen35 (standard GQA, 97 layers), NOT the
Qwen3-Next/DeltaNet hybrid the original scaffold guessed. Corrected the profile
to family=qwen35-dense and dropped the fictional GDN/linear-attention fields.
Fixes found while correcting the profile:
- Restored the required `attention_k_eq_v: false` field (compat.py reads it with
bracket notation — dropping it broke load_profiles for the whole catalog).
- Fixed a duplicate-key bug in the weights block: a second `hf_repo:` (DavidAU
base, no MTP head) shadowed PiehSoft's MTP GGUF under yaml.safe_load
last-key-wins — setup.sh would have fetched the wrong, head-less repo.
- Added qwen35-dense to llama-cpp-local supported_model_families (the C10 gate).
Renamed the weights slug mtp-q6k → piehsoft-q6k to follow the provider-quant
idiom used by every other GGUF compose (ubergarm-iq4ks, unsloth-q4km,
byteshape-iq4xs) and to drop the redundant mtp in mtp-q6k/mtp.yml. The serving
filename stays mtp.yml (KV isn't encoded in GGUF-engine filenames). Renamed the
compose dir + on-disk weights dir + all path references in lockstep.
Validation (2× 3090, server-cuda-b9570): verify-full 8/8, verify-stress 8/8
(ceiling ladder filled to 120K / 91% of n_ctx, 0 MiB VRAM growth, needle recall
clean through 90K), 8-pack 105/150 with MTP off==on (spec-dec lossless),
soak-continuous PASS. Guard suite 42/0.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
New opt-in flag: after a catalog model is up + ready, switch.sh --owui upserts
an OpenAI connection in Open WebUI (host.docker.internal:<port>) so it appears
in the chat picker — no manual Admin->Connections step. scripts/lib/owui-register.sh
is conditional (no-op if OWUI not running), idempotent (skips if already present),
and drives OWUI's admin config API via a token forged from the container secret.
Validated end-to-end on Deckard-40B (:8199).
Guard suite 41/42; the 1 failure (test-measurement-record) is pre-existing
bench-output-format drift, unrelated to this change (no switch.sh/owui reference).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The MTP-injected GGUF is NOT a local build (the scaffold's comment was wrong) —
it's published at PiehSoft/Qwen3.6-40B-Deckard-MTP-Q6_K. Wire hf_repo + files so
setup.sh/launch.sh auto-fetch it, and correct the compose/profile comments.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Dense 40B uncensored Qwen3.6 community merge, dual-3090 llama.cpp with
embedded MTP head (Q6_K GGUF, 31 GB). First dual llama.cpp compose in
the catalog; first 'category' field in the registry (uncensored).
Changes:
- Compose: models/qwen3.6-40b-deckard/llama-cpp/compose/dual/mtp-q6k/mtp.yml
Status: 🧪 Unverified (quality /150 + soak pending)
Config: -ngl 99 -ts 1,1 -fa on --spec-type draft-mtp --spec-draft-n-max 2
--cache-type-k q8_0 --cache-type-v q8_0 -c 131072
- Registry: llamacpp/deckard40B-dual-mtp (category=uncensored, experimental)
- DEFAULTS: (qwen3.6-40b-deckard, llamacpp, dual) → slug
- Model profile: scripts/lib/profiles/models/qwen3.6-40b-deckard.yml
- Drafter: qwen-mtp-builtin model_compat extended with qwen3.6-40b-deckard
- LiteLLM route: deckard-40b → :8199
- launch.sh: suggest_default_variant case for Deckard
- BENCHMARKS.md: new section with validated MTP n=2 numbers
- Test fixtures: registry count 43→44, disk 44→49, models 5→6
Wrinkle decisions (see PR description):
1. Dual llama.cpp: uses 'count: all' + '-ts 1,1' (tensor-split layer-split),
matching the validated serving config. No launcher changes needed — the
compose's deploy section handles GPU reservation directly.
2. Engine pin: uses rolling server-cuda tag (matching engine profile spec),
not the b9246 pin other composes use. The validated build (2026-06-09
digest 1c4ff61a) is newer than b9246 and has draft-mtp working. No engine
pin bump for other models.
NOT DONE (live validation — GPUs busy with quality eval):
- Boot/bench/soak/quality NOT run — maintainer to validate post-eval
- Status stays 🧪 until full gate passes
* Add vllm/qwen-27b-dual-balanced + record 3-way 8-pack A/B (tie)
New "balanced" dual tier: cyankiwi AWQ-BF16-INT4 (int4 group-32 + BF16 mtp,
compressed-tensors) + int8-PTH KV + MTP n=3, TP=2 @262K. Live-validated
2026-06-07 (Marlin WNA16, KV pool 370K tok / 1.41x — the LARGEST of the dual
family, since int4 weights free more VRAM than fp8's 8-bit; ~67 TPS decode).
3-way 8-pack A/B (--full, same harness, same day 2026-06-07):
fast (autoround+fp8KV) 109 · balanced (awq+int8KV) 105 · max (fp8+int8KV) 110
→ a TIE (deterministic packs 64/64/65; spread within +/-5-7 noise).
The short-context 8-pack does NOT separate the quants. So:
- fast (vllm/dual) stays the default;
- balanced/max are framed by SERVING characteristics (KV-pool headroom,
int8-PTH fidelity), NOT behavioral quality — headers/docs say so explicitly.
- The dimension where int8-PTH should win (long-ctx NIAH recall) is an open
follow-up, not claimed here.
Also corrects the stale fast-tier 129/150 baseline: that was the 2026-05-09
harness; today's benchlocal-cli (verifier fixes since) scores the same fast
config at 109/150. Annotated as not-comparable across all four compose headers.
Wiring: awq-bf16-int4 weights_variant (compressed-tensors, cyankiwi repo) +
registry entry (vllm-stable, kv int8_per_token_head, port 8016, SKIP kvcalc,
experimental) + dual-max status_note updated with the A/B + DUAL_CARD rows +
A/B footnote. test-compose-registry-disk 42->43 / 43->44. Full gate green 43/0;
compat C4/C10/C14 pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* Correct balanced KV-pool claim: fast has the largest pool, not balanced
Measured v0.22.0 @262K, TP=2: fast 622K/2.37× > balanced 370K/1.41× > max
295K/1.13×. My initial framing claimed balanced had "the largest KV pool of
the dual family" — wrong: I only compared it to max and forgot fast. Fast's
17.5 GB autoround weights are far lighter than balanced's 27 GB AWQ, so fast
has the biggest pool by a wide margin.
This reframes balanced honestly: it is DOMINATED by the fast tier — slower
(~67 vs ~89 code TPS), smaller KV pool, and tied/below on the 8-pack. Its
only possible edge is int8-PTH KV fidelity > fast's fp8_e5m2 (same size — a
fidelity bet, not a memory one), which is UNPROVEN (the short-ctx 8-pack is
blind to it). Keep only if the long-ctx NIAH A/B (#470) proves it; else
deprecate. Fixed across the compose header, registry status_note, and
DUAL_CARD (rows + footnote); fast row now cites its 622K/2.37× pool.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
A symmetric 4-slug family for the Qwen vLLM path:
*-fast = AutoRound INT4 + fp8_e5m2 KV (peak TPS, the proven path)
*-max = official FP8 + int8-PTH KV (higher fidelity @ 262K)
Slugs:
vllm/qwen-27b-dual-fast alias of vllm/dual (AutoRound INT4, TP=2) — production
vllm/qwen-27b-dual-max FP8 + int8-PTH, TP=2 — 🧪 live-validated 2026-06-07
vllm/qwen-27b-multi-fast AutoRound INT4, TP=4 — 🧪 cross-rig (dual is the proxy)
vllm/qwen-27b-multi-max FP8 + int8-PTH, TP=4 — 🧪 cross-rig (dual-max is the proxy)
dual-max validated live on this 2-card rig: FP8 weights load via MarlinFP8
W8A16 on Ampere sm_86, int8_per_token_head KV keeps the full 262K (KV pool
295K tok / 1.13x concurrency), MTP n=3 active, coherent. The multi4 configs
are byte-identical to their dual sibling apart from TP + gpu-count, so they
ship Experimental until a real >=4x 3090 host validates them (this dev rig
has 2 cards). No multi4 DEFAULT added — the resolver degrades to vllm/dual
(honest; no model carries a multi default), and the slugs stay reachable by name.
Catalog wiring:
- models/qwen3.6-27b.yml: new fp8 weights_variant (official e4m3 release,
layer-split safetensors + embedded mtp.safetensors + vision tower)
- vllm-stable.yml: + fp8 weight format (Marlin W8A16 on Ampere) and
+ int8_per_token_head KV — NATIVE on stock v0.22.0 for uniform-head-dim
models (qwen3-next); features flag stays false (no overlay, unlike Gemma's
#40391-provided int8-PTH on vllm-gemma-stable)
- compose_registry.py: 4 entries; test-compose-registry-disk counts 38->42 / 40->43
Full gate green (42/42); compat C4/C5/C10/C14 pass for all 4 slugs.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
n-sweep @370W (dual TP=2): n2 72.9/86.1 . n3 74.0/87.7 . n4 71.6/87.8 wall-TPS
(all within ~3% - the QAT-int4's fast acceptance decay caps the spec benefit).
n=3 is the best balance (top narrative, code tied with n=4, ~20% less drafting:
43% draft-accept vs n=4's 37%), so SPEC_N_MAX default 4->3. The w4a16 compose had
inherited n=4 from the autoround clone, but autoround's slower decay (88/73/59/48%
per-pos) earns n=4 while the QAT-int4's faster decay (64/39/25% at n=3) favors a
lower n - optimal MTP n is quant-specific. Updates the compose default + Quality
field, registry status_note, BENCHMARKS row. Quality unchanged (n = speed only).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
vllm/gemma-31b-qat-w4a16-dual A/B vs the autoround-int4 default (gemma-int8-mtp),
same int8-PTH KV + assistant MTP n=4 (byte-identical speculative-config):
- 8-pack 109/150 vs 105 (+4, within +/-5-7 noise = quality tie; real instructfollow
edge IF 15-vs-8 offset by hermes/cli/TC/RM).
- WEAKER spec-decode: MTP accept-len ~2.4 vs autoround's ~3.9 (per-position
acceptance lower at every position) -> ~30% lower TPS (71.6/87.8 @370W <
autoround 106/139 @230W despite MORE power; AL is the clean power-indep signal).
- Boots clean on stock vllm-gemma-stable (no #44494 workaround - tower arch).
Comparable quality but slower -> autoround-int4 stays the default; stays Experimental.
Updates the compose Quality field + registry status_note + a BENCHMARKS row.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Repoint the single-card Gemma-4-26B-A4B slug in place from the bf16/16K
AWQ+MTP path to an INT8-per-token-head long-context path. Same weights
(cyankiwi AWQ-4bit, Marlin WNA16 MoE), same external MTP drafter (n=4),
same gemma4 tool-call; only the KV format changes (bf16 -> int8_per_token_head)
via the vendored vLLM PR #40391 hybrid-SWA KV page-size overlay. The dual
(vllm/gemma-26ba4b-dual) is untouched.
INT8-PTH (1 byte/token) lifts the single-card ceiling from bf16 16K to a
176K default. Live-validated on 1x RTX 3090 (2026-06-06): #40391 applies
cleanly, int8_per_token_head KV inits, Marlin WNA16 MoE backend, MTP
SpeculativeConfig active; KV pool 183,357 tok >= 176K; coherent generation
(post cudagraph-warmup) + clean gemma4 tool-call. mem_util 0.94, not 0.96:
0.96 passed the upfront KV check but OOM'd in the drafter's later cudagraph
capture (240 MiB free, needed 256) on a single 24 GB card.
- engine vllm-gemma-stable: + gemma4-swa-moe family, + awq/compressed-tensors
weight formats
- patches.yml: + gemma-a4b-vllm-pr40391-rebased (model-scoped copy of the
patch dir; reaches the compose, passes test-patch-attribution)
- compose single/awq/mtp.yml -> single/awq/int8.yml (full rewrite to the
#40391 overlay + entrypoint + int8 KV; Status: Experimental)
- registry + profile_runtime: engine/kv/max_ctx/mem_util/compose_path/entrypoint
- test-generate-from-profile: repoint the clean-derived-seed fixture off this
slug (now overlay-carrying) to vllm/gemma-26ba4b-dual + a tp= override
Status stays Experimental; rebench-full + soak are the #464 follow-up. Full
gate suite green (test-submit-bench pre-existing/environmental, fails on
baseline too).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The C12 budget verdict used a flat absolute floor (`MIN_KV_GB = 0.05 if
qwen3-next-moe else 1.0`) and FAILed any config leaving <1 GB for the growing
KV pool. That false-FAILs KV-light models that demonstrably boot: gemma-4-26b
-a4b single-card AWQ + MTP projects 0.78 GB growing pool, yet the live boot
holds 17,490 tok >= 16,384 max_ctx and serves (sliding_window=1024 makes 25/30
layers' KV trivially cheap).
vLLM's real pre-check is token-capacity based: it boots iff the capped pool
holds >= ONE max_model_len sequence, NOT iff KV >= 1 GB. Growing KV scales
linearly with max_num_seqs, so one sequence's KV is
`kv_pool_requested_gb / max_num_seqs`. Use that as the floor — which also
generalizes (and removes) the old qwen3-next-moe 0.05 special-case.
CAP the floor at the legacy 1 GB: for dense/long-KV configs one sequence needs
many GB and a hard per-seq threshold would false-FAIL measured-working configs
sitting inside the estimator's +-1.5 GB band (e.g. gemma-dual-int8 @262K:
10.71 GB avail vs 10.84 GB/seq, 1.2% short but boots). Net: relax the floor
ONLY for KV-light models, leave dense on its exact prior >=1 GB behavior.
The fix lives in the estimator (where the gap is), NOT in disabling
kv_calc_supported for a config we actively ship the drafter on. Design
bounced with Codex (per-sequence floor); capped-at-1-GB refinement added here
after the pure per-seq form regressed the gemma-4-31b dense duals.
Verdicts after: gemma-26ba4b-single FAIL -> TIGHT (correct); dense duals
unchanged; --calibration still 7/7; full gate 42/42.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Per request: turn on the external MTP drafter for BOTH topologies and
ladder each compose to its empirically-validated max context.
- single/awq/base.yml -> single/awq/mtp.yml (rename per ADDING_MODELS.md:
drafter=MTP named, bf16 KV is the gemma4-on-Ampere default so NOT named).
Adds --speculative-config (gemma-26b-it-assistant n=4) + prefix-caching +
chunked-prefill (parity with the dual). Max ctx 8K -> 16K: KV pool 17,490
tok at 0.92 util on 1x 3090 with the drafter loaded (17,408 would leave
<100 tok headroom). Registry drafter None -> gemma-26b-it-assistant.
- dual/awq/mtp.yml: max ctx 32K -> 262,144 (model max). KV pool 806,821 tok
at 262144/0.92 on 2x 3090 — sliding_window=1024 keeps 25/30 layers' KV
cheap, so the model max fits with 3x headroom.
Both LIVE-validated on the rig 2026-06-06 (stock v0.22.0): single tp=1 +
MTP boots (SpeculativeConfig method=mtp n=4), coherent, clean gemma4
tool-call; dual tp=2 + MTP @262K boots, coherent, clean tool-call.
Registry max_ctx 16384/262144 + single compose_path -> mtp.yml; profile_runtime
mirrors (single gains the speculative_config slot, dual maxlen 262144).
test-generate-from-profile mk_einput now clears the base slug's drafter (the
clean-derived fixtures need a no-drafter seed; the drafter-refuse fixture sets
one explicitly) so it's stable now that gemma-26ba4b-single carries an MTP drafter.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The gemma-4-26b-a4b slugs were the last users of the purged
`vllm-nightly-clean` engine. Migrate them to AWQ-4bit on stock
`vllm-stable` v0.22.0, retiring the Ampere-dead AutoRound path.
Why AutoRound is dead on Ampere: AutoRound int4-MIXED packs to
`uint8b128`, which has no W4A16 kernel on sm_86 (Cutlass/Machete are
Hopper-only; Triton W4A16 takes only uint4b8/uint4). The cyankiwi
AWQ-4bit (pure uint4, compressed-tensors) resolves to Marlin WNA16 MoE
on sm_86 and serves cleanly.
Slug changes (registry 38 -> 36):
- REMOVE vllm/gemma-a4b, vllm/gemma-a4b-awq (AutoRound/AWQ-bf16 duals)
- RENAME vllm/gemma-a4b-single -> vllm/gemma-26ba4b-single (AWQ tp=1, :8040)
- RENAME vllm/gemma-a4b-awq-mtp -> vllm/gemma-26ba4b-dual (AWQ tp=2 + ext MTP, :8041)
Both on vllm-stable, kv bf16 (kv_cache_dtype=auto), kvcalc_key=SKIP
(experimental; rebench-full/soak pending before Production).
PR #40886 (AWQ compressed-tensors MoE key remap) is native in v0.22.0 —
the vendored overlay + install entrypoint are dropped; patch marked
deprecated_on 2026-06-06. vllm-nightly-clean deprecated (final
purged-nightly engine, now zero registry users); gemma4-swa-moe added
to vllm-stable supported_model_families; arch_patches engine_pin
flipped to [email protected].0 loads:true.
Boot-validated on 2x 3090 (2026-06-06): single tp=1 coherent + clean
gemma4 tool-call; dual tp=2 + external MTP (num_spec_tokens=4) coherent
+ clean tool-call, draft layers wired. 3 AutoRound/AWQ-bf16 composes
moved to _archive/ (kept for a future Hopper engine).
Gate: 42/42 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
#327 audit found all 3 DFlash slugs rode the purged nightly-e47c98ef image (the
same dead SHA as the deprecated vllm-nightly-full), and vllm/dual4-dflash was
mislabeled `production` while (a) on a dead image, (b) TP=4 (unrunnable on this
2-card rig), (c) DFlash is blocked on Qwen3-Next per the repo hardware truths
(DeltaNet rollback). Same treatment as the #254 Genesis tail:
- Archived dual-dflash, dual-dflash-noviz, dual4-dflash -> compose/_archive/
(+ manifest); removed their 3 registry entries (41 -> 38).
- DEFAULTS[(qwen,vllm,multi4)] removed -- no functional multi4 vLLM slug remains,
so the resolver degrades multi4 -> dual. The 4-card vLLM tier is now empty
(recoverable from git if demand returns).
- vllm-nightly-dflash engine -> stability: deprecated; its qwen3-next-hybrid arch
pin flipped loads:true -> false (zero registry users; not offered for derive).
- Cascade: calibration (-3 rows), kv-calc alias map + override, patches.yml
(-18 dflash refs), test fixtures (registry-disk 38/40, calibration 9/9;
launch-compat/list-topology-filter/model-default-resolver/pullgate-gates/pull
adapted to the now-empty multi4 + nightly-slug categories).
Completes the nightly-engine cleanup: vllm-nightly-mtp/full/dflash all deprecated
with zero registry users; only the gated vllm-nightly-clean (#326) remains.
42/42 gate green. Refs #327.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The Genesis-free 35B-A3B single-card preview was pinned to the purged
vllm-nightly-clean (dead image). Repoint to vllm-stable (v0.22.0, which
includes PR #42521 qwen3_5_moe weight loading) + refresh the now-stale
Genesis/nightly header prose. Boot-validated on v0.22.0 @ 0.95 earlier (#323);
resolver emits v0.22.0; 42/42 green. Refs #254.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replace the purged `vllm-nightly-clean` pin (nightly-bf610c2f, now 404 on
Docker Hub — so the launcher injected a dead image for every slug on it)
with a v0.22.0 two-engine design:
- `vllm-stable`: OVERLAY-FREE v0.22.0 — the broad successor to
vllm-nightly-clean. Serves Qwen3-Next (hybrid 27b + MoE 35B-A3B), generic
`dense` transformers, and uncurated derived-emission bases. Migrates
vllm/qwen-35b-a3b-dual, vllm/dual, vllm/minimal (live-validated on stock
v0.22.0: TP=2 + MTP + tool-call clean; marlin-pad confirmed unnecessary).
Added as the first loads:true arch pin on the qwen + dense rows.
- `vllm-gemma-stable`: KEPT as the OVERLAY-CARRYING v0.22.0 engine (#40391
per-head INT8 KV + #42006 tool-parser) — the Gemma 4 path. Unchanged.
- `vllm-pip-baseline`: the renamed pip `dense` lower-bound (frees the
`vllm-stable` name for the docker engine above).
Why two engines and not one: `vendored_overlays` is LOAD-BEARING, not
documentation. `derived_emittable` (CONTRACT-5) refuses any engine whose
`vendored_overlays != []` as a derived-emission base, and `diagnose-profile`
expects the Gemma overlay declared on its engine. A single engine cannot be
both overlay-free (for derived/Qwen bases) and overlay-carrying (for Gemma
provenance) — so the two stay split. Compose owns patch APPLICATION; engine
owns provenance/compatibility. (An attempt to fold both into one engine broke
4 tests via exactly these two consumers; the split is the validated shape.)
Also adds scripts/tests/test-compose-image-drift.sh: asserts every fixed
`${VLLM_IMAGE:-<tag>}` compose default equals its engine's `install.spec`
(catches bump-the-engine-forget-the-literal drift), and a docs/ADDING_MODELS.md
"Coherence rules" section codifying the compose/engine/patch separation.
42/42 gate green; resolver emits v0.22.0 for all migrated Qwen + Gemma slugs.
The remaining experimental/preview slugs still on vllm-nightly-clean migrate
in follow-up as each is validated, then that engine is deprecated.
Refs #254.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
vllm/qwen-a3b-preview-single OOM'd at the KV-cache check on stock vLLM
v0.22.0 — ~20 GB weights on one 24 GB card left 0.07 GiB for KV at
mem_util 0.92. v0.22.0's CUDA-graph memory profiling (default since
v0.21.0) trims effective KV vs the older nightly the preview was tuned on.
Bump the compose default + registry mem_util to 0.95 (the documented
headless-Linux single-card default). Boots clean: KV 0.78 GiB / 31,129
tokens, coherent. WSL2 users set 0.94 via .env (docs/FAQ.md). Surfaced
during #316 boot-validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The autoround-int4 weights entry pointed at
Qwen/Qwen3-MoE-A3B-Instruct-AutoRound-Int4-mixed, which never existed on
HF — a fresh `setup.sh qwen3.6-35b-a3b` 404'd on download (#316). Correct
to the real repo Intel/Qwen3.6-35B-A3B-int4-mixed-AutoRound (ungated;
11 safetensors byte-match our on-disk copy).
It stayed hidden because the weights were pre-downloaded on the maintainer
rig (never re-fetched) and the weights-registry guard test asserted the
same bad string. Guard corrected too; resolve-check hardening filed as
#320 and a revision-pin schema as #319.
Note: upstream renamed Intel/Qwen3.6-35B-A3B-int4-AutoRound -> -int4-mixed-
AutoRound and re-quantized 2026-05-01; current HEAD 65f69c7 re-validated on
2x3090 stock v0.22.0 TP=2 (verify-full 8/8, decode ~172 TPS, coherent).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Self-descriptive <engine>/gemma-12b-<topology>-<quant>[-mtp] slugs:
vllm/gemma-12b-mtp -> vllm/gemma-12b-dual-bf16-mtp
vllm/gemma-12b-int8-mtp -> vllm/gemma-12b-single-int8-mtp
beellama/gemma-12b -> beellama/gemma-12b-single-q8kxl
llamacpp/gemma-12b -> llamacpp/gemma-12b-single-q8kxl
Pruned the two no-MTP base composes (MTP is output-lossless and fits the full
262144, so the bases bought nothing): removed vllm/gemma-12b (dual bf16 base) and
vllm/gemma-12b-int8 (single int8 base) + their compose files.
Cross-file: registry keys/entries, kv-calc COMPOSE_ALIAS_TEXT, calibration anchors
(dropped the removed dual-base anchor → kv-calc --calibration now 18/18, test
expectations updated), BENCHMARKS rows (rename 4 / collapse 2 pairs into the MTP
rows), UPSTREAM #39914 row, ADDING_MODELS slug example, disk-count 58/59 -> 56/57.
Hard rename, no back-compat aliases (slugs were day-old 🧪, no users). All stay 🧪
pending per-compose soaks + the ephemeral arch-preview image caveat.
Guard suite 41/41 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
vllm/gemma-12b-int8-mtp — base int8 + assistant external drafter, single 3090.
MTP fits the FULL 262144 on one card (drafter resident, KV pool ~310K tok, 1.18x
at 262K, ~20.7 GB — no ctx step-down vs base.yml).
n-sweep 2026-06-04 (code-gen probe @262144), accept-len in parens:
n=2 96.7 (2.62) · n=3 115.5 (3.33) · n=4 117.0 (3.67) · n=5 122.5 (4.06) TPS
vs ~50 no-MTP. Default SPEC_N=4 (near-peak, matches dual, robust; probe is
code-only — MTP's best case, so higher n risks draft-waste on narrative).
SPEC_N=5 for code-max.
Wiring: compose_registry vllm/gemma-12b-int8-mtp (tp=1, drafter gemma-12b-it-
assistant), kv-calc alias gemma-single-int8-mtp, disk-count 57/58 -> 58/59.
Guard suite 41/41 green. 8-pack quality (int8, lossless vs MTP) folds in next.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
lucianommartins (gemma upstream) confirmed on vllm#39914 that the 12B 256K OOB
was a config.json bug, now fixed: google/gemma-4-12B-it (+ the -assistant draft)
ship max_position_embeddings=262144. With the corrected config the stock
vllm/vllm-openai:gemma4-unified image sizes the Gemma4 RoPE cos/sin cache to
262144 and serves the full 256K — the vendored vllm-gemma4-prope-longctx overlay
(which sized the cache from runtime max_model_len) is redundant.
Re-validated OVERLAY-FREE on 2x 3090 sm_86 TP=2 bf16 (2026-06-04): base + MTP both
boot 262144 on the stock image, NIAH exact-recall PASS at 140K/170K/200K/230K/241K,
MTP spec-decode active. No device-side asserts.
Changes:
- models/gemma-4-12b.yml: bf16 hf_repo unsloth/gemma-4-12b-it -> google/gemma-4-12B-it
(the unsloth mirror still ships 131072 as of 2026-06-04; google carries the fix).
NOTE: re-introduces the Google license gate on base.yml — the MTP draft was
already google-sourced, so the vLLM dual path is google-gated either way.
- base.yml + mtp.yml: remove overlay mount + entrypoint install.sh invoke; rewrite
Max-ctx / Status / weights headers (256K is stock now, no overlay).
- patches.yml: drop the gemma-vllm-gemma4-prope-longctx entry.
- engines/vllm-gemma4-unified.yml: vendored_overlays -> []; notes updated.
- compose_registry.py: comment + both status_notes (no overlay).
- diagnose_profile_cli.py: drop the patch->path map entries.
- delete models/gemma-4-12b/vllm/patches/vllm-gemma4-prope-longctx/.
- UPSTREAM.md: #39914 -> Resolved (config fix supersedes; overlay dropped).
Guard suite 41/41 green (incl. test-patch-attribution).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Implemented by Codex (mcp handoff); validated per the checkpoint report (Claude
re-validates live before the PR). Fixes the gemma4_unified prefill crash past 131072:
stock vllm/vllm-openai:gemma4-unified builds the RoPE cos/sin cache from
config.max_position_embeddings (131072), so positions above that index past the
AOT-compiled cache → CUDA device-side assert. Overlay sizes the cache from runtime
max_model_len and purges the stale TorchInductor cache (marker -cache-v2) so a fresh
262144 graph compiles.
- New overlay models/gemma-4-12b/vllm/patches/vllm-gemma4-prope-longctx/ (install.sh +
guarded diff + README), mounted + install-invoked by both dual composes.
- vllm/gemma-12b + vllm/gemma-12b-mtp default to 262144; registry max_ctx=262144;
models gemma-4-12b max_ctx_supported=262144; engine vendored_overlays + patches.yml
(gemma-vllm-gemma4-prope-longctx, verified) + diagnose path hints; UPSTREAM #39914 row.
- Codex live-validated: base NIAH 155K + 200K exact recall @ 262144 (KV 495,924 tok);
MTP spec-decode survives >131K. Cache shows rand_strided((262144,256/512)).
NOTE: test-compose-registry-disk still red until the single-card slugs are registered
(next commit). Re-test #39914 row: drop overlay when vLLM fixes it upstream.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Two LOW findings from the ship-with-notes review of 0f6f861:
- docstring: generalize the gemma 'full/sliding' layer counts (10/50 on 31B,
8/40 on 12B — read from spec, not hardcoded) + point at the measured override.
- calibration YAML: note the measured 22,816 B/tok/card is total-avail-KV/tokens,
NOT net of the ~0.08 GB/card sliding-fixed term (kv-calc adds that separately —
an intentional, tiny, conservative double-count, not an exact decomposition).
Comment-only; --calibration unchanged (19/19).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The 31B-derived gemma4-swa-dense global-only formula priced the 12B's growing KV
at 32,768 B/tok/card (8 full x 8 kv x 512 x bpe2 / TP2), but the live gemma4_unified
pool measured 22,816 B/tok/card (8.16 GiB / 384,019 tok @ 131K/TP2/0.90/MTP). Add a
gemma-12b-only measured override (measured_kv_growing_bpt_tp1=45,632) in the
gemma4-swa-dense KV branch — 31B keeps the formula (untouched, calibration byte-identical).
Token-capacity prediction 204K -> 315K vs measured 384K; residual is the conservative
shared cudagraph_overhead (handed to Codex to model principled-ly). Guard 41/41.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Registers the new gemma4_unified 12B model (vLLM PR #44429, merged 2026-06-03) as
catalog slugs vllm/gemma-12b (base) + vllm/gemma-12b-mtp (assistant drafter n=4),
so launch.sh/switch.sh resolve them by slug. Status: experimental.
- ModelProfile (family gemma4-unified; 48L/8 full+40 sliding, 16/8 GQA, head_dim
256/512, SWA 1024, max_ctx 131072 — past that vLLM CUDA-OOBs, vllm#39914).
- Engine profile vllm-gemma4-unified pins the gemma4-unified PREVIEW image
(NOT stable v0.22.0, which lacks the arch). Caveat: ephemeral tag — pin a digest
before any Production promotion.
- Drafter gemma-12b-it-assistant (n=4, 0.85GB), bf16 weights, 2 registry entries,
no DEFAULTS row (experimental). kv-calc made gemma-4-12b-aware via the shared
Gemma dense path + measured calibration anchor (384,019 tok / 8.16 GiB @ MTP/TP2/131072).
- ADDING_MODELS.md: name the central registry (compose_registry.py SoT → both
launchers) explicitly in the intro + path-3.
KNOWN: kv-calc under-predicts the live pool (~204K vs measured 384K tok, -47%) — the
shared Gemma dense formula over-prices gemma4_unified global-layer KV; --calibration
GB-verdict still PASSES (TIGHT). Recorded in the calibration YAML header; refine the
gemma4_unified global-KV model as a follow-up. Guard suite 41/41 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
A careful tok/s re-test (bench.sh narrative n=5, MEASURED no-spec controls,
three v0.3.0 images: efe856397 / e0663be / 63abcd3) shows DFlash prose is
net-POSITIVE everywhere — Qwen single Q5 +27% (45.7 vs 35.9 no-spec), Qwen
dual Q8 +52% (35.6 vs 23.4), Gemma single Q4 +28-31% (44.6 vs 34.8). The
earlier "v0.3.0-wide prose-acceptance regression (~0.07 AR / net-negative)"
was a DOUBLE error: (1) over-reading the noisy/prompt-dependent acceptance-rate
diagnostic (the same efe856397 image we logged at ~0.07 now reads ~0.32 AR at
the same tok/s — Anbeeld's #288 AR caution was right), and (2) a wrong no-spec
baseline (we'd used ~37; the real dual-Q8 no-spec is 23.4). The new adaptive-DM
HEAD 63abcd3 is neutral (tok/s flat). Build-arch ruled out (a 3090 runs identical
sm_86 SASS from a fat or single-arch binary).
- compose_registry.py: 5 beellama status_notes (measured slugs assert net-positive;
un-rebenched gemma-duals retract the claim without overclaiming).
- 5 beellama compose Caveats: same correction.
- docs/UPSTREAM.md row 38: regression clause retracted; prose-recovery half of
the promotion gate dropped.
- BENCHMARKS.md: added the v0.3.0 DFlash-vs-no-spec A/B note (no-spec baselines).
- (learnings/qwen3.6-27b.md + gemma-4-31b.md got dated append-only retractions;
reported to Anbeeld at discussion #288.)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The 3 beellama composes still defaulting to the old self-hosted
multiarch-b9459-07ac3ce (the 2 singles + the gemma-q4ks dual) and the two
single-card registry status_notes predate the #296 BEELLAMA_IMAGE
centralization. b9459 is the pre-v0.3.0 build — the gemma-q4ks dual even
documented that its default "STILL has the [multi-GPU DFlash] bug". A direct
`docker compose up` (no launcher) would pull that stale image.
- image: defaults → ghcr.io/noonghunna/beellama-cpp:multiarch-v0.3.0-efe856397
(matches the q8kxl duals; broad sm_86/89/120 compat for direct-compose).
- Caveats / Pull-the-image / status_note prose → launchers inject Anbeeld's
official server-cuda-v0.3.0 (sm_86/89); 5090/sm_120 uses the multiarch build.
- Drop the obsolete "no official Docker yet (v0.3.0 WIP)" claim; note the
v0.3.0 prose-DFlash acceptance regression (code unaffected) instead.
- Preserve the two historical b9459 references (the 2026-05-30 benchmark
attribution + the old-build prose-accept comparison).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Rename vllm/gemma-int8→gemma-int8-mtp + vllm/gemma-mtp→gemma-bf16-mtp (both carry MTP n=4; only KV format differs); kv-calc alias fix. Ladder gemma-bf16-mtp default 32K→131K (measured BF16 KV pool 196,527 tok @ 0.95; verify-stress 8/8, NIAH→120K, bench 118.8/154.3, zero TPS regression). Codify the dual-card max-context-priority rule in DUAL_CARD.md. Suite 41/41.
Single beellama/gemma-dflash 128K (caveats; 140K OOMs under load, CTX_SIZE=102400 agent-safe). New dual beellama/gemma-dflash-dual parked upstream-gated (262K works but DFlash multi-GPU broken in 07ac3ce; fixes on v0.3.0 dev branch, no release). switch.sh --list hides upstream-gated (parked) like deprecated with its own note. UPSTREAM row tracks Anbeeld #39. Suite 38/38.
Adds the max context size to switch.sh --list. registry-emit threads a ctx label through the variant TSV (before status_note); switch.sh renders it rightmost. Production -> bare ctx; caveats/NA -> folded into the health paren (comma-separated). Rounded to nearest K (163840->164K; 32768 reads 33K). Shows registry vs compose default as a single value when they match, 'validated/compose' (e.g. 164K/200K) when they drift -- the only 3 drifts are experimental lanes; all production/caveats match. Suite 38/38.
Bump vllm/gemma-mtp (dual, bf16+MTP) to stable v0.22.0 (validated 5/5 coherent). Deprecate vllm/gemma-mtp-tp1: fp8 KV is hardware-impossible for Gemma 4 on Ampere sm_86 (fp8e4nv kernel unsupported; fp8_e5m2 rejected by gemma4 attention allowlist; nvfp4 Blackwell-only), live-confirmed on v0.22.0. Remove the dead DEFAULTS[(gemma-4-31b,vllm,single)] row; repoint launch/setup/preflight single-card Gemma -> beellama/gemma-dflash. Also fixes a pre-existing unterminated NEXT_STEPS_NOTE string in setup.sh's gemma-4-31b arm. Suite 38/38.
* Deprecate Genesis vLLM composes; repoint vLLM single default to vllm/minimal
Mark all 8 Genesis-loading qwen3.6-27b vLLM composes deprecated (registry
status + compose-header Status): vllm/default, vllm/long-text, vllm/long-text-no-mtp,
vllm/long-vision, vllm/bounded-thinking, vllm/tools-text, vllm/dual-turbo,
vllm/dual-tq3-mtp-genesis. Genesis is on hold pending Sandermage's next stable
release; the stack is moving to stable vLLM + beellama single-card.
Repoint DEFAULTS[(qwen3.6-27b, vllm, single)]: vllm/default -> vllm/minimal (the
Genesis-free fp8-KV config). The `vllm/default` token now resolves single ->
vllm/minimal, dual -> vllm/dual. The model single-card default stays beellama/dflash.
Resolver guard tests updated (vllm/default single -> vllm/minimal); SINGLE_CARD.md
example commands redirected off the deprecated slugs. Suite 38/38.
Follow-ups: (1) bump vllm/minimal + vllm/dual to v0.22.0 (Genesis-free -> stable);
(2) SINGLE_CARD.md narrative reframe (single-card vLLM deprecated, beellama default).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* switch.sh --list: hide deprecated by default, reveal with --all
With 9 Genesis composes now deprecated, --list was cluttered. Hide deprecated
variants from the default --list (tally + display loops); --all reveals them
alongside other-topology variants. Footer shows "(+N deprecated hidden --all)"
so they stay discoverable, never silently dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Gemma single-card had NO functional default (vLLM single is upstream-gated at
head_dim=512; ik-llama walls ~24K; stock llama.cpp ~12 TPS). beellama is the
only viable fast single-card Gemma-4 path, so promote it to fill the gap.
Registry: beellama/gemma-dflash experimental -> caveats + new
DEFAULTS[(gemma-4-31b, beellama, single)]. Resolver now returns it for Gemma
single (beellama is #1 in ENGINE_PREFERENCE[single]); Gemma dual unchanged.
Validated #441: 131-200K ctx, 47/88 TPS, 8-pack 109/150 think-off /
114/150 think-on (reasoning net-positive on Gemma). Reuses the multi-arch
image from #271 (sm_86/89/120; sm_89/120 compiled-not-validated).
Docs: compose header (caveats + Quality + ctx ceiling), README Gemma row,
BENCHMARKS row marker + intro, UPSTREAM.md row. Resolver guard test flipped
from a degradation assertion to beellama/gemma-dflash; full suite 38/38.
Re-test trigger: re-point the default to the no-fork mainline path when
llama.cpp#23398 (Gemma-4 MTP) merges (docs/UPSTREAM.md).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Make `beellama/dflash` the single-GPU default for Qwen3.6-27B, served via
an unofficial multi-arch image, and demote ik-llama to the balanced alt.
Decision basis (single-3090, 370 W; bench.sh n=5 + quality-test --full):
- Code TPS ~100 vs ik 69 — fastest single-card 3090 code path.
- 8-pack 107/150 (71%) think-off vs ik 99/150 (66%); 113/150 think-on
(same-session). DFlash is output-lossless, so quality == the Q5_K_S target.
- Context ceiling ladder (measured): 130K comfortable, 160K usable
(115K-tok prefill OK, 0.8 GB headroom), 200K boots but OOMs on prefill.
Ships 102K default; raise to 160K via CTX_SIZE.
- ik still wins narrative TPS (63), VRAM, and shipped vision — kept as the
balanced alt (`--variant ik-llama/iq4ks-mtp`).
Registry: `beellama/dflash` experimental -> caveats + a new
DEFAULTS[(qwen3.6-27b, beellama, single)] row. beellama is already #1 in
ENGINE_PREFERENCE[single], so the resolver now returns it for Qwen single.
Image: unofficial multi-arch ghcr build
`beellama-cpp:multiarch-b9459-07ac3ce` (sm_86/89/120 = RTX 3090/4090/5090),
built from Anbeeld/beellama.cpp .devops/cuda.Dockerfile with
CUDA_DOCKER_ARCH="86;89;120" + GGML_CUDA_FA_ALL_QUANTS=ON. Engine reports
`ARCHS = 860,890,1200`; cuobjdump confirms all 3 SASS. CAVEAT: sm_89/sm_120
are compiled but UNVALIDATED — only sm_86/3090 is verified on-rig.
Also bumps the gemma beellama compose to the same multi-arch image (stays
experimental — not promoted). Docs: README single-card framing,
INFERENCE_ENGINES.md, UPSTREAM.md row, BENCHMARKS.md Qwen beellama row.
Resolver guard tests flipped to the new default; full suite 38/38.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
