Decision 2026-06-11: hold DiffusionGemma at 🧪 experimental; promote 🧪 → ⚠️
caveats when #45163 merges into a pinnable release (arch is in no release today
+ mutable :gemma tag = the blocker for caveats). soak-continuous PASS, bench +
8-pack done; the dLLM SSE/streaming-NIAH checks can't cleanly gate. Tracked in
club-3090 task + learnings re-test row.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Make `vllm/diffusiongemma-dual` usable through setup/pull/switch/launch, and
pivot the engine delivery from a baked local image to a SIDELOADED overlay on a
stock, pullable nightly (so non-rig users can actually run it).
Delivery (no baked image):
- Engine `vllm-diffusion-gemma` pins a STOCK `vllm/vllm-openai:nightly-2c9c07c8…`
and sideloads the vendored overlay at boot via install_script (entrypoint).
- `models/diffusiongemma-26b-a4b/vllm/patches/dgemma-overlay/` is the vendored
payload: the FULL stock-vs-`dgemma`-branch `vllm/` delta (123 .py) — a lean
PR-#45163-only overlay version-skews (`build_attn_metadata() got an unexpected
keyword argument 'causal'`: load-bearing dgemma changes live outside the PR
diff) — plus Codex's 3 fixes (marlin K-pad ×2 + diffusion_gemma TP-vocab/dtype).
install.sh cp's it over the installed vllm pkg, fail-loud arch assert.
- base.yml rewired: stock image + overlay mount + install_script entrypoint;
drops the 3 standalone marlin-k-pad mounts (folded into the overlay) and the
baked-image dependency. The dgemma-pr45163 Dockerfile is now the overlay-
regeneration helper, not the runtime engine.
Catalog wiring:
- Model profile `diffusiongemma-26b-a4b` (gemma4-swa-moe backbone + block
diffusion; text; valid_tp [1,2]; kv_calc_supported=false → kvcalc_key SKIP).
- Engine profile `vllm-diffusion-gemma` (stock install + vendored_overlays).
- compose_registry `vllm/diffusiongemma-dual` (port 8042, status upstream-gated).
NO DEFAULTS row — upstream-gated is non-functional, reachable only by slug.
- patches.yml `dgemma-vllm45163-sideload` (install_script + drift_guard) and the
diagnose_profile_cli overlay-path hint.
- docs/UPSTREAM.md row for vllm#45163 (re-pin + drop triggers).
Output-length fix carried in base.yml: lift the model's 256-tok max_new_tokens
default to 16384 via --override-generation-config (keeps the diffusion denoising
params; fixes OWUI truncation + next-turn echo).
Tests: bump the count fixtures for +1 model / +1 engine / +1 compose / +1 registry
entry (test-profiles-compat 6→7 models, 11→12 engines; test-compose-registry-disk
44→45 registry, 45→46 disk). Full gate green except test-compose-registry-disk
local-only redness from untracked nex-n2-mini WIP in the shared tree (CI-clean
validated green: tracked-only disk=46, registry=45, disk-only set all-allowed).
Validated live on 2x RTX 3090: stock nightly + sideloaded overlay boots, serves
coherent output, 262K, via `docker compose -f base.yml up` (the switch/launch path).
Upstream-gated until #45163 merges into a pinnable engine.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
From maintainer feedback on the P1 bundle:
- scripts/setup-image-studio.sh: pre-run plan + confirm prompt (--yes / CI=1 /
non-TTY auto-yes to never hang), --help/usage banner, and a proper "Get started"
block — create your admin account (first sign-up = admin; no creds pre-set),
pick the gemma-4-12b chat model, then 🖼️ to generate. States the fresh-vs-existing
volume wiring caveat.
- services/litellm/config.yaml: add a gemma-4-12b route (-> :8069), so the
image-studio chat brain is reachable through the gateway too (it's the one route
live in image-studio mode; the big-model routes are GPU-mutex with ComfyUI).
Open WebUI still points direct to :8069 by default for a clean picker.
- docs/IMAGE_STUDIO.md: architecture section + ASCII diagram (front-end -> chat /
image; the 2-GPU split; LiteLLM gateway), explicit first-run + how-to-generate-an-
image-in-chat steps, chat-routing explanation, and pin/v0.9.6/secret accuracy fixes.
Live-validated: gemma-4-12b responds through LiteLLM :4000; setup --help + bash -n clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- docs/IMAGE_STUDIO.md (new): the bundle — quickstart, two-front-ends, gpu-mode
modes, measured Ideogram-4 VRAM-by-resolution, chat-model options, the
PersistentConfig "image button missing" fix, single-GPU fallback.
- docs/README.md: IMAGE_STUDIO.md under the user track.
- README.md: one-line image-gen pointer.
- docs/FAQ.md: point the existing image-gen answer at the turnkey bundle.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* Add vllm/qwen-27b-dual-balanced + record 3-way 8-pack A/B (tie)
New "balanced" dual tier: cyankiwi AWQ-BF16-INT4 (int4 group-32 + BF16 mtp,
compressed-tensors) + int8-PTH KV + MTP n=3, TP=2 @262K. Live-validated
2026-06-07 (Marlin WNA16, KV pool 370K tok / 1.41x — the LARGEST of the dual
family, since int4 weights free more VRAM than fp8's 8-bit; ~67 TPS decode).
3-way 8-pack A/B (--full, same harness, same day 2026-06-07):
fast (autoround+fp8KV) 109 · balanced (awq+int8KV) 105 · max (fp8+int8KV) 110
→ a TIE (deterministic packs 64/64/65; spread within +/-5-7 noise).
The short-context 8-pack does NOT separate the quants. So:
- fast (vllm/dual) stays the default;
- balanced/max are framed by SERVING characteristics (KV-pool headroom,
int8-PTH fidelity), NOT behavioral quality — headers/docs say so explicitly.
- The dimension where int8-PTH should win (long-ctx NIAH recall) is an open
follow-up, not claimed here.
Also corrects the stale fast-tier 129/150 baseline: that was the 2026-05-09
harness; today's benchlocal-cli (verifier fixes since) scores the same fast
config at 109/150. Annotated as not-comparable across all four compose headers.
Wiring: awq-bf16-int4 weights_variant (compressed-tensors, cyankiwi repo) +
registry entry (vllm-stable, kv int8_per_token_head, port 8016, SKIP kvcalc,
experimental) + dual-max status_note updated with the A/B + DUAL_CARD rows +
A/B footnote. test-compose-registry-disk 42->43 / 43->44. Full gate green 43/0;
compat C4/C10/C14 pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* Correct balanced KV-pool claim: fast has the largest pool, not balanced
Measured v0.22.0 @262K, TP=2: fast 622K/2.37× > balanced 370K/1.41× > max
295K/1.13×. My initial framing claimed balanced had "the largest KV pool of
the dual family" — wrong: I only compared it to max and forgot fast. Fast's
17.5 GB autoround weights are far lighter than balanced's 27 GB AWQ, so fast
has the biggest pool by a wide margin.
This reframes balanced honestly: it is DOMINATED by the fast tier — slower
(~67 vs ~89 code TPS), smaller KV pool, and tied/below on the 8-pack. Its
only possible edge is int8-PTH KV fidelity > fast's fp8_e5m2 (same size — a
fidelity bet, not a memory one), which is UNPROVEN (the short-ctx 8-pack is
blind to it). Keep only if the long-ctx NIAH A/B (#470) proves it; else
deprecate. Fixed across the compose header, registry status_note, and
DUAL_CARD (rows + footnote); fast row now cites its 622K/2.37× pool.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
DUAL_CARD.md: annotate dual.yml as the "fast" tier (≡ vllm/qwen-27b-dual-fast)
and add the new "max accuracy" row (vllm/qwen-27b-dual-max — FP8 + int8-PTH KV,
🧪, boot/KV-validated, no TPS claim).
MULTI_CARD.md: the "Shipped TP=4 baselines" section documented vllm/dual4 +
vllm/dual4-dflash → composes that no longer exist (multi4 now has only
autoround-int4/mtp.yml + fp8/mtp.yml). Rewrote to vllm/qwen-27b-multi-fast +
-multi-max, carried Whamp's cross-rig fp8/MTP numbers onto multi-fast (same
config, renamed file; labelled cross-rig), and dropped the removed DFlash TP=4
variant (DFlash on Qwen3-Next vLLM is blocked, #39931 — kept its numbers as a
historical note). Fixed the recipe's stale filename (multi4.yml) + container/port
(8014 multi-fast / 8015 multi-max), and the "why one config" section → fast/max pair.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Five verify-or-get-bitten checks distilled from the gemma/qwen quant work:
arch registered in the TARGET engine image (Gemma4Unified→gemma4-unified image
vs Gemma4ForConditionalGeneration→stock v0.22.0); MTP key is family-specific
(mtp_num_hidden_layers vs num_nextn_predict_layers); --reasoning-parser required
or verify-full step 6 false-fails; don't infer a quant from its name (read
quantization_config); KV dtype constrained by the loader (compressed-tensors
can't fp8-KV → int8-PTH). + meta: verify capability against the running image,
not the profile.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Capture the quant/KV findings so others don't re-hit them:
DTYPE_MATRIX.md — new "KV-quant × checkpoint compatibility — the two Ampere traps":
(1) fp8 KV is rejected for compressed-tensors checkpoints (AWQ/FP8/INT8 weights),
flag-independent; auto_round/GPTQ unaffected. (2) int8_per_token_head is NATIVE
in stock v0.22.0 for uniform-head-dim models — #40391 is the Gemma-4-only
(interleaved 256/512 head-dim → page-size unification) adapter, don't copy it.
+ an Ampere KV-dtype picker table; tag the INT8-PTH row native/overlay status.
QUANTIZATION.md — new §4a "Picking a quant by fidelity (KLD) — and where QAT fits":
Phaelon74 KLD ranking (INT8 0.009 < FP8 0.023 < AWQ-BF16-INT4 0.042 < AWQ-INT4
0.051 < AutoRound 0.063); KLD is weights-only (KV-quant adds separate error);
QAT only out-earns PTQ at <=4-bit (8-bit PTQ already near-lossless); dual=fidelity /
single=fit tiering. + int8_per_token_head row + the fp8-guard caveat in §5.
FIX the stale §4 FP8 line: FP8 *weights* DO run on Ampere via Marlin W8A16 (not
"emulated / KV-only") and are a top-fidelity option.
FAQ.md — new Q: "My AWQ/FP8 model errors on --kv-cache-dtype fp8" → use int8-PTH.
Gate 42/42.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Post-#324 the Gemma duals run on vllm-gemma-stable (overlay-carrying), not
vllm-stable. Repoint that engine-pin row and add the missing vllm-stable
(overlay-free) row for the Qwen3-Next + dense + derived-base path. Pure
doc-fidelity; flagged during the #254 Genesis-cleanup.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Completes the coherence section with the three Codex-endorsed additions:
- Rule 7: label every patch by role (compose-local-runtime /
engine-provenance-only / load-bearing-gate / deprecated-history) — the
load-bearing-vs-provenance distinction that the #254 unify got wrong.
- Rule 8: engine identity is stable, version lives in install.spec; channels
(stable/rc/nightly) are separate engine profiles; deviate one compose by
re-pointing its registry engine, not by editing the shared profile.
- A compact 'change type -> tests that prove coherence' matrix.
UPSTREAM cross-link intentionally NOT added here — docs/UPSTREAM.md stays the
single tracker. Refs #254.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replace the purged `vllm-nightly-clean` pin (nightly-bf610c2f, now 404 on
Docker Hub — so the launcher injected a dead image for every slug on it)
with a v0.22.0 two-engine design:
- `vllm-stable`: OVERLAY-FREE v0.22.0 — the broad successor to
vllm-nightly-clean. Serves Qwen3-Next (hybrid 27b + MoE 35B-A3B), generic
`dense` transformers, and uncurated derived-emission bases. Migrates
vllm/qwen-35b-a3b-dual, vllm/dual, vllm/minimal (live-validated on stock
v0.22.0: TP=2 + MTP + tool-call clean; marlin-pad confirmed unnecessary).
Added as the first loads:true arch pin on the qwen + dense rows.
- `vllm-gemma-stable`: KEPT as the OVERLAY-CARRYING v0.22.0 engine (#40391
per-head INT8 KV + #42006 tool-parser) — the Gemma 4 path. Unchanged.
- `vllm-pip-baseline`: the renamed pip `dense` lower-bound (frees the
`vllm-stable` name for the docker engine above).
Why two engines and not one: `vendored_overlays` is LOAD-BEARING, not
documentation. `derived_emittable` (CONTRACT-5) refuses any engine whose
`vendored_overlays != []` as a derived-emission base, and `diagnose-profile`
expects the Gemma overlay declared on its engine. A single engine cannot be
both overlay-free (for derived/Qwen bases) and overlay-carrying (for Gemma
provenance) — so the two stay split. Compose owns patch APPLICATION; engine
owns provenance/compatibility. (An attempt to fold both into one engine broke
4 tests via exactly these two consumers; the split is the validated shape.)
Also adds scripts/tests/test-compose-image-drift.sh: asserts every fixed
`${VLLM_IMAGE:-<tag>}` compose default equals its engine's `install.spec`
(catches bump-the-engine-forget-the-literal drift), and a docs/ADDING_MODELS.md
"Coherence rules" section codifying the compose/engine/patch separation.
42/42 gate green; resolver emits v0.22.0 for all migrated Qwen + Gemma slugs.
The remaining experimental/preview slugs still on vllm-nightly-clean migrate
in follow-up as each is validated, then that engine is deprecated.
Refs #254.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add HF_HUB_OFFLINE + TRANSFORMERS_OFFLINE passthrough (default off) to all
31 vLLM composes, driven by one OFFLINE flag. Even with local weights and a
local --model path, vLLM/transformers resolve config/tokenizer metadata
against HF unless these are set, so air-gapped boots hang/fail (#318).
OFFLINE=1 (or either var) makes vLLM use local files only.
Compose only injects env vars it explicitly lists, so a host-shell
HF_HUB_OFFLINE never reached the container before this. Default 0 = unchanged
behavior online. Documents the air-gap flow in docs/FAQ.md (incl. the gated-
drafter gotcha for *-mtp composes).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
FAQ/DUAL_CARD/HARDWARE told users to `git clone vLLM to /opt/ai/engines/vllm/primary/`
and said the dual composes mount from there — both stale. The marlin-pad patch
(vllm#40361) is now vendored in-repo (models/qwen3.6-27b/vllm/patches/vllm-marlin-pad/
{marlin.py,MPLinearKernel.py}) and overlaid into the stock vLLM image automatically by
each dual compose (delivery_mechanism: site_package_overlay). No user clone, no rig path.
Genericizes the internal path out of the public docs AND fixes the no-longer-needed
clone step in one go. (setup.sh already notes the clone is "previous design".)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Promote the cross-cutting findings out of the omni README so all clubbers
find them:
- FAQ: new "Image & video generation" section — use ComfyUI (not the LLM
stack) on a free card; Open WebUI -> ComfyUI for a unified UI; open-weight
model shortlist; + "why does my image model OOM with a small transformer?"
(the ~8-24GB text-encoder hidden cost; GGUF Q5/Q6 near-lossless).
- DUAL_CARD: "multimodal & image/video models" heads-up — size the full
pipeline (encoder, not just transformer); fp8/int8 KV reaches full context
single-card before reaching for TP/PP on PCIe; ComfyUI for image/video.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
ADDING_MODELS.md gets the full rule + AGENTS.md (CLAUDE.md) the short form:
filename = serving-feature delta from a plain boot — <drafter>[-<kv>][-vision].yml,
suffix order drafter→KV→vision. base.yml = engine-default KV + no drafter; don't
name the default KV (no bf16.yml when bf16 is the default — that's base.yml), only
non-default KV (int8/fp8/tq3). Workload-tuned variants (long-text, tools-text,
bounded-thinking, minimal) keep a descriptive name — recognized exception. Never
encode the weights-quant (that's the <quant>/ dir) or topology (the path).
Existing filenames grandfathered (renaming re-paths registry compose_path).
Resolves the base.yml-vs-bf16.yml-vs-int8.yml spread. New composes only.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Codifies the slug body for NEW models: <engine>/<model>-<topology>-<quant>[-<feature>]
— the compose path components flattened, engine first (e.g. vllm/gemma-12b-dual-bf16-mtp).
Previously only the engine PREFIX rule + quant-slug=weights_variant + path layout were
documented; the body composition was convention-by-example and the catalog was
inconsistent (qwen 'vllm/dual' vs gemma-31b 'vllm/gemma-bf16-mtp' vs gemma-12b's explicit
scheme). Existing slugs are explicitly grandfathered — renaming shipped slugs breaks
switch.sh <slug>, so the convention applies to new models/variants only.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Self-descriptive <engine>/gemma-12b-<topology>-<quant>[-mtp] slugs:
vllm/gemma-12b-mtp -> vllm/gemma-12b-dual-bf16-mtp
vllm/gemma-12b-int8-mtp -> vllm/gemma-12b-single-int8-mtp
beellama/gemma-12b -> beellama/gemma-12b-single-q8kxl
llamacpp/gemma-12b -> llamacpp/gemma-12b-single-q8kxl
Pruned the two no-MTP base composes (MTP is output-lossless and fits the full
262144, so the bases bought nothing): removed vllm/gemma-12b (dual bf16 base) and
vllm/gemma-12b-int8 (single int8 base) + their compose files.
Cross-file: registry keys/entries, kv-calc COMPOSE_ALIAS_TEXT, calibration anchors
(dropped the removed dual-base anchor → kv-calc --calibration now 18/18, test
expectations updated), BENCHMARKS rows (rename 4 / collapse 2 pairs into the MTP
rows), UPSTREAM #39914 row, ADDING_MODELS slug example, disk-count 58/59 -> 56/57.
Hard rename, no back-compat aliases (slugs were day-old 🧪, no users). All stay 🧪
pending per-compose soaks + the ephemeral arch-preview image caveat.
Guard suite 41/41 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
lucianommartins (gemma upstream) confirmed on vllm#39914 that the 12B 256K OOB
was a config.json bug, now fixed: google/gemma-4-12B-it (+ the -assistant draft)
ship max_position_embeddings=262144. With the corrected config the stock
vllm/vllm-openai:gemma4-unified image sizes the Gemma4 RoPE cos/sin cache to
262144 and serves the full 256K — the vendored vllm-gemma4-prope-longctx overlay
(which sized the cache from runtime max_model_len) is redundant.
Re-validated OVERLAY-FREE on 2x 3090 sm_86 TP=2 bf16 (2026-06-04): base + MTP both
boot 262144 on the stock image, NIAH exact-recall PASS at 140K/170K/200K/230K/241K,
MTP spec-decode active. No device-side asserts.
Changes:
- models/gemma-4-12b.yml: bf16 hf_repo unsloth/gemma-4-12b-it -> google/gemma-4-12B-it
(the unsloth mirror still ships 131072 as of 2026-06-04; google carries the fix).
NOTE: re-introduces the Google license gate on base.yml — the MTP draft was
already google-sourced, so the vLLM dual path is google-gated either way.
- base.yml + mtp.yml: remove overlay mount + entrypoint install.sh invoke; rewrite
Max-ctx / Status / weights headers (256K is stock now, no overlay).
- patches.yml: drop the gemma-vllm-gemma4-prope-longctx entry.
- engines/vllm-gemma4-unified.yml: vendored_overlays -> []; notes updated.
- compose_registry.py: comment + both status_notes (no overlay).
- diagnose_profile_cli.py: drop the patch->path map entries.
- delete models/gemma-4-12b/vllm/patches/vllm-gemma4-prope-longctx/.
- UPSTREAM.md: #39914 -> Resolved (config fix supersedes; overlay dropped).
Guard suite 41/41 green (incl. test-patch-attribution).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Posted the runtime-sizing fix + club-3090 patch link as a follow-up on the
issue thread (issuecomment-4622760819). Tracker now reflects the engagement.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
WSL2 NAT gotcha: binding 0.0.0.0 (or BIND_HOST) only reaches the WSL2 VM IP, not the
Windows host LAN IP; launch.sh's 'localhost:8020' is a cosmetic display string, not the
bind. Fix on the Windows side: (A) .wslconfig networkingMode=mirrored (Win11 22H2+,
cleanest), or (B) netsh interface portproxy + a firewall rule (any Windows; WSL IP
changes on reboot). Clarifies .env URL= is the client target, not the server bind.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Implemented by Codex (mcp handoff); validated per the checkpoint report (Claude
re-validates live before the PR). Fixes the gemma4_unified prefill crash past 131072:
stock vllm/vllm-openai:gemma4-unified builds the RoPE cos/sin cache from
config.max_position_embeddings (131072), so positions above that index past the
AOT-compiled cache → CUDA device-side assert. Overlay sizes the cache from runtime
max_model_len and purges the stale TorchInductor cache (marker -cache-v2) so a fresh
262144 graph compiles.
- New overlay models/gemma-4-12b/vllm/patches/vllm-gemma4-prope-longctx/ (install.sh +
guarded diff + README), mounted + install-invoked by both dual composes.
- vllm/gemma-12b + vllm/gemma-12b-mtp default to 262144; registry max_ctx=262144;
models gemma-4-12b max_ctx_supported=262144; engine vendored_overlays + patches.yml
(gemma-vllm-gemma4-prope-longctx, verified) + diagnose path hints; UPSTREAM #39914 row.
- Codex live-validated: base NIAH 155K + 200K exact recall @ 262144 (KV 495,924 tok);
MTP spec-decode survives >131K. Cache shows rand_strided((262144,256/512)).
NOTE: test-compose-registry-disk still red until the single-card slugs are registered
(next commit). Re-test #39914 row: drop overlay when vLLM fixes it upstream.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Registers the new gemma4_unified 12B model (vLLM PR #44429, merged 2026-06-03) as
catalog slugs vllm/gemma-12b (base) + vllm/gemma-12b-mtp (assistant drafter n=4),
so launch.sh/switch.sh resolve them by slug. Status: experimental.
- ModelProfile (family gemma4-unified; 48L/8 full+40 sliding, 16/8 GQA, head_dim
256/512, SWA 1024, max_ctx 131072 — past that vLLM CUDA-OOBs, vllm#39914).
- Engine profile vllm-gemma4-unified pins the gemma4-unified PREVIEW image
(NOT stable v0.22.0, which lacks the arch). Caveat: ephemeral tag — pin a digest
before any Production promotion.
- Drafter gemma-12b-it-assistant (n=4, 0.85GB), bf16 weights, 2 registry entries,
no DEFAULTS row (experimental). kv-calc made gemma-4-12b-aware via the shared
Gemma dense path + measured calibration anchor (384,019 tok / 8.16 GiB @ MTP/TP2/131072).
- ADDING_MODELS.md: name the central registry (compose_registry.py SoT → both
launchers) explicitly in the intro + path-3.
KNOWN: kv-calc under-predicts the live pool (~204K vs measured 384K tok, -47%) — the
shared Gemma dense formula over-prices gemma4_unified global-layer KV; --calibration
GB-verdict still PASSES (TIGHT). Recorded in the calibration YAML header; refine the
gemma4_unified global-KV model as a follow-up. Guard suite 41/41 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The new gemma4_unified arch (vllm PR #44429, merged 2026-06-03) advertises 256K
via p-RoPE but config.json caps max_position_embeddings=131072 (official google ==
unsloth mirror). On the gemma4-unified preview image, NIAH is clean to ~124K but a
single-shot prefill past 131072 CUDA-OOB-crashes (vllm#39914 p-RoPE large-prefill);
a YaRN rope_scaling override is ignored (arch reads rope_parameters). Ship 131072 as
the dual default; re-test 256K on #39914 fix / image maturity. Data point added to
the #39914 thread.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The Profile (at-a-glance) header + required Status field is gate-tested
(test-compose-status-drift) but was documented only in CLAUDE.md/AGENTS.md
(the agent guide) — the human-contributor surfaces never stated it, so a
submission could omit Status and only fail in CI. Surface the requirement
(link, don't duplicate the schema) on every path:
- CONTRIBUTING: new "Profile header on every compose" rule next to one-per-PR.
- PR template: a "Profile header complete" checkbox for compose PRs.
- ADDING_MODELS Step 3: a "Profile header (mandatory)" subsection.
- BRING_YOUR_OWN §4: a header bullet for contributed composes.
Also reciprocal-link the two onboarding docs: ADDING_MODELS now points paths
1-2 (serve/tune/validate without the catalog) to BRING_YOUR_OWN, framing
itself as the promotion step after a config validates there.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
launch.sh/switch.sh are registry-derived and only resolve cataloged slugs, so
they can't list or boot a BYO model. Make explicit: boot the compose directly
via `docker compose -f <path> up -d` + env, drive evals by endpoint
(--url / MODEL= / URL=) not by slug, and pull.sh emits a standalone compose
(--out) you run directly without registration. switch.sh --list is only for
finding a template to copy.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Stock vLLM --kv-cache-dtype is only auto/fp8 (FP8 storage-only on Ampere).
The INT8 KV (INT8 per-token-head / PTH) this stack runs is a vendored engine
patch shipped via the int8.yml composes — you get it by using a patched
compose, not by setting a flag. Clarify so a BYO vLLM user doesn't expect
INT8 KV from stock vLLM.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
vLLM has a single --kv-cache-dtype for both K and V — no per-stream K/V
split. The -ctk/-ctv asymmetric K-high/V-low + -khad/-vhad pattern is
llama.cpp / ik-llama / beellama only. The prior text presented it as a
general lever, which would send a vLLM user after flags that don't exist.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
A user-track front door for testing your own model/compose without the
catalog: serve (pull.sh for safetensors / copy-a-compose for GGUF, any
engine, single or dual) → tune with the fast scripts (verify-full /
verify-stress / bench / quality-test --full|--medium) → validate with the
full rebench-full gate (optional for self-use, mandatory for a registry
contribution). Includes a tuning guide: real context ceiling, NIAH, KV
quant (asymmetric K-high/V-low), MTP/DFlash n-sweeps, batch/ubatch, A/B
discipline. Indexed in docs/README user track + a FAQ pointer.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The Copilot entry recommended `vllm/tools-text`, deprecated 2026-05-31 with the
rest of the Genesis single-card vLLM family. Rewrote to current reality:
- 2× 3090 → vllm/dual (262K); 1× 3090 → no dedicated ≥48K vLLM tools compose
remains, raise vllm/minimal's MAX_MODEL_LEN cautiously (single-card KV is tight,
cf. #35).
- Genesis P68/P69 greeting-stall paragraph reframed as moot — those patches
aren't in the stable v0.22.0 composes (vllm/dual, vllm/minimal).
- Added the qwen3_coder streaming-tool-call caveat (#145, open) since Copilot
streams, with the qwen3_xml workaround.
- Kept the low-max_tokens client caveat; cross-linked the new temperature entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Adds a FAQ entry for the #232 symptom (Hermes/Cline/OpenHands returns a
one-char/empty reply + finish_reason: stop mid-tool-task). Root cause is
sampling temperature 1.0 (Qwen3.6's model-card default); our composes default
to 0.6 server-side, but a client-sent temperature wins in vLLM, so agent
harnesses inheriting 1.0 must set it in their own provider config. Captures the
fix where a confused user will look instead of reopening #232.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
A careful tok/s re-test (bench.sh narrative n=5, MEASURED no-spec controls,
three v0.3.0 images: efe856397 / e0663be / 63abcd3) shows DFlash prose is
net-POSITIVE everywhere — Qwen single Q5 +27% (45.7 vs 35.9 no-spec), Qwen
dual Q8 +52% (35.6 vs 23.4), Gemma single Q4 +28-31% (44.6 vs 34.8). The
earlier "v0.3.0-wide prose-acceptance regression (~0.07 AR / net-negative)"
was a DOUBLE error: (1) over-reading the noisy/prompt-dependent acceptance-rate
diagnostic (the same efe856397 image we logged at ~0.07 now reads ~0.32 AR at
the same tok/s — Anbeeld's #288 AR caution was right), and (2) a wrong no-spec
baseline (we'd used ~37; the real dual-Q8 no-spec is 23.4). The new adaptive-DM
HEAD 63abcd3 is neutral (tok/s flat). Build-arch ruled out (a 3090 runs identical
sm_86 SASS from a fat or single-arch binary).
- compose_registry.py: 5 beellama status_notes (measured slugs assert net-positive;
un-rebenched gemma-duals retract the claim without overclaiming).
- 5 beellama compose Caveats: same correction.
- docs/UPSTREAM.md row 38: regression clause retracted; prose-recovery half of
the promotion gate dropped.
- BENCHMARKS.md: added the v0.3.0 DFlash-vs-no-spec A/B note (no-spec baselines).
- (learnings/qwen3.6-27b.md + gemma-4-31b.md got dated append-only retractions;
reported to Anbeeld at discussion #288.)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
DUAL_CARD.md still presented vllm/dual-dflash + dual-dflash-noviz as live
'peak code TPS' recommendations (pick-by-workload table + two sections with
setup steps), and the Gemma dflash row was a dead link (compose removed in
#451). They were pruned 2026-05-31 (superseded by vllm/dual; stranded on a
now-purged nightly). Add deprecation banners + redirect to the maintained
beellama DFlash duals; fix the stale UPSTREAM.md 'recommended DFlash path
today' line. Surfaces the rationale @hlo-world asked for in #297.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Three-panel format (Serving · Quality · Takeaways) for posting a config's
measured serving + quality numbers. Spec-dec is its own column; core 8-pack
stays /150 with optional humaneval+/lcb/aider in a separate table. Linked
from the docs index; first instance is the Qwen3.6-27B thinking A/B in #221.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Qwen3.6-27B dual+single v0.3.0 control: prose acceptance collapses identically
(single 0.085 == dual 0.090), and Qwen is DeltaNet (NOT SWA-windowed) — so the
SWA-mismatch suspect floated for Gemma is a red herring. It's a cross-model
v0.3.0 DFlash-on-prose regression (code unaffected; Qwen dual code 145 TPS).
Multi-GPU DFlash fix confirmed on both model families.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Rename vllm/gemma-int8→gemma-int8-mtp + vllm/gemma-mtp→gemma-bf16-mtp (both carry MTP n=4; only KV format differs); kv-calc alias fix. Ladder gemma-bf16-mtp default 32K→131K (measured BF16 KV pool 196,527 tok @ 0.95; verify-stress 8/8, NIAH→120K, bench 118.8/154.3, zero TPS regression). Codify the dual-card max-context-priority rule in DUAL_CARD.md. Suite 41/41.
Single beellama/gemma-dflash 128K (caveats; 140K OOMs under load, CTX_SIZE=102400 agent-safe). New dual beellama/gemma-dflash-dual parked upstream-gated (262K works but DFlash multi-GPU broken in 07ac3ce; fixes on v0.3.0 dev branch, no release). switch.sh --list hides upstream-gated (parked) like deprecated with its own note. UPSTREAM row tracks Anbeeld #39. Suite 38/38.
Bump vllm/gemma-mtp (dual, bf16+MTP) to stable v0.22.0 (validated 5/5 coherent). Deprecate vllm/gemma-mtp-tp1: fp8 KV is hardware-impossible for Gemma 4 on Ampere sm_86 (fp8e4nv kernel unsupported; fp8_e5m2 rejected by gemma4 attention allowlist; nvfp4 Blackwell-only), live-confirmed on v0.22.0. Remove the dead DEFAULTS[(gemma-4-31b,vllm,single)] row; repoint launch/setup/preflight single-card Gemma -> beellama/gemma-dflash. Also fixes a pre-existing unterminated NEXT_STEPS_NOTE string in setup.sh's gemma-4-31b arm. Suite 38/38.
* Deprecate Genesis vLLM composes; repoint vLLM single default to vllm/minimal
Mark all 8 Genesis-loading qwen3.6-27b vLLM composes deprecated (registry
status + compose-header Status): vllm/default, vllm/long-text, vllm/long-text-no-mtp,
vllm/long-vision, vllm/bounded-thinking, vllm/tools-text, vllm/dual-turbo,
vllm/dual-tq3-mtp-genesis. Genesis is on hold pending Sandermage's next stable
release; the stack is moving to stable vLLM + beellama single-card.
Repoint DEFAULTS[(qwen3.6-27b, vllm, single)]: vllm/default -> vllm/minimal (the
Genesis-free fp8-KV config). The `vllm/default` token now resolves single ->
vllm/minimal, dual -> vllm/dual. The model single-card default stays beellama/dflash.
Resolver guard tests updated (vllm/default single -> vllm/minimal); SINGLE_CARD.md
example commands redirected off the deprecated slugs. Suite 38/38.
Follow-ups: (1) bump vllm/minimal + vllm/dual to v0.22.0 (Genesis-free -> stable);
(2) SINGLE_CARD.md narrative reframe (single-card vLLM deprecated, beellama default).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* switch.sh --list: hide deprecated by default, reveal with --all
With 9 Genesis composes now deprecated, --list was cluttered. Hide deprecated
variants from the default --list (tally + display loops); --all reveals them
alongside other-topology variants. Footer shows "(+N deprecated hidden --all)"
so they stay discoverable, never silently dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Gemma single-card had NO functional default (vLLM single is upstream-gated at
head_dim=512; ik-llama walls ~24K; stock llama.cpp ~12 TPS). beellama is the
only viable fast single-card Gemma-4 path, so promote it to fill the gap.
Registry: beellama/gemma-dflash experimental -> caveats + new
DEFAULTS[(gemma-4-31b, beellama, single)]. Resolver now returns it for Gemma
single (beellama is #1 in ENGINE_PREFERENCE[single]); Gemma dual unchanged.
Validated #441: 131-200K ctx, 47/88 TPS, 8-pack 109/150 think-off /
114/150 think-on (reasoning net-positive on Gemma). Reuses the multi-arch
image from #271 (sm_86/89/120; sm_89/120 compiled-not-validated).
Docs: compose header (caveats + Quality + ctx ceiling), README Gemma row,
BENCHMARKS row marker + intro, UPSTREAM.md row. Resolver guard test flipped
from a degradation assertion to beellama/gemma-dflash; full suite 38/38.
Re-test trigger: re-point the default to the no-fork mainline path when
llama.cpp#23398 (Gemma-4 MTP) merges (docs/UPSTREAM.md).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Make `beellama/dflash` the single-GPU default for Qwen3.6-27B, served via
an unofficial multi-arch image, and demote ik-llama to the balanced alt.
Decision basis (single-3090, 370 W; bench.sh n=5 + quality-test --full):
- Code TPS ~100 vs ik 69 — fastest single-card 3090 code path.
- 8-pack 107/150 (71%) think-off vs ik 99/150 (66%); 113/150 think-on
(same-session). DFlash is output-lossless, so quality == the Q5_K_S target.
- Context ceiling ladder (measured): 130K comfortable, 160K usable
(115K-tok prefill OK, 0.8 GB headroom), 200K boots but OOMs on prefill.
Ships 102K default; raise to 160K via CTX_SIZE.
- ik still wins narrative TPS (63), VRAM, and shipped vision — kept as the
balanced alt (`--variant ik-llama/iq4ks-mtp`).
Registry: `beellama/dflash` experimental -> caveats + a new
DEFAULTS[(qwen3.6-27b, beellama, single)] row. beellama is already #1 in
ENGINE_PREFERENCE[single], so the resolver now returns it for Qwen single.
Image: unofficial multi-arch ghcr build
`beellama-cpp:multiarch-b9459-07ac3ce` (sm_86/89/120 = RTX 3090/4090/5090),
built from Anbeeld/beellama.cpp .devops/cuda.Dockerfile with
CUDA_DOCKER_ARCH="86;89;120" + GGML_CUDA_FA_ALL_QUANTS=ON. Engine reports
`ARCHS = 860,890,1200`; cuobjdump confirms all 3 SASS. CAVEAT: sm_89/sm_120
are compiled but UNVALIDATED — only sm_86/3090 is verified on-rig.
Also bumps the gemma beellama compose to the same multi-arch image (stays
experimental — not promoted). Docs: README single-card framing,
INFERENCE_ENGINES.md, UPSTREAM.md row, BENCHMARKS.md Qwen beellama row.
Resolver guard tests flipped to the new default; full suite 38/38.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Now that the unofficial sm_86 image is published + public on GHCR, flip the two
beellama composes' BEELLAMA_IMAGE default from the local-only beellama-cpp:local
to ghcr.io/noonghunna/beellama-cpp:sm86-b9459-07ac3ce (immutable commit-pinned
tag) — so 3090 users pull-and-run instead of building. Update caveats/quickstart
+ INFERENCE_ENGINES/UPSTREAM docs (pull for sm_86, build for other arches via
BEELLAMA_IMAGE override). Status stays experimental: the resolver still skips
(NA), so ik-llama remains the single-card default — no behavior change — pending
cross-rig validation before promotion.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Onboards Anbeeld's llama.cpp fork (beellama.cpp) as a registry engine
with two single-card DFlash composes, both experimental pending a
published Docker image:
- beellama/dflash Qwen3.6-27B Q5_K_S + Anbeeld DFlash-IQ4_XS,
q5_0/q4_1 KV, 102K ctx, port 8060
- beellama/gemma-dflash Gemma-4-31B Q4_K_S + Anbeeld DFlash-IQ4_XS,
q5_0/q4_1 KV, 102K ctx, port 8061
Composes reference the locally-built ${BEELLAMA_IMAGE:-beellama-cpp:local}
(entrypoint /app/llama-server) — upstream ships no pullable image, so
they ship as Status: Experimental and surface as (NA: experimental) in
--list (launch is --force-gated). Configs follow the fork's quickstart
docs (Qwen "Precision combo", Gemma "Balanced combo"), deviating only on
the stack-wide --reasoning off default.
Catalog wiring: engine profile beellama-local.yml; two DFlash GGUF
drafters (anbeeld-qwen-dflash, anbeeld-gemma-dflash); compose_registry.py
entries with kvcalc_key="SKIP" (llama.cpp-family, like ik-llama);
weights-map entries (target + draft GGUFs) on both models; q4_1 added to
rtx-3090 supported KV; INFERENCE_ENGINES + UPSTREAM notes; test counts
bumped (engines 8->9, drafters 6->8, registry 45->47, disk 46->48).
Default-resolver invariant preserved: beellama is #1 in
ENGINE_PREFERENCE[single], but its (NA) status + no DEFAULTS row means
the resolver skips it, so qwen3.6-27b single still resolves to
ik-llama/iq4ks-mtp (asserted by test-model-default-resolver.sh). It
auto-promotes only when a published image lands and the composes flip to
production. Full catalog suite green (only the pre-existing
test-submit-bench.sh fixture failure remains).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
