The silent-empty classifier flagged any HTTP-200 turn with decode_tps==0
and t_ms>=1s. But cmd_run zeroes decode_tps when decode_s < 0.1s OR
completion_tokens <= 0 — so a turn that DID produce output but decoded it
in a sub-100ms burst got mislabelled silent-empty. That burst case is
exactly a tool-call turn (small tool_calls payload, empty content) or a
block-diffusion canvas emitted all at once. The DiffusionGemma re-soak
(2026-06-11) false-flagged 4/25 tool-call turns this way.
Switch the discriminator to completion_tokens == 0 (genuine no-output)
when that column is present; fall back to the legacy decode_tps==0 proxy
for pre-2026-05-04 CSVs that lack the column. The summary.md label
already said '0 completion tokens' — now the logic matches it.
Validated: dgemma re-soak turn-log 4->0 silent-empty; synthetic genuine
empty (completion_tokens=0) still flagged; tool-call turn (ct=23,
decode_tps=0) no longer flagged; old column-less CSV still uses the
decode_tps==0 fallback.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The wrapper exposed --enable-thinking (force thinking on for every pack)
but had no force-off counterpart, so a clean all-off arm of a reasoning
A/B wasn't reachable through the wrapper — only the mixed per-pack
default. Add --no-thinking / NO_THINKING=1, forwarded to benchlocal-cli
--no-thinking, mutually exclusive with --enable-thinking. Keeps the
wrapper's localhost hermes-resolve + timeout sizing on both arms.
Validated live driving the DiffusionGemma 8-pack off/on A/B.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
steamEngineer's single-card beellama-qwen36-27b soak failed with
'no running club-3090 container found' — auto_container()'s glob
matched vllm-/llama-cpp-/ik-llama-/sglang- but not beellama-, so the
container fell through to the die(). (ik-llama report #365 soaked
fine because ik-llama-qwen36-27b *was* in the glob.)
Add beellama-qwen36-27b / -qwen36-35b-a3b / -gemma4-31b / -gemma4-12b
(note beellama names gemma as 'gemma4-31b', not 'gemma-4-31b'); the
'^' prefix anchor covers the -dual / -dual-dflash siblings. Update
the die() message + the two CONTAINER help-text blurbs to list
beellama-. Sibling of #403 (llama-cpp/ik glob gap).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
#358 wired the registry/profile/patches but missed setup.sh's own hardcoded
per-model dispatch, so `setup.sh diffusiongemma-26b-a4b` errored 'unsupported
model'. Add the model_label, the main weights dispatch (PRIMARY_WEIGHT_KEY=
diffusiongemma-26b-a4b:fp8 — fp8-only, default WEIGHTS=autoround is a no-op),
the Supported: list, and a next-steps hint (switch.sh --force / gpu-mode dgemma).
Verified: weights.py entry diffusiongemma-26b-a4b:fp8 → RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic;
test-model-weights-registry green. Completes the 'users can use setup' wiring.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds `gpu-mode dgemma` (alias `diffusiongemma`): stops all other GPU models
(it's TP=2, both cards — shared 8199 slot with deckard) then launches the
DiffusionGemma compose (base.yml, PORT=8199) + wires it into Open WebUI.
Mirrors mode_deckard. Also stops it in mode_off.
Dual-card, so before switching to another model run `gpu-mode off` first
(single-card modes don't individually stop it — noted in usage). The mode
launches whatever base.yml is on the tree (the official :gemma image once
the dgemma image-adoption PR lands).
Validated live on 2x RTX 3090: `gpu-mode dgemma` stops peers, recreates the
container, starts services, owui-registers :8199, serves. bash -n clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add FORCE_TOKENS=N: forces EXACTLY N output tokens per run (max_tokens +
min_tokens + ignore_eos), overriding MAX_TOKENS_NARR/CODE. Required to bench
DIFFUSION LMs at a chosen output size — they self-terminate early (~1-2K words),
so raising the cap alone won't lengthen the generation; you must force the
length to measure sustained throughput. Off (0) by default → no behavior change
for existing runs. vLLM-oriented (ignore_eos/min_tokens).
Validated on diffusiongemma-26b-a4b: FORCE_TOKENS=2000 → exactly 2000 tokens,
~257 wall_TPS vs ~177 at the default self-terminated narrative (block-parallel
diffusion throughput rises with output size — invisible to the default bench).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Make `vllm/diffusiongemma-dual` usable through setup/pull/switch/launch, and
pivot the engine delivery from a baked local image to a SIDELOADED overlay on a
stock, pullable nightly (so non-rig users can actually run it).
Delivery (no baked image):
- Engine `vllm-diffusion-gemma` pins a STOCK `vllm/vllm-openai:nightly-2c9c07c8…`
and sideloads the vendored overlay at boot via install_script (entrypoint).
- `models/diffusiongemma-26b-a4b/vllm/patches/dgemma-overlay/` is the vendored
payload: the FULL stock-vs-`dgemma`-branch `vllm/` delta (123 .py) — a lean
PR-#45163-only overlay version-skews (`build_attn_metadata() got an unexpected
keyword argument 'causal'`: load-bearing dgemma changes live outside the PR
diff) — plus Codex's 3 fixes (marlin K-pad ×2 + diffusion_gemma TP-vocab/dtype).
install.sh cp's it over the installed vllm pkg, fail-loud arch assert.
- base.yml rewired: stock image + overlay mount + install_script entrypoint;
drops the 3 standalone marlin-k-pad mounts (folded into the overlay) and the
baked-image dependency. The dgemma-pr45163 Dockerfile is now the overlay-
regeneration helper, not the runtime engine.
Catalog wiring:
- Model profile `diffusiongemma-26b-a4b` (gemma4-swa-moe backbone + block
diffusion; text; valid_tp [1,2]; kv_calc_supported=false → kvcalc_key SKIP).
- Engine profile `vllm-diffusion-gemma` (stock install + vendored_overlays).
- compose_registry `vllm/diffusiongemma-dual` (port 8042, status upstream-gated).
NO DEFAULTS row — upstream-gated is non-functional, reachable only by slug.
- patches.yml `dgemma-vllm45163-sideload` (install_script + drift_guard) and the
diagnose_profile_cli overlay-path hint.
- docs/UPSTREAM.md row for vllm#45163 (re-pin + drop triggers).
Output-length fix carried in base.yml: lift the model's 256-tok max_new_tokens
default to 16384 via --override-generation-config (keeps the diffusion denoising
params; fixes OWUI truncation + next-turn echo).
Tests: bump the count fixtures for +1 model / +1 engine / +1 compose / +1 registry
entry (test-profiles-compat 6→7 models, 11→12 engines; test-compose-registry-disk
44→45 registry, 45→46 disk). Full gate green except test-compose-registry-disk
local-only redness from untracked nex-n2-mini WIP in the shared tree (CI-clean
validated green: tracked-only disk=46, registry=45, disk-only set all-allowed).
Validated live on 2x RTX 3090: stock nightly + sideloaded overlay boots, serves
coherent output, 262K, via `docker compose -f base.yml up` (the switch/launch path).
Upstream-gated until #45163 merges into a pinnable engine.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
`gpu-mode` is a fixed mode dispatcher; there was no way to launch the
newly-promoted Deckard-40B through it. Add a `deckard` mode mirroring the
27b/gemma pattern:
- DECKARD_DIR + start_deckard/stop_deckard helpers (compose_at on the
piehsoft-q6k/mtp.yml compose; MODEL_DIR resolves from .env via --env-file,
same as the vLLM modes).
- mode_deckard: stops all other GPU models (Deckard layer-splits across both
cards), boots Deckard on :8199, brings up the support services, and wires it
into Open WebUI via the switch.sh --owui helper (best-effort; Deckard isn't
in the LiteLLM gateway config).
- stop_deckard added after every `stop_all_27b` call site (all 10 GPU-exclusive
modes) so switching AWAY from Deckard frees both cards first.
- usage() + dispatch case arm.
Live-validated: `gpu-mode deckard` boots the piehsoft-q6k project, container
healthy on :8199 serving deckard-40b, mount /mnt/models/huggingface, OWUI
connection recognized. bash -n clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The test feeds `BENCH_MOCK=1 bash scripts/bench.sh` output into the emitter,
but didn't pin the engine — so bench.sh's preflight autodetect adopted whatever
container happened to be running. bench.sh's BENCH_MOCK has two branches:
PP_MODE=log emits the NARRATIVE block (with decode_TPS, which this test
asserts), while PP_MODE=fallback (ENGINE_KIND=llamacpp) emits a prompt-
processing block with NO decode_TPS. When autodetect landed on a llama.cpp
container (e.g. a Deckard/ik dual up on the rig), the fallback mock fired and
build_record(result_class="boot-fit-measured") raised MeasuredRecordError —
a flaky, rig-state-dependent failure.
Pin CONTAINER=none + PREFLIGHT_NO_AUTODETECT=1 + ENGINE_KIND=vllm on the sample
capture so the mock branch is deterministic regardless of what's running.
Verified: passes 3× consecutively with the llama.cpp container up (the prior
flake condition), and the prompt-processing fallback is confirmed as the cause.
Fixes#478
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The arch-confirm blocker named in the 🧪 status note is resolved: the GGUF
header reports general.architecture=qwen35 (standard GQA, 97 layers), NOT the
Qwen3-Next/DeltaNet hybrid the original scaffold guessed. Corrected the profile
to family=qwen35-dense and dropped the fictional GDN/linear-attention fields.
Fixes found while correcting the profile:
- Restored the required `attention_k_eq_v: false` field (compat.py reads it with
bracket notation — dropping it broke load_profiles for the whole catalog).
- Fixed a duplicate-key bug in the weights block: a second `hf_repo:` (DavidAU
base, no MTP head) shadowed PiehSoft's MTP GGUF under yaml.safe_load
last-key-wins — setup.sh would have fetched the wrong, head-less repo.
- Added qwen35-dense to llama-cpp-local supported_model_families (the C10 gate).
Renamed the weights slug mtp-q6k → piehsoft-q6k to follow the provider-quant
idiom used by every other GGUF compose (ubergarm-iq4ks, unsloth-q4km,
byteshape-iq4xs) and to drop the redundant mtp in mtp-q6k/mtp.yml. The serving
filename stays mtp.yml (KV isn't encoded in GGUF-engine filenames). Renamed the
compose dir + on-disk weights dir + all path references in lockstep.
Validation (2× 3090, server-cuda-b9570): verify-full 8/8, verify-stress 8/8
(ceiling ladder filled to 120K / 91% of n_ctx, 0 MiB VRAM growth, needle recall
clean through 90K), 8-pack 105/150 with MTP off==on (spec-dec lossless),
soak-continuous PASS. Guard suite 42/0.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
New opt-in flag: after a catalog model is up + ready, switch.sh --owui upserts
an OpenAI connection in Open WebUI (host.docker.internal:<port>) so it appears
in the chat picker — no manual Admin->Connections step. scripts/lib/owui-register.sh
is conditional (no-op if OWUI not running), idempotent (skips if already present),
and drives OWUI's admin config API via a token forged from the container secret.
Validated end-to-end on Deckard-40B (:8199).
Guard suite 41/42; the 1 failure (test-measurement-record) is pre-existing
bench-output-format drift, unrelated to this change (no switch.sh/owui reference).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The MTP-injected GGUF is NOT a local build (the scaffold's comment was wrong) —
it's published at PiehSoft/Qwen3.6-40B-Deckard-MTP-Q6_K. Wire hf_repo + files so
setup.sh/launch.sh auto-fetch it, and correct the compose/profile comments.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Vet fixes on the Deckard-40B catalog PR:
- Engine: pin immutable :server-cuda-b9570 (the validated 2026-06-09 build,
digest 1c4ff61a) instead of the rolling :server-cuda tag — rolling regressed
at b9282 with a crash loop (#187), and b9570 is what we benched. The shared
llama-cpp-local engine pins b9246 (for the 27B, too old for Deckard-40B), so
pin the newer build at the compose level via LLAMACPP_IMAGE.
- test-compose-registry-disk: disk count 49 -> 45 (the correct +1 delta for
Deckard's one compose; 49 had absorbed 4 untracked nex-n2-mini WIP composes).
- Compose header status word: 'Unverified' -> 'Experimental' (canonical 🧪).
Full guard suite 42/42 green in a clean checkout (nex-n2-mini parked).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Dense 40B uncensored Qwen3.6 community merge, dual-3090 llama.cpp with
embedded MTP head (Q6_K GGUF, 31 GB). First dual llama.cpp compose in
the catalog; first 'category' field in the registry (uncensored).
Changes:
- Compose: models/qwen3.6-40b-deckard/llama-cpp/compose/dual/mtp-q6k/mtp.yml
Status: 🧪 Unverified (quality /150 + soak pending)
Config: -ngl 99 -ts 1,1 -fa on --spec-type draft-mtp --spec-draft-n-max 2
--cache-type-k q8_0 --cache-type-v q8_0 -c 131072
- Registry: llamacpp/deckard40B-dual-mtp (category=uncensored, experimental)
- DEFAULTS: (qwen3.6-40b-deckard, llamacpp, dual) → slug
- Model profile: scripts/lib/profiles/models/qwen3.6-40b-deckard.yml
- Drafter: qwen-mtp-builtin model_compat extended with qwen3.6-40b-deckard
- LiteLLM route: deckard-40b → :8199
- launch.sh: suggest_default_variant case for Deckard
- BENCHMARKS.md: new section with validated MTP n=2 numbers
- Test fixtures: registry count 43→44, disk 44→49, models 5→6
Wrinkle decisions (see PR description):
1. Dual llama.cpp: uses 'count: all' + '-ts 1,1' (tensor-split layer-split),
matching the validated serving config. No launcher changes needed — the
compose's deploy section handles GPU reservation directly.
2. Engine pin: uses rolling server-cuda tag (matching engine profile spec),
not the b9246 pin other composes use. The validated build (2026-06-09
digest 1c4ff61a) is newer than b9246 and has draft-mtp working. No engine
pin bump for other models.
NOT DONE (live validation — GPUs busy with quality eval):
- Boot/bench/soak/quality NOT run — maintainer to validate post-eval
- Status stays 🧪 until full gate passes
Reuse scripts/preflight.sh (docker + compose v2 + daemon, >=1 GPU, free-disk at
the docker root and the models mount, idle-VRAM warn), gated by SKIP_BUILD /
SKIP_DOWNLOAD so we only require the space we'll actually use. Plus image-studio
specifics: require the `hf` CLI when downloading, and warn (don't fail) if the
gemma-4-12b GGUF chat model isn't present yet — image gen still works; the warning
prints the exact `hf download` to fetch it. Hard-fails surface a Fix: hint before
the ~30-min build/download instead of dying partway through.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
From maintainer feedback on the P1 bundle:
- scripts/setup-image-studio.sh: pre-run plan + confirm prompt (--yes / CI=1 /
non-TTY auto-yes to never hang), --help/usage banner, and a proper "Get started"
block — create your admin account (first sign-up = admin; no creds pre-set),
pick the gemma-4-12b chat model, then 🖼️ to generate. States the fresh-vs-existing
volume wiring caveat.
- services/litellm/config.yaml: add a gemma-4-12b route (-> :8069), so the
image-studio chat brain is reachable through the gateway too (it's the one route
live in image-studio mode; the big-model routes are GPU-mutex with ComfyUI).
Open WebUI still points direct to :8069 by default for a clean picker.
- docs/IMAGE_STUDIO.md: architecture section + ASCII diagram (front-end -> chat /
image; the 2-GPU split; LiteLLM gateway), explicit first-run + how-to-generate-an-
image-in-chat steps, chat-routing explanation, and pin/v0.9.6/secret accuracy fixes.
Live-validated: gemma-4-12b responds through LiteLLM :4000; setup --help + bash -n clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* Add vllm/qwen-27b-dual-balanced + record 3-way 8-pack A/B (tie)
New "balanced" dual tier: cyankiwi AWQ-BF16-INT4 (int4 group-32 + BF16 mtp,
compressed-tensors) + int8-PTH KV + MTP n=3, TP=2 @262K. Live-validated
2026-06-07 (Marlin WNA16, KV pool 370K tok / 1.41x — the LARGEST of the dual
family, since int4 weights free more VRAM than fp8's 8-bit; ~67 TPS decode).
3-way 8-pack A/B (--full, same harness, same day 2026-06-07):
fast (autoround+fp8KV) 109 · balanced (awq+int8KV) 105 · max (fp8+int8KV) 110
→ a TIE (deterministic packs 64/64/65; spread within +/-5-7 noise).
The short-context 8-pack does NOT separate the quants. So:
- fast (vllm/dual) stays the default;
- balanced/max are framed by SERVING characteristics (KV-pool headroom,
int8-PTH fidelity), NOT behavioral quality — headers/docs say so explicitly.
- The dimension where int8-PTH should win (long-ctx NIAH recall) is an open
follow-up, not claimed here.
Also corrects the stale fast-tier 129/150 baseline: that was the 2026-05-09
harness; today's benchlocal-cli (verifier fixes since) scores the same fast
config at 109/150. Annotated as not-comparable across all four compose headers.
Wiring: awq-bf16-int4 weights_variant (compressed-tensors, cyankiwi repo) +
registry entry (vllm-stable, kv int8_per_token_head, port 8016, SKIP kvcalc,
experimental) + dual-max status_note updated with the A/B + DUAL_CARD rows +
A/B footnote. test-compose-registry-disk 42->43 / 43->44. Full gate green 43/0;
compat C4/C10/C14 pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* Correct balanced KV-pool claim: fast has the largest pool, not balanced
Measured v0.22.0 @262K, TP=2: fast 622K/2.37× > balanced 370K/1.41× > max
295K/1.13×. My initial framing claimed balanced had "the largest KV pool of
the dual family" — wrong: I only compared it to max and forgot fast. Fast's
17.5 GB autoround weights are far lighter than balanced's 27 GB AWQ, so fast
has the biggest pool by a wide margin.
This reframes balanced honestly: it is DOMINATED by the fast tier — slower
(~67 vs ~89 code TPS), smaller KV pool, and tied/below on the 8-pack. Its
only possible edge is int8-PTH KV fidelity > fast's fp8_e5m2 (same size — a
fidelity bet, not a memory one), which is UNPROVEN (the short-ctx 8-pack is
blind to it). Keep only if the long-ctx NIAH A/B (#470) proves it; else
deprecate. Fixed across the compose header, registry status_note, and
DUAL_CARD (rows + footnote); fast row now cites its 622K/2.37× pool.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
A symmetric 4-slug family for the Qwen vLLM path:
*-fast = AutoRound INT4 + fp8_e5m2 KV (peak TPS, the proven path)
*-max = official FP8 + int8-PTH KV (higher fidelity @ 262K)
Slugs:
vllm/qwen-27b-dual-fast alias of vllm/dual (AutoRound INT4, TP=2) — production
vllm/qwen-27b-dual-max FP8 + int8-PTH, TP=2 — 🧪 live-validated 2026-06-07
vllm/qwen-27b-multi-fast AutoRound INT4, TP=4 — 🧪 cross-rig (dual is the proxy)
vllm/qwen-27b-multi-max FP8 + int8-PTH, TP=4 — 🧪 cross-rig (dual-max is the proxy)
dual-max validated live on this 2-card rig: FP8 weights load via MarlinFP8
W8A16 on Ampere sm_86, int8_per_token_head KV keeps the full 262K (KV pool
295K tok / 1.13x concurrency), MTP n=3 active, coherent. The multi4 configs
are byte-identical to their dual sibling apart from TP + gpu-count, so they
ship Experimental until a real >=4x 3090 host validates them (this dev rig
has 2 cards). No multi4 DEFAULT added — the resolver degrades to vllm/dual
(honest; no model carries a multi default), and the slugs stay reachable by name.
Catalog wiring:
- models/qwen3.6-27b.yml: new fp8 weights_variant (official e4m3 release,
layer-split safetensors + embedded mtp.safetensors + vision tower)
- vllm-stable.yml: + fp8 weight format (Marlin W8A16 on Ampere) and
+ int8_per_token_head KV — NATIVE on stock v0.22.0 for uniform-head-dim
models (qwen3-next); features flag stays false (no overlay, unlike Gemma's
#40391-provided int8-PTH on vllm-gemma-stable)
- compose_registry.py: 4 entries; test-compose-registry-disk counts 38->42 / 40->43
Full gate green (42/42); compat C4/C5/C10/C14 pass for all 4 slugs.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
n-sweep @370W (dual TP=2): n2 72.9/86.1 . n3 74.0/87.7 . n4 71.6/87.8 wall-TPS
(all within ~3% - the QAT-int4's fast acceptance decay caps the spec benefit).
n=3 is the best balance (top narrative, code tied with n=4, ~20% less drafting:
43% draft-accept vs n=4's 37%), so SPEC_N_MAX default 4->3. The w4a16 compose had
inherited n=4 from the autoround clone, but autoround's slower decay (88/73/59/48%
per-pos) earns n=4 while the QAT-int4's faster decay (64/39/25% at n=3) favors a
lower n - optimal MTP n is quant-specific. Updates the compose default + Quality
field, registry status_note, BENCHMARKS row. Quality unchanged (n = speed only).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
vllm/gemma-31b-qat-w4a16-dual A/B vs the autoround-int4 default (gemma-int8-mtp),
same int8-PTH KV + assistant MTP n=4 (byte-identical speculative-config):
- 8-pack 109/150 vs 105 (+4, within +/-5-7 noise = quality tie; real instructfollow
edge IF 15-vs-8 offset by hermes/cli/TC/RM).
- WEAKER spec-decode: MTP accept-len ~2.4 vs autoround's ~3.9 (per-position
acceptance lower at every position) -> ~30% lower TPS (71.6/87.8 @370W <
autoround 106/139 @230W despite MORE power; AL is the clean power-indep signal).
- Boots clean on stock vllm-gemma-stable (no #44494 workaround - tower arch).
Comparable quality but slower -> autoround-int4 stays the default; stays Experimental.
Updates the compose Quality field + registry status_note + a BENCHMARKS row.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Repoint the single-card Gemma-4-26B-A4B slug in place from the bf16/16K
AWQ+MTP path to an INT8-per-token-head long-context path. Same weights
(cyankiwi AWQ-4bit, Marlin WNA16 MoE), same external MTP drafter (n=4),
same gemma4 tool-call; only the KV format changes (bf16 -> int8_per_token_head)
via the vendored vLLM PR #40391 hybrid-SWA KV page-size overlay. The dual
(vllm/gemma-26ba4b-dual) is untouched.
INT8-PTH (1 byte/token) lifts the single-card ceiling from bf16 16K to a
176K default. Live-validated on 1x RTX 3090 (2026-06-06): #40391 applies
cleanly, int8_per_token_head KV inits, Marlin WNA16 MoE backend, MTP
SpeculativeConfig active; KV pool 183,357 tok >= 176K; coherent generation
(post cudagraph-warmup) + clean gemma4 tool-call. mem_util 0.94, not 0.96:
0.96 passed the upfront KV check but OOM'd in the drafter's later cudagraph
capture (240 MiB free, needed 256) on a single 24 GB card.
- engine vllm-gemma-stable: + gemma4-swa-moe family, + awq/compressed-tensors
weight formats
- patches.yml: + gemma-a4b-vllm-pr40391-rebased (model-scoped copy of the
patch dir; reaches the compose, passes test-patch-attribution)
- compose single/awq/mtp.yml -> single/awq/int8.yml (full rewrite to the
#40391 overlay + entrypoint + int8 KV; Status: Experimental)
- registry + profile_runtime: engine/kv/max_ctx/mem_util/compose_path/entrypoint
- test-generate-from-profile: repoint the clean-derived-seed fixture off this
slug (now overlay-carrying) to vllm/gemma-26ba4b-dual + a tp= override
Status stays Experimental; rebench-full + soak are the #464 follow-up. Full
gate suite green (test-submit-bench pre-existing/environmental, fails on
baseline too).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Both 3090s normally run capped at 230W (quiet/cool), enforced at boot by
the enabled nvidia-power-cap.service oneshot. The cap suppresses benchmark
TPS — a documented artifact (e.g. an ik_llama "+18-20%" result that was
really just 230W vs 370W) — so maintainers need a quick way to uncap for a
true-TPS bench, then re-cap.
Adds `gpu-mode power-cap <on|off|status>` (alias: `powercap`):
- on: restart nvidia-power-cap.service (reusing the service as the single
source of truth for 230W). restart, NOT start: the unit is already
active from boot, and `start` on a RemainAfterExit oneshot is a no-op
that would silently fail to re-apply. Falls back to direct
`nvidia-smi -pl 230` if the service is missing.
- off: read each card's Default Power Limit from nvidia-smi and apply it.
Defaults differ per card here (GPU0 370W, GPU1 420W), so the value is
read, never hardcoded; nvidia-smi has no reset flag. Session-scoped:
a reboot or driver reload re-caps via the service (never disabled).
- status: print per-GPU enforced/default/min/max limits.
Also surfaces a one-line per-card cap state in `gpu-mode status` and echoes
the resulting enforced limit after on/off. Errors loudly (no nvidia-smi, no
sudo) in the script's existing style.
Live-tested on the rig: status -> off (370/420) -> status -> on (230/230),
verified each transition with `nvidia-smi -q -d POWER`; left at 230W.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The C12 budget verdict used a flat absolute floor (`MIN_KV_GB = 0.05 if
qwen3-next-moe else 1.0`) and FAILed any config leaving <1 GB for the growing
KV pool. That false-FAILs KV-light models that demonstrably boot: gemma-4-26b
-a4b single-card AWQ + MTP projects 0.78 GB growing pool, yet the live boot
holds 17,490 tok >= 16,384 max_ctx and serves (sliding_window=1024 makes 25/30
layers' KV trivially cheap).
vLLM's real pre-check is token-capacity based: it boots iff the capped pool
holds >= ONE max_model_len sequence, NOT iff KV >= 1 GB. Growing KV scales
linearly with max_num_seqs, so one sequence's KV is
`kv_pool_requested_gb / max_num_seqs`. Use that as the floor — which also
generalizes (and removes) the old qwen3-next-moe 0.05 special-case.
CAP the floor at the legacy 1 GB: for dense/long-KV configs one sequence needs
many GB and a hard per-seq threshold would false-FAIL measured-working configs
sitting inside the estimator's +-1.5 GB band (e.g. gemma-dual-int8 @262K:
10.71 GB avail vs 10.84 GB/seq, 1.2% short but boots). Net: relax the floor
ONLY for KV-light models, leave dense on its exact prior >=1 GB behavior.
The fix lives in the estimator (where the gap is), NOT in disabling
kv_calc_supported for a config we actively ship the drafter on. Design
bounced with Codex (per-sequence floor); capped-at-1-GB refinement added here
after the pure per-seq form regressed the gemma-4-31b dense duals.
Verdicts after: gemma-26ba4b-single FAIL -> TIGHT (correct); dense duals
unchanged; --calibration still 7/7; full gate 42/42.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Per request: turn on the external MTP drafter for BOTH topologies and
ladder each compose to its empirically-validated max context.
- single/awq/base.yml -> single/awq/mtp.yml (rename per ADDING_MODELS.md:
drafter=MTP named, bf16 KV is the gemma4-on-Ampere default so NOT named).
Adds --speculative-config (gemma-26b-it-assistant n=4) + prefix-caching +
chunked-prefill (parity with the dual). Max ctx 8K -> 16K: KV pool 17,490
tok at 0.92 util on 1x 3090 with the drafter loaded (17,408 would leave
<100 tok headroom). Registry drafter None -> gemma-26b-it-assistant.
- dual/awq/mtp.yml: max ctx 32K -> 262,144 (model max). KV pool 806,821 tok
at 262144/0.92 on 2x 3090 — sliding_window=1024 keeps 25/30 layers' KV
cheap, so the model max fits with 3x headroom.
Both LIVE-validated on the rig 2026-06-06 (stock v0.22.0): single tp=1 +
MTP boots (SpeculativeConfig method=mtp n=4), coherent, clean gemma4
tool-call; dual tp=2 + MTP @262K boots, coherent, clean tool-call.
Registry max_ctx 16384/262144 + single compose_path -> mtp.yml; profile_runtime
mirrors (single gains the speculative_config slot, dual maxlen 262144).
test-generate-from-profile mk_einput now clears the base slug's drafter (the
clean-derived fixtures need a no-drafter seed; the drafter-refuse fixture sets
one explicitly) so it's stable now that gemma-26ba4b-single carries an MTP drafter.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The gemma-4-26b-a4b slugs were the last users of the purged
`vllm-nightly-clean` engine. Migrate them to AWQ-4bit on stock
`vllm-stable` v0.22.0, retiring the Ampere-dead AutoRound path.
Why AutoRound is dead on Ampere: AutoRound int4-MIXED packs to
`uint8b128`, which has no W4A16 kernel on sm_86 (Cutlass/Machete are
Hopper-only; Triton W4A16 takes only uint4b8/uint4). The cyankiwi
AWQ-4bit (pure uint4, compressed-tensors) resolves to Marlin WNA16 MoE
on sm_86 and serves cleanly.
Slug changes (registry 38 -> 36):
- REMOVE vllm/gemma-a4b, vllm/gemma-a4b-awq (AutoRound/AWQ-bf16 duals)
- RENAME vllm/gemma-a4b-single -> vllm/gemma-26ba4b-single (AWQ tp=1, :8040)
- RENAME vllm/gemma-a4b-awq-mtp -> vllm/gemma-26ba4b-dual (AWQ tp=2 + ext MTP, :8041)
Both on vllm-stable, kv bf16 (kv_cache_dtype=auto), kvcalc_key=SKIP
(experimental; rebench-full/soak pending before Production).
PR #40886 (AWQ compressed-tensors MoE key remap) is native in v0.22.0 —
the vendored overlay + install entrypoint are dropped; patch marked
deprecated_on 2026-06-06. vllm-nightly-clean deprecated (final
purged-nightly engine, now zero registry users); gemma4-swa-moe added
to vllm-stable supported_model_families; arch_patches engine_pin
flipped to [email protected].0 loads:true.
Boot-validated on 2x 3090 (2026-06-06): single tp=1 coherent + clean
gemma4 tool-call; dual tp=2 + external MTP (num_spec_tokens=4) coherent
+ clean tool-call, draft layers wired. 3 AutoRound/AWQ-bf16 composes
moved to _archive/ (kept for a future Hopper engine).
Gate: 42/42 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
#327 audit found all 3 DFlash slugs rode the purged nightly-e47c98ef image (the
same dead SHA as the deprecated vllm-nightly-full), and vllm/dual4-dflash was
mislabeled `production` while (a) on a dead image, (b) TP=4 (unrunnable on this
2-card rig), (c) DFlash is blocked on Qwen3-Next per the repo hardware truths
(DeltaNet rollback). Same treatment as the #254 Genesis tail:
- Archived dual-dflash, dual-dflash-noviz, dual4-dflash -> compose/_archive/
(+ manifest); removed their 3 registry entries (41 -> 38).
- DEFAULTS[(qwen,vllm,multi4)] removed -- no functional multi4 vLLM slug remains,
so the resolver degrades multi4 -> dual. The 4-card vLLM tier is now empty
(recoverable from git if demand returns).
- vllm-nightly-dflash engine -> stability: deprecated; its qwen3-next-hybrid arch
pin flipped loads:true -> false (zero registry users; not offered for derive).
- Cascade: calibration (-3 rows), kv-calc alias map + override, patches.yml
(-18 dflash refs), test fixtures (registry-disk 38/40, calibration 9/9;
launch-compat/list-topology-filter/model-default-resolver/pullgate-gates/pull
adapted to the now-empty multi4 + nightly-slug categories).
Completes the nightly-engine cleanup: vllm-nightly-mtp/full/dflash all deprecated
with zero registry users; only the gated vllm-nightly-clean (#326) remains.
42/42 gate green. Refs #327.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Refresh the remaining test fixtures after archiving Genesis and purged-nightly vLLM composes.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The Genesis-free 35B-A3B single-card preview was pinned to the purged
vllm-nightly-clean (dead image). Repoint to vllm-stable (v0.22.0, which
includes PR #42521 qwen3_5_moe weight loading) + refresh the now-stale
Genesis/nightly header prose. Boot-validated on v0.22.0 @ 0.95 earlier (#323);
resolver emits v0.22.0; 42/42 green. Refs #254.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replace the purged `vllm-nightly-clean` pin (nightly-bf610c2f, now 404 on
Docker Hub — so the launcher injected a dead image for every slug on it)
with a v0.22.0 two-engine design:
- `vllm-stable`: OVERLAY-FREE v0.22.0 — the broad successor to
vllm-nightly-clean. Serves Qwen3-Next (hybrid 27b + MoE 35B-A3B), generic
`dense` transformers, and uncurated derived-emission bases. Migrates
vllm/qwen-35b-a3b-dual, vllm/dual, vllm/minimal (live-validated on stock
v0.22.0: TP=2 + MTP + tool-call clean; marlin-pad confirmed unnecessary).
Added as the first loads:true arch pin on the qwen + dense rows.
- `vllm-gemma-stable`: KEPT as the OVERLAY-CARRYING v0.22.0 engine (#40391
per-head INT8 KV + #42006 tool-parser) — the Gemma 4 path. Unchanged.
- `vllm-pip-baseline`: the renamed pip `dense` lower-bound (frees the
`vllm-stable` name for the docker engine above).
Why two engines and not one: `vendored_overlays` is LOAD-BEARING, not
documentation. `derived_emittable` (CONTRACT-5) refuses any engine whose
`vendored_overlays != []` as a derived-emission base, and `diagnose-profile`
expects the Gemma overlay declared on its engine. A single engine cannot be
both overlay-free (for derived/Qwen bases) and overlay-carrying (for Gemma
provenance) — so the two stay split. Compose owns patch APPLICATION; engine
owns provenance/compatibility. (An attempt to fold both into one engine broke
4 tests via exactly these two consumers; the split is the validated shape.)
Also adds scripts/tests/test-compose-image-drift.sh: asserts every fixed
`${VLLM_IMAGE:-<tag>}` compose default equals its engine's `install.spec`
(catches bump-the-engine-forget-the-literal drift), and a docs/ADDING_MODELS.md
"Coherence rules" section codifying the compose/engine/patch separation.
42/42 gate green; resolver emits v0.22.0 for all migrated Qwen + Gemma slugs.
The remaining experimental/preview slugs still on vllm-nightly-clean migrate
in follow-up as each is validated, then that engine is deprecated.
Refs #254.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
vllm/qwen-a3b-preview-single OOM'd at the KV-cache check on stock vLLM
v0.22.0 — ~20 GB weights on one 24 GB card left 0.07 GiB for KV at
mem_util 0.92. v0.22.0's CUDA-graph memory profiling (default since
v0.21.0) trims effective KV vs the older nightly the preview was tuned on.
Bump the compose default + registry mem_util to 0.95 (the documented
headless-Linux single-card default). Boots clean: KV 0.78 GiB / 31,129
tokens, coherent. WSL2 users set 0.94 via .env (docs/FAQ.md). Surfaced
during #316 boot-validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The autoround-int4 weights entry pointed at
Qwen/Qwen3-MoE-A3B-Instruct-AutoRound-Int4-mixed, which never existed on
HF — a fresh `setup.sh qwen3.6-35b-a3b` 404'd on download (#316). Correct
to the real repo Intel/Qwen3.6-35B-A3B-int4-mixed-AutoRound (ungated;
11 safetensors byte-match our on-disk copy).
It stayed hidden because the weights were pre-downloaded on the maintainer
rig (never re-fetched) and the weights-registry guard test asserted the
same bad string. Guard corrected too; resolve-check hardening filed as
#320 and a revision-pin schema as #319.
Note: upstream renamed Intel/Qwen3.6-35B-A3B-int4-AutoRound -> -int4-mixed-
AutoRound and re-quantized 2026-05-01; current HEAD 65f69c7 re-validated on
2x3090 stock v0.22.0 TP=2 (verify-full 8/8, decode ~172 TPS, coherent).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Self-contained vLLM-Omni deploy for Qwen3-Omni-30B-A3B-Instruct (int4
AutoRound) on 2x RTX 3090 -- the omni model (text/image/audio/video in,
text+speech out), served via vLLM-Omni, stage-parallel (thinker->GPU0,
talker+code2wav->GPU1).
TEXT path validated on-rig: full 65K context via fp8 KV, NIAH recall PASS
at 60K (depth 50% & 90%), coherent. Audio/speech out present but
unvalidated. Pin vllm/vllm-omni:v0.20.0 -- rc images are version-skew
-broken. TEXT requests must send "modalities":["text"] (bypasses the
fragile talker stage).
Deliberately NOT wired into compose_registry.py: custom-engine,
direct-`docker compose`-only deploy. test-compose-registry-disk bumped
57->58 + disk-only whitelist extended to allow this compose alongside the
parked sglang archives. Full static gate suite 41/41 green.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
preflight_compose_deps falsely reported "missing model files" for any compose
whose vLLM --model / --speculative-config path uses an env-default, e.g.
--model /root/.cache/huggingface/${MODEL_SUBDIR:-gemma-4-12b-autoround-int8}
The path grep's char-class excluded `:` and `}`, truncating the token to
`${MODEL_SUBDIR` before _preflight_compose_path_default (which already resolves
${VAR:-default}) ever ran → switch.sh aborted launching ALL 4 gemma-12b composes.
Two hardenings:
- path grep: stop only at real delimiters (quote/space/comma), not `:`/`}`, so
`${VAR:-default}` survives to the resolver. Also skip comment lines (`^\s*#`)
so a path written in prose (e.g. "...expects it at /root/.cache/.../assistant.")
isn't mistaken for a dependency.
- _preflight_compose_path_default: strip a trailing `}` left when a path is the
DEFAULT inside an outer ${VAR:-/root/.cache/huggingface/<path>} (the grep
anchors mid-expansion → captures `<path>}`).
Verified: all 4 gemma-12b composes now pass preflight_compose_deps; qwen3.6-27b
+ gemma-4-31b composes still pass (no regression). Guard suite 41/41 (the lone
transient fail was bench-mock autodetect colliding with a concurrently-running
soak service — unrelated, passes with PREFLIGHT_NO_AUTODETECT=1).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Self-descriptive <engine>/gemma-12b-<topology>-<quant>[-mtp] slugs:
vllm/gemma-12b-mtp -> vllm/gemma-12b-dual-bf16-mtp
vllm/gemma-12b-int8-mtp -> vllm/gemma-12b-single-int8-mtp
beellama/gemma-12b -> beellama/gemma-12b-single-q8kxl
llamacpp/gemma-12b -> llamacpp/gemma-12b-single-q8kxl
Pruned the two no-MTP base composes (MTP is output-lossless and fits the full
262144, so the bases bought nothing): removed vllm/gemma-12b (dual bf16 base) and
vllm/gemma-12b-int8 (single int8 base) + their compose files.
Cross-file: registry keys/entries, kv-calc COMPOSE_ALIAS_TEXT, calibration anchors
(dropped the removed dual-base anchor → kv-calc --calibration now 18/18, test
expectations updated), BENCHMARKS rows (rename 4 / collapse 2 pairs into the MTP
rows), UPSTREAM #39914 row, ADDING_MODELS slug example, disk-count 58/59 -> 56/57.
Hard rename, no back-compat aliases (slugs were day-old 🧪, no users). All stay 🧪
pending per-compose soaks + the ephemeral arch-preview image caveat.
Guard suite 41/41 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
bench.sh / verify.sh / verify-stress.sh share preflight_autodetect_endpoint,
which only recognised a hardcoded list of container-name prefixes (qwen36-27b,
gemma-4-31b, ...). Any model on a differently-named container (gemma-4-12b,
qwen-35b-a3b, beellama, any BYO compose) fell through and the tools silently
defaulted to port 8020 → ran against the wrong endpoint (#310, syangsao).
Fix: detect by the ENGINE-INTERNAL port mapping instead — vLLM 8000 /
llama.cpp 8080 / sglang 30000. Prefer a recognised club-3090 engine-family
prefix when several match; otherwise take the first and print how to override
(CONTAINER=/URL=). Validated live: picks vllm-gemma-4-12b-int8-mtp on :8038
(missed by the old allowlist) and ignores non-engine containers (benchlocal
sandboxes :9000, litellm :4000). URL/CONTAINER env still win.
Guard suite 41/41 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
vllm/gemma-12b-int8-mtp — base int8 + assistant external drafter, single 3090.
MTP fits the FULL 262144 on one card (drafter resident, KV pool ~310K tok, 1.18x
at 262K, ~20.7 GB — no ctx step-down vs base.yml).
n-sweep 2026-06-04 (code-gen probe @262144), accept-len in parens:
n=2 96.7 (2.62) · n=3 115.5 (3.33) · n=4 117.0 (3.67) · n=5 122.5 (4.06) TPS
vs ~50 no-MTP. Default SPEC_N=4 (near-peak, matches dual, robust; probe is
code-only — MTP's best case, so higher n risks draft-waste on narrative).
SPEC_N=5 for code-max.
Wiring: compose_registry vllm/gemma-12b-int8-mtp (tp=1, drafter gemma-12b-it-
assistant), kv-calc alias gemma-single-int8-mtp, disk-count 57/58 -> 58/59.
Guard suite 41/41 green. 8-pack quality (int8, lossless vs MTP) folds in next.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
