The gemma-4-26b-a4b slugs were the last users of the purged
`vllm-nightly-clean` engine. Migrate them to AWQ-4bit on stock
`vllm-stable` v0.22.0, retiring the Ampere-dead AutoRound path.
Why AutoRound is dead on Ampere: AutoRound int4-MIXED packs to
`uint8b128`, which has no W4A16 kernel on sm_86 (Cutlass/Machete are
Hopper-only; Triton W4A16 takes only uint4b8/uint4). The cyankiwi
AWQ-4bit (pure uint4, compressed-tensors) resolves to Marlin WNA16 MoE
on sm_86 and serves cleanly.
Slug changes (registry 38 -> 36):
- REMOVE vllm/gemma-a4b, vllm/gemma-a4b-awq (AutoRound/AWQ-bf16 duals)
- RENAME vllm/gemma-a4b-single -> vllm/gemma-26ba4b-single (AWQ tp=1, :8040)
- RENAME vllm/gemma-a4b-awq-mtp -> vllm/gemma-26ba4b-dual (AWQ tp=2 + ext MTP, :8041)
Both on vllm-stable, kv bf16 (kv_cache_dtype=auto), kvcalc_key=SKIP
(experimental; rebench-full/soak pending before Production).
PR #40886 (AWQ compressed-tensors MoE key remap) is native in v0.22.0 —
the vendored overlay + install entrypoint are dropped; patch marked
deprecated_on 2026-06-06. vllm-nightly-clean deprecated (final
purged-nightly engine, now zero registry users); gemma4-swa-moe added
to vllm-stable supported_model_families; arch_patches engine_pin
flipped to [email protected].0 loads:true.
Boot-validated on 2x 3090 (2026-06-06): single tp=1 coherent + clean
gemma4 tool-call; dual tp=2 + external MTP (num_spec_tokens=4) coherent
+ clean tool-call, draft layers wired. 3 AutoRound/AWQ-bf16 composes
moved to _archive/ (kept for a future Hopper engine).
Gate: 42/42 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
#327 audit found all 3 DFlash slugs rode the purged nightly-e47c98ef image (the
same dead SHA as the deprecated vllm-nightly-full), and vllm/dual4-dflash was
mislabeled `production` while (a) on a dead image, (b) TP=4 (unrunnable on this
2-card rig), (c) DFlash is blocked on Qwen3-Next per the repo hardware truths
(DeltaNet rollback). Same treatment as the #254 Genesis tail:
- Archived dual-dflash, dual-dflash-noviz, dual4-dflash -> compose/_archive/
(+ manifest); removed their 3 registry entries (41 -> 38).
- DEFAULTS[(qwen,vllm,multi4)] removed -- no functional multi4 vLLM slug remains,
so the resolver degrades multi4 -> dual. The 4-card vLLM tier is now empty
(recoverable from git if demand returns).
- vllm-nightly-dflash engine -> stability: deprecated; its qwen3-next-hybrid arch
pin flipped loads:true -> false (zero registry users; not offered for derive).
- Cascade: calibration (-3 rows), kv-calc alias map + override, patches.yml
(-18 dflash refs), test fixtures (registry-disk 38/40, calibration 9/9;
launch-compat/list-topology-filter/model-default-resolver/pullgate-gates/pull
adapted to the now-empty multi4 + nightly-slug categories).
Completes the nightly-engine cleanup: vllm-nightly-mtp/full/dflash all deprecated
with zero registry users; only the gated vllm-nightly-clean (#326) remains.
42/42 gate green. Refs #327.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Self-descriptive <engine>/gemma-12b-<topology>-<quant>[-mtp] slugs:
vllm/gemma-12b-mtp -> vllm/gemma-12b-dual-bf16-mtp
vllm/gemma-12b-int8-mtp -> vllm/gemma-12b-single-int8-mtp
beellama/gemma-12b -> beellama/gemma-12b-single-q8kxl
llamacpp/gemma-12b -> llamacpp/gemma-12b-single-q8kxl
Pruned the two no-MTP base composes (MTP is output-lossless and fits the full
262144, so the bases bought nothing): removed vllm/gemma-12b (dual bf16 base) and
vllm/gemma-12b-int8 (single int8 base) + their compose files.
Cross-file: registry keys/entries, kv-calc COMPOSE_ALIAS_TEXT, calibration anchors
(dropped the removed dual-base anchor → kv-calc --calibration now 18/18, test
expectations updated), BENCHMARKS rows (rename 4 / collapse 2 pairs into the MTP
rows), UPSTREAM #39914 row, ADDING_MODELS slug example, disk-count 58/59 -> 56/57.
Hard rename, no back-compat aliases (slugs were day-old 🧪, no users). All stay 🧪
pending per-compose soaks + the ephemeral arch-preview image caveat.
Guard suite 41/41 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Registers the new gemma4_unified 12B model (vLLM PR #44429, merged 2026-06-03) as
catalog slugs vllm/gemma-12b (base) + vllm/gemma-12b-mtp (assistant drafter n=4),
so launch.sh/switch.sh resolve them by slug. Status: experimental.
- ModelProfile (family gemma4-unified; 48L/8 full+40 sliding, 16/8 GQA, head_dim
256/512, SWA 1024, max_ctx 131072 — past that vLLM CUDA-OOBs, vllm#39914).
- Engine profile vllm-gemma4-unified pins the gemma4-unified PREVIEW image
(NOT stable v0.22.0, which lacks the arch). Caveat: ephemeral tag — pin a digest
before any Production promotion.
- Drafter gemma-12b-it-assistant (n=4, 0.85GB), bf16 weights, 2 registry entries,
no DEFAULTS row (experimental). kv-calc made gemma-4-12b-aware via the shared
Gemma dense path + measured calibration anchor (384,019 tok / 8.16 GiB @ MTP/TP2/131072).
- ADDING_MODELS.md: name the central registry (compose_registry.py SoT → both
launchers) explicitly in the intro + path-3.
KNOWN: kv-calc under-predicts the live pool (~204K vs measured 384K tok, -47%) — the
shared Gemma dense formula over-prices gemma4_unified global-layer KV; --calibration
GB-verdict still PASSES (TIGHT). Recorded in the calibration YAML header; refine the
gemma4_unified global-KV model as a follow-up. Guard suite 41/41 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
fp8-mtp.yml was only ever an A/B reference -- MTP is net-negative on this MoE
at vLLM TP=2 (n2 ~= -45% / n3 ~= -51% wall TPS; the built-in-MTP draft forward
pays inter-GPU sync the acceptance can't amortize). It served no production
purpose, so remove it and its wiring.
Removed:
- models/qwen3.6-35b-a3b/vllm/compose/dual/autoround-int4/fp8-mtp.yml
- compose_registry.py entry `vllm/qwen-a3b-preview-mtp` (port 8052)
- its profile_runtime.yml capture block + calibration anchor (-> 17 anchors)
- the kv-calc COMPOSE_ALIAS_TEXT token
Test/doc updates for the new counts:
- test-kv-generic-dense + test-submit-pull: calibration contract 18/18 -> 17/17
- test-compose-registry-disk: registry 50->49, disk 51->50 composes
- test-report-calib: container ref -> live vllm-qwen36-35b-a3b-dual
- BENCHMARKS.md + KV_MATH.md notes
The MTP-net-negative finding stays recorded in the BENCHMARKS production row
and learnings/qwen3.6-35b-a3b.md. Full tests/ suite green (sole red is the
pre-existing fixture-dependent test-submit-bench, identical on clean master).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Every gemma-4-31b vLLM compose pinned a purged Docker Hub nightly
(e47c98ef / bf610c2f), un-bootable for fresh users (#250, #167 class).
Prune the 9-variant set to 3 and repoint survivors onto the immutable
stable release tag vllm/vllm-openai:v0.21.0 (never :latest).
Survivors: vllm/gemma-mtp (bf16 dual, default), vllm/gemma-int8 (int8-PTH
dual), vllm/gemma-mtp-tp1 (fp8 single). Dropped: gemma-dflash,
gemma-dflash-int8, gemma-int8-tq3, gemma-bf16, gemma-awq; the 262K path
folds into gemma-int8 via a CTX env override. Gemma is decoupled from the
shared Qwen nightly profiles via a new vllm-gemma-stable engine profile;
Qwen nightly profiles are untouched.
Diagnose decoupling: the pruned gemma dflash patch dirs doubled as the
diagnose tool's cross-model disk-source proxies for two Qwen overlays
(vllm-pr41703-dflash, vllm-pr42102-dflash-kv-quant). Those capabilities
are image-baked in the pinned nightly, not mountable files, so mark them
image_baked and have diagnose skip the disk-source check for image-baked
overlays. Runtime-neutral (no Qwen compose mounts them).
switch.sh launcher: handle the new VLLM_IMAGE engine-pin export (the twin
loop in launch.sh had it; switch.sh did not -> "unexpected engine pin
export"), and guard the VLLM_NIGHTLY_SHA echo for the image-only stable
profile (was unbound under set -u).
Validated on 2x RTX 3090 (Ampere, v0.21.0): full scripts/tests/*.sh suite
35/0; live boots of gemma-mtp (bf16) and gemma-int8 serve coherent output,
qwen vllm/dual regression-clean. gemma-mtp-tp1 (fp8_e4m3) is correctly
SM-gated (required_sm=9.0) -- confirmed Ampere-incompatible (Triton
"fp8e4nv not supported on sm_86"), a documented 32 GB+/Hopper variant;
beellama remains the single-card gemma path on Ampere.
Refs #451, #250, #167.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The gh-less paste fallback embedded the absolute bundle dir into the
PUBLIC issue body ("full redacted bundle at `/abs/.../.pull-captures/...`"),
violating the acceptance that nothing the on-ramp tells a user to share
contains an unredacted absolute path. Render a repo-relative
`.pull-captures/<slug>/<ts>` pointer instead. Strengthen the gh-less
leak assertion to a rig-independent check (the absolute bundle dir must
not appear; only the relative pointer may) — the prior /opt|/home check
passed under a tmp sandbox dir and missed this.
CONTRACT-1.2: pull prints the honest one-line on-ramp pointer whenever a
gate bundle was emitted for the run, keyed on the V1-recorded capture dir
— explicitly NOT gated on the exit code (the bypassable no-arch-row C0
advisory path exits 0 yet emits the #1 §10-R9 bundle). Gate path stays
I/O-free: a single stdout line, no network/prompt/auto-send. It does not
classify (suppression is loop-side at submit).
CONTRACT-1.3: scripts/pull.sh --submit-last / --submit <dir> is a distinct
top-level verb parsed before the slug/--profile-like requirement.
--submit-last re-reads the V1 shared .last marker at submit (the race
defense — surfaces the CURRENT bundle, never a silent wrong-bundle).
Re-shows bundle identity + the exact already-redacted payload, requires an
explicit y before any network, then reuses the shipped F5 dedup.submit
(effective_dedup_hash, bounded loop:dedup-<hash> labels, +1-or-open,
collision-safe verify, suppression/review-queue) — not reimplemented.
gh-less fallback runs post-F2 classification, gated on should_file:
should_file=True -> a prefilled public issues/new URL with the
loop:dedup-<hash> label and the deterministic title template; review-
queued (unknown / correct-refusal) -> the local _review-queue spool path
and the no-public-issue line, with NO public issues/new URL. Never raises;
degrades to the local spool + printed paste-path. Console is never a
submission source — only the redacted artifact is emitted.
New scripts/tests/test-submit-pull.sh (mocked gh, zero network): the
.last-marker race re-read, the bundle-emitted-but-exit-0 surfacing,
F5-reuse, the gh-less should_file branch with no public URL for review-
queued, gate-path I/O-free, and leak-hygiene. Full shipped suite green in
the CI condition; kv-calc --calibration unchanged at 22/22; safetensors
decision path byte-unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
