`auto` mode previously only detected NVLink — PCIe P2P (a patched consumer-GPU driver:
tinygrad/geohot/aikitoria) required setting NVLINK_MODE=pcie_p2p by hand. Now, after the
NVLink check comes up empty, `auto` probes `nvidia-smi topo -p2p r`; if P2P is reported OK
between ALL pairs, it auto-promotes to PCIe P2P (custom all-reduce ON, NCCL_P2P_LEVEL=PHB).
So any user on a P2P-capable layout (shared root complex / switch) + a patched driver gets
it automatically.
- New `_pcie_p2p_available()` helper: parses the p2p matrix, skips header/legend rows (a data
row carries the self-"X"), requires every off-diagonal cell == OK (full P2P; partial → off).
- `pcie_p2p` stays as a FORCE override (assert the patch is loaded, bypass the probe).
- No-op for stock GeForce drivers (software-disable P2P → "CNS") and cards on separate root
complexes — both correctly stay off. Verified on this 2× 3090 rig: `auto` still resolves
"P2P DISABLED" (CNS).
- New test-detect-nvlink-autop2p.sh (6 mocked cases: enable on OK, stay off on CNS, NVLink
wins, multi-GPU all-OK vs mixed/none). Existing alloc-conf test unchanged. Gate 58/58.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Replace the separate `image-studio` / `video-studio` / `comfyui` scenes with
a single `ai-studio` scene: ComfyUI holds both GPUs always, and image / video /
audio are picked as *lanes* in OWUI rather than gpu-mode switches. Removes the
gemma-12b chat brain from the studio bundle (kept on disk, just unwired) and
points OWUI's DEFAULT_MODELS at the qwen director.
- gpu-mode.sh: `mode_ai_studio` replaces the two studio modes; drop the gemma
start/stop + the GPU0-pin split; preflight now reads the shared manifest.
- scripts/lib/studio-models.tsv: single source-of-truth model manifest (modality,
label, root, rel_path, size, installer) consumed by both gpu-mode preflight and
the c3 cockpit, so the two can't drift.
- download_{director,video,audio,ace_step,stable_audio,studio}_models.sh: idempotent
fetchers; download_studio_models.sh orchestrates "grab everything".
- setup-video-studio.sh + repointed setup-image-studio.sh → `gpu-mode ai-studio`.
- litellm: drop the gemma-4-12b :8069 route; comfyui compose/entrypoint GPU-pin
comments corrected (no scene sets the pin now).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Four changes from review:
1. comfyui shows engine "comfyui" (not the generic "studio") — both running
(_docker_ps_stack_containers) and stopped (_merge_known_services); other GPU
rig services / studio stacks stay "studio".
2. Orchestration: remove the bottom Power-cap strip — the cap already shows on the
GPU cards at the top ("(cap NNNW)"), so the strip duplicated it. populate_power_cap
keeps caching the per-GPU cap for the cards; the [c] menu still adjusts it.
3. gpu-mode: remove the standalone `comfyui` scene (mode_comfyui + dispatch + TSV
row + usage help) — comfyui runs via image-studio / video-studio now. test EXPECT
+ cockpit _STUDIO_SCENES updated.
4. Containers now lists the STOPPED studio sidecars (studio-director/-gallery/
-orchestrator/-image-shim/-tts) + comfyui: _stopped_studio_containers reads
`docker container ls -a` for scene-managed service/stack containers that exist but
aren't running and aren't top-level services/ dirs (engines excluded as noise).
[s] on them uses service_start_or_restart — compose-up for a real service dir,
`docker restart` for an existing sidecar (its dir name ≠ container name).
Full cockpit suite 702 passed; bash gates 56/0.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01PoXV1FuKmTiDPB15qQvLjB
gemma-int8 was a literal alias for gemma (same INT8 PTH container :8032), and
gemma-mtp (the bf16/32K/no-overlay fallback) isn't needed as a scene. Keep only
the 'gemma' scene (INT8 PTH KV, 262K, vision, :8032 — mode_gemma_int8).
Removed: the gemma-int8 + gemma-mtp TSV rows, their dispatch cases + help lines,
mode_gemma (the bf16 fallback mode) and start_gemma_mtp. Kept stop_gemma_mtp /
stop_gemma_int8 / stop_all_gemma — mode_off + the other gemma modes call them to
defensively tear down a stray bf16/int8 gemma container (the bf16 compose stays
serveable via its catalog slug).
test-gpu-mode-list EXPECT trimmed; repo gates 56/0. Cockpit untouched (the scene
table is data-driven — the dropped scenes just disappear from --list-modes).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Scene-catalog cleanup for the Orchestration tab (maintainer-directed):
- Regroup: 'serving' group → 'models'; 'chat' moves serving→ops (browser-chat
infra, no GPU model); 'diffusiongemma' moves studio→models (it IS a model).
Groups are now models / studio / ops. gpu-mode's plain + --json renders are
data-driven, so this is a TSV edit + the contract comment.
- Remove 'bigmodel': it was ~redundant with 'off' (both stop everything). Its one
useful extra — the freed VRAM/RAM readout — is folded into mode_off; the cache
drop was marginal (Linux auto-reclaims page cache; nothing for VRAM) and the
llama-server hint belongs in docs.
- Remove the 'diffusiongemma'/'dgemma' SCENE (TSV/dispatch/help/mode_/start_): a
niche dLLM whose scene duplicated the catalog slug vllm/diffusiongemma-dual (the
way to serve it). stop_diffusiongemma is KEPT — mode_off + the studio modes
call it to defensively clear a stray dgemma container.
- Cockpit scene filter switched from "ops-except-off" to a deny-list of the COMMAND
modes (power-cap / prune / prune-all) so 'chat' (now ops) still shows; off/chat +
models/studio scenes appear, the command-modes don't.
Updated test-gpu-mode-list EXPECT/GROUPS + the cockpit filter test. Repo gates
56/0; cockpit suite 673 (+1 skip) unchanged (cockpit code untouched by the
gpu-mode.sh scene removals).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A catalog slug's compose can mount a SEPARATE weight artifact beyond its core
weights — a DFlash draft model or an mmproj vision projector, from a different
subdir. The cockpit Download fetched only the core, so those 8 slugs (5 DFlash
+ 3 GGUF-vision) would download, read "present", offer Start, then fail to boot
for the missing companion.
The registry is the single source of truth: the cockpit gets every catalog slug
from it, so the slug's required artifacts come from it too.
- compose_registry.py: new per-slug `weights_companions` field — the extra
weight-variant keys a slug needs beyond `weights_variant`. Set on the 5 DFlash
slugs (-> anbeeld-dflash-iq4xs) + 3 vision slugs (-> gguf_mmproj_f16). Per-slug,
so a text slug sharing a vision GGUF variant (e.g. llamacpp/default on
unsloth-q4km) does NOT over-fetch the ~1GB mmproj.
- registry-emit --json: emit `weights_companions` + `drafter` + `vision` (vision
derived from the vision-coding workload). drafter/vision also enable a future
catalog badge. test-registry-json contract updated.
- setup.sh: read WEIGHT_EXTRA_KEYS (a <model>:<variant> list) and fetch those
alongside the core via the existing companion-download loop + per-file SHA
verify. setup.sh stays the weights-layer puller (model profiles); it does NOT
depend on the slug registry, so the historical pre-registry "download + test a
new model before it has a slug" path is untouched.
- cockpit: run_weights_download passes entry.weights_companions as
WEIGHT_EXTRA_KEYS (model-qualified); weights_state is companion-aware (a slug
is PARTIAL, Start gated + Download offered, until BOTH core and companions are
on disk); CatalogEntry gains weights_companions / drafter / vision (attached to
the row like `source`, no shared-core schema change).
+5 tests (row-attach, companion env injection, companion-aware state, setup.sh
WEIGHT_EXTRA_KEYS, registry-json contract). Suite: tui-core 66, serve-cockpit
665 (+1 skip), repo gates 56/0.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The serve-cockpit Download action runs `WEIGHT_KEY=<model>:<variant> bash
scripts/setup.sh <model>` to fetch an exact catalog entry. But the friendly
model-name dispatch (`case "${MODEL_NAME}"`) hard-`exit 1`s on any model outside
the five flagship families — BEFORE the WEIGHT_KEY override a few lines down.
So a download of any other catalog model (e.g. llamacpp/vibethinker-3b-single)
died instantly with "unsupported model", never fetched a byte, and the cockpit
then reported "download incomplete (partial) — check disk / HF_TOKEN".
WEIGHT_KEY is documented as the authoritative "exact catalog entry" fetch-now
flow: the recipe fully specifies <model>:<variant> and load_weight_recipe()
already validates that the key's model matches MODEL_NAME. So when WEIGHT_KEY
is set, the unknown-model `*)` branch is now a no-op (the override below sets
PRIMARY_WEIGHT_KEY); the friendly guard still fails fast for a typo'd / truly
unsupported positional when WEIGHT_KEY is absent.
Validated: `WEIGHT_KEY=vibethinker-3b:prithivmlmods-q8 SKIP_MODEL=1 setup.sh
vibethinker-3b` now resolves the recipe (prithivMLmods/VibeThinker-3B-GGUF →
vibethinker-3b-gguf/prithivmlmods-q8) and reaches the download; an unknown
model with no WEIGHT_KEY still errors. +regression guard in test-setup-picker.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
From the live-rig UX audit:
- A6 Catalog "fits" respects LIVE free-VRAM, not an empty card: a row whose vram_est
exceeds live per-GPU free-VRAM (from the estate poll) is downgraded ●→⚠/✗ with a
reason; with no live data the column reads "(vs empty card)". Never fabricates a
free number; the live-serving model's own row is exempt (its VRAM is counted as
used, so it is provably fitting).
- A7 Serving panel shows the REAL running config: probes /v1/models max_model_len
(running ctx) + docker inspect image, and badges "config differs from catalog slug
<slug>" only when the probe diverges from the slug's CONFIGURED ctx (registry
max_ctx, exact int) — not the kv-calc fit ceiling. Registry-derived fields stay
labelled "(per catalog slug)".
- A4 Targeted serving verbs: [k] stop / [b] restart resolve the serving container by
matched_slug and route through the existing confirm/reconcile gate (vs [o] stop-ALL
which kills co-resident services); [n] switch jumps to Run·Catalog.
- #4/N7 Doctor: [y] re-runs the (read-only) doctor read; the full-validation report is
surfaced from Doctor; clear issues offer the obvious next action.
- A9 ③ Gate ladder shows each step's last outcome (·/⟳/✓/✗) from the run's exit.
Threads a new configured_ctx field through the registry --json contract
(registry-emit.sh → VariantRow → CatalogEntry) so the A7 badge compares the
configured ctx; unifies the K-label convention on registry-emit's ÷1000 so the
running-ctx and catalog labels match. All registry-parity gates pass.
Verified by an implement→gate→repair→verify→critic workflow; folded the findings: the
serving model's own row no longer false-"✗ won't fit"; the divergence badge compares
configured vs probed (not the fit ceiling) so an honest 262K serve on the 295K-fit
qwen doesn't false-badge; the ladder glyph awaits run completion before resolving
⟳→✓/✗ (was stuck at ⟳).
Suite green: cockpit 444; registry-json / switch- / launch-parity / compose-disk gates pass.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Dual composes inject PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,...
for the PCIe (custom-all-reduce OFF) path. On NVLink / patched-PCIe-P2P
rigs detect_nvlink.sh turns custom all-reduce ON, where expandable_segments
backs allocations with a cuMemMap VA range that cudaIpcGetMemHandle rejects
during graph-buffer registration -> custom_all_reduce.cuh:455 'invalid
argument', worker dies at boot.
The prior fix clobbered PYTORCH_CUDA_ALLOC_CONF to max_split_size_mb:512,
discarding any other knob the user set (custom max_split_size_mb,
garbage_collection_threshold, ...). Instead strip ONLY the expandable_segments
token on the P2P path and keep the rest; fall back to max_split_size_mb:512
when nothing remains. The PCIe path keeps expandable_segments untouched.
Strengthen test-detect-nvlink-alloc-conf.sh to assert the surgical contract
(preserve neighbors, strip in any position, no dangling commas) and add the
vllm#42609 row to docs/UPSTREAM.md (its "DP>1 AND TP>1" title under-scopes
the bug; the same cuh:455 crash fires TP-only -- cf. vllm#42583 / vllm#45604).
Validated on a 2x RTX 3090 NVLink rig (TP=2, NV4): before, boot crashes at
custom_all_reduce.cuh:455 'invalid argument'; after, the dual stack boots
clean (custom AR ON, expandable_segments stripped), /v1/models ready, serves
a completion.
Co-authored-by: Cursor <[email protected]>
Dual composes inject PYTORCH_CUDA_ALLOC_CONF with expandable_segments:True for the PCIe path. detect_nvlink.sh used \, which kept that value on NVLink rigs and crashed custom all-reduce at CUDA graph profiling (vllm#42609). Assign max_split_size_mb:512 unconditionally when P2P is on.
Co-authored-by: Cursor <[email protected]>
Generate a per-instance Compose override for estate boots so CUDA_VISIBLE_DEVICES and NVIDIA_VISIBLE_DEVICES are set inside each container. This prevents parallel vLLM estate instances from both binding to physical GPUs 0/1 when Docker exposes all reserved devices.\n\nValidated with test-diagnose-estate, test-estate-json, test-profiles-compat, test-launch-compat, and a live 4x3090 estate boot.
--fit-all --card <c> --json walks COMPOSE_REGISTRY once and emits
{card, card_vram_gb, variants:{slug: verdict}} — the same per-slug verdict
as --fit (skip for non-vLLM kvcalc_key=SKIP), in a SINGLE process. Measured:
all 53 slugs in 0.22s vs 53× per-slug --fit (~11s serial). Backs the cockpit
catalog's fit column without N subprocess spawns. test-kv-calc-fit.sh gains
section (e): shape + per-slug equivalence + skip handling.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
scripts/tests/test-compose-nvlink-escape.sh asserts no shipped compose
carries an un-escaped runtime token in a `bash -c` entrypoint —
${_NVLINK_ENABLED}, a bash ${VAR[@]} array, or $@ / ${VLLM_ENFORCE_EAGER:+}
in a detect_nvlink entrypoint. Compose v5.1+ interpolates entrypoint string
values at parse time, so an un-escaped ${_NVLINK_ENABLED:-0} is baked to its
default before the container runs → the NVLink branch dies (always PCIe).
This was live on 9 composes until #436 (lmcache via #429/#433); the guard
prevents it regressing.
Scans 54 composes (13 detect_nvlink entrypoints); negative-tested to fire on
the un-escaped form. Full guard suite 55/55.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Data-layer contracts the cockpit (and any jq user) consumes — all strictly
additive (existing human output byte-identical), full guard suite green (54/54):
- registry-emit.sh --json : {variants,defaults,profiles{engines,models,hardware,drafters}}
- tools/kv-calc.py --fit <slug|model> --card <gpu> --json : structured fit verdict
- gpu-mode.sh --list-modes [--json] : scene catalog (serving/studio/ops)
- estate_cli.py report-state/diagnose --json : structured estate read
- pull.sh --profile-like --dry-run --json : structured swap_path (not a message blob)
- health.sh CONTAINER= : Doctor probes any engine container (was qwen36-27b-hardcoded)
- switch.sh --explain <slug> [--json] : joined registry/engine/model/hw/drafter + fit + bench
Built + adversarially reviewed via workflow. The review caught a real
switch<->kv-calc seam defect (switch fed hyphenated 'rtx-3090', kv-calc matched
only 'rtx3090' -> fit silently 'unavailable' on the 3090 rig); fixed: kv-calc
accepts hyphenated hardware-profile ids + hyphen-strip fallback; switch surfaces
kv-calc's structured verdict regardless of RC and renders its real keys; both
tests now exercise the seam. No shared-module edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
benchlocal-cli already supports --max-tokens (overrides the per-pack ~1024
completion budget for BOTH arms, #28), but quality-test.sh only forwarded
--thinking-max-tokens. So a VERBOSE model that self-truncates the deterministic
packs (finish_reason=length before its final ANSWER:/</solution> line) could not
be benched at a higher budget through our wrapper.
Surfaced on llamacpp/qwen27b-pi-reasoning (a reasoning fine-tune that reasons in
visible content even thinking-off): ~5-10 reasonmath/bugfind/cli misses in the
thinking-OFF 8-pack were truncations, not wrong answers (e.g. RM-05 had 4/5
checkpoints right but ran out of tokens). The thinking-ON pass (16384 budget)
did not truncate — so the gap is purely the deterministic budget, which had no
knob in our wrapper until now.
- quality-test.sh: add --max-tokens N + MAX_TOKENS env passthrough (mirrors
--thinking-max-tokens exactly: same int validation, help, ENV doc, forward + echo).
- rebench-full.sh: plumb MAX_TOKENS into BOTH 8-pack passes (off + on) + doc it.
- test-quality-thinking.sh: assert env + flag forms forward --max-tokens and that
a non-integer is rejected (mock-benchlocal harness, hermetic).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
Three related [C0]/eligibility pull-gate fixes for the curated-swap surface —
an uncurated derive (abliterated / fine-tune) of a model we already serve.
1. Wrapper-arch alias. `pull.sh --profile-like` was false-aborting at [C0] with
"no arch_patches matrix row for 'Qwen3_5ForConditionalGeneration'". That is
the OUTER multimodal wrapper class the weights report; the patch matrix is
keyed on the inner canonical `Qwen3NextForCausalLM`. arch_patches.yml is a
closed key-set, so the alias lives in the editable arch_model_xref.
- profile_runtime.yml: `config_architectures: [Qwen3_5ForConditionalGeneration]`
on the Qwen3NextForCausalLM xref entry.
- generate_compose.py: `resolve_arch_from_config()` maps a config.json
architectures[0] string -> (canonical_arch, arch_row) via that alias.
- gates.py [C0]: resolve the wrapper arch via the alias before declaring
NO_ARCH_ROW. The hybrid now reports ENGINE_SUPPORTED.
2. GGUF axis. `supported_weight_formats` was declared on every engine but never
enforced (only `kv_format` was). The deriver blocks GGUF on the derive path,
but the curated registry / curated-swap path had no such guard. gates.py [C0]
now rejects a `gguf` weight_format on an engine whose supported_weight_formats
lacks `gguf` (structural axis; matches the `gguf` token only, so a derive's
raw dtype spelling bf16/float16 is never false-rejected).
3. Won't-fit size advisory. The eligibility no-fit-model abort for a hybrid/MoE
derive now appends (a) an actionable NOTE pointing at the curated-swap path +
docs/BRING_YOUR_OWN.md, and (b) a coarse weights-only VRAM verdict: when the
raw weights exceed the detected topology's total VRAM they won't fit at ANY
KV, so say so concretely (the huihui abliterated bf16 ~54 GB vs 2×24 GB case)
instead of a generic stop. `_weights_oversize_advisory()` is pure/total —
empty when it fits / size unknown / headless.
Docs + tests:
- BRING_YOUR_OWN.md: new section C — "Swap a curated model for a fine-tune /
abliterated variant -> reuse its compose" (artifact↔engine + quant + MTP
caveats, worked example).
- test-pullgate-gates.sh: wrapper-arch ALIAS [C0] case + resolve_arch_from_config()
unit + GGUF-on-vLLM runtime-incompatible + no-false-positive control +
_weights_oversize_advisory() unit (oversize / fits / headless / malformed).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
Layers an LMCache tiered persistent prefix-KV cache (MP/HMA connector) onto the
dual-max fidelity profile (FP8 + int8-PTH KV + MTP n=3 @262K). Controlled A/B
shows ZERO decode penalty (74 narr / 94 code TPS == without LMCache, MTP intact
~83% accept); reuses cached prefix KV across long / multi-session workloads
(cold->warm TTFT ~7-8x) instead of re-prefilling.
- New vllm-lmcache engine profile: lmcache/vllm-openai DIGEST-pinned (the tag is
mutable — bundled vLLM 0.22.1-dev then 0.23.1-dev under the same tag).
- preflight_lmcache_ram guard: rejects --l1-size-gb over-allocation even under
--force (the l1=100-on-94GB host-OOM that forced a reboot, #133).
- env-gated L2 disk tier hook (LMCACHE_L2_ADAPTER, off by default).
- INTERNALS.md RAM/disk sizing section; registry entry + count bumps; full
guard suite green.
Incubating (hidden, --force to launch): runs LMCache's third-party image with a
newer vLLM than our v0.22.0 pin; L2 latency unmeasured on-rig; 38 GB on-demand.
Refs #133.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
The performance-max VibeThinker path — prithivMLmods Q8_0 GGUF on mainline
llama.cpp, single 3090, q8_0 KV, full 131K, -b 4096 -ub 2048. A sibling to
the vLLM bf16 compose (#418), and the better one on every axis.
Why it's better than the vLLM sibling:
- ~166 TPS decode (vs ~110 vLLM bf16), ~6.2 GB at full 131K (vs ~9.8 GB),
prefill ~6,630 tok/s (-b 4096/-ub 2048 = +20%), boot ~9s.
- Q8_0 is near-lossless → quality INTACT, where vLLM's fp8 weight-quant broke
this quant-sensitive 3B (non-terminating empty output). KV quant is fine
(storage-only); fp8 *weights* are the problem.
Validated 2026-06-16 (tool-free packs, temp 0.6, thinking-on):
gsm-symbolic-30 100% · instructfollow-15 100% · reasonmath-15 80% ·
structoutput-15 80% · dataextract-15 40%.
dataextract is a genuine extraction ceiling — full temp sweep {0:27%,
0.6:40%, 1.0:27%} can't lift it (value mismatches persist). A reasoning
specialist, not an extractor. No tool-calling. verify-full 5/9 by design.
Wiring: registry entry + dense-family added to llama-cpp-local engine +
prithivmlmods-q8 weights variant + disk/registry counts (51/52). No MTP head
in the GGUF (plain qwen2 conversion) → no spec-dec. Guard suite 47/47.
Status: 🐣 incubating (hidden from switch.sh --list; --force to launch).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Introduce a new pre-experimental status tier, `incubating`, for composes that
work but aren't ready for the actionable list — niche specialists or models
that fail the standard functional gate by design. Incubating composes are
HIDDEN from `switch.sh --list` by default (revealed by `--list --all`) and
launch-gated behind `--force` (non-functional), so a half-validated model is
catalogued and discoverable without cluttering the recommended set.
Tier wiring (reuses the existing status plumbing — no new emit column):
- compose_registry.py: `incubating` in STATUS_VALUES + `🐣` in COMPOSE_STATUS_EMOJI
(stays out of FUNCTIONAL_STATUSES → --force-gated, never auto-defaulted)
- switch.sh: both --list filters skip incubating unless --all, with a
`(+N incubating hidden — --all)` header note
- AGENTS.md: Status enum row + Caveats-required references
- docs/ADDING_MODELS.md: new rule — NEW MODELS START at 🐣 Incubating, promote
up the enum (🐣 → 🧪 → ⚠️/✅) as they earn the actionable list
First occupant — VibeThinker-3B (WeiboAI, Qwen2 dense reasoning fine-tune):
- `vllm/vibethinker-3b-single` — bf16 weights + fp8_e5m2 KV, single 3090,
full 131072 ctx, mem_util 0.40 (~9.8 GB, single-concurrency sized),
--reasoning-parser qwen3, no tool-calling
- Live-validated 2026-06-16: serves clean correct reasoning/code (~110 TPS),
qwen3 parser splits <think>. fp8 WEIGHTS rejected (break the quant-sensitive
3B: non-terminating empty output). Always-reasoning + no-tools → fails
verify-full's fixed-small-budget checks (5/9) by design → incubating, not gate-passing.
Guard suite 47/47 green (nex-n2-mini parked aside; it's an unrelated local experiment).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
adds an optional `revision:` key per weights variant. weights.py emits
WEIGHT_REVISION; setup.sh threads it into `hf download --revision` and
pins the post-download sha-verify etag lookup to the same revision, so a
stale pin can't false-fail against a newer HEAD. preflight's manual hint
mirrors the flag. unset = track HEAD, so behavior is unchanged for every
current entry (nothing sets revision: today).
this is the weights half of #316: upstream quant repos re-quant
silently, and we had no lever to pin the bytes a BENCHMARKS row was
measured against. engine images already pin; weights didn't. mechanism
only, no real entry is pinned in this PR (that's a per-entry,
rig-validated call that's yours to make).
refs #319, #316
adds scripts/tests/test-hf-repos-resolve.sh: an opt-in, network-gated
check that every hf_repo / hf_repos entry in the weights registry
actually resolves on HF, not just string-matches the profile. catches
the renamed / never-existed repo class from #316 that the string-only
guard in test-model-weights-registry.sh can't see.
gated behind CLUB3090_CHECK_HF_REPOS=1 and self-skips otherwise, so the
default `for t in scripts/tests/*.sh` sweep stays offline-safe and green.
follows redirects like `hf download` does, so a rename or recase resolves
and is flagged (not failed); only a real 404 fails. gated 401/403 counts
as exists; network / rate-limit codes are reported but not counted, to
avoid spurious reds.
refs #320, #316
#145 / vLLM #39056 are STREAMING-only tool-call bugs, and our gates were
structurally blind to them: benchlocal-cli runs every pack non-streaming
(benchlocal-cli#68, closed wrong-layer), and verify-full had check_tools
(non-streaming) and check_streaming (no tools) as SEPARATE checks — the bug
lives in their untested intersection (tools × streaming × thinking-on). That's
how the MTP-streaming drop (#39598, un-mitigated since Genesis retired) shipped
silently into the dual composes.
- scripts/verify-full.sh: new [6/9] check_streaming_tools — stream:true + tools
+ tool_choice=auto + enable_thinking=true, reassembles the SSE deltas, and
asserts delta.tool_calls (get_weather) + finish_reason=tool_calls + no
<tool_call> leak into delta.content. Uses tool_choice=auto (the clean path on
v0.22.0); hard-fails on the leak signature (the #145/#39056 class). NB:
tool_choice=required + MTP is a known-open drop (#39598) — the check notes it
but uses auto so it's a green gate on shipped composes. Gated by SKIP_TOOLS.
Renumbered the suite 8 -> 9 checks. Live-validated: vllm/dual all 9 pass.
- scripts/stream-toolcall-probe.py: the deep-sweep instrument behind the #400
investigation (thinking-on, multi-prompt, auto+required, SSE reassembly,
PASS/DROP/OTHER classification, exit-1 on any DROP). It's what proved
qwen3_coder == qwen3_xml and isolated the #39598 MTP-gating. Run it twice
(control vs candidate) for a streaming A/B.
- scripts/tests/test-stream-toolcall-probe.sh: offline mock-SSE gate (clean +
#145-drop modes) — no GPU needed.
Full suite green (test-compose-registry-disk fails pre-existing on an unrelated
untracked model dir).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
The rolling ghcr.io/ikawrakow/ik-llama-cpp:cu13-server tag moved (2026-05-23
-> 2026-06-10, digest 5f914f1c) and the new build REJECTS the legacy
speculative-decode flags: "legacy speculative option '--multi-token-prediction'
is disabled; use --spec-type mtp:n_max=1,p_min=0.0". Every ik-llama MTP compose
crash-restart-loops on a fresh pull (reported by furfix, Discord — single 3090
WSL). The binary's own --help confirms `-mtp`/`--draft-*`/`--spec-stage` are all
rejected in favour of the unified `--spec-type SPEC[:k=v,...]`.
Flag migration (13 composes):
- 11 MTP composes: --multi-token-prediction + --draft-max N + --draft-p-min P
-> --spec-type mtp:n_max=N,p_min=P (per-compose N preserved).
- 2 two-stage composes: the ngram-mod + mtp cascade used the rejected
--spec-stage with legacy keys -> --spec-type ngram-mod:...,ngram_size_n=... +
--spec-type mtp:...,p_min=... (spec-ngram-size-n -> ngram_size_n,
draft-p-min -> p_min, the canonical keys).
Digest-pin (decided to stop the recurrence — this is the 2nd flag churn from
the moving tag; ik-llama has no engine profile, so the per-compose image:
default is the only pin point): :cu13-server -> @sha256:5f914f1c... in all 13
ik composes + the preflight test fixture + 3 docs (IK_LLAMA / SINGLE_CARD /
INFERENCE_ENGINES). VLLM-style central injection isn't available for ik-llama;
a follow-up could add an engine profile so future pins are one-line.
Live-validated on the pinned image (single 3090):
- ik-llama/iq4ks-mtp: healthy, 0 restarts, no legacy error, MTP context ready,
served a completion, draft acceptance 0.62 (26/42).
- ik-llama/iq4ks-two-stage: healthy, 0 restarts, both stages load
(ngram_mod n=16 + MTP context ready), speculative decoding initialized.
Full test suite green (test-compose-registry-disk pre-existing on an unrelated
untracked model dir). The 3 untracked nex-n2-mini ik composes also carry the
rolling tag but are out of scope here (in-progress catalog work).
Co-authored-by: noonghunna <[email protected]>
Phase 1 (offline): the wrapper, the corpus home, and the test gate. Phase 2
(GPU) curates the actual baselines against the live RECOMMENDED_DEFAULT_MODELS
configs.
- scripts/quality-baseline.sh: thin wrapper over quality-test.sh --full that
captures (--capture -> --save-json) or diffs (default -> --previous-result)
an n>=3 aggregate per (registry-slug, thinking-mode). no-thinking is
canonical (temp-0); enable-thinking is the reasoning-on companion. --dry-run
prints the resolved command; extra args pass through to benchlocal-cli.
- scripts/quality-test.sh: forward --repeat / --previous-result and honor a
--save-json path override, so the wrapper's blessed layout works.
- results/baselines/: committed corpus home (whitelisted in .gitignore) + a
README documenting the convention, usage, and an empty index table.
- scripts/tests/test-quality-baseline.sh: offline gate (--dry-run) — asserts
command/path resolution per mode + the required-slug / valid-mode /
positive-repeat / missing-baseline guards.
- docs/QUALITY_TEST.md: regression-baseline subsection pointing at the corpus.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
bench-agentic.sh drove the ramp with tool_choice='required' and RAISED if a
turn returned no parseable tool call → the main loop caught it and broke,
aborting at the first miss. Reachable depth was bounded by tool-call
reliability at depth, not the TURNS/context budget — so on flaky parsers the
ramp stopped below the ~35K DeltaNet degrade zone the producer exists to
characterize (field-observed abort at turn 11 / ~22K).
Fix: on a no-parseable-tool-call turn, synthesize a tool call so the prompt
keeps growing by the same fixed ~chars (the fixture tool_result is injected
regardless and dominates the growth), flag tool_call_missed, and continue.
Genuine transport errors (HTTP/timeout) still propagate and stop the ramp.
Per-turn rows mark misses; a summary line reports "tool-call misses: N/M".
The optional non-tool RAMP_MODE the issue floats is largely subsumed — the
synthesize-on-miss path already lets the ramp reach configured depth on any
engine regardless of tool-call reliability.
Test: scripts/tests/test-bench-agentic-ramp.sh — mock SSE endpoint; asserts the
ramp reaches turn 3 under 100% tool-call misses (counted) AND the success path
is unchanged (no false misses). Offline, no GPU.
Caveat: guarantees reaching configured TURNS; whether the 15-turn fixture
exceeds 35K is separate (extend the fixture if not). Sibling bench-agentic fix
#498 stays separate.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
Make `vllm/diffusiongemma-dual` usable through setup/pull/switch/launch, and
pivot the engine delivery from a baked local image to a SIDELOADED overlay on a
stock, pullable nightly (so non-rig users can actually run it).
Delivery (no baked image):
- Engine `vllm-diffusion-gemma` pins a STOCK `vllm/vllm-openai:nightly-2c9c07c8…`
and sideloads the vendored overlay at boot via install_script (entrypoint).
- `models/diffusiongemma-26b-a4b/vllm/patches/dgemma-overlay/` is the vendored
payload: the FULL stock-vs-`dgemma`-branch `vllm/` delta (123 .py) — a lean
PR-#45163-only overlay version-skews (`build_attn_metadata() got an unexpected
keyword argument 'causal'`: load-bearing dgemma changes live outside the PR
diff) — plus Codex's 3 fixes (marlin K-pad ×2 + diffusion_gemma TP-vocab/dtype).
install.sh cp's it over the installed vllm pkg, fail-loud arch assert.
- base.yml rewired: stock image + overlay mount + install_script entrypoint;
drops the 3 standalone marlin-k-pad mounts (folded into the overlay) and the
baked-image dependency. The dgemma-pr45163 Dockerfile is now the overlay-
regeneration helper, not the runtime engine.
Catalog wiring:
- Model profile `diffusiongemma-26b-a4b` (gemma4-swa-moe backbone + block
diffusion; text; valid_tp [1,2]; kv_calc_supported=false → kvcalc_key SKIP).
- Engine profile `vllm-diffusion-gemma` (stock install + vendored_overlays).
- compose_registry `vllm/diffusiongemma-dual` (port 8042, status upstream-gated).
NO DEFAULTS row — upstream-gated is non-functional, reachable only by slug.
- patches.yml `dgemma-vllm45163-sideload` (install_script + drift_guard) and the
diagnose_profile_cli overlay-path hint.
- docs/UPSTREAM.md row for vllm#45163 (re-pin + drop triggers).
Output-length fix carried in base.yml: lift the model's 256-tok max_new_tokens
default to 16384 via --override-generation-config (keeps the diffusion denoising
params; fixes OWUI truncation + next-turn echo).
Tests: bump the count fixtures for +1 model / +1 engine / +1 compose / +1 registry
entry (test-profiles-compat 6→7 models, 11→12 engines; test-compose-registry-disk
44→45 registry, 45→46 disk). Full gate green except test-compose-registry-disk
local-only redness from untracked nex-n2-mini WIP in the shared tree (CI-clean
validated green: tracked-only disk=46, registry=45, disk-only set all-allowed).
Validated live on 2x RTX 3090: stock nightly + sideloaded overlay boots, serves
coherent output, 262K, via `docker compose -f base.yml up` (the switch/launch path).
Upstream-gated until #45163 merges into a pinnable engine.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The test feeds `BENCH_MOCK=1 bash scripts/bench.sh` output into the emitter,
but didn't pin the engine — so bench.sh's preflight autodetect adopted whatever
container happened to be running. bench.sh's BENCH_MOCK has two branches:
PP_MODE=log emits the NARRATIVE block (with decode_TPS, which this test
asserts), while PP_MODE=fallback (ENGINE_KIND=llamacpp) emits a prompt-
processing block with NO decode_TPS. When autodetect landed on a llama.cpp
container (e.g. a Deckard/ik dual up on the rig), the fallback mock fired and
build_record(result_class="boot-fit-measured") raised MeasuredRecordError —
a flaky, rig-state-dependent failure.
Pin CONTAINER=none + PREFLIGHT_NO_AUTODETECT=1 + ENGINE_KIND=vllm on the sample
capture so the mock branch is deterministic regardless of what's running.
Verified: passes 3× consecutively with the llama.cpp container up (the prior
flake condition), and the prompt-processing fallback is confirmed as the cause.
Fixes#478
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Vet fixes on the Deckard-40B catalog PR:
- Engine: pin immutable :server-cuda-b9570 (the validated 2026-06-09 build,
digest 1c4ff61a) instead of the rolling :server-cuda tag — rolling regressed
at b9282 with a crash loop (#187), and b9570 is what we benched. The shared
llama-cpp-local engine pins b9246 (for the 27B, too old for Deckard-40B), so
pin the newer build at the compose level via LLAMACPP_IMAGE.
- test-compose-registry-disk: disk count 49 -> 45 (the correct +1 delta for
Deckard's one compose; 49 had absorbed 4 untracked nex-n2-mini WIP composes).
- Compose header status word: 'Unverified' -> 'Experimental' (canonical 🧪).
Full guard suite 42/42 green in a clean checkout (nex-n2-mini parked).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Dense 40B uncensored Qwen3.6 community merge, dual-3090 llama.cpp with
embedded MTP head (Q6_K GGUF, 31 GB). First dual llama.cpp compose in
the catalog; first 'category' field in the registry (uncensored).
Changes:
- Compose: models/qwen3.6-40b-deckard/llama-cpp/compose/dual/mtp-q6k/mtp.yml
Status: 🧪 Unverified (quality /150 + soak pending)
Config: -ngl 99 -ts 1,1 -fa on --spec-type draft-mtp --spec-draft-n-max 2
--cache-type-k q8_0 --cache-type-v q8_0 -c 131072
- Registry: llamacpp/deckard40B-dual-mtp (category=uncensored, experimental)
- DEFAULTS: (qwen3.6-40b-deckard, llamacpp, dual) → slug
- Model profile: scripts/lib/profiles/models/qwen3.6-40b-deckard.yml
- Drafter: qwen-mtp-builtin model_compat extended with qwen3.6-40b-deckard
- LiteLLM route: deckard-40b → :8199
- launch.sh: suggest_default_variant case for Deckard
- BENCHMARKS.md: new section with validated MTP n=2 numbers
- Test fixtures: registry count 43→44, disk 44→49, models 5→6
Wrinkle decisions (see PR description):
1. Dual llama.cpp: uses 'count: all' + '-ts 1,1' (tensor-split layer-split),
matching the validated serving config. No launcher changes needed — the
compose's deploy section handles GPU reservation directly.
2. Engine pin: uses rolling server-cuda tag (matching engine profile spec),
not the b9246 pin other composes use. The validated build (2026-06-09
digest 1c4ff61a) is newer than b9246 and has draft-mtp working. No engine
pin bump for other models.
NOT DONE (live validation — GPUs busy with quality eval):
- Boot/bench/soak/quality NOT run — maintainer to validate post-eval
- Status stays 🧪 until full gate passes
* Add vllm/qwen-27b-dual-balanced + record 3-way 8-pack A/B (tie)
New "balanced" dual tier: cyankiwi AWQ-BF16-INT4 (int4 group-32 + BF16 mtp,
compressed-tensors) + int8-PTH KV + MTP n=3, TP=2 @262K. Live-validated
2026-06-07 (Marlin WNA16, KV pool 370K tok / 1.41x — the LARGEST of the dual
family, since int4 weights free more VRAM than fp8's 8-bit; ~67 TPS decode).
3-way 8-pack A/B (--full, same harness, same day 2026-06-07):
fast (autoround+fp8KV) 109 · balanced (awq+int8KV) 105 · max (fp8+int8KV) 110
→ a TIE (deterministic packs 64/64/65; spread within +/-5-7 noise).
The short-context 8-pack does NOT separate the quants. So:
- fast (vllm/dual) stays the default;
- balanced/max are framed by SERVING characteristics (KV-pool headroom,
int8-PTH fidelity), NOT behavioral quality — headers/docs say so explicitly.
- The dimension where int8-PTH should win (long-ctx NIAH recall) is an open
follow-up, not claimed here.
Also corrects the stale fast-tier 129/150 baseline: that was the 2026-05-09
harness; today's benchlocal-cli (verifier fixes since) scores the same fast
config at 109/150. Annotated as not-comparable across all four compose headers.
Wiring: awq-bf16-int4 weights_variant (compressed-tensors, cyankiwi repo) +
registry entry (vllm-stable, kv int8_per_token_head, port 8016, SKIP kvcalc,
experimental) + dual-max status_note updated with the A/B + DUAL_CARD rows +
A/B footnote. test-compose-registry-disk 42->43 / 43->44. Full gate green 43/0;
compat C4/C10/C14 pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* Correct balanced KV-pool claim: fast has the largest pool, not balanced
Measured v0.22.0 @262K, TP=2: fast 622K/2.37× > balanced 370K/1.41× > max
295K/1.13×. My initial framing claimed balanced had "the largest KV pool of
the dual family" — wrong: I only compared it to max and forgot fast. Fast's
17.5 GB autoround weights are far lighter than balanced's 27 GB AWQ, so fast
has the biggest pool by a wide margin.
This reframes balanced honestly: it is DOMINATED by the fast tier — slower
(~67 vs ~89 code TPS), smaller KV pool, and tied/below on the 8-pack. Its
only possible edge is int8-PTH KV fidelity > fast's fp8_e5m2 (same size — a
fidelity bet, not a memory one), which is UNPROVEN (the short-ctx 8-pack is
blind to it). Keep only if the long-ctx NIAH A/B (#470) proves it; else
deprecate. Fixed across the compose header, registry status_note, and
DUAL_CARD (rows + footnote); fast row now cites its 622K/2.37× pool.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
A symmetric 4-slug family for the Qwen vLLM path:
*-fast = AutoRound INT4 + fp8_e5m2 KV (peak TPS, the proven path)
*-max = official FP8 + int8-PTH KV (higher fidelity @ 262K)
Slugs:
vllm/qwen-27b-dual-fast alias of vllm/dual (AutoRound INT4, TP=2) — production
vllm/qwen-27b-dual-max FP8 + int8-PTH, TP=2 — 🧪 live-validated 2026-06-07
vllm/qwen-27b-multi-fast AutoRound INT4, TP=4 — 🧪 cross-rig (dual is the proxy)
vllm/qwen-27b-multi-max FP8 + int8-PTH, TP=4 — 🧪 cross-rig (dual-max is the proxy)
dual-max validated live on this 2-card rig: FP8 weights load via MarlinFP8
W8A16 on Ampere sm_86, int8_per_token_head KV keeps the full 262K (KV pool
295K tok / 1.13x concurrency), MTP n=3 active, coherent. The multi4 configs
are byte-identical to their dual sibling apart from TP + gpu-count, so they
ship Experimental until a real >=4x 3090 host validates them (this dev rig
has 2 cards). No multi4 DEFAULT added — the resolver degrades to vllm/dual
(honest; no model carries a multi default), and the slugs stay reachable by name.
Catalog wiring:
- models/qwen3.6-27b.yml: new fp8 weights_variant (official e4m3 release,
layer-split safetensors + embedded mtp.safetensors + vision tower)
- vllm-stable.yml: + fp8 weight format (Marlin W8A16 on Ampere) and
+ int8_per_token_head KV — NATIVE on stock v0.22.0 for uniform-head-dim
models (qwen3-next); features flag stays false (no overlay, unlike Gemma's
#40391-provided int8-PTH on vllm-gemma-stable)
- compose_registry.py: 4 entries; test-compose-registry-disk counts 38->42 / 40->43
Full gate green (42/42); compat C4/C5/C10/C14 pass for all 4 slugs.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Repoint the single-card Gemma-4-26B-A4B slug in place from the bf16/16K
AWQ+MTP path to an INT8-per-token-head long-context path. Same weights
(cyankiwi AWQ-4bit, Marlin WNA16 MoE), same external MTP drafter (n=4),
same gemma4 tool-call; only the KV format changes (bf16 -> int8_per_token_head)
via the vendored vLLM PR #40391 hybrid-SWA KV page-size overlay. The dual
(vllm/gemma-26ba4b-dual) is untouched.
INT8-PTH (1 byte/token) lifts the single-card ceiling from bf16 16K to a
176K default. Live-validated on 1x RTX 3090 (2026-06-06): #40391 applies
cleanly, int8_per_token_head KV inits, Marlin WNA16 MoE backend, MTP
SpeculativeConfig active; KV pool 183,357 tok >= 176K; coherent generation
(post cudagraph-warmup) + clean gemma4 tool-call. mem_util 0.94, not 0.96:
0.96 passed the upfront KV check but OOM'd in the drafter's later cudagraph
capture (240 MiB free, needed 256) on a single 24 GB card.
- engine vllm-gemma-stable: + gemma4-swa-moe family, + awq/compressed-tensors
weight formats
- patches.yml: + gemma-a4b-vllm-pr40391-rebased (model-scoped copy of the
patch dir; reaches the compose, passes test-patch-attribution)
- compose single/awq/mtp.yml -> single/awq/int8.yml (full rewrite to the
#40391 overlay + entrypoint + int8 KV; Status: Experimental)
- registry + profile_runtime: engine/kv/max_ctx/mem_util/compose_path/entrypoint
- test-generate-from-profile: repoint the clean-derived-seed fixture off this
slug (now overlay-carrying) to vllm/gemma-26ba4b-dual + a tp= override
Status stays Experimental; rebench-full + soak are the #464 follow-up. Full
gate suite green (test-submit-bench pre-existing/environmental, fails on
baseline too).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Per request: turn on the external MTP drafter for BOTH topologies and
ladder each compose to its empirically-validated max context.
- single/awq/base.yml -> single/awq/mtp.yml (rename per ADDING_MODELS.md:
drafter=MTP named, bf16 KV is the gemma4-on-Ampere default so NOT named).
Adds --speculative-config (gemma-26b-it-assistant n=4) + prefix-caching +
chunked-prefill (parity with the dual). Max ctx 8K -> 16K: KV pool 17,490
tok at 0.92 util on 1x 3090 with the drafter loaded (17,408 would leave
<100 tok headroom). Registry drafter None -> gemma-26b-it-assistant.
- dual/awq/mtp.yml: max ctx 32K -> 262,144 (model max). KV pool 806,821 tok
at 262144/0.92 on 2x 3090 — sliding_window=1024 keeps 25/30 layers' KV
cheap, so the model max fits with 3x headroom.
Both LIVE-validated on the rig 2026-06-06 (stock v0.22.0): single tp=1 +
MTP boots (SpeculativeConfig method=mtp n=4), coherent, clean gemma4
tool-call; dual tp=2 + MTP @262K boots, coherent, clean tool-call.
Registry max_ctx 16384/262144 + single compose_path -> mtp.yml; profile_runtime
mirrors (single gains the speculative_config slot, dual maxlen 262144).
test-generate-from-profile mk_einput now clears the base slug's drafter (the
clean-derived fixtures need a no-drafter seed; the drafter-refuse fixture sets
one explicitly) so it's stable now that gemma-26ba4b-single carries an MTP drafter.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The gemma-4-26b-a4b slugs were the last users of the purged
`vllm-nightly-clean` engine. Migrate them to AWQ-4bit on stock
`vllm-stable` v0.22.0, retiring the Ampere-dead AutoRound path.
Why AutoRound is dead on Ampere: AutoRound int4-MIXED packs to
`uint8b128`, which has no W4A16 kernel on sm_86 (Cutlass/Machete are
Hopper-only; Triton W4A16 takes only uint4b8/uint4). The cyankiwi
AWQ-4bit (pure uint4, compressed-tensors) resolves to Marlin WNA16 MoE
on sm_86 and serves cleanly.
Slug changes (registry 38 -> 36):
- REMOVE vllm/gemma-a4b, vllm/gemma-a4b-awq (AutoRound/AWQ-bf16 duals)
- RENAME vllm/gemma-a4b-single -> vllm/gemma-26ba4b-single (AWQ tp=1, :8040)
- RENAME vllm/gemma-a4b-awq-mtp -> vllm/gemma-26ba4b-dual (AWQ tp=2 + ext MTP, :8041)
Both on vllm-stable, kv bf16 (kv_cache_dtype=auto), kvcalc_key=SKIP
(experimental; rebench-full/soak pending before Production).
PR #40886 (AWQ compressed-tensors MoE key remap) is native in v0.22.0 —
the vendored overlay + install entrypoint are dropped; patch marked
deprecated_on 2026-06-06. vllm-nightly-clean deprecated (final
purged-nightly engine, now zero registry users); gemma4-swa-moe added
to vllm-stable supported_model_families; arch_patches engine_pin
flipped to [email protected].0 loads:true.
Boot-validated on 2x 3090 (2026-06-06): single tp=1 coherent + clean
gemma4 tool-call; dual tp=2 + external MTP (num_spec_tokens=4) coherent
+ clean tool-call, draft layers wired. 3 AutoRound/AWQ-bf16 composes
moved to _archive/ (kept for a future Hopper engine).
Gate: 42/42 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
#327 audit found all 3 DFlash slugs rode the purged nightly-e47c98ef image (the
same dead SHA as the deprecated vllm-nightly-full), and vllm/dual4-dflash was
mislabeled `production` while (a) on a dead image, (b) TP=4 (unrunnable on this
2-card rig), (c) DFlash is blocked on Qwen3-Next per the repo hardware truths
(DeltaNet rollback). Same treatment as the #254 Genesis tail:
- Archived dual-dflash, dual-dflash-noviz, dual4-dflash -> compose/_archive/
(+ manifest); removed their 3 registry entries (41 -> 38).
- DEFAULTS[(qwen,vllm,multi4)] removed -- no functional multi4 vLLM slug remains,
so the resolver degrades multi4 -> dual. The 4-card vLLM tier is now empty
(recoverable from git if demand returns).
- vllm-nightly-dflash engine -> stability: deprecated; its qwen3-next-hybrid arch
pin flipped loads:true -> false (zero registry users; not offered for derive).
- Cascade: calibration (-3 rows), kv-calc alias map + override, patches.yml
(-18 dflash refs), test fixtures (registry-disk 38/40, calibration 9/9;
launch-compat/list-topology-filter/model-default-resolver/pullgate-gates/pull
adapted to the now-empty multi4 + nightly-slug categories).
Completes the nightly-engine cleanup: vllm-nightly-mtp/full/dflash all deprecated
with zero registry users; only the gated vllm-nightly-clean (#326) remains.
42/42 gate green. Refs #327.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Refresh the remaining test fixtures after archiving Genesis and purged-nightly vLLM composes.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replace the purged `vllm-nightly-clean` pin (nightly-bf610c2f, now 404 on
Docker Hub — so the launcher injected a dead image for every slug on it)
with a v0.22.0 two-engine design:
- `vllm-stable`: OVERLAY-FREE v0.22.0 — the broad successor to
vllm-nightly-clean. Serves Qwen3-Next (hybrid 27b + MoE 35B-A3B), generic
`dense` transformers, and uncurated derived-emission bases. Migrates
vllm/qwen-35b-a3b-dual, vllm/dual, vllm/minimal (live-validated on stock
v0.22.0: TP=2 + MTP + tool-call clean; marlin-pad confirmed unnecessary).
Added as the first loads:true arch pin on the qwen + dense rows.
- `vllm-gemma-stable`: KEPT as the OVERLAY-CARRYING v0.22.0 engine (#40391
per-head INT8 KV + #42006 tool-parser) — the Gemma 4 path. Unchanged.
- `vllm-pip-baseline`: the renamed pip `dense` lower-bound (frees the
`vllm-stable` name for the docker engine above).
Why two engines and not one: `vendored_overlays` is LOAD-BEARING, not
documentation. `derived_emittable` (CONTRACT-5) refuses any engine whose
`vendored_overlays != []` as a derived-emission base, and `diagnose-profile`
expects the Gemma overlay declared on its engine. A single engine cannot be
both overlay-free (for derived/Qwen bases) and overlay-carrying (for Gemma
provenance) — so the two stay split. Compose owns patch APPLICATION; engine
owns provenance/compatibility. (An attempt to fold both into one engine broke
4 tests via exactly these two consumers; the split is the validated shape.)
Also adds scripts/tests/test-compose-image-drift.sh: asserts every fixed
`${VLLM_IMAGE:-<tag>}` compose default equals its engine's `install.spec`
(catches bump-the-engine-forget-the-literal drift), and a docs/ADDING_MODELS.md
"Coherence rules" section codifying the compose/engine/patch separation.
42/42 gate green; resolver emits v0.22.0 for all migrated Qwen + Gemma slugs.
The remaining experimental/preview slugs still on vllm-nightly-clean migrate
in follow-up as each is validated, then that engine is deprecated.
Refs #254.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The autoround-int4 weights entry pointed at
Qwen/Qwen3-MoE-A3B-Instruct-AutoRound-Int4-mixed, which never existed on
HF — a fresh `setup.sh qwen3.6-35b-a3b` 404'd on download (#316). Correct
to the real repo Intel/Qwen3.6-35B-A3B-int4-mixed-AutoRound (ungated;
11 safetensors byte-match our on-disk copy).
It stayed hidden because the weights were pre-downloaded on the maintainer
rig (never re-fetched) and the weights-registry guard test asserted the
same bad string. Guard corrected too; resolve-check hardening filed as
#320 and a revision-pin schema as #319.
Note: upstream renamed Intel/Qwen3.6-35B-A3B-int4-AutoRound -> -int4-mixed-
AutoRound and re-quantized 2026-05-01; current HEAD 65f69c7 re-validated on
2x3090 stock v0.22.0 TP=2 (verify-full 8/8, decode ~172 TPS, coherent).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Self-contained vLLM-Omni deploy for Qwen3-Omni-30B-A3B-Instruct (int4
AutoRound) on 2x RTX 3090 -- the omni model (text/image/audio/video in,
text+speech out), served via vLLM-Omni, stage-parallel (thinker->GPU0,
talker+code2wav->GPU1).
TEXT path validated on-rig: full 65K context via fp8 KV, NIAH recall PASS
at 60K (depth 50% & 90%), coherent. Audio/speech out present but
unvalidated. Pin vllm/vllm-omni:v0.20.0 -- rc images are version-skew
-broken. TEXT requests must send "modalities":["text"] (bypasses the
fragile talker stage).
Deliberately NOT wired into compose_registry.py: custom-engine,
direct-`docker compose`-only deploy. test-compose-registry-disk bumped
57->58 + disk-only whitelist extended to allow this compose alongside the
parked sglang archives. Full static gate suite 41/41 green.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Self-descriptive <engine>/gemma-12b-<topology>-<quant>[-mtp] slugs:
vllm/gemma-12b-mtp -> vllm/gemma-12b-dual-bf16-mtp
vllm/gemma-12b-int8-mtp -> vllm/gemma-12b-single-int8-mtp
beellama/gemma-12b -> beellama/gemma-12b-single-q8kxl
llamacpp/gemma-12b -> llamacpp/gemma-12b-single-q8kxl
Pruned the two no-MTP base composes (MTP is output-lossless and fits the full
262144, so the bases bought nothing): removed vllm/gemma-12b (dual bf16 base) and
vllm/gemma-12b-int8 (single int8 base) + their compose files.
Cross-file: registry keys/entries, kv-calc COMPOSE_ALIAS_TEXT, calibration anchors
(dropped the removed dual-base anchor → kv-calc --calibration now 18/18, test
expectations updated), BENCHMARKS rows (rename 4 / collapse 2 pairs into the MTP
rows), UPSTREAM #39914 row, ADDING_MODELS slug example, disk-count 58/59 -> 56/57.
Hard rename, no back-compat aliases (slugs were day-old 🧪, no users). All stay 🧪
pending per-compose soaks + the ephemeral arch-preview image caveat.
Guard suite 41/41 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
