The prior 4-card validation (@Whamp #446) was on the OLD int8-PTH KV. #595 flips
to fp8/e4m3; the fp8 config is validated on the 2-card dual-max proxy (all gates
green) but not yet re-confirmed at TP=4. Request a fresh 4-card report before
upgrading ⚠️ caveats -> ✅.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Comment-only polish on the KV flip (no runtime change):
- fp8 -> e4m3 runs at scale=1.0 (checkpoint is weight-only; calculate_kv_scales
is disabled on Qwen3-Next hybrid), not "loads the checkpoint's scales"
- must be `fp8` not `fp8_e5m2` (e5m2 hard-rejected with fp8 checkpoints)
- backend FlashInfer (int8-PTH is TRITON_ATTN-only) -> flat decode at depth
- quality 109 ties int8-PTH 107; soak-continuous PASS (0 growth, 509 MB margin)
- fix comparison-table column spacing
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
@ryanmpelletier rebuilt the benchlocal sandboxes and ran the full 8-pack on
his 4x 3090 (v0.24.0, thinking-off): TOTAL 108/150 — within +-1 of the 109/150
we carry from the 2-card fast tier (vllm/dual). So multi-fast's quality is now
MEASURED TP-invariant on real 4-card hardware, not assumed.
- baselines.yml: ryan's 4x3090-pcie submission gains quality_8pk: 108/150 +
quality_env (harness provenance — the first quality ingest into a submission
row, the friction-#8 / slice-3e hook: a quality number carries its harness
fingerprint so it's reproducible).
- compose header Quality: 'open follow-up' -> the measured 108/150 confirmation.
Closes the last open item on the multi-fast promotion (bench + soak + quality
all confirmed on 4-card). Does NOT touch multi-max's caveat (that needs an
fp8 4-card report, not INT4).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The 4-card FP8 max tier is byte-identical to the now-production
vllm/qwen-27b-dual-max apart from TP=4 + gpu-count. We can't self-validate
(2-card dev rig), so it rides @Whamp's cross-rig full chain — #446, 4x 3090:
verify-full + verify-stress 7/7 + soak-continuous PASS (85/102).
Status experimental -> ⚠️ Production w/ caveats (registry 'caveats' + header
Caveats line + drift guard green). The caveat, stated honestly: that
validation was on an OLDER engine (pre-v0.24.0 pin) + a non-standard rig
(aikitoria P2P kernel, mixed x4/x16/x8/x16 lanes), single report — no clean
v0.24.0 4-card datapoint yet. A fresh one (like @ryanmpelletier's for
multi-fast) upgrades it to bare ✅.
No baseline row inducted: @Whamp's number is under-specified (engine version
unstated, non-standard rig), so inducting it as THE bar would mislead — the
status_note carries the provenance + caveats instead. No DEFAULTS change.
Removed the now-contradictory '🧪 Experimental until a ≥4x host validates it'
prose from the header.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The FP8 max-accuracy tier had bench (83.1/108.2) + quality (107/150) on the
2x3090 reference rig but no soak-continuous run — the one missing gate item.
Ran the full operational gate fresh on the v0.24.0 pin:
- verify-full 9/9
- verify-stress: all 6 rungs, fillable to 240,636 tok clean (91%)
- soak-continuous PASS: 0 err / 0 growth / 100% retention, p50 decode 85
That clears the production bar.
- registry + compose header: status experimental -> production (drift guard
green). status_note also de-staled: dropped the '~56 TPS' probe / 'slowest of
the three' framing (corrected to decode 83/108, the slow axis is PREFILL/TTFT
from MarlinFP8 W8A16 on Ampere, not decode) + the full-gate results.
- baselines.yml: row comment notes the soak PASS completing the gate.
- No DEFAULTS[(qwen,vllm,dual)] change -> vllm/dual (fast) stays the dual
default; dual-max just joins the actionable list as the max-fidelity tier.
Bonus for the 5090 crowd: dual-max is now a non-experimental config, so the
launch command drops --force (bash scripts/switch.sh vllm/qwen-27b-dual-max).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
#588 fixed the fp8-weights DeepGEMM crash but the _deepgemm_env gate (and my
guard) matched weights_variant == 'fp8' EXACTLY — missing agents-a1, whose
weights are 'fp8-dynamic' (compressed-tensors FP8). Those are genuine FP8
weights that route FP8 GEMM, so a 5090 running vllm/agents-a1-dual would still
hit the 'recipe not found' crash the whole framework is meant to prevent.
- _deepgemm_env gate: weights_variant == 'fp8' -> startswith('fp8'), catching
both 'fp8' and 'fp8-dynamic' (compressed-tensors). INT4/AWQ/W8A8/bf16 never
route FP8 GEMM so they stay correctly excluded.
- agents-a1 compose: add the - VLLM_USE_DEEP_GEMM pass-through.
- test-deepgemm-fp8-parity: broadened to startswith('fp8') so it now covers
agents-a1 (5 fp8-family composes) + any future fp8-* variant.
- test-launch-compat: new case — fp8-dynamic on 5090 injects VLLM_USE_DEEP_GEMM=0.
Now ALL fp8-family vLLM slugs are 5090-safe via the launcher (dual-max,
multi-max, dual-lmcache, diffusiongemma-dual, agents-a1). Convention: fp8
weight variants must be named fp8-* for the gate/guard to catch them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
#580 added the consumer-Blackwell DeepGEMM auto-disable pass-through to
dual-max, but the other fp8-weights vLLM composes — hand-maintained
parallel copies, no extends/include — had silently drifted without it:
multi-max, dual-lmcache, diffusiongemma-dual. On a 5090 / PRO 6000 / GB10
(sm_120/121) the launcher's _deepgemm_env injects VLLM_USE_DEEP_GEMM=0, but
with no receiving line docker-compose drops it → the fp8-GEMM 'recipe not
found' boot crash #580 fixed for dual-max (disc #571).
- Add the pass-through (+ the shared comment) to all
three. All 4 fp8-weights vLLM composes now carry it.
- NEW test-deepgemm-fp8-parity: asserts every fp8-weights vLLM compose has
the pass-through — REDs on the exact drift class that caused this (verified
it fails when the line is removed, passes when restored). Scope matches the
_deepgemm_env gate (weights_variant==fp8 + vllm engine); INT4/AWQ/bf16 never
invoke DeepGEMM so they're correctly excluded.
Removes ONE 5090 blocker (fp8 crash) — NOT a blanket 'runs on 5090': beellama
composes stay broken on sm_120 (Anbeeld#85, CUDA 12.4), llama.cpp/INT4 on
sm_120 remain unvalidated (no 5090 on the reference rig).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
@ryanmpelletier's flawless 4x 3090 report (#584) is the cross-rig
validation the compose header required ('until a real >=4x 3090 host
validates it'): v0.24.0, no power caps, all x16, verify-full 9/9,
verify-stress needles clean to 240K, soak-continuous PASS, bench n=5.
Third concordant 4-card validation (after @alanspires 6xVFIO + @Whamp),
first on v0.24.0.
- registry + compose header: status experimental -> production (both, drift
guard green). No DEFAULTS[(qwen,vllm,multi4)] entry -> no resolver cascade;
promotion just moves it onto the actionable list. Quality is TP-invariant,
carried from the vllm/dual proxy (109/150); 4-card 8-pack is the open
confirmation (report.sh runs none).
- baselines.yml: vllm/qwen-27b-multi-fast gains its 4x3090-pcie submission
(tier: submitted; decode 74.76/90.83, prefill 1288->1175, KV pool
1.77M/6.77x, NIAH 240K). Submission-only by construction - the 2-card
reference rig can never produce a local 4-card row.
- BENCHMARKS.md: ryan's row (all-x16 74.8/90.8 beats @Whamp's mixed-lane
59/74 on the same fp8-KV config -> PCIe lane width matters for TP=4).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The dual-max FP8 2x3090 decode bench (83.1/108.2, v0.24.0, 2026-06-30) lived
only in BENCHMARKS.md:122 + disc #515 (c17490565) — never inducted into
baselines.yml, so c3 / the registry-emit join showed only guybrush01's
2x5090 submission and no local bar. Surfaced while diagnosing #585.
- baselines.yml: vllm/qwen-27b-dual-max gains its primary tier: local row
(83.1/108.2 · TTFT 158 · prefill 1364->875 · 8-pack 107/150 · NIAH 240K ·
v0.24.0 = current pin -> FRESH); guybrush's 2x5090 submission preserved.
- compose header + DUAL_CARD.md: the stale '~56 TPS' probe (and the now-false
'slowest of the three' framing) -> real decode 83/108; the genuine
tradeoffs (smallest KV pool 295K/1.13x, slowest prefill/TTFT 158ms from
FP8's compute-heavy Marlin W8A16 dequant) kept. BENCHMARKS.md already
corrected; this closes the two spots that still read ~56.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Two fixes from guybrush01's fp8-weights 5090 run (disc #571), which needed both
to boot the fp8w arm on consumer Blackwell.
1) VLLM_USE_DEEP_GEMM: vLLM's DeepGEMM fp8-GEMM path is built for Hopper (sm_90)
+ datacenter Blackwell (sm_100/103). Consumer Blackwell (5090 / PRO 6000 /
GB10, sm_120/121) has no recipe and hard-fails "recipe not found" at boot.
Add _deepgemm_env: for fp8-weights slugs, inject VLLM_USE_DEEP_GEMM=0 on the
consumer SMs (sm_120/121 confirmed-broken; sm_89 Ada added proactively — it
routes fp8 via Marlin/CUTLASS so disabling is a harmless no-op that pre-empts
the same wall for 4090 owners). Hopper/datacenter untouched. dual/fp8/mtp.yml
gains a pass-through env; both launchers whitelist the export.
2) arch-ab.sh fp8w arm: add --force. vllm/qwen-27b-dual-max is status=experimental
so switch.sh gates it without --force.
test-launch-compat locks 5090/Ada-down / Hopper-keep / non-fp8-skip / user-pin.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The #246 Phase 2 first pass — spend a bigger card's KV pool on
concurrency, weights-invariant. Ships the machinery + measurement; the
32 GB envelope rows land from volunteer probe runs (baselines-style
plumbing-first).
Why concurrency-only (agreed 2026-07-05, design doc): most headline
configs are already at 262K model-max on 24 GB (GGUF single, dual vLLM),
so the extra 32 GB pool serves MORE STREAMS, not more context. The
context lever (bigger max_model_len) only helps single-card vLLM and is
Cliff-2b-capped anyway — deferred.
- Composes: `--max-num-seqs` parametrized to `${MAX_NUM_SEQS:-N}` on the
two pilot slugs (dual=2, minimal=1). Weights-invariant, default kept.
- envelopes.yml (empty): measured per-(slug, card-class) `max_num_seqs`,
born-from-measurement (a `validated` block is required — no guesses).
- launch_compat.py `_envelope_env`: injects MAX_NUM_SEQS from a row when
it EXCEEDS the compose default; 24 GB / no-row / heterogeneous /
user-env-set -> no-op. Both launchers whitelist MAX_NUM_SEQS.
- test-envelopes: schema + injection contract (inject / no-row /
heterogeneous / user-env-wins / no-gain).
- concurrency-probe.sh: the measurement — N concurrent long streams x R
rounds, separating expected pool-fill from a real leak (post-warm
growth). Live-validated N=2 on the dual-3090 (0 MB post-warm growth,
all rounds clean — the shipped default is sound).
Reuses the Phase 1 injection seam; independent of the KV verdict.
Full scripts gate 67/67.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Root cause of club-3090 #548, pinned from the stack trace: A1's
promotion accidentally committed the whole torch_compile cache (3,667
files, ~20 MB) compiled on THIS sm_86 rig, and the compose warm-start
mount fed it to every fresh pull. On sm_120 the AOT graph (baked
against Marlin-processed FP8 layers -> reads layer.workspace) loads
onto Cutlass-processed layers -> AttributeError -> restart loop.
vLLM's AOT cache key doesn't include arch/kernel selection, so the
cross-arch hit is silent (upstream issue to follow; UPSTREAM.md row
updated with the pinned mechanism).
Fix = the house pattern every other model already has: cache contents
gitignored (cache/.gitignore + README), directory kept for the mount,
local files untouched (our warm-start intact). Fresh users pay one
~60-90s compile on first boot and warm-start locally thereafter --
against THEIR OWN silicon's kernel selection.
Also retires the recurring dirty-tree noise from best_config files
updating during our own runs.
Full scripts gate 65/65.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The single-card Gemma-4 default was the catalog's only default with no
baseline row (wave-2 PRIORITY gap). Full gate on Anbeeld's OFFICIAL
v0.3.2-preview digest (the current beellama-local pin), rebench tag
gemma-dflash-regate:
- decode 44.91 narr / 79.76 code (n=5, CV 2.1%/5.6%), TTFT 115 ms
- NIAH ladder clean to 117,513 tok (91% of 128K); ceiling margin 929 MB
(< 1024 MB bar -> caveat stays, but up from ~177 MB pre-v0.3.2)
- 8-pack current harness: 108/150 off / 113/150 on (+5 in-band wash,
think-off default re-confirmed)
- soak PASS (p50 59.8, 100.6% retention, 0 growth, 0/100 silent-empty)
Row seeded via catalog-baseline.sh (first non-vLLM engine through the
producer flow); BENCHMARKS row + compose header (Quality + margin
numbers) updated to match.
Tooling fallout the gate exposed:
- rebench-full.sh hardcoded RUNS:-3/WARMUPS:-1, silently under-running
the bench protocol on every orchestrated gate. The A1 "n=3 env leak"
attribution was wrong -- it was this line (baselines comment
corrected). Fixed to protocol 5/3; n=3 had flattered gemma-dflash
code TPS by +8%.
- catalog-baseline.sh counted prefill-probe run-lines into its bench-n
gate (read n=8 for an n=5 log); now reads the summary headers.
Full scripts gate 64/64.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The A1 compose shipped (#546) without the 'Hardware metadata' block every
sibling carries, so preflight_compose_gpu_fit logged 'WARN: compose has
no hardware metadata; allowing boot' and skipped enforcement — visible in
guybrush01's #548 report. Add the block (min-vram 24 / gpu-count 2 /
TP 2 / Requires-sm 8.0+ — Marlin FP8 weight-only needs sm_80, unlike the
sibling's INT4 7.5+); preflight now parses + enforces (live rc=0).
Also propagate the #548 finding as Caveats (4): Blackwell sm_120 hits an
upstream vLLM v0.24.0 kernel-selection bug at boot ('QKVParallelLinear'
has no attribute 'workspace' — Cutlass W8A8 selected, a Marlin-path
attribute expected). Workaround shipped as a commented env line
(VLLM_TEST_FORCE_FP8_MARLIN=1 — forces the exact weight-only path this
compose's gate validated; verified present in the v0.24.0 image) +
mirrored into the registry status_note so switch.sh's NOTE carries it.
Gates: compose_meta_get parses all 4 keys; docker compose config valid;
test-compose-status-drift / registry-disk / preflight-gpu-fit green;
full scripts gate green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
InternScience's 35B agentic MoE (Qwen3-Next MoE arch; the card declares its
OWN base -> own model entry, not a qwen fine-tune slug), served from the
official FP8-dynamic compressed-tensors checkpoint on stock vLLM v0.24.0,
dual 3090 TP=2, full 262K. The agentic thinking-ON specialist.
First model onboarded END-TO-END through the Bring & Validate lane (T2
producer-zero): pull.sh route-C sibling swap (post deriver fix) ->
generate-compose -> documented swap -> full gate -> ④ vs-bar -> ⑤ scaffold.
Full gate PASS (rebench tag agents-a1-fp8-dual, 2026-07-03):
- bench 153.9/154.0 decode TPS (n=5, CV 0.1%, TTFT ~130ms), ~22.0 GB/card
- verify-stress 8/8 — staggered NIAH exact-recall to 240K (91%), VRAM Δ0
- soak PASS (0 growth, 0/100 silent-empty, 99.8% retention)
- 8-pack --full OFF 105/150 · ON 110/150 (post benchlocal #79+#81 harness):
toolcall 15/15 OFF · IF 15/15 ON · cli-40 thinking-ON 23/40 (the stack's
highest; base 17/40) · hermes 12/20 OFF, REGRESSES to 9/20 thinking-ON
(verified model behavior — disclosed as caveat 1)
vs qwen3.6-35b-a3b: general capability TIES (ON 110=110), decode −13% —
NOT a general upgrade; reach for it on tool/CLI-agent work thinking-ON.
Catalog wiring: compose dual/fp8-dynamic/fp8.yml (non-default KV named per
convention; OWN port 8072 — sibling-shared ports masquerade the slug in
estate detection); model profile (geometry verified == 35B-A3B from
config.json; mtp_num_hidden_layers=0 — safetensors header shows 0 mtp
tensors, config's 1 is an inherited default); registry entry status=caveats;
kv-calc agents-a1 spec + alias (fit-all prices, calibration 100%);
weights.py aliases (hf auto-fetch); LiteLLM route :8072; BENCHMARKS section;
count bumps (registry 57, disk 58, models 11).
Full catalog suite 60/60 (1 = known worktree-fixture submit-bench).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The shared served-model-name `qwen3.6-27b-autoround` (from #490) mislabels the
non-autoround 27b scenes (fp8 dual-max, lmcache): /v1/models advertises
"autoround" while `root` points at qwen3.6-27b-fp8. Same class on gemma-4-31b —
after the v0.24.0 consolidation the default is cyankiwi qat-AWQ-INT4 (bf16 KV),
yet the LiteLLM route still targeted `gemma-4-31b-autoround` (a latent #482 drift).
Fix WITHOUT breaking anything, via vLLM multi-served-name:
- Every 27b scene now serves `qwen3.6-27b <its-quant-name>`; every gemma-31b
scene serves `gemma-4-31b <its-quant-name>`. The neutral name is PRIMARY
(honest /v1/models id); the quant-specific name is retained as a live ALIAS.
- LiteLLM: add `qwen3.6-27b` / `gemma-4-31b` canonical public routes; keep the
`-autoround` routes as back-compat aliases (same upstream). Repairs the gemma drift.
- Migrate our own MODEL= defaults + docs (bench/verify/quality/launch/setup, c3,
tui-core, EXAMPLES, ...) to the neutral name. Weights slugs (`-autoround-int4`)
untouched; CHANGELOG + results/ history left as-is.
Retiring the `-autoround` alias entirely is a deliberate later step once nothing
still asks for it.
Live-verified on-rig (single/minimal, stock v0.24.0): /v1/models lists BOTH names
(root=...-autoround-int4); chat to `qwen3.6-27b` AND `qwen3.6-27b-autoround` both
return 200; `qwen3.6-27b-fp8` correctly 404s. Full shell gate 59/59 (1 = known
worktree-fixture); c3 pytest 41 passed; served-name arg-order + YAML validated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
3 active qwen composes (qwen/minimal, qwen/dual/fp8-mtp, qwen-35b-a3b/dual/fp8)
carried a stale `# Engine-profile: vllm-nightly-clean` docstring — the registry
moved them to vllm-stable (the nightly-clean successor, #254) but the header
comments weren't updated. Cosmetic only: the launcher injects VLLM_IMAGE from the
registry engine's install.spec, NOT from this header (confirmed: launch/preflight
don't parse Engine-profile for resolution). Also corrected the qwen-35b prose that
still said "v0.22.0" (image is v0.24.0) + the "aligning tracked in #254" note (#254
shipped). No functional change — the nightly images were purged; nothing launched them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The DiffusionGemma arch is NATIVE in v0.24.0 (PR #45163 merged 2026-06-12), so
the vllm-diffusion-gemma engine drops the mutable/branch :gemma digest for the
same pinned stable tag the rest of the stack uses. The 3 Ampere/TP fix-mounts
STAY (not in #45163): marlin.py + marlin_utils_fp8.py (sm_86 fp8 MoE Marlin
K-pad — #45295's dense marlin-pad in v0.24.0 does NOT cover marlin_moe_wna16)
and diffusion_gemma.py (TP-vocab; native #45163 copy is still TP=1).
Also fixes the compose for v0.24.0's arg-parser: the bare positional model must
be the --model flag (the entrypoint prepends --disable-custom-all-reduce).
Live-validated 2026-07-02: the actual compose boots + serves coherent at TP=2 on
stock v0.24.0 (Paris / primary colors / 42). Suite 59 pass / 1 pre-existing
(test-submit-bench worktree-fixture-absent).
- engine vllm-diffusion-gemma.yml: install.spec :gemma@digest -> v0.24.0 + provenance
- compose base.yml: image default -> v0.24.0 + --model flag fix + header
- patches.yml dgemma-gemma-image-fixes: reason/drift_guard/drop_when -> v0.24.0
- BENCHMARKS + UPSTREAM #45163 row: reflect the bump
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
README + DUAL_CARD: single bf16 vllm/gemma-31b-dual @224K default (stock
v0.24.0, overlay-free); the v0.22.0 int8-PTH/bf16-mtp/qat-w4a16 composes are
deprecated. UPSTREAM #40391 row: the 31b DEFAULT moved to bf16 (int8-PTH craters
recall on v0.24.0 without #40391); the #40391/#42006 overlays + vllm-gemma-stable
STAY for the 26b-a4b-single + the deprecated 31b composes. 262K int8-PTH returns
overlay-free when #40391 merges.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Gemma-4-31B onto stock vLLM v0.24.0, overlay-free on vllm-stable — the
first slice of the gemma consolidation (Option A1).
- New dual/qat-awq-int4/int8.yml: cyankiwi QAT-AWQ-INT4 weights (lm_head
excluded → dodges the v0.24.0 tie_weights crash that autoround hits),
int8-PTH KV @262K, util 0.965 (bf16 lm_head margin), native gemma4
tool+reasoning parsers. Overlay-free: #40391 (KV page-align) is native
in v0.24.0; #42006 N/A.
- MTP DISABLED with a documented re-enable caveat: Gemma-4 MTP x tool-calling
is broken on v0.24.0 (upstream vLLM #39043; MTP fix#42006 closed-unmerged).
A/B-proven on-rig: MTP-off tools PASS, MTP-on tools FAIL. Re-enable when a
stable vLLM ships the fix.
- Registry: vllm/gemma-31b-dual on vllm-stable; qat-awq-int4 weights map.
- vllm-stable.features.int8_per_token_head -> true: #40391 is native in
v0.24.0, so gemma int8-PTH no longer needs the overlay-carrying engine.
Live-validated dual on stock v0.24.0: 267K KV pool, coherent, streaming
multi-tool 3/3 (all args, no leak). Suite green (registry/compat/diagnose).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The caveat said "~58 GB free" as a flat number, but preflight_lmcache_ram
actually hard-fails the launch unless MemAvailable >= L1 + 28 GB (the +28
covers vLLM + OS), computed from the live L1 so it tracks the
LMCACHE_L1_GB override (l1=10 -> ~38 GB). The static Requires-min-ram-gb:
58 header is documentation only — nothing in scripts/ reads it; the guard
keys off the lmcache-l1-gb header. Also note the gate is --force-proof.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The LiteLLM gateway routes qwen3.6-27b-autoround → :8010, but the fp8 /
awq / lmcache 27b scenes served scene-specific names (qwen3.6-27b-fp8,
qwen3.6-27b-awq-bf16-int4). Bring one of those up as the :8010 primary
(e.g. via gpu-mode PORT override) and the gateway 404s on a served-name
mismatch (#482).
Standardize every 27b serving scene's --served-model-name to the
canonical qwen3.6-27b-autoround so the route matches whichever scene is
on :8010. The quant still differs by compose path/port — only the served
name is unified. Weights --model paths are untouched. Document the
invariant in services/litellm/config.yaml.
Full test suite green (58/58).
Closes#482
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
DeepReinforce agentic-coding RL fine-tune of Qwen3.6-35B-A3B (same qwen35moe arch, no MTP head, text-only). ik_llama dual 3090, Q8_0, full 262K, drafter-free ngram (opt-in). Gate PASS: ~108.8/108.6 TPS, verify-stress 8/8, soak PASS, 8-pack 105/105. Ties the base on the 8-pack (110), EDGES it on aider (15/30 vs 12-13) -> coding-leaning lane. Catalog suite 58/58 green.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
From alexpolo1's bare-metal 2x 3090 + NVMe re-run (#423, 2026-06-24, after #454):
1. Warm-L2 rehydrate is COMPUTE-bound, not disk-bound (5.30 s = 405 ms disk +
37 ms H2D + ~4.8 s Mamba/GDN-state recompute) — a faster SSD won't speed
warm-load on this hybrid; the L2 tier buys persistence/capacity, not warm
speed. Closes the lmcache.yml "L2 latency not yet measured on-rig" caveat.
2. An aborted prefill banks only the completed prefix (46/320) — LMCache
commits a session's blocks only on request completion, so the first long
prefill must run to completion or it re-prefills from scratch.
INTERNALS.md LMCache section + lmcache.yml header. Comment/prose only.
Port the standard NVLink auto-detect (detect_nvlink mount + NVLINK_MODE
+ conditional custom-all-reduce entrypoint, v5.1-safe escaped form) to
the 35B-A3B dual production default. Unlike the escape-fixes, this adds
an explicit --disable-custom-all-reduce on the PCIe branch where there
was none before (it relied on the image default + NCCL_P2P_DISABLE=1).
Almost certainly PCIe-neutral — vLLM auto-disables custom all-reduce
without P2P on consumer 3090s — but this is the only commit in the PR
that changes the validated PCIe path on a ✅ Production compose, so it
wants a rebench-full confirm on a free rig before being treated as
validated. Parses + resolves correctly + passes the guard suite
(47/47); not yet re-benched (rig busy). Hold/merge this commit on the
rebench result.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
These two dual vLLM composes lacked the standard NVLink auto-detect the
other dual composes carry (detect_nvlink mount + NVLINK_MODE + the
conditional custom-all-reduce entrypoint). Port it with the v5.1-safe
escaped form. PCIe-behaviour-neutral on this rig; enables the NVLink
path on NVLink rigs.
- gemma-4-26b-a4b (Experimental): reference-identical port (stock image,
--model flag form).
- diffusiongemma (Experimental): the :gemma image's ["vllm","serve"]
entrypoint is overridden so it can source detect_nvlink; the former
static VLLM_DISABLE_CUSTOM_ALL_REDUCE=1 is replaced by the conditional
--disable-custom-all-reduce flag.
Both parse + resolve correctly on Compose v5.1.4 and pass the guard
suite (47/47). The NVLink branch is untested here (PCIe rig); the
diffusiongemma :gemma-image entrypoint override in particular wants a
boot-check on NVLink hardware.
Also document why qwen3-omni is exempt rather than missing: it is
stage-parallel (thinker->GPU0, talker+code2wav->GPU1), so there is no TP
all-reduce to toggle and the cross-stage connector uses host shared
memory — re-enabling P2P would regress, not help.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Compose v5.1+ interpolates entrypoint strings at parse time, so the
un-escaped ${_NVLINK_ENABLED:-0} in these composes' boot scripts was
resolved to "0" (the host env lacks the var) BEFORE the container ran,
making `if [ "0" = "1" ]` always false. detect_nvlink.sh still ran, but
its result was ignored: every one of these dual/multi vLLM composes
silently took the PCIe branch regardless of hardware. On an NVLink rig
running Compose v5.1+, NVLink never engaged.
Escape the three entrypoint-bash tokens ($${_NVLINK_ENABLED:-0},
$${VLLM_ENFORCE_EAGER:+...}, "$$@") so Compose passes them through and
bash expands them at runtime. PCIe-behaviour-neutral (both the broken
and fixed forms resolve to the PCIe branch on a PCIe rig); the fix only
restores the NVLink path on NVLink rigs. Same root cause as the lmcache
fix in #429/#433, applied repo-wide.
Confirmed via `docker compose config` on Compose v5.1.4: all 9 now
resolve to a live ${_NVLINK_ENABLED} instead of a baked 0. Guard suite
47/47.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Port detect_nvlink.sh from dual-max: mount script, NVLINK_MODE env, ipc: host, and conditional --disable-custom-all-reduce. On PCIe path unset PYTORCH_CUDA_ALLOC_CONF after detect_nvlink because expandable_segments is incompatible with LMCacheMPConnector. Experimental — needs on-rig validation on NVLink rigs.
Co-authored-by: Cursor <cursoragent@cursor.com>
Docker Compose v5.1+ interpolates dollar signs in inline entrypoint scripts. Bash array expansion is invalid Compose syntax and blocks switch.sh before the container starts. Escape dollar signs as doubled dollar signs so bash receives the intended script at runtime.
Co-authored-by: Cursor <cursoragent@cursor.com>
Full rebench-full (--with-8pack-thinking=both) completed 2026-06-18 for
llamacpp/qwen27b-pi-reasoning-single. Replaces the earlier inferred "≈ base
50/59 @370W" note (+ the PENDING-ladder caveats) with the measured numbers, and
adds the BENCHMARKS.md single-card row.
- Bench @370W: 47.9/55.3 decode narr/code (n=5, CV<2%); @230W cap 28.5/32.9.
- verify-stress 8/8 (NIAH->183K), 8-pack 104/150 off / 106/150 on, soak PASS
(0 MiB growth, 0/100 silent-empty, p50 54.5, 102% retention).
- MTP head NOT weaker than base: matched-power A/B dead-even (73% vs 72% accept);
47.9/55.3 @370W ~on par with base 50.3/58.9. Power-sensitive (mainline -42%
370->230W) — the earlier "slow" 28/33 was the rig's 230W cap, not the model.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The merged llamacpp/qwen27b-pi-reasoning-single docs claimed its embedded MTP
head was "~45% below base / weaker" — that was a power-cap measurement artifact,
not a model property. A matched-power A/B (230W, same engine/KV/n=2/prompt) shows
the head performs IDENTICALLY to the base Qwen3.6-27B MTP head: 73% vs 72% draft
acceptance, 41.2 vs 41.2 t/s. The base llamacpp/default 50.3/58.9 figure I
compared against is a 370W BENCHMARKS number; mainline is -42% from 370->230W, so
the comparison was only valid at matched power (cf. the ik-llama/iq4ks-mtp
BENCHMARKS row, which documents exactly this trap). The 28.5/33.4 decode numbers
are correct AS 230W measurements; at 370W this config matches base's 50/59.
Corrected the status_note, compose header, and weights manual_note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* LMCache: in-repo L2 default + LMCACHE_L2=1 toggle, preflight disk-check, corrected sizing
- LMCACHE_L2=1 enables the L2 disk tier at a gitignored repo-root lmcache-kv/ (zero
config); LMCACHE_L2_ADAPTER still fully overrides. L2 stays OFF by default.
- preflight_lmcache_ram now also soft-warns on low L2 disk space at the L2 host dir.
- CORRECTED capacity (measured > estimated): the LMCache offload cache is
~131 KB/token MEASURED (lmcache_mp_l1_memory_usage_bytes 4.93 GB + 4.6 GB on L2
disk, 36,808-token session) — ~7x the GPU's 18.9 int8-PTH rate, which only sets
max servable context. Prior docs used 18.9 for capacity and overstated it ~7x:
--l1-size-gb 30 holds ~4 x 50K sessions (not ~33), <1 full 262K.
- Added a RAM/disk-vs-context sizing table to INTERNALS.md.
- Measured L2 rehydrate: 4.8s vs 43s cold re-prefill (~9x), cross-restart persistence
confirmed; use the fs adapter (not nixl_store — this image's NIXL is broken).
Refs #133.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* LMCache: document fp8 serde is broken on this model (no reduced-size L2 yet)
Measured 2026-06-17: LMCache's fp8 serde shape-errors on this model's KV
('[2,16,1584,260]' invalid — wrong layout for the hybrid GDN/head_dim-256/int8-PTH
source), 60 serialize fails, L2 stays empty. CacheGen-MP is 'coming soon'. So L2 is
uncompressed (~131 KB/token); the fp8-serde toggle was NOT wired (it doesn't work).
Re-test trigger noted. Refs #133.
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Measured the L2 disk tier on-rig (#133 follow-up): rehydrate 4.8s vs 43s cold
re-prefill (~9x) for a 37K session, cross-restart persistence confirmed
(0 L1 / 46 L2 retained keys post-restart). Two corrections to the shipped docs:
- L2 disk footprint ~125 KB/token measured (~33 GB per 262K session), not the
4.72 GB GPU-KV figure (L2 stores ~7x lower-density).
- Use the fs adapter, not nixl_store — this image's NIXL backend is broken.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Layers an LMCache tiered persistent prefix-KV cache (MP/HMA connector) onto the
dual-max fidelity profile (FP8 + int8-PTH KV + MTP n=3 @262K). Controlled A/B
shows ZERO decode penalty (74 narr / 94 code TPS == without LMCache, MTP intact
~83% accept); reuses cached prefix KV across long / multi-session workloads
(cold->warm TTFT ~7-8x) instead of re-prefilling.
- New vllm-lmcache engine profile: lmcache/vllm-openai DIGEST-pinned (the tag is
mutable — bundled vLLM 0.22.1-dev then 0.23.1-dev under the same tag).
- preflight_lmcache_ram guard: rejects --l1-size-gb over-allocation even under
--force (the l1=100-on-94GB host-OOM that forced a reboot, #133).
- env-gated L2 disk tier hook (LMCACHE_L2_ADAPTER, off by default).
- INTERNALS.md RAM/disk sizing section; registry entry + count bumps; full
guard suite green.
Incubating (hidden, --force to launch): runs LMCache's third-party image with a
newer vLLM than our v0.22.0 pin; L2 latency unmeasured on-rig; 38 GB on-demand.
Refs #133.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Code packs run 2026-06-16 (tool-free — sandbox executes the model's code, no
tool-calls): humaneval-plus-30 29/30 (97%), lcb-v6-30 25/30 (83%, 2 losses =
token_limit on the hardest). Substantiates the 'strong one-shot/competition
coding' characterization with our own numbers (matches the authors' ~96% LeetCode).
Folded into the llamacpp compose Quality field + registry status_note.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The performance-max VibeThinker path — prithivMLmods Q8_0 GGUF on mainline
llama.cpp, single 3090, q8_0 KV, full 131K, -b 4096 -ub 2048. A sibling to
the vLLM bf16 compose (#418), and the better one on every axis.
Why it's better than the vLLM sibling:
- ~166 TPS decode (vs ~110 vLLM bf16), ~6.2 GB at full 131K (vs ~9.8 GB),
prefill ~6,630 tok/s (-b 4096/-ub 2048 = +20%), boot ~9s.
- Q8_0 is near-lossless → quality INTACT, where vLLM's fp8 weight-quant broke
this quant-sensitive 3B (non-terminating empty output). KV quant is fine
(storage-only); fp8 *weights* are the problem.
Validated 2026-06-16 (tool-free packs, temp 0.6, thinking-on):
gsm-symbolic-30 100% · instructfollow-15 100% · reasonmath-15 80% ·
structoutput-15 80% · dataextract-15 40%.
dataextract is a genuine extraction ceiling — full temp sweep {0:27%,
0.6:40%, 1.0:27%} can't lift it (value mismatches persist). A reasoning
specialist, not an extractor. No tool-calling. verify-full 5/9 by design.
Wiring: registry entry + dense-family added to llama-cpp-local engine +
prithivmlmods-q8 weights variant + disk/registry counts (51/52). No MTP head
in the GGUF (plain qwen2 conversion) → no spec-dec. Guard suite 47/47.
Status: 🐣 incubating (hidden from switch.sh --list; --force to launch).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce a new pre-experimental status tier, `incubating`, for composes that
work but aren't ready for the actionable list — niche specialists or models
that fail the standard functional gate by design. Incubating composes are
HIDDEN from `switch.sh --list` by default (revealed by `--list --all`) and
launch-gated behind `--force` (non-functional), so a half-validated model is
catalogued and discoverable without cluttering the recommended set.
Tier wiring (reuses the existing status plumbing — no new emit column):
- compose_registry.py: `incubating` in STATUS_VALUES + `🐣` in COMPOSE_STATUS_EMOJI
(stays out of FUNCTIONAL_STATUSES → --force-gated, never auto-defaulted)
- switch.sh: both --list filters skip incubating unless --all, with a
`(+N incubating hidden — --all)` header note
- AGENTS.md: Status enum row + Caveats-required references
- docs/ADDING_MODELS.md: new rule — NEW MODELS START at 🐣 Incubating, promote
up the enum (🐣 → 🧪 → ⚠️/✅) as they earn the actionable list
First occupant — VibeThinker-3B (WeiboAI, Qwen2 dense reasoning fine-tune):
- `vllm/vibethinker-3b-single` — bf16 weights + fp8_e5m2 KV, single 3090,
full 131072 ctx, mem_util 0.40 (~9.8 GB, single-concurrency sized),
--reasoning-parser qwen3, no tool-calling
- Live-validated 2026-06-16: serves clean correct reasoning/code (~110 TPS),
qwen3 parser splits <think>. fp8 WEIGHTS rejected (break the quant-sensitive
3B: non-terminating empty output). Always-reasoning + no-tools → fails
verify-full's fixed-small-budget checks (5/9) by design → incubating, not gate-passing.
Guard suite 47/47 green (nex-n2-mini parked aside; it's an unrelated local experiment).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
