InternScience's 35B agentic MoE (Qwen3-Next MoE arch; the card declares its
OWN base -> own model entry, not a qwen fine-tune slug), served from the
official FP8-dynamic compressed-tensors checkpoint on stock vLLM v0.24.0,
dual 3090 TP=2, full 262K. The agentic thinking-ON specialist.
First model onboarded END-TO-END through the Bring & Validate lane (T2
producer-zero): pull.sh route-C sibling swap (post deriver fix) ->
generate-compose -> documented swap -> full gate -> ④ vs-bar -> ⑤ scaffold.
Full gate PASS (rebench tag agents-a1-fp8-dual, 2026-07-03):
- bench 153.9/154.0 decode TPS (n=5, CV 0.1%, TTFT ~130ms), ~22.0 GB/card
- verify-stress 8/8 — staggered NIAH exact-recall to 240K (91%), VRAM Δ0
- soak PASS (0 growth, 0/100 silent-empty, 99.8% retention)
- 8-pack --full OFF 105/150 · ON 110/150 (post benchlocal #79+#81 harness):
toolcall 15/15 OFF · IF 15/15 ON · cli-40 thinking-ON 23/40 (the stack's
highest; base 17/40) · hermes 12/20 OFF, REGRESSES to 9/20 thinking-ON
(verified model behavior — disclosed as caveat 1)
vs qwen3.6-35b-a3b: general capability TIES (ON 110=110), decode −13% —
NOT a general upgrade; reach for it on tool/CLI-agent work thinking-ON.
Catalog wiring: compose dual/fp8-dynamic/fp8.yml (non-default KV named per
convention; OWN port 8072 — sibling-shared ports masquerade the slug in
estate detection); model profile (geometry verified == 35B-A3B from
config.json; mtp_num_hidden_layers=0 — safetensors header shows 0 mtp
tensors, config's 1 is an inherited default); registry entry status=caveats;
kv-calc agents-a1 spec + alias (fit-all prices, calibration 100%);
weights.py aliases (hf auto-fetch); LiteLLM route :8072; BENCHMARKS section;
count bumps (registry 57, disk 58, models 11).
Full catalog suite 60/60 (1 = known worktree-fixture submit-bench).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The shared served-model-name `qwen3.6-27b-autoround` (from #490) mislabels the
non-autoround 27b scenes (fp8 dual-max, lmcache): /v1/models advertises
"autoround" while `root` points at qwen3.6-27b-fp8. Same class on gemma-4-31b —
after the v0.24.0 consolidation the default is cyankiwi qat-AWQ-INT4 (bf16 KV),
yet the LiteLLM route still targeted `gemma-4-31b-autoround` (a latent #482 drift).
Fix WITHOUT breaking anything, via vLLM multi-served-name:
- Every 27b scene now serves `qwen3.6-27b <its-quant-name>`; every gemma-31b
scene serves `gemma-4-31b <its-quant-name>`. The neutral name is PRIMARY
(honest /v1/models id); the quant-specific name is retained as a live ALIAS.
- LiteLLM: add `qwen3.6-27b` / `gemma-4-31b` canonical public routes; keep the
`-autoround` routes as back-compat aliases (same upstream). Repairs the gemma drift.
- Migrate our own MODEL= defaults + docs (bench/verify/quality/launch/setup, c3,
tui-core, EXAMPLES, ...) to the neutral name. Weights slugs (`-autoround-int4`)
untouched; CHANGELOG + results/ history left as-is.
Retiring the `-autoround` alias entirely is a deliberate later step once nothing
still asks for it.
Live-verified on-rig (single/minimal, stock v0.24.0): /v1/models lists BOTH names
(root=...-autoround-int4); chat to `qwen3.6-27b` AND `qwen3.6-27b-autoround` both
return 200; `qwen3.6-27b-fp8` correctly 404s. Full shell gate 59/59 (1 = known
worktree-fixture); c3 pytest 41 passed; served-name arg-order + YAML validated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
3 active qwen composes (qwen/minimal, qwen/dual/fp8-mtp, qwen-35b-a3b/dual/fp8)
carried a stale `# Engine-profile: vllm-nightly-clean` docstring — the registry
moved them to vllm-stable (the nightly-clean successor, #254) but the header
comments weren't updated. Cosmetic only: the launcher injects VLLM_IMAGE from the
registry engine's install.spec, NOT from this header (confirmed: launch/preflight
don't parse Engine-profile for resolution). Also corrected the qwen-35b prose that
still said "v0.22.0" (image is v0.24.0) + the "aligning tracked in #254" note (#254
shipped). No functional change — the nightly images were purged; nothing launched them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The DiffusionGemma arch is NATIVE in v0.24.0 (PR #45163 merged 2026-06-12), so
the vllm-diffusion-gemma engine drops the mutable/branch :gemma digest for the
same pinned stable tag the rest of the stack uses. The 3 Ampere/TP fix-mounts
STAY (not in #45163): marlin.py + marlin_utils_fp8.py (sm_86 fp8 MoE Marlin
K-pad — #45295's dense marlin-pad in v0.24.0 does NOT cover marlin_moe_wna16)
and diffusion_gemma.py (TP-vocab; native #45163 copy is still TP=1).
Also fixes the compose for v0.24.0's arg-parser: the bare positional model must
be the --model flag (the entrypoint prepends --disable-custom-all-reduce).
Live-validated 2026-07-02: the actual compose boots + serves coherent at TP=2 on
stock v0.24.0 (Paris / primary colors / 42). Suite 59 pass / 1 pre-existing
(test-submit-bench worktree-fixture-absent).
- engine vllm-diffusion-gemma.yml: install.spec :gemma@digest -> v0.24.0 + provenance
- compose base.yml: image default -> v0.24.0 + --model flag fix + header
- patches.yml dgemma-gemma-image-fixes: reason/drift_guard/drop_when -> v0.24.0
- BENCHMARKS + UPSTREAM #45163 row: reflect the bump
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
README + DUAL_CARD: single bf16 vllm/gemma-31b-dual @224K default (stock
v0.24.0, overlay-free); the v0.22.0 int8-PTH/bf16-mtp/qat-w4a16 composes are
deprecated. UPSTREAM #40391 row: the 31b DEFAULT moved to bf16 (int8-PTH craters
recall on v0.24.0 without #40391); the #40391/#42006 overlays + vllm-gemma-stable
STAY for the 26b-a4b-single + the deprecated 31b composes. 262K int8-PTH returns
overlay-free when #40391 merges.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Gemma-4-31B onto stock vLLM v0.24.0, overlay-free on vllm-stable — the
first slice of the gemma consolidation (Option A1).
- New dual/qat-awq-int4/int8.yml: cyankiwi QAT-AWQ-INT4 weights (lm_head
excluded → dodges the v0.24.0 tie_weights crash that autoround hits),
int8-PTH KV @262K, util 0.965 (bf16 lm_head margin), native gemma4
tool+reasoning parsers. Overlay-free: #40391 (KV page-align) is native
in v0.24.0; #42006 N/A.
- MTP DISABLED with a documented re-enable caveat: Gemma-4 MTP x tool-calling
is broken on v0.24.0 (upstream vLLM #39043; MTP fix#42006 closed-unmerged).
A/B-proven on-rig: MTP-off tools PASS, MTP-on tools FAIL. Re-enable when a
stable vLLM ships the fix.
- Registry: vllm/gemma-31b-dual on vllm-stable; qat-awq-int4 weights map.
- vllm-stable.features.int8_per_token_head -> true: #40391 is native in
v0.24.0, so gemma int8-PTH no longer needs the overlay-carrying engine.
Live-validated dual on stock v0.24.0: 267K KV pool, coherent, streaming
multi-tool 3/3 (all args, no leak). Suite green (registry/compat/diagnose).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The caveat said "~58 GB free" as a flat number, but preflight_lmcache_ram
actually hard-fails the launch unless MemAvailable >= L1 + 28 GB (the +28
covers vLLM + OS), computed from the live L1 so it tracks the
LMCACHE_L1_GB override (l1=10 -> ~38 GB). The static Requires-min-ram-gb:
58 header is documentation only — nothing in scripts/ reads it; the guard
keys off the lmcache-l1-gb header. Also note the gate is --force-proof.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The LiteLLM gateway routes qwen3.6-27b-autoround → :8010, but the fp8 /
awq / lmcache 27b scenes served scene-specific names (qwen3.6-27b-fp8,
qwen3.6-27b-awq-bf16-int4). Bring one of those up as the :8010 primary
(e.g. via gpu-mode PORT override) and the gateway 404s on a served-name
mismatch (#482).
Standardize every 27b serving scene's --served-model-name to the
canonical qwen3.6-27b-autoround so the route matches whichever scene is
on :8010. The quant still differs by compose path/port — only the served
name is unified. Weights --model paths are untouched. Document the
invariant in services/litellm/config.yaml.
Full test suite green (58/58).
Closes#482
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
DeepReinforce agentic-coding RL fine-tune of Qwen3.6-35B-A3B (same qwen35moe arch, no MTP head, text-only). ik_llama dual 3090, Q8_0, full 262K, drafter-free ngram (opt-in). Gate PASS: ~108.8/108.6 TPS, verify-stress 8/8, soak PASS, 8-pack 105/105. Ties the base on the 8-pack (110), EDGES it on aider (15/30 vs 12-13) -> coding-leaning lane. Catalog suite 58/58 green.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
From alexpolo1's bare-metal 2x 3090 + NVMe re-run (#423, 2026-06-24, after #454):
1. Warm-L2 rehydrate is COMPUTE-bound, not disk-bound (5.30 s = 405 ms disk +
37 ms H2D + ~4.8 s Mamba/GDN-state recompute) — a faster SSD won't speed
warm-load on this hybrid; the L2 tier buys persistence/capacity, not warm
speed. Closes the lmcache.yml "L2 latency not yet measured on-rig" caveat.
2. An aborted prefill banks only the completed prefix (46/320) — LMCache
commits a session's blocks only on request completion, so the first long
prefill must run to completion or it re-prefills from scratch.
INTERNALS.md LMCache section + lmcache.yml header. Comment/prose only.
Port the standard NVLink auto-detect (detect_nvlink mount + NVLINK_MODE
+ conditional custom-all-reduce entrypoint, v5.1-safe escaped form) to
the 35B-A3B dual production default. Unlike the escape-fixes, this adds
an explicit --disable-custom-all-reduce on the PCIe branch where there
was none before (it relied on the image default + NCCL_P2P_DISABLE=1).
Almost certainly PCIe-neutral — vLLM auto-disables custom all-reduce
without P2P on consumer 3090s — but this is the only commit in the PR
that changes the validated PCIe path on a ✅ Production compose, so it
wants a rebench-full confirm on a free rig before being treated as
validated. Parses + resolves correctly + passes the guard suite
(47/47); not yet re-benched (rig busy). Hold/merge this commit on the
rebench result.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
These two dual vLLM composes lacked the standard NVLink auto-detect the
other dual composes carry (detect_nvlink mount + NVLINK_MODE + the
conditional custom-all-reduce entrypoint). Port it with the v5.1-safe
escaped form. PCIe-behaviour-neutral on this rig; enables the NVLink
path on NVLink rigs.
- gemma-4-26b-a4b (Experimental): reference-identical port (stock image,
--model flag form).
- diffusiongemma (Experimental): the :gemma image's ["vllm","serve"]
entrypoint is overridden so it can source detect_nvlink; the former
static VLLM_DISABLE_CUSTOM_ALL_REDUCE=1 is replaced by the conditional
--disable-custom-all-reduce flag.
Both parse + resolve correctly on Compose v5.1.4 and pass the guard
suite (47/47). The NVLink branch is untested here (PCIe rig); the
diffusiongemma :gemma-image entrypoint override in particular wants a
boot-check on NVLink hardware.
Also document why qwen3-omni is exempt rather than missing: it is
stage-parallel (thinker->GPU0, talker+code2wav->GPU1), so there is no TP
all-reduce to toggle and the cross-stage connector uses host shared
memory — re-enabling P2P would regress, not help.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Compose v5.1+ interpolates entrypoint strings at parse time, so the
un-escaped ${_NVLINK_ENABLED:-0} in these composes' boot scripts was
resolved to "0" (the host env lacks the var) BEFORE the container ran,
making `if [ "0" = "1" ]` always false. detect_nvlink.sh still ran, but
its result was ignored: every one of these dual/multi vLLM composes
silently took the PCIe branch regardless of hardware. On an NVLink rig
running Compose v5.1+, NVLink never engaged.
Escape the three entrypoint-bash tokens ($${_NVLINK_ENABLED:-0},
$${VLLM_ENFORCE_EAGER:+...}, "$$@") so Compose passes them through and
bash expands them at runtime. PCIe-behaviour-neutral (both the broken
and fixed forms resolve to the PCIe branch on a PCIe rig); the fix only
restores the NVLink path on NVLink rigs. Same root cause as the lmcache
fix in #429/#433, applied repo-wide.
Confirmed via `docker compose config` on Compose v5.1.4: all 9 now
resolve to a live ${_NVLINK_ENABLED} instead of a baked 0. Guard suite
47/47.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Port detect_nvlink.sh from dual-max: mount script, NVLINK_MODE env, ipc: host, and conditional --disable-custom-all-reduce. On PCIe path unset PYTORCH_CUDA_ALLOC_CONF after detect_nvlink because expandable_segments is incompatible with LMCacheMPConnector. Experimental — needs on-rig validation on NVLink rigs.
Co-authored-by: Cursor <cursoragent@cursor.com>
Docker Compose v5.1+ interpolates dollar signs in inline entrypoint scripts. Bash array expansion is invalid Compose syntax and blocks switch.sh before the container starts. Escape dollar signs as doubled dollar signs so bash receives the intended script at runtime.
Co-authored-by: Cursor <cursoragent@cursor.com>
Full rebench-full (--with-8pack-thinking=both) completed 2026-06-18 for
llamacpp/qwen27b-pi-reasoning-single. Replaces the earlier inferred "≈ base
50/59 @370W" note (+ the PENDING-ladder caveats) with the measured numbers, and
adds the BENCHMARKS.md single-card row.
- Bench @370W: 47.9/55.3 decode narr/code (n=5, CV<2%); @230W cap 28.5/32.9.
- verify-stress 8/8 (NIAH->183K), 8-pack 104/150 off / 106/150 on, soak PASS
(0 MiB growth, 0/100 silent-empty, p50 54.5, 102% retention).
- MTP head NOT weaker than base: matched-power A/B dead-even (73% vs 72% accept);
47.9/55.3 @370W ~on par with base 50.3/58.9. Power-sensitive (mainline -42%
370->230W) — the earlier "slow" 28/33 was the rig's 230W cap, not the model.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The merged llamacpp/qwen27b-pi-reasoning-single docs claimed its embedded MTP
head was "~45% below base / weaker" — that was a power-cap measurement artifact,
not a model property. A matched-power A/B (230W, same engine/KV/n=2/prompt) shows
the head performs IDENTICALLY to the base Qwen3.6-27B MTP head: 73% vs 72% draft
acceptance, 41.2 vs 41.2 t/s. The base llamacpp/default 50.3/58.9 figure I
compared against is a 370W BENCHMARKS number; mainline is -42% from 370->230W, so
the comparison was only valid at matched power (cf. the ik-llama/iq4ks-mtp
BENCHMARKS row, which documents exactly this trap). The 28.5/33.4 decode numbers
are correct AS 230W measurements; at 370W this config matches base's 50/59.
Corrected the status_note, compose header, and weights manual_note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* LMCache: in-repo L2 default + LMCACHE_L2=1 toggle, preflight disk-check, corrected sizing
- LMCACHE_L2=1 enables the L2 disk tier at a gitignored repo-root lmcache-kv/ (zero
config); LMCACHE_L2_ADAPTER still fully overrides. L2 stays OFF by default.
- preflight_lmcache_ram now also soft-warns on low L2 disk space at the L2 host dir.
- CORRECTED capacity (measured > estimated): the LMCache offload cache is
~131 KB/token MEASURED (lmcache_mp_l1_memory_usage_bytes 4.93 GB + 4.6 GB on L2
disk, 36,808-token session) — ~7x the GPU's 18.9 int8-PTH rate, which only sets
max servable context. Prior docs used 18.9 for capacity and overstated it ~7x:
--l1-size-gb 30 holds ~4 x 50K sessions (not ~33), <1 full 262K.
- Added a RAM/disk-vs-context sizing table to INTERNALS.md.
- Measured L2 rehydrate: 4.8s vs 43s cold re-prefill (~9x), cross-restart persistence
confirmed; use the fs adapter (not nixl_store — this image's NIXL is broken).
Refs #133.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* LMCache: document fp8 serde is broken on this model (no reduced-size L2 yet)
Measured 2026-06-17: LMCache's fp8 serde shape-errors on this model's KV
('[2,16,1584,260]' invalid — wrong layout for the hybrid GDN/head_dim-256/int8-PTH
source), 60 serialize fails, L2 stays empty. CacheGen-MP is 'coming soon'. So L2 is
uncompressed (~131 KB/token); the fp8-serde toggle was NOT wired (it doesn't work).
Re-test trigger noted. Refs #133.
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Measured the L2 disk tier on-rig (#133 follow-up): rehydrate 4.8s vs 43s cold
re-prefill (~9x) for a 37K session, cross-restart persistence confirmed
(0 L1 / 46 L2 retained keys post-restart). Two corrections to the shipped docs:
- L2 disk footprint ~125 KB/token measured (~33 GB per 262K session), not the
4.72 GB GPU-KV figure (L2 stores ~7x lower-density).
- Use the fs adapter, not nixl_store — this image's NIXL backend is broken.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Layers an LMCache tiered persistent prefix-KV cache (MP/HMA connector) onto the
dual-max fidelity profile (FP8 + int8-PTH KV + MTP n=3 @262K). Controlled A/B
shows ZERO decode penalty (74 narr / 94 code TPS == without LMCache, MTP intact
~83% accept); reuses cached prefix KV across long / multi-session workloads
(cold->warm TTFT ~7-8x) instead of re-prefilling.
- New vllm-lmcache engine profile: lmcache/vllm-openai DIGEST-pinned (the tag is
mutable — bundled vLLM 0.22.1-dev then 0.23.1-dev under the same tag).
- preflight_lmcache_ram guard: rejects --l1-size-gb over-allocation even under
--force (the l1=100-on-94GB host-OOM that forced a reboot, #133).
- env-gated L2 disk tier hook (LMCACHE_L2_ADAPTER, off by default).
- INTERNALS.md RAM/disk sizing section; registry entry + count bumps; full
guard suite green.
Incubating (hidden, --force to launch): runs LMCache's third-party image with a
newer vLLM than our v0.22.0 pin; L2 latency unmeasured on-rig; 38 GB on-demand.
Refs #133.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Code packs run 2026-06-16 (tool-free — sandbox executes the model's code, no
tool-calls): humaneval-plus-30 29/30 (97%), lcb-v6-30 25/30 (83%, 2 losses =
token_limit on the hardest). Substantiates the 'strong one-shot/competition
coding' characterization with our own numbers (matches the authors' ~96% LeetCode).
Folded into the llamacpp compose Quality field + registry status_note.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The performance-max VibeThinker path — prithivMLmods Q8_0 GGUF on mainline
llama.cpp, single 3090, q8_0 KV, full 131K, -b 4096 -ub 2048. A sibling to
the vLLM bf16 compose (#418), and the better one on every axis.
Why it's better than the vLLM sibling:
- ~166 TPS decode (vs ~110 vLLM bf16), ~6.2 GB at full 131K (vs ~9.8 GB),
prefill ~6,630 tok/s (-b 4096/-ub 2048 = +20%), boot ~9s.
- Q8_0 is near-lossless → quality INTACT, where vLLM's fp8 weight-quant broke
this quant-sensitive 3B (non-terminating empty output). KV quant is fine
(storage-only); fp8 *weights* are the problem.
Validated 2026-06-16 (tool-free packs, temp 0.6, thinking-on):
gsm-symbolic-30 100% · instructfollow-15 100% · reasonmath-15 80% ·
structoutput-15 80% · dataextract-15 40%.
dataextract is a genuine extraction ceiling — full temp sweep {0:27%,
0.6:40%, 1.0:27%} can't lift it (value mismatches persist). A reasoning
specialist, not an extractor. No tool-calling. verify-full 5/9 by design.
Wiring: registry entry + dense-family added to llama-cpp-local engine +
prithivmlmods-q8 weights variant + disk/registry counts (51/52). No MTP head
in the GGUF (plain qwen2 conversion) → no spec-dec. Guard suite 47/47.
Status: 🐣 incubating (hidden from switch.sh --list; --force to launch).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce a new pre-experimental status tier, `incubating`, for composes that
work but aren't ready for the actionable list — niche specialists or models
that fail the standard functional gate by design. Incubating composes are
HIDDEN from `switch.sh --list` by default (revealed by `--list --all`) and
launch-gated behind `--force` (non-functional), so a half-validated model is
catalogued and discoverable without cluttering the recommended set.
Tier wiring (reuses the existing status plumbing — no new emit column):
- compose_registry.py: `incubating` in STATUS_VALUES + `🐣` in COMPOSE_STATUS_EMOJI
(stays out of FUNCTIONAL_STATUSES → --force-gated, never auto-defaulted)
- switch.sh: both --list filters skip incubating unless --all, with a
`(+N incubating hidden — --all)` header note
- AGENTS.md: Status enum row + Caveats-required references
- docs/ADDING_MODELS.md: new rule — NEW MODELS START at 🐣 Incubating, promote
up the enum (🐣 → 🧪 → ⚠️/✅) as they earn the actionable list
First occupant — VibeThinker-3B (WeiboAI, Qwen2 dense reasoning fine-tune):
- `vllm/vibethinker-3b-single` — bf16 weights + fp8_e5m2 KV, single 3090,
full 131072 ctx, mem_util 0.40 (~9.8 GB, single-concurrency sized),
--reasoning-parser qwen3, no tool-calling
- Live-validated 2026-06-16: serves clean correct reasoning/code (~110 TPS),
qwen3 parser splits <think>. fp8 WEIGHTS rejected (break the quant-sensitive
3B: non-terminating empty output). Always-reasoning + no-tools → fails
verify-full's fixed-small-budget checks (5/9) by design → incubating, not gate-passing.
Guard suite 47/47 green (nex-n2-mini parked aside; it's an unrelated local experiment).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
deckard-40b loops at default sampling (reported by milano @ 262K ctx). The
compose shipped --repeat-penalty 1.0 (no penalty) and llama.cpp's default
repeat-last-n 64 → degenerate loops (a 10-word phrase repeating up to ~46x in a
2500-token gen, intermittent at temp 0.6). milano's own rep 1.05 + presence 0.05
only takes that to ~18 ("helps then loops again").
Change (validated quality-neutral by a same-session symmetric 8-pack A/B):
- --repeat-penalty 1.0 -> 1.1 (env REPEAT_PENALTY)
- add --repeat-last-n 256 (env REPEAT_LAST_N; was llama.cpp default 64)
→ together these cut looping ~5x (max 46 -> 9 in a per-request sweep) and are
gentle on code (unlike DRY).
- wire DRY sampler env knobs, DEFAULT OFF (--dry-multiplier 0.0 / --dry-base
1.75 / --dry-allowed-length 2): DRY is the strongest loop-breaker but
over-suppresses legitimate repetition in CODE, so it's opt-in (DRY_MULTIPLIER=0.8)
for severe long-ctx loops. Header env-docs updated.
Symmetric 8-pack A/B (same harness, 2026-06-13, think-off, MTP n=2):
OLD rep1.0 99/150 vs NEW rep1.1 100/150 (Δ +1 = noise; per-pack deltas
bidirectional). NB: OLD measured 99 today vs the #350 historical 105 — that
6-pt gap is harness drift since 2026-06-10, NOT this change (the change is +1).
So: anti-loop win at zero quality cost.
Reasoning is OFF by default on this compose, so --reasoning-budget (a common
suggestion) is a no-op here unless the user enables reasoning.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The rolling ghcr.io/ikawrakow/ik-llama-cpp:cu13-server tag moved (2026-05-23
-> 2026-06-10, digest 5f914f1c) and the new build REJECTS the legacy
speculative-decode flags: "legacy speculative option '--multi-token-prediction'
is disabled; use --spec-type mtp:n_max=1,p_min=0.0". Every ik-llama MTP compose
crash-restart-loops on a fresh pull (reported by furfix, Discord — single 3090
WSL). The binary's own --help confirms `-mtp`/`--draft-*`/`--spec-stage` are all
rejected in favour of the unified `--spec-type SPEC[:k=v,...]`.
Flag migration (13 composes):
- 11 MTP composes: --multi-token-prediction + --draft-max N + --draft-p-min P
-> --spec-type mtp:n_max=N,p_min=P (per-compose N preserved).
- 2 two-stage composes: the ngram-mod + mtp cascade used the rejected
--spec-stage with legacy keys -> --spec-type ngram-mod:...,ngram_size_n=... +
--spec-type mtp:...,p_min=... (spec-ngram-size-n -> ngram_size_n,
draft-p-min -> p_min, the canonical keys).
Digest-pin (decided to stop the recurrence — this is the 2nd flag churn from
the moving tag; ik-llama has no engine profile, so the per-compose image:
default is the only pin point): :cu13-server -> @sha256:5f914f1c... in all 13
ik composes + the preflight test fixture + 3 docs (IK_LLAMA / SINGLE_CARD /
INFERENCE_ENGINES). VLLM-style central injection isn't available for ik-llama;
a follow-up could add an engine profile so future pins are one-line.
Live-validated on the pinned image (single 3090):
- ik-llama/iq4ks-mtp: healthy, 0 restarts, no legacy error, MTP context ready,
served a completion, draft acceptance 0.62 (26/42).
- ik-llama/iq4ks-two-stage: healthy, 0 restarts, both stages load
(ngram_mod n=16 + MTP context ready), speculative decoding initialized.
Full test suite green (test-compose-registry-disk pre-existing on an unrelated
untracked model dir). The 3 untracked nex-n2-mini ik composes also carry the
rolling tag but are out of scope here (in-progress catalog work).
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Make `vllm/diffusiongemma-dual` usable through setup/pull/switch/launch, and
pivot the engine delivery from a baked local image to a SIDELOADED overlay on a
stock, pullable nightly (so non-rig users can actually run it).
Delivery (no baked image):
- Engine `vllm-diffusion-gemma` pins a STOCK `vllm/vllm-openai:nightly-2c9c07c8…`
and sideloads the vendored overlay at boot via install_script (entrypoint).
- `models/diffusiongemma-26b-a4b/vllm/patches/dgemma-overlay/` is the vendored
payload: the FULL stock-vs-`dgemma`-branch `vllm/` delta (123 .py) — a lean
PR-#45163-only overlay version-skews (`build_attn_metadata() got an unexpected
keyword argument 'causal'`: load-bearing dgemma changes live outside the PR
diff) — plus Codex's 3 fixes (marlin K-pad ×2 + diffusion_gemma TP-vocab/dtype).
install.sh cp's it over the installed vllm pkg, fail-loud arch assert.
- base.yml rewired: stock image + overlay mount + install_script entrypoint;
drops the 3 standalone marlin-k-pad mounts (folded into the overlay) and the
baked-image dependency. The dgemma-pr45163 Dockerfile is now the overlay-
regeneration helper, not the runtime engine.
Catalog wiring:
- Model profile `diffusiongemma-26b-a4b` (gemma4-swa-moe backbone + block
diffusion; text; valid_tp [1,2]; kv_calc_supported=false → kvcalc_key SKIP).
- Engine profile `vllm-diffusion-gemma` (stock install + vendored_overlays).
- compose_registry `vllm/diffusiongemma-dual` (port 8042, status upstream-gated).
NO DEFAULTS row — upstream-gated is non-functional, reachable only by slug.
- patches.yml `dgemma-vllm45163-sideload` (install_script + drift_guard) and the
diagnose_profile_cli overlay-path hint.
- docs/UPSTREAM.md row for vllm#45163 (re-pin + drop triggers).
Output-length fix carried in base.yml: lift the model's 256-tok max_new_tokens
default to 16384 via --override-generation-config (keeps the diffusion denoising
params; fixes OWUI truncation + next-turn echo).
Tests: bump the count fixtures for +1 model / +1 engine / +1 compose / +1 registry
entry (test-profiles-compat 6→7 models, 11→12 engines; test-compose-registry-disk
44→45 registry, 45→46 disk). Full gate green except test-compose-registry-disk
local-only redness from untracked nex-n2-mini WIP in the shared tree (CI-clean
validated green: tracked-only disk=46, registry=45, disk-only set all-allowed).
Validated live on 2x RTX 3090: stock nightly + sideloaded overlay boots, serves
coherent output, 262K, via `docker compose -f base.yml up` (the switch/launch path).
Upstream-gated until #45163 merges into a pinnable engine.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
🧪 Experimental compose for DiffusionGemma-26B-A4B (Google's first vLLM
dLLM) on 2x RTX 3090 Ampere, via the UNMERGED vllm#45163 dgemma branch.
- base.yml: TP=2, 262K ctx, eager, TRITON_ATTN, entropy-bound diffusion
sampler, gemma4 tool+reasoning parsers. Mounts the three fix-overlays.
- marlin-k-pad/{marlin,marlin_utils_fp8}.py: Codex's Marlin sub-tile-K pad
(lets fp8 Marlin W8A16 tile DiffusionGemma's K=352/1056 dims within
Ampere's 99 KB shared mem -- the original "fp8 dead on Ampere" blocker).
- marlin-k-pad/diffusion_gemma.py: Codex's TP-vocab soft-embedding fix
+ the :656 dtype cast.
- dgemma-pr45163/Dockerfile: builds vllm-dgemma:pr45163 (June-10 nightly
+ dgemma overlay).
base.yml also lifts the model's generation_config max_new_tokens 256 -> 16384
via --override-generation-config (NOT --generation-config vllm, which would
strip the diffusion denoising params). The 256 default was applied as a hard
max_tokens cap -> OWUI answers truncated at 256 (finish_reason=length) and the
no-EOS turn made the next turn echo the previous answer; both fixed.
Engine is a LOCAL image from unmerged PRs -- NOT pinnable/reproducible for
other rigs. Do not promote to master / the registry until vllm#45163 merges
into a pinnable nightly (see REGISTRY_PREP.md, left untracked).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The arch-confirm blocker named in the 🧪 status note is resolved: the GGUF
header reports general.architecture=qwen35 (standard GQA, 97 layers), NOT the
Qwen3-Next/DeltaNet hybrid the original scaffold guessed. Corrected the profile
to family=qwen35-dense and dropped the fictional GDN/linear-attention fields.
Fixes found while correcting the profile:
- Restored the required `attention_k_eq_v: false` field (compat.py reads it with
bracket notation — dropping it broke load_profiles for the whole catalog).
- Fixed a duplicate-key bug in the weights block: a second `hf_repo:` (DavidAU
base, no MTP head) shadowed PiehSoft's MTP GGUF under yaml.safe_load
last-key-wins — setup.sh would have fetched the wrong, head-less repo.
- Added qwen35-dense to llama-cpp-local supported_model_families (the C10 gate).
Renamed the weights slug mtp-q6k → piehsoft-q6k to follow the provider-quant
idiom used by every other GGUF compose (ubergarm-iq4ks, unsloth-q4km,
byteshape-iq4xs) and to drop the redundant mtp in mtp-q6k/mtp.yml. The serving
filename stays mtp.yml (KV isn't encoded in GGUF-engine filenames). Renamed the
compose dir + on-disk weights dir + all path references in lockstep.
Validation (2× 3090, server-cuda-b9570): verify-full 8/8, verify-stress 8/8
(ceiling ladder filled to 120K / 91% of n_ctx, 0 MiB VRAM growth, needle recall
clean through 90K), 8-pack 105/150 with MTP off==on (spec-dec lossless),
soak-continuous PASS. Guard suite 42/0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The MTP-injected GGUF is NOT a local build (the scaffold's comment was wrong) —
it's published at PiehSoft/Qwen3.6-40B-Deckard-MTP-Q6_K. Wire hf_repo + files so
setup.sh/launch.sh auto-fetch it, and correct the compose/profile comments.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Vet fixes on the Deckard-40B catalog PR:
- Engine: pin immutable :server-cuda-b9570 (the validated 2026-06-09 build,
digest 1c4ff61a) instead of the rolling :server-cuda tag — rolling regressed
at b9282 with a crash loop (#187), and b9570 is what we benched. The shared
llama-cpp-local engine pins b9246 (for the 27B, too old for Deckard-40B), so
pin the newer build at the compose level via LLAMACPP_IMAGE.
- test-compose-registry-disk: disk count 49 -> 45 (the correct +1 delta for
Deckard's one compose; 49 had absorbed 4 untracked nex-n2-mini WIP composes).
- Compose header status word: 'Unverified' -> 'Experimental' (canonical 🧪).
Full guard suite 42/42 green in a clean checkout (nex-n2-mini parked).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dense 40B uncensored Qwen3.6 community merge, dual-3090 llama.cpp with
embedded MTP head (Q6_K GGUF, 31 GB). First dual llama.cpp compose in
the catalog; first 'category' field in the registry (uncensored).
Changes:
- Compose: models/qwen3.6-40b-deckard/llama-cpp/compose/dual/mtp-q6k/mtp.yml
Status: 🧪 Unverified (quality /150 + soak pending)
Config: -ngl 99 -ts 1,1 -fa on --spec-type draft-mtp --spec-draft-n-max 2
--cache-type-k q8_0 --cache-type-v q8_0 -c 131072
- Registry: llamacpp/deckard40B-dual-mtp (category=uncensored, experimental)
- DEFAULTS: (qwen3.6-40b-deckard, llamacpp, dual) → slug
- Model profile: scripts/lib/profiles/models/qwen3.6-40b-deckard.yml
- Drafter: qwen-mtp-builtin model_compat extended with qwen3.6-40b-deckard
- LiteLLM route: deckard-40b → :8199
- launch.sh: suggest_default_variant case for Deckard
- BENCHMARKS.md: new section with validated MTP n=2 numbers
- Test fixtures: registry count 43→44, disk 44→49, models 5→6
Wrinkle decisions (see PR description):
1. Dual llama.cpp: uses 'count: all' + '-ts 1,1' (tensor-split layer-split),
matching the validated serving config. No launcher changes needed — the
compose's deploy section handles GPU reservation directly.
2. Engine pin: uses rolling server-cuda tag (matching engine profile spec),
not the b9246 pin other composes use. The validated build (2026-06-09
digest 1c4ff61a) is newer than b9246 and has draft-mtp working. No engine
pin bump for other models.
NOT DONE (live validation — GPUs busy with quality eval):
- Boot/bench/soak/quality NOT run — maintainer to validate post-eval
- Status stays 🧪 until full gate passes
* Add vllm/qwen-27b-dual-balanced + record 3-way 8-pack A/B (tie)
New "balanced" dual tier: cyankiwi AWQ-BF16-INT4 (int4 group-32 + BF16 mtp,
compressed-tensors) + int8-PTH KV + MTP n=3, TP=2 @262K. Live-validated
2026-06-07 (Marlin WNA16, KV pool 370K tok / 1.41x — the LARGEST of the dual
family, since int4 weights free more VRAM than fp8's 8-bit; ~67 TPS decode).
3-way 8-pack A/B (--full, same harness, same day 2026-06-07):
fast (autoround+fp8KV) 109 · balanced (awq+int8KV) 105 · max (fp8+int8KV) 110
→ a TIE (deterministic packs 64/64/65; spread within +/-5-7 noise).
The short-context 8-pack does NOT separate the quants. So:
- fast (vllm/dual) stays the default;
- balanced/max are framed by SERVING characteristics (KV-pool headroom,
int8-PTH fidelity), NOT behavioral quality — headers/docs say so explicitly.
- The dimension where int8-PTH should win (long-ctx NIAH recall) is an open
follow-up, not claimed here.
Also corrects the stale fast-tier 129/150 baseline: that was the 2026-05-09
harness; today's benchlocal-cli (verifier fixes since) scores the same fast
config at 109/150. Annotated as not-comparable across all four compose headers.
Wiring: awq-bf16-int4 weights_variant (compressed-tensors, cyankiwi repo) +
registry entry (vllm-stable, kv int8_per_token_head, port 8016, SKIP kvcalc,
experimental) + dual-max status_note updated with the A/B + DUAL_CARD rows +
A/B footnote. test-compose-registry-disk 42->43 / 43->44. Full gate green 43/0;
compat C4/C10/C14 pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Correct balanced KV-pool claim: fast has the largest pool, not balanced
Measured v0.22.0 @262K, TP=2: fast 622K/2.37× > balanced 370K/1.41× > max
295K/1.13×. My initial framing claimed balanced had "the largest KV pool of
the dual family" — wrong: I only compared it to max and forgot fast. Fast's
17.5 GB autoround weights are far lighter than balanced's 27 GB AWQ, so fast
has the biggest pool by a wide margin.
This reframes balanced honestly: it is DOMINATED by the fast tier — slower
(~67 vs ~89 code TPS), smaller KV pool, and tied/below on the 8-pack. Its
only possible edge is int8-PTH KV fidelity > fast's fp8_e5m2 (same size — a
fidelity bet, not a memory one), which is UNPROVEN (the short-ctx 8-pack is
blind to it). Keep only if the long-ctx NIAH A/B (#470) proves it; else
deprecate. Fixed across the compose header, registry status_note, and
DUAL_CARD (rows + footnote); fast row now cites its 622K/2.37× pool.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A symmetric 4-slug family for the Qwen vLLM path:
*-fast = AutoRound INT4 + fp8_e5m2 KV (peak TPS, the proven path)
*-max = official FP8 + int8-PTH KV (higher fidelity @ 262K)
Slugs:
vllm/qwen-27b-dual-fast alias of vllm/dual (AutoRound INT4, TP=2) — production
vllm/qwen-27b-dual-max FP8 + int8-PTH, TP=2 — 🧪 live-validated 2026-06-07
vllm/qwen-27b-multi-fast AutoRound INT4, TP=4 — 🧪 cross-rig (dual is the proxy)
vllm/qwen-27b-multi-max FP8 + int8-PTH, TP=4 — 🧪 cross-rig (dual-max is the proxy)
dual-max validated live on this 2-card rig: FP8 weights load via MarlinFP8
W8A16 on Ampere sm_86, int8_per_token_head KV keeps the full 262K (KV pool
295K tok / 1.13x concurrency), MTP n=3 active, coherent. The multi4 configs
are byte-identical to their dual sibling apart from TP + gpu-count, so they
ship Experimental until a real >=4x 3090 host validates them (this dev rig
has 2 cards). No multi4 DEFAULT added — the resolver degrades to vllm/dual
(honest; no model carries a multi default), and the slugs stay reachable by name.
Catalog wiring:
- models/qwen3.6-27b.yml: new fp8 weights_variant (official e4m3 release,
layer-split safetensors + embedded mtp.safetensors + vision tower)
- vllm-stable.yml: + fp8 weight format (Marlin W8A16 on Ampere) and
+ int8_per_token_head KV — NATIVE on stock v0.22.0 for uniform-head-dim
models (qwen3-next); features flag stays false (no overlay, unlike Gemma's
#40391-provided int8-PTH on vllm-gemma-stable)
- compose_registry.py: 4 entries; test-compose-registry-disk counts 38->42 / 40->43
Full gate green (42/42); compat C4/C5/C10/C14 pass for all 4 slugs.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
n-sweep @370W (dual TP=2): n2 72.9/86.1 . n3 74.0/87.7 . n4 71.6/87.8 wall-TPS
(all within ~3% - the QAT-int4's fast acceptance decay caps the spec benefit).
n=3 is the best balance (top narrative, code tied with n=4, ~20% less drafting:
43% draft-accept vs n=4's 37%), so SPEC_N_MAX default 4->3. The w4a16 compose had
inherited n=4 from the autoround clone, but autoround's slower decay (88/73/59/48%
per-pos) earns n=4 while the QAT-int4's faster decay (64/39/25% at n=3) favors a
lower n - optimal MTP n is quant-specific. Updates the compose default + Quality
field, registry status_note, BENCHMARKS row. Quality unchanged (n = speed only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
