First 8-pack on vllm/tess-dual-nvfp4 (both modes, same-night GGUF
comparator 116/117). Gap is cli-40-concentrated (−8 off); deterministic
packs tie or better (RM-off 14/15 best-ever Tess). Per the
pre-registered rule the huginnfork NVFP4A16 weights-only A/B + FP8
precision arm are the next gates. BENCHMARKS vLLM subsection + registry
note + compose Quality line; Grok-§3 discriminator results recorded in
learnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Shipped-config quality refresh on b9967 + 16K budget (native template,
both modes, no pack regression vs the 2026-07-09 basis; first toolcall
15/15 OFF). The streaming+thinking finish=length caveat is non-repro
3/3 on this config — Status ⚠️→✅, caveat history kept as a RESOLVED
note, registry caveats→production, Quality lines refreshed. Drift +
parity + registry-json guards green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Fresh-boot shipped compose on the new pin: 63.4/73.5 wall (+22%/+8% vs
b9246 row), MTP accept 0.737. EAGLE3 external draft (self-converted per
the #674 recipe) at n=2: 47.5/61.3, accept 0.408 — no EAGLE3 compose.
Head-vs-feed mechanism note recorded.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
3/3 streaming tool-calls with thinking-ON clean on the live shipped
serve (incl. a parallel 2-tool call); finish=tool_calls, reasoning
separated. Caveat retained pending the full-gate refresh; engine-build
vs budget attribution unresolved.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Sean's reproducible conversion recipe (metadata-only --target-model-dir
trick, draft-quant quality-free finding) folded into the row Notes per
his suggestion; links the #674 comment as canonical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
First single-RTX-4090 data point for Tess-4-27B with an Eagle3 draft on
mainline llama.cpp b9932. 8-pack quality-test.sh --full:
124/150 (82.7%) think-on · 115/150 (76.7%) think-off.
Matched to the same-rig 2026-07-09 DFlash baseline (125 on / 117 off) it
lands within 1-2 scenarios (single-run noise), confirming Eagle3 is
quality-equivalent to DFlash and that the min-p 0.75->0 sampler change
costs no measurable quality. Includes the spec-draft-n-max sweep showing
n_max=2 is optimal (deeper drafting is strictly worse for this model).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
In-worker apply_model probe: all 16 full-attention layers carry the
shipped calibrated values byte-exact (0.023-0.271, none at 1.0), so the
provider A/B was calibrated-scales vs scale=1.0 — and tied. Corrects the
"load-on-hybrid unverified, assume scale=1.0" claims in the nvfp4-fast
compose header, registry status_note, and both BENCHMARKS rows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Decode 66.9/90.4 vs 69.7/85.5, 8-pack 109 vs 110, NIAH clean to 240K.
MTP head loads through the compressed-tensors path (first observed).
Catalog verdict: no second 27B slug — nvidia-pinned stays, unsloth BYO.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
First confirmed MoE-FP4-on-Ampere boot (MARLIN NvFp4 MoE backend, weight-only
W4A16 execution). Decode 179.5/179.4 vs the shipped tier's 182.3/182.3 and
8-pack 103/150 vs 104-equivalent — statistical ties at full 262K. Native-FP4
rigs need their own quality run (W4A4 activations are not exercised here).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The safetensors config (model_type qwen3_5_text) shows 48 linear_attention
+ 16 full_attention layers (full_attention_interval=4). The earlier
"standard GQA, 64 layers" claim was inferred from the GGUF header, which
only describes the full-attention geometry — llama.cpp's qwen35 arch
implements the hybrid internally. No behavioral results change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
First NVFP4 boot on Ampere — 69.7/85.5 decode, MTP n=3 accept 97-99%,
8-pack think-off 110/150 (ties the fp8 tier's 109). Capability/quality
datapoint behind the #670 unhide; not a recommended Ampere serving path
(no speed edge vs AutoRound).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* concurrency-probe: surface aggregate tok/s in verdict, RESULT, and sweep
The probe already computed aggregate throughput (summed completion
tokens / wall) per round but only printed it in the round table — the
verdict, RESULT line, and sweep knee reported per-stream only. Users
asking "what total throughput at N agents?" (the Bruno question) had
no first-party number to point at.
- verdict: aggregate tok/s (N streams) next to per-stream
- RESULT: agg_tps= field (machine-readable, sweep-parseable)
- SWEEP: per-N summary table (per-stream vs aggregate) + aggregate in
the knee line
Knee logic unchanged (still per-stream-floor based). Offline guard
test-concurrency-probe passes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* concurrency-probe: re-resolve served model after each SWEEP boot
The top-of-script MODEL autodetect runs before SWEEP boots anything on
$URL, so it silently falls back to the default (qwen3.6-27b) — and every
request against any other model 404s, reading as errors=N per round (hit
live sweeping vllm/qwen-35b-a3b-dual on :8051; the 27B sweep only worked
because the fallback equals its served name). Re-resolve from /v1/models
after each boot; an explicit caller MODEL= pin is respected (MODEL_PINNED).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* FAQ + BENCHMARKS: first-party concurrency/aggregate-throughput matrix
Measured via the extended concurrency-probe (this PR) on 2x3090:
- dense 27B dual-fast: batching knee N=2 (~104 decode-agg @16K ctx),
collapses to ~54 by N=8; generation shape 211 agg @ N=8
- 35B-A3B MoE dual: ~250-270 decode-agg FLAT N=2..16 @16K ctx;
generation shape 1,037 tok/s aggregate @ N=16 (92.8/stream, clean)
New FAQ entry: how to serve multiple coding agents (MAX_NUM_SEQS,
ctx-vs-concurrency, model pick by architecture, honest aggregate-vs-
per-stream caveats, prefill-bound note, probe invocation).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
n-sweep 2026-07-09 (dual 262K, server-side decode tok/s + draft
acceptance, 3 runs/arm after warm): n=2 peaks at 57.9 tok/s / 0.62
accept; throughput falls monotonically past it (34.9 / 0.27 @ n=6) as
Q4 draft acceptance craters. Confirms the shipped MTP_DRAFT_N_MAX=2
default (inherited from Deckard, now Tess-validated). Comment/doc note
only — no config change; full table in the per-model learnings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The original OOM at 98K was caused by MTP, not the context length. Two
5090s (#613 @guybrush01, #617 @paulp83) swept the space: MTP-on at 98K
leaves no room for the draft head + cudagraphs + GDN prefill scratch and
OOMs — but at 65K + util 0.85 it PASSES, with MTP's ~2x intact:
verify-stress all-pass, decode 129/155 TPS (vs 71/71 with SPEC=off), MTP
accept ~3.2, ~1.4 GB VRAM free.
So the default keeps MTP and trades context, rather than dropping MTP:
- compose single/nvfp4/mtp.yml: MAX_MODEL_LEN 98304 -> 65536,
GPU_MEMORY_UTILIZATION 0.92 -> 0.85, header updated to the swept config.
- registry: max_ctx 98304 -> 65536, mem_util 0.92 -> 0.85, status_note.
- BENCHMARKS: guybrush's SPEC=on+65K row (129/155 decode, accept 3.2).
- baselines.yml: 27B single-nvfp4 submission (stale=false, v0.24.0).
SPEC=off stays exposed as the max-ctx fallback (81K/98K @ 71 TPS) and the
tight-system-RAM path (MTP's draft load OOM-kills a 28 GB host, #617).
80 GB+ cards raise MAX_MODEL_LEN toward 262K with MTP on. Stays 🧪 (8-pack
quality still owed). Scoped guards green (baselines / status-drift /
switch+launch parity / profiles-compat).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The ⚠️-caveat required a clean v0.24.0 4-card fp8/e4m3-KV report to
upgrade to ✅. Two landed:
- #584 (@ryanmpelletier, 4× 3090 x16, Proxmox): attention backend
FLASHINFER (the #595 int8-PTH→fp8/e4m3 flip live at TP=4), verify-full
9/9, verify-stress clean to 240,634 tok, soak-continuous PASS, bench
n=5, and the full 8-pack at 111/150 — ties the 2-card dual-max proxy
(109/150), so max-tier quality is TP-invariant (measured, not assumed).
- #625 (@MoppelMat, 4× 3090 bare metal, mixed x4/x8, 300 W): a second
independent 4-card rig — decode 79/102, prefill-90K clean, soak PASS.
Decode beats ryan's x16 rig (bare metal); prefill trails ~30% (narrow
lanes are the bandwidth-bound bottleneck; decode is lane-insensitive
at TP=4) — a clean decode(compute)/prefill(interconnect) split.
Changes:
- compose header: Status ⚠️ → ✅, drop the Caveats block, Quality now
cites the 4-card 111/150, prose updated to the promotion evidence.
- registry: status caveats → production + refreshed status_note.
- baselines.yml: replace @Whamp's doubly-stale #446 int8-PTH row (v0.22.0)
with ryan's fresh v0.24.0 fp8-KV submission (TPS + 8-pack + ctx + soak);
MoppelMat noted as the corroborating 2nd rig. Emit computes stale=false.
- BENCHMARKS.md: append both v0.24.0 4-card rows (append-only history).
- MULTI_CARD.md: drop the "no 4-card bench yet / needs TP=4 confirmation"
prose now that both reports have landed.
Guards green: test-baselines, test-compose-status-drift,
test-{switch,launch}-registry-parity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
@paulp83's single-5090 run is the first time the NVFP4 tier has booted
anywhere. Boots clean (vLLM v0.24.0, quant=modelopt_mixed, fp8 KV @131K,
MTP off as shipped), verify-full 9/9, verify-stress needle-clean (9.8K +
29K), soak-continuous PASS (15 MiB growth / 100% retention / 0 err / p50
311). Decode 255.8 narr / 257.9 code TPS @ 60 ms TTFT — ~2.5-3x our 2x3090
AutoRound tier (native FP4 GEMM). VRAM 30.6/32 GB @131K — tight but flat.
- registry: de-blind the status_note (drop "never booted" → first 5090
validation with numbers); stays 🧪 (8-pack quality still owed — that run's
benchlocal sandboxes weren't built).
- baselines.yml: first NVFP4 row — a 1x5090-pcie submission (narr/code TPS,
clean@29K, soak PASS, engine v0.24.0, tier submitted); surfaces ⑂-labelled
in c3 like the multi-max submission. No primary row (we can't run it).
- compose header: AUTHORED BLIND block → FIRST COMMUNITY VALIDATION with the
gate results + the 98304 fallback note.
- BENCHMARKS.md: the first NVFP4 row in the 35B-A3B section.
Confirms the two design calls: MTP-off (speculative_config=None) and fp8 KV
at scale=1.0 (declared-not-shipped scales) both loaded correctly on the
modelopt path. learnings + memory updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Per llm-compressor's W8A8 guide, INT8 W8A8 quant is unsupported on compute
capability >= 10.0 (RTX 5090 sm_120, RTX 6000-Blackwell). Record it where it
matters: the experimental W8A8 "prefill corner" BENCHMARKS row (a factor
against promotion — can't serve the Blackwell community; FP8 weights stay the
Blackwell-safe 8-bit path) + a "silicon != kernel" note on the DTYPE_MATRIX
Blackwell line (the INT8 TCs exist, but the W8A8 kernel doesn't build there).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
@ryanmpelletier's flawless 4x 3090 report (#584) is the cross-rig
validation the compose header required ('until a real >=4x 3090 host
validates it'): v0.24.0, no power caps, all x16, verify-full 9/9,
verify-stress needles clean to 240K, soak-continuous PASS, bench n=5.
Third concordant 4-card validation (after @alanspires 6xVFIO + @Whamp),
first on v0.24.0.
- registry + compose header: status experimental -> production (both, drift
guard green). No DEFAULTS[(qwen,vllm,multi4)] entry -> no resolver cascade;
promotion just moves it onto the actionable list. Quality is TP-invariant,
carried from the vllm/dual proxy (109/150); 4-card 8-pack is the open
confirmation (report.sh runs none).
- baselines.yml: vllm/qwen-27b-multi-fast gains its 4x3090-pcie submission
(tier: submitted; decode 74.76/90.83, prefill 1288->1175, KV pool
1.77M/6.77x, NIAH 240K). Submission-only by construction - the 2-card
reference rig can never produce a local 4-card row.
- BENCHMARKS.md: ryan's row (all-x16 74.8/90.8 beats @Whamp's mixed-lane
59/74 on the same fp8-KV config -> PCIe lane width matters for TP=4).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The single-card Gemma-4 default was the catalog's only default with no
baseline row (wave-2 PRIORITY gap). Full gate on Anbeeld's OFFICIAL
v0.3.2-preview digest (the current beellama-local pin), rebench tag
gemma-dflash-regate:
- decode 44.91 narr / 79.76 code (n=5, CV 2.1%/5.6%), TTFT 115 ms
- NIAH ladder clean to 117,513 tok (91% of 128K); ceiling margin 929 MB
(< 1024 MB bar -> caveat stays, but up from ~177 MB pre-v0.3.2)
- 8-pack current harness: 108/150 off / 113/150 on (+5 in-band wash,
think-off default re-confirmed)
- soak PASS (p50 59.8, 100.6% retention, 0 growth, 0/100 silent-empty)
Row seeded via catalog-baseline.sh (first non-vLLM engine through the
producer flow); BENCHMARKS row + compose header (Quality + margin
numbers) updated to match.
Tooling fallout the gate exposed:
- rebench-full.sh hardcoded RUNS:-3/WARMUPS:-1, silently under-running
the bench protocol on every orchestrated gate. The A1 "n=3 env leak"
attribution was wrong -- it was this line (baselines comment
corrected). Fixed to protocol 5/3; n=3 had flattered gemma-dflash
code TPS by +8%.
- catalog-baseline.sh counted prefill-probe run-lines into its bench-n
gate (read n=8 for an n=5 log); now reads the summary headers.
Full scripts gate 64/64.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The producer wiring: a validated gate run now flows into the corpus
automatically and into the shipped bar via one reviewed command.
- scripts/catalog-baseline.sh (§2.3, one mechanism / three entry points):
gate-validates the tag dir (verify-full pass HARD · bench n>=3 HARD,
n<5 WARN vs the canonical target · quality present unless --tps-only),
extracts decode TPS/TTFT/8pk both arms/ctx_validated (NIAH-ladder
verdict incl. token count)/provenance (pin via the same resolution the
emit join uses; rig+power via nvidia-smi, all overridable), UPSERTS the
baselines.yml row textually (comments preserved), prints a unified
diff. Never commits — rows ship via PR. --baselines-file for tests.
- rebench-full.sh: on completion, EXACT-container-match the served
engine to a registry slug (identity semantics — never port/substring,
the F9 rule) and append a fingerprint-complete #249 record (engine_pin
via resolve_variant_pin/compose fallback · hardware · power-cap ·
quality_8pk extensions · soak status) to the gitignored corpus, then
print the catalog-baseline.sh induction prompt. BYO/swap serves skip
with a note (no registry identity to record against).
- measurement_record.py: --quality-8pk / --quality-8pk-think-on ->
measured_extensions (the designed extension namespace; frozen schema
untouched).
- test-catalog-baseline.sh: hermetic (synthetic tag dir + a COPY of
baselines.yml) — extraction, dry-run no-write, add->replace upsert,
refusals (missing verify/quality/unknown slug), real-file checksum
guard.
TRUTH CORRECTIONS the tool's first live run surfaced (run against the
real agents-a1-fp8-dual tag): the A1 gate's bench was n=3 (a RUNS=3 env
leak into the overnight gate — 'n=5' was misstated in publications), the
BENCHMARKS TPS pair was order-garbled (artifact: narrative 154.0 / code
153.8), and the #81-rescored think-on total (110) was never materialized
back into the tag JSON (reads 108) — materialization gap documented in
the row comment. baselines.yml + BENCHMARKS.md corrected.
Live-validated: rebench record block ran against the serving A1 → first
corpus record written (vllm-agents-a1-dual__e9d28e22.jsonl: pin v0.24.0,
rtx-3090, 370W, quality extensions); induction dry-run reproduces the
row from the tag. Deferred to 2b/2c: c3 local-overlay + richer badges;
the bench 10K/90K prefill/TTFT probe. Gates: test-catalog-baseline +
test-measurement-record + test-baselines green; full scripts gate green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
First cross-rig confirmation: decode 152.2/152.3 (CV 0.0%) within ~1% of
the gate rig despite one card on an OCuLink eGPU dock at PCIe 4.0 x4 and
both capped 250W — single-stream TP=2 decode is VRAM-bandwidth-bound.
Full battery mirrors the gate (NIAH 6/6 to 240K, soak 100% retention,
agentic TTFT sub-linear); boot ran custom all-reduce via the PCIe-P2P
auto-detect, clean throughout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
InternScience's 35B agentic MoE (Qwen3-Next MoE arch; the card declares its
OWN base -> own model entry, not a qwen fine-tune slug), served from the
official FP8-dynamic compressed-tensors checkpoint on stock vLLM v0.24.0,
dual 3090 TP=2, full 262K. The agentic thinking-ON specialist.
First model onboarded END-TO-END through the Bring & Validate lane (T2
producer-zero): pull.sh route-C sibling swap (post deriver fix) ->
generate-compose -> documented swap -> full gate -> ④ vs-bar -> ⑤ scaffold.
Full gate PASS (rebench tag agents-a1-fp8-dual, 2026-07-03):
- bench 153.9/154.0 decode TPS (n=5, CV 0.1%, TTFT ~130ms), ~22.0 GB/card
- verify-stress 8/8 — staggered NIAH exact-recall to 240K (91%), VRAM Δ0
- soak PASS (0 growth, 0/100 silent-empty, 99.8% retention)
- 8-pack --full OFF 105/150 · ON 110/150 (post benchlocal #79+#81 harness):
toolcall 15/15 OFF · IF 15/15 ON · cli-40 thinking-ON 23/40 (the stack's
highest; base 17/40) · hermes 12/20 OFF, REGRESSES to 9/20 thinking-ON
(verified model behavior — disclosed as caveat 1)
vs qwen3.6-35b-a3b: general capability TIES (ON 110=110), decode −13% —
NOT a general upgrade; reach for it on tool/CLI-agent work thinking-ON.
Catalog wiring: compose dual/fp8-dynamic/fp8.yml (non-default KV named per
convention; OWN port 8072 — sibling-shared ports masquerade the slug in
estate detection); model profile (geometry verified == 35B-A3B from
config.json; mtp_num_hidden_layers=0 — safetensors header shows 0 mtp
tensors, config's 1 is an inherited default); registry entry status=caveats;
kv-calc agents-a1 spec + alias (fit-all prices, calibration 100%);
weights.py aliases (hf auto-fetch); LiteLLM route :8072; BENCHMARKS section;
count bumps (registry 57, disk 58, models 11).
Full catalog suite 60/60 (1 = known worktree-fixture submit-bench).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The shared served-model-name `qwen3.6-27b-autoround` (from #490) mislabels the
non-autoround 27b scenes (fp8 dual-max, lmcache): /v1/models advertises
"autoround" while `root` points at qwen3.6-27b-fp8. Same class on gemma-4-31b —
after the v0.24.0 consolidation the default is cyankiwi qat-AWQ-INT4 (bf16 KV),
yet the LiteLLM route still targeted `gemma-4-31b-autoround` (a latent #482 drift).
Fix WITHOUT breaking anything, via vLLM multi-served-name:
- Every 27b scene now serves `qwen3.6-27b <its-quant-name>`; every gemma-31b
scene serves `gemma-4-31b <its-quant-name>`. The neutral name is PRIMARY
(honest /v1/models id); the quant-specific name is retained as a live ALIAS.
- LiteLLM: add `qwen3.6-27b` / `gemma-4-31b` canonical public routes; keep the
`-autoround` routes as back-compat aliases (same upstream). Repairs the gemma drift.
- Migrate our own MODEL= defaults + docs (bench/verify/quality/launch/setup, c3,
tui-core, EXAMPLES, ...) to the neutral name. Weights slugs (`-autoround-int4`)
untouched; CHANGELOG + results/ history left as-is.
Retiring the `-autoround` alias entirely is a deliberate later step once nothing
still asks for it.
Live-verified on-rig (single/minimal, stock v0.24.0): /v1/models lists BOTH names
(root=...-autoround-int4); chat to `qwen3.6-27b` AND `qwen3.6-27b-autoround` both
return 200; `qwen3.6-27b-fp8` correctly 404s. Full shell gate 59/59 (1 = known
worktree-fixture); c3 pytest 41 passed; served-name arg-order + YAML validated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The DiffusionGemma arch is NATIVE in v0.24.0 (PR #45163 merged 2026-06-12), so
the vllm-diffusion-gemma engine drops the mutable/branch :gemma digest for the
same pinned stable tag the rest of the stack uses. The 3 Ampere/TP fix-mounts
STAY (not in #45163): marlin.py + marlin_utils_fp8.py (sm_86 fp8 MoE Marlin
K-pad — #45295's dense marlin-pad in v0.24.0 does NOT cover marlin_moe_wna16)
and diffusion_gemma.py (TP-vocab; native #45163 copy is still TP=1).
Also fixes the compose for v0.24.0's arg-parser: the bare positional model must
be the --model flag (the entrypoint prepends --disable-custom-all-reduce).
Live-validated 2026-07-02: the actual compose boots + serves coherent at TP=2 on
stock v0.24.0 (Paris / primary colors / 42). Suite 59 pass / 1 pre-existing
(test-submit-bench worktree-fixture-absent).
- engine vllm-diffusion-gemma.yml: install.spec :gemma@digest -> v0.24.0 + provenance
- compose base.yml: image default -> v0.24.0 + --model flag fix + header
- patches.yml dgemma-gemma-image-fixes: reason/drift_guard/drop_when -> v0.24.0
- BENCHMARKS + UPSTREAM #45163 row: reflect the bump
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The serving row (2c686e9) captured TPS/soak/NIAH from the report; this
adds the quality-test.sh --full 8-pack (110/150) they posted on #474.
Lands on our dual-3090 baseline (109/150) within n=1 noise — confirms
8-pack quality is weights+KV+sampler-bound, not GPU-arch-bound.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Same-rig NVLink/PHB toggle on 2x 3090 PCIe 3.0 x16 (#488). Records both
arms as measured data. Within-rig delta is +56-60% (consistent across
soak/narr/code), but power was not pinned across the two runs (325/298 W
NVLink vs 228/253 W PHB, ~8 h apart) and the magnitude is larger than the
power-controlled #77 A/B (+15%). PCIe 3.0 x16 ~= 4.0 x8 in bandwidth, so
the bus alone doesn't explain it. Notes flag the magnitude as an upper
bound pending a power-matched re-run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Both DeepReinforce agentic-coding fine-tunes land in-band with the base Qwen on the 8-pack; the 35B edges the base on aider (15/30 vs 12-13), the 9B is a lean-footprint niche. ik_llama GGUF, ngram, 262K. Slugs ik-llama/ornith9b-single (#477) + ik-llama/ornith35b-dual (#479).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
multi-fast fp8 (59/74), multi-max int8-PTH (85/102), and the standout INT4+int8-PTH (92.50/121.85). Decomposes to +57% narr from KV format (fp8->int8-PTH) + 9% from int4-vs-fp8 weights. P2P-kernel installed, engagement TBD.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Full rebench-full (--with-8pack-thinking=both) completed 2026-06-18 for
llamacpp/qwen27b-pi-reasoning-single. Replaces the earlier inferred "≈ base
50/59 @370W" note (+ the PENDING-ladder caveats) with the measured numbers, and
adds the BENCHMARKS.md single-card row.
- Bench @370W: 47.9/55.3 decode narr/code (n=5, CV<2%); @230W cap 28.5/32.9.
- verify-stress 8/8 (NIAH->183K), 8-pack 104/150 off / 106/150 on, soak PASS
(0 MiB growth, 0/100 silent-empty, p50 54.5, 102% retention).
- MTP head NOT weaker than base: matched-power A/B dead-even (73% vs 72% accept);
47.9/55.3 @370W ~on par with base 50.3/58.9. Power-sensitive (mainline -42%
370->230W) — the earlier "slow" 28/33 was the rig's 230W cap, not the model.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First cross-rig confirmation of the shipped vllm/diffusiongemma-dual compose
(@steamEngineer, 2x 3090 NVLink). Decode 294/370 — notably above the PCIe
reference (~177/180), plausibly the NVLink TP=2 all-reduce, though block-
diffusion variance is high. NIAH usable to 184K, verify-stress 8/8; the lone
verify-full fail is the SSE-chunk check (false-fail for a block-diffusion model).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
deucebucket's Qwen3.6-35B-A3B-Cerebellum-v3 (mainline llama.cpp single-3090,
ablation-informed mixed-precision GGUF, 11.96 GB, ~15.1 GB peak @131K, 147.9/146.2
TPS, det 63/75). Landed as a credited author-reported data point rather than a
catalog compose (PR #393 closed): its headline value is the 16 GB fit, which we
have no 16 GB card to validate or support; on 24 GB it's context/quality-dominated
by the ik-llama apex-fit/byteshape siblings. README llama.cpp ❌→✅ correction
(559a3fe) was the durable fix from the same report.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The arch-confirm blocker named in the 🧪 status note is resolved: the GGUF
header reports general.architecture=qwen35 (standard GQA, 97 layers), NOT the
Qwen3-Next/DeltaNet hybrid the original scaffold guessed. Corrected the profile
to family=qwen35-dense and dropped the fictional GDN/linear-attention fields.
Fixes found while correcting the profile:
- Restored the required `attention_k_eq_v: false` field (compat.py reads it with
bracket notation — dropping it broke load_profiles for the whole catalog).
- Fixed a duplicate-key bug in the weights block: a second `hf_repo:` (DavidAU
base, no MTP head) shadowed PiehSoft's MTP GGUF under yaml.safe_load
last-key-wins — setup.sh would have fetched the wrong, head-less repo.
- Added qwen35-dense to llama-cpp-local supported_model_families (the C10 gate).
Renamed the weights slug mtp-q6k → piehsoft-q6k to follow the provider-quant
idiom used by every other GGUF compose (ubergarm-iq4ks, unsloth-q4km,
byteshape-iq4xs) and to drop the redundant mtp in mtp-q6k/mtp.yml. The serving
filename stays mtp.yml (KV isn't encoded in GGUF-engine filenames). Renamed the
compose dir + on-disk weights dir + all path references in lockstep.
Validation (2× 3090, server-cuda-b9570): verify-full 8/8, verify-stress 8/8
(ceiling ladder filled to 120K / 91% of n_ctx, 0 MiB VRAM growth, needle recall
clean through 90K), 8-pack 105/150 with MTP off==on (spec-dec lossless),
soak-continuous PASS. Guard suite 42/0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dense 40B uncensored Qwen3.6 community merge, dual-3090 llama.cpp with
embedded MTP head (Q6_K GGUF, 31 GB). First dual llama.cpp compose in
the catalog; first 'category' field in the registry (uncensored).
Changes:
- Compose: models/qwen3.6-40b-deckard/llama-cpp/compose/dual/mtp-q6k/mtp.yml
Status: 🧪 Unverified (quality /150 + soak pending)
Config: -ngl 99 -ts 1,1 -fa on --spec-type draft-mtp --spec-draft-n-max 2
--cache-type-k q8_0 --cache-type-v q8_0 -c 131072
- Registry: llamacpp/deckard40B-dual-mtp (category=uncensored, experimental)
- DEFAULTS: (qwen3.6-40b-deckard, llamacpp, dual) → slug
- Model profile: scripts/lib/profiles/models/qwen3.6-40b-deckard.yml
- Drafter: qwen-mtp-builtin model_compat extended with qwen3.6-40b-deckard
- LiteLLM route: deckard-40b → :8199
- launch.sh: suggest_default_variant case for Deckard
- BENCHMARKS.md: new section with validated MTP n=2 numbers
- Test fixtures: registry count 43→44, disk 44→49, models 5→6
Wrinkle decisions (see PR description):
1. Dual llama.cpp: uses 'count: all' + '-ts 1,1' (tensor-split layer-split),
matching the validated serving config. No launcher changes needed — the
compose's deploy section handles GPU reservation directly.
2. Engine pin: uses rolling server-cuda tag (matching engine profile spec),
not the b9246 pin other composes use. The validated build (2026-06-09
digest 1c4ff61a) is newer than b9246 and has draft-mtp working. No engine
pin bump for other models.
NOT DONE (live validation — GPUs busy with quality eval):
- Boot/bench/soak/quality NOT run — maintainer to validate post-eval
- Status stays 🧪 until full gate passes
n-sweep @370W (dual TP=2): n2 72.9/86.1 . n3 74.0/87.7 . n4 71.6/87.8 wall-TPS
(all within ~3% - the QAT-int4's fast acceptance decay caps the spec benefit).
n=3 is the best balance (top narrative, code tied with n=4, ~20% less drafting:
43% draft-accept vs n=4's 37%), so SPEC_N_MAX default 4->3. The w4a16 compose had
inherited n=4 from the autoround clone, but autoround's slower decay (88/73/59/48%
per-pos) earns n=4 while the QAT-int4's faster decay (64/39/25% at n=3) favors a
lower n - optimal MTP n is quant-specific. Updates the compose default + Quality
field, registry status_note, BENCHMARKS row. Quality unchanged (n = speed only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
vllm/gemma-31b-qat-w4a16-dual A/B vs the autoround-int4 default (gemma-int8-mtp),
same int8-PTH KV + assistant MTP n=4 (byte-identical speculative-config):
- 8-pack 109/150 vs 105 (+4, within +/-5-7 noise = quality tie; real instructfollow
edge IF 15-vs-8 offset by hermes/cli/TC/RM).
- WEAKER spec-decode: MTP accept-len ~2.4 vs autoround's ~3.9 (per-position
acceptance lower at every position) -> ~30% lower TPS (71.6/87.8 @370W <
autoround 106/139 @230W despite MORE power; AL is the clean power-indep signal).
- Boots clean on stock vllm-gemma-stable (no #44494 workaround - tower arch).
Comparable quality but slower -> autoround-int4 stays the default; stays Experimental.
Updates the compose Quality field + registry status_note + a BENCHMARKS row.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
