Per llm-compressor's W8A8 guide, INT8 W8A8 quant is unsupported on compute
capability >= 10.0 (RTX 5090 sm_120, RTX 6000-Blackwell). Record it where it
matters: the experimental W8A8 "prefill corner" BENCHMARKS row (a factor
against promotion — can't serve the Blackwell community; FP8 weights stay the
Blackwell-safe 8-bit path) + a "silicon != kernel" note on the DTYPE_MATRIX
Blackwell line (the INT8 TCs exist, but the W8A8 kernel doesn't build there).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The dual-max FP8 2x3090 decode bench (83.1/108.2, v0.24.0, 2026-06-30) lived
only in BENCHMARKS.md:122 + disc #515 (c17490565) — never inducted into
baselines.yml, so c3 / the registry-emit join showed only guybrush01's
2x5090 submission and no local bar. Surfaced while diagnosing #585.
- baselines.yml: vllm/qwen-27b-dual-max gains its primary tier: local row
(83.1/108.2 · TTFT 158 · prefill 1364->875 · 8-pack 107/150 · NIAH 240K ·
v0.24.0 = current pin -> FRESH); guybrush's 2x5090 submission preserved.
- compose header + DUAL_CARD.md: the stale '~56 TPS' probe (and the now-false
'slowest of the three' framing) -> real decode 83/108; the genuine
tradeoffs (smallest KV pool 295K/1.13x, slowest prefill/TTFT 158ms from
FP8's compute-heavy Marlin W8A16 dequant) kept. BENCHMARKS.md already
corrected; this closes the two spots that still read ~56.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Adds a subsection to DTYPE_MATRIX's two-axes section clarifying that the
FMHA-kernel arch gating is a vLLM-family (vLLM + SGLang) phenomenon.
The llama.cpp family (mainline / ik-llama / beellama) is always-dequant:
KV quant is storage-only on every arch (dequant inside the FA kernel, no
FP8/FP4 tensor cores even on Hopper), so no consumer-vs-datacenter split
— q4_0 KV behaves the same on a 3090/4090/5090/Spark, which is why our
single-card GGUF configs hit 262K anywhere. GGUF weight quant is
dequant-to-FP16 too, so the native-FP8/NVFP4-weights win is vLLM-only.
Division of labor: native low-precision COMPUTE wins are vLLM-only (and
mostly datacenter for KV); the CAPACITY win (KV compression for long
ctx) is delivered arch-agnostically by the GGUF family — the right tool
for a consumer card that wants big context.
Verified: llama.cpp #22411 / #24109, ik_llama #1142.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Consolidates the validated findings from the #246 A/B + #571 volunteer
data + vLLM/TRT-LLM source review. New DTYPE_MATRIX section "Having the
Tensor Cores ≠ using them" makes the two load-bearing distinctions
explicit:
- Axis 1 — weights vs KV are gated SEPARATELY. FP8/FP4 *weights* work
wherever the TCs exist (sm_89+ FP8, all Blackwell FP4). KV is
different because it rides the attention (FMHA) kernel.
- Axis 2 — KV compute vs KV storage. Native FP8/FP4 *attention* compute
needs FA3 (Hopper sm_90 only) or trtllm-gen FMHA (datacenter Blackwell
sm_100/103 only). On EVERY consumer card — Ada 4090 (sm_89), consumer
Blackwell 5090/PRO-6000 (sm_120), DGX Spark GB10 (sm_121) — FP8 KV is
storage-only and nvfp4 KV doesn't work.
Consequences documented (both empirically confirmed): e4m3 ≡ e5m2 in
speed on consumer cards (86.73 vs 86.66 on a 5090, disc #571 — a
precision choice, not a perf lever); nvfp4 KV crashes on consumer
Blackwell (#43562). DGX Spark = same sm_12x family as the 5090.
Corrected now-wrong claims: DTYPE_MATRIX "Ada/Blackwell get a real win
on FP8 KV" (false — Hopper/DC-only) + the per-arch Ada/Blackwell-consumer
rows; QUANTIZATION + HARDWARE "native FP8 compute on sm_89+" → storage-
only + precision framing. Compute-win KV on consumer cards = INT8-PTH.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Two 5090 owners hit `--kv-cache-dtype nvfp4 requires sm100f` crashing
mid-boot on the #246 A/B (disc #571). Root-caused from vLLM #43562 /
TRT-LLM #10241: nvfp4 KV forces the trtllm-gen FP4 FMHA, built ONLY for
datacenter Blackwell sm_100/sm_103. Consumer Blackwell (sm_120/121 —
RTX 5090 / PRO 6000 Blackwell) is a HIGHER cc number but a different
family with no FMHA build. NVFP4 *weights* work there; only the KV path
doesn't. Our #246 gate had used a numeric ">=10.0" floor that wrongly
passed sm_120 — a floor can't express "sm_100/103 but not the
numerically-higher sm_120".
- gates.py: new `_ARCH_KERNEL_SM_FAMILY` allowlist ({nvfp4: sm_100/103});
dropped nvfp4 from the numeric `_ARCH_KERNEL_SM` floor; family-membership
reject with the FMHA reason + fp8_e4m3 fallback.
- arch-ab.sh: nvfp4 arm now refuses on consumer Blackwell (not just
<sm_10), naming the FMHA gap + the fp8_e4m3 path; dropped nvfp4 from the
recommended arms in help.
- hardware profiles: removed nvfp4 from rtx-5090 / rtx-6000-pro-blackwell
KV lists (both sm_120) + added a why-not note.
- docs (DTYPE_MATRIX / HARDWARE / KV_MATH / QUANTIZATION): corrected the
"Blackwell sm >= 10.0" framing to "datacenter sm_100/103 only".
- UPSTREAM.md: #43562 / TRT-LLM #10241 row + re-test trigger.
- test-arch-ab: nvfp4 refuses on sm_86 AND sm_120, allowed on sm_100;
the dual-5090 all-arms test drops nvfp4.
Full scripts gate 66/66.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Root cause of club-3090 #548, pinned from the stack trace: A1's
promotion accidentally committed the whole torch_compile cache (3,667
files, ~20 MB) compiled on THIS sm_86 rig, and the compose warm-start
mount fed it to every fresh pull. On sm_120 the AOT graph (baked
against Marlin-processed FP8 layers -> reads layer.workspace) loads
onto Cutlass-processed layers -> AttributeError -> restart loop.
vLLM's AOT cache key doesn't include arch/kernel selection, so the
cross-arch hit is silent (upstream issue to follow; UPSTREAM.md row
updated with the pinned mechanism).
Fix = the house pattern every other model already has: cache contents
gitignored (cache/.gitignore + README), directory kept for the mount,
local files untouched (our warm-start intact). Fresh users pay one
~60-90s compile on first boot and warm-start locally thereafter --
against THEIR OWN silicon's kernel selection.
Also retires the recurring dirty-tree noise from best_config files
updating during our own runs.
Full scripts gate 65/65.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
launch.sh/switch.sh now detect GPU arch and export KV_CACHE_DTYPE=
fp8_e4m3 (native FP8 compute) on sm_89+ cards for the pilot slugs
(vllm/dual, vllm/minimal). The injected value is the hardware
profiles' dormant kv_format_default.balanced -- one source of truth
shared with the pull gates; the Ampere no-op is data equality
(3090-class balanced = fp8_e5m2 = compose default -> nothing emitted),
not a code branch.
Injection guards (all load-bearing):
- pilot allowlist only; expansion gated on the #246 cross-rig A/B
- per-variant: registry kv_format == fp8_e5m2 only (int8-PTH/TQ/bf16
slugs never touched -- compressed-tensors weights reject fp8 KV)
- vLLM-family variants only; explicit user KV_CACHE_DTYPE= wins;
unmapped cards / heterogeneous rigs / no nvidia-smi -> no injection
- direct `docker compose up` keeps Ampere-safe compose defaults
Rides the existing resolve-variant-pin export seam (new optional
--gpu-spec); VLLM_ATTENTION_BACKEND is whitelisted but ships no value
(vLLM auto-detect stays the default until measured). Preflight banner
names the detected arch class.
Consistency fixes the injection exposed:
- gates.py _ARCH_KERNEL_SM: fp8_e4m3 9.0 -> 8.9 (vLLM's real floor is
SM89+; we'd otherwise inject e4m3 on 4090s our own pull-gate calls
unloadable) + new nvfp4: 10.0 row (v0.24.0 literal had NO gate --
a 3090 pull of an nvfp4 config wouldn't have been rejected)
- Blackwell hardware profiles: nvfp4 declared as CANDIDATE capability
(engine list unchanged until validated -- gates take the intersection)
- kv-calc: projected nvfp4 rows (bytes/elem + activation coefs),
calibration unchanged
Docs: HARDWARE.md new section, KV_MATH/QUANTIZATION/DTYPE_MATRIX rows
(incl. retiring the Genesis-era "e4m3 undertuned per #51" advisory in
favor of the A/B).
Validated: 8-case injection matrix in test-launch-compat; live no-op
on the real 2x3090 (spec built, nothing injected); faked 4090 through
the real bash detection path emits e4m3; compose interpolation both
ways; kv-calc --calibration green; full scripts gate 64/64.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Closes the rescore-materialization gap the induction tool's first live
run surfaced (#562): the benchlocal-cli 'rescore' subcommand ALREADY
supports write-back (--in-place / --output) — the gap was practice, not
a missing upstream feature. The T2 rescore ran to stdout only, so the
published A1 thinking-on 110/150 diverged from the tag artifact (108).
- A1's rescore MATERIALIZED into the tag JSONs (rescore --in-place; OFF
105 unchanged, ON 108→110) — induction now extracts 110 and the
regenerated corpus record carries 110; artifact, corpus, baseline row
and publication all agree. Pre-rescore JSON backed up outside the tag.
- docs/QUALITY_TEST.md: 'Rescoring saved results — MATERIALIZE, don't
just read' — the rule (a rescore that changes a published number must
be written back in the same session), the command, what rescore can't
re-run (sandbox packs), and the A1 case as the cautionary example.
- catalog-baseline.sh header: reads-artifacts-as-truth warning + the
header's gate text updated to the actual n>=3/warn-under-5 behavior.
- baselines.yml A1 comment: gap note -> materialized + doc pointer.
Guards green (test-catalog-baseline, test-baselines).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Investigation of a Discord cross-rig report (WSL, 3060Ti+4090: verify
fails on the pinned v0.3.2-preview digest, passes on the old noonghunna
snapshot; NOT reproducible on our 3090s — verify-full all-pass) surfaced
three doc-level truths worth recording:
- The 50-series gap is a TOOLCHAIN ceiling: Anbeeld's CI builds on the
Dockerfile-default CUDA 12.4, whose nvcc cannot target sm_120 (no
cubin, max PTX compute_90) — every official tag lacks Blackwell.
Upstream ask filed: Anbeeld#85 (CUDA_VERSION 12.8.1 + arch list incl
120); when it lands we retire the noonghunna snapshot entirely.
- Three registry status_notes claimed launchers inject 'server-cuda-
v0.3.0' — stale since the 2026-06-12 pin bump; they inject the
v0.3.2-preview digest from engines/beellama-local.yml install.spec.
Fixed all three (qwen dflash, gemma-12b, gemma dflash) + honest
labeling of the noonghunna snapshot as v0.3.0-feature-level and
unmaintained (predates KVarN + v0.3.1 fixes).
- The engine-notes self-build guidance was outdated: FA_ALL_QUANTS is
hardcoded in Anbeeld's cuda.Dockerfile since our PR Anbeeld#48, so a
self-build needs only CUDA_DOCKER_ARCH (+ CUDA_VERSION=12.8.1 for
sm_120). Recipe verified against his master Dockerfile 2026-07-04.
UPSTREAM.md beellama row updated (dated entry + next-triggers; the 'no
official image' claim struck through as historical). Gates: YAML +
registry import clean; status-drift / profiles-compat / switch-parity
green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Adds a stdlib HTTP control plane that wraps scripts/switch.sh so a harness
can POST /switch and block until the new model is serving. Introduces no new
orchestration logic — switch.sh stays the single source of truth (registry
lookup, down/up, readiness).
- tools/model-switch/server.py: GET /healthz|/status|/models, POST /switch
({slug}|{model}); registry-validated; /health readiness (works with or
without VLLM_API_KEY); single-flight lock; refuses to start unauthenticated
on a non-loopback bind.
- scripts/systemd/club3090-model-switch.service: host daemon unit.
- scripts/tests/test-model-switch.sh: hermetic HTTP/auth/validation contract.
- docs/EXAMPLES.md, .env.example: usage + config.
Mirrors the existing stdlib HTTP style (services/studio/*); zero new deps.
Experimental/opt-in per the repo's staging convention.
The shared served-model-name `qwen3.6-27b-autoround` (from #490) mislabels the
non-autoround 27b scenes (fp8 dual-max, lmcache): /v1/models advertises
"autoround" while `root` points at qwen3.6-27b-fp8. Same class on gemma-4-31b —
after the v0.24.0 consolidation the default is cyankiwi qat-AWQ-INT4 (bf16 KV),
yet the LiteLLM route still targeted `gemma-4-31b-autoround` (a latent #482 drift).
Fix WITHOUT breaking anything, via vLLM multi-served-name:
- Every 27b scene now serves `qwen3.6-27b <its-quant-name>`; every gemma-31b
scene serves `gemma-4-31b <its-quant-name>`. The neutral name is PRIMARY
(honest /v1/models id); the quant-specific name is retained as a live ALIAS.
- LiteLLM: add `qwen3.6-27b` / `gemma-4-31b` canonical public routes; keep the
`-autoround` routes as back-compat aliases (same upstream). Repairs the gemma drift.
- Migrate our own MODEL= defaults + docs (bench/verify/quality/launch/setup, c3,
tui-core, EXAMPLES, ...) to the neutral name. Weights slugs (`-autoround-int4`)
untouched; CHANGELOG + results/ history left as-is.
Retiring the `-autoround` alias entirely is a deliberate later step once nothing
still asks for it.
Live-verified on-rig (single/minimal, stock v0.24.0): /v1/models lists BOTH names
(root=...-autoround-int4); chat to `qwen3.6-27b` AND `qwen3.6-27b-autoround` both
return 200; `qwen3.6-27b-fp8` correctly 404s. Full shell gate 59/59 (1 = known
worktree-fixture); c3 pytest 41 passed; served-name arg-order + YAML validated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Our vllm#40361 sub-tile-n Marlin pad was closed-superseded by mgoin's
vllm#45295 (consolidated marlin_padded_nk across all dense Marlin paths),
native in vLLM v0.24.0. `vllm-stable` now pins v0.24.0 and the AutoRound
INT4 TP=2 path (vllm/dual) boots clean without the overlay (validated in
#533 Phase 0b). No live compose mounts the patch (archive-only), so this
is a tracking-only de-registration — the cleanup deferred from #533.
- patches.yml: qwen-vllm-marlin-pad -> deprecated (upstream.status
open->merged, load_bearing_when [], delivery none, drift_guard null),
mirroring the gemma-vllm-pr41800 merged-and-dropped precedent. Kept as
history (foundational false; entry not deleted).
- arch_patches.yml: correct the stale kernel_constraints note (#40361 ->
#45295 native in v0.24.0). required_patches / marlin_alignment_required
unchanged: the alignment is a real arch property (now satisfied stock),
and deprecated patches stay listed per the pr41800 precedent.
- UPSTREAM.md: mark the #40361 / #40354 / v0.24.0-bump marlin rows DONE
(native in v0.24.0, patch de-registered, no live mount) and correct the
stale "composes still mount it" line (all mounts are under _archive/).
Full shell gate green (59/59; the 1 = known worktree-fixture-absent
test-submit-bench). test-patch-attribution (reads both registry files) passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The DiffusionGemma arch is NATIVE in v0.24.0 (PR #45163 merged 2026-06-12), so
the vllm-diffusion-gemma engine drops the mutable/branch :gemma digest for the
same pinned stable tag the rest of the stack uses. The 3 Ampere/TP fix-mounts
STAY (not in #45163): marlin.py + marlin_utils_fp8.py (sm_86 fp8 MoE Marlin
K-pad — #45295's dense marlin-pad in v0.24.0 does NOT cover marlin_moe_wna16)
and diffusion_gemma.py (TP-vocab; native #45163 copy is still TP=1).
Also fixes the compose for v0.24.0's arg-parser: the bare positional model must
be the --model flag (the entrypoint prepends --disable-custom-all-reduce).
Live-validated 2026-07-02: the actual compose boots + serves coherent at TP=2 on
stock v0.24.0 (Paris / primary colors / 42). Suite 59 pass / 1 pre-existing
(test-submit-bench worktree-fixture-absent).
- engine vllm-diffusion-gemma.yml: install.spec :gemma@digest -> v0.24.0 + provenance
- compose base.yml: image default -> v0.24.0 + --model flag fix + header
- patches.yml dgemma-gemma-image-fixes: reason/drift_guard/drop_when -> v0.24.0
- BENCHMARKS + UPSTREAM #45163 row: reflect the bump
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
README + DUAL_CARD: single bf16 vllm/gemma-31b-dual @224K default (stock
v0.24.0, overlay-free); the v0.22.0 int8-PTH/bf16-mtp/qat-w4a16 composes are
deprecated. UPSTREAM #40391 row: the 31b DEFAULT moved to bf16 (int8-PTH craters
recall on v0.24.0 without #40391); the #40391/#42006 overlays + vllm-gemma-stable
STAY for the 26b-a4b-single + the deprecated 31b composes. 262K int8-PTH returns
overlay-free when #40391 merges.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The fast/balanced/max composes aren't rungs on a single fidelity axis —
they're corners of a 3-axis trade-space (weight-fidelity / decode+context
/ prefill-TTFT). §4b names the axes, the corner map, the legitimacy rule
(a differentiator no instrument can measure isn't a tier), scheme-vs-
algorithm, and why "balanced" stays provisional. It fills the prefill
corner with the measured INT8 W8A8 vs FP8 result (v0.24.0): quality tied
107/150, W8A8 wins prefill/TTFT (native CUTLASS INT8), FP8 wins decode —
a prefill-vs-decode tradeoff, not a replacement. W8A8 qualifies as an
analyzed tier; shipping it as a slug is deferred (not committed).
- §4a "Tiering principle" now points into §4b (topology projection).
- DUAL_CARD balanced/max footnote cross-links §4b for the rationale.
- QUANTIZATION.md added to the docs/README.md index.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Folds the current-design rationale from the parked production-agent design doc
into the public architecture page (leaving out roadmap/staging/target arch):
- "Why this shape — creative within bounds": serial dissolves the VRAM problem
on a PCIe rig (the hard part is creative orchestration); the model proposes,
the executor disposes (never trusted to sequence GPU ops); creative in the
film-making layer, never the control plane.
- Cross-shot consistency framed as the quality ceiling the Character Bible +
continuity modes attack.
- "Transport-success ≠ real success" added to Key decisions — the executor
ffprobe-validates the actual output (duration/audio/placeholder), not just
"a file appeared".
- Reproducible productions/<job_id>/ + typed manifest (seeds, workflow versions,
exact ffmpeg cmd, validator results) added to the troubleshooting section.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
- Expand the continuity modes into a table; spell out the DEFAULT "storyboard"
mode (each shot gets its own keyframe sharing one style bible, then animated
i2v from that keyframe) vs hero / chain / none.
- Add "Diagnosing & troubleshooting": health checks for the 3 moving parts
(pipe / 4B :8090 / planner :8195), the offline plan dry-run
(run.py --backend synthetic — the key diagnostic), live job inspection
(/job/<id>), the offline unit tests, and a symptom -> cause -> where-to-look
table (intent misreads, off-topic plans, thinking runaway, research-fail-open,
the decade duration mis-parse).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Two pieces of feedback:
- The decision-flow diagrams read like a code walkthrough (function names,
file:line, FLOOR/CONTROLLER). Rewrite Shape A + the Production Director's
Stage 1 ("how it responds to you") and Stage 2 ("what 'go' builds") as
behavioral flows — what the agent does from the user's side — and move the
code-level pointers into a compact "under the hood" aside + the existing
system-prompt map. Lighten the comparison table + the double-gate decision
to drop residual code identifiers.
- Mine the recent studio commits (#502/#513/#523) for agent-behavior detail
worth surfacing: the **Character Bible** (recurring characters defined once
with a fixed look + seed, referenced per shot for visual consistency),
**continuity modes** (storyboard/hero/chain/none), and the **honesty fix**
(the director won't claim it can browse arbitrary pages). Added to Stage 2,
the comparison table (new "Visual continuity" row), and Key decisions.
Also surfaces SearXNG as the documentary-research backend (agents doc callout +
the ai-studio README services line).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Add a prominent status banner at the top of agents-architecture.md and reinforce
it in the challenges intro: the Production Director is a work in progress, NOT
recommended for production use — the multi-turn chat is still buggy and does not
follow instructions reliably yet. The single-shot lanes are simpler/further along
but carry the listed rough edges. Sets reader expectations up front.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Documents the open director limitations as of #519-#524: no cross-lane routing
(manual lane pick → documentary-in-LTX gets a scenic clip), the 4B capability
ceiling (thinking-OFF + borderline critic, 27B/35B = the injectable upgrade path),
shallow single-shot-lane chat, the decade duration-parse bug (_target_seconds
reads "1980s"/"90s" as seconds — verified), fragmented prompt sources, and the
pending live end-to-end validation. Each row pairs the symptom with a path
forward (parked Q4 router / 27B critic / regex fix).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Documents the AI Studio agent architecture, which wasn't written down: the one
4B director (qwen3.5-4b-uncensored @:8090, thinking-OFF) behind all 13 lanes, the
two agent shapes (single-shot craft-or-decline vs the conversational plan-then-
execute Production Director), the Production decision flow (conversation controller
→ confirm latch → decide_action → /produce planning pipeline), a per-lane
comparison table, and the system-prompt map (8 prompts across AGENTS.md /
director_intent.py / prompts.py / inline DIRECTOR_*_SYS). Links to the editable
sources (AGENTS.md, director_intent.py, the :8195 server). Indexed from the
ai-studio README.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
`quality-test.sh --full` (which docs tell cross-rig contributors to run)
needs the 3 sandboxed packs' Docker images — but they aren't auto-pulled,
and benchlocal-cli's mid-run hint pointed at a relative
`tools/build-sandboxes.sh` that's only valid inside a benchlocal-cli
checkout (absent on a pip install). guybrush01 followed it into a dead
path (#492).
- quality-test.sh: on --full / --sandboxed-only, preflight the
benchlocal-sandbox-{bugfind,cli,hermes} images UP FRONT. If missing,
print the correct clone + build steps (our context, not benchlocal-cli's
broken relative path) + the --medium no-Docker fallback, then continue
with the deterministic packs.
- QUALITY_TEST.md: document that --full needs the sandbox images built
first ("needs Docker" undersold it) — the clone + build-sandboxes.sh
step, with --medium / --no-sandboxed as the no-Docker path.
Full suite green (59/59). Pairs with benchlocal-cli#69 (the upstream hint).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
setup-ai-studio.sh's director download honored MODEL_DIR, but the ComfyUI
assets (image/video/audio) defaulted to a hardcoded /mnt/models/comfyui — so
any rig whose models don't live under /mnt failed mid-download with
"mkdir: Permission denied" (sumo Discord report, 2026-06-27). Two more walls
behind it: the disk-space preflight checked the hardcoded /mnt path and had no
override (an idempotent resume under-threshold was blocked), and gpu-mode.sh —
which setup calls at step 3 — hardcoded CLUB3090_DIR=/opt/ai/github/club-3090,
breaking on any other clone.
- New services/comfyui/comfyui-paths.sh: derive COMFYUI_ROOT / COMFYUI_MODELS_DIR
as a "comfyui" sibling of MODEL_DIR (read from repo-root .env when unset).
Backward-compatible on the rig (/mnt/models/huggingface -> /mnt/models/comfyui);
explicit overrides respected. Sourced by setup-ai-studio.sh,
download_studio_models.sh, and gpu-mode.sh so the download target, the disk
check, and the container mounts all agree.
- setup-ai-studio.sh: disk check uses the derived path; add SKIP_DISK_CHECK=1 to
bypass the free-space preflight independently of SKIP_DOWNLOAD.
- gpu-mode.sh: auto-detect CLUB3090_DIR from the script's real location (resolving
the /usr/local/bin symlink), overridable via env — removes the hardcoded path.
- The 3 legacy hardcoded download scripts (flux2/hunyuan_llava/models) honor
COMFYUI_MODELS_DIR.
- New test-comfyui-paths.sh guards the derivation; docs/ai-studio config note.
Full suite green (59/59).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
New dedicated guide for getting the most out of a PCIe-only multi-GPU
rig: reading nvidia-smi topo -m (why PHB not PIX), NUMA/NPS1, triple-slot
placement + M.2 lane-sharing, BIOS (Above 4G / ReBAR / ACS / IOMMU),
enabling consumer P2P via the patched driver + NVLINK_MODE=pcie_p2p,
realistic measured expectations (cross-rig #91/#95/#77), verifying
engagement via report.sh's new field, and troubleshooting. Generic
guidance with ROMED8-2T as a labeled example board.
Cross-referenced from HARDWARE.md (NVLink section), DUAL_CARD.md (NVLink
auto-detection), FAQ.md (Do I need NVLink?), and the docs index.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Director placement lever: env-driven CPU / GPU0 / GPU1 (backend)
The studio director's GPU/CPU placement is now a single lever,
STUDIO_DIRECTOR_DEVICE (gpu0 | gpu1 | cpu, default gpu0), read from the rig
.env by gpu-mode's start_studio_director and translated into the compose
env (-ngl + CUDA_VISIBLE_DEVICES + device_ids):
- gpu0 (default): -ngl 99, GPU0 — fast craft (~50-100 tok/s), ~4.6 GB,
coexists with the image lanes. Unchanged from before.
- gpu1: -ngl 99, GPU1 — only when GPU1 has room (NOT during a video render;
GPU1 is the DisTorch DiT donor).
- cpu: -ngl 0, CUDA_VISIBLE_DEVICES="" — frees ~4.6 GB off GPU0 (lifts the
single-card Wan window 121→161 frames) at ~single-digit tok/s craft.
Compose now reads ${DIRECTOR_NGL:-99} + ${STUDIO_DIRECTOR_CUDA-0} (no-colon
so an explicit empty value = CPU survives). Default (no override) preserves
current GPU0 behaviour exactly.
Live-validated: CPU mode starts with GPU0 full (gemma12b), adds 0 MiB VRAM
to GPU0, serves on :8090, generates (~5 tok/s CPU). The c3 Settings field
that writes STUDIO_DIRECTOR_DEVICE follows in the next commit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* c3 Settings: Director placement (CPU / GPU0 / GPU1) + CPU thread cap
Exposes the director-placement lever in the c3 Settings modal so users pick
GPU0 (fast, default) / GPU1 / CPU without hand-editing the .env.
- services.py: director_device() reads STUDIO_DIRECTOR_DEVICE from the repo
.env (default gpu0, validates the value); set_repo_env_var() upserts a key
in place (preserves other lines, no duplicates, creates the file if absent).
- app.py: SettingsScreen gains a "Director placement" Select; apply_settings
persists the choice to the repo .env (the SHARED config gpu-mode reads —
distinct from c3-settings.json for MODEL_DIR/HF_TOKEN). Applies on the next
ai-studio start.
- compose: CPU thread cap — -t ${DIRECTOR_THREADS:-8} bounds CPU use so the
director doesn't starve OWUI's embedder/reranker (also CPU). The ~2.6 GB
GGUF loads into system RAM (mmap'd; resident in page cache, not run from SSD).
- tests: +6 data-layer (TestDirectorPlacement) + 1 headless apply-settings
round-trip (persists STUDIO_DIRECTOR_DEVICE, idempotent re-apply). Full
suite green (728), settings/director subset 13/13.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* tests: fix stale scene names in gpu-mode-list EXPECT
The scene catalog renamed its dispatch keywords to qwen27b / gemma-31b,
but the test's EXPECT spot-check map still referenced the old 27b / gemma
short names — so the JSON-shape assertion had been red on master. Point
EXPECT at the canonical names the catalog now emits.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* studio: run director in chat scene as the catalog-support layer
Bring the uncensored director up in the `chat` scene (honoring the
STUDIO_DIRECTOR_DEVICE placement knob), framing chat as the supporting-
infra home — OWUI + LiteLLM + Qdrant + SearXNG + director — for ad-hoc
Catalog models launched via `switch.sh --owui`.
A CPU-placed director uses no GPU, so it's the always-on path: it survives
scene switches and stays live in OWUI. New _director_evict_if_gpu helper
frees only a GPU-resident director when a dual-card LLM scene claims the
cards; mode_off stops it outright. Also brings mode_gemma_int8 in line with
its dual-card siblings (it was missing the studio teardown entirely).
Docs: requirements.md gains a "Chat scene — the Catalog-support layer"
section + reframes director placement around the unified knob / c3 Setting;
video.md note synced.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* studio: disable director thinking on CPU only (latency)
The uncensored director is an "Aggressive" reasoning fine-tune that emits
a full <think> trace before every answer. On GPU that's cheap and the
trace lands in reasoning_content (content stays clean), so leave it on.
On CPU (~14 tok/s) the trace dominates latency, so gpu-mode now passes
`--jinja --reasoning off` for the cpu placement only — forcing the
template's enable_thinking=false (this fine-tune ignores /no_think and
--reasoning-budget 0, but honors --reasoning off).
Wired via a new DIRECTOR_THINK_ARGS compose param (empty on GPU). Live:
CPU director now answers in one pass, no reasoning trace, craft quality
intact (full cinematic spec, finish=stop).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* c3: Containers-pane director start honors placement + enumerate sidecars
Two consistency fixes for the Containers tab:
1. Starting studio-director from Containers now honors STUDIO_DIRECTOR_DEVICE
instead of falling back to the GPU0/thinking-on compose default. New
director_compose_env() mirrors gpu-mode's start_studio_director translation
(NGL/CUDA/GPU/THINK_ARGS), injected as an `env K=V …` prefix on the compose
up cmd (process env wins over --env-file). cpu → -ngl 0 + --reasoning off.
2. The nested studio sidecars (director/gallery/orchestrator/image-shim/
step-voice/tts) now enumerate when STOPPED, so they're startable rows — not
only visible while running. New STUDIO_SIDECARS map is the single SoT for
resolving the container-name → services/studio/<sub>/ project (fixing the
director↔enhancer name mismatch that previously returned None → docker
restart, which fails on a fresh install).
+8 tests (director_compose_env cpu/gpu, director resolves to enhancer with the
env prefix, sidecar enumeration). Live: c3 service_start plan starts the director
CPU + no-think (argv -ngl 0 --reasoning off, GPU0 free, clean generation).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
image.md: replace the "evaluated and dropped (cloud-only)" note with the lane
write-up — local Krea2 landed in ComfyUI v0.26.0 (#14589); aesthetic/stylized,
aligned (Z-Image stays the uncensored pick). README: 11→12 lanes, +Krea row,
image-lane count 3→5.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Make a fresh clone go straight to generating, with the single-command UX the Image
Studio beta had — but for the consolidated image+video+audio studio.
- scripts/setup-ai-studio.sh (NEW, canonical): preflight (docker/gpu/~120 GB disk) →
build ComfyUI image → download_studio_models.sh (full roster) → gpu-mode ai-studio →
install the OWUI Studio pipe → onboarding. Flags: --yes / SKIP_BUILD / SKIP_DOWNLOAD / SKIP_PIPE.
- push-pipe-to-owui.sh now **UPSERTs**: installs the OWUI `function` row if absent (the
one previously-manual step — paste into Admin → Functions), else updates it. First-time
install just needs an OWUI admin account to exist. Validated: INSERT against a DB copy
(clean row + valid meta JSON), UPDATE live on the running OWUI.
- Retire the now-misnamed setup-image-studio.sh / setup-video-studio.sh → thin redirects to
setup-ai-studio.sh (they only pulled one modality post-consolidation and still echoed the
removed `gpu-mode image-studio`/`video-studio` scenes).
- Docs: services/studio/README.md (install is automated) + docs/ai-studio/README.md "Bring it up"
now leads with the one-command path.
Gate: scripts/tests 57/57; setup scripts syntax-clean; no test references the old script bodies.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Record the 2026-06-24 finding: the image lanes ship the fast/distilled checkpoints
(Z-Image-Turbo, HiDream-O1-Dev), and tuning them is a dead end (measured — Z-Image
steps/sampler/shift identical; HiDream negative-prompt no-op; noise_scale is a calibrated
constant that blanks the image if changed). The only real quality lever is the non-distilled
sibling: Z-Image base (50-step CFG — fits our card, clean drop-in) and HiDream-I1-Full
(50-step CFG — won't fit 2048² with the director resident; needs the director-off-GPU0 lever
+ 1024²/DisTorch). Parked as a future option — the current lanes are strong as-is.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
- requirements.md (new): hardware / software / disk / RAM to self-host AI Studio, expressed
generically (2× 24 GB Ampere+, PCIe ok / no NVLink; ~120 GB disk; 32 GB+ RAM), single-vs-dual-card,
and the director-placement VRAM lever. Linked from README's deep-dive index.
- video.md: a "Wan2.2 — tuning & limits" section (recipe + the sampler finding, 480p default /
720p-DisTorch valve, the corrected length-ceiling table, i2v-seeded chaining) + Wan model files in
the manifest. The ceiling table now shows the director as the single-card swing factor (121↔161)
and adds the DisTorch both-card column.
- Director placement documented as a lever (default GPU0; STUDIO_DIRECTOR_GPU / -ngl 0 relocate it),
with the caveat that GPU1 is unsafe for the LTX lanes (their 22 GB donor) — CPU is universal.
- The director carries its model name (Qwen3.5-4B-Uncensored) in the services manifest, matching the
lane-naming convention.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
- README.md: 9→11 lanes, +Wan2.2 / +Z-Image rows, video labels aligned to the unified
`Studio · Video (Model)` naming.
- image.md: +Z-Image lane (table + section) + the "Krea2 dropped — cloud-only, no local
detection" note.
- video.md: four video lanes now (LTX / Sulphur / 10Eros / Wan2.2); Wan caveats (text→video,
no synced audio, 832×480) in the capabilities table; 8→11-lane matrix ref.
- services/studio/README.md: 9→11 picker entries + the two new workflow-graph rows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Sweep the studio docs to match the consolidation: `image-studio` /
`video-studio` / `comfyui` scene refs → the single `ai-studio` scene, drop the
gemma-12b chat-on-the-other-card framing, and document the new pieces.
- README: 8→9 lanes (+10Eros); "Lanes vs modes" → "One scene, lanes inside it".
- video.md: model manifest LoRA `-384`→`-384-1.1`, +10Eros unet row, lane matrix
+🔓 10Eros, VRAM note → `ai-studio`.
- audio.md: step-voice ⊕-video mutex bullet (c3 guards it) + start-from-c3 note.
- image.md / FAQ.md / services/studio/README.md: scene-name + bring-up fixups.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
UPSTREAM.md rows authored by qwen max while assisting on this repo and left
uncommitted; landing them on master per the direct-commit-for-docs convention.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
expandable_segments around the cumem sleep-mode pool, not the
custom_all_reduce graph-buffer registration path — so it does not cover
this crash. The open in-engine fix for our path is #43923; that's the
drop condition for the detect_nvlink.sh guard.
Co-authored-by: Cursor <cursoragent@cursor.com>
Dual composes inject PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,...
for the PCIe (custom-all-reduce OFF) path. On NVLink / patched-PCIe-P2P
rigs detect_nvlink.sh turns custom all-reduce ON, where expandable_segments
backs allocations with a cuMemMap VA range that cudaIpcGetMemHandle rejects
during graph-buffer registration -> custom_all_reduce.cuh:455 'invalid
argument', worker dies at boot.
The prior fix clobbered PYTORCH_CUDA_ALLOC_CONF to max_split_size_mb:512,
discarding any other knob the user set (custom max_split_size_mb,
garbage_collection_threshold, ...). Instead strip ONLY the expandable_segments
token on the P2P path and keep the rest; fall back to max_split_size_mb:512
when nothing remains. The PCIe path keeps expandable_segments untouched.
Strengthen test-detect-nvlink-alloc-conf.sh to assert the surgical contract
(preserve neighbors, strip in any position, no dangling commas) and add the
vllm#42609 row to docs/UPSTREAM.md (its "DP>1 AND TP>1" title under-scopes
the bug; the same cuh:455 crash fires TP-only -- cf. vllm#42583 / vllm#45604).
Validated on a 2x RTX 3090 NVLink rig (TP=2, NV4): before, boot crashes at
custom_all_reduce.cuh:455 'invalid argument'; after, the dual stack boots
clean (custom AR ON, expandable_segments stripped), /v1/models ready, serves
a completion.
Co-authored-by: Cursor <cursoragent@cursor.com>
Three related [C0]/eligibility pull-gate fixes for the curated-swap surface —
an uncurated derive (abliterated / fine-tune) of a model we already serve.
1. Wrapper-arch alias. `pull.sh --profile-like` was false-aborting at [C0] with
"no arch_patches matrix row for 'Qwen3_5ForConditionalGeneration'". That is
the OUTER multimodal wrapper class the weights report; the patch matrix is
keyed on the inner canonical `Qwen3NextForCausalLM`. arch_patches.yml is a
closed key-set, so the alias lives in the editable arch_model_xref.
- profile_runtime.yml: `config_architectures: [Qwen3_5ForConditionalGeneration]`
on the Qwen3NextForCausalLM xref entry.
- generate_compose.py: `resolve_arch_from_config()` maps a config.json
architectures[0] string -> (canonical_arch, arch_row) via that alias.
- gates.py [C0]: resolve the wrapper arch via the alias before declaring
NO_ARCH_ROW. The hybrid now reports ENGINE_SUPPORTED.
2. GGUF axis. `supported_weight_formats` was declared on every engine but never
enforced (only `kv_format` was). The deriver blocks GGUF on the derive path,
but the curated registry / curated-swap path had no such guard. gates.py [C0]
now rejects a `gguf` weight_format on an engine whose supported_weight_formats
lacks `gguf` (structural axis; matches the `gguf` token only, so a derive's
raw dtype spelling bf16/float16 is never false-rejected).
3. Won't-fit size advisory. The eligibility no-fit-model abort for a hybrid/MoE
derive now appends (a) an actionable NOTE pointing at the curated-swap path +
docs/BRING_YOUR_OWN.md, and (b) a coarse weights-only VRAM verdict: when the
raw weights exceed the detected topology's total VRAM they won't fit at ANY
KV, so say so concretely (the huihui abliterated bf16 ~54 GB vs 2×24 GB case)
instead of a generic stop. `_weights_oversize_advisory()` is pure/total —
empty when it fits / size unknown / headless.
Docs + tests:
- BRING_YOUR_OWN.md: new section C — "Swap a curated model for a fine-tune /
abliterated variant -> reuse its compose" (artifact↔engine + quant + MTP
caveats, worked example).
- test-pullgate-gates.sh: wrapper-arch ALIAS [C0] case + resolve_arch_from_config()
unit + GGUF-on-vLLM runtime-incompatible + no-false-positive control +
_weights_oversize_advisory() unit (oversize / fits / headless / malformed).
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Introduce a new pre-experimental status tier, `incubating`, for composes that
work but aren't ready for the actionable list — niche specialists or models
that fail the standard functional gate by design. Incubating composes are
HIDDEN from `switch.sh --list` by default (revealed by `--list --all`) and
launch-gated behind `--force` (non-functional), so a half-validated model is
catalogued and discoverable without cluttering the recommended set.
Tier wiring (reuses the existing status plumbing — no new emit column):
- compose_registry.py: `incubating` in STATUS_VALUES + `🐣` in COMPOSE_STATUS_EMOJI
(stays out of FUNCTIONAL_STATUSES → --force-gated, never auto-defaulted)
- switch.sh: both --list filters skip incubating unless --all, with a
`(+N incubating hidden — --all)` header note
- AGENTS.md: Status enum row + Caveats-required references
- docs/ADDING_MODELS.md: new rule — NEW MODELS START at 🐣 Incubating, promote
up the enum (🐣 → 🧪 → ⚠️/✅) as they earn the actionable list
First occupant — VibeThinker-3B (WeiboAI, Qwen2 dense reasoning fine-tune):
- `vllm/vibethinker-3b-single` — bf16 weights + fp8_e5m2 KV, single 3090,
full 131072 ctx, mem_util 0.40 (~9.8 GB, single-concurrency sized),
--reasoning-parser qwen3, no tool-calling
- Live-validated 2026-06-16: serves clean correct reasoning/code (~110 TPS),
qwen3 parser splits <think>. fp8 WEIGHTS rejected (break the quant-sensitive
3B: non-terminating empty output). Always-reasoning + no-tools → fails
verify-full's fixed-small-budget checks (5/9) by design → incubating, not gate-passing.
Guard suite 47/47 green (nex-n2-mini parked aside; it's an unrelated local experiment).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Distills the #350 Deckard-40B post into a reusable 'we shipped X' skeleton
that wraps a Results Card (intro+credits / Results Card / getting it / run it
/ what'd help / credits). References RESULTS_CARD.md for the measurement panel
rather than duplicating it; wired into the docs/README.md reference index.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
adds an optional `revision:` key per weights variant. weights.py emits
WEIGHT_REVISION; setup.sh threads it into `hf download --revision` and
pins the post-download sha-verify etag lookup to the same revision, so a
stale pin can't false-fail against a newer HEAD. preflight's manual hint
mirrors the flag. unset = track HEAD, so behavior is unchanged for every
current entry (nothing sets revision: today).
this is the weights half of #316: upstream quant repos re-quant
silently, and we had no lever to pin the bytes a BENCHMARKS row was
measured against. engine images already pin; weights didn't. mechanism
only, no real entry is pinned in this PR (that's a per-entry,
rig-validated call that's yours to make).
refs #319, #316
The streaming sweep for the qwen3_xml PR (#400) surfaced the real picture:
- #39056 auto+thinking+streaming is fixed natively on v0.22.0; the qwen3_coder→
qwen3_xml swap is a no-op (byte-identical A/B, both parsers) → PR #400 closed.
- The residual #145 failure is tool_choice=required + thinking + streaming, and
it's MTP-gated: confirmed 2026-06-13 (scripts/stream-toolcall-probe.py) MTP n=3
dropped 13/20, no-MTP clean 0/20, parser-independent. That's #39598 (MTP
streaming early-return) resurfacing — its Genesis P64 mitigation was retired
with Genesis (#182/#254), so the drop shipped silently back into the dual
composes (benchlocal is non-streaming; verify-full's streaming check has no
tools, so neither caught it).
Updated both rows; mitigation = tool_choice=auto (clean w/ MTP) or no-MTP;
durable fix is upstream (#45413 is parser-side, likely doesn't cover the
spec-decode-streaming path).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The rolling ghcr.io/ikawrakow/ik-llama-cpp:cu13-server tag moved (2026-05-23
-> 2026-06-10, digest 5f914f1c) and the new build REJECTS the legacy
speculative-decode flags: "legacy speculative option '--multi-token-prediction'
is disabled; use --spec-type mtp:n_max=1,p_min=0.0". Every ik-llama MTP compose
crash-restart-loops on a fresh pull (reported by furfix, Discord — single 3090
WSL). The binary's own --help confirms `-mtp`/`--draft-*`/`--spec-stage` are all
rejected in favour of the unified `--spec-type SPEC[:k=v,...]`.
Flag migration (13 composes):
- 11 MTP composes: --multi-token-prediction + --draft-max N + --draft-p-min P
-> --spec-type mtp:n_max=N,p_min=P (per-compose N preserved).
- 2 two-stage composes: the ngram-mod + mtp cascade used the rejected
--spec-stage with legacy keys -> --spec-type ngram-mod:...,ngram_size_n=... +
--spec-type mtp:...,p_min=... (spec-ngram-size-n -> ngram_size_n,
draft-p-min -> p_min, the canonical keys).
Digest-pin (decided to stop the recurrence — this is the 2nd flag churn from
the moving tag; ik-llama has no engine profile, so the per-compose image:
default is the only pin point): :cu13-server -> @sha256:5f914f1c... in all 13
ik composes + the preflight test fixture + 3 docs (IK_LLAMA / SINGLE_CARD /
INFERENCE_ENGINES). VLLM-style central injection isn't available for ik-llama;
a follow-up could add an engine profile so future pins are one-line.
Live-validated on the pinned image (single 3090):
- ik-llama/iq4ks-mtp: healthy, 0 restarts, no legacy error, MTP context ready,
served a completion, draft acceptance 0.62 (26/42).
- ik-llama/iq4ks-two-stage: healthy, 0 restarts, both stages load
(ngram_mod n=16 + MTP context ready), speculative decoding initialized.
Full test suite green (test-compose-registry-disk pre-existing on an unrelated
untracked model dir). The 3 untracked nex-n2-mini ik composes also carry the
rolling tag but are out of scope here (in-progress catalog work).
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
