v0.26.0 (comfyanonymous/ComfyUI #14589) adds native LOCAL Krea2 support —
the earlier "cloud-only, dropped" verdict was pin-specific (cb9f6394
predated it; only the cloud Krea2ImageNode existed → local load failed
"Could not detect model type").
- entrypoint.sh: COMFYUI_REF default cb9f6394 -> f6c162dd (v0.26.0); reword
the pin comment (Krea2 + Qwen3-VL text-gen; re-validate ALL lanes on bump).
- docker-compose.yml: pin comment v0.26.0 + Krea2.
- download_krea.sh (new): fetches the 3 Comfy-Org/Krea-2 assets (turbo fp8
DiT + Qwen3-VL-4B encoder + Qwen-Image VAE) into the ComfyUI models tree.
- download_studio_models.sh + studio-models.tsv: add Krea as an image-lane
model (roster <-> manifest kept mirrored).
NOT yet merge-ready: the v0.26.0 pin is a load-bearing bump — the custom
HiDream-O1 node, ComfyUI-GGUF, and DisTorch multi-GPU all ride it. On-rig
re-validation of every studio lane (Krea renders + the existing 9 still
work) is the merge gate. The OWUI lane graph (krea2.json workflow +
studio_pipe entry) is authored at that validation step (the Krea sampler /
Qwen3-VL encode graph differs from Z-Image — not clonable blind).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
`auto` mode previously only detected NVLink — PCIe P2P (a patched consumer-GPU driver:
tinygrad/geohot/aikitoria) required setting NVLINK_MODE=pcie_p2p by hand. Now, after the
NVLink check comes up empty, `auto` probes `nvidia-smi topo -p2p r`; if P2P is reported OK
between ALL pairs, it auto-promotes to PCIe P2P (custom all-reduce ON, NCCL_P2P_LEVEL=PHB).
So any user on a P2P-capable layout (shared root complex / switch) + a patched driver gets
it automatically.
- New `_pcie_p2p_available()` helper: parses the p2p matrix, skips header/legend rows (a data
row carries the self-"X"), requires every off-diagonal cell == OK (full P2P; partial → off).
- `pcie_p2p` stays as a FORCE override (assert the patch is loaded, bypass the probe).
- No-op for stock GeForce drivers (software-disable P2P → "CNS") and cards on separate root
complexes — both correctly stay off. Verified on this 2× 3090 rig: `auto` still resolves
"P2P DISABLED" (CNS).
- New test-detect-nvlink-autop2p.sh (6 mocked cases: enable on OK, stay off on CNS, NVLink
wins, multi-GPU all-OK vs mixed/none). Existing alloc-conf test unchanged. Gate 58/58.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Make a fresh clone go straight to generating, with the single-command UX the Image
Studio beta had — but for the consolidated image+video+audio studio.
- scripts/setup-ai-studio.sh (NEW, canonical): preflight (docker/gpu/~120 GB disk) →
build ComfyUI image → download_studio_models.sh (full roster) → gpu-mode ai-studio →
install the OWUI Studio pipe → onboarding. Flags: --yes / SKIP_BUILD / SKIP_DOWNLOAD / SKIP_PIPE.
- push-pipe-to-owui.sh now **UPSERTs**: installs the OWUI `function` row if absent (the
one previously-manual step — paste into Admin → Functions), else updates it. First-time
install just needs an OWUI admin account to exist. Validated: INSERT against a DB copy
(clean row + valid meta JSON), UPDATE live on the running OWUI.
- Retire the now-misnamed setup-image-studio.sh / setup-video-studio.sh → thin redirects to
setup-ai-studio.sh (they only pulled one modality post-consolidation and still echoed the
removed `gpu-mode image-studio`/`video-studio` scenes).
- Docs: services/studio/README.md (install is automated) + docs/ai-studio/README.md "Bring it up"
now leads with the one-command path.
Gate: scripts/tests 57/57; setup scripts syntax-clean; no test references the old script bodies.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Wire the two new lanes into the install/preflight surfaces so a fresh rig fetches them
and c3 / gpu-mode know they're expected:
- download_zimage.sh — Z-Image-Turbo fp8 + Qwen3-4B encoder + flux ae VAE (~12 GB).
- download_wan.sh — Wan2.2-Rapid Mega NSFW Q8 GGUF + umt5 encoder + Wan 2.1 VAE (~25 GB).
- download_studio_models.sh ("grab all") calls both (idempotent — skips what's present).
- studio-models.tsv: +image/Z-Image, +video/Wan2.2 rows (the shared manifest read by both
c3's StudioModel loader and gpu-mode preflight). Roster note records Krea2 as dropped.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Replace the separate `image-studio` / `video-studio` / `comfyui` scenes with
a single `ai-studio` scene: ComfyUI holds both GPUs always, and image / video /
audio are picked as *lanes* in OWUI rather than gpu-mode switches. Removes the
gemma-12b chat brain from the studio bundle (kept on disk, just unwired) and
points OWUI's DEFAULT_MODELS at the qwen director.
- gpu-mode.sh: `mode_ai_studio` replaces the two studio modes; drop the gemma
start/stop + the GPU0-pin split; preflight now reads the shared manifest.
- scripts/lib/studio-models.tsv: single source-of-truth model manifest (modality,
label, root, rel_path, size, installer) consumed by both gpu-mode preflight and
the c3 cockpit, so the two can't drift.
- download_{director,video,audio,ace_step,stable_audio,studio}_models.sh: idempotent
fetchers; download_studio_models.sh orchestrates "grab everything".
- setup-video-studio.sh + repointed setup-image-studio.sh → `gpu-mode ai-studio`.
- litellm: drop the gemma-4-12b :8069 route; comfyui compose/entrypoint GPU-pin
comments corrected (no scene sets the pin now).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
gpu-mode only STARTS the studio bundle; the models are fetched by
setup-image-studio.sh. So `gpu-mode image-studio` (or the cockpit, which calls
gpu-mode) on a box that skipped the installer booted broken services. Gate both
paths on the models being present.
- gpu-mode: preflight_studio_models() runs at the top of mode_image_studio /
mode_video_studio — checks the director GGUF (both) + the Ideogram set (image),
and on a miss prints `Fix: bash scripts/setup-image-studio.sh` and refuses to
start, instead of tearing down the LLMs then booting weightless containers.
Mirrors preflight_compose_deps; skip with STUDIO_NO_PREFLIGHT=1.
- cockpit: the studio guard now checks studio_ready() = comfyui-local image AND
the director GGUF on disk (was image-only), so it catches a missing director
BEFORE the scene-switch confirm. Cached flag renamed _comfyui_ready →
_studio_ready; STUDIO_DIRECTOR_REL kept in lock-step with the setup download +
the gpu-mode preflight.
Full cockpit suite 706 passed; bash gates 56/0.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01PoXV1FuKmTiDPB15qQvLjB
The studio sidecar images build on scene start, but their MODELS were never
downloaded by any script — so a fresh user's studio-director (the 🖼️ prompt
crafter, in both image- and video-studio), studio-tts, and studio-step-voice
booted with no weights. Add the downloads, all repo-verified:
- setup-image-studio.sh now also fetches the director GGUF + mmproj
(HauhauCS/Qwen3.5-4B-Uncensored-HauhauCS-Aggressive — commit matched the
on-disk metadata) into MODEL_DIR/qwen3.5-4b-gguf/…, closing the image-studio
🖼️ button gap.
- services/comfyui/download_kokoro.sh — kokoro-v1.0.onnx + voices-v1.0.bin from
the kokoro-onnx release assets (the format studio-tts loads).
- services/comfyui/download_step_audio.sh — stepfun-ai/Step-Audio-EditX +
Step-Audio-Tokenizer for studio-step-voice.
Paths mirror each sidecar compose's ${…_DIR} default. bash gates 56/0.
NOT included: the LTX-2.3 / Sulphur video-model set (video-studio) — ~12+ files
across several uploaders; needs a per-file source audit, tracked separately.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01PoXV1FuKmTiDPB15qQvLjB
Four changes from review:
1. comfyui shows engine "comfyui" (not the generic "studio") — both running
(_docker_ps_stack_containers) and stopped (_merge_known_services); other GPU
rig services / studio stacks stay "studio".
2. Orchestration: remove the bottom Power-cap strip — the cap already shows on the
GPU cards at the top ("(cap NNNW)"), so the strip duplicated it. populate_power_cap
keeps caching the per-GPU cap for the cards; the [c] menu still adjusts it.
3. gpu-mode: remove the standalone `comfyui` scene (mode_comfyui + dispatch + TSV
row + usage help) — comfyui runs via image-studio / video-studio now. test EXPECT
+ cockpit _STUDIO_SCENES updated.
4. Containers now lists the STOPPED studio sidecars (studio-director/-gallery/
-orchestrator/-image-shim/-tts) + comfyui: _stopped_studio_containers reads
`docker container ls -a` for scene-managed service/stack containers that exist but
aren't running and aren't top-level services/ dirs (engines excluded as noise).
[s] on them uses service_start_or_restart — compose-up for a real service dir,
`docker restart` for an existing sidecar (its dir name ≠ container name).
Full cockpit suite 702 passed; bash gates 56/0.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01PoXV1FuKmTiDPB15qQvLjB
Ollama was dropped from serving 2026-05-10 (Qwen/Gemma route through LiteLLM
directly), but its services/ollama/ compose dir was kept "for manual spin-up" —
so the cockpit Containers tab (which lists every services/*/docker-compose.yml)
surfaced it as a stopped service that nothing in the stack starts. Remove it.
- Delete services/ollama/.
- gpu-mode.sh: drop the 9 defensive `stop_service ollama` calls, the :11434
status readout, and fix the stale `chat` usage-help line (it listed Ollama +
omitted SearXNG; the authoritative scene TSV was already correct).
- services/openwebui: drop the dead OLLAMA_BASE_URL env; set ENABLE_OLLAMA_API
=false so Open WebUI doesn't probe a now-dead :11434 on boot.
- cockpit services.py: drop ollama from the supporting-service docstring/comment
examples (it's no longer a known service).
The SERVICES=() array already excluded ollama, so no scene behaviour changes.
bash gates 56/0; cockpit container/known-service tests green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01PoXV1FuKmTiDPB15qQvLjB
gemma-int8 was a literal alias for gemma (same INT8 PTH container :8032), and
gemma-mtp (the bf16/32K/no-overlay fallback) isn't needed as a scene. Keep only
the 'gemma' scene (INT8 PTH KV, 262K, vision, :8032 — mode_gemma_int8).
Removed: the gemma-int8 + gemma-mtp TSV rows, their dispatch cases + help lines,
mode_gemma (the bf16 fallback mode) and start_gemma_mtp. Kept stop_gemma_mtp /
stop_gemma_int8 / stop_all_gemma — mode_off + the other gemma modes call them to
defensively tear down a stray bf16/int8 gemma container (the bf16 compose stays
serveable via its catalog slug).
test-gpu-mode-list EXPECT trimmed; repo gates 56/0. Cockpit untouched (the scene
table is data-driven — the dropped scenes just disappear from --list-modes).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Scene-catalog cleanup for the Orchestration tab (maintainer-directed):
- Regroup: 'serving' group → 'models'; 'chat' moves serving→ops (browser-chat
infra, no GPU model); 'diffusiongemma' moves studio→models (it IS a model).
Groups are now models / studio / ops. gpu-mode's plain + --json renders are
data-driven, so this is a TSV edit + the contract comment.
- Remove 'bigmodel': it was ~redundant with 'off' (both stop everything). Its one
useful extra — the freed VRAM/RAM readout — is folded into mode_off; the cache
drop was marginal (Linux auto-reclaims page cache; nothing for VRAM) and the
llama-server hint belongs in docs.
- Remove the 'diffusiongemma'/'dgemma' SCENE (TSV/dispatch/help/mode_/start_): a
niche dLLM whose scene duplicated the catalog slug vllm/diffusiongemma-dual (the
way to serve it). stop_diffusiongemma is KEPT — mode_off + the studio modes
call it to defensively clear a stray dgemma container.
- Cockpit scene filter switched from "ops-except-off" to a deny-list of the COMMAND
modes (power-cap / prune / prune-all) so 'chat' (now ops) still shows; off/chat +
models/studio scenes appear, the command-modes don't.
Updated test-gpu-mode-list EXPECT/GROUPS + the cockpit filter test. Repo gates
56/0; cockpit suite 673 (+1 skip) unchanged (cockpit code untouched by the
gpu-mode.sh scene removals).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Reworks the Orchestration tab's control set per maintainer direction:
- Power cap → a MENU ([c] opens PowerCapMenuScreen) instead of a stateful on/off
toggle: [d] apply default cap (230W) · [u] clear cap (uncap to hw default) ·
[m] custom wattage (type a value). Each choice routes through the standard
confirm gate. gpu-mode.sh power-cap now ALSO accepts a numeric wattage (applied
to both cards, validated against each card's [min,max]); power_cap_set(<W>)
builds it. (The menu blurs its Input on open + uses priority shortcut bindings
so d/u/m reach the screen — a focused Input otherwise swallows printable keys.)
- Prune / prune-all REMOVED from the cockpit (was Orchestration-only): binding,
action, palette entry, help/footer text, the prune() service method + its tests.
`gpu-mode prune[-all]` stays on the CLI for manual cleanup.
- Power-cap SWEEP [w] moved Orchestration → Doctor (a tuning/diagnostic bench, not
a live-estate control): context-gate → tab-doctor, action guard updated, surfaced
in the Doctor hint + help.
- Help/docstrings refreshed (Orchestration + the stale Doctor help line now lists
the real v/V/R/F verbs + the relocated [w]).
Scene start/stop path unchanged (⏎ → reconcile-gated scene_switch → gpu-mode
<mode>). +3 power-cap tests (menu→default/clear/custom, cap-sweep on Doctor),
power_cap_set custom-wattage + reject-invalid tests; per-tab check_action updated.
Suite: serve-cockpit 670 (+1 skip), tui-core 66.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A catalog slug's compose can mount a SEPARATE weight artifact beyond its core
weights — a DFlash draft model or an mmproj vision projector, from a different
subdir. The cockpit Download fetched only the core, so those 8 slugs (5 DFlash
+ 3 GGUF-vision) would download, read "present", offer Start, then fail to boot
for the missing companion.
The registry is the single source of truth: the cockpit gets every catalog slug
from it, so the slug's required artifacts come from it too.
- compose_registry.py: new per-slug `weights_companions` field — the extra
weight-variant keys a slug needs beyond `weights_variant`. Set on the 5 DFlash
slugs (-> anbeeld-dflash-iq4xs) + 3 vision slugs (-> gguf_mmproj_f16). Per-slug,
so a text slug sharing a vision GGUF variant (e.g. llamacpp/default on
unsloth-q4km) does NOT over-fetch the ~1GB mmproj.
- registry-emit --json: emit `weights_companions` + `drafter` + `vision` (vision
derived from the vision-coding workload). drafter/vision also enable a future
catalog badge. test-registry-json contract updated.
- setup.sh: read WEIGHT_EXTRA_KEYS (a <model>:<variant> list) and fetch those
alongside the core via the existing companion-download loop + per-file SHA
verify. setup.sh stays the weights-layer puller (model profiles); it does NOT
depend on the slug registry, so the historical pre-registry "download + test a
new model before it has a slug" path is untouched.
- cockpit: run_weights_download passes entry.weights_companions as
WEIGHT_EXTRA_KEYS (model-qualified); weights_state is companion-aware (a slug
is PARTIAL, Start gated + Download offered, until BOTH core and companions are
on disk); CatalogEntry gains weights_companions / drafter / vision (attached to
the row like `source`, no shared-core schema change).
+5 tests (row-attach, companion env injection, companion-aware state, setup.sh
WEIGHT_EXTRA_KEYS, registry-json contract). Suite: tui-core 66, serve-cockpit
665 (+1 skip), repo gates 56/0.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The serve-cockpit Download action runs `WEIGHT_KEY=<model>:<variant> bash
scripts/setup.sh <model>` to fetch an exact catalog entry. But the friendly
model-name dispatch (`case "${MODEL_NAME}"`) hard-`exit 1`s on any model outside
the five flagship families — BEFORE the WEIGHT_KEY override a few lines down.
So a download of any other catalog model (e.g. llamacpp/vibethinker-3b-single)
died instantly with "unsupported model", never fetched a byte, and the cockpit
then reported "download incomplete (partial) — check disk / HF_TOKEN".
WEIGHT_KEY is documented as the authoritative "exact catalog entry" fetch-now
flow: the recipe fully specifies <model>:<variant> and load_weight_recipe()
already validates that the key's model matches MODEL_NAME. So when WEIGHT_KEY
is set, the unknown-model `*)` branch is now a no-op (the override below sets
PRIMARY_WEIGHT_KEY); the friendly guard still fails fast for a typo'd / truly
unsupported positional when WEIGHT_KEY is absent.
Validated: `WEIGHT_KEY=vibethinker-3b:prithivmlmods-q8 SKIP_MODEL=1 setup.sh
vibethinker-3b` now resolves the recipe (prithivMLmods/VibeThinker-3B-GGUF →
vibethinker-3b-gguf/prithivmlmods-q8) and reaches the download; an unknown
model with no WEIGHT_KEY still errors. +regression guard in test-setup-picker.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Fill in the HuggingFace source repo for five weight variants that had no direct
download recipe (were "manual"), each verified live against the HF API:
- gemma-4-31b:bf16 → google/gemma-4-31B-it
- qwen3.6-27b:awq → cyankiwi/Qwen3.6-27B-AWQ-INT4
- qwen3.6-35b-a3b:gptq_int4 → palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4
- qwen3.6-35b-a3b:dflash → z-lab/Qwen3.6-35B-A3B-DFlash
- qwen3.6-35b-a3b:dflash_gguf → abhinand/Qwen3.6-35B-A3B-DFlash-GGUF
These are now fetchable via `WEIGHT_KEY=<model>:<variant> bash scripts/setup.sh
<model>` (and, once a compose adopts them, one-click in the c3 Download UX).
Weights-registry tests green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Batch-emit the static weights metadata for every (model, variant) with a
local_subdir — subdir / hf_repo / size_gb / verify_glob / status / kind — as a
single JSON array. The serve-cockpit TUI shells this once to learn where each
slug's weights live + how to fetch them, then stats the dirs itself against its
own (user-configurable) model dir for the download-vs-serve state. Purely
additive: a new subcommand, no FS access, `entry`/`lookup` unchanged.
Foundation for the upcoming Download UX (Download-vs-Start, listing progress).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
From the live-rig UX audit:
- A6 Catalog "fits" respects LIVE free-VRAM, not an empty card: a row whose vram_est
exceeds live per-GPU free-VRAM (from the estate poll) is downgraded ●→⚠/✗ with a
reason; with no live data the column reads "(vs empty card)". Never fabricates a
free number; the live-serving model's own row is exempt (its VRAM is counted as
used, so it is provably fitting).
- A7 Serving panel shows the REAL running config: probes /v1/models max_model_len
(running ctx) + docker inspect image, and badges "config differs from catalog slug
<slug>" only when the probe diverges from the slug's CONFIGURED ctx (registry
max_ctx, exact int) — not the kv-calc fit ceiling. Registry-derived fields stay
labelled "(per catalog slug)".
- A4 Targeted serving verbs: [k] stop / [b] restart resolve the serving container by
matched_slug and route through the existing confirm/reconcile gate (vs [o] stop-ALL
which kills co-resident services); [n] switch jumps to Run·Catalog.
- #4/N7 Doctor: [y] re-runs the (read-only) doctor read; the full-validation report is
surfaced from Doctor; clear issues offer the obvious next action.
- A9 ③ Gate ladder shows each step's last outcome (·/⟳/✓/✗) from the run's exit.
Threads a new configured_ctx field through the registry --json contract
(registry-emit.sh → VariantRow → CatalogEntry) so the A7 badge compares the
configured ctx; unifies the K-label convention on registry-emit's ÷1000 so the
running-ctx and catalog labels match. All registry-parity gates pass.
Verified by an implement→gate→repair→verify→critic workflow; folded the findings: the
serving model's own row no longer false-"✗ won't fit"; the divergence badge compares
configured vs probed (not the fit ceiling) so an honest 262K serve on the 295K-fit
qwen doesn't false-badge; the ladder glyph awaits run completion before resolving
⟳→✓/✗ (was stuck at ⟳).
Suite green: cockpit 444; registry-json / switch- / launch-parity / compose-disk gates pass.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Dual composes inject PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,...
for the PCIe (custom-all-reduce OFF) path. On NVLink / patched-PCIe-P2P
rigs detect_nvlink.sh turns custom all-reduce ON, where expandable_segments
backs allocations with a cuMemMap VA range that cudaIpcGetMemHandle rejects
during graph-buffer registration -> custom_all_reduce.cuh:455 'invalid
argument', worker dies at boot.
The prior fix clobbered PYTORCH_CUDA_ALLOC_CONF to max_split_size_mb:512,
discarding any other knob the user set (custom max_split_size_mb,
garbage_collection_threshold, ...). Instead strip ONLY the expandable_segments
token on the P2P path and keep the rest; fall back to max_split_size_mb:512
when nothing remains. The PCIe path keeps expandable_segments untouched.
Strengthen test-detect-nvlink-alloc-conf.sh to assert the surgical contract
(preserve neighbors, strip in any position, no dangling commas) and add the
vllm#42609 row to docs/UPSTREAM.md (its "DP>1 AND TP>1" title under-scopes
the bug; the same cuh:455 crash fires TP-only -- cf. vllm#42583 / vllm#45604).
Validated on a 2x RTX 3090 NVLink rig (TP=2, NV4): before, boot crashes at
custom_all_reduce.cuh:455 'invalid argument'; after, the dual stack boots
clean (custom AR ON, expandable_segments stripped), /v1/models ready, serves
a completion.
Co-authored-by: Cursor <[email protected]>
Dual composes inject PYTORCH_CUDA_ALLOC_CONF with expandable_segments:True for the PCIe path. detect_nvlink.sh used \, which kept that value on NVLink rigs and crashed custom all-reduce at CUDA graph profiling (vllm#42609). Assign max_split_size_mb:512 unconditionally when P2P is on.
Co-authored-by: Cursor <[email protected]>
Generate a per-instance Compose override for estate boots so CUDA_VISIBLE_DEVICES and NVIDIA_VISIBLE_DEVICES are set inside each container. This prevents parallel vLLM estate instances from both binding to physical GPUs 0/1 when Docker exposes all reserved devices.\n\nValidated with test-diagnose-estate, test-estate-json, test-profiles-compat, test-launch-compat, and a live 4x3090 estate boot.
--fit-all --card <c> --json walks COMPOSE_REGISTRY once and emits
{card, card_vram_gb, variants:{slug: verdict}} — the same per-slug verdict
as --fit (skip for non-vLLM kvcalc_key=SKIP), in a SINGLE process. Measured:
all 53 slugs in 0.22s vs 53× per-slug --fit (~11s serial). Backs the cockpit
catalog's fit column without N subprocess spawns. test-kv-calc-fit.sh gains
section (e): shape + per-slug equivalence + skip handling.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
scripts/tests/test-compose-nvlink-escape.sh asserts no shipped compose
carries an un-escaped runtime token in a `bash -c` entrypoint —
${_NVLINK_ENABLED}, a bash ${VAR[@]} array, or $@ / ${VLLM_ENFORCE_EAGER:+}
in a detect_nvlink entrypoint. Compose v5.1+ interpolates entrypoint string
values at parse time, so an un-escaped ${_NVLINK_ENABLED:-0} is baked to its
default before the container runs → the NVLink branch dies (always PCIe).
This was live on 9 composes until #436 (lmcache via #429/#433); the guard
prevents it regressing.
Scans 54 composes (13 detect_nvlink entrypoints); negative-tested to fire on
the un-escaped form. Full guard suite 55/55.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Data-layer contracts the cockpit (and any jq user) consumes — all strictly
additive (existing human output byte-identical), full guard suite green (54/54):
- registry-emit.sh --json : {variants,defaults,profiles{engines,models,hardware,drafters}}
- tools/kv-calc.py --fit <slug|model> --card <gpu> --json : structured fit verdict
- gpu-mode.sh --list-modes [--json] : scene catalog (serving/studio/ops)
- estate_cli.py report-state/diagnose --json : structured estate read
- pull.sh --profile-like --dry-run --json : structured swap_path (not a message blob)
- health.sh CONTAINER= : Doctor probes any engine container (was qwen36-27b-hardcoded)
- switch.sh --explain <slug> [--json] : joined registry/engine/model/hw/drafter + fit + bench
Built + adversarially reviewed via workflow. The review caught a real
switch<->kv-calc seam defect (switch fed hyphenated 'rtx-3090', kv-calc matched
only 'rtx3090' -> fit silently 'unavailable' on the 3090 rig); fixed: kv-calc
accepts hyphenated hardware-profile ids + hyphen-strip fallback; switch surfaces
kv-calc's structured verdict regardless of RC and renders its real keys; both
tests now exercise the seam. No shared-module edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A lazydocker-style terminal UI for the stack's test scripts (verify / bench /
verify-stress-NIAH / quality / soak / rebench-full): auto-detects the running
model + endpoint, injects MODEL=/URL=, streams live progress, and browses the
results/ history. Built by Qwen Max from docs/club3090-test-tui-prompt.md.
- scripts/c3t — launcher that derives the repo root from its own location and
runs the tool from tools/test-console's own venv (no global installs).
- tools/test-console/ — Python + Textual package, pinned pyproject + uv.lock,
and an 83-test offline pytest suite (parsers vs fixtures, mocked docker /
/v1/models detection, BENCH_MOCK bench parse) — all green, no GPU required.
Fixed before merge: __main__.py and a test fixture hardcoded the rig path
/opt/ai/github/club-3090 — now derived from __file__ (parents[3]; override via
C3T_REPO_ROOT) so it works from any clone, not just this rig. .gitignore
excludes .venv / caches.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
Full rebench-full (--with-8pack-thinking=both) completed 2026-06-18 for
llamacpp/qwen27b-pi-reasoning-single. Replaces the earlier inferred "≈ base
50/59 @370W" note (+ the PENDING-ladder caveats) with the measured numbers, and
adds the BENCHMARKS.md single-card row.
- Bench @370W: 47.9/55.3 decode narr/code (n=5, CV<2%); @230W cap 28.5/32.9.
- verify-stress 8/8 (NIAH->183K), 8-pack 104/150 off / 106/150 on, soak PASS
(0 MiB growth, 0/100 silent-empty, p50 54.5, 102% retention).
- MTP head NOT weaker than base: matched-power A/B dead-even (73% vs 72% accept);
47.9/55.3 @370W ~on par with base 50.3/58.9. Power-sensitive (mainline -42%
370->230W) — the earlier "slow" 28/33 was the rig's 230W cap, not the model.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
benchlocal-cli already supports --max-tokens (overrides the per-pack ~1024
completion budget for BOTH arms, #28), but quality-test.sh only forwarded
--thinking-max-tokens. So a VERBOSE model that self-truncates the deterministic
packs (finish_reason=length before its final ANSWER:/</solution> line) could not
be benched at a higher budget through our wrapper.
Surfaced on llamacpp/qwen27b-pi-reasoning (a reasoning fine-tune that reasons in
visible content even thinking-off): ~5-10 reasonmath/bugfind/cli misses in the
thinking-OFF 8-pack were truncations, not wrong answers (e.g. RM-05 had 4/5
checkpoints right but ran out of tokens). The thinking-ON pass (16384 budget)
did not truncate — so the gap is purely the deterministic budget, which had no
knob in our wrapper until now.
- quality-test.sh: add --max-tokens N + MAX_TOKENS env passthrough (mirrors
--thinking-max-tokens exactly: same int validation, help, ENV doc, forward + echo).
- rebench-full.sh: plumb MAX_TOKENS into BOTH 8-pack passes (off + on) + doc it.
- test-quality-thinking.sh: assert env + flag forms forward --max-tokens and that
a non-integer is rejected (mock-benchlocal harness, hermetic).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
The merged llamacpp/qwen27b-pi-reasoning-single docs claimed its embedded MTP
head was "~45% below base / weaker" — that was a power-cap measurement artifact,
not a model property. A matched-power A/B (230W, same engine/KV/n=2/prompt) shows
the head performs IDENTICALLY to the base Qwen3.6-27B MTP head: 73% vs 72% draft
acceptance, 41.2 vs 41.2 t/s. The base llamacpp/default 50.3/58.9 figure I
compared against is a 370W BENCHMARKS number; mainline is -42% from 370->230W, so
the comparison was only valid at matched power (cf. the ik-llama/iq4ks-mtp
BENCHMARKS row, which documents exactly this trap). The 28.5/33.4 decode numbers
are correct AS 230W measurements; at 370W this config matches base's 50/59.
Corrected the status_note, compose header, and weights manual_note.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Three related [C0]/eligibility pull-gate fixes for the curated-swap surface —
an uncurated derive (abliterated / fine-tune) of a model we already serve.
1. Wrapper-arch alias. `pull.sh --profile-like` was false-aborting at [C0] with
"no arch_patches matrix row for 'Qwen3_5ForConditionalGeneration'". That is
the OUTER multimodal wrapper class the weights report; the patch matrix is
keyed on the inner canonical `Qwen3NextForCausalLM`. arch_patches.yml is a
closed key-set, so the alias lives in the editable arch_model_xref.
- profile_runtime.yml: `config_architectures: [Qwen3_5ForConditionalGeneration]`
on the Qwen3NextForCausalLM xref entry.
- generate_compose.py: `resolve_arch_from_config()` maps a config.json
architectures[0] string -> (canonical_arch, arch_row) via that alias.
- gates.py [C0]: resolve the wrapper arch via the alias before declaring
NO_ARCH_ROW. The hybrid now reports ENGINE_SUPPORTED.
2. GGUF axis. `supported_weight_formats` was declared on every engine but never
enforced (only `kv_format` was). The deriver blocks GGUF on the derive path,
but the curated registry / curated-swap path had no such guard. gates.py [C0]
now rejects a `gguf` weight_format on an engine whose supported_weight_formats
lacks `gguf` (structural axis; matches the `gguf` token only, so a derive's
raw dtype spelling bf16/float16 is never false-rejected).
3. Won't-fit size advisory. The eligibility no-fit-model abort for a hybrid/MoE
derive now appends (a) an actionable NOTE pointing at the curated-swap path +
docs/BRING_YOUR_OWN.md, and (b) a coarse weights-only VRAM verdict: when the
raw weights exceed the detected topology's total VRAM they won't fit at ANY
KV, so say so concretely (the huihui abliterated bf16 ~54 GB vs 2×24 GB case)
instead of a generic stop. `_weights_oversize_advisory()` is pure/total —
empty when it fits / size unknown / headless.
Docs + tests:
- BRING_YOUR_OWN.md: new section C — "Swap a curated model for a fine-tune /
abliterated variant -> reuse its compose" (artifact↔engine + quant + MTP
caveats, worked example).
- test-pullgate-gates.sh: wrapper-arch ALIAS [C0] case + resolve_arch_from_config()
unit + GGUF-on-vLLM runtime-incompatible + no-false-positive control +
_weights_oversize_advisory() unit (oversize / fits / headless / malformed).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
* LMCache: in-repo L2 default + LMCACHE_L2=1 toggle, preflight disk-check, corrected sizing
- LMCACHE_L2=1 enables the L2 disk tier at a gitignored repo-root lmcache-kv/ (zero
config); LMCACHE_L2_ADAPTER still fully overrides. L2 stays OFF by default.
- preflight_lmcache_ram now also soft-warns on low L2 disk space at the L2 host dir.
- CORRECTED capacity (measured > estimated): the LMCache offload cache is
~131 KB/token MEASURED (lmcache_mp_l1_memory_usage_bytes 4.93 GB + 4.6 GB on L2
disk, 36,808-token session) — ~7x the GPU's 18.9 int8-PTH rate, which only sets
max servable context. Prior docs used 18.9 for capacity and overstated it ~7x:
--l1-size-gb 30 holds ~4 x 50K sessions (not ~33), <1 full 262K.
- Added a RAM/disk-vs-context sizing table to INTERNALS.md.
- Measured L2 rehydrate: 4.8s vs 43s cold re-prefill (~9x), cross-restart persistence
confirmed; use the fs adapter (not nixl_store — this image's NIXL is broken).
Refs #133.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
* LMCache: document fp8 serde is broken on this model (no reduced-size L2 yet)
Measured 2026-06-17: LMCache's fp8 serde shape-errors on this model's KV
('[2,16,1584,260]' invalid — wrong layout for the hybrid GDN/head_dim-256/int8-PTH
source), 60 serialize fails, L2 stays empty. CacheGen-MP is 'coming soon'. So L2 is
uncompressed (~131 KB/token); the fp8-serde toggle was NOT wired (it doesn't work).
Re-test trigger noted. Refs #133.
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
Layers an LMCache tiered persistent prefix-KV cache (MP/HMA connector) onto the
dual-max fidelity profile (FP8 + int8-PTH KV + MTP n=3 @262K). Controlled A/B
shows ZERO decode penalty (74 narr / 94 code TPS == without LMCache, MTP intact
~83% accept); reuses cached prefix KV across long / multi-session workloads
(cold->warm TTFT ~7-8x) instead of re-prefilling.
- New vllm-lmcache engine profile: lmcache/vllm-openai DIGEST-pinned (the tag is
mutable — bundled vLLM 0.22.1-dev then 0.23.1-dev under the same tag).
- preflight_lmcache_ram guard: rejects --l1-size-gb over-allocation even under
--force (the l1=100-on-94GB host-OOM that forced a reboot, #133).
- env-gated L2 disk tier hook (LMCACHE_L2_ADAPTER, off by default).
- INTERNALS.md RAM/disk sizing section; registry entry + count bumps; full
guard suite green.
Incubating (hidden, --force to launch): runs LMCache's third-party image with a
newer vLLM than our v0.22.0 pin; L2 latency unmeasured on-rig; 38 GB on-demand.
Refs #133.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
Code packs run 2026-06-16 (tool-free — sandbox executes the model's code, no
tool-calls): humaneval-plus-30 29/30 (97%), lcb-v6-30 25/30 (83%, 2 losses =
token_limit on the hardest). Substantiates the 'strong one-shot/competition
coding' characterization with our own numbers (matches the authors' ~96% LeetCode).
Folded into the llamacpp compose Quality field + registry status_note.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The performance-max VibeThinker path — prithivMLmods Q8_0 GGUF on mainline
llama.cpp, single 3090, q8_0 KV, full 131K, -b 4096 -ub 2048. A sibling to
the vLLM bf16 compose (#418), and the better one on every axis.
Why it's better than the vLLM sibling:
- ~166 TPS decode (vs ~110 vLLM bf16), ~6.2 GB at full 131K (vs ~9.8 GB),
prefill ~6,630 tok/s (-b 4096/-ub 2048 = +20%), boot ~9s.
- Q8_0 is near-lossless → quality INTACT, where vLLM's fp8 weight-quant broke
this quant-sensitive 3B (non-terminating empty output). KV quant is fine
(storage-only); fp8 *weights* are the problem.
Validated 2026-06-16 (tool-free packs, temp 0.6, thinking-on):
gsm-symbolic-30 100% · instructfollow-15 100% · reasonmath-15 80% ·
structoutput-15 80% · dataextract-15 40%.
dataextract is a genuine extraction ceiling — full temp sweep {0:27%,
0.6:40%, 1.0:27%} can't lift it (value mismatches persist). A reasoning
specialist, not an extractor. No tool-calling. verify-full 5/9 by design.
Wiring: registry entry + dense-family added to llama-cpp-local engine +
prithivmlmods-q8 weights variant + disk/registry counts (51/52). No MTP head
in the GGUF (plain qwen2 conversion) → no spec-dec. Guard suite 47/47.
Status: 🐣 incubating (hidden from switch.sh --list; --force to launch).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Introduce a new pre-experimental status tier, `incubating`, for composes that
work but aren't ready for the actionable list — niche specialists or models
that fail the standard functional gate by design. Incubating composes are
HIDDEN from `switch.sh --list` by default (revealed by `--list --all`) and
launch-gated behind `--force` (non-functional), so a half-validated model is
catalogued and discoverable without cluttering the recommended set.
Tier wiring (reuses the existing status plumbing — no new emit column):
- compose_registry.py: `incubating` in STATUS_VALUES + `🐣` in COMPOSE_STATUS_EMOJI
(stays out of FUNCTIONAL_STATUSES → --force-gated, never auto-defaulted)
- switch.sh: both --list filters skip incubating unless --all, with a
`(+N incubating hidden — --all)` header note
- AGENTS.md: Status enum row + Caveats-required references
- docs/ADDING_MODELS.md: new rule — NEW MODELS START at 🐣 Incubating, promote
up the enum (🐣 → 🧪 → ⚠️/✅) as they earn the actionable list
First occupant — VibeThinker-3B (WeiboAI, Qwen2 dense reasoning fine-tune):
- `vllm/vibethinker-3b-single` — bf16 weights + fp8_e5m2 KV, single 3090,
full 131072 ctx, mem_util 0.40 (~9.8 GB, single-concurrency sized),
--reasoning-parser qwen3, no tool-calling
- Live-validated 2026-06-16: serves clean correct reasoning/code (~110 TPS),
qwen3 parser splits <think>. fp8 WEIGHTS rejected (break the quant-sensitive
3B: non-terminating empty output). Always-reasoning + no-tools → fails
verify-full's fixed-small-budget checks (5/9) by design → incubating, not gate-passing.
Guard suite 47/47 green (nex-n2-mini parked aside; it's an unrelated local experiment).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
soak-test.sh's auto_container() had drifted into a hardcoded model-name
allowlist (qwen36-27b / qwen36-35b-a3b / gemma-4-31b + beellama gemma
variants). Any shipped compose outside that list silently failed auto-detect
with "no running club-3090 container found" — #405 hit it on the shipped
diffusiongemma-26b-a4b dual compose (soak "failed to launch" through no
fault of the contributor's rig).
Re-syncs with the canonical preflight.sh::preflight_autodetect_endpoint
(which this function's own comment claims to mirror): detect by ENGINE-INTERNAL
port mapping (vLLM 8000 / llama.cpp 8080 / sglang 30000), model-agnostic, then
prefer a recognised engine-family prefix. Same bug class as the #310 preflight
fix — now any compose is found regardless of model.
Verified: unit-tested the pipeline against the #405 container
(vllm-diffusiongemma-26b-a4b-fp8-tp2 + its 8020->8000 mapping) plus qwen/
llama-cpp/sglang variants — all detected; a port-80 studio container is
correctly excluded. bash -n clean; full guard suite green.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
First use of the new `revision:` lever (#408). Pins the
morikomorizz/Qwen3.6-35B-A3B-Uncensored-HauhauCS-MTP fetch to commit
49a080db7406cd98f94f0f2a18539bbcc1520444 — the exact bytes the BENCHMARKS
row and #411 Results Card were measured against.
Verified: repo last-modified 2026-06-10 predates our 2026-06-14 download,
and the HF x-linked-etag at this sha equals our local file's sha256
(76a0d4c2...). weights.py round-trips revision: -> WEIGHT_REVISION; guard
suite green. Guards against a silent upstream re-quant (the #316 class).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
adds an optional `revision:` key per weights variant. weights.py emits
WEIGHT_REVISION; setup.sh threads it into `hf download --revision` and
pins the post-download sha-verify etag lookup to the same revision, so a
stale pin can't false-fail against a newer HEAD. preflight's manual hint
mirrors the flag. unset = track HEAD, so behavior is unchanged for every
current entry (nothing sets revision: today).
this is the weights half of #316: upstream quant repos re-quant
silently, and we had no lever to pin the bytes a BENCHMARKS row was
measured against. engine images already pin; weights didn't. mechanism
only, no real entry is pinned in this PR (that's a per-entry,
rig-validated call that's yours to make).
refs #319, #316
adds scripts/tests/test-hf-repos-resolve.sh: an opt-in, network-gated
check that every hf_repo / hf_repos entry in the weights registry
actually resolves on HF, not just string-matches the profile. catches
the renamed / never-existed repo class from #316 that the string-only
guard in test-model-weights-registry.sh can't see.
gated behind CLUB3090_CHECK_HF_REPOS=1 and self-skips otherwise, so the
default `for t in scripts/tests/*.sh` sweep stays offline-safe and green.
follows redirects like `hf download` does, so a rename or recase resolves
and is flagged (not failed); only a real 404 fails. gated 401/403 counts
as exists; network / rate-limit codes are reported but not counted, to
avoid spurious reds.
refs #320, #316
#145 / vLLM #39056 are STREAMING-only tool-call bugs, and our gates were
structurally blind to them: benchlocal-cli runs every pack non-streaming
(benchlocal-cli#68, closed wrong-layer), and verify-full had check_tools
(non-streaming) and check_streaming (no tools) as SEPARATE checks — the bug
lives in their untested intersection (tools × streaming × thinking-on). That's
how the MTP-streaming drop (#39598, un-mitigated since Genesis retired) shipped
silently into the dual composes.
- scripts/verify-full.sh: new [6/9] check_streaming_tools — stream:true + tools
+ tool_choice=auto + enable_thinking=true, reassembles the SSE deltas, and
asserts delta.tool_calls (get_weather) + finish_reason=tool_calls + no
<tool_call> leak into delta.content. Uses tool_choice=auto (the clean path on
v0.22.0); hard-fails on the leak signature (the #145/#39056 class). NB:
tool_choice=required + MTP is a known-open drop (#39598) — the check notes it
but uses auto so it's a green gate on shipped composes. Gated by SKIP_TOOLS.
Renumbered the suite 8 -> 9 checks. Live-validated: vllm/dual all 9 pass.
- scripts/stream-toolcall-probe.py: the deep-sweep instrument behind the #400
investigation (thinking-on, multi-prompt, auto+required, SSE reassembly,
PASS/DROP/OTHER classification, exit-1 on any DROP). It's what proved
qwen3_coder == qwen3_xml and isolated the #39598 MTP-gating. Run it twice
(control vs candidate) for a streaming A/B.
- scripts/tests/test-stream-toolcall-probe.sh: offline mock-SSE gate (clean +
#145-drop modes) — no GPU needed.
Full suite green (test-compose-registry-disk fails pre-existing on an unrelated
untracked model dir).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
The rolling ghcr.io/ikawrakow/ik-llama-cpp:cu13-server tag moved (2026-05-23
-> 2026-06-10, digest 5f914f1c) and the new build REJECTS the legacy
speculative-decode flags: "legacy speculative option '--multi-token-prediction'
is disabled; use --spec-type mtp:n_max=1,p_min=0.0". Every ik-llama MTP compose
crash-restart-loops on a fresh pull (reported by furfix, Discord — single 3090
WSL). The binary's own --help confirms `-mtp`/`--draft-*`/`--spec-stage` are all
rejected in favour of the unified `--spec-type SPEC[:k=v,...]`.
Flag migration (13 composes):
- 11 MTP composes: --multi-token-prediction + --draft-max N + --draft-p-min P
-> --spec-type mtp:n_max=N,p_min=P (per-compose N preserved).
- 2 two-stage composes: the ngram-mod + mtp cascade used the rejected
--spec-stage with legacy keys -> --spec-type ngram-mod:...,ngram_size_n=... +
--spec-type mtp:...,p_min=... (spec-ngram-size-n -> ngram_size_n,
draft-p-min -> p_min, the canonical keys).
Digest-pin (decided to stop the recurrence — this is the 2nd flag churn from
the moving tag; ik-llama has no engine profile, so the per-compose image:
default is the only pin point): :cu13-server -> @sha256:5f914f1c... in all 13
ik composes + the preflight test fixture + 3 docs (IK_LLAMA / SINGLE_CARD /
INFERENCE_ENGINES). VLLM-style central injection isn't available for ik-llama;
a follow-up could add an engine profile so future pins are one-line.
Live-validated on the pinned image (single 3090):
- ik-llama/iq4ks-mtp: healthy, 0 restarts, no legacy error, MTP context ready,
served a completion, draft acceptance 0.62 (26/42).
- ik-llama/iq4ks-two-stage: healthy, 0 restarts, both stages load
(ngram_mod n=16 + MTP context ready), speculative decoding initialized.
Full test suite green (test-compose-registry-disk pre-existing on an unrelated
untracked model dir). The 3 untracked nex-n2-mini ik composes also carry the
rolling tag but are out of scope here (in-progress catalog work).
Co-authored-by: noonghunna <[email protected]>
setup.sh hard-coded GENESIS_PIN=7b9fd319 and the qwen3.6-27b case set
NEEDS_GENESIS=1, so the default `bash setup.sh qwen3.6-27b` cloned + checked
out Sandermage's Genesis tree at a stale pin on the most common setup path —
the original #182 harm being that the stale pin silently reverts the PN59
Cliff-2b fix.
That harm is already neutralized in practice (the live qwen3.6-27b autoround
composes run overlay-free on stock vllm/vllm-openai:v0.22.0 with `Genesis: none`
and no _genesis mount), but the clone still fired — wasted work, and a latent
landmine. It also contradicted the model profile, which already declares
qwen3.6-27b.requires_genesis: false.
Genesis's last unique justification on this stack was turboquant_3bit_nc KV;
that's now covered by beellama KVarN, and we're not reviving Genesis near-term.
- NEEDS_GENESIS now defaults to 0 and is env-overridable (`${NEEDS_GENESIS:-0}`).
- The qwen3.6-27b dispatch no longer flips it to 1 (matches requires_genesis:false).
- Opt-in preserved for reviving an archived TQ3 compose:
`NEEDS_GENESIS=1 bash scripts/setup.sh qwen3.6-27b` still clones the tree.
- Replaced the stale v7.69-dev narrative comment with the current opt-in state.
No live config needs Genesis (registry has zero non-deprecated slugs on a
genesis_equipped engine), so nothing changes for any shipped path. preflight's
genesis-pin drift check self-skips when the tree isn't cloned. Full test suite
green (test-compose-registry-disk fails pre-existing on an unrelated untracked
model dir; this change touches only setup.sh).
Closes#182.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
Phase 1 (offline): the wrapper, the corpus home, and the test gate. Phase 2
(GPU) curates the actual baselines against the live RECOMMENDED_DEFAULT_MODELS
configs.
- scripts/quality-baseline.sh: thin wrapper over quality-test.sh --full that
captures (--capture -> --save-json) or diffs (default -> --previous-result)
an n>=3 aggregate per (registry-slug, thinking-mode). no-thinking is
canonical (temp-0); enable-thinking is the reasoning-on companion. --dry-run
prints the resolved command; extra args pass through to benchlocal-cli.
- scripts/quality-test.sh: forward --repeat / --previous-result and honor a
--save-json path override, so the wrapper's blessed layout works.
- results/baselines/: committed corpus home (whitelisted in .gitignore) + a
README documenting the convention, usage, and an empty index table.
- scripts/tests/test-quality-baseline.sh: offline gate (--dry-run) — asserts
command/path resolution per mode + the required-slug / valid-mode /
positive-repeat / missing-baseline guards.
- docs/QUALITY_TEST.md: regression-baseline subsection pointing at the corpus.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
bench-agentic.sh drove the ramp with tool_choice='required' and RAISED if a
turn returned no parseable tool call → the main loop caught it and broke,
aborting at the first miss. Reachable depth was bounded by tool-call
reliability at depth, not the TURNS/context budget — so on flaky parsers the
ramp stopped below the ~35K DeltaNet degrade zone the producer exists to
characterize (field-observed abort at turn 11 / ~22K).
Fix: on a no-parseable-tool-call turn, synthesize a tool call so the prompt
keeps growing by the same fixed ~chars (the fixture tool_result is injected
regardless and dominates the growth), flag tool_call_missed, and continue.
Genuine transport errors (HTTP/timeout) still propagate and stop the ramp.
Per-turn rows mark misses; a summary line reports "tool-call misses: N/M".
The optional non-tool RAMP_MODE the issue floats is largely subsumed — the
synthesize-on-miss path already lets the ramp reach configured depth on any
engine regardless of tool-call reliability.
Test: scripts/tests/test-bench-agentic-ramp.sh — mock SSE endpoint; asserts the
ramp reaches turn 3 under 100% tool-call misses (counted) AND the success path
is unchanged (no false misses). Offline, no GPU.
Caveat: guarantees reaching configured TURNS; whether the 15-turn fixture
exceeds 35K is separate (extend the fixture if not). Sibling bench-agentic fix
#498 stays separate.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
rebench-full ran the 8-pack quality eval (both think-OFF and think-ON,
~1.5-2.5 hr) by DEFAULT — the longest phase, dominating wall time even when
you only need "does this boot / serve / recall / soak". Flip it: the 8-pack
is now opt-in.
(omit) fast structural gates only (verify + bench + stress + soak)
--with-8pack-thinking 8-pack, reasoning OFF (--full --no-thinking)
--with-8pack-thinking=off same
--with-8pack-thinking=on 8-pack, reasoning ON (--full --enable-thinking)
--with-8pack-thinking=both both passes (the production-promotion gate)
The =off pass now FORCES --no-thinking (all 8 packs think-OFF) for a clean
with/without-reasoning A/B — previously bare `--full` used pack-default MIXED
thinking, which wasn't a true "reasoning off". quality-test.sh runs exactly one
mode per call, so "both" == two invocations; the think-ON pass fires ONLY for
=on/=both, never accidentally.
Naming realigned to benchlocal-cli #65's pack-set-vs-thinking-mode vocabulary
(the issue's --with-quality=off,on -> --with-8pack-thinking[=off|on|both];
"thinking" is explicit in the name so off/on can't be misread as "skip the
pack"). ENABLE_THINKING env no longer drives the 8-pack (only bench.sh).
Promotion-gate call sites updated to --with-8pack-thinking=both: ADDING_MODELS.md
Step 8, QUALITY_TEST.md, TQ3_MTP_GENESIS.md.
Validated: bash -n clean; bad-value guard exits 2 early; --help updated;
test-quality-thinking green (quality-test.sh untouched).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
