Everything the #246 runner dogfood on this rig surfaced, in one PR:
- verify-stress STRESS_FAST=1 (opt-in; gates keep full mode): skips
probe 7's large fresh needles (near-duplicate the ladder's depths)
and caps the ceiling ladder at 2 rungs (~2/3 anchor + ceiling).
MEASURED on the identical vllm/dual boot: 861s -> 361s (-58%).
Depth coverage stays 4-anchor (bench probe 10K/90K + mid + ceiling).
- VRAM margin attribution: compose `count: all` (DeviceRequests
Count:-1) and VISIBLE_DEVICES=all now resolve as a DETERMINED
"all host GPUs" -- kills the spurious per-rung WARN on dual rigs.
- VRAM margin aggregation: MIN per-card free instead of sum. TP OOMs
on the first card to run out; the sum overstated dual margins ~2x
(906 MB reported where the honest figure was 453 MB). SEMANTIC
TIGHTENING: multi-GPU margin advisories now fire against the real
per-card floor. Ceiling test updated to the new contract.
- arch-ab: lean tier exports STRESS_FAST=1 (per-arm ~30 -> ~20 min);
margin advisory surfaced in the summary table + a mid-run "ladder
CLEAN, not a failure" note so first-time runners don't abort on the
rc=1 advisory; bundle print points at the #246 test thread.
Full scripts gate 65/65. The e4m3-below-sm_8.9 refusal is the prior
commit on this branch.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
vLLM hard-rejects fp8_e4m3 KV below SM 8.9 at boot -- without the guard
a bare run on an Ampere rig (default arms include e4m3) sits in
switch.sh's ready-wait until the 10-minute timeout instead of getting a
clear refusal. The message says why: on Ampere there is no arch delta
to measure. --arms e5m2 (control-only) stays possible.
test-arch-ab: bare-run-on-3090 refusal + control-only-on-3090 pass.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
report.sh --full-calibration instead of the bare default: seconds, no
test re-runs, and the predicted-vs-actual VRAM matrix on the volunteer
card class is direct Phase 2 input (32 GB envelope calibration).
Deliberately NOT --full -- that re-runs the whole battery (~43 min)
against only the last arm's container; the arms already carry that data.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
report.sh default mode (~2 s, path/host/user redaction ON) rides in the
tarball — PCIe lane width / driver / power caps / WSL-vs-bare-metal is
the context that makes cross-rig A/B variance interpretable (the same
mechanism that auto-flagged the x4-lane slot in #158).
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
One command per volunteer rig: runs the pilot variant once per KV-dtype
arm (fresh symmetric boot each), verify-full + bench n=5 + NIAH ladder
per arm -- soak and quality packs deliberately skipped (quality
consolidates at promotion time per the #246 test plan; ~25-30 min/arm).
Ends with a per-arm comparison table and ONE attachable tarball.
Arms: e5m2 (control) / e4m3 (native FP8 KV, sm_89+) / nvfp4 (Blackwell
opt-in, refused below sm 10.0) / fp8w (vllm/qwen-27b-dual-max STOCK --
FP8-weights checkpoints reject fp8 KV, so this arm measures the
native-FP8-WEIGHTS lift instead; dual-rig only).
Guard rails: variant auto-pick by GPU count (2+ -> vllm/dual, 1 ->
vllm/minimal); KV arms are explicit KV_CACHE_DTYPE pins (unambiguous on
any launcher version); the running container's --kv-cache-dtype is
asserted live before any bench time is spent; --dry-run / --resume /
--full escape hatch.
test-arch-ab: hermetic plan/refusal contract via CLUB3090_FAKE_GPUS.
Full scripts gate 65/65.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
launch.sh/switch.sh now detect GPU arch and export KV_CACHE_DTYPE=
fp8_e4m3 (native FP8 compute) on sm_89+ cards for the pilot slugs
(vllm/dual, vllm/minimal). The injected value is the hardware
profiles' dormant kv_format_default.balanced -- one source of truth
shared with the pull gates; the Ampere no-op is data equality
(3090-class balanced = fp8_e5m2 = compose default -> nothing emitted),
not a code branch.
Injection guards (all load-bearing):
- pilot allowlist only; expansion gated on the #246 cross-rig A/B
- per-variant: registry kv_format == fp8_e5m2 only (int8-PTH/TQ/bf16
slugs never touched -- compressed-tensors weights reject fp8 KV)
- vLLM-family variants only; explicit user KV_CACHE_DTYPE= wins;
unmapped cards / heterogeneous rigs / no nvidia-smi -> no injection
- direct `docker compose up` keeps Ampere-safe compose defaults
Rides the existing resolve-variant-pin export seam (new optional
--gpu-spec); VLLM_ATTENTION_BACKEND is whitelisted but ships no value
(vLLM auto-detect stays the default until measured). Preflight banner
names the detected arch class.
Consistency fixes the injection exposed:
- gates.py _ARCH_KERNEL_SM: fp8_e4m3 9.0 -> 8.9 (vLLM's real floor is
SM89+; we'd otherwise inject e4m3 on 4090s our own pull-gate calls
unloadable) + new nvfp4: 10.0 row (v0.24.0 literal had NO gate --
a 3090 pull of an nvfp4 config wouldn't have been rejected)
- Blackwell hardware profiles: nvfp4 declared as CANDIDATE capability
(engine list unchanged until validated -- gates take the intersection)
- kv-calc: projected nvfp4 rows (bytes/elem + activation coefs),
calibration unchanged
Docs: HARDWARE.md new section, KV_MATH/QUANTIZATION/DTYPE_MATRIX rows
(incl. retiring the Genesis-era "e4m3 undertuned per #51" advisory in
favor of the A/B).
Validated: 8-case injection matrix in test-launch-compat; live no-op
on the real 2x3090 (spec built, nothing injected); faked 4090 through
the real bash detection path emits e4m3; compose interpolation both
ways; kv-calc --calibration green; full scripts gate 64/64.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The single-card Gemma-4 default was the catalog's only default with no
baseline row (wave-2 PRIORITY gap). Full gate on Anbeeld's OFFICIAL
v0.3.2-preview digest (the current beellama-local pin), rebench tag
gemma-dflash-regate:
- decode 44.91 narr / 79.76 code (n=5, CV 2.1%/5.6%), TTFT 115 ms
- NIAH ladder clean to 117,513 tok (91% of 128K); ceiling margin 929 MB
(< 1024 MB bar -> caveat stays, but up from ~177 MB pre-v0.3.2)
- 8-pack current harness: 108/150 off / 113/150 on (+5 in-band wash,
think-off default re-confirmed)
- soak PASS (p50 59.8, 100.6% retention, 0 growth, 0/100 silent-empty)
Row seeded via catalog-baseline.sh (first non-vLLM engine through the
producer flow); BENCHMARKS row + compose header (Quality + margin
numbers) updated to match.
Tooling fallout the gate exposed:
- rebench-full.sh hardcoded RUNS:-3/WARMUPS:-1, silently under-running
the bench protocol on every orchestrated gate. The A1 "n=3 env leak"
attribution was wrong -- it was this line (baselines comment
corrected). Fixed to protocol 5/3; n=3 had flattered gemma-dflash
code TPS by +8%.
- catalog-baseline.sh counted prefill-probe run-lines into its bench-n
gate (read n=8 for an n=5 log); now reads the summary headers.
Full scripts gate 64/64.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Reviewed dispositions of the wave-1 gap list (verdict sheet approved
2026-07-04):
- vllm/dual + vllm/qwen-27b-dual-fast (alias mirror): FRESH from the
2026-06-30 v0.24.0 pin-bump gate row (decode 70.7/93.5, NIAH->240K,
soak PASS) — measured on the current vllm-stable pin, so no
born-stale archaeology needed after all.
- ik-llama/iq4ks-mtp-vision: carried decode (60.39/72.4) from iq4ks-mtp
per the 2026-05-25 vision re-tune row; carried qualifies because the
vision delta doesn't touch the decode path.
- ik-llama/apex-fit-q8q5: 2026-05-28 row (decode 105.63/156.80, n=5),
NIAH clean@180K.
- vllm/gemma-26ba4b-single: 2026-06-06 row (decode 169.2/219.9) + full
8-pack (98 off / 109 think-on), NIAH clean@161K — FRESH (gemma-stable
still pins v0.22.0).
- vllm/agents-a1-dual: decode pair order corrected from the tag
artifact (narr 154.0 / code 153.8 — wave-1 had them swapped).
- Footer: SEED WAVE 2 list -> KNOWN GAPS dispositions (vllm/minimal,
gemma-12b pair, beellama/gemma-dflash PRIORITY re-gate, deckard).
catalog-baseline.sh: insertion anchor updated for the retitled footer
(accepts both titles); test-catalog-baseline.sh now guarantees the ADD
path (strips the seeded vllm/dual row from its copy) and asserts added
rows land before the footer — the marker-drift class this bump caught.
Guards: test-baselines 15 rows joined, same 4 wave-1 stale WARNs, all
5 new rows fresh. Full scripts gate 64/64.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Closes the rescore-materialization gap the induction tool's first live
run surfaced (#562): the benchlocal-cli 'rescore' subcommand ALREADY
supports write-back (--in-place / --output) — the gap was practice, not
a missing upstream feature. The T2 rescore ran to stdout only, so the
published A1 thinking-on 110/150 diverged from the tag artifact (108).
- A1's rescore MATERIALIZED into the tag JSONs (rescore --in-place; OFF
105 unchanged, ON 108→110) — induction now extracts 110 and the
regenerated corpus record carries 110; artifact, corpus, baseline row
and publication all agree. Pre-rescore JSON backed up outside the tag.
- docs/QUALITY_TEST.md: 'Rescoring saved results — MATERIALIZE, don't
just read' — the rule (a rescore that changes a published number must
be written back in the same session), the command, what rescore can't
re-run (sandbox packs), and the A1 case as the cautionary example.
- catalog-baseline.sh header: reads-artifacts-as-truth warning + the
header's gate text updated to the actual n>=3/warn-under-5 behavior.
- baselines.yml A1 comment: gap note -> materialized + doc pointer.
Guards green (test-catalog-baseline, test-baselines).
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Decode-only TPS can't express real trade-offs anymore (the W8A8-vs-FP8
result was a prefill-corner-vs-decode-corner split) — this adds the
CANONICAL prefill/TTFT measurement to bench.sh per the design's sourcing
rule, with the protocol gotchas productized:
- bench.sh PREFILL PROBE (default-on; PREFILL_PROBE=0 / PREFILL_DEPTHS /
PREFILL_RUNS): warm + n measured per depth (10K + 90K anchors; 90K is
inside the DeltaNet degradation regime and pairs with the NIAH ladder's
~94K rung). CACHE-BUSTED: fresh salted haystack per request — composes
serve enable_prefix_caching, an identical prompt re-measures the CACHE
HIT (vLLM's prefix cache is block-chained; unique first line breaks the
chain). SELF-CALIBRATING: word-count heuristics overshoot tokens ~1.3x;
the warmup's reported prompt_toks scales the measured runs (target^2/
actual) to within ~4% of the requested depth. Depths exceeding the
served ctx SKIP with a note. DUAL METRIC, labeled: prompt_tokens/TTFT =
client-observed (user-truth: incl tokenization+transfer+scheduling) AND
the vLLM stats-log windowed rate = engine-internal (compute-truth) —
at 93K on A1 they differ by ~7s of non-prefill overhead (5.5K vs ~10K
t/s); never cross-compare kinds (stack LEARNINGS row added).
- measurement_record parser: per-block pass -> prefill_tps_by_ctx +
ttft_ms_by_ctx extensions; the canonical short-prompt ttft_s is
PROTECTED from the probe blocks (the old last-occurrence rule would
have swallowed the 90K block's 17s TTFT).
- catalog-baseline.sh: rows gain prefill_tps {10k: N, 90k: M} (parsed
via THE record parser, no second grammar) + ANCHOR CALIBRATION at
induction: the probe's deep anchor vs the NIAH ladder's nearest rung —
agreement (0.7-1.3) certifies the ladder's whole depth curve; A1 live:
probe 5459-5584 t/s @93K vs ladder 7403 @94K = ratio 0.74-0.75, OK.
Divergence warns with an investigate message (design: a finding).
- test-baselines schema: prefill_tps = dict of numeric depth points.
test-catalog-baseline fixture: probe blocks + TTFT-pollution guard +
anchor-OK assertion.
Live-validated 3x against the serving A1 (262K): 10K = 7977 t/s CV 1.1%
TTFT 1.25s; 93K = 5584 t/s CV 0.5% TTFT 16.1s; engine-log ~10K t/s.
Full scripts gate green.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The consumer half of the producer wiring: THIS RIG's corpus records now
render next to the shipped bar in the catalog preview — the design's
'c3 overlays the bar with your numbers, badged'.
- LocalMeasured projection + CatalogEntry.local_measurement: the newest
#249 corpus record per variant slug (decode point, both 8pk arms, pin,
date). One canonical-short decode point per record — the narr/code
pair split is a parser note for 2c.
- services.local_measurements(): pure-fs corpus scan; newest-per-slug by
the record's _recorded_at stamp (NEW — rebench-full now stamps it at
write; mtime+line-order fallback covers pre-stamp records); malformed
lines skipped. Joins in the same enrichment pass — still zero
subprocess on the catalog path.
- Preview rework: the measured line is the BAR with provenance
(date · rig · submitted_by from the joined baseline); a stale bar gets
an explicit detail line ('† bar measured on <pin> — current <pin>;
re-bench owed'); a slug this rig has gated adds
'yours ~<decode> · 8pk <P/150> (<date> · this rig)'.
Live-verified: the corpus's first record (A1, from 2a's live run) joins
and renders 'yours ~154 decode · 8pk 105/150'. Tests: overlay semantics
(newest-wins by stamp, malformed-skip, entry join, no-record → None) +
the three-state preview (fresh bar + provenance + yours · stale pin
detail · no-yours). c3 suite 766/766; scripts gate green serial.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The producer wiring: a validated gate run now flows into the corpus
automatically and into the shipped bar via one reviewed command.
- scripts/catalog-baseline.sh (§2.3, one mechanism / three entry points):
gate-validates the tag dir (verify-full pass HARD · bench n>=3 HARD,
n<5 WARN vs the canonical target · quality present unless --tps-only),
extracts decode TPS/TTFT/8pk both arms/ctx_validated (NIAH-ladder
verdict incl. token count)/provenance (pin via the same resolution the
emit join uses; rig+power via nvidia-smi, all overridable), UPSERTS the
baselines.yml row textually (comments preserved), prints a unified
diff. Never commits — rows ship via PR. --baselines-file for tests.
- rebench-full.sh: on completion, EXACT-container-match the served
engine to a registry slug (identity semantics — never port/substring,
the F9 rule) and append a fingerprint-complete #249 record (engine_pin
via resolve_variant_pin/compose fallback · hardware · power-cap ·
quality_8pk extensions · soak status) to the gitignored corpus, then
print the catalog-baseline.sh induction prompt. BYO/swap serves skip
with a note (no registry identity to record against).
- measurement_record.py: --quality-8pk / --quality-8pk-think-on ->
measured_extensions (the designed extension namespace; frozen schema
untouched).
- test-catalog-baseline.sh: hermetic (synthetic tag dir + a COPY of
baselines.yml) — extraction, dry-run no-write, add->replace upsert,
refusals (missing verify/quality/unknown slug), real-file checksum
guard.
TRUTH CORRECTIONS the tool's first live run surfaced (run against the
real agents-a1-fp8-dual tag): the A1 gate's bench was n=3 (a RUNS=3 env
leak into the overnight gate — 'n=5' was misstated in publications), the
BENCHMARKS TPS pair was order-garbled (artifact: narrative 154.0 / code
153.8), and the #81-rescored think-on total (110) was never materialized
back into the tag JSON (reads 108) — materialization gap documented in
the row comment. baselines.yml + BENCHMARKS.md corrected.
Live-validated: rebench record block ran against the serving A1 → first
corpus record written (vllm-agents-a1-dual__e9d28e22.jsonl: pin v0.24.0,
rtx-3090, 370W, quality extensions); induction dry-run reproduces the
row from the tag. Deferred to 2b/2c: c3 local-overlay + richer badges;
the bench 10K/90K prefill/TTFT probe. Gates: test-catalog-baseline +
test-measurement-record + test-baselines green; full scripts gate green.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The catalog's measured columns (TPS / 8pk) now come from ONE productized
source: scripts/lib/profiles/baselines.yml — the shipped, PR-reviewed
'bar' (accepted display projection of a validated gate run, with pin/rig/
power provenance) — joined per-slug into registry-emit --json with an
emit-computed staleness verdict. Consumers never read baselines.yml or
BENCHMARKS.md directly (BENCHMARKS stays the public human/cross-rig
ledger). Design: the catalog-baselines note (2026-07-02, §2/§5 slice 1).
- baselines.yml SEED WAVE 1: 10 rows with airtight traceability only
(rebench tags on disk: agents-a1 golden specimen + gemma-31b-dual;
unambiguous decode-class BENCHMARKS rows for the rest). 4 rows are
HONESTLY born-stale with documented reasons (llamacpp rolling-tag-era
x2, beellama pre-#296 image, 35B v0.22.0) — the guard's demo cases.
9 slugs still owed rows are listed as wave-2 gaps, no guessed numbers.
- registry-emit join: per-variant 'baseline' field; current pin resolved
the way launchers actually resolve it (engine-profile install.spec,
compose-image-default fallback for ik/llama.cpp) → 'stale' =
measured-pin != current-pin, null when undeterminable.
- test-baselines.sh: schema + slug-membership + ctx-parity (compose ctx
default == registry max_ctx, functional slugs) RED; pin-staleness
WARN-only (pin bumps must not block on immediate re-bench — the debt
stays visible). test-registry-json gains the 'baseline' contract key.
- c3: enrich_measurements = pure in-memory map off the joined field —
deletes BOTH the per-slug --explain fan-out (~4s/slug, the #439
option-3 leg) and the BENCHMARKS.md scrape from the catalog path
(Explain modal + cross-rig explorer keep their readers). Stale rows
render a † on the TPS cell + a status-line legend; full badge/overlay
treatment is slice 2. Measurement gains source='baseline' + stale.
- results/baselines/README: the three-store relationship (regression
corpus vs measurement records vs display bar) + the no-drift rule the
slice-2 induction tool enforces.
Verified: guard test green (10 rows, 4 stale-warned); live join emits
57 variants / 10 baselines; live c3 catalog shows all 10 with daggers on
exactly the stale 4 (A1 serving during the check: 154/154 - 105/150);
c3 suite 764/764; full scripts gate green serial. (First gate pass
tripped classifier/dedup by running CONCURRENTLY with the c3 suite —
.pull-captures pollution class; serial rerun clean.)
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Investigation of a Discord cross-rig report (WSL, 3060Ti+4090: verify
fails on the pinned v0.3.2-preview digest, passes on the old noonghunna
snapshot; NOT reproducible on our 3090s — verify-full all-pass) surfaced
three doc-level truths worth recording:
- The 50-series gap is a TOOLCHAIN ceiling: Anbeeld's CI builds on the
Dockerfile-default CUDA 12.4, whose nvcc cannot target sm_120 (no
cubin, max PTX compute_90) — every official tag lacks Blackwell.
Upstream ask filed: Anbeeld#85 (CUDA_VERSION 12.8.1 + arch list incl
120); when it lands we retire the noonghunna snapshot entirely.
- Three registry status_notes claimed launchers inject 'server-cuda-
v0.3.0' — stale since the 2026-06-12 pin bump; they inject the
v0.3.2-preview digest from engines/beellama-local.yml install.spec.
Fixed all three (qwen dflash, gemma-12b, gemma dflash) + honest
labeling of the noonghunna snapshot as v0.3.0-feature-level and
unmaintained (predates KVarN + v0.3.1 fixes).
- The engine-notes self-build guidance was outdated: FA_ALL_QUANTS is
hardcoded in Anbeeld's cuda.Dockerfile since our PR Anbeeld#48, so a
self-build needs only CUDA_DOCKER_ARCH (+ CUDA_VERSION=12.8.1 for
sm_120). Recipe verified against his master Dockerfile 2026-07-04.
UPSTREAM.md beellama row updated (dated entry + next-triggers; the 'no
official image' claim struck through as historical). Gates: YAML +
registry import clean; status-drift / profiles-compat / switch-parity
green.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The A1 compose shipped (#546) without the 'Hardware metadata' block every
sibling carries, so preflight_compose_gpu_fit logged 'WARN: compose has
no hardware metadata; allowing boot' and skipped enforcement — visible in
guybrush01's #548 report. Add the block (min-vram 24 / gpu-count 2 /
TP 2 / Requires-sm 8.0+ — Marlin FP8 weight-only needs sm_80, unlike the
sibling's INT4 7.5+); preflight now parses + enforces (live rc=0).
Also propagate the #548 finding as Caveats (4): Blackwell sm_120 hits an
upstream vLLM v0.24.0 kernel-selection bug at boot ('QKVParallelLinear'
has no attribute 'workspace' — Cutlass W8A8 selected, a Marlin-path
attribute expected). Workaround shipped as a commented env line
(VLLM_TEST_FORCE_FP8_MARLIN=1 — forces the exact weight-only path this
compose's gate validated; verified present in the v0.24.0 image) +
mirrored into the registry status_note so switch.sh's NOTE carries it.
Gates: compose_meta_get parses all 4 keys; docker compose config valid;
test-compose-status-drift / registry-disk / preflight-gpu-fit green;
full scripts gate green.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
F2 — the Catalog preview strip clipped a WRAPPING caveat line: border
eats 2 rows of max-height 6 → 4 content lines; the dual-fast caveat
wrapped past that and lost its tail. max-height 6→8 + overflow-y auto
(height stays auto — short previews don't grow). Regression test pins a
wrapped-caveat entry to its full height with the tail visible.
F5 — the #544 deliberate deferrals:
- wait_gpu_vram_settle wired into ALL model/studio scene handlers
(27b, 35b-a3b, gemma-12b, deckard, ai-studio) — was gemma-int8 only;
every scene switch now lets the torn-down scene's VRAM release before
the next boot (#535 class).
- mode_off gains an engine-prefix CATCH-ALL: the enumerated stop_*
lists cover gpu-mode scenes, but a catalog-launched engine
(switch.sh <slug>) survived 'off' — caught LIVE during validation
when off left vllm-qwen36-27b-minimal serving and the 27b TP=2 scene
booted straight into its residue (the exact #535 failure). Any
remaining vllm-/llama-cpp-/ik-llama-/sglang-/beellama- container is
now stopped, with a named notice.
- c3 preflight-error visibility (the third residual): verified
already-plumbed — switch.sh's #544 refusal exits fast, its
[preflight] ERROR lines stream into the serve pane, serve_failed
stamps ✗ + [!] capture (pinned by existing tests). No change needed.
Validation: bash -n + full scripts gate green (by exit code); c3 suite
763/763; LIVE scene cycle off → 27b → off on the rig — boot ready
(qwen3.6-27b, 21.5 GiB/card), fixed off left ZERO engine containers
(verified with a non-enumerated probe container that the catch-all
stopped), both GPUs at 1 MiB after.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
estate.yml is a desired-state PLAN, but report-state reported its
instances as 'active' without checking docker — so every consumer
treated a leftover ~/.club3090/estate.yml as live GPU claims. The T1.1
audit hit this on an EMPTY rig: the c3 serve confirm warned '⚠ Starting
this will STOP estate llama-gpu0 (GPU [0]), estate llama-gpu1 (GPU [1])'
with nothing running. Scary + wrong for a fresh user.
estate_cli (the layer that owns estate liveness — fix here, not a c3
filter):
- report-state --json: each instance gains 'container' (the
club3090-<name> it would own) + 'running' (docker probe: true / false
/ null when docker itself is unavailable), payload gains
'running_count'. Additive — existing keys unchanged.
- report-state text view: per instance '— running' / '— down (plan
only)' / '— liveness unknown'.
c3 reconcile gate (consumer): an estate instance is a claim unless
running == false. true / null / MISSING (older CLI output) all stay
claims — the dual-writer gate fails CLOSED on unknown liveness, and a
mid-boot instance (running container, VRAM not yet allocated) still
conflicts.
Tests: estate JSON contract test extended (running/container/
running_count; running asserted 'in (False, None)' so the test passes
with or without docker); 3 new c3 gate tests (stale plan on empty rig →
SAFE; running:true → claim even with idle GPUs; running:null → fail
closed; the legacy no-key fixtures double as the missing-key case).
Scripts gate all green; c3 suite 758/758. Live repro on this rig (which
has the exact stale estate.yml): running_count 0, reconcile now reports
estate_claims=[] while keeping the TRUE conflict (the actually-serving
vllm container).
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The T1 audit's top adoption gap: even while a model SERVES, the best
endpoint info anywhere was a bare port — no scheme/host/path, no auth
note, no copy. "Point your agent at this URL" is the whole reason the
stack exists, and the URL was underivable from the UI.
One shared derivation (layer rule; #512 precedence env/.env ->
c3_lan_ip subshell -> localhost), four surfaces:
- switch.sh: ready-line on successful boot —
"▶ API: http://<lan>:<port>/v1 (model: <served-id> · OpenAI-compatible
· no auth)" — CLI parity; served id probed from /v1/models.
- services.lan_ip(): c3 consumes the SAME derivation, session-cached.
- Orchestration serving card: bare ":8020" -> full URL + "no auth ·
[u] copy".
- Estate rail: compact "api :8020/v1 · [u] copy" (rail is ~30 cols; the
full URL lives on the card).
- NEW [u] copy-API-URL key on every Run & Operate tab (kept separate
from [Y] so row-copy semantics are untouched); honest notify no-op
when nothing serves.
F3b (readiness lag): api_booting only cleared when the HEAVY docker+
health batch re-ran, so "⏳ booting" outlived actual readiness by a poll
cycle (audit: >=25s stale across three surfaces). While booting, the
fast GPU tick now piggybacks a 1.5s /v1/models probe; on 200 it pulls
the next heavy poll forward (burst regime) — every booting surface
flips within a tick or two of the API answering. Bounded: fires only in
the booting state.
Verified: c3 pytest suite 516 passed; live serve via switch.sh prints
the ready-line with the real LAN IP + the neutral served name
(http://192.168.86.33:8020/v1 · qwen3.6-27b · no auth).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Adds a stdlib HTTP control plane that wraps scripts/switch.sh so a harness
can POST /switch and block until the new model is serving. Introduces no new
orchestration logic — switch.sh stays the single source of truth (registry
lookup, down/up, readiness).
- tools/model-switch/server.py: GET /healthz|/status|/models, POST /switch
({slug}|{model}); registry-validated; /health readiness (works with or
without VLLM_API_KEY); single-flight lock; refuses to start unauthenticated
on a non-loopback bind.
- scripts/systemd/club3090-model-switch.service: host daemon unit.
- scripts/tests/test-model-switch.sh: hermetic HTTP/auth/validation contract.
- docs/EXAMPLES.md, .env.example: usage + config.
Mirrors the existing stdlib HTTP style (services/studio/*); zero new deps.
Experimental/opt-in per the repo's staging convention.
InternScience's 35B agentic MoE (Qwen3-Next MoE arch; the card declares its
OWN base -> own model entry, not a qwen fine-tune slug), served from the
official FP8-dynamic compressed-tensors checkpoint on stock vLLM v0.24.0,
dual 3090 TP=2, full 262K. The agentic thinking-ON specialist.
First model onboarded END-TO-END through the Bring & Validate lane (T2
producer-zero): pull.sh route-C sibling swap (post deriver fix) ->
generate-compose -> documented swap -> full gate -> ④ vs-bar -> ⑤ scaffold.
Full gate PASS (rebench tag agents-a1-fp8-dual, 2026-07-03):
- bench 153.9/154.0 decode TPS (n=5, CV 0.1%, TTFT ~130ms), ~22.0 GB/card
- verify-stress 8/8 — staggered NIAH exact-recall to 240K (91%), VRAM Δ0
- soak PASS (0 growth, 0/100 silent-empty, 99.8% retention)
- 8-pack --full OFF 105/150 · ON 110/150 (post benchlocal #79+#81 harness):
toolcall 15/15 OFF · IF 15/15 ON · cli-40 thinking-ON 23/40 (the stack's
highest; base 17/40) · hermes 12/20 OFF, REGRESSES to 9/20 thinking-ON
(verified model behavior — disclosed as caveat 1)
vs qwen3.6-35b-a3b: general capability TIES (ON 110=110), decode −13% —
NOT a general upgrade; reach for it on tool/CLI-agent work thinking-ON.
Catalog wiring: compose dual/fp8-dynamic/fp8.yml (non-default KV named per
convention; OWN port 8072 — sibling-shared ports masquerade the slug in
estate detection); model profile (geometry verified == 35B-A3B from
config.json; mtp_num_hidden_layers=0 — safetensors header shows 0 mtp
tensors, config's 1 is an inherited default); registry entry status=caveats;
kv-calc agents-a1 spec + alias (fit-all prices, calibration 100%);
weights.py aliases (hf auto-fetch); LiteLLM route :8072; BENCHMARKS section;
count bumps (registry 57, disk 58, models 11).
Full catalog suite 60/60 (1 = known worktree-fixture submit-bench).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
pull.sh (and c3's ① Bring pane) rejected EVERY llm-compressor checkpoint
(FP8-dynamic, INT8 W8A8, INT4 pack-quantized) with `quant-dtype-unknown
(bits undeterminable)`: _quant_bpw only read top-level bits keys + method-
name heuristics, and "compressed-tensors" as a method matches none — but
the bits live nested at config_groups.<g>.weights.num_bits. Parse them
(widest group wins — mixed-precision groups exist; the widest dominates
the VRAM footprint the fit-check prices).
Found by the T2 producer-zero dogfood (Agents-A1-FP8-dynamic): step ①
hard-stopped with arch=null + swap_path=null; post-fix the lane resolves
the arch and emits the honest route-C verdict (sibling qwen3.6-35b-a3b,
BRING_YOUR_OWN swap pointer) — validated live, and the swapped compose
boots + serves on stock v0.24.0 (Marlin weight-only FP8 MoE on sm_86).
+3 deriver test cases: FP8-dynamic (the A1 shape), INT4 pack-quantized,
mixed groups -> widest.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Switching gpu-mode ai-studio → gemma on a desktop rig (GNOME running on the GPUs,
~1.3 GiB/card) leaves < the 0.95 util budget free, so gemma failed vLLM's boot
free-memory check (`free < util × total`); with restart:unless-stopped it restart-
looped and switch.sh waited the full 600s READY_TIMEOUT for an endpoint that never
came. The existing gpu_preflight only enforces a coarse 80%-free floor (the reporter
had ~90% free), so it sailed through. qwen (0.92) fit → that path worked, which is
why *only* gemma broke.
- preflight.sh: new preflight_compose_gpu_fit — parses the effective
GPU_MEMORY_UTILIZATION (env override or compose default) + TP card-count, compares
per-card free VRAM to `util × total × 0.98` (~ vLLM's own mem_get_info total), and
HARD-fails (unless --force) with an actionable message (free VRAM / lower util). A
~10s settle-retry covers teardown lag (docker `down` returns before CUDA frees).
- switch.sh: call it in up_variant (vllm-only, after down_running so the retry also
covers the just-torn-down container). launch.sh delegates to switch.sh → covered.
- gpu-mode.sh: wait_gpu_vram_settle after scene teardown in the gemma handler, so the
incoming TP=2 model boots into freed VRAM instead of ai-studio's residue.
- test-preflight-gpu-fit.sh: mocks nvidia-smi + the #535 numbers — fit / short+message
/ --force / env-override / single-card.
Full shell gate 60/60 (1 = known worktree-fixture). Validated end-to-end against the
real gemma compose: idle rig fits; simulated 22060 MiB free → instant "GPU 1 has 21.5
GiB free, needs ~22.3" + lower-util hint (matches vLLM's 22.38 threshold).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The shared served-model-name `qwen3.6-27b-autoround` (from #490) mislabels the
non-autoround 27b scenes (fp8 dual-max, lmcache): /v1/models advertises
"autoround" while `root` points at qwen3.6-27b-fp8. Same class on gemma-4-31b —
after the v0.24.0 consolidation the default is cyankiwi qat-AWQ-INT4 (bf16 KV),
yet the LiteLLM route still targeted `gemma-4-31b-autoround` (a latent #482 drift).
Fix WITHOUT breaking anything, via vLLM multi-served-name:
- Every 27b scene now serves `qwen3.6-27b <its-quant-name>`; every gemma-31b
scene serves `gemma-4-31b <its-quant-name>`. The neutral name is PRIMARY
(honest /v1/models id); the quant-specific name is retained as a live ALIAS.
- LiteLLM: add `qwen3.6-27b` / `gemma-4-31b` canonical public routes; keep the
`-autoround` routes as back-compat aliases (same upstream). Repairs the gemma drift.
- Migrate our own MODEL= defaults + docs (bench/verify/quality/launch/setup, c3,
tui-core, EXAMPLES, ...) to the neutral name. Weights slugs (`-autoround-int4`)
untouched; CHANGELOG + results/ history left as-is.
Retiring the `-autoround` alias entirely is a deliberate later step once nothing
still asks for it.
Live-verified on-rig (single/minimal, stock v0.24.0): /v1/models lists BOTH names
(root=...-autoround-int4); chat to `qwen3.6-27b` AND `qwen3.6-27b-autoround` both
return 200; `qwen3.6-27b-fp8` correctly 404s. Full shell gate 59/59 (1 = known
worktree-fixture); c3 pytest 41 passed; served-name arg-order + YAML validated.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Our vllm#40361 sub-tile-n Marlin pad was closed-superseded by mgoin's
vllm#45295 (consolidated marlin_padded_nk across all dense Marlin paths),
native in vLLM v0.24.0. `vllm-stable` now pins v0.24.0 and the AutoRound
INT4 TP=2 path (vllm/dual) boots clean without the overlay (validated in
#533 Phase 0b). No live compose mounts the patch (archive-only), so this
is a tracking-only de-registration — the cleanup deferred from #533.
- patches.yml: qwen-vllm-marlin-pad -> deprecated (upstream.status
open->merged, load_bearing_when [], delivery none, drift_guard null),
mirroring the gemma-vllm-pr41800 merged-and-dropped precedent. Kept as
history (foundational false; entry not deleted).
- arch_patches.yml: correct the stale kernel_constraints note (#40361 ->
#45295 native in v0.24.0). required_patches / marlin_alignment_required
unchanged: the alignment is a real arch property (now satisfied stock),
and deprecated patches stay listed per the pr41800 precedent.
- UPSTREAM.md: mark the #40361 / #40354 / v0.24.0-bump marlin rows DONE
(native in v0.24.0, patch de-registered, no live mount) and correct the
stale "composes still mount it" line (all mounts are under _archive/).
Full shell gate green (59/59; the 1 = known worktree-fixture-absent
test-submit-bench). test-patch-attribution (reads both registry files) passes.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The DiffusionGemma arch is NATIVE in v0.24.0 (PR #45163 merged 2026-06-12), so
the vllm-diffusion-gemma engine drops the mutable/branch :gemma digest for the
same pinned stable tag the rest of the stack uses. The 3 Ampere/TP fix-mounts
STAY (not in #45163): marlin.py + marlin_utils_fp8.py (sm_86 fp8 MoE Marlin
K-pad — #45295's dense marlin-pad in v0.24.0 does NOT cover marlin_moe_wna16)
and diffusion_gemma.py (TP-vocab; native #45163 copy is still TP=1).
Also fixes the compose for v0.24.0's arg-parser: the bare positional model must
be the --model flag (the entrypoint prepends --disable-custom-all-reduce).
Live-validated 2026-07-02: the actual compose boots + serves coherent at TP=2 on
stock v0.24.0 (Paris / primary colors / 42). Suite 59 pass / 1 pre-existing
(test-submit-bench worktree-fixture-absent).
- engine vllm-diffusion-gemma.yml: install.spec :gemma@digest -> v0.24.0 + provenance
- compose base.yml: image default -> v0.24.0 + --model flag fix + header
- patches.yml dgemma-gemma-image-fixes: reason/drift_guard/drop_when -> v0.24.0
- BENCHMARKS + UPSTREAM #45163 row: reflect the bump
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Gemma-4-31B onto stock vLLM v0.24.0, overlay-free on vllm-stable — the
first slice of the gemma consolidation (Option A1).
- New dual/qat-awq-int4/int8.yml: cyankiwi QAT-AWQ-INT4 weights (lm_head
excluded → dodges the v0.24.0 tie_weights crash that autoround hits),
int8-PTH KV @262K, util 0.965 (bf16 lm_head margin), native gemma4
tool+reasoning parsers. Overlay-free: #40391 (KV page-align) is native
in v0.24.0; #42006 N/A.
- MTP DISABLED with a documented re-enable caveat: Gemma-4 MTP x tool-calling
is broken on v0.24.0 (upstream vLLM #39043; MTP fix#42006 closed-unmerged).
A/B-proven on-rig: MTP-off tools PASS, MTP-on tools FAIL. Re-enable when a
stable vLLM ships the fix.
- Registry: vllm/gemma-31b-dual on vllm-stable; qat-awq-int4 weights map.
- vllm-stable.features.int8_per_token_head -> true: #40391 is native in
v0.24.0, so gemma int8-PTH no longer needs the overlay-carrying engine.
Live-validated dual on stock v0.24.0: 267K KV pool, coherent, streaming
multi-tool 3/3 (all args, no leak). Suite green (registry/compat/diagnose).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
#531 pinned COMFYUI_ROOT for ComfyUI's input/models mount, but the 5 output
consumers (gallery :8189, orchestrator, tts, step-voice, production) mount a
separate COMFYUI_OUTPUT_DIR that was never derived from COMFYUI_ROOT — it fell
back to /mnt/models/comfyui/output. On any non-/mnt rig, ComfyUI wrote renders
to $COMFYUI_ROOT/output while the gallery served the empty /mnt default -> 404
on generated media (the '_output_images_will_be_put_here' placeholder was the tell).
- comfyui-paths.sh: derive + export COMFYUI_OUTPUT_DIR=$COMFYUI_ROOT/output
- c3_persist_comfy_root: pin it to .env PER-VAR (adds OUTPUT_DIR even when ROOT
was already pinned by #531 -> existing users self-heal on next studio start)
- test: +7 assertions incl. the ROOT-already-pinned migration case
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
Root cause of "model dropdowns empty" on any rig whose MODEL_DIR isn't the /mnt
layout: the comfyui compose mounts `${COMFYUI_ROOT:-/mnt/models/comfyui}/models`
via `sudo docker compose --env-file .env`, but COMFYUI_ROOT is only derived
in-shell by comfyui-paths.sh — sudo strips the exported var and .env carries only
MODEL_DIR, so the compose falls back to the /mnt default and mounts an EMPTY tree.
Downloads (which key off the MODEL_DIR-derived COMFYUI_ROOT, e.g. $HOME/comfyui)
land elsewhere → ComfyUI sees no models → loaders 400 with "not in []" and the
HiDream node reports "not installed". Invisible on the maintainer rig only because
its /mnt layout makes the two paths coincide.
Fix: persist the resolved COMFYUI_ROOT into repo-root .env (write-if-absent, never
clobbers a hand-set value; no-op under C3_PATHS_NO_ENV / unwritable .env) — same
mechanism already used for LANIP. Called from c3_ensure_comfy_models_dir (every
downloader) and gpu-mode start_comfyui (the launch), so the --env-file'd compose
mounts the SAME tree the downloads went into.
- services/comfyui/comfyui-paths.sh: add c3_persist_comfy_root(); call it from
c3_ensure_comfy_models_dir. C3_ENV_FILE override for testability.
- scripts/gpu-mode.sh: start_comfyui pins COMFYUI_ROOT before the compose up.
- scripts/tests/test-comfyui-paths.sh: + cases H (writes when absent) / I (respects
existing). 13/13 green; test-studio-derig green; real .env not polluted.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
Both lanes were fully wired (workflow + custom node + OWUI lane) but their
weights were never fetched by setup-ai-studio.sh -> download_studio_models.sh,
so selecting either returned a cryptic "ComfyUI generation error" — the log
shows FileNotFoundError: <model> is not installed (download_if_missing: false).
- HiDream: download_hidream_o1.sh existed and is correct, but the roster never
invoked it. Wire it in.
- Chroma: had no downloader at all. Its UNET (Chroma1-HD-fp8mixed), the shared
t5xxl_fp16 encoder, and the flux ae VAE relied on the legacy (now-unwired)
download_models.sh already being on disk. Add download_chroma.sh (mirrors
download_zimage.sh's split_files staging; ungated sources) and wire it in.
- studio-models.tsv: add the chroma + hidream rows so c3's missing-model check,
gpu-mode preflight, and the download modal know about them.
Repo file paths verified against the live HF API. The downloads themselves are
not run-verified here (no 120 GB pull); the script mirrors the proven sibling
downloaders. test-studio-derig.sh green.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
The help text claimed "the two packs that default thinking-on: instructfollow-15,
reasonmath-15" but the authoritative benchlocal-cli pack metadata
(default_thinking: on) covers four: instructfollow-15, reasonmath-15, bugfind-15,
hermesagent-20 (plus the --reasoning suite). Stale since bugfind/hermes were
flipped on upstream. Help-text only; no behavior change.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
On a fresh install OWUI has no admin account yet, so step 4 (push-pipe-to-owui.sh) is EXPECTED
to be skipped — the Studio lanes can't install without an admin user. The old guidance was a
yellow warning mid-flow + a buried parenthetical in step 1 of the onboarding, which a reporter
scrolled past (#510: "no Studio function / no valves").
Track PIPE_OK from the install result and, when the pipe was skipped, surface installing it as a
LOUD, renumbered onboarding step ("⚠ INSTALL THE STUDIO LANES … bash services/studio/push-pipe-
to-owui.sh"), instead of a passing warning. When the pipe DID install, the step disappears and the
flow stays clean. Also point the browser_base step at the real location (Admin → Functions → Studio).
Verified both renders (skipped → prominent install step + renumber; installed → clean 3-step).
Co-authored-by: noonghunna <[email protected]>
AI Studio generation bugs (e.g. #510 "ComfyUI generation error" on the HiDream lane) weren't
diagnosable — report.sh captured hardware/stack but not the ComfyUI log where the actual
workflow-execution error lives, so reporters had no easy way to include it.
Add an opt-in `--studio` flag (off by default — verbose, only relevant for studio bugs) that
appends the container log tails for the AI Studio stack: comfyui (400 lines — the generation
engine, carries the error trace) + studio-director / orchestrator / image-shim / tts /
step-voice / gallery (200 lines each). Each in a collapsible <details> block with its run
status, ANSI colour stripped, and piped through the same redact() as the rest of the report.
If no studio containers exist, it says so.
Also: add the `--studio` row to the bug-report issue template (the bug-type → command table)
so studio reporters reach for it.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
`hostname -I` is net-tools only; CachyOS (and any GNU inetutils host) lacks it, so c3_lan_ip
returned empty → the studio URLs fell back to `localhost` (and the reporter hit a setup abort
they attributed to it).
- c3_lan_ip: try `hostname -I`, then fall back to the portable `ip -4 -o addr show scope global`.
Always returns 0 (callers no longer need a `|| true` guard).
- .env is now the source of truth for LANIP, like MODEL_DIR: comfyui-paths.sh reads LANIP from
repo-root .env at source time (env still wins), and a new c3_resolve_lanip() PERSISTS an
auto-detected IP back to .env (user-editable) — or, if nothing detects, falls back to localhost
and tells the user to set LANIP in .env. Precedence: shell-env > .env > auto-detect > localhost.
- setup-ai-studio.sh calls c3_resolve_lanip (the persister); gpu-mode keeps reading (it now picks
up the .env value + the portable detect, no write on a mode switch).
- derig guard: assert the `ip` fallback exists (no hostname-only regression) + accept the
c3_resolve_lanip wrapper.
Verified: net-tools host, faithful CachyOS host (hostname rejects -I → ip fallback), pinned
.env/env (reused, not overwritten), and no-detect (localhost + instruction). Real repo .env
untouched by the tests.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
Quick-win batch from the Codex director-lane review (all confirmed against code):
F1 — the OWUI Production lane misread creation asks as questions: "can you make a
30s noir short?" / "do a 1-min documentary on Pakistan" start with can/do and end
with "?", so the old keyword gate excluded them from brief detection and dove into
chit-chat with no brief. A GENERATION REQUEST now takes precedence over question-
shape, so a creation ask phrased as a question is captured as the brief.
F13 — extracted the conversation-intent classifiers out of pipe()'s untestable
locals into services/studio/director_intent.py (stdlib-only, table-tested). The pipe
bake injects its source verbatim (same pattern as AGENTS.md/workflows), so the
deployed pipe and the tests can't drift. +11 intent tests incl. the real failing
transcript.
F8 — the planner's capability registry (capabilities.yaml) described ONLY Wan, so
the 4B planned against Wan's 16fps/silent physics even when LTX (24fps, native
audio) was pinned. Added ltx/sulphur/10eros contracts; prompt_slice now filters to
the PINNED lane and describes audio per-lane (no more hardcoded "SILENT"). Drift
guard: test_capabilities_match_wired_video_lanes.
F4 — pipe sized shots with round(secs/5) while the server uses ceil; aligned to ceil
(must match planner.derive_shots so the proposal == what /produce builds).
F11 — the production video-lane valve still called ltx/sulphur/10eros "roadmap (not
yet wired)" — they render + were live-validated. Fixed the text + a derig guard.
Out of scope (Codex Batch 2/3): structured LLM intent controller, semantic critic,
long-plan decomposition, selective thinking-on, cross-lane router.
Tests: 103 production+intent unittests + studio derig guard all green; pipe
rebuilt + py_compile-clean + baked intent block exec-verified.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
comfyui-paths.sh: default MODEL_DIR to $HOME/models (user-owned) instead of the rig /mnt path when unconfigured; c3_ensure_comfy_models_dir() fail-fast; shared c3_lan_ip() used by gpu-mode + setup. Follow-up to #505/#503/#504. Full suite 60/60.
`quality-test.sh --full` (which docs tell cross-rig contributors to run)
needs the 3 sandboxed packs' Docker images — but they aren't auto-pulled,
and benchlocal-cli's mid-run hint pointed at a relative
`tools/build-sandboxes.sh` that's only valid inside a benchlocal-cli
checkout (absent on a pip install). guybrush01 followed it into a dead
path (#492).
- quality-test.sh: on --full / --sandboxed-only, preflight the
benchlocal-sandbox-{bugfind,cli,hermes} images UP FRONT. If missing,
print the correct clone + build steps (our context, not benchlocal-cli's
broken relative path) + the --medium no-Docker fallback, then continue
with the deterministic packs.
- QUALITY_TEST.md: document that --full needs the sandbox images built
first ("needs Docker" undersold it) — the clone + build-sandboxes.sh
step, with --medium / --no-sandboxed as the no-Docker path.
Full suite green (59/59). Pairs with benchlocal-cli#69 (the upstream hint).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
setup-ai-studio.sh's director download honored MODEL_DIR, but the ComfyUI
assets (image/video/audio) defaulted to a hardcoded /mnt/models/comfyui — so
any rig whose models don't live under /mnt failed mid-download with
"mkdir: Permission denied" (sumo Discord report, 2026-06-27). Two more walls
behind it: the disk-space preflight checked the hardcoded /mnt path and had no
override (an idempotent resume under-threshold was blocked), and gpu-mode.sh —
which setup calls at step 3 — hardcoded CLUB3090_DIR=/opt/ai/github/club-3090,
breaking on any other clone.
- New services/comfyui/comfyui-paths.sh: derive COMFYUI_ROOT / COMFYUI_MODELS_DIR
as a "comfyui" sibling of MODEL_DIR (read from repo-root .env when unset).
Backward-compatible on the rig (/mnt/models/huggingface -> /mnt/models/comfyui);
explicit overrides respected. Sourced by setup-ai-studio.sh,
download_studio_models.sh, and gpu-mode.sh so the download target, the disk
check, and the container mounts all agree.
- setup-ai-studio.sh: disk check uses the derived path; add SKIP_DISK_CHECK=1 to
bypass the free-space preflight independently of SKIP_DOWNLOAD.
- gpu-mode.sh: auto-detect CLUB3090_DIR from the script's real location (resolving
the /usr/local/bin symlink), overridable via env — removes the hardcoded path.
- The 3 legacy hardcoded download scripts (flux2/hunyuan_llava/models) honor
COMFYUI_MODELS_DIR.
- New test-comfyui-paths.sh guards the derivation; docs/ai-studio config note.
Full suite green (59/59).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
report.sh captured P2P *capability* thoroughly (topo -m, topo -p2p, lspci
LnkSta/ACS) but never surfaced P2P *engagement* — whether P2P is actually
ON for the running serving container. The [nvlink] boot line from
detect_nvlink.sh (which states the resolved NCCL_P2P_LEVEL + custom-all-
reduce state) was only present implicitly in the raw "first 200 lines"
log dump, and missing entirely when no container was up or when the line
scrolled past line 200 (likelier on 3-4 GPU boots).
Add an "Interconnect / P2P engagement" field to Boot log highlights: grep
the WHOLE boot log for the [nvlink] line + the live container's
NCCL_P2P/NVLINK_MODE env, printed as a labeled block. Always prints
something (explicit "undetermined" note otherwise) so a reviewer never
has to guess. Closes the round-trip we hit on #446 and #488.
Full suite green (58/58).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
The nvidia-power-cap.service unit shipped in #485 caps both cards at 250W (rtx-3090.yml prefill operating point), but gpu-mode.sh's comments, echoes, and the service-missing fallback (-pl 230) all still said 230W. Align them so the messaging + fallback match the unit.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
scripts/gpu-mode.sh `power-cap on` restarts nvidia-power-cap.service as
the single-source-of-truth enforcer, but the repo never shipped the unit
— so the path silently fell back to a direct `nvidia-smi -pl 230` and no
cap survived reboot.
Ship the Type=oneshot unit (250W both 3090s — the rtx-3090.yml
`power_cap_w_prefill` point; higher caps trigger GPU1 fall-off-bus on the
X670E rig). Ordered After=gpu-tune.service so on rigs where another
oneshot also sets -pl at boot, this unit is the last writer; After= on an
absent unit is a no-op, so the file stays portable. See #483.
Co-authored-by: Alex <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
DeepReinforce agentic-coding RL fine-tune of Qwen3.6-35B-A3B (same qwen35moe arch, no MTP head, text-only). ik_llama dual 3090, Q8_0, full 262K, drafter-free ngram (opt-in). Gate PASS: ~108.8/108.6 TPS, verify-stress 8/8, soak PASS, 8-pack 105/105. Ties the base on the 8-pack (110), EDGES it on aider (15/30 vs 12-13) -> coding-leaning lane. Catalog suite 58/58 green.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
