Two tiers per #665: Tier 1 = never passed anywhere (10 — capability
gap), Tier 2 = reachable but rare (6 — probability gap; per-scenario
evidence inline). RM-13 moved out of never-set on the vLLM A0 OFF pass,
shrinking Sean's proposed 11 to 10. Same 16 selections, guard green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
First 8-pack on vllm/tess-dual-nvfp4 (both modes, same-night GGUF
comparator 116/117). Gap is cli-40-concentrated (−8 off); deterministic
packs tie or better (RM-off 14/15 best-ever Tess). Per the
pre-registered rule the huginnfork NVFP4A16 weights-only A/B + FP8
precision arm are the next gates. BENCHMARKS vLLM subsection + registry
note + compose Quality line; Grok-§3 discriminator results recorded in
learnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Shipped-config quality refresh on b9967 + 16K budget (native template,
both modes, no pack regression vs the 2026-07-09 basis; first toolcall
15/15 OFF). The streaming+thinking finish=length caveat is non-repro
3/3 on this config — Status ⚠️→✅, caveat history kept as a RESOLVED
note, registry caveats→production, Quality lines refreshed. Drift +
parity + registry-json guards green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
--quantization modelopt matches NVIDIA's export only; unsloth's NVFP4
declares compressed-tensors and errors against the flag (first Discord
report 2026-07-12). Inline note: delete the flag pair for unsloth,
auto-detect takes over; provider A/B tied so either checkpoint is fine.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The probes section showed only single-leg examples and left the
REASONING=on boot requirement for ON probes buried in the full-eval
section. Example now pairs OFF+ON with the boot warning inline, and the
curated-set table carries measured runtimes (model-floor: 3.5 min OFF /
11 min ON single-draw on the Tess dual @ b9967; engine-window ~40 s).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
quality-test.sh passes through --scenario / --scenarios-file /
--incremental / --resume / --allow-partial (benchlocal #84/#85): bare
selections derive their pack set (custom mode), sandbox preflight fires
when a selection touches Docker packs, and --resume refuses
mode/selection/thinking/sampling/timeout flags rather than fork the
restored config. Selection runs print the partial-result warning.
scripts/scenario-sets/ ships the two curated Tess probe sets with
provenance headers: tess4-model-floor.txt (14 fails-everywhere + 2
thinking-only across 2 rigs / 2 drafters / 2 engine builds — the
retrain-target list) and tess4-engine-window.txt (CLI-25/31/32, the
b9932→b9967 discriminators for cheap engine-arm checks).
rerun-failed-packs.sh now re-runs a prior run's failures as ONE
selection run with --incremental durability (was: whole-pack loops).
Guards: test-scenario-sets.sh (format/provenance/passthrough) new,
test-rerun-failed-packs.sh updated; live-validated against the running
Tess serve (Set B: 3/3 pass, partial-labeled, sandbox auto-enabled).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Fresh-boot shipped compose on the new pin: 63.4/73.5 wall (+22%/+8% vs
b9246 row), MTP accept 0.737. EAGLE3 external draft (self-converted per
the #674 recipe) at n=2: 47.5/61.3, accept 0.408 — no EAGLE3 compose.
Head-vs-feed mechanism note recorded.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
AGENTS.md said llama.cpp tracks the rolling server-cuda tag, but it has
been build-pinned since the #187 crash-loop (b9246, now b9967 via #680)
with zero vendored patches. Codify that as pinning trigger (2): a
rolling tag that shipped a regression under a validated name earns a
stability pin, which outlives patches. Add the bump-on-a-cadence rule
(stale pins cost think-ON quality: b9246→b9967 +4) and point at
engine-pin-bump.sh for the mechanical half.
UPSTREAM.md: llama.cpp image row rewritten to the pinned reality
(consumers 2→8, #680 bump recorded); vllm-stable inventory row bumped
v0.22.0→v0.24.0 (stale since 2026-06-30) with the #682 hold-for-arm
staging noted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
3/3 streaming tool-calls with thinking-ON clean on the live shipped
serve (incl. a parallel 2-tool call); finish=tool_calls, reasoning
separated. Caveat retained pending the full-gate refresh; engine-build
vs budget attribution unresolved.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Sean's reproducible conversion recipe (metadata-only --target-model-dir
trick, draft-quant quality-free finding) folded into the row Notes per
his suggestion; links the #674 comment as canonical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
First single-RTX-4090 data point for Tess-4-27B with an Eagle3 draft on
mainline llama.cpp b9932. 8-pack quality-test.sh --full:
124/150 (82.7%) think-on · 115/150 (76.7%) think-off.
Matched to the same-rig 2026-07-09 DFlash baseline (125 on / 117 off) it
lands within 1-2 scenarios (single-run noise), confirming Eagle3 is
quality-equivalent to DFlash and that the min-p 0.75->0 sampler change
costs no measurable quality. Includes the spec-draft-n-max sweep showing
n_max=2 is optimal (deeper drafting is strictly worse for this model).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Generalizes what beellama-pin-bump.sh does for the one digest-pinned
engine to every engine profile: rewrites install.spec + the display_name
tag echo + every registered non-deprecated compose's image: default, and
REPORTS (never edits) the pin strings that encode validation claims —
pin-asserting fixtures, arch_patches.yml loads:true rows, UPSTREAM.md
rows, hand-written launcher suggestion strings, and prose/history —
bucketed so the six lines that need hands aren't buried under 100+
baselines/compose-header mentions. --check dry-runs with a unified diff.
The judgment half (patch re-validation, live boot + warm-up, trackers)
stays with the CLAUDE.md checklist, which the script prints. Paid for
by hand three times: the v0.22→v0.24 vLLM bump, the b9967 llama.cpp
bump (#680), and the upcoming v0.25 evaluation.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
b9246 was 52 days / 721 builds old. Validated on Tess dual (full quality
A/B, 2026-07-11): think-OFF total tie 115=115 (greedy-leg engine
isolation — totals neutral, though 4 scenarios flipped each way),
think-ON 122 vs the 118x3 pin — the lift is thinking-path-specific;
3 previously-100%-reproducible failures now pass (CLI-31 safety refusal,
CLI-20 archive precision, RM-04 math). Boots clean, draft-mtp works.
Also unifies the pin landscape: Deckard's b9570 and the ROLLING tags on
gemma-12b/vibethinker (a #187-policy violation — rolling crash-looped at
b9282) all move to the same validated build. Engine-profile spec now
matches the composes. Non-Tess slugs: boot re-validation is spot-checked
(smoke to follow on PR); bench numbers stay dated to their measured pin;
baselines rows go stale by design (the staleness badge is the feature).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
migtissera NVFP4 via Marlin W4A16 fallback: 62.4 tok/s decode spec-off
TP=2 @131K (BENCHMARKS 2026-07-11) vs the llama.cpp entry's 57.9 with
MTP — first vLLM-servable Tess on consumer cards. Ships experimental:
the 8-pack (the vLLM quality program A0 baseline) + stress/soak are the
promotion gate; pin-fallback rule to huginnfork NVFP4A16 documented in
compose + weights manual_note.
Catalog plumbing: nvfp4 weights variant; qwen35-dense added to
vllm-stable supported_model_families (live-validated); froggeric
template coverage extended to the tess compose (shared vendored copy,
patches.yml); kv_calc_supported=false on the model (hybrid — KV on
16/64 layers, naive math ~4x over; wire at promotion). Also finishes
the 2026-07-11 hybrid correction in the two spots today's sweep missed
(model YAML comment, llamacpp registry status_note). Counts 63→64 /
64→65. Full suite 77/78 (submit-bench = worktree fixture artifact);
diagnose-profile GREEN.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
benchlocal-cli's finest run granularity is --pack (no per-scenario run
filter — benchlocal-cli#82), so after a --full run the "are these
failures real?" question cost another 1-2 h full re-run. This script
parses a saved RunResult, re-runs ONLY the packs containing failures
(through quality-test.sh, keeping its hermes-env/timeout guards),
matches the original run's thinking mode from the JSON, passes
--previous-result for benchlocal's own per-scenario delta, and prints a
consolidated REPRODUCED / FIXED(flake) / NEW-regression verdict.
Supports --repeat N passthrough for flakiness rates and RERUN_DRY=1.
Guard test included.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Think-ON total neutral (118=118; baseline had no truncations to cure),
toolcall recovered 13 -> 15/15 under thinking. Default stays as cross-
rig insurance per Sean's #665 evidence. Full table in learnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Caps the think phase so reasoning can't consume the entire completion
budget when a user enables REASONING=on — the token_limit truncation
class @seanyourhighness eliminated in #665 (2 -> 0 on his 4090 run,
think-on 8-pack 125/150). Inert under the shipped thinking-off default.
Expected to mitigate the streaming-toolcall+thinking caveat (same
mechanism); first-party think-on validation running now — Quality line
update follows when it lands.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Piping a multi-hour quality run through tail buffers everything until
exit: no live progress, no partial results, interrupt loses the run.
Redirect to a file and summarize from it. Learned on the 2026-07-11
template A/B; benchlocal-cli#82 tracks the upstream resume half.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
"Production w/ caveats" means works-with-disclosed-limits, but ❗ reads
error-adjacent. The ✅ emoji can't be tinted (color-font green is baked
in), so use the text-presentation U+2714 checkmark styled orange1 —
same shape as production's check, color signals the caveat. Markup-
wrapped in _STATUS_GLYPH; the catalog DataTable cell now renders via
Text.from_markup (the other four call sites are already markup
contexts). Legend updated. c3 tests: 248 + 61 targeted headless pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Full gate complete on the reference 2x3090: verify-stress 8/8 (NIAH
clean to 240,635 tok = 91% of 262K, ceiling margin 1,801 MB, zero drift
across the ladder) + soak-continuous PASS (0 errors, 0/25 silent-empty,
0 VRAM growth, p50 215.9, 100% retention), joining the existing bench
(179.5/179.4 decode) and 8-pack (103/150 tie) results.
Caveats (why not full Production): streaming tool-calls + thinking-on
hit finish=length (known family class, verify-full check 6); native-FP4
quality remains unvalidated (all numbers are the Ampere W4A16 bound).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
In-worker apply_model probe: all 16 full-attention layers carry the
shipped calibrated values byte-exact (0.023-0.271, none at 1.0), so the
provider A/B was calibrated-scales vs scale=1.0 — and tied. Corrects the
"load-on-hybrid unverified, assume scale=1.0" claims in the nvfp4-fast
compose header, registry status_note, and both BENCHMARKS rows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Add vllm/qwen-35b-a3b-dual-nvfp4-fast: Ampere-validated unsloth NVFP4 MoE
New weights variant nvfp4-fast (unsloth/Qwen3.6-35B-A3B-NVFP4-Fast,
compressed-tensors mixed W4A4+FP8, auto-detected quant) + dual compose
(port 8080) + registry entry. First-party validated on the reference
2x3090 2026-07-11 — first MoE-FP4 fallback boot (MARLIN NvFp4 MoE
backend): decode 179.5/179.4 and 8-pack think-off 103/150, double
statistical tie with the AutoRound tier at full 262K (22.46 GB/card).
Ships 🧪 until verify-full/stress/soak run; native-FP4 (sm_90+) quality
remains the arc's missing datapoint (W4A4 activations unexercised on
Ampere). Registry counts 62→63, disk 63→64.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Fix MAX_ARG_STRLEN cliff: pass registry emit via temp file, not env
The full registry JSON (and the intermediate tab rows) were passed to
python heredocs through single environment variables. A single env value
is capped at MAX_ARG_STRLEN (~128 KB on Linux); the emit crossed it at
63 registry entries — "Argument list too long" in test-baselines and
test-registry-json, with the runtime REGISTRY_TAB pass in
registry-emit.sh sitting a few KB from the same cliff. All three now
write to a mktemp file and pass the path; python reads it with
encoding="utf-8".
Full suite: 76/77 green (test-submit-bench needs gitignored
results/rebench fixtures absent in a fresh worktree — env artifact).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Move nvfp4-fast slug to port 8081 — 8080 is Open WebUI on common rigs
Caught by the boot-the-actual-compose gate: the health probe on 8080 hit
Open WebUI and verify-full ran against the wrong service. 8081 is free
and unclaimed in the registry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Record actual gate results in nvfp4-fast compose header
switch.sh --force boot 272s + verify-full 7/1/2: the single failure is
streaming-toolcall+thinking-on finish=length — the known family caveat
class, not compose-specific. Stress/soak remain the 🧪→promotion gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Decode 66.9/90.4 vs 69.7/85.5, 8-pack 109 vs 110, NIAH clean to 240K.
MTP head loads through the compressed-tensors path (first observed).
Catalog verdict: no second 27B slug — nvidia-pinned stays, unsloth BYO.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
ADDING_MODELS.md: the canonical Step-4 and worked-example registry
entries pinned the deprecated vllm-nightly-mtp engine (purged image,
0 registry users) and the archived TQ3 KV path; the Step-6 calibration
schema taught vllm-nightly-<sha> + genesis_pin: v7.72.2 (both retired);
the worked-example weights block used the old list format the doc's own
schema section forbids. All examples now match current practice
(vllm-stable, release-tag pins, genesis_pin: null, map-format weights).
Added "The lightweight path" — the new-quant/new-slug checklist the doc
previously punted to a design-doc reference (full suite, diagnose-
profile, boot-the-actual-compose are the bolded skip-at-your-peril
items — each shipped a real gap this week).
AGENTS.md: at-a-glance routing line to the new section.
BRING_YOUR_OWN.md: pointer to the c3 Bring & Validate producer funnel
(same flow, guided) + --engine enum fixed to the documented
vllm|llama-cpp|sglang|other set.
Audit basis: full staleness sweep 2026-07-11 (every script/flag/path/
cross-link in both docs verified; BYO otherwise current).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
First confirmed MoE-FP4-on-Ampere boot (MARLIN NvFp4 MoE backend, weight-only
W4A16 execution). Decode 179.5/179.4 vs the shipped tier's 182.3/182.3 and
8-pack 103/150 vs 104-equivalent — statistical ties at full 262K. Native-FP4
rigs need their own quality run (W4A4 activations are not exercised here).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The safetensors config (model_type qwen3_5_text) shows 48 linear_attention
+ 16 full_attention layers (full_attention_interval=4). The earlier
"standard GQA, 64 layers" claim was inferred from the GGUF header, which
only describes the full-attention geometry — llama.cpp's qwen35 arch
implements the hybrid internally. No behavioral results change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Add spec-sweep.sh: draft-depth n-sweep for spec-decode drafters
The n-sweep has been hand-rolled repeatedly (Deckard MTP, Tess MTP
2026-07-09, EAGLE3 2026-07-11, gemma n=4-vs-8, DFlash n=5/7/8) — same
shape every time. This standardizes it as the tuning sibling of
concurrency-probe.sh (streams) and power-cap-sweep.sh (watts).
Two engine paths:
- llama.cpp FAST PATH (no reboots): per-request speculative.n_max +
timings.draft_n/draft_n_accepted/predicted_per_second — full curve
against one live server in ~2 min. Capability probe REFUSES (exit 3)
when the server doesn't report draft_n, so an ignored field can't
fabricate a flat curve.
- vLLM: reboot per arm via switch.sh (SPEC_N_MAX=<n>; SPEC=off for the
n=0 baseline arm) — no per-request draft-depth knob exists.
Output: curve table + sweet-spot line (flags a net-negative drafter
when n=0 wins) + machine-readable RESULT rows. SWEEP_DRY plans without
booting. Offline guards: scripts/tests/test-spec-sweep.sh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* spec-sweep: enforce the probe differ-check + llama.cpp reboot fallback
Live validation caught the gap: llama-server b9246 silently IGNORES
per-request speculative.n_max (probe draft_n identical across n) and the
first validation produced exactly the fake flat curve the probe was
designed to prevent — the differ-check was documented but not enforced.
- probe now requires draft_n(n1) != draft_n(n4); ignored field ->
reboot-per-arm fallback via MTP_DRAFT_N_MAX=<n> switch.sh (llama.cpp
boots ~15s; slug default config restored after the sweep)
- n=0 baseline guard: a compose with a hardcoded drafter that ignores
SPEC=off gets marked SPEC-OFF-IGNORED instead of lying
- validated against ground truth: reproduces the hand-rolled 2026-07-09
Tess MTP curve (n1 55.67/0.74 · n2 55.86/0.58 · n3 48.8/0.47 vs
55.6/0.75 · 57.9/0.62 · 49.3/0.46) in 3m27s
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First NVFP4 boot on Ampere — 69.7/85.5 decode, MTP n=3 accept 97-99%,
8-pack think-off 110/150 (ties the fp8 tier's 109). Capability/quality
datapoint behind the #670 unhide; not a recommended Ampere serving path
(no speed edge vs AutoRound).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
NVFP4 runs on sub-sm_90 cards via the Marlin W4A16 weight-only fallback
since vLLM v0.24 (floor = capability 7.5, marlin_utils_fp4). Live-confirmed
on 2x RTX 3090 sm_86 2026-07-11: the shipped dual/nvfp4/mtp.yml boots at
262K + fp8 KV + MTP n=3 (accept 97%+), 69.7/85.5 decode TPS, and scores
8-pack think-off 110/150 — a statistical tie with the fp8 production
tier's 109. c3 hiding these slugs as "incompatible-hw" was factually wrong.
Semantic split — required_sm stays the NATIVE floor; new registry field
`fallback_sm` = the weight-only-fallback floor:
- kv-calc: in the band [fallback_sm, required_sm) the fit is PRICED
normally and annotated with `hw_fallback` {required_sm, card_sm, note}
instead of returning incompatible-hw (both --fit and --fit-all).
- c3: FitVerdict carries hw_fallback; fits-* + fallback -> "⚑" glyph,
VISIBLE by default. incompatible-hw hiding stays for true incompatibles.
- gates.py/compat.py: the hard SM floor becomes fallback_sm when present
(else required_sm, unchanged).
- Registry: fallback_sm=7.5 on the 4 NVFP4 slugs + status_note honesty
pass ("REFUSES on Ampere" -> fallback reality + measured numbers);
same corrections in the 4 compose headers.
- test-kv-calc-fit section (f) rewritten to the new contract.
On Ampere these slugs still carry the honest downside in the note: no
speed edge (~20% slower than the AutoRound tier for the same model) —
their sub-sm_90 value is models where NVFP4 is the only quant.
Guards green: kv-calc-fit, registry-disk, status-drift, switch/launch
parity, profiles-compat, pullgate, pull, registry-json/emit, diagnose,
launch-compat, default resolvers, kv-calc --calibration; c3 248 fast
tests + headless incompatible-hw hide tests.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Agent-facing version of the eval guidance shipped user-facing in
ANNOUNCEMENT_TEMPLATE.md §7 (da063d7): behavioral quality (8-pack, both
reasoning modes, benchlocal + sandbox one-time setup, REASONING=on for
the thinking-ON leg) + operational health (report.sh --full), the
non-overlap guarantee, and the don't-pair-rebench-full-with-report-full
rule with the pick-by-goal split.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* concurrency-probe: surface aggregate tok/s in verdict, RESULT, and sweep
The probe already computed aggregate throughput (summed completion
tokens / wall) per round but only printed it in the round table — the
verdict, RESULT line, and sweep knee reported per-stream only. Users
asking "what total throughput at N agents?" (the Bruno question) had
no first-party number to point at.
- verdict: aggregate tok/s (N streams) next to per-stream
- RESULT: agg_tps= field (machine-readable, sweep-parseable)
- SWEEP: per-N summary table (per-stream vs aggregate) + aggregate in
the knee line
Knee logic unchanged (still per-stream-floor based). Offline guard
test-concurrency-probe passes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* concurrency-probe: re-resolve served model after each SWEEP boot
The top-of-script MODEL autodetect runs before SWEEP boots anything on
$URL, so it silently falls back to the default (qwen3.6-27b) — and every
request against any other model 404s, reading as errors=N per round (hit
live sweeping vllm/qwen-35b-a3b-dual on :8051; the 27B sweep only worked
because the fallback equals its served name). Re-resolve from /v1/models
after each boot; an explicit caller MODEL= pin is respected (MODEL_PINNED).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* FAQ + BENCHMARKS: first-party concurrency/aggregate-throughput matrix
Measured via the extended concurrency-probe (this PR) on 2x3090:
- dense 27B dual-fast: batching knee N=2 (~104 decode-agg @16K ctx),
collapses to ~54 by N=8; generation shape 211 agg @ N=8
- 35B-A3B MoE dual: ~250-270 decode-agg FLAT N=2..16 @16K ctx;
generation shape 1,037 tok/s aggregate @ N=16 (92.8/stream, clean)
New FAQ entry: how to serve multiple coding agents (MAX_NUM_SEQS,
ctx-vs-concurrency, model pick by architecture, honest aggregate-vs-
per-stream caveats, prefill-bound note, probe invocation).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@seanyourhighness diagnosed a hard HTTP 500 at turn 2 benching Tess-4-27B:
tool_choice=required + max_tokens=150 truncates the tool-call JSON on a
reasoning model (finish=length), and the harness replays that unterminated
`arguments` string in `messages` — llama-server then throws in
common/chat.cpp func_args_not_string() re-parsing the client history, so a
turn-1 clip poisons every subsequent turn.
Two-part fix:
- max_tokens 150 -> 600 (600 completes the call per the report).
- _safe_args guard: validate each accumulated tool_call `arguments` parses as
JSON before replaying it in `messages`; substitute "{}" if it doesn't, so a
single clip can't kill the whole ramp even if a slower model still truncates.
The upstream half (malformed client-supplied arguments -> HTTP 500 instead of
4xx, template-independent) is being filed against llama.cpp separately.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two independent RTX 5090 validations agree within noise:
- #619 (@paulp83): first NVFP4 boot on Blackwell — verify-full 9/9,
verify-stress needle-clean, soak-continuous PASS.
- #612 + #652 (@guybrush01): verify-full 9/9, verify-stress to 120K
(91% of 131K), soak PASS, p50 ~311 TPS. (#652 re-confirms #612 within
noise — one rig confirmed twice.)
status experimental -> caveats + compose header Status/Caveats. Caveat =
8-pack quality NOT yet cross-rig-measured (sandboxes weren't built on
these runs; quality run pending, gated on #492) — reverts to experimental
if a quality run regresses. NVFP4 stays Blackwell/Hopper-only (sm_90+).
Unchanged: 27b-single-nvfp4 (shipped util OOMs a 5090 — #613/#617, needs a
config fix) and both dual-nvfp4 slugs (no reports) remain experimental.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Standardize a reproduce/contribute section on model announcements:
build the benchlocal sandboxes, run quality-test.sh (8-pack, both
thinking) + report.sh --full (operational) — the two non-overlapping
passes. Bumps What'd help → §8, Credits → §9. First used on the
Tess-4-27B announcement (Discussions #662).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The `-h|--help` case ran `sed -n '2,55p' "$0"`, but the script cd's to the
repo root before arg-parsing — so a relative-path invocation from another
cwd left `$0` unresolvable and (under set -e) `--help` printed nothing. It
also dumped raw #-comment lines and truncated at line 55, omitting the
endpoint-first + env-override sections.
Replace with a `usage()` heredoc: cwd-independent, complete (all flags + env
overrides + examples), cleanly formatted. `-h` and `--help` both print it.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
n-sweep 2026-07-09 (dual 262K, server-side decode tok/s + draft
acceptance, 3 runs/arm after warm): n=2 peaks at 57.9 tok/s / 0.62
accept; throughput falls monotonically past it (34.9 / 0.27 @ n=6) as
Q4 draft acceptance craters. Confirms the shipped MTP_DRAFT_N_MAX=2
default (inherited from Deckard, now Tess-validated). Comment/doc note
only — no config change; full table in the per-model learnings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Pressing [D] on a brought GGUF repo ran pull.sh, the SAFETENSORS path — it aborts unsupported-format (no config.json) on a GGUF repo, so nothing downloaded ("download did not complete"). Route-G already handled fit + serve; download was the last gap.
- run_bring_download(gguf_includes=[...]): fetch those files directly into the pull dir via hf download --local-dir <pull> --include <pat>... with the hf-download.sh guards (HF_HUB_DISABLE_XET=1 resumable, HF_TOKEN from token file). Else pull.sh unchanged.
- app._gguf_download_includes(): picked quant file(s) + vision mmproj + quant-matched external MTP drafter glob. Wired into run_bring_download_worker.
Validated: mock-runner confirms the hf-download cmd vs pull.sh unchanged; a real hf download --include of the bartowski mmproj completed exit 0. +2 services tests, 232 green.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Route-G only wired spec-decode when a SEPARATE mtp-*.gguf drafter was present
(migtissera-style). A GGUF with an EMBEDDED MTP head (nextn baked into the main
quant — bartowski/unsloth-style) got the sibling's --spec-type stripped and
nothing re-added, so it served as a base model with no spec-decode.
- gguf_has_embedded_mtp(path): a minimal stdlib GGUF-metadata parser (the gguf
package isn't in the c3 venv) reads <arch>.nextn_predict_layers; >=1 = embedded
MTP. Early-returns before the big tokenizer arrays; never raises.
- _rewrite_gguf_command(embedded_mtp=...): external drafter → --spec-draft-model
+ --spec-type draft-mtp (unchanged, wins if both); else embedded → --spec-type
draft-mtp with NO draft model (activates the nextn head).
- run_gguf_emit_and_serve computes embedded_mtp on the main gguf when no external
drafter was found.
Live-validated on the embedded-MTP unsloth GGUF: llama.cpp logs "creating MTP
draft context against the target model" (self-speculation, no separate file),
"adding speculative implementation 'draft-mtp'", serves ("2+2" -> "Four").
Distinct from #654's external case ("loading draft model .../mtp-Tess..."). +5
tests (parser via synthetic mini-GGUF + both wiring branches). 230 services +
17 phase-4 green.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Clicking Serve for a brought GGUF (route-G) died with 'No module named
scripts': c3 runs from tools/serve-cockpit/, so the repo root isn't on sys.path,
yet emit_gguf_compose (and swap_apply for route-C) do
`from scripts.lib.profiles.compose_registry import …`. Only ONE of the three
call sites guarded this with a sys.path.insert; the emit site surfaced the
ImportError to the user, and fit-check's topology detection silently degraded.
Fix once in CockpitData.__init__ (add repo_root to sys.path) so every call site
works. Validated by simulating c3's runtime (repo root NOT pre-added): scripts
is unimportable before init, importable after, emit returns no error. This is
the exact condition #654's validation harness masked by inserting the path.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The emit-swap group asserted PRE-P2b behavior and crashed on any rig (not env-
specific): P2b (swap_apply served-name + SPEC env-gating) now emits
--served-model-name ${SERVED_NAME:-<name>} and LIFTS --speculative-config out of
the command into a ${SPEC:-on}-gated entrypoint (${DRAFTER_METHOD:-mtp}). The
test still checked the plain served-name + spec-config-in-command, so _val()
returned None on the moved flag -> 'x' in None -> TypeError crash. Updated 3
assertions to P2b's shape. test-pull-swap.sh now green (also unblocks the route-C
location asserts added in #655).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Companion to #654 (route-G): the safetensors weight-swap path had the same
"compose in the project tree" problem the user flagged — swap_apply.emit_swap_
compose wrote _brought-<san>.yml NEXT TO the sibling compose (so its relative
../ mounts resolved), dropping throwaway files beside catalog composes.
Route-C's MOUNT was already fine (it dir-mounts the brought weights to a distinct
/brought-model), so this is location-only: absolutize every sibling volume via a
new _absolutize_volume (${VAR:-../rel} → ${VAR:-/abs}, plain ../rel → /abs; the
vLLM sibling has 5 relative mounts — HF cache, torch_compile, triton, the
froggeric chat template, detect_nvlink) so the compose is relocatable, then write
it to the runtime dir beside the pull dir (<MODEL_DIR>/.cache/huggingface/
club3090/composes/), falling back beside the sibling.
Validated at emit level: all relative sources absolutized, ${MODEL_DIR} env var
preserved, brought mount intact, compose lands in the runtime dir not the repo.
+2 assertions in test-pull-swap.sh's emit group. (That .sh has a PRE-EXISTING
env failure on this rig — identical on master, unrelated.)
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First live boot of route-G (#650, maintainer-gated) surfaced two bugs the
compile-only tests couldn't:
- BOOT FAILURE (ExitCode 128): emit_gguf_compose bind-mounted the 3 brought
GGUFs as individual files INTO /models/brought.* — but the sibling already
mounts MODEL_DIR → /models, so OCI can't create those mountpoints inside an
already-mounted /models ("read-only file system"). Fix: the brought files live
UNDER MODEL_DIR (the pull dir), so address each at /models/<realpath-relative-
to-MODEL_DIR> (realpath follows the pull-dir symlink) and mount MODEL_DIR ONCE.
Files outside MODEL_DIR get their own /brought/<tag>-<name> mount (never /models).
A /models mount is guaranteed even if a sibling lacks one.
- WRONG LOCATION: the compose was written next to the sibling in the project
tree (to inherit relative mounts). Now the volume is absolute (MODEL_DIR →
/models, dropping the ${MODEL_DIR:-../rel} fallback) so the compose is
relocatable — written to a runtime dir on the model disk
(MODEL_DIR/.cache/huggingface/club3090/composes/), not the repo.
Live-validated end-to-end on the c3-EMITTED compose: Tess-4-27B-GGUF loads
(main + mmproj vision + the brought MTP drafter — spec-decode engages) and
serves ("capital of France" -> "Paris"). Tests: 3 new (single-mount, runtime-
dir, outside-MODEL_DIR) + 5 existing emit tests updated to /models/<rel>. 13
phase-4 + 229 services green.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a 'Where weights land' section: MODEL_DIR is the single root; curated
catalog models live top-level (compose-referenced), BYO pulls derive to
.cache/huggingface/club3090/pulls/<repo-slug>/ — the SAME path pull.sh and the
c3 Bring [D] button compute, so a pull by either is discovered by both. Makes
the two intentional tiers explicit for users bringing their own weights.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Live dogfood 2026-07-09 (migtissera/Tess-4-27B, 54G bf16 on a 48G rig): the §2b
topology floor correctly hid every slug (single ≤21.6G, dual ≤43.2G, no
multi-card slug), but the funnel showed only the ✎ custom-slug sentinel with no
explanation — and the custom Input never appeared. Two cascading bugs:
- Bug A: when the size floor empties the recommended list, surface an honest
verdict — "won't fit" (compat slugs exist but all exceed VRAM; bring a smaller
quant) vs "no catalog recipe matches this artifact" (no compatible engine at
all) — instead of a silent empty dropdown.
- Bug B: with only the sentinel option, _set_select_options pre-selects it under
prevent(Select.Changed), so the Changed-gated custom-Input reveal never fires
and re-picking the sole sentinel is a no-op. reveal_slug_stage now reveals the
Input eagerly iff the sentinel is the sole option (kept hidden when real slugs
exist).
Neither is a regression — the floor (#582) + reveal are byte-identical to
pre-Grok; a 55G repo is the first to expose the all-hidden edge. Tests:
test_uiux_funnel_oversized.py (custom-hatch reachable + won't-fit verdict + no
false verdict at normal size). 254 c3 tests green.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three review nits from the phase 1/3 merges, all copy/label only — no logic
change (245 c3 tests green):
- action_serve_untested: the user-facing dead-end toast said "the bring-your-own
weight-swap is a pending follow-up" — false since #644 wired apply-swap. Now
states the real reason (fit-check resolved no sibling recipe or catalog profile
to serve) and points back to ① Bring.
- Its docstring still claimed "this does NOT serve the brought model's weights ...
DEFERRED follow-up," contradicting its own now-wired Route-C body. Rewritten to
the two actual modes (brought-weights serve vs untested reproduction).
- Disk-fit preflight cards labeled sizes "GiB" while the math is decimal GB
(size_gb * 1e9 / free / 1e9). Relabeled the disk-fit size/free/need to "GB"
(both the catalog and bring preflight cards). VRAM "GiB" labels left as-is —
those are genuinely binary.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
