Diagnosis + state for making Hermes reliable on :8010. Root cause =
tool-call format mismatch (GLM markup falls through vLLM qwen3_coder
parser AND Hermes <tool_call>-only fallback). vLLM server verified
clean (12/12 non-stream, 8/8 stream); trigger is in Hermes's exact
request. Proxy now tees req+resp for next-step capture. 4 coldcase
bugs fixed live (audio_type, TTS promisify, clue-graph, phase labels).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TRR1YavMkQniZ7YYTwZ9dK
Adds SESSION-REPORT.md (LMCache tuning, coldcase big-model test, Hermes
agentic audio_type fix) and the L1 A/B run logs. Branch-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reconciles the earlier L2=0 finding. With L2 on (production shape), a 60 GB
L1 serves warm reads ~1.6x faster on average (~2x on the evicted half) than
a 30 GB L1 for a ~50 GB working set: older sessions that spill past a 30 GB
L1 fall to L2 disk (~3-5s) but stay in RAM at L1=60 (~1.3-2.4s).
Two-part model: L2 disk decides *whether* a read is warm (retention); L1 RAM
decides *how fast* (speed). L1 is a fast write-through front for L2, not a
standalone store — hence L2=0 gave 0/8 yet L2=1+L1=60 is fastest. Decision:
keep L1=60, it is not reclaimable headroom.
- Add SALT env to lmcache-retention-test.sh (fresh keys per A/B leg, no L2 collision)
- Add _local-lmcache-tuning/l1-ab-run.sh (two-leg orchestrator + comparison)
- Record result + reconciliation in HANDOFF.md
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Clean L2-off experiment: 50.6 GB working set, L2 on = 8/8 warm,
L2 off = 0/8 (all re-prefill 45-71s incl. MRU session 8). The warm
benefit is the disk tier; L1=60 RAM retained nothing standalone.
Corrects the earlier 'L1=60 retains the working set' claim.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Durability capture so session-ephemeral scratchpad work isn't lost:
HANDOFF.md, run1/run2 result logs, the l2off orchestrator, and a tmux
watcher that preserves the in-flight L2=0 result on completion.
Branch-only (rig-local test state) — never for master.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bench.sh's PREFILL_PROBE salts every request fresh, so it measures cold
prefill + the immediate warm hit but never fills past L1 capacity — it
can't show retention, which is the whole point of raising LMCACHE_L1_GB.
This wrapper inserts N distinct large prefixes (sum sized between the
30 GB and 60 GB capacity lines), then re-reads them: session 1 is
LRU-oldest, so Round-2 TTFT per session is the retention curve. All-warm
= L1 held the working set (would have evicted earliest at L1=30).
Reuses bench.sh's streaming-TTFT method; sizes haystacks via /tokenize.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rig-specific tuning — this rig has 124 GB RAM (110 free), so L1=60
clears the preflight gate (L1+28=88) with headroom. shm_size must be
>= L1 or the MP connector falls back to slow pickle.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cliff 3 (CLIFFS.md) walked readers to "use llama.cpp, vLLM is hopeless
for agentic Qwen3-Next" with zero mention that LMCache is the measured
mitigation for exactly that cliff (#423: 40K prefix warm-loads 0.5-2.4s
L1 vs 34-43s cold). Native prefix cache / generic CPU-offload hit Cliff
3; LMCache's HMA hybrid recurrent-state caching is the differentiator.
Add a bounded mitigation subsection (measured for resumed prefixes,
flagged untested for growing loops), revise the recommendation to two
paths, and forward-link INTERNALS' LMCache section to Cliff 3.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The #665 finding written up as methodology: /150 totals are pass@1 at
pack-contract sampling; churners give a model a much higher pass@N
ceiling (~139 vs ~116-118 on Tess). Both halves get rules — verifier-
guided best-of-N as a legitimate client-side serving technique with a
sized cost table, and the reporting discipline that keeps harvested
numbers out of pass@1 columns. Credit to seanyourhighness's 7-draw
window.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Two tiers per #665: Tier 1 = never passed anywhere (10 — capability
gap), Tier 2 = reachable but rare (6 — probability gap; per-scenario
evidence inline). RM-13 moved out of never-set on the vLLM A0 OFF pass,
shrinking Sean's proposed 11 to 10. Same 16 selections, guard green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
First 8-pack on vllm/tess-dual-nvfp4 (both modes, same-night GGUF
comparator 116/117). Gap is cli-40-concentrated (−8 off); deterministic
packs tie or better (RM-off 14/15 best-ever Tess). Per the
pre-registered rule the huginnfork NVFP4A16 weights-only A/B + FP8
precision arm are the next gates. BENCHMARKS vLLM subsection + registry
note + compose Quality line; Grok-§3 discriminator results recorded in
learnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Shipped-config quality refresh on b9967 + 16K budget (native template,
both modes, no pack regression vs the 2026-07-09 basis; first toolcall
15/15 OFF). The streaming+thinking finish=length caveat is non-repro
3/3 on this config — Status ⚠️→✅, caveat history kept as a RESOLVED
note, registry caveats→production, Quality lines refreshed. Drift +
parity + registry-json guards green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
--quantization modelopt matches NVIDIA's export only; unsloth's NVFP4
declares compressed-tensors and errors against the flag (first Discord
report 2026-07-12). Inline note: delete the flag pair for unsloth,
auto-detect takes over; provider A/B tied so either checkpoint is fine.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The probes section showed only single-leg examples and left the
REASONING=on boot requirement for ON probes buried in the full-eval
section. Example now pairs OFF+ON with the boot warning inline, and the
curated-set table carries measured runtimes (model-floor: 3.5 min OFF /
11 min ON single-draw on the Tess dual @ b9967; engine-window ~40 s).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
quality-test.sh passes through --scenario / --scenarios-file /
--incremental / --resume / --allow-partial (benchlocal #84/#85): bare
selections derive their pack set (custom mode), sandbox preflight fires
when a selection touches Docker packs, and --resume refuses
mode/selection/thinking/sampling/timeout flags rather than fork the
restored config. Selection runs print the partial-result warning.
scripts/scenario-sets/ ships the two curated Tess probe sets with
provenance headers: tess4-model-floor.txt (14 fails-everywhere + 2
thinking-only across 2 rigs / 2 drafters / 2 engine builds — the
retrain-target list) and tess4-engine-window.txt (CLI-25/31/32, the
b9932→b9967 discriminators for cheap engine-arm checks).
rerun-failed-packs.sh now re-runs a prior run's failures as ONE
selection run with --incremental durability (was: whole-pack loops).
Guards: test-scenario-sets.sh (format/provenance/passthrough) new,
test-rerun-failed-packs.sh updated; live-validated against the running
Tess serve (Set B: 3/3 pass, partial-labeled, sandbox auto-enabled).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Fresh-boot shipped compose on the new pin: 63.4/73.5 wall (+22%/+8% vs
b9246 row), MTP accept 0.737. EAGLE3 external draft (self-converted per
the #674 recipe) at n=2: 47.5/61.3, accept 0.408 — no EAGLE3 compose.
Head-vs-feed mechanism note recorded.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
AGENTS.md said llama.cpp tracks the rolling server-cuda tag, but it has
been build-pinned since the #187 crash-loop (b9246, now b9967 via #680)
with zero vendored patches. Codify that as pinning trigger (2): a
rolling tag that shipped a regression under a validated name earns a
stability pin, which outlives patches. Add the bump-on-a-cadence rule
(stale pins cost think-ON quality: b9246→b9967 +4) and point at
engine-pin-bump.sh for the mechanical half.
UPSTREAM.md: llama.cpp image row rewritten to the pinned reality
(consumers 2→8, #680 bump recorded); vllm-stable inventory row bumped
v0.22.0→v0.24.0 (stale since 2026-06-30) with the #682 hold-for-arm
staging noted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
3/3 streaming tool-calls with thinking-ON clean on the live shipped
serve (incl. a parallel 2-tool call); finish=tool_calls, reasoning
separated. Caveat retained pending the full-gate refresh; engine-build
vs budget attribution unresolved.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Sean's reproducible conversion recipe (metadata-only --target-model-dir
trick, draft-quant quality-free finding) folded into the row Notes per
his suggestion; links the #674 comment as canonical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
First single-RTX-4090 data point for Tess-4-27B with an Eagle3 draft on
mainline llama.cpp b9932. 8-pack quality-test.sh --full:
124/150 (82.7%) think-on · 115/150 (76.7%) think-off.
Matched to the same-rig 2026-07-09 DFlash baseline (125 on / 117 off) it
lands within 1-2 scenarios (single-run noise), confirming Eagle3 is
quality-equivalent to DFlash and that the min-p 0.75->0 sampler change
costs no measurable quality. Includes the spec-draft-n-max sweep showing
n_max=2 is optimal (deeper drafting is strictly worse for this model).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Generalizes what beellama-pin-bump.sh does for the one digest-pinned
engine to every engine profile: rewrites install.spec + the display_name
tag echo + every registered non-deprecated compose's image: default, and
REPORTS (never edits) the pin strings that encode validation claims —
pin-asserting fixtures, arch_patches.yml loads:true rows, UPSTREAM.md
rows, hand-written launcher suggestion strings, and prose/history —
bucketed so the six lines that need hands aren't buried under 100+
baselines/compose-header mentions. --check dry-runs with a unified diff.
The judgment half (patch re-validation, live boot + warm-up, trackers)
stays with the CLAUDE.md checklist, which the script prints. Paid for
by hand three times: the v0.22→v0.24 vLLM bump, the b9967 llama.cpp
bump (#680), and the upcoming v0.25 evaluation.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
b9246 was 52 days / 721 builds old. Validated on Tess dual (full quality
A/B, 2026-07-11): think-OFF total tie 115=115 (greedy-leg engine
isolation — totals neutral, though 4 scenarios flipped each way),
think-ON 122 vs the 118x3 pin — the lift is thinking-path-specific;
3 previously-100%-reproducible failures now pass (CLI-31 safety refusal,
CLI-20 archive precision, RM-04 math). Boots clean, draft-mtp works.
Also unifies the pin landscape: Deckard's b9570 and the ROLLING tags on
gemma-12b/vibethinker (a #187-policy violation — rolling crash-looped at
b9282) all move to the same validated build. Engine-profile spec now
matches the composes. Non-Tess slugs: boot re-validation is spot-checked
(smoke to follow on PR); bench numbers stay dated to their measured pin;
baselines rows go stale by design (the staleness badge is the feature).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
migtissera NVFP4 via Marlin W4A16 fallback: 62.4 tok/s decode spec-off
TP=2 @131K (BENCHMARKS 2026-07-11) vs the llama.cpp entry's 57.9 with
MTP — first vLLM-servable Tess on consumer cards. Ships experimental:
the 8-pack (the vLLM quality program A0 baseline) + stress/soak are the
promotion gate; pin-fallback rule to huginnfork NVFP4A16 documented in
compose + weights manual_note.
Catalog plumbing: nvfp4 weights variant; qwen35-dense added to
vllm-stable supported_model_families (live-validated); froggeric
template coverage extended to the tess compose (shared vendored copy,
patches.yml); kv_calc_supported=false on the model (hybrid — KV on
16/64 layers, naive math ~4x over; wire at promotion). Also finishes
the 2026-07-11 hybrid correction in the two spots today's sweep missed
(model YAML comment, llamacpp registry status_note). Counts 63→64 /
64→65. Full suite 77/78 (submit-bench = worktree fixture artifact);
diagnose-profile GREEN.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
benchlocal-cli's finest run granularity is --pack (no per-scenario run
filter — benchlocal-cli#82), so after a --full run the "are these
failures real?" question cost another 1-2 h full re-run. This script
parses a saved RunResult, re-runs ONLY the packs containing failures
(through quality-test.sh, keeping its hermes-env/timeout guards),
matches the original run's thinking mode from the JSON, passes
--previous-result for benchlocal's own per-scenario delta, and prints a
consolidated REPRODUCED / FIXED(flake) / NEW-regression verdict.
Supports --repeat N passthrough for flakiness rates and RERUN_DRY=1.
Guard test included.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Think-ON total neutral (118=118; baseline had no truncations to cure),
toolcall recovered 13 -> 15/15 under thinking. Default stays as cross-
rig insurance per Sean's #665 evidence. Full table in learnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Caps the think phase so reasoning can't consume the entire completion
budget when a user enables REASONING=on — the token_limit truncation
class @seanyourhighness eliminated in #665 (2 -> 0 on his 4090 run,
think-on 8-pack 125/150). Inert under the shipped thinking-off default.
Expected to mitigate the streaming-toolcall+thinking caveat (same
mechanism); first-party think-on validation running now — Quality line
update follows when it lands.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Piping a multi-hour quality run through tail buffers everything until
exit: no live progress, no partial results, interrupt loses the run.
Redirect to a file and summarize from it. Learned on the 2026-07-11
template A/B; benchlocal-cli#82 tracks the upstream resume half.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
"Production w/ caveats" means works-with-disclosed-limits, but ❗ reads
error-adjacent. The ✅ emoji can't be tinted (color-font green is baked
in), so use the text-presentation U+2714 checkmark styled orange1 —
same shape as production's check, color signals the caveat. Markup-
wrapped in _STATUS_GLYPH; the catalog DataTable cell now renders via
Text.from_markup (the other four call sites are already markup
contexts). Legend updated. c3 tests: 248 + 61 targeted headless pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Full gate complete on the reference 2x3090: verify-stress 8/8 (NIAH
clean to 240,635 tok = 91% of 262K, ceiling margin 1,801 MB, zero drift
across the ladder) + soak-continuous PASS (0 errors, 0/25 silent-empty,
0 VRAM growth, p50 215.9, 100% retention), joining the existing bench
(179.5/179.4 decode) and 8-pack (103/150 tie) results.
Caveats (why not full Production): streaming tool-calls + thinking-on
hit finish=length (known family class, verify-full check 6); native-FP4
quality remains unvalidated (all numbers are the Ampere W4A16 bound).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
In-worker apply_model probe: all 16 full-attention layers carry the
shipped calibrated values byte-exact (0.023-0.271, none at 1.0), so the
provider A/B was calibrated-scales vs scale=1.0 — and tied. Corrects the
"load-on-hybrid unverified, assume scale=1.0" claims in the nvfp4-fast
compose header, registry status_note, and both BENCHMARKS rows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Add vllm/qwen-35b-a3b-dual-nvfp4-fast: Ampere-validated unsloth NVFP4 MoE
New weights variant nvfp4-fast (unsloth/Qwen3.6-35B-A3B-NVFP4-Fast,
compressed-tensors mixed W4A4+FP8, auto-detected quant) + dual compose
(port 8080) + registry entry. First-party validated on the reference
2x3090 2026-07-11 — first MoE-FP4 fallback boot (MARLIN NvFp4 MoE
backend): decode 179.5/179.4 and 8-pack think-off 103/150, double
statistical tie with the AutoRound tier at full 262K (22.46 GB/card).
Ships 🧪 until verify-full/stress/soak run; native-FP4 (sm_90+) quality
remains the arc's missing datapoint (W4A4 activations unexercised on
Ampere). Registry counts 62→63, disk 63→64.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Fix MAX_ARG_STRLEN cliff: pass registry emit via temp file, not env
The full registry JSON (and the intermediate tab rows) were passed to
python heredocs through single environment variables. A single env value
is capped at MAX_ARG_STRLEN (~128 KB on Linux); the emit crossed it at
63 registry entries — "Argument list too long" in test-baselines and
test-registry-json, with the runtime REGISTRY_TAB pass in
registry-emit.sh sitting a few KB from the same cliff. All three now
write to a mktemp file and pass the path; python reads it with
encoding="utf-8".
Full suite: 76/77 green (test-submit-bench needs gitignored
results/rebench fixtures absent in a fresh worktree — env artifact).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Move nvfp4-fast slug to port 8081 — 8080 is Open WebUI on common rigs
Caught by the boot-the-actual-compose gate: the health probe on 8080 hit
Open WebUI and verify-full ran against the wrong service. 8081 is free
and unclaimed in the registry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Record actual gate results in nvfp4-fast compose header
switch.sh --force boot 272s + verify-full 7/1/2: the single failure is
streaming-toolcall+thinking-on finish=length — the known family caveat
class, not compose-specific. Stress/soak remain the 🧪→promotion gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Decode 66.9/90.4 vs 69.7/85.5, 8-pack 109 vs 110, NIAH clean to 240K.
MTP head loads through the compressed-tensors path (first observed).
Catalog verdict: no second 27B slug — nvidia-pinned stays, unsloth BYO.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
ADDING_MODELS.md: the canonical Step-4 and worked-example registry
entries pinned the deprecated vllm-nightly-mtp engine (purged image,
0 registry users) and the archived TQ3 KV path; the Step-6 calibration
schema taught vllm-nightly-<sha> + genesis_pin: v7.72.2 (both retired);
the worked-example weights block used the old list format the doc's own
schema section forbids. All examples now match current practice
(vllm-stable, release-tag pins, genesis_pin: null, map-format weights).
Added "The lightweight path" — the new-quant/new-slug checklist the doc
previously punted to a design-doc reference (full suite, diagnose-
profile, boot-the-actual-compose are the bolded skip-at-your-peril
items — each shipped a real gap this week).
AGENTS.md: at-a-glance routing line to the new section.
BRING_YOUR_OWN.md: pointer to the c3 Bring & Validate producer funnel
(same flow, guided) + --engine enum fixed to the documented
vllm|llama-cpp|sglang|other set.
Audit basis: full staleness sweep 2026-07-11 (every script/flag/path/
cross-link in both docs verified; BYO otherwise current).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
First confirmed MoE-FP4-on-Ampere boot (MARLIN NvFp4 MoE backend, weight-only
W4A16 execution). Decode 179.5/179.4 vs the shipped tier's 182.3/182.3 and
8-pack 103/150 vs 104-equivalent — statistical ties at full 262K. Native-FP4
rigs need their own quality run (W4A4 activations are not exercised here).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The safetensors config (model_type qwen3_5_text) shows 48 linear_attention
+ 16 full_attention layers (full_attention_interval=4). The earlier
"standard GQA, 64 layers" claim was inferred from the GGUF header, which
only describes the full-attention geometry — llama.cpp's qwen35 arch
implements the hybrid internally. No behavioral results change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* Add spec-sweep.sh: draft-depth n-sweep for spec-decode drafters
The n-sweep has been hand-rolled repeatedly (Deckard MTP, Tess MTP
2026-07-09, EAGLE3 2026-07-11, gemma n=4-vs-8, DFlash n=5/7/8) — same
shape every time. This standardizes it as the tuning sibling of
concurrency-probe.sh (streams) and power-cap-sweep.sh (watts).
Two engine paths:
- llama.cpp FAST PATH (no reboots): per-request speculative.n_max +
timings.draft_n/draft_n_accepted/predicted_per_second — full curve
against one live server in ~2 min. Capability probe REFUSES (exit 3)
when the server doesn't report draft_n, so an ignored field can't
fabricate a flat curve.
- vLLM: reboot per arm via switch.sh (SPEC_N_MAX=<n>; SPEC=off for the
n=0 baseline arm) — no per-request draft-depth knob exists.
Output: curve table + sweet-spot line (flags a net-negative drafter
when n=0 wins) + machine-readable RESULT rows. SWEEP_DRY plans without
booting. Offline guards: scripts/tests/test-spec-sweep.sh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* spec-sweep: enforce the probe differ-check + llama.cpp reboot fallback
Live validation caught the gap: llama-server b9246 silently IGNORES
per-request speculative.n_max (probe draft_n identical across n) and the
first validation produced exactly the fake flat curve the probe was
designed to prevent — the differ-check was documented but not enforced.
- probe now requires draft_n(n1) != draft_n(n4); ignored field ->
reboot-per-arm fallback via MTP_DRAFT_N_MAX=<n> switch.sh (llama.cpp
boots ~15s; slug default config restored after the sweep)
- n=0 baseline guard: a compose with a hardcoded drafter that ignores
SPEC=off gets marked SPEC-OFF-IGNORED instead of lying
- validated against ground truth: reproduces the hand-rolled 2026-07-09
Tess MTP curve (n1 55.67/0.74 · n2 55.86/0.58 · n3 48.8/0.47 vs
55.6/0.75 · 57.9/0.62 · 49.3/0.46) in 3m27s
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First NVFP4 boot on Ampere — 69.7/85.5 decode, MTP n=3 accept 97-99%,
8-pack think-off 110/150 (ties the fp8 tier's 109). Capability/quality
datapoint behind the #670 unhide; not a recommended Ampere serving path
(no speed edge vs AutoRound).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
NVFP4 runs on sub-sm_90 cards via the Marlin W4A16 weight-only fallback
since vLLM v0.24 (floor = capability 7.5, marlin_utils_fp4). Live-confirmed
on 2x RTX 3090 sm_86 2026-07-11: the shipped dual/nvfp4/mtp.yml boots at
262K + fp8 KV + MTP n=3 (accept 97%+), 69.7/85.5 decode TPS, and scores
8-pack think-off 110/150 — a statistical tie with the fp8 production
tier's 109. c3 hiding these slugs as "incompatible-hw" was factually wrong.
Semantic split — required_sm stays the NATIVE floor; new registry field
`fallback_sm` = the weight-only-fallback floor:
- kv-calc: in the band [fallback_sm, required_sm) the fit is PRICED
normally and annotated with `hw_fallback` {required_sm, card_sm, note}
instead of returning incompatible-hw (both --fit and --fit-all).
- c3: FitVerdict carries hw_fallback; fits-* + fallback -> "⚑" glyph,
VISIBLE by default. incompatible-hw hiding stays for true incompatibles.
- gates.py/compat.py: the hard SM floor becomes fallback_sm when present
(else required_sm, unchanged).
- Registry: fallback_sm=7.5 on the 4 NVFP4 slugs + status_note honesty
pass ("REFUSES on Ampere" -> fallback reality + measured numbers);
same corrections in the 4 compose headers.
- test-kv-calc-fit section (f) rewritten to the new contract.
On Ampere these slugs still carry the honest downside in the note: no
speed edge (~20% slower than the AutoRound tier for the same model) —
their sub-sm_90 value is models where NVFP4 is the only quant.
Guards green: kv-calc-fit, registry-disk, status-drift, switch/launch
parity, profiles-compat, pullgate, pull, registry-json/emit, diagnose,
launch-compat, default resolvers, kv-calc --calibration; c3 248 fast
tests + headless incompatible-hw hide tests.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Agent-facing version of the eval guidance shipped user-facing in
ANNOUNCEMENT_TEMPLATE.md §7 (da063d7): behavioral quality (8-pack, both
reasoning modes, benchlocal + sandbox one-time setup, REASONING=on for
the thinking-ON leg) + operational health (report.sh --full), the
non-overlap guarantee, and the don't-pair-rebench-full-with-report-full
rule with the pick-by-goal split.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* concurrency-probe: surface aggregate tok/s in verdict, RESULT, and sweep
The probe already computed aggregate throughput (summed completion
tokens / wall) per round but only printed it in the round table — the
verdict, RESULT line, and sweep knee reported per-stream only. Users
asking "what total throughput at N agents?" (the Bruno question) had
no first-party number to point at.
- verdict: aggregate tok/s (N streams) next to per-stream
- RESULT: agg_tps= field (machine-readable, sweep-parseable)
- SWEEP: per-N summary table (per-stream vs aggregate) + aggregate in
the knee line
Knee logic unchanged (still per-stream-floor based). Offline guard
test-concurrency-probe passes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* concurrency-probe: re-resolve served model after each SWEEP boot
The top-of-script MODEL autodetect runs before SWEEP boots anything on
$URL, so it silently falls back to the default (qwen3.6-27b) — and every
request against any other model 404s, reading as errors=N per round (hit
live sweeping vllm/qwen-35b-a3b-dual on :8051; the 27B sweep only worked
because the fallback equals its served name). Re-resolve from /v1/models
after each boot; an explicit caller MODEL= pin is respected (MODEL_PINNED).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* FAQ + BENCHMARKS: first-party concurrency/aggregate-throughput matrix
Measured via the extended concurrency-probe (this PR) on 2x3090:
- dense 27B dual-fast: batching knee N=2 (~104 decode-agg @16K ctx),
collapses to ~54 by N=8; generation shape 211 agg @ N=8
- 35B-A3B MoE dual: ~250-270 decode-agg FLAT N=2..16 @16K ctx;
generation shape 1,037 tok/s aggregate @ N=16 (92.8/stream, clean)
New FAQ entry: how to serve multiple coding agents (MAX_NUM_SEQS,
ctx-vs-concurrency, model pick by architecture, honest aggregate-vs-
per-stream caveats, prefill-bound note, probe invocation).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@seanyourhighness diagnosed a hard HTTP 500 at turn 2 benching Tess-4-27B:
tool_choice=required + max_tokens=150 truncates the tool-call JSON on a
reasoning model (finish=length), and the harness replays that unterminated
`arguments` string in `messages` — llama-server then throws in
common/chat.cpp func_args_not_string() re-parsing the client history, so a
turn-1 clip poisons every subsequent turn.
Two-part fix:
- max_tokens 150 -> 600 (600 completes the call per the report).
- _safe_args guard: validate each accumulated tool_call `arguments` parses as
JSON before replaying it in `messages`; substitute "{}" if it doesn't, so a
single clip can't kill the whole ramp even if a slower model still truncates.
The upstream half (malformed client-supplied arguments -> HTTP 500 instead of
4xx, template-independent) is being filed against llama.cpp separately.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two independent RTX 5090 validations agree within noise:
- #619 (@paulp83): first NVFP4 boot on Blackwell — verify-full 9/9,
verify-stress needle-clean, soak-continuous PASS.
- #612 + #652 (@guybrush01): verify-full 9/9, verify-stress to 120K
(91% of 131K), soak PASS, p50 ~311 TPS. (#652 re-confirms #612 within
noise — one rig confirmed twice.)
status experimental -> caveats + compose header Status/Caveats. Caveat =
8-pack quality NOT yet cross-rig-measured (sandboxes weren't built on
these runs; quality run pending, gated on #492) — reverts to experimental
if a quality run regresses. NVFP4 stays Blackwell/Hopper-only (sm_90+).
Unchanged: 27b-single-nvfp4 (shipped util OOMs a 5090 — #613/#617, needs a
config fix) and both dual-nvfp4 slugs (no reports) remain experimental.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Standardize a reproduce/contribute section on model announcements:
build the benchlocal sandboxes, run quality-test.sh (8-pack, both
thinking) + report.sh --full (operational) — the two non-overlapping
passes. Bumps What'd help → §8, Credits → §9. First used on the
Tess-4-27B announcement (Discussions #662).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The `-h|--help` case ran `sed -n '2,55p' "$0"`, but the script cd's to the
repo root before arg-parsing — so a relative-path invocation from another
cwd left `$0` unresolvable and (under set -e) `--help` printed nothing. It
also dumped raw #-comment lines and truncated at line 55, omitting the
endpoint-first + env-override sections.
Replace with a `usage()` heredoc: cwd-independent, complete (all flags + env
overrides + examples), cleanly formatted. `-h` and `--help` both print it.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
n-sweep 2026-07-09 (dual 262K, server-side decode tok/s + draft
acceptance, 3 runs/arm after warm): n=2 peaks at 57.9 tok/s / 0.62
accept; throughput falls monotonically past it (34.9 / 0.27 @ n=6) as
Q4 draft acceptance craters. Confirms the shipped MTP_DRAFT_N_MAX=2
default (inherited from Deckard, now Tess-validated). Comment/doc note
only — no config change; full table in the per-model learnings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
