The #665 finding written up as methodology: /150 totals are pass@1 at
pack-contract sampling; churners give a model a much higher pass@N
ceiling (~139 vs ~116-118 on Tess). Both halves get rules — verifier-
guided best-of-N as a legitimate client-side serving technique with a
sized cost table, and the reporting discipline that keeps harvested
numbers out of pass@1 columns. Credit to seanyourhighness's 7-draw
window.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The probes section showed only single-leg examples and left the
REASONING=on boot requirement for ON probes buried in the full-eval
section. Example now pairs OFF+ON with the boot warning inline, and the
curated-set table carries measured runtimes (model-floor: 3.5 min OFF /
11 min ON single-draw on the Tess dual @ b9967; engine-window ~40 s).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
quality-test.sh passes through --scenario / --scenarios-file /
--incremental / --resume / --allow-partial (benchlocal #84/#85): bare
selections derive their pack set (custom mode), sandbox preflight fires
when a selection touches Docker packs, and --resume refuses
mode/selection/thinking/sampling/timeout flags rather than fork the
restored config. Selection runs print the partial-result warning.
scripts/scenario-sets/ ships the two curated Tess probe sets with
provenance headers: tess4-model-floor.txt (14 fails-everywhere + 2
thinking-only across 2 rigs / 2 drafters / 2 engine builds — the
retrain-target list) and tess4-engine-window.txt (CLI-25/31/32, the
b9932→b9967 discriminators for cheap engine-arm checks).
rerun-failed-packs.sh now re-runs a prior run's failures as ONE
selection run with --incremental durability (was: whole-pack loops).
Guards: test-scenario-sets.sh (format/provenance/passthrough) new,
test-rerun-failed-packs.sh updated; live-validated against the running
Tess serve (Set B: 3/3 pass, partial-labeled, sandbox auto-enabled).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Closes the rescore-materialization gap the induction tool's first live
run surfaced (#562): the benchlocal-cli 'rescore' subcommand ALREADY
supports write-back (--in-place / --output) — the gap was practice, not
a missing upstream feature. The T2 rescore ran to stdout only, so the
published A1 thinking-on 110/150 diverged from the tag artifact (108).
- A1's rescore MATERIALIZED into the tag JSONs (rescore --in-place; OFF
105 unchanged, ON 108→110) — induction now extracts 110 and the
regenerated corpus record carries 110; artifact, corpus, baseline row
and publication all agree. Pre-rescore JSON backed up outside the tag.
- docs/QUALITY_TEST.md: 'Rescoring saved results — MATERIALIZE, don't
just read' — the rule (a rescore that changes a published number must
be written back in the same session), the command, what rescore can't
re-run (sandbox packs), and the A1 case as the cautionary example.
- catalog-baseline.sh header: reads-artifacts-as-truth warning + the
header's gate text updated to the actual n>=3/warn-under-5 behavior.
- baselines.yml A1 comment: gap note -> materialized + doc pointer.
Guards green (test-catalog-baseline, test-baselines).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The shared served-model-name `qwen3.6-27b-autoround` (from #490) mislabels the
non-autoround 27b scenes (fp8 dual-max, lmcache): /v1/models advertises
"autoround" while `root` points at qwen3.6-27b-fp8. Same class on gemma-4-31b —
after the v0.24.0 consolidation the default is cyankiwi qat-AWQ-INT4 (bf16 KV),
yet the LiteLLM route still targeted `gemma-4-31b-autoround` (a latent #482 drift).
Fix WITHOUT breaking anything, via vLLM multi-served-name:
- Every 27b scene now serves `qwen3.6-27b <its-quant-name>`; every gemma-31b
scene serves `gemma-4-31b <its-quant-name>`. The neutral name is PRIMARY
(honest /v1/models id); the quant-specific name is retained as a live ALIAS.
- LiteLLM: add `qwen3.6-27b` / `gemma-4-31b` canonical public routes; keep the
`-autoround` routes as back-compat aliases (same upstream). Repairs the gemma drift.
- Migrate our own MODEL= defaults + docs (bench/verify/quality/launch/setup, c3,
tui-core, EXAMPLES, ...) to the neutral name. Weights slugs (`-autoround-int4`)
untouched; CHANGELOG + results/ history left as-is.
Retiring the `-autoround` alias entirely is a deliberate later step once nothing
still asks for it.
Live-verified on-rig (single/minimal, stock v0.24.0): /v1/models lists BOTH names
(root=...-autoround-int4); chat to `qwen3.6-27b` AND `qwen3.6-27b-autoround` both
return 200; `qwen3.6-27b-fp8` correctly 404s. Full shell gate 59/59 (1 = known
worktree-fixture); c3 pytest 41 passed; served-name arg-order + YAML validated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
`quality-test.sh --full` (which docs tell cross-rig contributors to run)
needs the 3 sandboxed packs' Docker images — but they aren't auto-pulled,
and benchlocal-cli's mid-run hint pointed at a relative
`tools/build-sandboxes.sh` that's only valid inside a benchlocal-cli
checkout (absent on a pip install). guybrush01 followed it into a dead
path (#492).
- quality-test.sh: on --full / --sandboxed-only, preflight the
benchlocal-sandbox-{bugfind,cli,hermes} images UP FRONT. If missing,
print the correct clone + build steps (our context, not benchlocal-cli's
broken relative path) + the --medium no-Docker fallback, then continue
with the deterministic packs.
- QUALITY_TEST.md: document that --full needs the sandbox images built
first ("needs Docker" undersold it) — the clone + build-sandboxes.sh
step, with --medium / --no-sandboxed as the no-Docker path.
Full suite green (59/59). Pairs with benchlocal-cli#69 (the upstream hint).
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Phase 1 (offline): the wrapper, the corpus home, and the test gate. Phase 2
(GPU) curates the actual baselines against the live RECOMMENDED_DEFAULT_MODELS
configs.
- scripts/quality-baseline.sh: thin wrapper over quality-test.sh --full that
captures (--capture -> --save-json) or diffs (default -> --previous-result)
an n>=3 aggregate per (registry-slug, thinking-mode). no-thinking is
canonical (temp-0); enable-thinking is the reasoning-on companion. --dry-run
prints the resolved command; extra args pass through to benchlocal-cli.
- scripts/quality-test.sh: forward --repeat / --previous-result and honor a
--save-json path override, so the wrapper's blessed layout works.
- results/baselines/: committed corpus home (whitelisted in .gitignore) + a
README documenting the convention, usage, and an empty index table.
- scripts/tests/test-quality-baseline.sh: offline gate (--dry-run) — asserts
command/path resolution per mode + the required-slug / valid-mode /
positive-repeat / missing-baseline guards.
- docs/QUALITY_TEST.md: regression-baseline subsection pointing at the corpus.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
rebench-full ran the 8-pack quality eval (both think-OFF and think-ON,
~1.5-2.5 hr) by DEFAULT — the longest phase, dominating wall time even when
you only need "does this boot / serve / recall / soak". Flip it: the 8-pack
is now opt-in.
(omit) fast structural gates only (verify + bench + stress + soak)
--with-8pack-thinking 8-pack, reasoning OFF (--full --no-thinking)
--with-8pack-thinking=off same
--with-8pack-thinking=on 8-pack, reasoning ON (--full --enable-thinking)
--with-8pack-thinking=both both passes (the production-promotion gate)
The =off pass now FORCES --no-thinking (all 8 packs think-OFF) for a clean
with/without-reasoning A/B — previously bare `--full` used pack-default MIXED
thinking, which wasn't a true "reasoning off". quality-test.sh runs exactly one
mode per call, so "both" == two invocations; the think-ON pass fires ONLY for
=on/=both, never accidentally.
Naming realigned to benchlocal-cli #65's pack-set-vs-thinking-mode vocabulary
(the issue's --with-quality=off,on -> --with-8pack-thinking[=off|on|both];
"thinking" is explicit in the name so off/on can't be misread as "skip the
pack"). ENABLE_THINKING env no longer drives the 8-pack (only bench.sh).
Promotion-gate call sites updated to --with-8pack-thinking=both: ADDING_MODELS.md
Step 8, QUALITY_TEST.md, TQ3_MTP_GENESIS.md.
Validated: bash -n clean; bad-value guard exits 2 early; --help updated;
test-quality-thinking green (quality-test.sh untouched).
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The 'Failure breakdown:' block printed at the end of every run already shows
failure_mode + full detail per failed scenario — the quickest read. Add a
'Diagnosing failures' section (breakdown first, then JSON / benchlocal-cli
inspect for full trace / older runs / filter / diff) and a quality-test.sh
footer pointing at it. Fix the QUALITY_TEST.md sample, which showed a grouped
format + invented failure modes (missing_field/wrong_value) that the tool
never emits, to the real flat per-line format + real failure_mode vocabulary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
QUALITY_TEST.md gains a 'Per-scenario timeouts' section (precedence ladder,
over-budget rationale, the planned opt-in soak-derived per-depth tier #114) +
points to the benchlocal-cli README for the canonical formula/flags. CLAUDE.md
(agent guide) gains a concise pointer + the 'don't hand-set --timeout-per-case'
convention. Timeout-sizing was previously undocumented in both.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The benchlocal-cli `--progress` flag emits per-scenario `[N/M]` lines
to stderr — critical for the long-running modes (--full ~30-40 min,
--reasoning ~similar, --pack aider-polyglot-30 ~25-30 min) which
otherwise go dark for the whole duration. The wrapper never forwarded
the flag, so users (including agents) ran blind unless they bypassed
the wrapper.
Make --progress on by default, --no-progress as the opt-out for CI /
log-volume-sensitive contexts. Mirror PROGRESS=0/1 env var. Document
the convention in AGENTS.md so agents see it as policy, not just
private/personal memory.
Convention surfaced after wasting 25 min on a blind --full run during
autonomous validation: the wrapper logged its own progress line and
nothing else for the entire 30+ min window.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Symlink direction: repo convention is AGENTS.md canonical, CLAUDE.md a
symlink -> AGENTS.md (opposite of the maintainer system, where AGENTS.md ->
CLAUDE.md). The repo had it backwards (CLAUDE.md real, AGENTS.md -> CLAUDE.md);
flip so AGENTS.md is the real file and CLAUDE.md symlinks to it.
Localhost gotcha: HermesAgent-20 runs its agent INSIDE the Docker sandbox and
calls the model over the network, so a localhost endpoint is the container's
own loopback, not the host. scripts/quality-test.sh already auto-sets
BENCHLOCAL_HERMES_RESOLVE_LOCALHOST=1 for localhost URLs, but invoking
benchlocal-cli directly bypasses that guard and hermes silently scores ~0/20.
- AGENTS.md (now canonical): "run quality via the wrapper, not raw benchlocal-cli."
- docs/QUALITY_TEST.md: new Limitations item + annotate the direct-CLI example
(its localhost example was the exact trap). Failure signature is uniform
~timeout-length latencies + flat GPU, not turn_count (0 for hermes regardless).
Surfaced 2026-05-27: Gemma-4 8-pack hermes 1/20 (artifact) -> 13/20 with the var set.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#195.
Add ENABLE_THINKING support to bench.sh and pass --enable-thinking / --thinking-max-tokens through quality-test.sh. Propagate the same env through rebench-full.sh, warn on likely reasoning-on server/request-off mismatches, and document the workflow for reasoning-model evals.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
BugFind-15 / HermesAgent-20 / CLI-40 are no longer stubbed — they run as
Docker-sandboxed verifier packs, included in --full by default (--no-sandboxed
to skip). Correct the pack count (5 deterministic + 3 sandboxed = 8), the
--quick/--medium/--full mode definitions (ReasonMath is in --medium now), the
run examples (drop the dead ENABLE_SANDBOXED=stubbed note), Limitations, and
the illustrative output (5-pack --medium). Adds the AiderPolyglot-30 pointer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
QUALITY_TEST.md gains a 'Sampling & temperature' section (canonical temp-0
default vs --sampling-from-server vs --temperature; compose-recommended
defaults; per-pack-delta caveat). BENCHMARKS.md methodology note clarifies
the 8-pack runs at temp 0 by default + how to eval at a model's recommended temp.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lands the Gemma 4 31B work that was in flight last week + Qwopus3.6-27B
preview compose. Captures four distinct serving paths for Gemma 4:
- dual/awq.yml — AWQ-4bit (text-only, simplest path)
- dual/dflash-int8.yml — DFlash + INT8 PTH KV (262K ctx, full pipeline)
- vllm-gemma4-dflash-int8/ — vendored patches stacking DFlash spec-decode
+ INT8 PTH KV across model_executor + v1/spec_decode + v1/attention +
v1/worker (~13 patched files; PR #42102 + #40391-rebased + tool-parser
fixes#42006 + #41991 stacked)
- vllm-gemma4-fp8-ampere/ — earlier Phase 2 attempt before INT8 PTH
reframe (kept for forensics); Ampere has no native FP8 tensor cores
- vllm-perheadkv-hybridpage-fix/ — hybrid-page bug fix surfaced during
Phase 3
- vllm-pr40391-perheadkv/ — PR #40391 vendored at the tree level
(separate from the rebased variant under refs/jianc99-dflash-gemma4)
Plus:
- models/qwen3.6-27b/vllm/compose/dual/qwopus-bf16mtp.yml — preview compose
for Carnice AutoRound Recipe D output (port 8071, NOT production —
see club-3090-todo.md for known gaps + cheap A/Bs).
- AGENTS.md — codify compose naming + profile-schema + experimental-compose
conventions that the new files follow.
- docs/QUALITY_TEST.md — runbook for `quality-test.sh` + the benchlocal-cli
packs it wraps.
CHANGELOG.md narrative entries for these are added separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
