The 'Failure breakdown:' block printed at the end of every run already shows
failure_mode + full detail per failed scenario — the quickest read. Add a
'Diagnosing failures' section (breakdown first, then JSON / benchlocal-cli
inspect for full trace / older runs / filter / diff) and a quality-test.sh
footer pointing at it. Fix the QUALITY_TEST.md sample, which showed a grouped
format + invented failure modes (missing_field/wrong_value) that the tool
never emits, to the real flat per-line format + real failure_mode vocabulary.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
QUALITY_TEST.md gains a 'Per-scenario timeouts' section (precedence ladder,
over-budget rationale, the planned opt-in soak-derived per-depth tier #114) +
points to the benchlocal-cli README for the canonical formula/flags. CLAUDE.md
(agent guide) gains a concise pointer + the 'don't hand-set --timeout-per-case'
convention. Timeout-sizing was previously undocumented in both.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The benchlocal-cli `--progress` flag emits per-scenario `[N/M]` lines
to stderr — critical for the long-running modes (--full ~30-40 min,
--reasoning ~similar, --pack aider-polyglot-30 ~25-30 min) which
otherwise go dark for the whole duration. The wrapper never forwarded
the flag, so users (including agents) ran blind unless they bypassed
the wrapper.
Make --progress on by default, --no-progress as the opt-out for CI /
log-volume-sensitive contexts. Mirror PROGRESS=0/1 env var. Document
the convention in AGENTS.md so agents see it as policy, not just
private/personal memory.
Convention surfaced after wasting 25 min on a blind --full run during
autonomous validation: the wrapper logged its own progress line and
nothing else for the entire 30+ min window.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Symlink direction: repo convention is AGENTS.md canonical, CLAUDE.md a
symlink -> AGENTS.md (opposite of the maintainer system, where AGENTS.md ->
CLAUDE.md). The repo had it backwards (CLAUDE.md real, AGENTS.md -> CLAUDE.md);
flip so AGENTS.md is the real file and CLAUDE.md symlinks to it.
Localhost gotcha: HermesAgent-20 runs its agent INSIDE the Docker sandbox and
calls the model over the network, so a localhost endpoint is the container's
own loopback, not the host. scripts/quality-test.sh already auto-sets
BENCHLOCAL_HERMES_RESOLVE_LOCALHOST=1 for localhost URLs, but invoking
benchlocal-cli directly bypasses that guard and hermes silently scores ~0/20.
- AGENTS.md (now canonical): "run quality via the wrapper, not raw benchlocal-cli."
- docs/QUALITY_TEST.md: new Limitations item + annotate the direct-CLI example
(its localhost example was the exact trap). Failure signature is uniform
~timeout-length latencies + flat GPU, not turn_count (0 for hermes regardless).
Surfaced 2026-05-27: Gemma-4 8-pack hermes 1/20 (artifact) -> 13/20 with the var set.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes#195.
Add ENABLE_THINKING support to bench.sh and pass --enable-thinking / --thinking-max-tokens through quality-test.sh. Propagate the same env through rebench-full.sh, warn on likely reasoning-on server/request-off mismatches, and document the workflow for reasoning-model evals.
Co-authored-by: noonghunna <[email protected]>
BugFind-15 / HermesAgent-20 / CLI-40 are no longer stubbed — they run as
Docker-sandboxed verifier packs, included in --full by default (--no-sandboxed
to skip). Correct the pack count (5 deterministic + 3 sandboxed = 8), the
--quick/--medium/--full mode definitions (ReasonMath is in --medium now), the
run examples (drop the dead ENABLE_SANDBOXED=stubbed note), Limitations, and
the illustrative output (5-pack --medium). Adds the AiderPolyglot-30 pointer.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
QUALITY_TEST.md gains a 'Sampling & temperature' section (canonical temp-0
default vs --sampling-from-server vs --temperature; compose-recommended
defaults; per-pack-delta caveat). BENCHMARKS.md methodology note clarifies
the 8-pack runs at temp 0 by default + how to eval at a model's recommended temp.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Lands the Gemma 4 31B work that was in flight last week + Qwopus3.6-27B
preview compose. Captures four distinct serving paths for Gemma 4:
- dual/awq.yml — AWQ-4bit (text-only, simplest path)
- dual/dflash-int8.yml — DFlash + INT8 PTH KV (262K ctx, full pipeline)
- vllm-gemma4-dflash-int8/ — vendored patches stacking DFlash spec-decode
+ INT8 PTH KV across model_executor + v1/spec_decode + v1/attention +
v1/worker (~13 patched files; PR #42102 + #40391-rebased + tool-parser
fixes#42006 + #41991 stacked)
- vllm-gemma4-fp8-ampere/ — earlier Phase 2 attempt before INT8 PTH
reframe (kept for forensics); Ampere has no native FP8 tensor cores
- vllm-perheadkv-hybridpage-fix/ — hybrid-page bug fix surfaced during
Phase 3
- vllm-pr40391-perheadkv/ — PR #40391 vendored at the tree level
(separate from the rebased variant under refs/jianc99-dflash-gemma4)
Plus:
- models/qwen3.6-27b/vllm/compose/dual/qwopus-bf16mtp.yml — preview compose
for Carnice AutoRound Recipe D output (port 8071, NOT production —
see club-3090-todo.md for known gaps + cheap A/Bs).
- AGENTS.md — codify compose naming + profile-schema + experimental-compose
conventions that the new files follow.
- docs/QUALITY_TEST.md — runbook for `quality-test.sh` + the benchlocal-cli
packs it wraps.
CHANGELOG.md narrative entries for these are added separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
