Community rigs hit two silent quality-test failure classes: (1) runs
"skipping" the sandboxed packs because the Docker images were never built
(the #492 warning exists but --sandboxed-only still warn-and-ran a
guaranteed-useless run), and (2) rigs that pulled a benchlocal-cli update
kept scoring on OLD sandbox images until told to rebuild manually — nothing
detected the drift. And report.sh collected none of this, so triage had to
ask.
quality-test.sh:
- --sandboxed-only + missing images/Docker → HARD FAIL up front with the
build instructions (a run with zero runnable packs is never intended).
--full keeps warn-and-continue (deterministic packs are still useful).
- NEW staleness heuristic: each sandbox image's Created is compared against
the benchlocal-cli console-script mtime (rewritten on every (re)install —
portable across pip-from-git AND editable-checkout installs). Image older
than CLI → warning naming the image + both dates + the rebuild command.
Explicitly labelled heuristic (an unrelated reinstall trips it) → WARN,
never abort.
report.sh:
- New "Quality tooling (benchlocal-cli + sandboxes)" section: CLI path /
version (via the console-script's own interpreter) / install date, the 4
sandbox images with build dates + per-image OLDER-than-CLI flags + rebuild
hint, and the newest results/quality/*.json. All best-effort; piped
through redact (verified leak-clean live).
Exact-version staleness (image labels + a `sandbox-status` command) needs
upstream benchlocal-cli changes — tracked in the todo.
Verified live: hard-fail branch (docker stub), warn-continue branch,
staleness branch (fresh CLI vs real images), report section rendering.
test-quality-baseline / test-quality-thinking / test-report-calib green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
ea9c233fef