benchlocal-cli already supports --max-tokens (overrides the per-pack ~1024
completion budget for BOTH arms, #28), but quality-test.sh only forwarded
--thinking-max-tokens. So a VERBOSE model that self-truncates the deterministic
packs (finish_reason=length before its final ANSWER:/</solution> line) could not
be benched at a higher budget through our wrapper.
Surfaced on llamacpp/qwen27b-pi-reasoning (a reasoning fine-tune that reasons in
visible content even thinking-off): ~5-10 reasonmath/bugfind/cli misses in the
thinking-OFF 8-pack were truncations, not wrong answers (e.g. RM-05 had 4/5
checkpoints right but ran out of tokens). The thinking-ON pass (16384 budget)
did not truncate — so the gap is purely the deterministic budget, which had no
knob in our wrapper until now.
- quality-test.sh: add --max-tokens N + MAX_TOKENS env passthrough (mirrors
--thinking-max-tokens exactly: same int validation, help, ENV doc, forward + echo).
- rebench-full.sh: plumb MAX_TOKENS into BOTH 8-pack passes (off + on) + doc it.
- test-quality-thinking.sh: assert env + flag forms forward --max-tokens and that
a non-integer is rejected (mock-benchlocal harness, hermetic).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
