Re-ran the agent comparison now that MTP-off removed the tool-call
corruption. 4 seeded bug classes x 2 runs x 2 agents, same backend,
same prompts, graders validated (seeded fails / clean passes).
PI 8/8 HERMES 8/8
The earlier 'Pi 3/3 vs Hermes 2/3' was measured on the corrupting
server -- it was the SERVER, not the agent. On a clean server the two
are indistinguishable on these tasks.
Two of MY grader bugs surfaced and were fixed before scoring:
- t2 want_any held 'Number(' / 'String(id)' = invalid regex. Hidden
because parseInt matched first (any() short-circuits); it crashed
only on a fix that avoided parseInt, scoring a PASS as FAIL.
- t3 accepted only one of two valid fixes. generateMysteryImages
declares exec = promisify(execFile) but never calls exec -- dead
code even in the clean original. Deleting it is more minimal than
restoring imports for an unused var. Both agents found that; the
grader called it FAIL.
Uncorrected the bench read 7/8 vs 6/8 -- both were grader artifacts.
213d888c90