alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions 38 Packages Projects Releases Wiki Activity

docs: document quality-test timeout sizing (QUALITY_TEST.md + agent guide)

Browse Source 
QUALITY_TEST.md gains a 'Per-scenario timeouts' section (precedence ladder,
over-budget rationale, the planned opt-in soak-derived per-depth tier #114) +
points to the benchlocal-cli README for the canonical formula/flags. CLAUDE.md
(agent guide) gains a concise pointer + the 'don't hand-set --timeout-per-case'
convention. Timeout-sizing was previously undocumented in both.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
noonghunna
2026-05-28 23:11:56 +00:00
parent a289eccd5a
commit 9c333f42bb
2 changed files with 13 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
AGENTS.md
View File

@@ -199,7 +199,7 @@ References (orphan composes / patches sitting in this state as of 2026-05-09):
- `verify-full.sh` — functional (~1-2 min). Runs on every compose change.
- `verify-stress.sh` — boundary cases (longctx ladder + tool-prefill OOM ~5-10 min). Runs on cliff-related changes.
- `bench.sh` — canonical TPS bench (~3-5 min). Run when you change anything that could move TPS (compose flags, Genesis env vars, vLLM pin).
- `quality-test.sh` — behavioral quality (~10-30 min depending on `--quick` / `--medium` / `--full`). Wraps [`benchlocal-cli`](https://github.com/noonghunna/benchlocal-cli) — runs verifier-backed bench packs (ToolCall-15, InstructFollow-15, StructOutput-15, etc) against the running endpoint. Catches what operational tests miss: a compose can pass verify + stress + bench + soak and still ship with degraded tool-call accuracy or instruction-follow drift from quantization or Genesis env-flips. Run before promoting `Status: ✅ Production` and before any pin bump that could shift behavior. **Run via this wrapper, not raw `benchlocal-cli`** — it auto-detects endpoint/model and, for localhost URLs, sets `BENCHLOCAL_HERMES_RESOLVE_LOCALHOST=1` so the sandboxed HermesAgent can reach the host model (direct `benchlocal-cli` skips that → hermes silently scores ~0/20). **Live per-scenario `[N/M]` progress is on by default** (the wrapper forwards `--progress` to benchlocal-cli) so long runs don't go dark; pass `--no-progress` only for CI / log-volume contexts. See [`docs/QUALITY_TEST.md`](docs/QUALITY_TEST.md).
- `quality-test.sh` — behavioral quality (~10-30 min depending on `--quick` / `--medium` / `--full`). Wraps [`benchlocal-cli`](https://github.com/noonghunna/benchlocal-cli) — runs verifier-backed bench packs (ToolCall-15, InstructFollow-15, StructOutput-15, etc) against the running endpoint. Catches what operational tests miss: a compose can pass verify + stress + bench + soak and still ship with degraded tool-call accuracy or instruction-follow drift from quantization or Genesis env-flips. Run before promoting `Status: ✅ Production` and before any pin bump that could shift behavior. **Run via this wrapper, not raw `benchlocal-cli`** — it auto-detects endpoint/model and, for localhost URLs, sets `BENCHLOCAL_HERMES_RESOLVE_LOCALHOST=1` so the sandboxed HermesAgent can reach the host model (direct `benchlocal-cli` skips that → hermes silently scores ~0/20). **Live per-scenario `[N/M]` progress is on by default** (the wrapper forwards `--progress` to benchlocal-cli) so long runs don't go dark; pass `--no-progress` only for CI / log-volume contexts. **Timeout sizing:** the wrapper auto-scales per-scenario timeouts (startup decode-TPS probe × thinking-token multiplier), deliberately over-budgeting — the fix for the thinking-on spurious-timeout class (benchlocal-cli #54/#59). **Don't hand-set `--timeout-per-case` to "fix" a slow run unless you've confirmed the probe measured wrong.** A planned opt-in tier will size from a soak-derived per-depth TPS curve (#114). Full precedence + flags → [`docs/QUALITY_TEST.md`](docs/QUALITY_TEST.md) "Per-scenario timeouts".
- `soak-test.sh` — stability (30-60 min). Run before shipping config / Genesis / memory-policy changes — catches Cliff 2b.
The pipeline is layered: each script has a different question it answers ("does it serve / work / survive / fast / behave correctly / stay healthy"). Skipping any layer can mask regressions.

12
docs/QUALITY_TEST.md
View File

@@ -147,6 +147,18 @@ Quality: line for compose schema field (paste into compose YAML header):
Quality: ToolCall-15 14/15 (93%) · InstructFollow-15 13/15 (87%) · StructOutput-15 15/15 (100%) · DataExtract-15 12/15 (80%) · ReasonMath-15 11/15 (73%) (--medium, packs v1.0.x, 2026-05-09)
```
## Per-scenario timeouts
`quality-test.sh` forwards to `benchlocal-cli`, which sizes each scenario's timeout automatically — you rarely need to set one. Precedence (highest wins):
1. **Manual** — `--timeout-per-case N` (or `TIMEOUT_PER_CASE=N`): used verbatim.
2. **Auto-scaling (default)** — the budget scales by the endpoint's measured decode speed and, for thinking-on runs, by the thinking-token budget. A one-shot startup probe measures the rig's decode TPS (and fails fast if the endpoint is unreachable, rather than hanging). The scaling deliberately **over-budgets** — a timeout is a safety ceiling, not a target — which is what keeps thinking-on packs from spuriously timing out. Exact formula + flags (`--measured-tps` / `--reference-tps` / `--retry-on-timeout`): [benchlocal-cli README → Per-case timeouts](https://github.com/noonghunna/benchlocal-cli#per-case-timeouts).
3. **Static default** — the pack's built-in `default_max_seconds`.
**Don't hand-set `--timeout-per-case` to "fix" a slow run** unless you've confirmed the auto-probe measured wrong — the over-budget is intentional.
> **Planned (not yet built):** an *opt-in* tier that sizes timeouts from a **soak-derived per-context-depth TPS curve** — a real "how fast at depth X" measurement for your exact rig/config, captured into the runtime measurement-record — instead of the single empty-context startup probe. It would be strictly opt-in and fall back to the auto-probe/default when no curve exists; measured data is never required. Tracked at [#114](https://github.com/noonghunna/club-3090/pull/114).
## Sampling & temperature
By default the packs sample at **temperature 0** (greedy) — deterministic and reproducible, so scores are comparable across rigs and across runs. This is the **canonical** baseline, and it's what regression tracking and cross-config ranking should use.