diff --git a/AGENTS.md b/AGENTS.md index 3b2f9c06..f2f664ce 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -199,7 +199,7 @@ References (orphan composes / patches sitting in this state as of 2026-05-09): - `verify-full.sh` — functional (~1-2 min). Runs on every compose change. - `verify-stress.sh` — boundary cases (longctx ladder + tool-prefill OOM ~5-10 min). Runs on cliff-related changes. - `bench.sh` — canonical TPS bench (~3-5 min). Run when you change anything that could move TPS (compose flags, Genesis env vars, vLLM pin). -- `quality-test.sh` — behavioral quality (~10-30 min depending on `--quick` / `--medium` / `--full`). Wraps [`benchlocal-cli`](https://github.com/noonghunna/benchlocal-cli) — runs verifier-backed bench packs (ToolCall-15, InstructFollow-15, StructOutput-15, etc) against the running endpoint. Catches what operational tests miss: a compose can pass verify + stress + bench + soak and still ship with degraded tool-call accuracy or instruction-follow drift from quantization or Genesis env-flips. Run before promoting `Status: ✅ Production` and before any pin bump that could shift behavior. **Run via this wrapper, not raw `benchlocal-cli`** — it auto-detects endpoint/model and, for localhost URLs, sets `BENCHLOCAL_HERMES_RESOLVE_LOCALHOST=1` so the sandboxed HermesAgent can reach the host model (direct `benchlocal-cli` skips that → hermes silently scores ~0/20). **Live per-scenario `[N/M]` progress is on by default** (the wrapper forwards `--progress` to benchlocal-cli) so long runs don't go dark; pass `--no-progress` only for CI / log-volume contexts. See [`docs/QUALITY_TEST.md`](docs/QUALITY_TEST.md). +- `quality-test.sh` — behavioral quality (~10-30 min depending on `--quick` / `--medium` / `--full`). Wraps [`benchlocal-cli`](https://github.com/noonghunna/benchlocal-cli) — runs verifier-backed bench packs (ToolCall-15, InstructFollow-15, StructOutput-15, etc) against the running endpoint. Catches what operational tests miss: a compose can pass verify + stress + bench + soak and still ship with degraded tool-call accuracy or instruction-follow drift from quantization or Genesis env-flips. Run before promoting `Status: ✅ Production` and before any pin bump that could shift behavior. **Run via this wrapper, not raw `benchlocal-cli`** — it auto-detects endpoint/model and, for localhost URLs, sets `BENCHLOCAL_HERMES_RESOLVE_LOCALHOST=1` so the sandboxed HermesAgent can reach the host model (direct `benchlocal-cli` skips that → hermes silently scores ~0/20). **Live per-scenario `[N/M]` progress is on by default** (the wrapper forwards `--progress` to benchlocal-cli) so long runs don't go dark; pass `--no-progress` only for CI / log-volume contexts. **Timeout sizing:** the wrapper auto-scales per-scenario timeouts (startup decode-TPS probe × thinking-token multiplier), deliberately over-budgeting — the fix for the thinking-on spurious-timeout class (benchlocal-cli #54/#59). **Don't hand-set `--timeout-per-case` to "fix" a slow run unless you've confirmed the probe measured wrong.** A planned opt-in tier will size from a soak-derived per-depth TPS curve (#114). Full precedence + flags → [`docs/QUALITY_TEST.md`](docs/QUALITY_TEST.md) "Per-scenario timeouts". - `soak-test.sh` — stability (30-60 min). Run before shipping config / Genesis / memory-policy changes — catches Cliff 2b. The pipeline is layered: each script has a different question it answers ("does it serve / work / survive / fast / behave correctly / stay healthy"). Skipping any layer can mask regressions. diff --git a/docs/QUALITY_TEST.md b/docs/QUALITY_TEST.md index 7e91dd64..8bcb1367 100644 --- a/docs/QUALITY_TEST.md +++ b/docs/QUALITY_TEST.md @@ -147,6 +147,18 @@ Quality: line for compose schema field (paste into compose YAML header): Quality: ToolCall-15 14/15 (93%) · InstructFollow-15 13/15 (87%) · StructOutput-15 15/15 (100%) · DataExtract-15 12/15 (80%) · ReasonMath-15 11/15 (73%) (--medium, packs v1.0.x, 2026-05-09) ``` +## Per-scenario timeouts + +`quality-test.sh` forwards to `benchlocal-cli`, which sizes each scenario's timeout automatically — you rarely need to set one. Precedence (highest wins): + +1. **Manual** — `--timeout-per-case N` (or `TIMEOUT_PER_CASE=N`): used verbatim. +2. **Auto-scaling (default)** — the budget scales by the endpoint's measured decode speed and, for thinking-on runs, by the thinking-token budget. A one-shot startup probe measures the rig's decode TPS (and fails fast if the endpoint is unreachable, rather than hanging). The scaling deliberately **over-budgets** — a timeout is a safety ceiling, not a target — which is what keeps thinking-on packs from spuriously timing out. Exact formula + flags (`--measured-tps` / `--reference-tps` / `--retry-on-timeout`): [benchlocal-cli README → Per-case timeouts](https://github.com/noonghunna/benchlocal-cli#per-case-timeouts). +3. **Static default** — the pack's built-in `default_max_seconds`. + +**Don't hand-set `--timeout-per-case` to "fix" a slow run** unless you've confirmed the auto-probe measured wrong — the over-budget is intentional. + +> **Planned (not yet built):** an *opt-in* tier that sizes timeouts from a **soak-derived per-context-depth TPS curve** — a real "how fast at depth X" measurement for your exact rig/config, captured into the runtime measurement-record — instead of the single empty-context startup probe. It would be strictly opt-in and fall back to the auto-probe/default when no curve exists; measured data is never required. Tracked at [#114](https://github.com/noonghunna/club-3090/pull/114). + ## Sampling & temperature By default the packs sample at **temperature 0** (greedy) — deterministic and reproducible, so scores are comparable across rigs and across runs. This is the **canonical** baseline, and it's what regression tracking and cross-config ranking should use.