diff --git a/.github/ISSUE_TEMPLATE/numbers-from-your-rig.yml b/.github/ISSUE_TEMPLATE/numbers-from-your-rig.yml index e7bf09c3..bc91249f 100644 --- a/.github/ISSUE_TEMPLATE/numbers-from-your-rig.yml +++ b/.github/ISSUE_TEMPLATE/numbers-from-your-rig.yml @@ -13,14 +13,23 @@ body: The fastest way to share everything we need: ```bash - bash scripts/report.sh --bench > my-rig.md + bash scripts/report.sh --full > my-rig.md ``` That captures rig hardware (GPUs, power caps, NVLink, driver, OS), stack version (commit, Genesis pin, vLLM image SHA), boot log highlights (engine - config, KV pool sizing, patches applied), AND runs the canonical bench - (3 warmups + 5 measured runs of narrative + code prompts) — all in one - ~3-5 min pass. Paste the file contents below. + config, KV pool sizing, patches applied), runs **verify-full + verify-stress + 7/7 + SOAK_MODE=continuous + canonical bench** — all in one ~35-min pass. + Paste the file contents below. + + Soak-continuous is the only test that catches Cliff 2b (multi-turn + accumulating-context cliff at ~25K tokens). Until upstream lands a fix, + we ask for it on every cross-rig contribution. See + [docs/CLIFFS.md](https://github.com/noonghunna/club-3090/blob/master/docs/CLIFFS.md). + + If 35 min is too long, swap `--full` for `--bench` to skip verify/stress/soak + — the bench numbers alone are still useful, but flag in Notes that you + haven't run soak-continuous so we don't assume Cliff 2b is clean on your rig. The bench prompts are pinned so your numbers are directly comparable to ours: @@ -34,7 +43,7 @@ body: - type: textarea id: rig-bench-report attributes: - label: Rig + bench report — paste contents of `bash scripts/report.sh --bench` + label: Rig + bench report — paste contents of `bash scripts/report.sh --full` (or `--bench` if you couldn't run the full chain) description: | Full output. The script captures hardware, stack, boot state, AND canonical bench numbers in one pass. @@ -48,13 +57,13 @@ body: - type: textarea id: soak-continuous attributes: - label: Soak-continuous summary (optional but high-signal) + label: Soak-continuous summary (only fill in if you ran `--bench` instead of `--full`) description: | - If you can spare ~25 min, run the multi-turn accumulating-context soak - — it's the only test that catches Cliff 2b (single-card vLLM paths - OOM at ~25K accumulated tokens across multiple turns, regardless of - single-prompt context limits). Cross-rig validation here helps us - document which configs escape the cliff on which hardware. + If you used `bash scripts/report.sh --full` above, soak-continuous is + already included — leave this blank. + + If you ran `--bench` (faster, no soak), please run soak separately and + paste the summary here, or note "skipped (rig couldn't spare ~25 min)": ```bash SOAK_MODE=continuous SOAK_SESSIONS=5 SOAK_TURNS=5 \ @@ -62,11 +71,10 @@ body: bash scripts/soak-test.sh ``` - Paste the resulting `summary.md` (or just the PASS/FAIL + final - accumulated-token count). See [docs/CLIFFS.md](https://github.com/noonghunna/club-3090/blob/master/docs/CLIFFS.md) + Soak-continuous is the only test that catches Cliff 2b (multi-turn + accumulating-context cliff at ~25K tokens). See + [docs/CLIFFS.md](https://github.com/noonghunna/club-3090/blob/master/docs/CLIFFS.md) for the byte-level explanation. - - Leave blank if you didn't run it. render: markdown validations: required: false diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 92d07d8f..050d7681 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -22,23 +22,22 @@ what existing variant did you compare against, what's the trade-off? --> ## Verification -- [ ] **Rig report attached** — paste contents of `bash scripts/report.sh > my-rig.md` (or `--bench` / `--verify` if relevant) as a PR comment. Captures GENESIS_PIN, vLLM image SHA, container CUDA/Python, PCIe lanes, power caps, NVLink topology — everything we'd otherwise have to ask for one bullet at a time. -- [ ] **`bash scripts/verify-full.sh` PASSES** against this PR's compose. Output attached. -- [ ] **`bash scripts/verify-stress.sh` 7/7 PASSES** against this PR's compose. Output attached. - -### For new compose variants ONLY - -- [ ] **`SOAK_MODE=continuous` summary attached** — single-card variants: required (catches Cliff 2b at ~25K accumulated tokens, which `verify-stress` does not). Multi-card variants: strongly recommended. +- [ ] **Full rig + validation report attached** — single command captures everything: ```bash - SOAK_MODE=continuous SOAK_SESSIONS=5 SOAK_TURNS=5 \ - CONTAINER= ENDPOINT= \ - bash scripts/soak-test.sh + bash scripts/report.sh --full > my-rig.md ``` - Paste the resulting `summary.md` as a PR comment. See [docs/CLIFFS.md](../docs/CLIFFS.md) for why soak-continuous is the only test that catches the multi-turn cliff, and [Issue #41](https://github.com/noonghunna/club-3090/issues/41) for the validation matrix. -- [ ] **`bash scripts/bench.sh` run included** — 3 warmups + 5 measured runs. Report `wall_TPS`, `decode_TPS`, `TTFT`, peak VRAM/card. MTP `AL` if applicable. -- [ ] **BENCHMARKS row added** — under the appropriate model section, mirroring existing column shape. + Runs hardware + stack + boot log capture **plus** verify-full + verify-stress 7/7 + SOAK_MODE=continuous + canonical bench in one ~35-min pass. Paste contents as a PR comment. See [docs/CLIFFS.md](../docs/CLIFFS.md) for why the soak-continuous step is load-bearing (catches Cliff 2b, which verify-stress doesn't). +- [ ] **BENCHMARKS row added** — under the appropriate model section, mirroring existing column shape (incl. `Rig` column). - [ ] **CHANGELOG entry added** in `models//CHANGELOG.md`. +If you'd rather run the steps separately: + +- `bash scripts/report.sh > my-rig.md` (rig only, ~2 sec) +- `bash scripts/verify-full.sh` — fast functional smoke +- `bash scripts/verify-stress.sh` — 7/7 boundary checks incl. Cliff 2 needles +- `SOAK_MODE=continuous SOAK_SESSIONS=5 SOAK_TURNS=5 bash scripts/soak-test.sh` — required for new single-card composes (catches Cliff 2b) +- `bash scripts/bench.sh` — canonical TPS (3 warmups + 5 measured) + ### N/A justifications (if any boxes above are unchecked)