From 8a29b95da10f7ab45151fba866aa526f2a1796c2 Mon Sep 17 00:00:00 2001
From: noonghunna <10742901+noonghunna@users.noreply.github.com>
Date: Mon, 4 May 2026 01:37:18 +0000
Subject: [PATCH] feat(report.sh): --stress + --soak flags, --full now the
 canonical "everything" pass

scripts/report.sh now has four optional sections:
  --verify    verify-full.sh    (~1-2 min)
  --stress    verify-stress.sh  (~5-10 min, 7/7 incl. Cliff 2 needles)
  --soak      SOAK_MODE=continuous + summary.md embed (~25 min, catches Cliff 2b)
  --bench     bench.sh          (~3 min, canonical TPS)
  --full      all four          (~35 min total, the canonical cross-rig pass)

Why soak as its own flag: verify-full + verify-stress + bench all PASS on
configs that FAIL the multi-turn continuous soak (Cliff 2b at ~25K accumulated
tokens). Until upstream lands a fix, soak is the only test that catches the
agentic-workload failure mode that bit issues #41 / #42 / #43 / #45.

Propagated --full as the recommended single-command path through:
- README.md (bug/bench callout)
- CONTRIBUTING.md (Numbers from your rig + new compose variant gate)
- .github/PULL_REQUEST_TEMPLATE.md (one checkbox covers verify+stress+soak+bench)
- .github/ISSUE_TEMPLATE/numbers-from-your-rig.yml (single paste includes soak)
- BENCHMARKS.md ("How to add a row for your rig")

Backward compatible: existing --verify and --bench flags unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
 .../ISSUE_TEMPLATE/numbers-from-your-rig.yml  | 38 +++++++------
 .github/PULL_REQUEST_TEMPLATE.md              | 25 +++++----
 BENCHMARKS.md                                 |  2 +-
 CONTRIBUTING.md                               | 18 +++++--
 README.md                                     |  2 +-
 scripts/report.sh                             | 54 +++++++++++++++++--
 6 files changed, 102 insertions(+), 37 deletions(-)

diff --git a/.github/ISSUE_TEMPLATE/numbers-from-your-rig.yml b/.github/ISSUE_TEMPLATE/numbers-from-your-rig.yml
index e7bf09c3..bc91249f 100644
--- a/.github/ISSUE_TEMPLATE/numbers-from-your-rig.yml
+++ b/.github/ISSUE_TEMPLATE/numbers-from-your-rig.yml
@@ -13,14 +13,23 @@ body:
         The fastest way to share everything we need:
 
         ```bash
-        bash scripts/report.sh --bench > my-rig.md
+        bash scripts/report.sh --full > my-rig.md
         ```
 
         That captures rig hardware (GPUs, power caps, NVLink, driver, OS), stack
         version (commit, Genesis pin, vLLM image SHA), boot log highlights (engine
-        config, KV pool sizing, patches applied), AND runs the canonical bench
-        (3 warmups + 5 measured runs of narrative + code prompts) — all in one
-        ~3-5 min pass. Paste the file contents below.
+        config, KV pool sizing, patches applied), runs **verify-full + verify-stress
+        7/7 + SOAK_MODE=continuous + canonical bench** — all in one ~35-min pass.
+        Paste the file contents below.
+
+        Soak-continuous is the only test that catches Cliff 2b (multi-turn
+        accumulating-context cliff at ~25K tokens). Until upstream lands a fix,
+        we ask for it on every cross-rig contribution. See
+        [docs/CLIFFS.md](https://github.com/noonghunna/club-3090/blob/master/docs/CLIFFS.md).
+
+        If 35 min is too long, swap `--full` for `--bench` to skip verify/stress/soak
+        — the bench numbers alone are still useful, but flag in Notes that you
+        haven't run soak-continuous so we don't assume Cliff 2b is clean on your rig.
 
         The bench prompts are pinned so your numbers are directly comparable to
         ours:
@@ -34,7 +43,7 @@ body:
   - type: textarea
     id: rig-bench-report
     attributes:
-      label: Rig + bench report — paste contents of `bash scripts/report.sh --bench`
+      label: Rig + bench report — paste contents of `bash scripts/report.sh --full` (or `--bench` if you couldn't run the full chain)
       description: |
         Full output. The script captures hardware, stack, boot state, AND canonical
         bench numbers in one pass.
@@ -48,13 +57,13 @@ body:
   - type: textarea
     id: soak-continuous
     attributes:
-      label: Soak-continuous summary (optional but high-signal)
+      label: Soak-continuous summary (only fill in if you ran `--bench` instead of `--full`)
       description: |
-        If you can spare ~25 min, run the multi-turn accumulating-context soak
-        — it's the only test that catches Cliff 2b (single-card vLLM paths
-        OOM at ~25K accumulated tokens across multiple turns, regardless of
-        single-prompt context limits). Cross-rig validation here helps us
-        document which configs escape the cliff on which hardware.
+        If you used `bash scripts/report.sh --full` above, soak-continuous is
+        already included — leave this blank.
+
+        If you ran `--bench` (faster, no soak), please run soak separately and
+        paste the summary here, or note "skipped (rig couldn't spare ~25 min)":
 
         ```bash
         SOAK_MODE=continuous SOAK_SESSIONS=5 SOAK_TURNS=5 \
@@ -62,11 +71,10 @@ body:
           bash scripts/soak-test.sh
         ```
 
-        Paste the resulting `summary.md` (or just the PASS/FAIL + final
-        accumulated-token count). See [docs/CLIFFS.md](https://github.com/noonghunna/club-3090/blob/master/docs/CLIFFS.md)
+        Soak-continuous is the only test that catches Cliff 2b (multi-turn
+        accumulating-context cliff at ~25K tokens). See
+        [docs/CLIFFS.md](https://github.com/noonghunna/club-3090/blob/master/docs/CLIFFS.md)
         for the byte-level explanation.
-
-        Leave blank if you didn't run it.
       render: markdown
     validations:
       required: false
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 92d07d8f..050d7681 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -22,23 +22,22 @@ what existing variant did you compare against, what's the trade-off? -->
 
 ## Verification
 
-- [ ] **Rig report attached** — paste contents of `bash scripts/report.sh > my-rig.md` (or `--bench` / `--verify` if relevant) as a PR comment. Captures GENESIS_PIN, vLLM image SHA, container CUDA/Python, PCIe lanes, power caps, NVLink topology — everything we'd otherwise have to ask for one bullet at a time.
-- [ ] **`bash scripts/verify-full.sh` PASSES** against this PR's compose. Output attached.
-- [ ] **`bash scripts/verify-stress.sh` 7/7 PASSES** against this PR's compose. Output attached.
-
-### For new compose variants ONLY
-
-- [ ] **`SOAK_MODE=continuous` summary attached** — single-card variants: required (catches Cliff 2b at ~25K accumulated tokens, which `verify-stress` does not). Multi-card variants: strongly recommended.
+- [ ] **Full rig + validation report attached** — single command captures everything:
   ```bash
-  SOAK_MODE=continuous SOAK_SESSIONS=5 SOAK_TURNS=5 \
-    CONTAINER=<container-name> ENDPOINT=<http://localhost:port> \
-    bash scripts/soak-test.sh
+  bash scripts/report.sh --full > my-rig.md
   ```
-  Paste the resulting `summary.md` as a PR comment. See [docs/CLIFFS.md](../docs/CLIFFS.md) for why soak-continuous is the only test that catches the multi-turn cliff, and [Issue #41](https://github.com/noonghunna/club-3090/issues/41) for the validation matrix.
-- [ ] **`bash scripts/bench.sh` run included** — 3 warmups + 5 measured runs. Report `wall_TPS`, `decode_TPS`, `TTFT`, peak VRAM/card. MTP `AL` if applicable.
-- [ ] **BENCHMARKS row added** — under the appropriate model section, mirroring existing column shape.
+  Runs hardware + stack + boot log capture **plus** verify-full + verify-stress 7/7 + SOAK_MODE=continuous + canonical bench in one ~35-min pass. Paste contents as a PR comment. See [docs/CLIFFS.md](../docs/CLIFFS.md) for why the soak-continuous step is load-bearing (catches Cliff 2b, which verify-stress doesn't).
+- [ ] **BENCHMARKS row added** — under the appropriate model section, mirroring existing column shape (incl. `Rig` column).
 - [ ] **CHANGELOG entry added** in `models/<model>/CHANGELOG.md`.
 
+If you'd rather run the steps separately:
+
+- `bash scripts/report.sh > my-rig.md` (rig only, ~2 sec)
+- `bash scripts/verify-full.sh` — fast functional smoke
+- `bash scripts/verify-stress.sh` — 7/7 boundary checks incl. Cliff 2 needles
+- `SOAK_MODE=continuous SOAK_SESSIONS=5 SOAK_TURNS=5 bash scripts/soak-test.sh` — required for new single-card composes (catches Cliff 2b)
+- `bash scripts/bench.sh` — canonical TPS (3 warmups + 5 measured)
+
 ### N/A justifications (if any boxes above are unchecked)
 
 <!-- e.g. "N/A — short-prompt-only path; soak-continuous would not exercise the multi-turn regime"
diff --git a/BENCHMARKS.md b/BENCHMARKS.md
index 462b422b..ef8e4cdb 100644
--- a/BENCHMARKS.md
+++ b/BENCHMARKS.md
@@ -30,7 +30,7 @@ Cross-rig numbers are comparable because the prompt + sampling are pinned. Varia
 
 ## How to add a row for your rig
 
-1. Run `bash scripts/report.sh --bench > my-rig.md` — captures hardware (incl. power caps + PCIe lanes), stack version (vLLM image SHA, Genesis commit), AND the canonical bench numbers in one pass.
+1. Run `bash scripts/report.sh --full > my-rig.md` — captures hardware (incl. power caps + PCIe lanes), stack version (vLLM image SHA, Genesis commit), verify-full + verify-stress + **SOAK_MODE=continuous** + canonical bench numbers in one ~35-min pass. (Or `--bench` for the fast subset; soak-continuous catches Cliff 2b which the others don't.)
 2. Open the [Numbers from your rig](https://github.com/noonghunna/club-3090/issues/new?template=numbers-from-your-rig.yml) issue template, paste the report, mention which compose variant you ran.
 3. We'll append your numbers as a row in the appropriate table here, with `Rig` cell formatted `@your-handle (rig-shape)` — e.g. `@whamp (4× 3090 PCIe x4/x8/x16/x16, 300 W)`.
 
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 3a7a5422..1f274214 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -8,8 +8,8 @@ Thanks for being here. This repo collects working recipes for serving big LLMs o
 
 ### ✅ Yes please
 
-- **Numbers from your rig.** Different power caps, different motherboards, different models — we want all of it. Use the [Numbers from your rig](https://github.com/noonghunna/club-3090/issues/new?template=numbers-from-your-rig.yml) issue template (no PR needed). The template asks for `bash scripts/report.sh --bench > my-rig.md` — one pass captures hardware (incl. power caps + NVLink topology), stack version, AND the canonical bench numbers. High-signal contributions land in `BENCHMARKS` with attribution.
-- **Bug reports with the data we ask for.** The [bug report template](https://github.com/noonghunna/club-3090/issues/new?template=bug-report.yml) leads with `bash scripts/report.sh > my-rig.md` (add `--verify` to include verify-full output) — single command captures the rig state we'd otherwise ask for individually (hardware, container state, Genesis patches, KV pool sizing, engine config). With that paste, the first reply is usually a fix or a clear next step instead of "can you send me…".
+- **Numbers from your rig.** Different power caps, different motherboards, different models — we want all of it. Use the [Numbers from your rig](https://github.com/noonghunna/club-3090/issues/new?template=numbers-from-your-rig.yml) issue template (no PR needed). The template asks for `bash scripts/report.sh --full > my-rig.md` — one ~35-min pass captures hardware (incl. power caps + NVLink topology), stack version, verify-full + verify-stress 7/7, **SOAK_MODE=continuous summary (catches Cliff 2b)**, AND the canonical bench numbers. High-signal contributions land in `BENCHMARKS` with attribution.
+- **Bug reports with the data we ask for.** The [bug report template](https://github.com/noonghunna/club-3090/issues/new?template=bug-report.yml) leads with `bash scripts/report.sh > my-rig.md` (add `--verify` to include verify-full output, `--soak` to also run SOAK_MODE=continuous if you suspect a multi-turn agent cliff) — single command captures the rig state we'd otherwise ask for individually (hardware, container state, Genesis patches, KV pool sizing, engine config). With that paste, the first reply is usually a fix or a clear next step instead of "can you send me…".
 - **Bug reproductions / minimum repros for upstream issues.** vLLM / llama.cpp / Genesis bugs that affect this stack are most useful when they have a one-paragraph reduction. Drop them in an issue or open a draft PR adding a reproducer to `verify-stress.sh`.
 - **New compose variants with measured numbers.** If you've found a config combination that beats one we ship — better TPS, lower VRAM, cleaner stress profile — open a PR with: (a) the `docker-compose.<name>.yml`, (b) `verify-full.sh` output passing, (c) `verify-stress.sh` output passing, (d) a `bench.sh` run (3 warm + 5 measured) showing the delta against the closest existing variant. Bonus points: a footer in the compose file explaining which existing variant you compared against and why this one is better for which workload.
 - **New models.** Adding a model is a real lift but well-defined: clone the `models/qwen3.6-27b/` directory structure, populate the engine subdirs, follow the [canonical learnings template](https://github.com/noonghunna/club-3090/blob/master/CLAUDE.md) layout (this repo doesn't ship that file but the convention is documented in `models/qwen3.6-27b/INTERNALS.md`). Open an issue first to scope.
@@ -57,6 +57,16 @@ Thanks for being here. This repo collects working recipes for serving big LLMs o
 
 New compose files (`models/<model>/<engine>/compose/docker-compose.<name>.yml`) get a tighter checklist than other PRs because they ship as a "supported" path that other people boot blind. The PR template enumerates these — bullets here are the *why*:
 
+**Single command captures all of (1)–(5) in one paste:**
+
+```bash
+bash scripts/report.sh --full > my-rig.md
+```
+
+That runs ~35 min and captures rig + verify-full + verify-stress + soak-continuous + bench. Paste the file contents as a PR comment.
+
+Or run the steps individually if you'd rather:
+
 1. **Rig report** — `bash scripts/report.sh > my-rig.md`, paste as a PR comment. Captures GENESIS_PIN, vLLM image SHA, container CUDA/Python, PCIe lanes per card, power caps, NVLink topology in one pass. Without it future readers can't tell whether your numbers are reproducible against their environment or rig-specific. **This is a merge gate**, not a nice-to-have.
 2. **`verify-full.sh` PASS** — fast functional smoke. Confirms the variant boots and serves correctly on your rig.
 3. **`verify-stress.sh` 7/7 PASS** — boundary tests including Cliff 2 needle recall (probe 7: 60K + 90K needles). Required for any variant claiming long-context support.
@@ -67,10 +77,10 @@ New compose files (`models/<model>/<engine>/compose/docker-compose.<name>.yml`)
      bash scripts/soak-test.sh
    ```
 5. **`bench.sh` run** — 3 warmups + 5 measured runs of narrative + code prompts. Report `wall_TPS`, `decode_TPS`, `TTFT`, peak VRAM/card per run, MTP/DFlash AL where applicable.
-6. **BENCHMARKS row** — under the appropriate model section, mirroring existing column shape. Attribution is automatic.
+6. **BENCHMARKS row** — under the appropriate model section, mirroring existing column shape (incl. `Rig` column). Attribution is automatic.
 7. **CHANGELOG entry** — in `models/<model>/CHANGELOG.md`.
 
-If any of these don't apply to your variant, say so explicitly in the PR ("N/A — short-prompt-only path; soak-continuous would not exercise the multi-turn regime"). "Forgot to run" gets the PR put on hold; "explained why N/A" gets it merged.
+If any of (1)–(5) don't apply to your variant, say so explicitly in the PR ("N/A — short-prompt-only path; soak-continuous would not exercise the multi-turn regime"). "Forgot to run" gets the PR put on hold; "explained why N/A" gets it merged.
 
 ---
 
diff --git a/README.md b/README.md
index 55569028..b15a675d 100644
--- a/README.md
+++ b/README.md
@@ -98,7 +98,7 @@ bash scripts/update.sh
 
 For client snippets — Python (`openai` SDK + raw `requests`), TypeScript / Node, plus connection settings for Open WebUI, Cline, Cursor, and other OpenAI-compat clients — see [`docs/EXAMPLES.md`](docs/EXAMPLES.md). Common questions ("can I use a 4090?", "why MTP not EAGLE?", "why not Ollama?", "what's a prefill cliff?") have answers in [`docs/FAQ.md`](docs/FAQ.md). Trying to decide self-host vs cloud APIs vs other local options? [`docs/COMPARISONS.md`](docs/COMPARISONS.md). Want to contribute numbers, bug repros, or new variants? [`CONTRIBUTING.md`](CONTRIBUTING.md). Tracking the upstream issues and PRs we depend on or have filed? [`docs/UPSTREAM.md`](docs/UPSTREAM.md).
 
-**Hit an issue or want to share bench numbers?** Run `bash scripts/report.sh > my-rig.md` (add `--bench` to include canonical TPS) and paste into the [bug](https://github.com/noonghunna/club-3090/issues/new?template=bug-report.yml) or [bench](https://github.com/noonghunna/club-3090/issues/new?template=numbers-from-your-rig.yml) issue template — single command captures everything we'd otherwise ask for individually.
+**Hit an issue or want to share bench numbers?** Run `bash scripts/report.sh > my-rig.md` (add `--full` for the canonical "everything" pass: rig + verify-full + verify-stress 7/7 + SOAK_MODE=continuous + bench, ~35 min) and paste into the [bug](https://github.com/noonghunna/club-3090/issues/new?template=bug-report.yml) or [bench](https://github.com/noonghunna/club-3090/issues/new?template=numbers-from-your-rig.yml) issue template — single command captures everything we'd otherwise ask for individually.
 
 For llama.cpp (different engine, different recipe — useful for max context on single-card):
 ```bash
diff --git a/scripts/report.sh b/scripts/report.sh
index 2e4e6b52..a5f065fe 100755
--- a/scripts/report.sh
+++ b/scripts/report.sh
@@ -9,18 +9,28 @@
 # Usage:
 #   bash scripts/report.sh                   # default: hardware + stack + boot log highlights (~2 sec)
 #   bash scripts/report.sh --verify          # adds verify-full.sh output (~1-2 min)
+#   bash scripts/report.sh --stress          # adds verify-stress.sh 7/7 output (~5-10 min)
+#   bash scripts/report.sh --soak            # adds SOAK_MODE=continuous summary (~25 min) — catches Cliff 2b
 #   bash scripts/report.sh --bench           # adds bench.sh output (~3 min)
-#   bash scripts/report.sh --full            # both verify + bench (~5 min)
+#   bash scripts/report.sh --full            # ALL four: verify + stress + soak + bench (~35 min, the canonical "everything" pass for cross-rig contributions)
 #   bash scripts/report.sh --no-redact       # disable path/host/user redaction
 #   bash scripts/report.sh --container NAME  # override container auto-detection
 #   bash scripts/report.sh > my-rig.md       # capture for paste
 #
+# Why --soak is its own flag:
+#   verify-full + verify-stress + bench all PASS on configs that FAIL the
+#   multi-turn continuous soak (Cliff 2b at ~25K accumulated tokens). Until
+#   the upstream fix lands, soak is the only test that catches the agentic-
+#   workload failure mode. See docs/CLIFFS.md.
+#
 # By default, paths under user homes, hostnames, usernames, and HF tokens are
 # redacted. Use --no-redact for internal sharing only.
 
 set -uo pipefail
 
 DO_VERIFY=0
+DO_STRESS=0
+DO_SOAK=0
 DO_BENCH=0
 REDACT=1
 CONTAINER=""
@@ -32,8 +42,10 @@ print_help() {
 while [[ $# -gt 0 ]]; do
   case "$1" in
     --verify) DO_VERIFY=1; shift ;;
+    --stress) DO_STRESS=1; shift ;;
+    --soak) DO_SOAK=1; shift ;;
     --bench) DO_BENCH=1; shift ;;
-    --full) DO_VERIFY=1; DO_BENCH=1; shift ;;
+    --full) DO_VERIFY=1; DO_STRESS=1; DO_SOAK=1; DO_BENCH=1; shift ;;
     --no-redact) REDACT=0; shift ;;
     --container) CONTAINER="${2:-}"; shift 2 ;;
     -h|--help) print_help; exit 0 ;;
@@ -481,6 +493,42 @@ if [[ $DO_VERIFY -eq 1 ]]; then
   fi
 fi
 
+# ---------------------------------------------------------------------------
+# Optional: verify-stress
+# ---------------------------------------------------------------------------
+
+if [[ $DO_STRESS -eq 1 ]]; then
+  section "verify-stress.sh output"
+  if [[ -f scripts/verify-stress.sh ]]; then
+    bash scripts/verify-stress.sh 2>&1 | redact | details "verify-stress output (7 boundary checks incl. Cliff 2 needle recall)"
+  else
+    echo "_scripts/verify-stress.sh not found_"
+  fi
+fi
+
+# ---------------------------------------------------------------------------
+# Optional: soak-continuous (catches Cliff 2b — the only test that does)
+# ---------------------------------------------------------------------------
+
+if [[ $DO_SOAK -eq 1 ]]; then
+  section "soak-test.sh (SOAK_MODE=continuous) output"
+  if [[ -f scripts/soak-test.sh ]]; then
+    soak_run_dir="results/report-soak-$(date +%Y%m%d-%H%M%S)"
+    SOAK_MODE=continuous SOAK_SESSIONS=5 SOAK_TURNS=5 SOAK_OUTPUT="$soak_run_dir" \
+      bash scripts/soak-test.sh 2>&1 | redact | details "soak-test stdout (5-session × 5-turn ramping conversation, ~25 min)"
+    if [[ -f "$soak_run_dir/summary.md" ]]; then
+      echo
+      echo "**Soak summary** (\`$soak_run_dir/summary.md\`):"
+      echo
+      redact < "$soak_run_dir/summary.md"
+    else
+      echo "_soak summary.md not produced — check stdout above_"
+    fi
+  else
+    echo "_scripts/soak-test.sh not found_"
+  fi
+fi
+
 # ---------------------------------------------------------------------------
 # Optional: bench
 # ---------------------------------------------------------------------------
@@ -502,5 +550,5 @@ cat <<'EOF'
 
 ---
 
-_Generated by `bash scripts/report.sh`. Add `--verify` for verify-full output, `--bench` for canonical TPS bench, `--full` for both. Use `--no-redact` to disable redaction (internal sharing only)._
+_Generated by `bash scripts/report.sh`. Flags: `--verify` (verify-full), `--stress` (verify-stress 7/7 incl. Cliff 2 needles), `--soak` (SOAK_MODE=continuous, catches Cliff 2b), `--bench` (canonical TPS), `--full` (all four, ~35 min). Use `--no-redact` to disable redaction (internal sharing only)._
 EOF