alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions 38 Packages Projects Releases Wiki Activity

feat(report.sh): --stress + --soak flags, --full now the canonical "everything" pass

Browse Source 
scripts/report.sh now has four optional sections:
  --verify    verify-full.sh    (~1-2 min)
  --stress    verify-stress.sh  (~5-10 min, 7/7 incl. Cliff 2 needles)
  --soak      SOAK_MODE=continuous + summary.md embed (~25 min, catches Cliff 2b)
  --bench     bench.sh          (~3 min, canonical TPS)
  --full      all four          (~35 min total, the canonical cross-rig pass)

Why soak as its own flag: verify-full + verify-stress + bench all PASS on
configs that FAIL the multi-turn continuous soak (Cliff 2b at ~25K accumulated
tokens). Until upstream lands a fix, soak is the only test that catches the
agentic-workload failure mode that bit issues #41 / #42 / #43 / #45.

Propagated --full as the recommended single-command path through:
- README.md (bug/bench callout)
- CONTRIBUTING.md (Numbers from your rig + new compose variant gate)
- .github/PULL_REQUEST_TEMPLATE.md (one checkbox covers verify+stress+soak+bench)
- .github/ISSUE_TEMPLATE/numbers-from-your-rig.yml (single paste includes soak)
- BENCHMARKS.md ("How to add a row for your rig")

Backward compatible: existing --verify and --bench flags unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
noonghunna
2026-05-04 01:37:18 +00:00
parent d8e7f73eb2
commit 8a29b95da1
6 changed files with 102 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
.github/ISSUE_TEMPLATE/numbers-from-your-rig.yml vendored
View File

@@ -13,14 +13,23 @@ body:
The fastest way to share everything we need:
```bash
bash scripts/report.sh --bench > my-rig.md
bash scripts/report.sh --full > my-rig.md
```
That captures rig hardware (GPUs, power caps, NVLink, driver, OS), stack
version (commit, Genesis pin, vLLM image SHA), boot log highlights (engine
config, KV pool sizing, patches applied), AND runs the canonical bench
(3 warmups + 5 measured runs of narrative + code prompts) — all in one
~3-5 min pass. Paste the file contents below.
config, KV pool sizing, patches applied), runs **verify-full + verify-stress
7/7 + SOAK_MODE=continuous + canonical bench** — all in one ~35-min pass.
Paste the file contents below.
Soak-continuous is the only test that catches Cliff 2b (multi-turn
accumulating-context cliff at ~25K tokens). Until upstream lands a fix,
we ask for it on every cross-rig contribution. See
[docs/CLIFFS.md](https://github.com/noonghunna/club-3090/blob/master/docs/CLIFFS.md).
If 35 min is too long, swap `--full` for `--bench` to skip verify/stress/soak
— the bench numbers alone are still useful, but flag in Notes that you
haven't run soak-continuous so we don't assume Cliff 2b is clean on your rig.
The bench prompts are pinned so your numbers are directly comparable to
ours:
@@ -34,7 +43,7 @@ body:
- type: textarea
id: rig-bench-report
attributes:
label: Rig + bench report — paste contents of `bash scripts/report.sh --bench`
label: Rig + bench report — paste contents of `bash scripts/report.sh --full` (or `--bench` if you couldn't run the full chain)
description: |
Full output. The script captures hardware, stack, boot state, AND canonical
bench numbers in one pass.
@@ -48,13 +57,13 @@ body:
- type: textarea
id: soak-continuous
attributes:
label: Soak-continuous summary (optional but high-signal)
label: Soak-continuous summary (only fill in if you ran `--bench` instead of `--full`)
description: |
If you can spare ~25 min, run the multi-turn accumulating-context soak
— it's the only test that catches Cliff 2b (single-card vLLM paths
OOM at ~25K accumulated tokens across multiple turns, regardless of
single-prompt context limits). Cross-rig validation here helps us
document which configs escape the cliff on which hardware.
If you used `bash scripts/report.sh --full` above, soak-continuous is
already included — leave this blank.
If you ran `--bench` (faster, no soak), please run soak separately and
paste the summary here, or note "skipped (rig couldn't spare ~25 min)":
```bash
SOAK_MODE=continuous SOAK_SESSIONS=5 SOAK_TURNS=5 \
@@ -62,11 +71,10 @@ body:
bash scripts/soak-test.sh
```
Paste the resulting `summary.md` (or just the PASS/FAIL + final
accumulated-token count). See [docs/CLIFFS.md](https://github.com/noonghunna/club-3090/blob/master/docs/CLIFFS.md)
Soak-continuous is the only test that catches Cliff 2b (multi-turn
accumulating-context cliff at ~25K tokens). See
[docs/CLIFFS.md](https://github.com/noonghunna/club-3090/blob/master/docs/CLIFFS.md)
for the byte-level explanation.
Leave blank if you didn't run it.
render: markdown
validations:
required: false

25
.github/PULL_REQUEST_TEMPLATE.md vendored
View File

@@ -22,23 +22,22 @@ what existing variant did you compare against, what's the trade-off? -->
## Verification
- [ ] **Rig report attached** — paste contents of `bash scripts/report.sh > my-rig.md` (or `--bench` / `--verify` if relevant) as a PR comment. Captures GENESIS_PIN, vLLM image SHA, container CUDA/Python, PCIe lanes, power caps, NVLink topology — everything we'd otherwise have to ask for one bullet at a time.
- [ ] **`bash scripts/verify-full.sh` PASSES** against this PR's compose. Output attached.
- [ ] **`bash scripts/verify-stress.sh` 7/7 PASSES** against this PR's compose. Output attached.
### For new compose variants ONLY
- [ ] **`SOAK_MODE=continuous` summary attached** — single-card variants: required (catches Cliff 2b at ~25K accumulated tokens, which `verify-stress` does not). Multi-card variants: strongly recommended.
- [ ] **Full rig + validation report attached** — single command captures everything:
```bash
SOAK_MODE=continuous SOAK_SESSIONS=5 SOAK_TURNS=5 \
CONTAINER=<container-name> ENDPOINT=<http://localhost:port> \
bash scripts/soak-test.sh
bash scripts/report.sh --full > my-rig.md
```
Paste the resulting `summary.md` as a PR comment. See [docs/CLIFFS.md](../docs/CLIFFS.md) for why soak-continuous is the only test that catches the multi-turn cliff, and [Issue #41](https://github.com/noonghunna/club-3090/issues/41) for the validation matrix.
- [ ] **`bash scripts/bench.sh` run included** — 3 warmups + 5 measured runs. Report `wall_TPS`, `decode_TPS`, `TTFT`, peak VRAM/card. MTP `AL` if applicable.
- [ ] **BENCHMARKS row added** — under the appropriate model section, mirroring existing column shape.
Runs hardware + stack + boot log capture **plus** verify-full + verify-stress 7/7 + SOAK_MODE=continuous + canonical bench in one ~35-min pass. Paste contents as a PR comment. See [docs/CLIFFS.md](../docs/CLIFFS.md) for why the soak-continuous step is load-bearing (catches Cliff 2b, which verify-stress doesn't).
- [ ] **BENCHMARKS row added** — under the appropriate model section, mirroring existing column shape (incl. `Rig` column).
- [ ] **CHANGELOG entry added** in `models/<model>/CHANGELOG.md`.
If you'd rather run the steps separately:
- `bash scripts/report.sh > my-rig.md` (rig only, ~2 sec)
- `bash scripts/verify-full.sh` — fast functional smoke
- `bash scripts/verify-stress.sh` — 7/7 boundary checks incl. Cliff 2 needles
- `SOAK_MODE=continuous SOAK_SESSIONS=5 SOAK_TURNS=5 bash scripts/soak-test.sh` — required for new single-card composes (catches Cliff 2b)
- `bash scripts/bench.sh` — canonical TPS (3 warmups + 5 measured)
### N/A justifications (if any boxes above are unchecked)
<!-- e.g. "N/A — short-prompt-only path; soak-continuous would not exercise the multi-turn regime"

2
BENCHMARKS.md
View File

@@ -30,7 +30,7 @@ Cross-rig numbers are comparable because the prompt + sampling are pinned. Varia
## How to add a row for your rig
1. Run `bash scripts/report.sh --bench > my-rig.md` — captures hardware (incl. power caps + PCIe lanes), stack version (vLLM image SHA, Genesis commit), AND the canonical bench numbers in one pass.
1. Run `bash scripts/report.sh --full > my-rig.md` — captures hardware (incl. power caps + PCIe lanes), stack version (vLLM image SHA, Genesis commit), verify-full + verify-stress + **SOAK_MODE=continuous** + canonical bench numbers in one ~35-min pass. (Or `--bench` for the fast subset; soak-continuous catches Cliff 2b which the others don't.)
2. Open the [Numbers from your rig](https://github.com/noonghunna/club-3090/issues/new?template=numbers-from-your-rig.yml) issue template, paste the report, mention which compose variant you ran.
3. We'll append your numbers as a row in the appropriate table here, with `Rig` cell formatted `@your-handle (rig-shape)` — e.g. `@whamp (4× 3090 PCIe x4/x8/x16/x16, 300 W)`.

18
CONTRIBUTING.md
View File

@@ -8,8 +8,8 @@ Thanks for being here. This repo collects working recipes for serving big LLMs o
### ✅ Yes please
- **Numbers from your rig.** Different power caps, different motherboards, different models — we want all of it. Use the [Numbers from your rig](https://github.com/noonghunna/club-3090/issues/new?template=numbers-from-your-rig.yml) issue template (no PR needed). The template asks for `bash scripts/report.sh --bench > my-rig.md` — one pass captures hardware (incl. power caps + NVLink topology), stack version, AND the canonical bench numbers. High-signal contributions land in `BENCHMARKS` with attribution.
- **Bug reports with the data we ask for.** The [bug report template](https://github.com/noonghunna/club-3090/issues/new?template=bug-report.yml) leads with `bash scripts/report.sh > my-rig.md` (add `--verify` to include verify-full output) — single command captures the rig state we'd otherwise ask for individually (hardware, container state, Genesis patches, KV pool sizing, engine config). With that paste, the first reply is usually a fix or a clear next step instead of "can you send me…".
- **Numbers from your rig.** Different power caps, different motherboards, different models — we want all of it. Use the [Numbers from your rig](https://github.com/noonghunna/club-3090/issues/new?template=numbers-from-your-rig.yml) issue template (no PR needed). The template asks for `bash scripts/report.sh --full > my-rig.md` — one ~35-min pass captures hardware (incl. power caps + NVLink topology), stack version, verify-full + verify-stress 7/7, **SOAK_MODE=continuous summary (catches Cliff 2b)**, AND the canonical bench numbers. High-signal contributions land in `BENCHMARKS` with attribution.
- **Bug reports with the data we ask for.** The [bug report template](https://github.com/noonghunna/club-3090/issues/new?template=bug-report.yml) leads with `bash scripts/report.sh > my-rig.md` (add `--verify` to include verify-full output, `--soak` to also run SOAK_MODE=continuous if you suspect a multi-turn agent cliff) — single command captures the rig state we'd otherwise ask for individually (hardware, container state, Genesis patches, KV pool sizing, engine config). With that paste, the first reply is usually a fix or a clear next step instead of "can you send me…".
- **Bug reproductions / minimum repros for upstream issues.** vLLM / llama.cpp / Genesis bugs that affect this stack are most useful when they have a one-paragraph reduction. Drop them in an issue or open a draft PR adding a reproducer to `verify-stress.sh`.
- **New compose variants with measured numbers.** If you've found a config combination that beats one we ship — better TPS, lower VRAM, cleaner stress profile — open a PR with: (a) the `docker-compose.<name>.yml`, (b) `verify-full.sh` output passing, (c) `verify-stress.sh` output passing, (d) a `bench.sh` run (3 warm + 5 measured) showing the delta against the closest existing variant. Bonus points: a footer in the compose file explaining which existing variant you compared against and why this one is better for which workload.
- **New models.** Adding a model is a real lift but well-defined: clone the `models/qwen3.6-27b/` directory structure, populate the engine subdirs, follow the [canonical learnings template](https://github.com/noonghunna/club-3090/blob/master/CLAUDE.md) layout (this repo doesn't ship that file but the convention is documented in `models/qwen3.6-27b/INTERNALS.md`). Open an issue first to scope.
@@ -57,6 +57,16 @@ Thanks for being here. This repo collects working recipes for serving big LLMs o
New compose files (`models/<model>/<engine>/compose/docker-compose.<name>.yml`) get a tighter checklist than other PRs because they ship as a "supported" path that other people boot blind. The PR template enumerates these — bullets here are the *why*:
**Single command captures all of (1)–(5) in one paste:**
```bash
bash scripts/report.sh --full > my-rig.md
```
That runs ~35 min and captures rig + verify-full + verify-stress + soak-continuous + bench. Paste the file contents as a PR comment.
Or run the steps individually if you'd rather:
1. **Rig report** — `bash scripts/report.sh > my-rig.md`, paste as a PR comment. Captures GENESIS_PIN, vLLM image SHA, container CUDA/Python, PCIe lanes per card, power caps, NVLink topology in one pass. Without it future readers can't tell whether your numbers are reproducible against their environment or rig-specific. **This is a merge gate**, not a nice-to-have.
2. **`verify-full.sh` PASS** — fast functional smoke. Confirms the variant boots and serves correctly on your rig.
3. **`verify-stress.sh` 7/7 PASS** — boundary tests including Cliff 2 needle recall (probe 7: 60K + 90K needles). Required for any variant claiming long-context support.
@@ -67,10 +77,10 @@ New compose files (`models/<model>/<engine>/compose/docker-compose.<name>.yml`)
bash scripts/soak-test.sh
```
5. **`bench.sh` run** — 3 warmups + 5 measured runs of narrative + code prompts. Report `wall_TPS`, `decode_TPS`, `TTFT`, peak VRAM/card per run, MTP/DFlash AL where applicable.
6. **BENCHMARKS row** — under the appropriate model section, mirroring existing column shape. Attribution is automatic.
6. **BENCHMARKS row** — under the appropriate model section, mirroring existing column shape (incl. `Rig` column). Attribution is automatic.
7. **CHANGELOG entry** — in `models/<model>/CHANGELOG.md`.
If any of these don't apply to your variant, say so explicitly in the PR ("N/A — short-prompt-only path; soak-continuous would not exercise the multi-turn regime"). "Forgot to run" gets the PR put on hold; "explained why N/A" gets it merged.
If any of (1)–(5) don't apply to your variant, say so explicitly in the PR ("N/A — short-prompt-only path; soak-continuous would not exercise the multi-turn regime"). "Forgot to run" gets the PR put on hold; "explained why N/A" gets it merged.
---

2
README.md
View File

@@ -98,7 +98,7 @@ bash scripts/update.sh
For client snippets — Python (`openai` SDK + raw `requests`), TypeScript / Node, plus connection settings for Open WebUI, Cline, Cursor, and other OpenAI-compat clients — see [`docs/EXAMPLES.md`](docs/EXAMPLES.md). Common questions ("can I use a 4090?", "why MTP not EAGLE?", "why not Ollama?", "what's a prefill cliff?") have answers in [`docs/FAQ.md`](docs/FAQ.md). Trying to decide self-host vs cloud APIs vs other local options? [`docs/COMPARISONS.md`](docs/COMPARISONS.md). Want to contribute numbers, bug repros, or new variants? [`CONTRIBUTING.md`](CONTRIBUTING.md). Tracking the upstream issues and PRs we depend on or have filed? [`docs/UPSTREAM.md`](docs/UPSTREAM.md).
**Hit an issue or want to share bench numbers?** Run `bash scripts/report.sh > my-rig.md` (add `--bench` to include canonical TPS) and paste into the [bug](https://github.com/noonghunna/club-3090/issues/new?template=bug-report.yml) or [bench](https://github.com/noonghunna/club-3090/issues/new?template=numbers-from-your-rig.yml) issue template — single command captures everything we'd otherwise ask for individually.
**Hit an issue or want to share bench numbers?** Run `bash scripts/report.sh > my-rig.md` (add `--full` for the canonical "everything" pass: rig + verify-full + verify-stress 7/7 + SOAK_MODE=continuous + bench, ~35 min) and paste into the [bug](https://github.com/noonghunna/club-3090/issues/new?template=bug-report.yml) or [bench](https://github.com/noonghunna/club-3090/issues/new?template=numbers-from-your-rig.yml) issue template — single command captures everything we'd otherwise ask for individually.
For llama.cpp (different engine, different recipe — useful for max context on single-card):
```bash

54
scripts/report.sh
View File

@@ -9,18 +9,28 @@
# Usage:
# bash scripts/report.sh # default: hardware + stack + boot log highlights (~2 sec)
# bash scripts/report.sh --verify # adds verify-full.sh output (~1-2 min)
# bash scripts/report.sh --stress # adds verify-stress.sh 7/7 output (~5-10 min)
# bash scripts/report.sh --soak # adds SOAK_MODE=continuous summary (~25 min) — catches Cliff 2b
# bash scripts/report.sh --bench # adds bench.sh output (~3 min)
# bash scripts/report.sh --full # both verify + bench (~5 min)
# bash scripts/report.sh --full # ALL four: verify + stress + soak + bench (~35 min, the canonical "everything" pass for cross-rig contributions)
# bash scripts/report.sh --no-redact # disable path/host/user redaction
# bash scripts/report.sh --container NAME # override container auto-detection
# bash scripts/report.sh > my-rig.md # capture for paste
#
# Why --soak is its own flag:
# verify-full + verify-stress + bench all PASS on configs that FAIL the
# multi-turn continuous soak (Cliff 2b at ~25K accumulated tokens). Until
# the upstream fix lands, soak is the only test that catches the agentic-
# workload failure mode. See docs/CLIFFS.md.
#
# By default, paths under user homes, hostnames, usernames, and HF tokens are
# redacted. Use --no-redact for internal sharing only.
set -uo pipefail
DO_VERIFY=0
DO_STRESS=0
DO_SOAK=0
DO_BENCH=0
REDACT=1
CONTAINER=""
@@ -32,8 +42,10 @@ print_help() {
while [[ $# -gt 0 ]]; do
case "$1" in
--verify) DO_VERIFY=1; shift ;;
--stress) DO_STRESS=1; shift ;;
--soak) DO_SOAK=1; shift ;;
--bench) DO_BENCH=1; shift ;;
--full) DO_VERIFY=1; DO_BENCH=1; shift ;;
--full) DO_VERIFY=1; DO_STRESS=1; DO_SOAK=1; DO_BENCH=1; shift ;;
--no-redact) REDACT=0; shift ;;
--container) CONTAINER="${2:-}"; shift 2 ;;
-h|--help) print_help; exit 0 ;;
@@ -481,6 +493,42 @@ if [[ $DO_VERIFY -eq 1 ]]; then
fi
fi
# ---------------------------------------------------------------------------
# Optional: verify-stress
# ---------------------------------------------------------------------------
if [[ $DO_STRESS -eq 1 ]]; then
section "verify-stress.sh output"
if [[ -f scripts/verify-stress.sh ]]; then
bash scripts/verify-stress.sh 2>&1 | redact | details "verify-stress output (7 boundary checks incl. Cliff 2 needle recall)"
else
echo "_scripts/verify-stress.sh not found_"
fi
fi
# ---------------------------------------------------------------------------
# Optional: soak-continuous (catches Cliff 2b — the only test that does)
# ---------------------------------------------------------------------------
if [[ $DO_SOAK -eq 1 ]]; then
section "soak-test.sh (SOAK_MODE=continuous) output"
if [[ -f scripts/soak-test.sh ]]; then
soak_run_dir="results/report-soak-$(date +%Y%m%d-%H%M%S)"
SOAK_MODE=continuous SOAK_SESSIONS=5 SOAK_TURNS=5 SOAK_OUTPUT="$soak_run_dir" \
bash scripts/soak-test.sh 2>&1 | redact | details "soak-test stdout (5-session × 5-turn ramping conversation, ~25 min)"
if [[ -f "$soak_run_dir/summary.md" ]]; then
echo
echo "**Soak summary** (\`$soak_run_dir/summary.md\`):"
echo
redact < "$soak_run_dir/summary.md"
else
echo "_soak summary.md not produced — check stdout above_"
fi
else
echo "_scripts/soak-test.sh not found_"
fi
fi
# ---------------------------------------------------------------------------
# Optional: bench
# ---------------------------------------------------------------------------
@@ -502,5 +550,5 @@ cat <<'EOF'
---
_Generated by `bash scripts/report.sh`. Add `--verify` for verify-full output, `--bench` for canonical TPS bench, `--full` for both. Use `--no-redact` to disable redaction (internal sharing only)._
_Generated by `bash scripts/report.sh`. Flags: `--verify` (verify-full), `--stress` (verify-stress 7/7 incl. Cliff 2 needles), `--soak` (SOAK_MODE=continuous, catches Cliff 2b), `--bench` (canonical TPS), `--full` (all four, ~35 min). Use `--no-redact` to disable redaction (internal sharing only)._
EOF