7248ccb605caf53b40bc25cc474b94de7be88668
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 noonghunnaandClaude Opus 4.8
|
8468d360ad |
soak-test: add beellama- to container auto-detect glob (#362)
steamEngineer's single-card beellama-qwen36-27b soak failed with 'no running club-3090 container found' — auto_container()'s glob matched vllm-/llama-cpp-/ik-llama-/sglang- but not beellama-, so the container fell through to the die(). (ik-llama report #365 soaked fine because ik-llama-qwen36-27b *was* in the glob.) Add beellama-qwen36-27b / -qwen36-35b-a3b / -gemma4-31b / -gemma4-12b (note beellama names gemma as 'gemma4-31b', not 'gemma-4-31b'); the '^' prefix anchor covers the -dual / -dual-dflash siblings. Update the die() message + the two CONTAINER help-text blurbs to list beellama-. Sibling of #403 (llama-cpp/ik glob gap). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
|
5c34dbe989 |
soak-test.sh: add qwen3.6-35b-a3b to container auto-detect glob (#244)
The auto_container() grep hardcoded `qwen36-27b|gemma-4-31b` engine × model combinations, so `ik-llama-qwen36-35b-a3b-apex-fit` (and any future 35b-a3b sibling on vllm / llama-cpp / ik-llama) was not discovered → "no running club-3090 container found" → soak exited in seconds. Workaround was `CONTAINER=...` explicit; this is the regex fix. Updated: the grep itself, the error message, the env doc, and the --help text in scripts/soak-test.sh. Co-authored-by: noonghunna <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
d9fdab2961 |
fix(soak,preflight): recognize llama-cpp / ik-llama containers in autodetect (#403)
soak-test.sh::auto_container only matched vllm-* → rc=2 "no running container" on the soak step of llama.cpp / ik_llama rebench-full, aborting the chain before the aider step. Mirror preflight's canonical engine-prefix set into auto_container, and probe internal 8080/30000 (not just 8000) in endpoint_from_container. Also add ik-llama-qwen36-27b to preflight_autodetect_endpoint + preflight_running globs — the ik leg's autodetect was a latent gap (only llama-cpp was listed). Closes #403. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunna
|
9a039d8c92 |
docs(soak-test): clarify PASS verdict semantics — closes #140
Soak-test PASS only verifies "no failure signal on this sample at this depth," not "patches in the compose's overlay set are load-bearing for this workload." On TP=2 / llama.cpp configs the topology itself takes Cliff 2 off the table, so PASS on those composes can't attribute work to any specific patch. - soak-test.sh header docstring: new "PASS verdict semantics" block - soak-test.sh --help: matching "PASS VERDICT" section - soak-helper.py: PASS verdicts now print a one-line caveat pointing to scripts/soak-test.sh --help and docs/CLIFFS.md - docs/CLIFFS.md: callout in "Why TP=2 escapes" explaining what a clean dual.yml soak does and does not validate Patch-attribution path (rerun with overlays stripped) referenced in all three surfaces. |
||
|
noonghunnaandClaude Opus 4.7
|
c298b60f76 |
encourage-soak: template dropdown + script ergonomics + report reminder + Notes convention
Four small fixes addressing low soak-test compliance in cross-rig bench contributions. Audit of 5 recent #113/#107/#102/#104/#93 showed soak data IS being run but it's hidden in the main report and the dedicated template field comes out empty (template said "leave blank if you ran --full"). Older BENCHMARKS rows often omit soak verdict entirely. 1. **`.github/ISSUE_TEMPLATE/numbers-from-your-rig.yml`**: replace optional "soak summary" textarea with a required dropdown listing PASS / borderline / FAIL / Skipped+reason / Not-yet-run. Verdict is now grep-able even when the data is buried in the main report textarea. 2. **`scripts/soak-test.sh`**: add `--continuous` / `--quick` / `--fresh` flags + `--help` + cleaner usage docs. Was 5 env vars to invoke (`SOAK_MODE=continuous SOAK_SESSIONS=5 SOAK_TURNS=5 CONTAINER=... ENDPOINT=...`); now `bash scripts/soak-test.sh --continuous` does the same with auto-detect (existing logic preserved + exposed). Env vars still work for back-compat. 3. **`scripts/report.sh`**: when `--bench` (or partial) ran without `--soak`/`--full`, append a "⚠ Soak: not included" reminder block to the report so contributors know what's missing before pasting into the issue template. 4. **`BENCHMARKS.md`**: Notes-column convention — every row should start with explicit `Soak: ✓ PASS` / `⚠ borderline` / `✗ FAIL` / `—` so readers can grep at a glance. Updated 2 recent rows (ygafarov #113, JDWarner #107) to use the convention. Older rows backfill as the convention spreads. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunna
|
a8606e3439 |
scripts(verify-full, soak-test): decouple from docker/vLLM assumptions (#85, #87)
@lamentofhighborne (#85) submitted the first 1× 3090 cross-rig data on a llama.cpp HOST build (no Docker container) and had to write local verify-full-mtp.sh + verify-stress-mtp.sh adaptations because our shipped scripts assumed vLLM compose stack throughout. Two scripts fixed in this commit: ## verify-full.sh Add `detect_engine()` helper at startup. Probes: 1. /props endpoint → llama.cpp llama-server 2. /v1/chat/completions response.system_fingerprint → "vllm-*" or "sglang-*" 3. CONTAINER name pattern as last-resort fallback Cached as $ENGINE_KIND and surfaced in the script header. Step 2 (Genesis check) and step 8 (MTP acceptance via SpecDecoding log scrape) now skip with engine-aware messages on llamacpp/sglang/unknown instead of failing on missing docker or missing log format. Net: a contributor on a host-build llama.cpp endpoint sees: [2/8] Genesis patches applied ... ⊘ llama.cpp engine — Genesis is vLLM-only, not applicable (skipped) instead of the previous misleading "no Genesis marker in logs". ## soak-test.sh `docker` becomes soft-required. New `CONTAINER=none` (or implicit when docker isn't in PATH) puts the script into HOST_MODE which: - Skips all `docker ps`/`docker port`/`docker stats`/`docker inspect` - Uses URL env var directly (fall back to localhost:8020) - Tracks VRAM via bare `nvidia-smi --query-gpu=memory.used` (no docker stats) Smoke-tested against running gemma-mtp endpoint with CONTAINER=none: boots cleanly, "[soak] host mode: CONTAINER=none — skipping docker checks", VRAM tracking + per-turn decode runs via HTTP. ## Not in this commit (deferred) verify-stress.sh has the same engine-coupling pattern but the docker references are mostly diagnostic *hints* in error messages (telling users where to find logs). Tracked in #87 as a follow-up; not blocking host-build users today. ## Test plan - [x] vLLM compose path: no regression (engine=vllm detected, all docker-dependent checks ran as before) - [x] CONTAINER=none + vLLM endpoint: host mode triggers, docker checks skipped, all HTTP-based checks ran - [ ] llama.cpp host build: would route to "llamacpp" engine class via /props endpoint, skip Genesis + MTP-log checks with clear messages. @lamentofhighborne or any future host-build contributor can validate. |
||
|
|
e6e6c827ba |
add Gemma 4 31B + Google MTP drafter (first Ampere data) (#68)
* add Gemma 4 31B + Google MTP drafter — first Ampere consumer cross-rig data Adds models/gemma-4-31b/ tree with two compose variants (TP=2 + TP=1), vendored vLLM PR #41745 overlay, switch.sh / launch.sh wiring, plus the script-level extensions needed to make `bash scripts/{bench,verify-*, soak-test}.sh` auto-detect Gemma containers. Validated 2026-05-05 on 2× RTX 3090 (Ampere sm_86, PCIe-only): - canonical bench (3 warm + 5 measured per prompt): narrative wall TPS = 108.87 (CV 3.2%) code wall TPS = 142.25 (CV 2.3%) - soak-continuous (5 sessions × 5 turns × 4 prompts = 100 turns): verdict PASS, 0 errors, 0 silent-empty, 0 MiB VRAM growth, 98.3% TPS retention, p50 decode 111.79 - VRAM 22.5 GB/card. Speedup vs no-spec-decode baseline: 1.79× narr / 2.31× code. First published Ampere consumer numbers on Google's Gemma 4 MTP "assistant" drafters (released 2026-05-05). Discussion #67 has the announcement + upstream context. BENCHMARKS.md gets a new "Gemma 4 31B" section with both TP=2 (working) and TP=1 (upstream-blocked on Ampere) rows. Components landed: - models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp.yml (TP=2) - models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp-tp1.yml (TP=1 — preserved for re-test when upstream Ampere fp8 path lands) - models/gemma-4-31b/vllm/patches/vllm-gemma4-mtp/ — vendored overlay of vllm-project/vllm#41745 (lucianommartins/gemma4-mtp). 7 modified Python files RO-mounted over the stock nightly image. Same shape as vllm-marlin-pad. Drop the entire tree when PR merges + propagates. - scripts/switch.sh + launch.sh: vllm/gemma-mtp + vllm/gemma-mtp-tp1 variants registered. RUNNING_PATTERN extended for vllm-gemma-4-31b*. - scripts/preflight.sh: autodetect_endpoint extended for gemma containers. - scripts/soak-test.sh: container grep extended. - scripts/soak-helper.py: SOAK_NO_CHAT_TEMPLATE_KWARGS=1 env knob to skip the Qwen3-specific `chat_template_kwargs.enable_thinking` body field for non-Qwen models that reject it (Gemma 4, etc.). - BENCHMARKS.md: new Gemma 4 31B section with both TP rows. - docs/UPSTREAM.md: PR #41745 row + the Ampere fp8-blocked finding + transformers 5.8.0 dependency entry. Pre-merge dependencies (drop when both land): 1. vllm-project/vllm#41745 → drop the patches/ tree + the volume block 2. transformers ≥ 5.8.0 → drop the entrypoint pip install line Out of scope: - TP=1 single-card. Tested + upstream-blocked on Ampere consumer: fp8 KV needed for fit; fp8_e4m3 hits Triton "fp8e4nv not supported" on sm_86; fp8_e5m2 rejected by gemma4_mm.py:1336 allowlist. Compose is preserved with the failing config baked in for future re-test. - Gemma 4 26B-A4B MoE single-card. Active params ~4B → should fit cleanly without fp8. Queued as the obvious follow-up. - vllm/vllm-openai:gemma4-0505-cu129 image swap. The recipe-image is Hopper-tagged and may not include sm_86 kernels; testing it would collapse this whole 4-layer wrangle to a one-line image bump but needs explicit Ampere validation first. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * models/gemma-4-31b/vllm/cache: gitignore + README (mirror qwen3.6 pattern) Replaces the .gitkeep stubs with the standard .gitignore (ignore everything except .gitignore + README.md) + README documenting the cache lifecycle. Mirrors models/qwen3.6-27b/vllm/cache/ exactly. Boot times this enables (validated tonight): - cold first boot: ~3-7 min (TP=1 / TP=2) - warm subsequent boot: ~2-3 min (cache hit) Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> --------- Co-authored-by: noonghunna <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
f32d8a6972 |
fix(soak): flag silent-empty turns (HTTP 200 + 0 tokens) as warnings
Previously soak-test.sh verdict treated `status != 200 or error` as the only error class. Turns where the engine returned HTTP 200 OK but emitted zero completion tokens (commonly: xgrammar mask rejecting all candidates, client max_tokens exhausted by `<think>` block, or spec-decode returning an empty draft batch) slipped through and the run reported PASS while the workload-level failure was firing. Discovered when @efschu posted a soak-continuous run on `dual-turbo.yml` + fp8_e5m2 KV (club-3090#47) where 2 of 25 turns logged decode_tps=0.0 with status=200 — same silent-empty failure mode @stiggy2k16 reported in #43 — but the verdict was PASS because they weren't HTTP errors. Changes: - soak-test.sh: TURN_LOG header now includes `completion_tokens` column. - soak-helper.py: append-log writes completion_tokens. Summary detects silent-empty turns via `status == 200 && !error && decode_tps == 0 && t_ms >= 1000`. Counts >=50% silent-empty as a verdict failure; 1-49% as a warning (run can still PASS for Cliff 2b but flags the workload-level bug). Back-compat with pre-2026-05-04 CSVs because decode_tps was always recorded. - Console + markdown summary both surface `silent_empty N/M (X.X%)`. Tested against existing soak-20260503-151203/turn-log.csv (silent_empty 0/25, no false positives despite missing column) and a synthetic recreation of efschu's run (silent_empty 2/15 as expected, verdict stays PASS with a warning). Refs: #43, #47 Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunna
|
8d5bfd85f9 |
feat(soak-test): continuous-mode v2 fixtures + reproduces Cliff 2 at 25K accumulated context
Adds SOAK_MODE=continuous env knob to scripts/soak-test.sh for testing
context-accumulation accretion. Each session becomes a single multi-turn
agentic-coding conversation (system → tool_call → tool_result → ... ×5)
that ramps to ~22-25K accumulated context by turn 5 — the workload shape
that bit GuiPerPT in club-3090#41 but that v1 fresh-mode fixtures couldn't
reproduce.
scripts/soak-helper.py — continuous-mode primitives:
- CONTINUOUS_TURNS — 5 ramping turn specs (350 → 1500 max_tokens, with
tool_synth describing the synthetic filler the next turn injects)
- _filler_python_code / _filler_grep_output / _filler_command_output —
plausible synthetic content of configurable size so context grows
even when the model doesn't emit a real tool_call
- cmd_init_session — creates state file with system prompt
- cmd_request_continuous — appends new user message to state, generates
request body using accumulated history
- cmd_ingest — appends assistant response + synthetic tool result to
state. Falls back to synthesized assistant tool_call when the model
didn't emit one (so context still grows as designed).
- cmd_run extended to accumulate streamed content / reasoning_content /
tool_calls deltas (was metrics-only in v1) so cmd_ingest can re-use them.
scripts/soak-test.sh — gated branching:
- SOAK_MODE env (fresh|continuous, default: fresh — v1 backward compat)
- Continuous mode requires SOAK_TURNS=5 (turn shapes are a designed ramp;
partial sessions don't reach target ctx). Validated upfront with hard
error if mismatch.
- Per-session state file at results/<run>/states/state-s{N}.json
- Inserts init-session at start of each session; request-continuous +
ingest around each turn's HTTP call.
CROSS-RIG VALIDATION (this rig, RTX 3090, today):
Config Boot Max VRAM Growth OOM at
vllm/long-vision 145K + 0.95 21778 23818 +2040 MiB s1 t5 (~26K)
vllm/long-text 180K + 0.93 22434 23736 +3240 MiB s1 t4 (~21K)
Both crash with byte-identical stack trace to GuiPerPT's #41 report:
chunk_gated_delta_rule → chunk_fwd_o → torch.empty_like(v)
→ CUDA OOM, tried 38-50 MiB, ~32 MiB free.
Major finding: Cliff 2 fires at ~21-26K *accumulated* context, not just at
50-60K *single* prompts. Both shipping single-card variants are unsafe
under multi-turn agent traffic. Updates to FAQ + per-model docs pending —
this commit ships the diagnostic primitive that exposed the finding.
Practical impact for users to be communicated separately.
Mock-tested all helper commands end-to-end before live runs:
- init-session → 1 msg (system)
- request-continuous turn 1 → 2 msgs, ~1.9K bytes
- ingest turn 1 → 4 msgs, 22K bytes (filler injected)
- ... ramp through turn 5 → 15 msgs, 106K bytes (~26K toks)
- fallback synthesis verified when model returns empty tool_calls
|
||
|
noonghunna
|
8e9cf70d99 |
fix(soak-test, switch): calibration + boot-progress UX from first cross-rig runs
Three calibration fixes to soak-test + two boot-progress improvements to switch.sh, all from the first cross-rig soak runs on this rig today. soak-test.sh — baseline timing Brief said "capture warm baseline after first turn" but turn 3 ships a 12K-char tool-result paste that fills prefix cache by ~1000 MiB on the first hit. With baseline at turn 1, every healthy config false-positives with "growth > 200 MiB threshold" on session 1 alone. Fixed: capture baseline at end of session 1 (after all 5 turn shapes run once). Sessions 2-N then measure real accretion from steady state. soak-helper.py — decode_tps guard Thinking-mode requests where vLLM bundles all reasoning into the terminal streaming chunk produce wall ≈ ttft (no separate content delta visible). Old code computed decode_tps = completion_tokens / (wall - ttft) with wall - ttft ≈ 0, yielding ~2 billion TPS. Fixed: when ttft is None OR wall - ttft < 100ms, report decode_tps = 0 (caller filterable). soak-helper.py — TPS outlier filter in cmd_summary Defensive — even if a future helper bug regresses, we filter unrealistic decode_tps values (>500 t/s) from all summary computations. tps_retention metric was previously being inflated by 2e9 outliers in early sessions before the helper-bug landed mid-run. Cross-rig validation results: Config Baseline Max VRAM Growth Verdict vllm/long-text 180K + 0.93 23316 23316 0 MiB PASS vllm/long-vision 145K + 0.95 22778 22778 0 MiB PASS Both configs are soak-clean under v1 synthetic shapes. Long-vision was expected to FAIL based on issue #41 — but our 5-turn fixtures reset the conversation each turn while hermes accumulates context across turns. v1 catches a class of issue (raw VRAM accretion across requests); v2 fixtures will need session = continuous conversation to catch the hermes-shaped class. Methodology limitation documented in CHANGELOG. switch.sh — wait_ready crash detection + boot-stage progress Closes the "container crashed silently in 2s, you wait 600s for nothing" UX gap that surfaced today when long-text's MODEL_DIR was misconfigured. - Crash detection: docker inspect -f '{{.State.Running}}' between polls; if false, dump last 30 log lines + exit 1 immediately (5s vs 600s) - Boot-stage markers: grep docker logs for "Resolved architecture", "Loading weights", "Compilation finished", "Capturing CUDA graphs", "Application startup complete" — surface one line per phase transition so the wait isn't silent. Validated end-to-end on long-vision boot today: visible progress at 60s / 68s / 80s / 120s / 176s / 196s = ready. |
||
|
noonghunna
|
563a39e0d3 |
feat(scripts): add soak-test.sh — runtime VRAM accretion validation (closes gap from #41)
Implements docs/diagnostics/soak-test-codex-brief.md (Codex). Fills the
gap exposed by club-3090#41 where a config passes verify-full + verify-stress
but accretes VRAM under realistic multi-turn agent traffic and OOMs
mid-session. The third validation primitive completes the trio:
verify-full.sh — boots correctly?
verify-stress.sh — known cliffs fire under one-shot stress?
soak-test.sh — runtime accretion under multi-turn traffic? (NEW)
scripts/soak-test.sh (195 LOC bash entrypoint):
- Auto-detects container (vllm-qwen36-27b*) + endpoint (mapped port)
- Runs SOAK_SESSIONS=20 × SOAK_TURNS=5 sessions (~10-30 min)
- Warm baseline captured AFTER first completed turn (post torch.compile +
cudagraph capture stabilization)
- Per-turn nvidia-smi snapshot, docker stats baseline + final
- Fails on: engine-dead errors, VRAM growth > SOAK_MAX_GROWTH_MIB (200 MiB),
decode-TPS retention < 80%, or wall-clock > SOAK_TIMEOUT_S (1800s)
- Read-only against deployment; not invoked from launch.sh (opt-in)
scripts/soak-helper.py (432 LOC Python helper):
- 5 fixture shapes: small chat → tool-call inspection → 12K-char tool
result paste → parse_size code completion → reasoning-heavy thinking
problem. Realistic OpenAI-tool-format requests (3 tools: read_file,
grep, run_command).
- SSE stream parser captures TTFT (first content/reasoning_content/tool_calls
delta) + completion_tokens from usage chunk
- cmd_summary computes: p50/p95 TPS + TTFT, first-5/last-5 retention,
VRAM oscillation, slow-turn count. Emits markdown summary + exits
0/1/2 (pass/fail/inconclusive).
Validation done locally without live stack (Codex did not boot a vLLM
container):
- bash -n + py_compile both clean
- All 5 fixture shapes generate valid OpenAI-compat JSON requests
- PASS path: clean turn-log → exit 0
- FAIL paths: VRAM growth > threshold, engine-dead 500, TPS retention
< 80% — all correctly produce exit 1
- GuiPerPT #41 replay: synthetic 1.2 GiB growth across 20 sessions
correctly produces FAIL: "VRAM grew 1165 MiB > 200 MiB threshold"
Live cross-rig validation pending — recommended next:
CONTAINER=vllm-qwen36-27b-long-vision bash scripts/soak-test.sh
(expected: FAIL on the #41-style 0.95 mem-util config, validating that
the test discriminates correctly)
CONTAINER=vllm-qwen36-27b-long-text bash scripts/soak-test.sh
(expected: PASS on the known-good 0.93 baseline)
|