7248ccb605caf53b40bc25cc474b94de7be88668
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 noonghunna
|
b84249c805 |
fix(verify-stress): three live-caught bugs in ceiling ladder (#199)
Bug 1 — get_vram_free_mb inflated on multi-GPU hosts: Summed ALL GPUs → reported 24507 MB when GPU0 truly had 381 free (added idle GPU1's 24 GB). Margin gate was defeated on any host with more GPUs than the model uses. FIX: reads Docker HostConfig.DeviceRequests[0].DeviceIDs to identify the model's GPU(s), passes nvidia-smi -i <those>. Falls back to CUDA_VISIBLE_DEVICES / NVIDIA_VISIBLE_DEVICES env, then all-GPUs with a warning. Live-validated: 353 MB (GPU0 only) vs old 24479 MB. Bug 2 — filler→token ratio overshoots ~18x: The rung used scale = target_tokens / 3.5, treating scale as chars when it's actually block-repetition count. The 95K rung produced 1.77M tokens (674% of n_ctx) → HTTP 400 on a healthy engine. FIX: calibration probe before the ladder sends scale=100, reads back prompt_tokens, computes the real tok/scale_unit ratio. Live-validated: 95K rung now produces 94788 tokens (0.2% error, 36% of n_ctx). Bug 3 — no-op ladder reported PASS: A ladder that tested nothing (all rungs HTTP 400) said 'All stress checks passed.' A rung with target < n_ctx returning 400 is a sizing error, not a clean skip. FIX: distinguishes sizing errors (target < n_ctx + HTTP 400) from legitimate engine rejections (target > n_ctx + HTTP 400). Sizing errors FAIL the probe. 'All rungs skipped' now FAILs with a clear diagnostic instead of silently passing. Tests: 37 pass (9 new — GPU-scoped VRAM query for Bug 1, dual/single/ scoped GPU variants, streaming helper timing extraction with real llama.cpp response shape). All three bugs were caught by running the ladder against the live 262K compose on the dual-3090 dev rig. The unit tests alone didn't catch them — mocks fed the wrong shapes (top-level n_ctx, sum-all-GPUs, clean-skip semantics). Live validation is now part of the workflow. |
||
|
noonghunna
|
07d478c1ab |
feat(verify-stress): capture prefill throughput during NIAH rungs (#199)
Add prefill speed measurement to all NIAH rungs (probes 1, 7, and 8) so the ladder shows whether a context depth is *usable* (fast enough), not just whether it *fits*. Prefill is ~O(N²) over context, so t/s collapses as depth climbs — that latency cliff hits agent workloads (which re-prefill a growing context every turn) before the VRAM cliff. Implementation: - send_streaming_niah() helper: sends NIAH requests with stream:true, measures wall-clock TTFT (time to first token), extracts prefill throughput from the response. - Primary (llama.cpp): timings.prompt_per_second + timings.prompt_ms from the final streaming chunk (confirmed against live endpoint). - Fallback (cross-engine): prompt_tokens / TTFT_seconds when timings is absent (vLLM, SGLang). Output: each rung line now includes prefill data: ✓ 14500 tokens: recalled 'crimson otter 42' (got: ...) prefill=890 t/s (16s) ✓ rung 2/6: target=125K actual=124K tok (47%) recalled '...' prefill=512 t/s (242s) VRAM_free=15800MB Purely additive/informational — no change to rung pass/fail logic or the VRAM-margin gate. A slow-but-fits rung still passes. Tests: 36 pass (8 new — streaming helper timing extraction with mocked llama.cpp timings shape, vLLM cross-engine fallback, HTTP 500 handling). |
||
|
noonghunna
|
5a825a428a |
fix(verify-stress): add CTX_SIZE-scaled ceiling ladder (#199)
verify-stress.sh's NIAH ladder used **fixed rungs** (~10K / 30K / 60K / 90K) that **do not scale to the compose's CTX_SIZE**. Any compose with CTX_SIZE > ~115K was never exercised above ~90K — its top window was untested. #197 surfaced the consequence: a hermes agent accumulated 127K tokens and OOMed on a compose that had 'passed' at 90K max. Fix: probe 8 is now a staggered ladder that climbs from ~95K to ~92% of n_ctx in ~30K increments. Each rung: - sends a NIAH request at that depth - captures VRAM before/after - stops at the first SYSTEM failure (OOM, crash, timeout) or recall miss Recall miss = quality ceiling found: - HTTP 200 + wrong answer → log △, break out of the ladder, pass the probe - HTTP 500 / timeout / 000 → system wall, break + fail the probe This gives you the exact fillable ceiling and the VRAM slope, not just pass/fail at one arbitrary depth. Engine health check + auto-restart: After probes 7 and 8 (both crash-prone at deep contexts), verify-stress checks if the engine is still alive. If it crashed (OOM kill, inductor ICE), it attempts `docker restart` and polls for recovery (up to 120s). This prevents cascade failures in rebench-full.sh's subsequent steps (quality-test, soak-test, aider) that would otherwise run against a dead engine. No-op for CONTAINER=none (endpoint-first mode). Example output for a 262K compose: ✓ rung 1/6: target=95K actual=94K (36%) recalled 'crimson otter 42' VRAM=18200MB ✓ rung 2/6: target=125K actual=124K (47%) recalled 'amber falcon 17' VRAM=15800MB △ rung 3/6: target=155K actual=153K (58%) recall MISS — quality ceiling reached VRAM=12100MB ✓ ceiling ladder: quality ceiling at 153000 tok (58% of n_ctx=262144) — recall miss, passed up to 124000 tok Configuration: CEILING_START_TOKENS First rung target (default: 95000) CEILING_STEP_TOKENS Increment between rungs (default: 30000) CEILING_FRACTION Top rung as fraction of n_ctx (default: 0.92) VRAM_MARGIN_MB Warn if free VRAM drops below this (default: 1024) SKIP_CEILING=1 Skip the entire ladder Also adds: - get_n_ctx() helper: reads /props (llama.cpp) → /v1/models (vLLM) → docker inspect fallback - get_vram_free_mb() helper: sums nvidia-smi memory.free across GPUs - ensure_engine_alive(): health check + auto-restart after crash-prone probes - Probes 1/7 recall miss also downgraded to informational (yellow △) - Probe numbering updated from [N/7] to [N/8] throughout - test-verify-stress-ceiling.sh: 26 tests covering helpers + ladder math |
||
|
noonghunnaandClaude Opus 4.7
|
fe23eff8f0 |
docs: laptop EC-managed power + TQ3 vs fp8 KV naming-trap; verify-stress: auto-bump curl timeout under VLLM_ENFORCE_EAGER
Three documentation/script follow-ups from @easel's #102 re-bench on RTX 5090 Laptop: - HARDWARE.md: new "Laptop GPUs — EC-managed power" subsection. nvidia-smi -pl returns N/A on laptop-class GPUs (EC owns the envelope, not the OS). Documents clock-lock as the only software characterization path on laptops. - CLIFFS.md: new "naming trap" callout in the KV-format section. fp8_e5m2 is 8 bits/token; turboquant_3bit_nc packs 3 bits. At 180K on 24GB, TQ3 fits where fp8 OOMs (4.36 GiB available vs 6.64 GiB needed for fp8). Pin: TQ3 = long-context KV; fp8 = short-context throughput. - verify-stress.sh: auto-detect VLLM_ENFORCE_EAGER=1 in the running container's env via docker inspect; when set, bump STRESS_LONGCTX_TIMEOUT_S 300→600s and STRESS_TOOL_PREFILL_- TIMEOUT_S 240→480s. Eager-mode prefill at 60K-140K runs 200-290s and was false-positiving as HTTP 000 (curl timeout) in @easel's run. Both env vars also exposed for manual override. Refs: noonghunna/club-3090#102 Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunna
|
4f01abb2d5 |
verify-stress: engine-aware diagnostic hints (closes #87)
Adds the same detect_engine() helper as verify-full.sh + soak-test.sh
(parallel implementation; would deduplicate to scripts/preflight.sh in
a future cleanup but not blocking).
What changed:
- Engine class detected once at startup via /props endpoint + chat
completion system_fingerprint + container name fallback
- Header now surfaces engine class alongside container/model/URL
- Diagnostic hints in fail() messages now use ${LOG_CMD} which adapts:
- vllm/sglang → "docker logs ${CONTAINER} 2>&1 | tail -50"
- llamacpp+container → same
- llamacpp+CONTAINER=none (host build) → "check llama-server stdout/stderr where you launched it"
- unknown → "check your engine's stdout/stderr or container logs"
Two fail-message hints (lines 474 and 632) keep their specific grep
filters (empty_strided_cuda for Cliff 1 mech B, DS conv state for
genesis-vllm-patches#17) — those error patterns are vLLM-Genesis-
specific and a llama.cpp host-build user wouldn't hit them anyway.
Comment block in the script explains "Some failure-mode hints are
vLLM-specific" for clarity.
Closes the harness-engine-decoupling triplet started in
|
||
|
noonghunna
|
29718cac99 |
scripts: auto-detect running container + port in verify / bench (closes #52 promise)
Verify-full / verify-stress / bench previously hardcoded URL=http://localhost:8020 + CONTAINER=vllm-qwen36-27b. That assumption silently broke for anyone running a non-default variant — dual-turbo on 8011, dual-dflash on 8012, etc. — making the embedded report.sh chain emit false negatives. Reported by sudepo on club-3090#52. Add preflight_autodetect_endpoint() that: - scans `docker ps` for one of our container patterns (vllm-qwen36-27b* / llama-cpp-qwen36-27b*) - extracts the host port from its 0.0.0.0:<port>->{8000,8080}/tcp mapping - sets URL + CONTAINER, but ONLY for fields the user didn't already set explicitly (env-var override always wins) - prints one [autodetect] line so the user sees what was picked - falls back silently to the existing hardcoded defaults if nothing is detected (no behaviour regression for fresh setups) Wired into the three test scripts. Skip via PREFLIGHT_NO_AUTODETECT=1 for the rare case where the user wants to point at a non-running container or remote endpoint. Verified locally: - autodetect with no env vars → picks up vllm-qwen36-27b-dual-turbo on port 8011 (matches `docker ps`) - autodetect with URL=... CONTAINER=... env set → preserves both - all four scripts pass `bash -n` Direct commit per the docs/cosmetic-direct-to-master convention; this is small, additive, override-preserving and falls back to existing behaviour on detection miss. |
||
|
|
9af1a5245a |
PN30 dst-shaped temp fix: close DS conv state regression class on long-text
Background
----------
Sander shipped Genesis PN30 (a9977d8) to fix the `NotImplementedError` in
`vllm/model_executor/layers/mamba/mamba_utils.py:get_conv_copy_spec` that
fires on DS layout + spec-decode `num_accepted_tokens > 1`. PN30 materializes
`state[src_block_id, :, offset:].contiguous()` and raw-memcpys it into
`state[dest_block_id]`.
ChatGPT/Codex CLI cross-checked the patch and identified a layout-correctness
bug: PN30's `.contiguous()` produces a compact buffer (10240×5 for our config
at offset=1), but the destination block is strided by full state_len (10240×6).
Raw-memcpy packs the compact rows into a layout where row 1+ start at the
wrong destination offset → corrupts DS conv state row strides → eventual TQ
store CUDA assert at probe 4 (multi-turn agent shape) was the surfacing point,
not the root offender.
The corrected fix lives in `collect_mamba_copy_meta`, where both source and
destination block ids are known. For DS conv offset > 0:
tmp = state[dest_block_id].clone()
tmp[..., :tail].copy_(state[src_block_id, ..., offset:])
Then batch-memcpy the full tmp block to `state[dest_block_id]`. Preserves DS
row stride. Reuses PN30's existing module-level temp tensor list + post-batch
stream sync + clear lifecycle (no churn there).
What this commit adds
---------------------
1. **`patch_pn30_dst_shaped_temp_fix.py`** — setup-time text-patch over the
Genesis PN30 wiring file. Patches three sub-patches:
- `pN30_collect_mamba_copy_meta_dst_shaped_temp` (NEW) — adds dst-shaped
temp construction + lifecycle hookup in `collect_mamba_copy_meta`.
- `pN30_get_conv_copy_spec_contiguous` (modified) — old compact `.contiguous()`
fast path now fails closed with a clear error if the collect-time bypass
is ever missed; prevents silent corruption.
- `pN30_module_level_state` + `pN30_do_mamba_copy_block_cleanup` (unchanged)
reused as-is.
444 lines, idempotent via marker. Diagnosis credit: ChatGPT/Codex CLI.
2. **`scripts/setup.sh`** — invokes the PN30 patch after the Genesis checkout,
alongside the existing PN25 register-fix sidecar. Both run automatically
on `bash scripts/setup.sh qwen3.6-27b` after every fresh setup.
3. **`docker-compose.long-text.yml`** — re-enables `VLLM_SSM_CONV_STATE_LAYOUT=DS`
+ `GENESIS_ENABLE_PN30_DS_LAYOUT_SPEC_DECODE=1`, restores `--max-model-len=180000`
from the 145K SD-fallback. Net: +6% TPS and +35K context recovered.
4. **`scripts/verify-stress.sh`** — Cliff 2 (60K + 90K large rungs) deferred
to probe 7 so engine death from architectural OOM doesn't cascade-fail
probes 2-6. Probe 3 strictness relaxed: any HTTP 200 passes, since the
bug class (Cliff 1 mech B inductor leak) surfaces as 500, not low token
counts. The previous strict assertion was an over-applied lesson from the
andthattoo structured-CoT bench (where token count *was* meaningful).
5. **Other 3 TQ3 composes** (long-vision / bounded-thinking / dual-turbo) —
DS layout disable comments updated to point at the now-working PN30 fix.
These composes still need PN25/PN30 enable + per-config validation; this
commit ships long-text only as the validated path.
Validation (long-text 180K + 0.95 mem-util + DS + PN25 v3 + PN30 fix)
---------------------------------------------------------------------
verify-stress.sh fresh-engine run, all 7 probes:
| Probe | Result | Notes |
|--------------------------------------|--------|------------------------------------|
| 1 small needle (10K + 30K) | ✅ | activation budget safe |
| 2 25K tool RETURN | ✅ | sufficient activation headroom |
| 3 IDE-agent one-shot | ✅ | 66 tokens, finish=stop (probe-design fix) |
| 4 multi-turn agent | ✅ | **closed by PN30 fix** |
| 5 LCB-coding | ✅ | **closed by PN30 fix** |
| 6 reasoning 8192 | ✅ | 8192 tokens, finish=length |
| 7 large needle (60K + 90K) | ❌ | Cliff 2 architectural — expected |
6/7 pass. The 1 failure is architectural (DeltaNet GDN forward state OOM at
50-60K single-prompt on 24 GB single card) — pre-tracked, no fix possible
on single card.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Co-Authored-By: Codex CLI (ChatGPT) <[email protected]>
|
||
|
|
a62ad78a4e |
PN25 v3: close Cliff 1 mech B (club-3090#16) on long-text via setup-time Genesis backport
Background ---------- Cliff 1 mech B is the inductor-compiled FFN intermediate buffer leak that PN12 (eager-mode SiluAndMul.forward_cuda pool) doesn't reach. vLLM v0.20 with `compilation_config.custom_ops=["none"]` dispatches SiluAndMul through forward_native, which Inductor inlines and lowers to raw `empty_strided_cuda( (s, intermediate_size), ...)` — bypassing PN12's FFNIntermediateCache pool. Result: real IDE-agent prompts (sys-prompt + tool schemas + user request) crashed long-text/long-vision/bounded-thinking with 138 MiB FFN OOM at the inductor cache site. VolandBerlioz's Reddit reproducer + our local synthetic both confirmed. Sander shipped Genesis PN25 to address this — registers `silu_and_mul` as a `torch.library.custom_op` so Inductor treats it as opaque (can't inline). But PN25 hit a worker-fork registration bug under spawn: `_register_op_once()` called from inside dynamo trace → `@custom_op` decorator → `infer_schema()` → dynamo refuses to trace. Filed Sander/genesis-vllm-patches#16 with the trace + analysis. What this commit adds --------------------- A setup-time backport that lets us enable PN25 NOW, before Sander's upstream fix lands in our pinned version (or in case it doesn't transfer to TP=1). Two parts in `patch_pn25_genesis_register_fix.py`: 1. **Genesis-side change** to `silu_and_mul_customop.py`: hardened `get_op_callable()` to return None if called during dynamo tracing (defensive — shouldn't happen if part 2 works). 2. **Wiring change** to `patch_N25_silu_inductor_safe_pool.py`: text-patches `vllm/model_executor/layers/activation.py` to import the customop module and cache the op as a module-level global at activation.py import time. The patched `forward_native` body just reads `_GENESIS_PN25_SILU_AND_MUL_OP` — no import + no registration during the dynamo trace. Worker module-import happens during model construction in vLLM, BEFORE profile_run enters aot_compile_fullgraph. Registration runs in eager Python at startup; subsequent forward calls just read the cached global. Wired into setup.sh after Genesis checkout. Idempotent via marker. Also in this commit ------------------- - **long-text.yml backed off 214K + 0.985 → 180K + 0.95.** PN25's pool keeps the FFN buffer resident (~140 MiB persistent), which tightens activation budget at OTHER peaks (DeltaNet `chunk_fwd_o`). 0.985 left only 26 MiB free at 30K probe — OOM. 0.95 frees ~480 MiB for activation comfort. Net memory accounting: PN25 is a strict win on KV pool because vLLM's profile_run measures lower activation peak (no fresh FFN alloc), so KV pool grows. Max concurrency at 180K: 1.07x without PN25 → 1.49x with PN25 + 0.95 (or 1.64x at 0.97 if we'd held it). - **verify-stress.sh probe 3 hardened.** Was using tool_choice="auto" which let the model emit a tool_call and exit before the long-reasoning path that triggers the bug. Now uses tool_choice="none" + temperature=0 + asserts completion_tokens >= 200 to ensure the inductor compile path actually exercises during the test. Validation on long-text 180K + 0.95 + PN25 v3 --------------------------------------------- | Probe | Result | Notes | |---------------------------------|--------|----------------------------------| | 1.1 Long-ctx needle 9.8K | ✅ PASS | activation budget safe | | 1.2 Long-ctx needle 29K | ✅ PASS | activation budget safe at 30K | | 1.3 Long-ctx needle 60K | ❌ FAIL | Cliff 2 architectural (DeltaNet) | | 2 25K tool RETURN | ❌ FAIL | FA varlen workspace (Sander #15) | | 3 IDE-agent one-shot | ✅ PASS | **Closed by PN25 v3** ⭐ | | 4 Multi-turn agent | ✅ PASS | **Closed by PN25 v3** ⭐ | | 5 LCB-coding | ❌ FAIL | DS conv state (Sander #17) | | 6 Reasoning-heavy 8192 | ✅ PASS | pure reasoning works clean | The 4 failures are pre-tracked separately: - Cliff 2 (#1.3): architectural, no fix at single-card; route to dual or llama.cpp - 25K tool RETURN (#2): Sander shipped PN31 (`753344b`) — pending our cross-rig - LCB-coding (#5): Sander shipped PN30 (`a9977d8`) — pending our cross-rig Sander has since shipped PN25 fix upstream (`d92bcb3` on dev) using a slightly different approach (hasattr check on global registry). Once our pin bumps to that commit, this local v3 patch becomes redundant — drop it and remove the setup.sh hook in a follow-up. Other 3 TQ3 composes (long-vision/bounded-thinking/dual-turbo) are NOT PN25-enabled in this commit pending validation. Long-text is the only compose with PN25 active + verified. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> Co-Authored-By: Codex CLI (ChatGPT) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
5e745c5c85 |
verify-stress: add 3 probes to cover the bug shapes we missed
Closes #20 + #13 in linked commits; this commit ships the prevention surface for catching the new bug classes that surfaced today before they hit users. verify-stress.sh grew from 3 probes → 6 probes: | # | Probe | Catches | |---|-------|---------| | 1 | Long-context needle ladder (10K/30K/60K/90K) | unchanged | | 2 | 25K-token tool RETURN prefill | unchanged | | 3 | IDE-agent one-shot (sys + 10 tool schemas + user) | club-3090#16 (Cliff 1 mech B inductor leak) | | 4 | Multi-turn agent (sys + tools + 4-turn history) | inductor compile-path bugs that need prior assistant/tool messages to fire | | 5 | LCB-coding shape (LeetCode problem + structured plan) | genesis-vllm-patches#17 (DS conv state crash) | | 6 | Reasoning-heavy (math + max_tokens=8192) | spec-decode AL collapse, mamba_cache_mode='align' interactions over long generation | Each probe is fail-fast (one request, ~10-60s if green; instant 500 if the bug fires) and prints actionable hints on failure pointing at the relevant issue tracker + workaround. Probe #6 also asserts completion_tokens >= 500 — catches the silent failure mode where the engine returns 200 OK but stops generation early due to AL collapse or hidden truncation. The prior probes only checked HTTP status, which would have missed this regression class. Probe #5 prompt is a representative LeetCode subarray-sum problem that asks for the GOAL/STATE/ALGO/EDGE/VERIFY structured plan format used in our bounded-thinking bench — same shape that triggered the DS conv state crash on every request during the LCB v6 50-problem run earlier today. This commit also closes the loop on: - club-3090#20 (launch.sh port/container) — closed with `77ca576` reference - club-3090#13 (TurboQuant on hybrid) — closed with PR #23 + GENESIS_ENABLE_P4=1 reference Issues that remain open and tracked: - club-3090#16 (Cliff 1 mech B) — awaiting Genesis PN25 worker-fork fix - club-3090#24 (re-pin to main when Sander merges) - genesis-vllm-patches#16 (PN25 registration) — escalated, PR offered - genesis-vllm-patches#17 (DS conv state) — filed today Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
b62b6b1de4 |
walk back: Cliff 1 mech B reproduces on real IDE-agent prompts (club-3090#16)
PR #23's "synthetic 50K-token tool-prefill stress passes on v0.20" finding does NOT translate to real IDE-agent workloads. Reproduced 2026-05-01 PM: a 5,900-char system prompt + 10 typical tool schemas + 346-char user request + max_tokens=2000 crashes the engine on long-text / long-vision / bounded-thinking / dual-turbo with a 98 MiB OOM at: inductor_cache/.../py:1208 buf9 = empty_strided_cuda((s18, 17408), (17408, 1), torch.float16) torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 98.00 MiB. Same site as VolandBerlioz's Reddit reproducer. Genesis PN12 patches the eager `SiluAndMul.forward_cuda` but vLLM's torch.compile inductor inlines `forward_native`, bypassing the FFNIntermediateCache pool. PN25 (the proper compile-path opaque-op fix) is on Genesis dev but blocked by a worker-fork registration bug that fires `torch.library.infer_schema` during dynamo trace. Escalated to Sandermage as urgent (genesis-vllm-patches#16 escalation comment). What changes in this commit --------------------------- 1. **docs/SINGLE_CARD.md** — restructured the "limitation to know" section from one cliff to two. New Cliff 1 mech B section explicitly tells IDE-agent users (Cline / OpenCode / Roo / Claude Code / Cursor) to default to tools-text.yml until PN25 lands. 2. **docs/DUAL_CARD.md** — annotated the dual.yml entry as the strongly-recommended path for IDE coding agents (fp8 KV avoids the inductor inlining bug; dual-turbo TQ3 KV is affected). 3. **docs/UPSTREAM.md** — bumped PN25 from "✅ Closed in dev tip; opt-in" to "🔴 Shipped on dev BUT blocked by worker-fork registration bug". Added new row for the DS conv state crash (genesis-vllm-patches#17, filed today after LCB v6 bench triggered it on every request). 4. **Compose headers** (long-text.yml / long-vision.yml / bounded-thinking.yml / dual-turbo.yml) — prepended a "⚠️ NOT SAFE FOR IDE-AGENT WORKLOADS WITH TOOL SCHEMAS" warning to the top of each header with the specific failure mode + workaround. 5. **scripts/verify-stress.sh** — added a third check: IDE-agent one-shot prompt (sys + 10 tool schemas + user request + max_tokens=2000). Catches the bug on a fresh boot of any affected compose. Existing checks #1 (long-context needle) and #2 (25K-token tool RETURN prefill) didn't catch this — different prefill shape, different inductor compile path. TODO comments added for 3 more probes (multi-turn agent / LCB-coding / reasoning-heavy) — left for follow-up. Public corrections ------------------ Already posted (this commit only ships the docs + script changes): - club-3090#16 update with synthetic-prompt repro + walked-back claims - discussion #18 reply to @lolren correcting today's earlier recommendation (dual.yml is the safe path for IDE agents on dual-card; was overconfident earlier today saying long-* variants were closed) - Sandermage genesis-vllm-patches#16 escalation comment - Sandermage genesis-vllm-patches#17 (new — DS conv state crash) Why this matters ---------------- Every IDE-agent user on master who follows our published recommendations (long-text / long-vision / bounded-thinking) and sends a request with a tool schema in the system prompt is on a coin-flip with this bug. Our test surface didn't catch it because canonical narrative+code prompts don't trigger the offending inductor code path. The new verify-stress probe will catch it on every future boot. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
53d0663a50 |
genesis: bump pin v7.62 → v7.64 + add compile-safe FFN sidecar (#16)
Setup script: pin updated 917519b → 64dd18b. Verified clean on tools-text
(75K + fp8 KV + MTP, PN8 enabled): verify-full.sh 8/8 + verify-stress.sh
tool-prefill OK. v7.64 release notes adopted: PN17 is Sandermage's anchored
version of our P104 sidecar (FA2 softmax_lse runtime clamp), PN19 sets
max_split_size_mb=20 during model load.
long-text.yml (218K + 0.985 + TQ3 + MTP):
- Genesis pin v7.64 (PN17 enabled, PN19 disabled — costs ~120 MiB KV pool
on Ampere consumer; the documented "200-500 MiB win on H100" is negative
on our hardware).
- Drops max-model-len 218000 → 205000 because PN17 reserves ~120 MiB of KV
pool space at boot ("estimated maximum model length is 206400" is the
engine pre-check failure mode otherwise).
- P104 sidecar (patch_fa_max_seqlen_clamp.py) kept mounted but env-disabled;
PN17 covers the same path via Sandermage's anchored fix. Flip the env
var back on if PN17 turns out not to cover turboquant_attn.py for some
config.
- Mounts new patch_pn12_compile_safe_custom_op.py — opaque torch.library
custom_op for the inductor-compiled forward_native FFN path that the
eager-mode forward_cuda sidecar can't reach (issue #16, mech B). Only
active when GENESIS_ENABLE_PN12_FFN_INTERMEDIATE_POOL=1; harmless when
off. Forward_native body simplified to a single static-guard branch on
module-level _PN12_ENABLED so Dynamo specializes at trace time instead
of compiling both branches (the else-branch's plain F.silu/mul lowers
to empty_strided_cuda, defeating the patch).
verify-stress.sh: fixed silent false-positive where check_tool_prefill
returned 0 even after fail() because rm -f cleanup clobbered $? — now
captures rc before cleanup and propagates.
CLIFFS.md: cross-reference Sandermage's broader 8-cliff catalog and add
"vLLM pin compatibility status" section documenting the v0.20 workspace-
lock regression (PR #39226) that blocks our config and is unrelated to
Cliff 1/2.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
noonghunna
|
5060e22a6c |
Split verify-full.sh → verify-full.sh (fast functional) + verify-stress.sh (boundary)
Recent additions to verify-full.sh (#8 tool-prefill OOM, #9 cascade detection, #10 MTP AL) made the script slow — the longctx needle ladder (#7) alone could run 5+ min, and the full 10-check suite was approaching 10 min. Awkward for "is the stack functional" iteration during dev work. verify-full.sh (8 fast checks, ~1-2 min) 1. Server reachable 2. Genesis patches applied 3. Basic completion (Paris) 4. Tool calling 5. Streaming (SSE) 6. Thinking / reasoning mode 7. Output quality / cascade detection (was #9) 8. MTP acceptance length threshold (was #10) Run: after every config change to confirm the stack still serves cleanly. verify-stress.sh (2 boundary checks, ~5-10 min) 1. Long-context needle ladder (4 depths, 10K / 30K / 60K / 90K) — was #7 2. Tool response prefill OOM (~25K-token mock tool message) — was #8 Run: before publishing or when investigating prefill-OOM regressions specifically. Smoke-tested against dual.yml on dual-card: verify-full.sh: 8/8 green in 65 seconds verify-stress.sh: 2/2 green (skipped longctx for this smoke), 15s Same env-var conventions (URL, MODEL, CONTAINER, SKIP_LONGCTX, SKIP_TOOL_PREFILL, PREFILL_TARGET_CHARS). Doc updates: - top-level README repo layout: lists both scripts with timing/scope - docs/ARCHITECTURE.md: scripts/ section + design rules updated - models/qwen3.6-27b/USE_CASES.md: tool-prefill reference points at verify-stress.sh now - models/qwen3.6-27b/CHANGELOG.md: dated entry documenting the split |