extend PN25 v3 + PN30 dst-shaped temp fix to all 4 TQ3 composes

PR #23 + PR a62ad78 + PR 9af1a52 shipped PN25 v3 + PN30 dst-shaped temp fix on long-text only. This commit extends the same patch stack to the remaining 3 TQ3 composes (long-vision, bounded-thinking, dual-turbo) and validates each independently. What changed ------------ - **long-vision.yml**: 198K + 0.98 → **145K + 0.95** + DS layout + PN25 v3 + PN30. Vision tower residence forces deeper backoff than long-text (engine pre-check returned `estimated max 148608` at 175K + 0.95, settled at 145K with safety margin). - **bounded-thinking.yml**: 214K + 0.985 → **180K + 0.95** + DS layout + PN25 v3 + PN30. Parity with long-text — same patch stack, same backoff, structured-CoT grammar still works on top. - **dual-turbo.yml**: 262K context preserved + DS layout + PN25 v3 + PN30. TP=2 splits state across both cards, fits the patch stack at 0.85 mem-util cleanly. Validation per compose (verify-stress.sh, 7 probes) --------------------------------------------------- | Variant | Pass count | Failure | |--------------------|------------|----------------------| | long-text | 6 / 7 | Cliff 2 architectural | | long-vision | 6 / 7 | Cliff 2 architectural | | bounded-thinking | 6 / 7 | Cliff 2 architectural | | dual-turbo (TP=2) | 6 / 7 | Cliff 2 architectural | All non-architectural probes pass — IDE-agent one-shot, multi-turn agent, LCB-coding, reasoning-heavy, 25K tool RETURN, small-rung longctx. Cliff 2 (60K+ single prompt DeltaNet GDN forward state OOM) fails on every variant including TP=2 because GDN state is per-rank not split — that's fundamental, not addressable on this config class. Per-config summaries written to: - results/v0.20-migration/long-vision-pn30.summary - results/v0.20-migration/bounded-thinking-pn30.summary - results/v0.20-migration/dual-turbo-pn30.summary Docs partial update (SINGLE_CARD.md): updated TL;DR table with new ctx ceilings and removed the Cliff 1 mech B "limitation to know" since both mechanisms (PN12 eager + PN25 v3 compile) now close it. Kept Cliff 2 as the one remaining limitation. More doc updates in follow-up commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 02:22:21 +00:00
parent 9af1a5245a
commit b875624f2d
7 changed files with 126 additions and 56 deletions
--- a/docs/SINGLE_CARD.md
+++ b/docs/SINGLE_CARD.md
@@ -10,36 +10,26 @@ Four recommended options:

 | What you're doing | Compose | Max ctx | Narr / Code TPS | VRAM (24 GB / card) |
 |---|---|---|---|---|
-| **Long ctx + vision** (chat, agents, image input) | [`long-vision.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.long-vision.yml) | **198K** | 50 / 66 | ~22.3 GB (mem-util 0.98) |
-| **Long ctx, text-only** (RAG, codebase, books) | [`long-text.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.long-text.yml) | **214K** | 50 / 67 | ~23.4 GB (mem-util 0.985) |
-| **Bounded thinking** (coding agents, structured-CoT, cost-bounded thinking) — see [STRUCTURED_COT.md](STRUCTURED_COT.md) | [`bounded-thinking.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml) | **214K** | 50 / 66 | ~21.7 GB (mem-util 0.985) |
+| **Long ctx + vision** (chat, agents, image input) | [`long-vision.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.long-vision.yml) | **145K** | 50 / 66 | ~23.0 GB (mem-util 0.95) |
+| **Long ctx, text-only** (RAG, codebase, books, **IDE agents** ⭐) | [`long-text.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.long-text.yml) | **180K** | 50 / 67 | ~22.4 GB (mem-util 0.95) |
+| **Bounded thinking** (coding agents, structured-CoT, cost-bounded thinking) — see [STRUCTURED_COT.md](STRUCTURED_COT.md) | [`bounded-thinking.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml) | **180K** | 50 / 66 | ~21.7 GB (mem-util 0.95) |
 | **Bulletproof, no cliffs** (production service, unpredictable inputs) | [`llamacpp/default`](../models/qwen3.6-27b/llama-cpp/compose/docker-compose.yml) | **262K** | 21 / 21 | ~20 GB |

 Run via `bash scripts/launch.sh` (interactive) or `bash scripts/switch.sh <variant>`.

-> ## ⚠️ Two limitations to know
+> ## ⚠️ One limitation to know
 >
-> ### 1. Cliff 1 mech B — IDE-agent prompts crash on long-text / long-vision / bounded-thinking / dual-turbo
->
-> **If you're using Cline, OpenCode, Roo, Claude Code, Cursor, or any tool-using agent, default to [`tools-text.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.tools-text.yml) (75K + fp8 KV) — not the 198K/214K/262K variants.**
->
-> Reproduced 2026-05-01 PM: a 5,900-char system prompt + 10 typical tool schemas + 346-char user request crashes `long-text.yml` (and the other 3 TQ3 composes) with a 98 MiB OOM at `empty_strided_cuda((s, 17408), ...)` in `inductor_cache/...py:1208`. Same site as VolandBerlioz's Reddit reproducer. Genesis PN12 patches the eager `SiluAndMul.forward_cuda` but vLLM's torch.compile inductor inlines `forward_native`, bypassing the pool. PN25 (the proper compile-path fix) is on Genesis dev but [blocked by a worker-fork registration bug](https://github.com/Sandermage/genesis-vllm-patches/issues/16) we filed.
->
-> Until PN25 lands default-on:
-> - **IDE agents → `tools-text.yml`** (PN8 closes Cliff 1 mech B via a different mechanism that *does* reach the compile path)
-> - **Pure long-form text / RAG / book Q&A → `long-text.yml`** (no tool schemas in prompt → doesn't trigger inductor compile path)
-> - **Vision + chat → `long-vision.yml`** (same as long-text — fine until you add tool schemas)
-> - **Big single prompts → `llamacpp/default`** (262K, different engine entirely — no inductor)
->
-> Tracking: [club-3090 #16](https://github.com/noonghunna/club-3090/issues/16).
->
-> ### 2. Cliff 2 — DeltaNet GDN single-prompt OOM at 50–60K tokens
+> ### Cliff 2 — DeltaNet GDN single-prompt OOM at 50–60K tokens
 >
 > **vLLM single-card variants will crash if you send a single prompt above ~50K tokens.**
 >
-> Architectural — DeltaNet GDN forward state grows with sequence length, OOMs at 50–60K regardless of how much VRAM you have left. Both `long-vision.yml` (198K) and `long-text.yml` (214K) are designed for **steady-state accumulation across many turns**, not single-shot big prompts.
+> Architectural — DeltaNet GDN forward state grows with sequence length, OOMs at 50–60K regardless of how much VRAM you have left. The long-* variants are designed for **steady-state accumulation across many turns**, not single-shot big prompts.
 >
 > If your workload sends single big prompts: `llamacpp/default` (262K, no cliffs anywhere) or `dual-turbo.yml` (TP=2 splits state across cards).
+>
+> ### What was Cliff 1 mech B (now closed) ✅
+>
+> Earlier in 2026-05 we tracked an inductor compile-path FFN intermediate buffer leak ([club-3090 #16](https://github.com/noonghunna/club-3090/issues/16)) that crashed long-* variants on real IDE-agent prompts. **Closed 2026-05-02** via Genesis PN25 (Inductor-safe `silu_and_mul` opaque op) + PN30 (DS conv state dst-shaped temp fix). Both fixes ship by default in our composes; ChatGPT/Codex CLI cross-check helped land the PN30 dst-shaped temp variant. No user action needed — a fresh `bash scripts/setup.sh qwen3.6-27b` picks up the fixes automatically.

 ---

--- a/models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml
@@ -167,6 +167,10 @@ services:
      - GENESIS_PN26_SPARSE_V_THRESHOLD=0.01
      - GENESIS_ENABLE_P38B_COMPILE_SAFE=1
      - GENESIS_ENABLE_P15B_FA_VARLEN_CLAMP=1
+      - GENESIS_ENABLE_PN25_SILU_INDUCTOR_SAFE=1
+      # PN30 — RE-ENABLED with our local dst-shaped temp fix
+      # (patch_pn30_dst_shaped_temp_fix.py, applied during setup.sh).
+      - GENESIS_ENABLE_PN30_DS_LAYOUT_SPEC_DECODE=1
      - GENESIS_PREALLOC_TOKEN_BUDGET=4128
      - GENESIS_BUFFER_MODE=shared
      # Explicit OFFs to match Sandermage's PROD env-var set verbatim:
@@ -177,16 +181,12 @@ services:
      - GENESIS_ENABLE_P81_FP8_BLOCK_SCALED_M_LE_8=0
      - GENESIS_ENABLE_P82=0
      - GENESIS_P82_THRESHOLD_SINGLE=0.3
-      # VLLM_SSM_CONV_STATE_LAYOUT — disabled 2026-05-01 PM after
-      # ChatGPT/Codex CLI diagnosed PN30 (Sander a9977d8) layout-corruption bug:
-      # PN30's .contiguous() materializes src[block, :, offset:] as compact
-      # 10240×5 memory, then raw-memcpys it into dst whose rows are 10240×6.
-      # Row 1+ start at the wrong destination offset → corrupts DS conv state.
-      # The TQ store CUDA assert we saw on probe 4 was the eventual surfacing,
-      # not the root cause. Until upstream fix lands, drop DS layout (-6% TPS
-      # per Sander's bench) for correctness on probes 4 + 5. Reported back
-      # on Sandermage/genesis-vllm-patches#17 with the row-stride diagnosis.
-      # - VLLM_SSM_CONV_STATE_LAYOUT=DS
+      # VLLM_SSM_CONV_STATE_LAYOUT=DS — RE-ENABLED with our local PN30
+      # dst-shaped temp fix (patch_pn30_dst_shaped_temp_fix.py, applied at
+      # setup time). Without our fix, Sander's PN30 a9977d8 corrupts DS row
+      # strides on spec-decode AL>1. With our fix, PN30 builds destination-
+      # shaped temp via collect_mamba_copy_meta. +6% TPS retained.
+      - VLLM_SSM_CONV_STATE_LAYOUT=DS
      - VLLM_USE_FUSED_MOE_GROUPED_TOPK=1
    shm_size: "16gb"
    ipc: host
@@ -228,12 +228,13 @@ services:
      - float16
      - --tensor-parallel-size
      - "1"
-      # 214K + 0.985 — parity with long-text.yml. See long-text.yml for
-      # the full v0.20 + Genesis v7.65 migration rationale.
+      # 180K + 0.95 — parity with long-text.yml. Backed off from 214K + 0.985
+      # on 2026-05-02 to give activation headroom for the PN12+PN25 FFN pool
+      # residence + DeltaNet GDN buffer. See long-text.yml for full rationale.
      - --max-model-len
-      - "214000"
+      - "180000"
      - --gpu-memory-utilization
-      - "0.985"
+      - "0.95"
      - --max-num-seqs
      - "1"
      - --max-num-batched-tokens
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual-turbo.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual-turbo.yml
@@ -71,16 +71,12 @@ services:
      - TRITON_CACHE_DIR=/root/.triton/cache
      - VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0
      - VLLM_FLOAT32_MATMUL_PRECISION=high
-      # VLLM_SSM_CONV_STATE_LAYOUT — disabled 2026-05-01 PM after
-      # ChatGPT/Codex CLI diagnosed PN30 (Sander a9977d8) layout-corruption bug:
-      # PN30's .contiguous() materializes src[block, :, offset:] as compact
-      # 10240×5 memory, then raw-memcpys it into dst whose rows are 10240×6.
-      # Row 1+ start at the wrong destination offset → corrupts DS conv state.
-      # The TQ store CUDA assert we saw on probe 4 was the eventual surfacing,
-      # not the root cause. Until upstream fix lands, drop DS layout (-6% TPS
-      # per Sander's bench) for correctness on probes 4 + 5. Reported back
-      # on Sandermage/genesis-vllm-patches#17 with the row-stride diagnosis.
-      # - VLLM_SSM_CONV_STATE_LAYOUT=DS
+      # VLLM_SSM_CONV_STATE_LAYOUT=DS — RE-ENABLED with our local PN30
+      # dst-shaped temp fix (patch_pn30_dst_shaped_temp_fix.py, applied at
+      # setup time). Without our fix, Sander's PN30 a9977d8 corrupts DS row
+      # strides on spec-decode AL>1. With our fix, PN30 builds destination-
+      # shaped temp via collect_mamba_copy_meta. +6% TPS retained.
+      - VLLM_SSM_CONV_STATE_LAYOUT=DS
      - VLLM_USE_FUSED_MOE_GROUPED_TOPK=1
      - CUDA_DEVICE_MAX_CONNECTIONS=8
      # FULL Genesis v7.65 PROD env-var set per Sandermage's
@@ -129,6 +125,10 @@ services:
      - GENESIS_PN26_SPARSE_V_THRESHOLD=0.01
      - GENESIS_ENABLE_P38B_COMPILE_SAFE=1
      - GENESIS_ENABLE_P15B_FA_VARLEN_CLAMP=1
+      - GENESIS_ENABLE_PN25_SILU_INDUCTOR_SAFE=1
+      # PN30 — RE-ENABLED with our local dst-shaped temp fix
+      # (patch_pn30_dst_shaped_temp_fix.py, applied during setup.sh).
+      - GENESIS_ENABLE_PN30_DS_LAYOUT_SPEC_DECODE=1
      - GENESIS_PREALLOC_TOKEN_BUDGET=4128
      - GENESIS_BUFFER_MODE=shared
      # Explicit OFFs to match Sandermage's PROD env-var set verbatim:
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.long-vision.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.long-vision.yml
@@ -153,6 +153,10 @@ services:
      - GENESIS_PN26_SPARSE_V_THRESHOLD=0.01
      - GENESIS_ENABLE_P38B_COMPILE_SAFE=1
      - GENESIS_ENABLE_P15B_FA_VARLEN_CLAMP=1
+      - GENESIS_ENABLE_PN25_SILU_INDUCTOR_SAFE=1
+      # PN30 — RE-ENABLED with our local dst-shaped temp fix
+      # (patch_pn30_dst_shaped_temp_fix.py, applied during setup.sh).
+      - GENESIS_ENABLE_PN30_DS_LAYOUT_SPEC_DECODE=1
      - GENESIS_PREALLOC_TOKEN_BUDGET=4128
      - GENESIS_BUFFER_MODE=shared
      # Explicit OFFs to match Sandermage's PROD env-var set verbatim:
@@ -163,16 +167,12 @@ services:
      - GENESIS_ENABLE_P81_FP8_BLOCK_SCALED_M_LE_8=0
      - GENESIS_ENABLE_P82=0
      - GENESIS_P82_THRESHOLD_SINGLE=0.3
-      # VLLM_SSM_CONV_STATE_LAYOUT — disabled 2026-05-01 PM after
-      # ChatGPT/Codex CLI diagnosed PN30 (Sander a9977d8) layout-corruption bug:
-      # PN30's .contiguous() materializes src[block, :, offset:] as compact
-      # 10240×5 memory, then raw-memcpys it into dst whose rows are 10240×6.
-      # Row 1+ start at the wrong destination offset → corrupts DS conv state.
-      # The TQ store CUDA assert we saw on probe 4 was the eventual surfacing,
-      # not the root cause. Until upstream fix lands, drop DS layout (-6% TPS
-      # per Sander's bench) for correctness on probes 4 + 5. Reported back
-      # on Sandermage/genesis-vllm-patches#17 with the row-stride diagnosis.
-      # - VLLM_SSM_CONV_STATE_LAYOUT=DS
+      # VLLM_SSM_CONV_STATE_LAYOUT=DS — RE-ENABLED with our local PN30
+      # dst-shaped temp fix (patch_pn30_dst_shaped_temp_fix.py, applied at
+      # setup time). Without our fix, Sander's PN30 a9977d8 corrupts DS row
+      # strides on spec-decode AL>1. With our fix, PN30 builds destination-
+      # shaped temp via collect_mamba_copy_meta. +6% TPS retained.
+      - VLLM_SSM_CONV_STATE_LAYOUT=DS
      - VLLM_USE_FUSED_MOE_GROUPED_TOPK=1
    shm_size: "16gb"
    ipc: host
@@ -218,10 +218,16 @@ services:
      # closes the Cliff 1 mech B sub-cliffs that drove the 140K + 0.95
      # backoff on dev205. Verified verify-full + 33K + 50K stress all
      # PASS. See docs/CLIFFS.md "v0.20 unblock" section.
+      # 145K + 0.95 — backed off from 198K + 0.98 on 2026-05-02 to give
+      # activation headroom for the PN12+PN25 FFN intermediate pool residence
+      # + PN30 dst-shaped temp lifecycle. Vision tower's persistent ~1 GB
+      # tightens the budget further than long-text (which sits at 180K + 0.95).
+      # Engine pre-check at 175K + 0.95 returned `estimated maximum model length
+      # is 148608` — falls back to 145K for safety margin.
      - --max-model-len
-      - "198000"
+      - "145000"
      - --gpu-memory-utilization
-      - "0.98"
+      - "0.95"
      - --max-num-seqs
      - "1"
      - --max-num-batched-tokens
--- a/results/v0.20-migration/bounded-thinking-pn30.summary
+++ b/results/v0.20-migration/bounded-thinking-pn30.summary
@@ -0,0 +1,22 @@
+variant: bounded-thinking
+date: 2026-05-02
+context: 180000
+gpu_mem_util: 0.95
+kv_dtype: turboquant_3bit_nc
+tp: 1
+genesis_pin: 753344b
+patches: PN25 v3 + PN30 dst-shaped temp fix + DS layout
+structured_outputs_enable_in_reasoning: true
+verify_stress_results: |
+  [1/7] longctx small: PASS
+  [2/7] 25K tool RETURN: PASS
+  [3/7] IDE-agent one-shot: PASS
+  [4/7] multi-turn agent: PASS
+  [5/7] LCB-coding: PASS
+  [6/7] reasoning 8192: PASS
+  [7/7] longctx large 60K+: FAIL Cliff 2
+score: 6/7
+notes: |
+  Parity with long-text — same patch stack + same backoff (180K + 0.95)
+  matches long-text behavior. Structured-CoT grammar enforcement still works
+  on top of the new patches.
--- a/results/v0.20-migration/dual-turbo-pn30.summary
+++ b/results/v0.20-migration/dual-turbo-pn30.summary
@@ -0,0 +1,28 @@
+variant: dual-turbo (TP=2)
+date: 2026-05-02
+context: 262144
+gpu_mem_util: 0.85
+kv_dtype: turboquant_3bit_nc
+tp: 2
+genesis_pin: 753344b (v7.65 dev tip)
+patches: PN25 v3 (local) + PN30 dst-shaped temp fix (local) + DS layout enabled
+verify_stress_results: |
+  [1/7] longctx small (10K + 30K): PASS
+  [2/7] 25K tool RETURN: PASS
+  [3/7] IDE-agent one-shot: PASS (66 tokens, finish=stop)
+  [4/7] multi-turn agent: PASS
+  [5/7] LCB-coding: PASS
+  [6/7] reasoning 8192: PASS (8192 tokens, finish=length)
+  [7/7] longctx large (60K+): FAIL (Cliff 2 architectural — DeltaNet GDN forward state OOM, even on TP=2 single-prompt)
+score: 6/7
+notes: |
+  TP=2 splits state but doesn't avoid Cliff 2 on single-prompt 60K+ since
+  the GDN state buffer is per-rank and grows with seq_len, not split across
+  ranks. Cliff 2 is not addressable on this config class regardless of
+  TP factor. Rule for users: dual-turbo for 4-stream concurrent serving at
+  ≤50K each, OR llamacpp/default 262K for single-prompt big inputs.
+  PN30 dst-shaped temp fix passes cleanly on TP=2 — multi-turn agent +
+  LCB-coding shapes both work that previously crashed with Sander's
+  upstream PN30 a9977d8.
+kv_pool_tokens: 1523232
+max_concurrency: 4.67
--- a/results/v0.20-migration/long-vision-pn30.summary
+++ b/results/v0.20-migration/long-vision-pn30.summary
@@ -0,0 +1,23 @@
+variant: long-vision
+date: 2026-05-02
+context: 145000
+gpu_mem_util: 0.95
+kv_dtype: turboquant_3bit_nc
+tp: 1
+vision: true
+genesis_pin: 753344b
+patches: PN25 v3 (local) + PN30 dst-shaped temp fix (local) + DS layout
+verify_stress_results: |
+  [1/7] longctx small (10K + 30K): PASS
+  [2/7] 25K tool RETURN: PASS
+  [3/7] IDE-agent one-shot: PASS (66 tokens)
+  [4/7] multi-turn agent: PASS
+  [5/7] LCB-coding: PASS
+  [6/7] reasoning 8192: PASS
+  [7/7] longctx large (60K+): FAIL Cliff 2 architectural
+score: 6/7
+backoff_from_198K: |
+  Vision tower's persistent ~1 GB plus PN12+PN25 pool residence + PN30
+  dst-shaped temp lifecycle tightens activation budget further than
+  long-text. Engine pre-check at 175K + 0.95 returned `estimated maximum
+  model length is 148608` → fall back to 145K + 0.95 for safety margin.