From b875624f2d9ea41d6ead3a8563f9ef37ffbdb59c Mon Sep 17 00:00:00 2001 From: noonghunna <10742901+noonghunna@users.noreply.github.com> Date: Sat, 2 May 2026 02:22:21 +0000 Subject: [PATCH] extend PN25 v3 + PN30 dst-shaped temp fix to all 4 TQ3 composes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #23 + PR a62ad78 + PR 9af1a52 shipped PN25 v3 + PN30 dst-shaped temp fix on long-text only. This commit extends the same patch stack to the remaining 3 TQ3 composes (long-vision, bounded-thinking, dual-turbo) and validates each independently. What changed ------------ - **long-vision.yml**: 198K + 0.98 → **145K + 0.95** + DS layout + PN25 v3 + PN30. Vision tower residence forces deeper backoff than long-text (engine pre-check returned `estimated max 148608` at 175K + 0.95, settled at 145K with safety margin). - **bounded-thinking.yml**: 214K + 0.985 → **180K + 0.95** + DS layout + PN25 v3 + PN30. Parity with long-text — same patch stack, same backoff, structured-CoT grammar still works on top. - **dual-turbo.yml**: 262K context preserved + DS layout + PN25 v3 + PN30. TP=2 splits state across both cards, fits the patch stack at 0.85 mem-util cleanly. Validation per compose (verify-stress.sh, 7 probes) --------------------------------------------------- | Variant | Pass count | Failure | |--------------------|------------|----------------------| | long-text | 6 / 7 | Cliff 2 architectural | | long-vision | 6 / 7 | Cliff 2 architectural | | bounded-thinking | 6 / 7 | Cliff 2 architectural | | dual-turbo (TP=2) | 6 / 7 | Cliff 2 architectural | All non-architectural probes pass — IDE-agent one-shot, multi-turn agent, LCB-coding, reasoning-heavy, 25K tool RETURN, small-rung longctx. Cliff 2 (60K+ single prompt DeltaNet GDN forward state OOM) fails on every variant including TP=2 because GDN state is per-rank not split — that's fundamental, not addressable on this config class. Per-config summaries written to: - results/v0.20-migration/long-vision-pn30.summary - results/v0.20-migration/bounded-thinking-pn30.summary - results/v0.20-migration/dual-turbo-pn30.summary Docs partial update (SINGLE_CARD.md): updated TL;DR table with new ctx ceilings and removed the Cliff 1 mech B "limitation to know" since both mechanisms (PN12 eager + PN25 v3 compile) now close it. Kept Cliff 2 as the one remaining limitation. More doc updates in follow-up commit. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/SINGLE_CARD.md | 30 +++++++------------ .../docker-compose.bounded-thinking.yml | 29 +++++++++--------- .../compose/docker-compose.dual-turbo.yml | 20 ++++++------- .../compose/docker-compose.long-vision.yml | 30 +++++++++++-------- .../bounded-thinking-pn30.summary | 22 ++++++++++++++ .../v0.20-migration/dual-turbo-pn30.summary | 28 +++++++++++++++++ .../v0.20-migration/long-vision-pn30.summary | 23 ++++++++++++++ 7 files changed, 126 insertions(+), 56 deletions(-) create mode 100644 results/v0.20-migration/bounded-thinking-pn30.summary create mode 100644 results/v0.20-migration/dual-turbo-pn30.summary create mode 100644 results/v0.20-migration/long-vision-pn30.summary diff --git a/docs/SINGLE_CARD.md b/docs/SINGLE_CARD.md index 2117736a..0f898012 100644 --- a/docs/SINGLE_CARD.md +++ b/docs/SINGLE_CARD.md @@ -10,36 +10,26 @@ Four recommended options: | What you're doing | Compose | Max ctx | Narr / Code TPS | VRAM (24 GB / card) | |---|---|---|---|---| -| **Long ctx + vision** (chat, agents, image input) | [`long-vision.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.long-vision.yml) | **198K** | 50 / 66 | ~22.3 GB (mem-util 0.98) | -| **Long ctx, text-only** (RAG, codebase, books) | [`long-text.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.long-text.yml) | **214K** | 50 / 67 | ~23.4 GB (mem-util 0.985) | -| **Bounded thinking** (coding agents, structured-CoT, cost-bounded thinking) — see [STRUCTURED_COT.md](STRUCTURED_COT.md) | [`bounded-thinking.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml) | **214K** | 50 / 66 | ~21.7 GB (mem-util 0.985) | +| **Long ctx + vision** (chat, agents, image input) | [`long-vision.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.long-vision.yml) | **145K** | 50 / 66 | ~23.0 GB (mem-util 0.95) | +| **Long ctx, text-only** (RAG, codebase, books, **IDE agents** ⭐) | [`long-text.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.long-text.yml) | **180K** | 50 / 67 | ~22.4 GB (mem-util 0.95) | +| **Bounded thinking** (coding agents, structured-CoT, cost-bounded thinking) — see [STRUCTURED_COT.md](STRUCTURED_COT.md) | [`bounded-thinking.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml) | **180K** | 50 / 66 | ~21.7 GB (mem-util 0.95) | | **Bulletproof, no cliffs** (production service, unpredictable inputs) | [`llamacpp/default`](../models/qwen3.6-27b/llama-cpp/compose/docker-compose.yml) | **262K** | 21 / 21 | ~20 GB | Run via `bash scripts/launch.sh` (interactive) or `bash scripts/switch.sh `. -> ## ⚠️ Two limitations to know +> ## ⚠️ One limitation to know > -> ### 1. Cliff 1 mech B — IDE-agent prompts crash on long-text / long-vision / bounded-thinking / dual-turbo -> -> **If you're using Cline, OpenCode, Roo, Claude Code, Cursor, or any tool-using agent, default to [`tools-text.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.tools-text.yml) (75K + fp8 KV) — not the 198K/214K/262K variants.** -> -> Reproduced 2026-05-01 PM: a 5,900-char system prompt + 10 typical tool schemas + 346-char user request crashes `long-text.yml` (and the other 3 TQ3 composes) with a 98 MiB OOM at `empty_strided_cuda((s, 17408), ...)` in `inductor_cache/...py:1208`. Same site as VolandBerlioz's Reddit reproducer. Genesis PN12 patches the eager `SiluAndMul.forward_cuda` but vLLM's torch.compile inductor inlines `forward_native`, bypassing the pool. PN25 (the proper compile-path fix) is on Genesis dev but [blocked by a worker-fork registration bug](https://github.com/Sandermage/genesis-vllm-patches/issues/16) we filed. -> -> Until PN25 lands default-on: -> - **IDE agents → `tools-text.yml`** (PN8 closes Cliff 1 mech B via a different mechanism that *does* reach the compile path) -> - **Pure long-form text / RAG / book Q&A → `long-text.yml`** (no tool schemas in prompt → doesn't trigger inductor compile path) -> - **Vision + chat → `long-vision.yml`** (same as long-text — fine until you add tool schemas) -> - **Big single prompts → `llamacpp/default`** (262K, different engine entirely — no inductor) -> -> Tracking: [club-3090 #16](https://github.com/noonghunna/club-3090/issues/16). -> -> ### 2. Cliff 2 — DeltaNet GDN single-prompt OOM at 50–60K tokens +> ### Cliff 2 — DeltaNet GDN single-prompt OOM at 50–60K tokens > > **vLLM single-card variants will crash if you send a single prompt above ~50K tokens.** > -> Architectural — DeltaNet GDN forward state grows with sequence length, OOMs at 50–60K regardless of how much VRAM you have left. Both `long-vision.yml` (198K) and `long-text.yml` (214K) are designed for **steady-state accumulation across many turns**, not single-shot big prompts. +> Architectural — DeltaNet GDN forward state grows with sequence length, OOMs at 50–60K regardless of how much VRAM you have left. The long-* variants are designed for **steady-state accumulation across many turns**, not single-shot big prompts. > > If your workload sends single big prompts: `llamacpp/default` (262K, no cliffs anywhere) or `dual-turbo.yml` (TP=2 splits state across cards). +> +> ### What was Cliff 1 mech B (now closed) ✅ +> +> Earlier in 2026-05 we tracked an inductor compile-path FFN intermediate buffer leak ([club-3090 #16](https://github.com/noonghunna/club-3090/issues/16)) that crashed long-* variants on real IDE-agent prompts. **Closed 2026-05-02** via Genesis PN25 (Inductor-safe `silu_and_mul` opaque op) + PN30 (DS conv state dst-shaped temp fix). Both fixes ship by default in our composes; ChatGPT/Codex CLI cross-check helped land the PN30 dst-shaped temp variant. No user action needed — a fresh `bash scripts/setup.sh qwen3.6-27b` picks up the fixes automatically. --- diff --git a/models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml b/models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml index 385bd484..367ce1cb 100644 --- a/models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml +++ b/models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml @@ -167,6 +167,10 @@ services: - GENESIS_PN26_SPARSE_V_THRESHOLD=0.01 - GENESIS_ENABLE_P38B_COMPILE_SAFE=1 - GENESIS_ENABLE_P15B_FA_VARLEN_CLAMP=1 + - GENESIS_ENABLE_PN25_SILU_INDUCTOR_SAFE=1 + # PN30 — RE-ENABLED with our local dst-shaped temp fix + # (patch_pn30_dst_shaped_temp_fix.py, applied during setup.sh). + - GENESIS_ENABLE_PN30_DS_LAYOUT_SPEC_DECODE=1 - GENESIS_PREALLOC_TOKEN_BUDGET=4128 - GENESIS_BUFFER_MODE=shared # Explicit OFFs to match Sandermage's PROD env-var set verbatim: @@ -177,16 +181,12 @@ services: - GENESIS_ENABLE_P81_FP8_BLOCK_SCALED_M_LE_8=0 - GENESIS_ENABLE_P82=0 - GENESIS_P82_THRESHOLD_SINGLE=0.3 - # VLLM_SSM_CONV_STATE_LAYOUT — disabled 2026-05-01 PM after - # ChatGPT/Codex CLI diagnosed PN30 (Sander a9977d8) layout-corruption bug: - # PN30's .contiguous() materializes src[block, :, offset:] as compact - # 10240×5 memory, then raw-memcpys it into dst whose rows are 10240×6. - # Row 1+ start at the wrong destination offset → corrupts DS conv state. - # The TQ store CUDA assert we saw on probe 4 was the eventual surfacing, - # not the root cause. Until upstream fix lands, drop DS layout (-6% TPS - # per Sander's bench) for correctness on probes 4 + 5. Reported back - # on Sandermage/genesis-vllm-patches#17 with the row-stride diagnosis. - # - VLLM_SSM_CONV_STATE_LAYOUT=DS + # VLLM_SSM_CONV_STATE_LAYOUT=DS — RE-ENABLED with our local PN30 + # dst-shaped temp fix (patch_pn30_dst_shaped_temp_fix.py, applied at + # setup time). Without our fix, Sander's PN30 a9977d8 corrupts DS row + # strides on spec-decode AL>1. With our fix, PN30 builds destination- + # shaped temp via collect_mamba_copy_meta. +6% TPS retained. + - VLLM_SSM_CONV_STATE_LAYOUT=DS - VLLM_USE_FUSED_MOE_GROUPED_TOPK=1 shm_size: "16gb" ipc: host @@ -228,12 +228,13 @@ services: - float16 - --tensor-parallel-size - "1" - # 214K + 0.985 — parity with long-text.yml. See long-text.yml for - # the full v0.20 + Genesis v7.65 migration rationale. + # 180K + 0.95 — parity with long-text.yml. Backed off from 214K + 0.985 + # on 2026-05-02 to give activation headroom for the PN12+PN25 FFN pool + # residence + DeltaNet GDN buffer. See long-text.yml for full rationale. - --max-model-len - - "214000" + - "180000" - --gpu-memory-utilization - - "0.985" + - "0.95" - --max-num-seqs - "1" - --max-num-batched-tokens diff --git a/models/qwen3.6-27b/vllm/compose/docker-compose.dual-turbo.yml b/models/qwen3.6-27b/vllm/compose/docker-compose.dual-turbo.yml index 73257f0d..6aeacffa 100644 --- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual-turbo.yml +++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual-turbo.yml @@ -71,16 +71,12 @@ services: - TRITON_CACHE_DIR=/root/.triton/cache - VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 - VLLM_FLOAT32_MATMUL_PRECISION=high - # VLLM_SSM_CONV_STATE_LAYOUT — disabled 2026-05-01 PM after - # ChatGPT/Codex CLI diagnosed PN30 (Sander a9977d8) layout-corruption bug: - # PN30's .contiguous() materializes src[block, :, offset:] as compact - # 10240×5 memory, then raw-memcpys it into dst whose rows are 10240×6. - # Row 1+ start at the wrong destination offset → corrupts DS conv state. - # The TQ store CUDA assert we saw on probe 4 was the eventual surfacing, - # not the root cause. Until upstream fix lands, drop DS layout (-6% TPS - # per Sander's bench) for correctness on probes 4 + 5. Reported back - # on Sandermage/genesis-vllm-patches#17 with the row-stride diagnosis. - # - VLLM_SSM_CONV_STATE_LAYOUT=DS + # VLLM_SSM_CONV_STATE_LAYOUT=DS — RE-ENABLED with our local PN30 + # dst-shaped temp fix (patch_pn30_dst_shaped_temp_fix.py, applied at + # setup time). Without our fix, Sander's PN30 a9977d8 corrupts DS row + # strides on spec-decode AL>1. With our fix, PN30 builds destination- + # shaped temp via collect_mamba_copy_meta. +6% TPS retained. + - VLLM_SSM_CONV_STATE_LAYOUT=DS - VLLM_USE_FUSED_MOE_GROUPED_TOPK=1 - CUDA_DEVICE_MAX_CONNECTIONS=8 # FULL Genesis v7.65 PROD env-var set per Sandermage's @@ -129,6 +125,10 @@ services: - GENESIS_PN26_SPARSE_V_THRESHOLD=0.01 - GENESIS_ENABLE_P38B_COMPILE_SAFE=1 - GENESIS_ENABLE_P15B_FA_VARLEN_CLAMP=1 + - GENESIS_ENABLE_PN25_SILU_INDUCTOR_SAFE=1 + # PN30 — RE-ENABLED with our local dst-shaped temp fix + # (patch_pn30_dst_shaped_temp_fix.py, applied during setup.sh). + - GENESIS_ENABLE_PN30_DS_LAYOUT_SPEC_DECODE=1 - GENESIS_PREALLOC_TOKEN_BUDGET=4128 - GENESIS_BUFFER_MODE=shared # Explicit OFFs to match Sandermage's PROD env-var set verbatim: diff --git a/models/qwen3.6-27b/vllm/compose/docker-compose.long-vision.yml b/models/qwen3.6-27b/vllm/compose/docker-compose.long-vision.yml index 160c047a..112776ff 100644 --- a/models/qwen3.6-27b/vllm/compose/docker-compose.long-vision.yml +++ b/models/qwen3.6-27b/vllm/compose/docker-compose.long-vision.yml @@ -153,6 +153,10 @@ services: - GENESIS_PN26_SPARSE_V_THRESHOLD=0.01 - GENESIS_ENABLE_P38B_COMPILE_SAFE=1 - GENESIS_ENABLE_P15B_FA_VARLEN_CLAMP=1 + - GENESIS_ENABLE_PN25_SILU_INDUCTOR_SAFE=1 + # PN30 — RE-ENABLED with our local dst-shaped temp fix + # (patch_pn30_dst_shaped_temp_fix.py, applied during setup.sh). + - GENESIS_ENABLE_PN30_DS_LAYOUT_SPEC_DECODE=1 - GENESIS_PREALLOC_TOKEN_BUDGET=4128 - GENESIS_BUFFER_MODE=shared # Explicit OFFs to match Sandermage's PROD env-var set verbatim: @@ -163,16 +167,12 @@ services: - GENESIS_ENABLE_P81_FP8_BLOCK_SCALED_M_LE_8=0 - GENESIS_ENABLE_P82=0 - GENESIS_P82_THRESHOLD_SINGLE=0.3 - # VLLM_SSM_CONV_STATE_LAYOUT — disabled 2026-05-01 PM after - # ChatGPT/Codex CLI diagnosed PN30 (Sander a9977d8) layout-corruption bug: - # PN30's .contiguous() materializes src[block, :, offset:] as compact - # 10240×5 memory, then raw-memcpys it into dst whose rows are 10240×6. - # Row 1+ start at the wrong destination offset → corrupts DS conv state. - # The TQ store CUDA assert we saw on probe 4 was the eventual surfacing, - # not the root cause. Until upstream fix lands, drop DS layout (-6% TPS - # per Sander's bench) for correctness on probes 4 + 5. Reported back - # on Sandermage/genesis-vllm-patches#17 with the row-stride diagnosis. - # - VLLM_SSM_CONV_STATE_LAYOUT=DS + # VLLM_SSM_CONV_STATE_LAYOUT=DS — RE-ENABLED with our local PN30 + # dst-shaped temp fix (patch_pn30_dst_shaped_temp_fix.py, applied at + # setup time). Without our fix, Sander's PN30 a9977d8 corrupts DS row + # strides on spec-decode AL>1. With our fix, PN30 builds destination- + # shaped temp via collect_mamba_copy_meta. +6% TPS retained. + - VLLM_SSM_CONV_STATE_LAYOUT=DS - VLLM_USE_FUSED_MOE_GROUPED_TOPK=1 shm_size: "16gb" ipc: host @@ -218,10 +218,16 @@ services: # closes the Cliff 1 mech B sub-cliffs that drove the 140K + 0.95 # backoff on dev205. Verified verify-full + 33K + 50K stress all # PASS. See docs/CLIFFS.md "v0.20 unblock" section. + # 145K + 0.95 — backed off from 198K + 0.98 on 2026-05-02 to give + # activation headroom for the PN12+PN25 FFN intermediate pool residence + # + PN30 dst-shaped temp lifecycle. Vision tower's persistent ~1 GB + # tightens the budget further than long-text (which sits at 180K + 0.95). + # Engine pre-check at 175K + 0.95 returned `estimated maximum model length + # is 148608` — falls back to 145K for safety margin. - --max-model-len - - "198000" + - "145000" - --gpu-memory-utilization - - "0.98" + - "0.95" - --max-num-seqs - "1" - --max-num-batched-tokens diff --git a/results/v0.20-migration/bounded-thinking-pn30.summary b/results/v0.20-migration/bounded-thinking-pn30.summary new file mode 100644 index 00000000..9ba8c2bc --- /dev/null +++ b/results/v0.20-migration/bounded-thinking-pn30.summary @@ -0,0 +1,22 @@ +variant: bounded-thinking +date: 2026-05-02 +context: 180000 +gpu_mem_util: 0.95 +kv_dtype: turboquant_3bit_nc +tp: 1 +genesis_pin: 753344b +patches: PN25 v3 + PN30 dst-shaped temp fix + DS layout +structured_outputs_enable_in_reasoning: true +verify_stress_results: | + [1/7] longctx small: PASS + [2/7] 25K tool RETURN: PASS + [3/7] IDE-agent one-shot: PASS + [4/7] multi-turn agent: PASS + [5/7] LCB-coding: PASS + [6/7] reasoning 8192: PASS + [7/7] longctx large 60K+: FAIL Cliff 2 +score: 6/7 +notes: | + Parity with long-text — same patch stack + same backoff (180K + 0.95) + matches long-text behavior. Structured-CoT grammar enforcement still works + on top of the new patches. diff --git a/results/v0.20-migration/dual-turbo-pn30.summary b/results/v0.20-migration/dual-turbo-pn30.summary new file mode 100644 index 00000000..db7bef82 --- /dev/null +++ b/results/v0.20-migration/dual-turbo-pn30.summary @@ -0,0 +1,28 @@ +variant: dual-turbo (TP=2) +date: 2026-05-02 +context: 262144 +gpu_mem_util: 0.85 +kv_dtype: turboquant_3bit_nc +tp: 2 +genesis_pin: 753344b (v7.65 dev tip) +patches: PN25 v3 (local) + PN30 dst-shaped temp fix (local) + DS layout enabled +verify_stress_results: | + [1/7] longctx small (10K + 30K): PASS + [2/7] 25K tool RETURN: PASS + [3/7] IDE-agent one-shot: PASS (66 tokens, finish=stop) + [4/7] multi-turn agent: PASS + [5/7] LCB-coding: PASS + [6/7] reasoning 8192: PASS (8192 tokens, finish=length) + [7/7] longctx large (60K+): FAIL (Cliff 2 architectural — DeltaNet GDN forward state OOM, even on TP=2 single-prompt) +score: 6/7 +notes: | + TP=2 splits state but doesn't avoid Cliff 2 on single-prompt 60K+ since + the GDN state buffer is per-rank and grows with seq_len, not split across + ranks. Cliff 2 is not addressable on this config class regardless of + TP factor. Rule for users: dual-turbo for 4-stream concurrent serving at + ≤50K each, OR llamacpp/default 262K for single-prompt big inputs. + PN30 dst-shaped temp fix passes cleanly on TP=2 — multi-turn agent + + LCB-coding shapes both work that previously crashed with Sander's + upstream PN30 a9977d8. +kv_pool_tokens: 1523232 +max_concurrency: 4.67 diff --git a/results/v0.20-migration/long-vision-pn30.summary b/results/v0.20-migration/long-vision-pn30.summary new file mode 100644 index 00000000..d018b6e9 --- /dev/null +++ b/results/v0.20-migration/long-vision-pn30.summary @@ -0,0 +1,23 @@ +variant: long-vision +date: 2026-05-02 +context: 145000 +gpu_mem_util: 0.95 +kv_dtype: turboquant_3bit_nc +tp: 1 +vision: true +genesis_pin: 753344b +patches: PN25 v3 (local) + PN30 dst-shaped temp fix (local) + DS layout +verify_stress_results: | + [1/7] longctx small (10K + 30K): PASS + [2/7] 25K tool RETURN: PASS + [3/7] IDE-agent one-shot: PASS (66 tokens) + [4/7] multi-turn agent: PASS + [5/7] LCB-coding: PASS + [6/7] reasoning 8192: PASS + [7/7] longctx large (60K+): FAIL Cliff 2 architectural +score: 6/7 +backoff_from_198K: | + Vision tower's persistent ~1 GB plus PN12+PN25 pool residence + PN30 + dst-shaped temp lifecycle tightens activation budget further than + long-text. Engine pre-check at 175K + 0.95 returned `estimated maximum + model length is 148608` → fall back to 145K + 0.95 for safety margin.