PN25 v3: close Cliff 1 mech B (club-3090#16) on long-text via setup-time Genesis backport

Background
----------

Cliff 1 mech B is the inductor-compiled FFN intermediate buffer leak that
PN12 (eager-mode SiluAndMul.forward_cuda pool) doesn't reach. vLLM v0.20
with `compilation_config.custom_ops=["none"]` dispatches SiluAndMul through
forward_native, which Inductor inlines and lowers to raw `empty_strided_cuda(
(s, intermediate_size), ...)` — bypassing PN12's FFNIntermediateCache pool.

Result: real IDE-agent prompts (sys-prompt + tool schemas + user request)
crashed long-text/long-vision/bounded-thinking with 138 MiB FFN OOM at the
inductor cache site. VolandBerlioz's Reddit reproducer + our local synthetic
both confirmed.

Sander shipped Genesis PN25 to address this — registers `silu_and_mul` as a
`torch.library.custom_op` so Inductor treats it as opaque (can't inline).
But PN25 hit a worker-fork registration bug under spawn: `_register_op_once()`
called from inside dynamo trace → `@custom_op` decorator → `infer_schema()`
→ dynamo refuses to trace.

Filed Sander/genesis-vllm-patches#16 with the trace + analysis.

What this commit adds
---------------------

A setup-time backport that lets us enable PN25 NOW, before Sander's upstream
fix lands in our pinned version (or in case it doesn't transfer to TP=1).
Two parts in `patch_pn25_genesis_register_fix.py`:

1. **Genesis-side change** to `silu_and_mul_customop.py`: hardened
   `get_op_callable()` to return None if called during dynamo tracing
   (defensive — shouldn't happen if part 2 works).

2. **Wiring change** to `patch_N25_silu_inductor_safe_pool.py`: text-patches
   `vllm/model_executor/layers/activation.py` to import the customop module
   and cache the op as a module-level global at activation.py import time.
   The patched `forward_native` body just reads `_GENESIS_PN25_SILU_AND_MUL_OP`
   — no import + no registration during the dynamo trace.

Worker module-import happens during model construction in vLLM, BEFORE
profile_run enters aot_compile_fullgraph. Registration runs in eager
Python at startup; subsequent forward calls just read the cached global.

Wired into setup.sh after Genesis checkout. Idempotent via marker.

Also in this commit
-------------------

- **long-text.yml backed off 214K + 0.985 → 180K + 0.95.** PN25's pool keeps
  the FFN buffer resident (~140 MiB persistent), which tightens activation
  budget at OTHER peaks (DeltaNet `chunk_fwd_o`). 0.985 left only 26 MiB
  free at 30K probe — OOM. 0.95 frees ~480 MiB for activation comfort.

  Net memory accounting: PN25 is a strict win on KV pool because vLLM's
  profile_run measures lower activation peak (no fresh FFN alloc), so KV
  pool grows. Max concurrency at 180K: 1.07x without PN25 → 1.49x with
  PN25 + 0.95 (or 1.64x at 0.97 if we'd held it).

- **verify-stress.sh probe 3 hardened.** Was using tool_choice="auto" which
  let the model emit a tool_call and exit before the long-reasoning path
  that triggers the bug. Now uses tool_choice="none" + temperature=0 +
  asserts completion_tokens >= 200 to ensure the inductor compile path
  actually exercises during the test.

Validation on long-text 180K + 0.95 + PN25 v3
---------------------------------------------

| Probe                           | Result | Notes                            |
|---------------------------------|--------|----------------------------------|
| 1.1 Long-ctx needle 9.8K        | ✅ PASS | activation budget safe           |
| 1.2 Long-ctx needle 29K         | ✅ PASS | activation budget safe at 30K    |
| 1.3 Long-ctx needle 60K         | ❌ FAIL | Cliff 2 architectural (DeltaNet) |
| 2  25K tool RETURN              | ❌ FAIL | FA varlen workspace (Sander #15) |
| 3  IDE-agent one-shot           | ✅ PASS | **Closed by PN25 v3** ⭐         |
| 4  Multi-turn agent             | ✅ PASS | **Closed by PN25 v3** ⭐         |
| 5  LCB-coding                   | ❌ FAIL | DS conv state (Sander #17)       |
| 6  Reasoning-heavy 8192         | ✅ PASS | pure reasoning works clean       |

The 4 failures are pre-tracked separately:
- Cliff 2 (#1.3): architectural, no fix at single-card; route to dual or llama.cpp
- 25K tool RETURN (#2): Sander shipped PN31 (`753344b`) — pending our cross-rig
- LCB-coding (#5): Sander shipped PN30 (`a9977d8`) — pending our cross-rig

Sander has since shipped PN25 fix upstream (`d92bcb3` on dev) using a
slightly different approach (hasattr check on global registry). Once our
pin bumps to that commit, this local v3 patch becomes redundant — drop it
and remove the setup.sh hook in a follow-up.

Other 3 TQ3 composes (long-vision/bounded-thinking/dual-turbo) are NOT
PN25-enabled in this commit pending validation. Long-text is the only
compose with PN25 active + verified.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Codex CLI (ChatGPT) <noreply@openai.com>

models/qwen3.6-27b/vllm/compose/docker-compose.long-text.yml

@@ -160,6 +160,7 @@ services:
       - GENESIS_ENABLE_PN13_CUDA_GRAPH_LAMBDA_ARITY=1
       - GENESIS_ENABLE_PN14_TQ_DECODE_OOB_CLAMP=1
       - GENESIS_ENABLE_PN17_FA2_LSE_CLAMP=1
+      - GENESIS_ENABLE_PN25_SILU_INDUCTOR_SAFE=1
       - GENESIS_ENABLE_PN19_SCOPED_MAX_SPLIT=1
       - GENESIS_ENABLE_PN22_LOCAL_ARGMAX_TP=1
       # PN26b sparse-V Triton kernel — first SM86 (Ampere consumer) sparse-V
@@ -229,18 +230,36 @@ services:
       - float16
       - --tensor-parallel-size
       - "1"
-      # 214K + 0.985 — restored from intermediate 185K backoff after v0.20
+      # 180K + 0.97 — backed off from 214K + 0.985 on 2026-05-01 PM after
-      # migration (2026-05-01 PM). Engine ceiling on v0.20 + cudagraph
+      # verify-stress probe 1 (10K-token long-context needle) crashed with
-      # profiling-OFF is 214K (vs 218K on dev205); -4K is the cumulative
+      # GDN forward OOM (`fla.ops.chunk.chunk_gated_delta_rule_fwd_h`
-      # overhead of v0.20's PyTorch 2.11 + CUDAGraph profiling default-on.
+      # `k.new_empty(B, NT, H, V, K)` requested 98 MiB, 46 MiB free). At
-      # 50K-token cliff that fired on dev205 doesn't reproduce on v0.20 +
+      # 0.985 mem-util, activation budget was ~46 MiB at 10K prefill —
-      # Genesis v7.65 — verified verify-full + 33K + 50K stress all PASS.
+      # tight enough that ANY single prompt over a few thousand tokens was
-      # Cliff 2 (single-prompt ≥50–60K in GDN) is independent and still
+      # a coin flip. The "Cliff 2 fires only at 50-60K" framing was
-      # applies — route those to TP=2 dual.yml or llama.cpp.
+      # config-specific to the dev205 + 0.92 default; at tighter mem-util
+      # the GDN forward buffer outpaces available activation much earlier.
+      #
+      # New target (validated 2026-05-01 PM via verify-stress.sh):
+      # 180K + 0.97 → ~6.4 GB after weights, ~5.9 GB KV pool, ~500 MiB
+      # activation safety. Probe 1 (10K needle) and probe 6 (math reasoning
+      # at max_tokens=8192) PASS. Safe single-prompt envelope ~30K; full
+      # 180K context for steady-state turn accumulation only.
+      #
+      # IDE-agent workloads (sys + tool schemas) STILL crash here per
+      # club-3090#16 — different bug (Cliff 1 mech B inductor leak), not
+      # fixed by mem-util tuning. Use tools-text.yml for those.
       - --max-model-len
-      - "214000"
+      - "180000"
+      # 0.95 — backed off from 0.97 on 2026-05-01 PM after PN25 v3 closed
+      # Cliff 1 mech B at the FFN intermediate buffer (which now sits in
+      # PN12+PN25's resident FFNIntermediateCache pool). Pool residence
+      # adds ~140 MiB persistent VRAM, which tightened activation budget
+      # at DeltaNet `chunk_fwd_o` (`o = torch.empty_like(v)`, 50 MiB) —
+      # 0.97 left only 26 MiB free, OOM at 9.8K probe. 0.95 frees ~480 MiB
+      # activation budget, comfortable for 30K+ prompts.
       - --gpu-memory-utilization
-      - "0.985"
+      - "0.95"
       - --max-num-seqs
       - "1"
       - --max-num-batched-tokens

models/qwen3.6-27b/vllm/patches/patch_pn25_genesis_register_fix.py

@@ -0,0 +1,286 @@
+"""Genesis PN25 worker-spawn registration fix (setup-time patch).
+
+Filed upstream as
+[Sandermage/genesis-vllm-patches#16](https://github.com/Sandermage/genesis-vllm-patches/issues/16).
+This is our local backport while the upstream PR cycle plays out.
+
+Why setup-time, not boot-time
+-----------------------------
+
+The genesis tree is mounted read-only into the vLLM container; a boot-time
+sidecar can't write to it. The patch needs to land in the genesis checkout
+BEFORE the container starts, which means at setup time. setup.sh invokes
+this script after `git checkout <pin>` so the fix lands on every fresh
+setup.
+
+The bug
+-------
+
+`_register_op_once()` checks an in-process `_op_registered` flag. When
+vLLM workers spawn (vLLM uses spawn, not fork), they get a fresh Python
+interpreter — module-level state resets, including `_op_registered=False`.
+The C++ `torch.library` registry also doesn't survive across spawn.
+
+Worker's first call to the patched `SiluAndMul.forward_native` body calls
+`get_op_callable()` which calls `_register_op_once()`. The flag is False,
+so it tries to re-decorate via `@torch.library.custom_op(...)`. That call
+internally invokes `torch.library.infer_schema(fn)` to derive the op
+schema from type hints. **Dynamo refuses to trace through `infer_schema`**
+(it's marked as skipped). And worker's first `forward_native` call IS
+inside a dynamo `aot_compile_fullgraph` trace during profile_run. Engine
+dies at:
+
+    torch._dynamo.exc.Unsupported: Attempted to call function marked as skipped
+      Explanation: Dynamo developers have intentionally marked that the
+      function `infer_schema` ... should not be traced.
+
+The fix
+-------
+
+Do not register from the compiled forward path at all. In PyTorch 2.11,
+calling a `torch._dynamo.disable` wrapped function from vLLM's
+`fullgraph=True` compile path is itself an Unsupported graph break.
+
+Instead, patch Genesis PN25 so `activation.py` registers and caches the
+opaque op at module import time in each spawned worker:
+
+    _GENESIS_PN25_SILU_AND_MUL_OP = get_op_callable()
+
+Worker module import happens during model construction, before
+`profile_run` enters `aot_compile_fullgraph`. The patched
+`forward_native` then calls only that cached module global and never runs
+`torch.library.custom_op(...)` while Dynamo is tracing.
+"""
+import os
+import sys
+
+CUSTOMOP_TARGET = (
+    "models/qwen3.6-27b/vllm/patches/genesis/vllm/_genesis/kernels/"
+    "silu_and_mul_customop.py"
+)
+WIRING_TARGET = (
+    "models/qwen3.6-27b/vllm/patches/genesis/vllm/_genesis/wiring/hybrid/"
+    "patch_N25_silu_inductor_safe_pool.py"
+)
+
+MARKER_CUSTOMOP = "# club-3090: PN25 trace-safe get_op_callable v3"
+MARKER_WIRING = "club-3090: PN25 import-time op cache v3"
+
+OLD_GET_OP_CALLABLE = '''def get_op_callable():
+    """Return the registered op callable, or None if registration failed.
+
+    Used by the PN25 wiring patch to populate the replacement body.
+    Caller is responsible for graceful degradation on None.
+    """
+    if not _register_op_once():
+        return None
+    try:
+        return torch.ops.genesis.silu_and_mul_pooled
+    except (AttributeError, RuntimeError):
+        return None
+'''
+
+NEW_GET_OP_CALLABLE = f'''def get_op_callable():
+    """Return the registered op callable, or None if registration failed.
+
+    {MARKER_CUSTOMOP}
+
+    Intended path: activation.py imports this module and calls this function
+    while each spawned worker imports activation.py, before Dynamo tracing.
+    If a stale forward body calls this from inside fullgraph tracing, avoid
+    torch.library registration there because Dynamo cannot trace Library
+    construction or infer_schema.
+    """
+    if not _op_registered:
+        try:
+            compiler = getattr(torch, "compiler", None)
+            if compiler is not None and compiler.is_compiling():
+                log.debug(
+                    "[PN25] get_op_callable called during torch.compile "
+                    "before registration; returning None"
+                )
+                return None
+        except Exception:
+            pass
+        try:
+            dynamo = getattr(torch, "_dynamo", None)
+            if dynamo is not None and dynamo.is_compiling():
+                log.debug(
+                    "[PN25] get_op_callable called during Dynamo tracing "
+                    "before registration; returning None"
+                )
+                return None
+        except Exception:
+            pass
+
+    if not _register_op_once():
+        return None
+    try:
+        return torch.ops.genesis.silu_and_mul_pooled
+    except (AttributeError, RuntimeError):
+        return None
+'''
+
+OLD_FORWARD_REPLACEMENT = '''PN25_FORWARD_NATIVE_REPLACEMENT = (
+    "    @staticmethod\\n"
+    "    def forward_native(x: torch.Tensor) -> torch.Tensor:\\n"
+    "        \\"\\"\\"PyTorch-native — Genesis PN25 routes through opaque\\n"
+    "        custom op so torch.compile/Inductor cannot inline the FFN\\n"
+    "        intermediate alloc; pool from FFNIntermediateCache instead.\\n"
+    "        Falls back to vanilla math when registration unavailable.\\n"
+    "        \\"\\"\\"\\n"
+    "        try:\\n"
+    "            from vllm._genesis.kernels.silu_and_mul_customop import (\\n"
+    "                get_op_callable as _genesis_pn25_get_op,\\n"
+    "            )\\n"
+    "            _genesis_pn25_op = _genesis_pn25_get_op()\\n"
+    "            if _genesis_pn25_op is not None:\\n"
+    "                return _genesis_pn25_op(x)\\n"
+    "        except Exception:  # pragma: no cover — defensive fallback\\n"
+    "            pass\\n"
+    "        d = x.shape[-1] // 2\\n"
+    "        return F.silu(x[..., :d]) * x[..., d:]\\n"
+)
+'''
+
+NEW_FORWARD_REPLACEMENT = '''PN25_FORWARD_NATIVE_REPLACEMENT = (
+    "    @staticmethod\\n"
+    "    def forward_native(x: torch.Tensor) -> torch.Tensor:\\n"
+    "        \\"\\"\\"PyTorch-native — Genesis PN25 routes through an opaque\\n"
+    "        custom op cached at activation.py import time, before vLLM's\\n"
+    "        aot_compile_fullgraph trace. Do not register custom ops here.\\n"
+    "        \\"\\"\\"\\n"
+    "        _genesis_pn25_op = _GENESIS_PN25_SILU_AND_MUL_OP\\n"
+    "        if _genesis_pn25_op is not None:\\n"
+    "            return _genesis_pn25_op(x)\\n"
+    "        d = x.shape[-1] // 2\\n"
+    "        return F.silu(x[..., :d]) * x[..., d:]\\n"
+)
+'''
+
+IMPORT_CONSTANTS = f'''PN25_IMPORT_ANCHOR = (
+    "logger = init_logger(__name__)\\n"
+    "\\n"
+    "\\n"
+)
+
+PN25_IMPORT_REPLACEMENT = (
+    "logger = init_logger(__name__)\\n"
+    "\\n"
+    "# [Genesis PN25] {MARKER_WIRING}. Register/cache the opaque\\n"
+    "# silu_and_mul op while activation.py is imported in each spawned\\n"
+    "# worker, before vLLM enters aot_compile_fullgraph profile_run.\\n"
+    "try:\\n"
+    "    from vllm._genesis.kernels.silu_and_mul_customop import (\\n"
+    "        get_op_callable as _genesis_pn25_get_op_callable,\\n"
+    "    )\\n"
+    "    _GENESIS_PN25_SILU_AND_MUL_OP = _genesis_pn25_get_op_callable()\\n"
+    "except Exception:\\n"
+    "    _GENESIS_PN25_SILU_AND_MUL_OP = None\\n"
+    "\\n"
+    "\\n"
+)
+
+'''
+
+SUBPATCH_ANCHOR = '''        sub_patches=[
+            TextPatch(
+                name="pN25_silu_and_mul_forward_native_opaque",
+'''
+
+SUBPATCH_REPLACEMENT = '''        sub_patches=[
+            TextPatch(
+                name="pN25_silu_and_mul_import_time_register",
+                anchor=PN25_IMPORT_ANCHOR,
+                replacement=PN25_IMPORT_REPLACEMENT,
+                required=True,
+            ),
+            TextPatch(
+                name="pN25_silu_and_mul_forward_native_opaque",
+'''
+
+
+def _read(path: str) -> str | None:
+    if not os.path.isfile(path):
+        print(f"[pn25_register_fix] target not found: {path}", file=sys.stderr)
+        return None
+    with open(path, "r") as f:
+        return f.read()
+
+
+def _write(path: str, src: str) -> None:
+    with open(path, "w") as f:
+        f.write(src)
+
+
+def _patch_customop() -> bool:
+    src = _read(CUSTOMOP_TARGET)
+    if src is None:
+        return False
+    if MARKER_CUSTOMOP in src:
+        print("[pn25_register_fix v3] customop guard already applied")
+        return True
+    if OLD_GET_OP_CALLABLE not in src:
+        print(
+            "[pn25_register_fix v3] customop get_op_callable anchor not found",
+            file=sys.stderr,
+        )
+        return False
+    _write(CUSTOMOP_TARGET, src.replace(OLD_GET_OP_CALLABLE, NEW_GET_OP_CALLABLE, 1))
+    print("[pn25_register_fix v3] patched get_op_callable trace guard")
+    return True
+
+
+def _patch_wiring() -> bool:
+    src = _read(WIRING_TARGET)
+    if src is None:
+        return False
+
+    changed = False
+    if MARKER_WIRING not in src:
+        if OLD_FORWARD_REPLACEMENT not in src:
+            print(
+                "[pn25_register_fix v3] forward replacement anchor not found",
+                file=sys.stderr,
+            )
+            return False
+        src = src.replace(OLD_FORWARD_REPLACEMENT, NEW_FORWARD_REPLACEMENT, 1)
+        src = src.replace(
+            NEW_FORWARD_REPLACEMENT,
+            NEW_FORWARD_REPLACEMENT + "\n" + IMPORT_CONSTANTS,
+            1,
+        )
+        changed = True
+
+    if "pN25_silu_and_mul_import_time_register" not in src:
+        if SUBPATCH_ANCHOR not in src:
+            print(
+                "[pn25_register_fix v3] subpatch insertion anchor not found",
+                file=sys.stderr,
+            )
+            return False
+        src = src.replace(SUBPATCH_ANCHOR, SUBPATCH_REPLACEMENT, 1)
+        changed = True
+
+    if changed:
+        _write(WIRING_TARGET, src)
+        print("[pn25_register_fix v3] patched PN25 activation import-time cache")
+    else:
+        print("[pn25_register_fix v3] wiring already applied")
+    return True
+
+
+def main() -> int:
+    ok_customop = _patch_customop()
+    ok_wiring = _patch_wiring()
+    if not (ok_customop and ok_wiring):
+        return 1
+    print(
+        "[pn25_register_fix v3] done — PN25 registers in spawned workers "
+        "before fullgraph tracing; forward_native no longer registers in-graph"
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/setup.sh

@@ -129,6 +129,16 @@ if [[ "${SKIP_GENESIS:-0}" != "1" ]]; then
     exit 1
   fi
   echo "[genesis] Pinned to ${GENESIS_PIN} ($(cd "${GENESIS_DIR}" && git rev-parse --short HEAD))"
+
+  # PN25 worker-spawn registration fix (genesis-vllm-patches#16) — local
+  # backport while upstream PR cycle plays out. Idempotent: re-running is a
+  # no-op once the markers are present. Safe to remove after Sandermage ships
+  # the import-time custom-op registration fix.
+  if [[ -f "${ROOT_DIR}/models/qwen3.6-27b/vllm/patches/patch_pn25_genesis_register_fix.py" ]]; then
+    (cd "${ROOT_DIR}" && python3 models/qwen3.6-27b/vllm/patches/patch_pn25_genesis_register_fix.py) || {
+      echo "[genesis] WARN: PN25 register fix did not apply cleanly. PN25 may not work in workers." >&2
+    }
+  fi
 else
   echo "[genesis] SKIP_GENESIS=1 — not cloning."
 fi

scripts/verify-stress.sh

@@ -380,10 +380,15 @@ body = {
         {"role": "user", "content": user_text},
     ],
     "tools": tools,
-    "tool_choice": "auto",
+    # tool_choice="none" forces content-only output (no tool_calls),
+    # which makes the model go through long-reasoning + code emission —
+    # the path that triggers Cliff 1 mech B inductor leak. With "auto"
+    # the model can short-circuit by emitting a tool_call and exit
+    # before hitting the inductor-compiled reasoning forward, hiding
+    # the bug. We want the bug to surface deterministically.
+    "tool_choice": "none",
     "max_tokens": 2000,
-    "temperature": 0.6,
+    "temperature": 0.0,
-    "top_p": 0.95,
     "stream": False,
 }
 with open(os.environ['REQ_FILE'], 'w') as f:
@@ -396,10 +401,20 @@ PYEOF
   case "$http_code" in
     200)
       body="$(cat "${resp_file}")"
-      local finish content_len
+      local finish content_chars completion_tokens
       finish="$(echo "$body" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['choices'][0].get('finish_reason') or '?')" 2>/dev/null || echo "?")"
-      content_len="$(echo "$body" | python3 -c "import sys,json; d=json.load(sys.stdin); m=d['choices'][0].get('message') or {}; c=m.get('content') or ''; tc=m.get('tool_calls') or []; print(f'{len(c)},{len(tc)}')" 2>/dev/null || echo "0,0")"
+      content_chars="$(echo "$body" | python3 -c "import sys,json; d=json.load(sys.stdin); m=d['choices'][0].get('message') or {}; print(len(m.get('content') or ''))" 2>/dev/null || echo "0")"
-      pass "IDE-agent one-shot OK — finish=${finish} content_chars=${content_len%,*} tool_calls=${content_len##*,}"
+      completion_tokens="$(echo "$body" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('usage',{}).get('completion_tokens', 0))" 2>/dev/null || echo "0")"
+      # Hardened: tool_choice=none forces content-only generation, so the
+      # model MUST go through the long-reasoning + code-emission path.
+      # If completion_tokens is suspiciously low, the model didn't actually
+      # exercise the inductor compile path that Cliff 1 mech B fires from.
+      if [[ "$completion_tokens" -lt 200 ]]; then
+        fail "HTTP 200 but only ${completion_tokens} completion tokens (finish=${finish})" \
+             "Probe expects long-form content generation to exercise the inductor compile path. <200 tokens means the bug surface wasn't actually tested. Check finish_reason — if 'tool_calls' despite tool_choice=none, the engine ignored the constraint."
+      else
+        pass "IDE-agent one-shot OK — ${completion_tokens} completion tokens (${content_chars} chars), finish=${finish}"
+      fi
       ;;
     500)
       fail "HTTP 500 — likely Cliff 1 mech B (inductor FFN intermediate OOM)" \