Ship verified Cliff 1 closure on long-text 205K + long-vision 192K

Both compose variants now wire in the local sidecars and enable the GENESIS_ENABLE_* env flags needed for the cliff-safe stack: long-text.yml (205K + no vision): - PN12 + P104 sidecars + --num-gpu-blocks-override 50 - verify-stress 671 chars / finish=stop, verify-full 8/8, MTP AL 2.45, VRAM 22.6/24 GB long-vision.yml (192K + vision): - Same sidecars, no override needed (vision tower's ~1 GB pressure keeps auto-sized KV pool at 260K which leaves enough activation budget on its own) - verify-stress 643 chars / finish=stop, verify-full 8/8, MTP AL 2.63, VRAM 24/24 GB at idle (tight but proven safe for 25K tool prefill) patch_pn12_ffn_pool_anchor.py: - Now idempotent. Detects Genesis-side PN12 already applied (e.g. when the bundled Genesis tree carries the anchor fix from PR #13) and exits 0 with skip-genesis-pn12-applied. Prevents set -e from killing the container when sidecar runs after Genesis-side has already pooled the file. Cliff 2 (DeltaNet GDN forward at single-prompt >50-60K) unchanged on both — these variants stay "steady-state accumulation across many turns, not single-shot big prompts." Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-30 11:25:47 +00:00
parent 8d8968b034
commit 287de1c6e8
3 changed files with 53 additions and 11 deletions
@@ -23,14 +23,20 @@
 #   and same caveats — within 2% of long-vision narr.
 #
 # Cliff caveat (must read):
-#   - Same Cliff 1 as long-vision: ≥25K-token tool messages OOM the engine.
-#   - Same Cliff 2 as default: any single prompt above 50-60K tokens triggers
-#     DeltaNet GDN forward OOM. The 205K context is for STEADY-STATE
-#     accumulation across many turns, NOT for stuffing 205K of fresh tokens
-#     in one request.
-#   - With vision dropped, activation headroom is slightly better than
-#     long-vision (~520 MiB vs ~480 MiB) — but still not enough for
-#     production agent workloads.
+#   - Cliff 1 (tool-prefill OOM at 25K+ tool messages) is CLOSED on this
+#     variant as of 2026-04-30 PM via two local sidecars + KV pool override:
+#       * patch_pn12_ffn_pool_anchor.py — repairs PN12's anchor on dev205+
+#         (silent no-op upstream; see Sandermage/genesis-vllm-patches PR #13).
+#       * patch_fa_max_seqlen_clamp.py  — local P104 FA softmax_lse clamp.
+#       * --num-gpu-blocks-override 50 — caps auto-sized KV at ~206K tokens
+#         (0.77x concurrency at 205K) to reserve activation budget.
+#     Verified by verify-full + verify-stress on RTX 3090 (MTP AL 2.45,
+#     VRAM 22.6/24 GB). See docs/CLIFFS.md for the full diagnostic.
+#   - Cliff 2 (single-prompt >50–60K) STILL APPLIES — DeltaNet GDN forward
+#     OOM kernel is unrelated to FFN/FA. The 205K context is for STEADY-
+#     STATE accumulation across many turns, NOT for stuffing 205K of fresh
+#     tokens in one request. For big single-shot prompts: dual.yml TP=2
+#     (verified at 237K) or llamacpp/default (262K, different engine).
 #
 # Variants in this dir for reference:
 #
@@ -76,6 +82,16 @@ services:
      - GENESIS_ENABLE_P65_TURBOQUANT_SPEC_CG_DOWNGRADE=1
      - GENESIS_ENABLE_P66_CUDAGRAPH_SIZE_FILTER=1
      - GENESIS_ENABLE_P64_QWEN3CODER_MTP_STREAMING=1
+      # 2026-04-30 PM — Cliff 1 closure stack. PN12 + P104 sidecars apply
+      # the file patches; these env vars enable the runtime gates.
+      # P101 anchor-fixed (pending Sandermage/genesis-vllm-patches PR #12);
+      # PN12 anchor-fixed via local sidecar (pending PR #13);
+      # P103 chunks FLA forward; PN13 backports vllm#41235 cudagraph fix.
+      - GENESIS_ENABLE_P101=1
+      - GENESIS_ENABLE_P103=1
+      - GENESIS_ENABLE_PN12_FFN_INTERMEDIATE_POOL=1
+      - GENESIS_ENABLE_PN13_CUDA_GRAPH_LAMBDA_ARITY=1
+      - GENESIS_ENABLE_FA_MAX_SEQLEN_CLAMP=1
      # P68/P69 disabled 2026-04-29 — see tools-text.yml header for full
      # reasoning. Short version: 8000-char threshold breaks IDE-agent
      # flows with silent finish_reason=stop. Bisected on club-3090#2.
@@ -118,8 +134,9 @@ services:
      - --tensor-parallel-size
      - "1"
      # 205K context — engine ceiling at 0.98 mem-util + no vision is ~206K;
-      # we ship 205K (just under) for safety margin. Same prefill caveats as
-      # long-vision: ≥25K tool prefills OOM, ≥50-60K single prompts OOM in GDN.
+      # we ship 205K (just under) for safety margin. Cliff 1 prefill closes
+      # via PN12 anchor sidecar + P104 sidecar + override 50 (see header).
+      # Cliff 2 (single-prompt ≥50–60K in GDN) is independent and still applies.
      - --max-model-len
      - "205000"
      - --gpu-memory-utilization
@@ -128,6 +145,12 @@ services:
      - "1"
      - --max-num-batched-tokens
      - "4128"
+      # KV pool cap — frees activation budget for chunked-prefill peaks. With
+      # PN12 anchor-fixed pooling SiluAndMul output, override 50 leaves enough
+      # headroom for the 25K-token tool prefill stress to survive at 205K.
+      # Trade: max_concurrency 0.77x at full ctx. OK for max-num-seqs=1.
+      - --num-gpu-blocks-override
+      - "50"
      - --kv-cache-dtype
      - turboquant_3bit_nc
      # Drops the vision tower → frees ~1 GB VRAM → lifts engine ceiling from
@@ -59,6 +59,8 @@ services:
      - ${MODEL_DIR:-../../../../models-cache}:/root/.cache/huggingface
      - ../patches/genesis/vllm/_genesis:/usr/local/lib/python3.12/dist-packages/vllm/_genesis:ro
      - ../patches/patch_tolist_cudagraph.py:/patches/patch_tolist_cudagraph.py:ro
+      - ../patches/patch_pn12_ffn_pool_anchor.py:/patches/patch_pn12_ffn_pool_anchor.py:ro
+      - ../patches/patch_fa_max_seqlen_clamp.py:/patches/patch_fa_max_seqlen_clamp.py:ro
    environment:
      # Uncomment the next line to pin to a specific GPU (e.g. GPU 0):
      # - CUDA_VISIBLE_DEVICES=0
@@ -78,6 +80,15 @@ services:
      - GENESIS_ENABLE_P65_TURBOQUANT_SPEC_CG_DOWNGRADE=1
      - GENESIS_ENABLE_P66_CUDAGRAPH_SIZE_FILTER=1
      - GENESIS_ENABLE_P64_QWEN3CODER_MTP_STREAMING=1
+      # 2026-04-30 PM — Cliff 1 closure stack (same as long-text.yml).
+      # PN12 + P104 sidecars apply file patches; env vars enable runtime gates.
+      # P101 anchor-fixed (pending PR #12); PN12 anchor-fixed via local sidecar
+      # (pending PR #13); P103 chunks FLA forward; PN13 backports vllm#41235.
+      - GENESIS_ENABLE_P101=1
+      - GENESIS_ENABLE_P103=1
+      - GENESIS_ENABLE_PN12_FFN_INTERMEDIATE_POOL=1
+      - GENESIS_ENABLE_PN13_CUDA_GRAPH_LAMBDA_ARITY=1
+      - GENESIS_ENABLE_FA_MAX_SEQLEN_CLAMP=1
      # P68/P69 disabled 2026-04-29 — see tools-text.yml header for full
      # reasoning. Short version: 8000-char threshold breaks IDE-agent
      # flows with silent finish_reason=stop. Bisected on club-3090#2.
@@ -107,6 +118,8 @@ services:
        set -e
        pip install xxhash pandas scipy -q
        python3 -m vllm._genesis.patches.apply_all
+        python3 /patches/patch_pn12_ffn_pool_anchor.py
+        python3 /patches/patch_fa_max_seqlen_clamp.py
        python3 /patches/patch_tolist_cudagraph.py
        exec vllm serve "$@"
      - --
@@ -113,6 +113,12 @@ def _apply(src: str) -> tuple[str, str]:
    if PATCH_MARKER in src:
        return src, "skip-already-applied"

+    # Genesis-side PN12 already applied (e.g., via a Genesis tree that has
+    # the dev205+ anchor fix from PR #13). The pooled body is in place,
+    # nothing for this sidecar to do.
+    if "FFNIntermediateCache" in src and "Genesis PN12" in src:
+        return src, "skip-genesis-pn12-applied"
+
    bounds = _class_body_bounds(src)
    if bounds is None:
        return src, "anchor-not-found-class"
@@ -142,7 +148,7 @@ def main() -> int:
    log.info("%s SiluAndMul.forward_cuda: %s", PATCH_TAG, status)

    if patched == src:
-        if status == "skip-already-applied":
+        if status in ("skip-already-applied", "skip-genesis-pn12-applied"):
            return 0
        log.error("%s no changes written; %s", PATCH_TAG, status)
        return 1