Push verified ceilings: long-text 218K, long-vision 198K

Bisected new ceilings now that anchor-fixed PN12 actually pools FFN
intermediates. Findings:

long-text.yml: 205K → 218K (+6%)
  - 0.98 mem-util engine ceiling = 206K (just verified vs prior 205K).
  - Bumping to 0.985 raises engine ceiling to 218K (vLLM-reported 218784).
  - At 218K + 0.985 + no vision + no override: verify-stress + verify-full
    all pass, MTP AL 2.66, VRAM 23.7/24 GB.
  - Dropped --num-gpu-blocks-override 50 — no longer needed at 0.985 because
    the anchor-fixed PN12 cuts allocator churn enough that the natural
    activation budget at this mem-util is sufficient.

long-vision.yml: 192K → 198K (+3%)
  - 0.98 mem-util engine ceiling = 198K (vLLM-reported 198144).
  - 0.985 + vision REOPENS Cliff 1 — more goes to KV at the expense of
    activation budget, and the vision tower's persistent ~1 GB allocation
    leaves no room to absorb the change. So 0.98 stays the right balance
    for the vision variant.
  - At 198K + 0.98 + vision + sidecars: verify-stress passes.

0.99 mem-util ruled out — driver/system reserves ~440 MiB on this hardware
(23.56 GiB visible to vLLM out of 24 GB), so vLLM's startup memory check
fails at 0.99. 0.985 is the practical max.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
noonghunna
2026-04-30 12:43:25 +00:00
co-authored by Claude Opus 4.7
parent 287de1c6e8
commit f3e5b5217c
2 changed files with 31 additions and 26 deletions
@@ -1,10 +1,12 @@
# ===========================================================================
# Long-text variant — engine-ceiling 205K context, text-only (no vision).
# Long-text variant — 218K context, text-only (no vision).
#
# Stack: same as the default + long-vision (vLLM nightly + Genesis v7.14 +
# TurboQuant 3-bit KV + MTP n=3) — but with the vision tower disabled to
# free ~1 GB of VRAM, lifting the engine ceiling from 192K (long-vision)
# to ~206K. We ship 205K (just under the absolute ceiling for safety).
# Stack: same as default + long-vision (vLLM nightly + Genesis v7.62.x with
# the PN12 anchor fix + TurboQuant 3-bit KV + MTP n=3) — but with the vision
# tower disabled to free ~1 GB of VRAM. With the anchor-fixed PN12 actually
# pooling FFN intermediates, we can run at 0.985 mem-util and 218K
# max-model-len without the override 50 KV cap that previous 205K config
# needed. Engine ceiling at 0.985 + no vision is 218K (vLLM-reported).
#
# When to pick this over long-vision:
# - You don't need image input
@@ -24,14 +26,19 @@
#
# Cliff caveat (must read):
# - Cliff 1 (tool-prefill OOM at 25K+ tool messages) is CLOSED on this
# variant as of 2026-04-30 PM via two local sidecars + KV pool override:
# variant as of 2026-04-30 PM via two local sidecars + 0.985 mem-util:
# * patch_pn12_ffn_pool_anchor.py — repairs PN12's anchor on dev205+
# (silent no-op upstream; see Sandermage/genesis-vllm-patches PR #13).
# Idempotent: skips if Genesis-side PN12 is already applied.
# * patch_fa_max_seqlen_clamp.py — local P104 FA softmax_lse clamp.
# * --num-gpu-blocks-override 50 — caps auto-sized KV at ~206K tokens
# (0.77x concurrency at 205K) to reserve activation budget.
# Verified by verify-full + verify-stress on RTX 3090 (MTP AL 2.45,
# VRAM 22.6/24 GB). See docs/CLIFFS.md for the full diagnostic.
# * --gpu-memory-utilization 0.985 (up from 0.98) — gives ~120 MiB
# more to vLLM, mostly KV pool. With anchor-fixed PN12 reducing FFN
# allocator churn, the activation budget at 218K + 0.985 stays large
# enough for 25K-token tool prefills.
# Verified by verify-full + verify-stress on RTX 3090 (MTP AL 2.66,
# VRAM 23.7/24 GB). See docs/CLIFFS.md for the full diagnostic.
# Note: 0.99 mem-util is too aggressive on this hardware (driver/system
# reserves ~440 MiB; vLLM startup check fails). 0.985 is the practical max.
# - Cliff 2 (single-prompt >5060K) STILL APPLIES — DeltaNet GDN forward
# OOM kernel is unrelated to FFN/FA. The 205K context is for STEADY-
# STATE accumulation across many turns, NOT for stuffing 205K of fresh
@@ -133,24 +140,18 @@ services:
- float16
- --tensor-parallel-size
- "1"
# 205K context — engine ceiling at 0.98 mem-util + no vision is ~206K;
# we ship 205K (just under) for safety margin. Cliff 1 prefill closes
# via PN12 anchor sidecar + P104 sidecar + override 50 (see header).
# Cliff 2 (single-prompt ≥5060K in GDN) is independent and still applies.
# 218K context — engine ceiling at 0.985 mem-util + no vision is 218784
# (vLLM-reported); we ship 218K for safety margin. Cliff 1 prefill closes
# via PN12 anchor sidecar + P104 sidecar (see header). Cliff 2
# (single-prompt ≥5060K in GDN) is independent and still applies.
- --max-model-len
- "205000"
- "218000"
- --gpu-memory-utilization
- "0.98"
- "0.985"
- --max-num-seqs
- "1"
- --max-num-batched-tokens
- "4128"
# KV pool cap — frees activation budget for chunked-prefill peaks. With
# PN12 anchor-fixed pooling SiluAndMul output, override 50 leaves enough
# headroom for the 25K-token tool prefill stress to survive at 205K.
# Trade: max_concurrency 0.77x at full ctx. OK for max-num-seqs=1.
- --num-gpu-blocks-override
- "50"
- --kv-cache-dtype
- turboquant_3bit_nc
# Drops the vision tower → frees ~1 GB VRAM → lifts engine ceiling from
@@ -134,11 +134,15 @@ services:
- float16
- --tensor-parallel-size
- "1"
# 192K context — engine ceiling at 0.98 mem-util is ~206K; we ship 192K
# to leave a small safety margin. Activation headroom at 0.98 is ~480 MiB
# — too tight for ≥25K-token tool prefills (Cliff 1).
# 198K context — engine ceiling at 0.98 mem-util + vision is 198144
# (vLLM-reported); we ship 198K for safety margin. With anchor-fixed
# PN12 + P104 sidecars, Cliff 1 prefill closes at this ceiling.
# Note: pushing mem-util to 0.985 reopens Cliff 1 here (more goes to
# KV pool, less to activation budget) — vision tower's persistent
# allocation makes 0.98 the right balance. Long-text drops vision
# and runs at 0.985 + 218K cleanly.
- --max-model-len
- "192000"
- "198000"
- --gpu-memory-utilization
- "0.98"
- --max-num-seqs