Instrumented _genesis_continuation_prefill with a call counter (later
reverted) and ran the 33K-token tool-prefill stress on long-text 185K
+ 0.975 + TQ3 KV. The patched body never executed despite dispatcher
reporting "rebound" at boot. Live turboquant_attn.py:903 in the running
container is still the original torch.cat site.
Same architectural class as PN12 forward_native: vLLM's
aot_compile_fullgraph captures the call chain at compile time, baking
in the original method body; class-attribute rebind doesn't update the
compiled artifact. Sandermage's PROD configs use fp8 KV (not
TurboQuant) so the call site never fires there and the silent no-op
isn't visible. Our TQ3-KV configs surface it.
Practical impact on shipped configs: zero — 33K stress passes anyway
(the line 903 cliff fires only at ~50K-token single-shot prefills).
But P38's persistent K_full/V_full reservation (~700 MiB on 27B at
185K) is currently dead memory until either Genesis converts P38 to
the torch.library.custom_op pattern (mirroring what PN25 does for
forward_native) or the underlying compile-time capture changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
After 383b5cc shipped 175K + 0.97 (text) and 140K + 0.95 (vision),
audit showed the cliffs the backoff was protecting against fire on
every config we ship — they're independent of max-model-len. So the
context capacity was wasted protection.
Push text-only ceilings up:
long-text: 175K + 0.97 → 185K + 0.975
bounded-thinking: 175K + 0.97 → 185K + 0.975
Vision stays at 140K + 0.95: tried 185K + 0.98, 185K + 0.975, 160K
+ 0.97; all reopened Cliff 2 (DeltaNet GDN forward buffer) at the
130K-char stress class. Vision tower's ~1 GiB persistent + the new
patches' persistent allocations (P38 K_full/V_full ~750 MiB at 185K
+ compile-safe sidecar ~138 MiB) leave too little headroom for the
GDN intermediate buffer at 30K+ token prefills on this variant.
P37 disabled on vision (was on for parity with long-text but P37's
MoE intermediate cache pool is no-op on dense Qwen3.6-27B and the
env gate doesn't free memory anyway).
Verification at the new ceilings:
long-text 185K + 0.975: verify-full 8/8 (MTP AL 2.66),
130K-char tool-prefill stress PASS
long-vision 140K + 0.95: verify-full 8/8 (MTP AL 3.27),
130K-char tool-prefill stress PASS
bounded-thinking 185K + 0.975: not re-booted in this final state
(config identical to long-text +
one --structured-outputs flag,
no memory delta expected)
Docs updated: SINGLE_CARD.md picker table + activation-budget +
per-variant blurbs; engines/VLLM.md TL;DR + KV cache table; engines/
LLAMA_CPP.md "when to use vLLM"; STRUCTURED_COT.md "When to pick
this over long-text"; models/qwen3.6-27b/README.md per-variant lines;
docs/CLIFFS.md "Update 2026-05-01 PM" with full bisection sweep and
final decision.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Setup script: pin updated 917519b → 64dd18b. Verified clean on tools-text
(75K + fp8 KV + MTP, PN8 enabled): verify-full.sh 8/8 + verify-stress.sh
tool-prefill OK. v7.64 release notes adopted: PN17 is Sandermage's anchored
version of our P104 sidecar (FA2 softmax_lse runtime clamp), PN19 sets
max_split_size_mb=20 during model load.
long-text.yml (218K + 0.985 + TQ3 + MTP):
- Genesis pin v7.64 (PN17 enabled, PN19 disabled — costs ~120 MiB KV pool
on Ampere consumer; the documented "200-500 MiB win on H100" is negative
on our hardware).
- Drops max-model-len 218000 → 205000 because PN17 reserves ~120 MiB of KV
pool space at boot ("estimated maximum model length is 206400" is the
engine pre-check failure mode otherwise).
- P104 sidecar (patch_fa_max_seqlen_clamp.py) kept mounted but env-disabled;
PN17 covers the same path via Sandermage's anchored fix. Flip the env
var back on if PN17 turns out not to cover turboquant_attn.py for some
config.
- Mounts new patch_pn12_compile_safe_custom_op.py — opaque torch.library
custom_op for the inductor-compiled forward_native FFN path that the
eager-mode forward_cuda sidecar can't reach (issue #16, mech B). Only
active when GENESIS_ENABLE_PN12_FFN_INTERMEDIATE_POOL=1; harmless when
off. Forward_native body simplified to a single static-guard branch on
module-level _PN12_ENABLED so Dynamo specializes at trace time instead
of compiling both branches (the else-branch's plain F.silu/mul lowers
to empty_strided_cuda, defeating the patch).
verify-stress.sh: fixed silent false-positive where check_tool_prefill
returned 0 even after fail() because rm -f cleanup clobbered $? — now
captures rc before cleanup and propagates.
CLIFFS.md: cross-reference Sandermage's broader 8-cliff catalog and add
"vLLM pin compatibility status" section documenting the v0.20 workspace-
lock regression (PR #39226) that blocks our config and is unrelated to
Cliff 1/2.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
CLIFFS.md: replace single-config narrative with shipped-configs table
(long-text 218K, long-vision 198K) plus the bisection table that
established each ceiling.
CHANGELOG.md: add bisection summary to the 2026-04-30 PM entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Initial 'PN12 is partial / architectural wall' framing was wrong. PN12
was silently no-op'd on dev205+ — same anchor-drift bug class as P101.
Once a local sidecar repairs the anchor, Cliff 1 closes at 205K with
verify-full + verify-stress passing and MTP n=3 active.
Sandermage's PN12 design intent was correct; we don't need a
gate_up_proj pool extension. The anchor fix is the missing piece.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
PR #12 (P101 anchor drift fix) is now open on Sandermage's repo.
P104 stays held back pending Sandermage's response on issue #11 to
avoid piling on his in-progress mechanism-B / FFN-pool work.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Tested PN12 + PN13 + P101(anchor-fixed) + P103 + P104 + 50-block-override
on long-text 205K + 0.98 + TQ3 + no-vision. Result: Cliff 1 STILL
fires at 138 MiB / 130 MiB free, same FFN buffer signature.
PN12 only pools SiluAndMul.forward_cuda output (step 3 of FFN forward).
The OOM site is gate_proj or up_proj output (steps 1/2), each shape
[max_num_batched_tokens, intermediate_size] = 138 MiB, fresh-allocated
per layer per step. PN12 cuts allocator churn from 4× per layer to 3×;
significant but insufficient.
Holding off on commenting at issue #11 while Sandermage is actively
shipping. Will share data when he resurfaces. Local Genesis branches
(P101 anchor fix, P104) ready when needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Add "architectural wall" section documenting empirical conclusion from
the Codex agent's 5-hour build session: P101 + P103 + P104 + 50-block-
override on long-text fail at the same FFN buffer (138 MiB / 130.5 MiB
free) regardless of max_model_len in [175K, 205K]. max_num_batched_tokens
is pinned at 4128 by Mamba block_size, sizing the FFN intermediate
buffer at 138 MiB constant. This is the architectural wall for
TQ3 + single-card + MTP at 24GB.
Document what an actual complete fix would require (chunked FFN forward,
drop MTP, FA3/FlashInfer/FlashQLA Ampere path, dual-card TP=2, llama.cpp).
Document explicitly what P104 + P101 anchor fix DO unblock (Genesis
community + future variant optionality, not our currently shipped configs).
Revise recommended path forward: ship P104 + P101 anchor fix as Genesis
PRs (community benefit), keep current shipped composes correct, route
users to dual-card or llama.cpp for genuine cliff-free long-context.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
5-hour Codex agent session shipped two Genesis contributions on a local
branch (club-3090-cliff1-prep in the Genesis clone), waiting for upstream
PR submission. Empirical finding: TQ3+MTP+single-card hits an architectural
wall — Cliff 1 mechanism A (FA2 softmax_lse) closes via P104; mechanism B
(FFN intermediate buffer) is bounded by max_num_batched_tokens which is
pinned at 4128 by Mamba block_size.
P101 anchor drift discovery: P101 was silently no-op'd on dev205+ for
anyone enabling GENESIS_ENABLE_P101=1. apply_all reported "applied"
misleadingly. Fix updates anchor to match upstream torch.arange form.
P104 (new): env-gated FA max_seqlen_k runtime clamp, follows Genesis
text-patch infra, ~260 lines. PR-ready.
Empirical: 205K + 50-block-override + P101+P103+P104 still hits 138 MiB
FFN buffer with 130.5 MiB free. 175K + same stack: identical signature.
max_model_len is not the dominant variable.
No shipped config changes — default 48K + tools-text 75K stay correct.
Updates limited to documentation (CLIFFS.md, UPSTREAM.md, CHANGELOG.md).
Branch cliff1-fa-clamp NOT merged to master — waiting on user review +
upstream PR decision before merging.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Tested Sandermage's existing P101 (TQ continuation 64-token slicing)
+ P103 (FLA Cliff 2 chunked fwd) — both opt-in, default-OFF, never
enabled by us. Hypothesis was they'd close Cliff 1 directly.
Empirical result: they don't fully close Cliff 1. P101 reroutes
around the FA2 softmax_lse mechanism (the one ChatGPT/DeepSeek
pointed at), but Cliff 1 has a SECOND mechanism — the FFN
intermediate buffer (max_num_batched_tokens × intermediate_size
= 4128 × 17408 × 2 bytes = 138 MiB per chunk). With vision tower's
~500 MiB pressure, the FFN buffer dominates and fires Cliff 1
even with P101 active.
Three test runs (all P101+P103 enabled):
- 192K + 0.98 + vision: FFN buffer OOM (138 MiB / 130 MiB free)
- 175K + 0.97 + vision: FFN buffer OOM (138 MiB / 110 MiB free)
- 205K + 0.98 + no-vision: FA2 softmax_lse OOM (50 MiB / 50 MiB)
So the dominant Cliff 1 mechanism depends on whether vision is on.
The proposed FA2 clamp at issue #11 still useful for the tools-text
/ long-text-no-vision case but wouldn't fully unlock long-vision —
FFN buffer is downstream and architectural.
Updates:
- CLIFFS.md: revised "Root cause" section with dual mechanism, table
of which fires under which configs, what each mitigation closes
- CHANGELOG: documented the discovery
- Posted cross-rig data to Sandermage/genesis-vllm-patches#11 (the
pleasant surprise — confirms his patches work for what they do,
identifies the second mechanism we missed)
No shipped config changes. Default 48K + tools-text 75K stay correct.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Three corrections + clarifications from ChatGPT consultation on the
proposed Cliff 1 clamp:
1. Clamp formula correction: use min(attn_metadata.max_seq_len,
actual_max_seq_len_for_this_batch) — NOT chunk size. Chunk size
is the Q dimension; softmax_lse pads on the K dimension which
spans accumulated prompt. Clamping to chunk size would break
continuation prefill.
2. Specific guards documented: FA2/Ampere only, runtime-not-capture,
never below max(seqused_k). Each guard maps to a concrete failure
mode if violated.
3. Implementation shape: env-gated (GENESIS_FA2_CLAMP_MAX_SEQLEN=1),
diagnostic logging at the call site (num_actual_tokens,
max_query_len, attn_metadata.max_seq_len, seq_lens.max()), and
test progression starting at 86K (known-fail) before pushing
higher.
4. Added "Don't pursue --max-num-batched-tokens=2048 as primary fix"
to dead-ends — touches Q dimension; cap leak is on K.
5. Tightened recommended path forward — keep default capped at 48K
UNTIL clamp verified, then cautiously re-open 75K/86K/128K.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Single canonical reference for everything we know about Cliff 1
(FA2 softmax_lse cap-leak) and Cliff 2 (fla.ops GDN forward
intermediate buffer): TL;DR table, empirical bisection with stack
traces, root-cause walk-through, why earlier "FFN intermediate
buffer" framing was wrong, why mem-util doesn't help, why PN8
closes Cliff 1 on tools-text but not on TQ3 paths, why llama.cpp
dodges both structurally, alternative attention backends with
feasibility, who-can-fix-it landscape (Sandermage, Tri Dao, fla-org,
QwenLM, us at any difficulty), recommended path forward, and
re-test triggers.
Cross-linked from FAQ.md and README.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
