f6613c869a
Six rounds of cross-rig bisect with Codex/ChatGPT diagnosis closes Cliff 2 at 60K on TP=1 + 24GB. Two shippable variants: Balanced MTP (long-text.yml updated): - Genesis v7.69 dev tip (commit 2db18df) - Codex r1 P103 gate fix applied to nested Genesis tree (cu_seqlens=[0,T] treated as dense single-seq, not multi-seq varlen). Sent to Sander as v7.70 proposal — diff in /tmp/cliff2_v770_cu_seqlens_response.md. - vllm#35975 backport sidecar (patch_inputs_embeds_optional.py) - mem-util 0.93 (down from 0.95) - max_model_len 180000 (admission ceiling at this mem-util) - MTP K=3 retained - 60K probe: HTTP 200 in 623s, recall correct, MTP AL=4.00 Max-context safety (long-text-no-mtp.yml — NEW): - Same patches, but MTP off + mem-util 0.95 - max_model_len 200000 (admission unlocked by removing MTP residency) - 60K probe: HTTP 200 in 537s, recall correct - 90K probe: indeterminate within 25-min curl budget - For long single-shot RAG / codebase analysis Diagnostic chain: 1. Codex r1 identified P103 gate as too broad (cu_seqlens != None bypasses chunking even for single-seq [0,T] case). Applied fix. 2. T=4128 distribution showed chunked path never engages on real serving (vLLM's outer chunked-prefill caps T well below MAX_T). 3. Codex r2: real Cliff 2 source is residency, not gate logic. 4. PR #35975 backport (skip inputs_embeds for text-only) frees 444 MiB at boot — necessary but not sufficient at 0.95 mem-util. 5. mem-util sweep at 0.92/0.93 with MTP+#35975 closed Cliff 2 at 60K. 6. MTP-off + 0.95 + 200K admission validated max-context variant. Codex's P103 gate fix is semantically correct and worth shipping in Genesis v7.70 even though it's not what closes 60K Cliff 2 on this config (the FLA call sees T=4128 already, well below MAX_T). Full diagnostic trail: results/v0.20-migration/v769-codex-r1-test.summary Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
178 lines
7.5 KiB
Plaintext
178 lines
7.5 KiB
Plaintext
# v7.69 Retest — Branch v7.69-cliff2-test
|
||
Date: 2026-05-02 PM
|
||
Genesis pin: 2db18df (v7.69 dev tip)
|
||
vLLM image: nightly-7a1eb8ac2ec4ea69338c51dc7afd4b15010abfa8 (v0.20.1rc1.dev16)
|
||
Hardware: 1× RTX 3090 (24 GB), SM 8.6 Ampere consumer, TP=1
|
||
Compose: long-text.yml @ 180K + 0.95 (back to original target — DS layout active)
|
||
|
||
## Env bundle (Cliff 2 closure recipe per Sander's v7.69 reply)
|
||
|
||
GENESIS_ENABLE_P103=1
|
||
GENESIS_ENABLE_PN32_GDN_CHUNKED_PREFILL=1
|
||
GENESIS_PN32_GDN_CHUNK_SIZE=8192
|
||
GENESIS_PN32_GDN_CHUNK_THRESHOLD=16384
|
||
GENESIS_FLA_FWD_H_MAX_T=16384
|
||
GENESIS_ENABLE_PN34_WORKSPACE_LOCK_RELAX=1
|
||
VLLM_SSM_CONV_STATE_LAYOUT=DS
|
||
|
||
## Sidecars dropped on this branch
|
||
- patch_pn25_genesis_register_fix.py (replaced by Sander's PN25 v7.68)
|
||
- patch_pn30_dst_shaped_temp_fix.py (replaced by Sander's PN30 v7.68 + v7.69 part3 marker fix)
|
||
- patch_workspace_lock_disable.py (replaced by Sander's PN34, opt-in env)
|
||
|
||
## ✅ F1 — PN30 v7.68 part3 drift-marker fix CONFIRMED WORKING
|
||
v7.68 dev tip (18e65e3) had part3 skip with `upstream_merged` because
|
||
its drift-markers list contained the bare prefix `[Genesis PN30` which
|
||
also matched parts 1+2's own markers. v7.69 (2db18df) tightened part3's
|
||
drift markers to `[Genesis PN30 v7.68 dst-shaped]` — specific enough to
|
||
not collide.
|
||
|
||
Boot log:
|
||
[PN30 v1/worker/mamba_utils.py — collect_mamba_copy_meta dst-shaped DS
|
||
temp (issue #17, v7.68)] applied 1 sub-patches:
|
||
pN30_collect_mamba_copy_meta_dst_shaped_temp
|
||
[Genesis] applied: PN30 DS conv state + spec-decode AL>1 (issue #17)
|
||
— PN30 v7.68 applied: ...
|
||
|
||
DS layout active (`Setting SSM conv state layout to DS`). Part1 + part2
|
||
+ part3 all APPLY clean, no upstream_merged false-positive. Apply_all
|
||
elapsed: clean, 0 failures.
|
||
|
||
## ✅ F2 — P103 chunk.py self-install hook CONFIRMED FIRES
|
||
v7.68 P103 logged "rebound at 0 caller sites" because `setattr` ran in
|
||
the entrypoint shell process and was lost on `exec vllm serve` (worker
|
||
spawn replaced the process image).
|
||
|
||
v7.69 P103 ships a text-patched self-install hook appended to
|
||
chunk.py end-of-file:
|
||
|
||
try:
|
||
import os as _genesis_p103_os
|
||
if _genesis_p103_os.environ.get("GENESIS_ENABLE_P103", ...):
|
||
from vllm._genesis.wiring.hybrid.patch_103_fla_cliff2_chunked
|
||
import _genesis_p103_install_at_import as _genesis_p103_install
|
||
_genesis_p103_install(globals())
|
||
except Exception:
|
||
pass
|
||
|
||
Confirmed firing on TP=1 spawn. Cliff 2 trace path:
|
||
|
||
vllm/model_executor/layers/fla/ops/chunk.py:111 forward
|
||
→ vllm/_genesis/wiring/hybrid/patch_103_fla_cliff2_chunked.py:203
|
||
chunked_fwd ← P103 wrapper engaged at runtime
|
||
→ return original_fwd(...) ← BUT falls through to fallback
|
||
|
||
The wrap IS engaged in workers. F2 is verified at the
|
||
"hook fires correctly" level. Boot log:
|
||
[Genesis P103 self-install] wrapper installed in chunk.py at
|
||
module-import time (survives `exec vllm serve` + worker spawn)
|
||
|
||
## ⚠️ F3 — Cliff 2 NOT closed at 60K despite P103 + PN32 v2
|
||
|
||
Probes pass / fail summary:
|
||
[1/7] Long-context needle 10K ✅ recalled 'silver otter 16'
|
||
[1/7] Long-context needle 30K ✅ recalled 'amber iguana 95'
|
||
(was crashing on v7.68; v7.69 fix)
|
||
[2/7] Tool prefill 25K ✅ tool_call clean
|
||
[3/7] IDE-agent one-shot ✅ 66 completion tokens
|
||
[4/7] Multi-turn agent ✅
|
||
[5/7] LCB-coding shape ✅
|
||
[6/7] Reasoning max=8192 ✅
|
||
[7/7] Long-context needle 60K ❌ HTTP 500 (Cliff 2 OOM)
|
||
[7/7] Long-context needle 90K ❌ HTTP 500
|
||
|
||
Cliff 2 OOM trace (probe 7 60K rung):
|
||
|
||
File "vllm/model_executor/layers/fla/ops/chunk.py", line 111, in forward
|
||
g, o, A, final_state, w, h, v_new = chunk_gated_delta_rule_fwd(
|
||
File "vllm/_genesis/wiring/hybrid/patch_103_fla_cliff2_chunked.py",
|
||
line 203, in chunked_fwd
|
||
return original_fwd( ← P103 fallback path
|
||
File "vllm/model_executor/layers/fla/ops/chunk.py", line 72,
|
||
in chunk_gated_delta_rule_fwd
|
||
o = chunk_fwd_o(
|
||
File "vllm/model_executor/layers/fla/ops/chunk_o.py", line 161,
|
||
in chunk_fwd_o
|
||
o = torch.empty_like(v)
|
||
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 50.00 MiB.
|
||
GPU 0 has a total capacity of 23.56 GiB of which 24.50 MiB is free.
|
||
|
||
### Why P103's chunked path didn't engage
|
||
|
||
P103's gate is `T <= MAX_T or cu_seqlens != None`. At 60K prompt
|
||
serving:
|
||
- T = 60000 (well above MAX_T=16384)
|
||
- But cu_seqlens != None (vLLM serving ALWAYS passes cu_seqlens)
|
||
→ P103 falls through to original_fwd
|
||
|
||
### Why PN32 v2 didn't engage
|
||
|
||
PN32 v2's docstring says "Multi-seq bypasses to original" — i.e. it
|
||
only chunks when there's exactly one sequence. With MTP K=3
|
||
spec-decode, the dispatcher reports `uniform_query_len=4` per
|
||
P66 cudagraph filter log. Whether PN32's gate sees this as
|
||
"multi-seq" wasn't traceable at runtime, but the trace doesn't
|
||
show PN32's chunked path.
|
||
|
||
### Net: F3 closure works only when cu_seqlens=None AND single-seq
|
||
|
||
Both P103 and PN32 v2 have gates that require cu_seqlens=None
|
||
and/or single-sequence to engage. vLLM serving always passes
|
||
cu_seqlens (it's the per-sequence boundary marker that enables
|
||
batched serving). So neither chunking path fires under real
|
||
serving, only synthetic test scenarios.
|
||
|
||
## Other engine state observations on v7.69
|
||
|
||
Engine survived probe 7 OOM gracefully (HTTP 500 + EngineDeadError
|
||
exception → API server shutdown). Compare to v7.66 where Cliff 2 OOM
|
||
killed the EngineCore process abruptly. Suggests vLLM's
|
||
`async_llm.output_handler` exception handling is cleaner on v0.20+.
|
||
Engine still came back up clean on next request batch (could re-test
|
||
small probes after large probe failed).
|
||
|
||
## Cross-rig regressions: NONE
|
||
TPS on smoke benchmark equivalent to v7.66 + sidecars (same engine
|
||
path under cu_seqlens != None, no overhead from PN32 v2 in fallback
|
||
mode).
|
||
|
||
## Recommendation back to Sander
|
||
|
||
Two options for v7.70:
|
||
|
||
(a) Ship F3 fix that handles cu_seqlens != None — slice metadata
|
||
along with tensors per chunk; thread initial_state correctly
|
||
across chunk boundaries even when cu_seqlens describes a
|
||
multi-sequence batch. Per Sander's own caveat in v7.69 reply:
|
||
"Multi-seq bypasses to original (chunking across cu_seqlens
|
||
boundaries needs inner state-cache surgery not exposed at this
|
||
layer)" — this is the surgery that needs to happen.
|
||
|
||
(b) Keep current "synthetic-only" gate but document explicitly
|
||
that P103 + PN32 only close Cliff 2 in benchmarks where
|
||
cu_seqlens=None is passed manually. Real serving stays
|
||
behind the cliff. Operators on TP=1 + 24GB should plan for
|
||
50-60K single-prompt ceiling and route longer prompts to
|
||
TP=2 (`dual.yml`) or llama.cpp.
|
||
|
||
Our preference: (a) — but we recognize this is non-trivial. Happy to
|
||
engage ChatGPT/Codex CLI on the cu_seqlens-aware chunking design and
|
||
share back as we did for PN30 dst-shaped + PN25 import-time
|
||
registration.
|
||
|
||
## What stayed working
|
||
- All 4 of v7.68's other accepted-and-folded patches (PN25 v7.68
|
||
TP=1 register, PN34 workspace_lock, PN30 v7.68 part1+part2, P101)
|
||
remain solid.
|
||
- Tool calls clean across IDE-agent shapes (probes 3 + 4).
|
||
- 30K single-prompt now safe (was unsafe on v7.68 due to PN30
|
||
being broken; v7.69 closes that regression).
|
||
|
||
## Net: v7.69 is a clear improvement over both v7.66 and v7.68 dev tip
|
||
|
||
If we accept that Cliff 2 is fundamentally still architectural for
|
||
single-24GB-GPU long-prompt and route around it, v7.69 is our
|
||
best stable single-card config to date. Recommend Sander tag v7.69
|
||
stable and we move to master once the 30K-now-safe data is
|
||
cross-confirmed on Sander's planned A5000-rig.
|