Cliff 3 (CLIFFS.md) walked readers to "use llama.cpp, vLLM is hopeless
for agentic Qwen3-Next" with zero mention that LMCache is the measured
mitigation for exactly that cliff (#423: 40K prefix warm-loads 0.5-2.4s
L1 vs 34-43s cold). Native prefix cache / generic CPU-offload hit Cliff
3; LMCache's HMA hybrid recurrent-state caching is the differentiator.
Add a bounded mitigation subsection (measured for resumed prefixes,
flagged untested for growing loops), revise the recommendation to two
paths, and forward-link INTERNALS' LMCache section to Cliff 3.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Second way the needle test gives a false all-clear (distinct from the #200
fixed-depth issue): it's insensitive to KV-cache-quant distributional drift.
Per Anbeeld's Qwen3.6-27B / RTX 3090 KV-quant benchmarks, needle at 32K scores
100% across every cache mode while 99.9%-pct KLD tail precision falls 100%->54%.
So verify-stress 7/7 (incl. 91K needle) does NOT certify a KV-quant choice is
tail-safe for code/JSON/agent workloads. Prefer >= q5_0/q4_1 (asymmetric) there.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Repo-wide follow-up to the compose quant-layer move (9821c94). The
mechanical move + registry/launch/test rewire covered the launchable
surface; a full-tree sweep found compose-path refs the move invalidated
in docs, two functional scripts, and one half-migrated mapping.
Functional fixes:
- bench-row-formatter.sh infer_compose_path(): 6 dual entries
(int8-tq3, tq3-mtp-genesis, tq3-nomtp, tq3-mtp, int8, bf16) were left
at bare dual/<file>.yml while the rest were migrated -> would emit
dead compose paths into BENCHMARKS rows.
- residency-instrument/run-instrumented-soak.sh: case->COMPOSE_FILE
paths (long-text, long-text-no-mtp, tools-text, dual default) now
resolve under <quant>/.
Docs: README layout line + tree, engine/model READMEs, patch-README
quick-recipes, diagnostics, FAQ/CLIFFS/KV_MATH/DTYPE/MULTI/SINGLE/
STRUCTURED_COT/TQ3/UPSTREAM, issue template, sglang cross-refs
(-> vllm prod path). Per-model targets: qwen-vllm->autoround-int4,
llama-cpp->unsloth-q4km, gemma defaults (bf16-mtp/fp8-mtp),
carnice->own slug dir.
Intentionally left as historical/append-only records: CHANGELOG x2,
BENCHMARKS row-labels (live paths already correct in row bodies),
calibration source: provenance citations, switch.sh/parity history
comments. Separate follow-ups: gpu-mode.sh (#417 deprecated-repo
repoint), bench-row-formatter compose_display() docker-compose.yml
branch (PR-B). Flagged pre-existing-stale: dual/int8-tq3.yml in
pr40798/pr40914 READMEs (predate this refactor; ambiguous target).
Guard tests (registry-disk, mounts-resolve, switch-parity,
launch-compat) all PASS post-edit. Leak-clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The shipped llamacpp/mtp default CTX_SIZE=262144 boots, reserves its KV
pool, and passes verify-stress(91K) but only *fills* to ~125K before
OOMing on the flash-attention transient scratch at high fill
(launch_fattn/cuMemCreate) — not the GDN cliff, not the reserved pool.
Measured via the #200 ceiling-ladder probe (single 3090, Q4_K_M, q4_0 KV,
MTP n=2, -ub 512): 262K walls at 155K; 200K fills 183K (1177 MB free,
clears the 1024 MB margin); 224K is knife-edge. Calibrated rates: ~25
MB/1K KV reservation, ~7.9 MB/1K at-fill growth. Surfaced diagnosing
syangsao's single-card OOM (#197).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to commit 1b2a76c which added the speed-vs-context section to
models/qwen3.6-27b/llama-cpp/README.md. The new learning (smaller -ub
buys more ctx, especially when mmproj F16 is loaded) needs to be
discoverable from the high-level user-facing docs too — not just buried
in the model-specific README.
- SINGLE_CARD.md: corrected the "49K is the safe-headroom max with mmproj"
claim (it's the speed-optimal max; ub=512 lever pushes to 192K) +
prominent override snippet right under the profile description.
- CLIFFS.md: added a 2026-05-20 refinement note pointing out that `-ub`
is doing two jobs (cliff-survival + KV-budget), and the optimal value
is configuration-conditional on vision-on/off.
- FAQ.md: inline note in the canonical switch.sh recipe for mtp-vision
pointing users at the override.
All three back-reference the README's full sweep data as the canonical
source; surfaced by @JensJN in #170.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Soak-test PASS only verifies "no failure signal on this sample at this
depth," not "patches in the compose's overlay set are load-bearing for
this workload." On TP=2 / llama.cpp configs the topology itself takes
Cliff 2 off the table, so PASS on those composes can't attribute work
to any specific patch.
- soak-test.sh header docstring: new "PASS verdict semantics" block
- soak-test.sh --help: matching "PASS VERDICT" section
- soak-helper.py: PASS verdicts now print a one-line caveat pointing
to scripts/soak-test.sh --help and docs/CLIFFS.md
- docs/CLIFFS.md: callout in "Why TP=2 escapes" explaining what a clean
dual.yml soak does and does not validate
Patch-attribution path (rerun with overlays stripped) referenced in all
three surfaces.
Two findings from @easel's deep cross-rig writeup on issue #102 (5090
Laptop WSL2):
1. **HARDWARE.md** — new "GPU memory budget on WSL2" subsection. WSL2
container CUDA-context consumes ~1.31 GiB before vLLM profiler runs.
`gpu_memory_utilization=0.95` crashes at boot; 0.944 works. Validated
on 2 machines. Formula: `(vram_total - 1.31) / vram_total`.
Recommendation: `GPU_MEMORY_UTILIZATION=0.94` in `.env` on WSL2.
2. **CLIFFS.md** — new section "Cliff 3 — DeltaNet SSM state is not
prefix-cacheable (the prefill cliff)". This is a structural finding
that explains a class of failures we'd been describing without
naming. @easel's warm-cache run (68.7% KV-block hit, turn 10 at
35.6K tokens took 577s — 2.3× the cold-start 254s) is the smoking
gun: prefix cache helps attention but DeltaNet's recurrent state
`h_t = f(h_{t-1}, x_t)` must be recomputed from scratch every turn.
PN32 fixes OOM stability; nothing fixes the O(n) prefill scaling
because the architecture itself is sequential.
Practical ceiling on single-card vLLM (any 24 GB Qwen3-Next config):
sub-30s TTFT only below 5K accumulated tokens. 22-35K = 3-4 min/turn.
~74K = 10+ min client timeout.
Elevates "for single-card agentic Qwen3-Next, use llama.cpp" from
implied to explicit in the docs. Dual-card extends the envelope to
25-30K accumulated; deep sessions (50K+) still need llama.cpp on
either topology.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Compose files now live under `<model>/<engine>/compose/<topology>/<file>.yml`,
with topology as a folder rather than a filename prefix. Solves all 7
inconsistencies surfaced in the post-rename audit (single-card composes
without `single-` prefix, unsuffixed `docker-compose.yml` ambiguity,
fine-tunes encoding model name in filename, etc.) by making the directory
hierarchy enforce the convention.
Layout:
models/<model>/<engine>/compose/<topology>/<feature>.yml
Where:
- <model>: qwen3.6-27b, gemma-4-31b
- <engine>: vllm, llama-cpp, sglang
- <topology>: single, dual, multi3, multi4, multi8
- <feature>: docker-compose.yml (default) | turbo.yml | dflash.yml | etc.
Each topology subdir has a `docker-compose.yml` for the recommended
starter — bare `cd <topology> && docker compose up` works because
docker compose finds that filename automatically. Variants drop the
`docker-compose.` prefix since they're invoked via `-f` flag.
27 compose file moves total:
- 18 Qwen vLLM composes redistributed across single/dual/multi4
- 2 Qwen llama-cpp composes into single/
- 6 Gemma vLLM composes redistributed across single/dual
- 1 untracked qwopus-bf16mtp moved to dual/
Inside each compose: relative paths to `../patches/` and `../cache/`
bumped to `../../patches/` / `../../cache/`, and `../../../../models-cache`
to `../../../../../models-cache` (one extra `..` for the new depth).
Reference updates across 148 files (BENCHMARKS, all docs, CHANGELOGs,
sibling-table cross-references in compose headers, scripts, patch
READMEs, .github issue templates, tools/residency-instrument).
scripts/switch.sh VARIANTS map updated; tags themselves unchanged
(`vllm/dual` → `dual/docker-compose.yml`, `vllm/dual4` → `multi4/docker-compose.yml`,
`vllm/gemma-mtp` → `gemma-4-31b/.../dual/docker-compose.yml`, etc.).
AGENTS.md "Compose layout" section rewritten to describe the new
hierarchy, with concrete examples and the fine-tune exception
(`dual/carnice-bf16mtp.yml` carries the fine-tune name as a filename
prefix until the fine-tune graduates to its own model directory).
All switch.sh paths verified to resolve to actual files post-move.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three documentation/script follow-ups from @easel's #102 re-bench
on RTX 5090 Laptop:
- HARDWARE.md: new "Laptop GPUs — EC-managed power" subsection.
nvidia-smi -pl returns N/A on laptop-class GPUs (EC owns the
envelope, not the OS). Documents clock-lock as the only
software characterization path on laptops.
- CLIFFS.md: new "naming trap" callout in the KV-format section.
fp8_e5m2 is 8 bits/token; turboquant_3bit_nc packs 3 bits.
At 180K on 24GB, TQ3 fits where fp8 OOMs (4.36 GiB available
vs 6.64 GiB needed for fp8). Pin: TQ3 = long-context KV;
fp8 = short-context throughput.
- verify-stress.sh: auto-detect VLLM_ENFORCE_EAGER=1 in the
running container's env via docker inspect; when set, bump
STRESS_LONGCTX_TIMEOUT_S 300→600s and STRESS_TOOL_PREFILL_-
TIMEOUT_S 240→480s. Eager-mode prefill at 60K-140K runs
200-290s and was false-positiving as HTTP 000 (curl timeout)
in @easel's run. Both env vars also exposed for manual override.
Refs: noonghunna/club-3090#102
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three additions across docs/ that came out of today's reply trail:
1. NEW docs/CONTAINER_RUNTIMES.md — non-Docker / non-bare-metal
environmental notes. Captures:
- Soft-warn pattern (setup.sh docker check is now non-load-bearing
after 2f8ed19, launch.sh / switch.sh keep hard check)
- Podman / Podman Compose env override (COMPOSE_BIN already supported)
- microk8s integration story — open invitation for @apnar to PR a
manifest example after their 5090 + 3090 Ti benches
- Proxmox VE 8.x / kernel 6.17.x asyncio crash class — full
elimination trail from #49 (Genesis ruled out, torch.compile ruled
out, multiproc ruled out, default-runtime: nvidia ruled out, --init
ruled out, vLLM upstream ruled out via my Probe B reproduction).
Verdict: environmental, parked. Re-check triggers listed.
2. docs/HARDWARE.md — added "Note for WSL2 / Windows users" covering
TDR class diagnosis from #50 (RossNE99). Concrete PowerShell registry
edit (TdrDelay=60s) + three escalating fix paths + WSL2-specific
gotchas (pin_memory auto-off, host RAM for paged load). Plus a
pointer to CONTAINER_RUNTIMES.md for non-Docker users.
3. docs/CLIFFS.md — refined the "rig-class caveat" callout to point at
CONTAINER_RUNTIMES.md for the full Proxmox elimination trail (vs
inline summary).
These are observation notes for users hitting environmental issues that
aren't club-3090 bugs. Not recipes for non-default runtime setups
(those need cross-rig validation we don't have CI for).
Refs: #49, #50, disc #48
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@lexhoefsloot's bisect on club-3090#49 surfaced that the v7.66 baseline
my CLIFFS.md called "known good" actually crashes on his rig
(3× 3090 / Proxmox VE / Debian 12 / kernel 6.17.2 / default-runtime:
nvidia / dual-turbo + override-pve compose) with an uvloop event-loop
trace BEFORE engine initialization. The crash:
- predates v7.66 (asyncio bug exists at fc89395 too)
- is independent of GENESIS_ENABLE_P87 (P87=0 doesn't fix it)
- masks-then-fires across the v7.66..v7.69 range (PN30 marker
collisions on intermediate commits)
The v7.66/v7.69 stability verdict still holds on the rig class it was
measured on (bare-metal Ubuntu 2× 3090 PCIe, default Docker runtime)
but isn't a universal claim. Adding a callout near the top of the pin
status section so future readers landing on a non-baseline rig
understand "known good" is environment-conditional.
Refs: #49
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- setup.sh: demote `preflight_docker` from hard ✗ to soft ⚠. setup.sh
fetches genesis + models but never invokes docker until launch.sh /
`docker compose up`, so hard-failing blocks non-docker container-runtime
users (microk8s, podman, k8s, manual) for no reason. launch.sh keeps the
hard check because it actually invokes docker.
Reported in disc #48 (apnar).
- dual-turbo.yml: default `GENESIS_ENABLE_P87=0`. The marlin pad-sub-tile-n
fix is already vendored at ../patches/vllm-marlin-pad/marlin.py and
RO-mounted over the target file (lines 53-54). Letting Genesis re-do the
patch fails with [Errno 30] read-only filesystem and `set -e` propagates
exit-1 from `apply_all` before `vllm serve` runs.
Reported in #49 (lexhoefsloot).
- CLIFFS.md: top-level pin header now says "Genesis v7.69 (2db18df)" instead
of v7.66 (fc89395). Master moved to v7.69 on 2026-05-02 PM (Sander cut
v7.69 with all 3 cross-rig sidecars accept-and-folded; full closure recipe
was already documented in the v7.69 section, just the top-of-file header
+ the older v7.68 verdict line + table caption hadn't been updated).
Removed the duplicated v7.66 stub section (lines 60-66 + 68-103 were the
same paragraph twice). setup.sh:140 has been on `2db18df` already.
Reported in #49 (lexhoefsloot).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds arxiv citations to the Cliff 2 mechanism (CLIFFS.md) and the
TurboQuant KV format (HARDWARE.md), with a new "Academic references"
section in CLIFFS.md gathering all relevant papers in one place.
Headline finding: PerfMamba (arxiv 2511.22849) directly documents the
Cliff 2 mechanism in the parent architecture. At seq 2048, Mamba-2's
SSM consumes 33.5% more memory than Mamba-1 due to "block-wise state
materialization" — the same pattern Qwen3-Next inherits via Gated
DeltaNet. Activation peak scales as O(γ·D·N·L). That's the formal
scaling we'd been describing empirically.
What the literature doesn't cover (and is club-3090's contribution):
the activation-peak interaction with KV quantization format choice
(TQ3 vs fp8) and the per-VRAM-class budget consequences for consumer
Ampere deployments. PerfMamba describes the mechanism; we describe
the application-side trade-offs.
Citations added:
- arxiv 2511.22849 (PerfMamba) — Cliff 2 root mechanism
- arxiv 2504.19874 (TurboQuant ICLR 2026) — TQ3 KV technique
- NVlabs/GatedDeltaNet ICLR 2025 — Qwen3-Next architecture
- arxiv 2312.00752 (Mamba) — baseline for PerfMamba's deltas
- arxiv 2309.06180 (PagedAttention) — vLLM foundation
- arxiv 2502.01070 (FP8 across accelerators) — fp8 KV
- arxiv 2512.01644 (Systematic Char. of LLM Inference) — recent context
- arxiv 2503.08311 (Mind the Memory Gap) — peak memory patterns
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 20 GB Ampere finding from @efschu was already in HARDWARE.md (commit
124f08c) but not cross-referenced from the surfaces a user lands on first
when hitting the symptom. This commit closes the gap:
- docs/CLIFFS.md — new "KV format choice tunes the boundary" subsection
under Cliff 2 root-cause. Generalizes from efschu's specific 20 GB finding
to the principle: variant matrix is per-card-budget × KV-format-tradeoff
aware; shipped defaults are tuned for 24 GB / 3090; users on different
VRAM classes may need to override --kv-cache-dtype.
- docs/DUAL_CARD.md — dual-turbo picker row gets a "20 GB Ampere users:
override TQ3 → fp8_e5m2; see HARDWARE.md + #47" inline pointer.
- models/qwen3.6-27b/vllm/compose/docker-compose.dual-turbo.yml — comment
block above the --kv-cache-dtype line documenting the rationale, the
swap rule, and the cross-link to HARDWARE.md.
- CHANGELOG.md — records the lesson as a stack-level finding so it's
discoverable in repo history. Notes future work on KV_FORMAT env knob
+ preflight in #219.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After the full Phase 3 5-grammar A/B (HE+ 164 + LCB v6 50, n=214 problems
× 5 conditions = 1070 generations), bounded-thinking.yml is updated to
recommend the DeepSeek scratchpad grammar (PLAN/NOTE×0-15/VERDICT FSM at
tools/grammar-eval/deepseek-scratchpad.gbnf) as the default.
Phase 3 results (combined HE+ + LCB v6, all FSM-enforced):
- DeepSeek scratchpad: 87.4% Pass@1 (+1 net vs andthattoo, +4pp on LCB)
- andthattoo G/A/E: 86.9% (the originally-published technique)
- Holiday tagline: 86.4%
- PROMPT_TERSE (no FSM): 82.2% (Phase 2's n=30 win was subset-selection bias)
- FREE (no constraint): 78.0% (baseline)
Phase 1 reproducibility is exact: HE+ FSM Δ +4.3pp / LCB v6 Δ +24.0pp,
both match Phase 1's published numbers — validates the bench harness.
Compose ships unchanged engine-side (same vLLM image, same Genesis stack,
same TQ3 KV, same MTP n=3, same enable_in_reasoning flag). Only the
docstring's recommended-grammar pointer + the on-disk grammar files in
tools/grammar-eval/ change. Three grammars are now validated and available
client-side via extra_body={"structured_outputs": {"grammar": ...}}:
- DeepSeek scratchpad (default, best LCB)
- andthattoo G/A/E (originally-published, ~4× tighter think budget)
- Holiday tagline (extreme 24-token compression, wins LCB by 4pp too)
Combined-accuracy spread is within noise (0.5pp at n=214), so we ship one
compose rather than three siblings — choice is at the client, not at the
compose level.
Bug fix bundled: tools/grammar-eval/subset-bench.py --full --include-lcb
mode now correctly threads dataset kind through run_condition so LCB
problems use mod.run_tests_livecodebench instead of HE+ assertion-based
testing. Without this, the Phase 3 LCB shard crashed with KeyError:
'prompt' on the HE→LCB transition.
This wraps active research on bounded-thinking. Reopen if upstream FSM-
regress cluster behavior changes (Genesis pin bump, vLLM grammar engine
swap, model-family change), or if user demand surfaces for a sibling
compose pinning a non-default grammar.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Today's full validation matrix exposed Cliff 2b — DeltaNet GDN forward
fires under accumulated multi-turn context (~21-26K), not just at single
prompts >50-60K. All 6 single-card vLLM variants FAIL v2 continuous;
only vllm/dual (TP=2) and llamacpp/default survive cleanly. Three issues
filed today (#41, #42, #43) all map to the same class. Docs needed to
reflect this so users don't keep landing on long-text/long-vision/
tools-text and burning time tuning what won't fix.
docs/SINGLE_CARD.md
- New ⚠️ section at top: single-card vLLM unsafe for hermes/openhands/
OpenCode/Cline/OpenClaw/Aider/Cursor with retained context. Routing
tree to dual.yml or llamacpp/default.
- Existing "One limitation" split into Cliff 2a (single-prompt, mostly
closed v7.69) and Cliff 2b (multi-turn, NOT closed). Reasons each
can/can't be tuned at config layer.
docs/CLIFFS.md
- TL;DR table extended from 2 cliffs to 3 (Cliff 2a + 2b separated).
- New section "Why TP=2 escapes" — per-card head sharding halves the
GDN live-tensor sizes (48 MiB → 24 MiB on v/u/o etc., 97 → 49 on h).
Concrete byte math; total per-card live FLA set drops from ~500 MiB
to ~250 MiB. Validated 0 MiB growth on dual.yml v2 continuous.
- New section "Why llama.cpp escapes" — three concrete differences:
(1) different GDN kernel (own CUDA, smaller per-step working buffers
vs FLA Triton), (2) ggml manual allocator (no PyTorch caching layer
fragmentation), (3) no JIT/Triton autotune (pre-compiled, static
layout from boot). Trade ~3× decode speed for cliff-immunity.
docs/FAQ.md
- New troubleshooting entry: "My hermes / openhands / OpenCode / Cline /
OpenClaw / Cursor session OOMs after a few turns. What do I do?"
Lists every dead-end we tested today so users don't repeat them:
mem-util tuning, MTP-off, max-num-batched-tokens (Mamba block_size
floor blocks <4128), TRITON_CACHE_AUTOTUNING (Blackwell-only recovery),
expandable_segments (already on), empty_cache (reclaims but cliff
fires next turn). Routing answer + soak-test repro command.
CHANGELOG.md
- Dated entry above the soak-test v2 entry summarizing the docs sweep.
Codex residency pilot data backs every claim:
results/residency-20260503-codex-pilot3/ (initial)
results/residency-20260503-empty-cache-idle/ (empty_cache experiment)
Investigation memo: docs/diagnostics/cliff2-handoff-results.md (gitignored).
Sandermage shipped both companion fixes within hours of our reports:
- P38B (#14 fix) — text-patch source-level hook in _continuation_prefill.
Source edit survives aot_compile_fullgraph capture; different from our
PN12→PN25 torch.library.custom_op route, both reach the same end.
- P15B (#15 fix) — direct backport of our suggestion path 1. Clamps
max_seqlen_k at TQ wrapper boundary from cu_seqlens_k. One GPU→CPU
sync/call, acceptable on the infrequent continuation-prefill path.
Both opt-in via env (GENESIS_ENABLE_P38B_COMPILE_SAFE=1 / P15B_FA_VARLEN_
CLAMP=1) on Sandermage's dev branch. Will land in v7.65 release.
UPSTREAM.md rows for #14 + #15 + PN25 updated to status "Fix on dev,
pending v7.65" with the env vars to enable when adopting. Added P98
marker false-positive row (we side-noted it on Genesis #9 thread —
awaiting Sander's call on a marker fix).
CLIFFS.md "Update 2026-05-01 PM" section extended with the P38B/P15B
landing news + a cross-reference to the v0.20 path: empirically the 50K
cliff doesn't reproduce on v0.20 either, so we have two independent
paths to the same outcome. Holding both until v7.65 ships so the master
migration is one coherent PR (pin + Genesis + sidecar cleanup + context
restoration to 218K/198K).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Updated "vLLM pin compatibility status" with the empirical unblock:
1. Sandermage's P98 auto-skips on v0.20 (drift marker false-positive).
2. Local patch_workspace_lock_disable.py sidecar relaxes the strict
assertion to a one-shot WARNING.
3. With the sidecar + Genesis v7.64 + compile-safe sidecar:
verify-full 8/8, 33K-token stress PASS, 50K-token stress PASS.
The 50K-stress-PASS is the big signal — that cliff fires on EVERY dev205
config (long-text trips line 903; long-vision trips line 909→394→300).
Suggests v0.20 implicitly resolves Genesis #14 (P38 silent no-op) and
#15 (FA varlen workspace) for our configs, possibly via vllm#40092's
TQ FA3/FA4 prefill paths changing the workspace allocator behavior.
Cross-validation across long-vision + bounded-thinking + dual variants
needed before considering a master pin bump from dev205.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Instrumented _genesis_continuation_prefill with a call counter (later
reverted) and ran the 33K-token tool-prefill stress on long-text 185K
+ 0.975 + TQ3 KV. The patched body never executed despite dispatcher
reporting "rebound" at boot. Live turboquant_attn.py:903 in the running
container is still the original torch.cat site.
Same architectural class as PN12 forward_native: vLLM's
aot_compile_fullgraph captures the call chain at compile time, baking
in the original method body; class-attribute rebind doesn't update the
compiled artifact. Sandermage's PROD configs use fp8 KV (not
TurboQuant) so the call site never fires there and the silent no-op
isn't visible. Our TQ3-KV configs surface it.
Practical impact on shipped configs: zero — 33K stress passes anyway
(the line 903 cliff fires only at ~50K-token single-shot prefills).
But P38's persistent K_full/V_full reservation (~700 MiB on 27B at
185K) is currently dead memory until either Genesis converts P38 to
the torch.library.custom_op pattern (mirroring what PN25 does for
forward_native) or the underlying compile-time capture changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After 383b5cc shipped 175K + 0.97 (text) and 140K + 0.95 (vision),
audit showed the cliffs the backoff was protecting against fire on
every config we ship — they're independent of max-model-len. So the
context capacity was wasted protection.
Push text-only ceilings up:
long-text: 175K + 0.97 → 185K + 0.975
bounded-thinking: 175K + 0.97 → 185K + 0.975
Vision stays at 140K + 0.95: tried 185K + 0.98, 185K + 0.975, 160K
+ 0.97; all reopened Cliff 2 (DeltaNet GDN forward buffer) at the
130K-char stress class. Vision tower's ~1 GiB persistent + the new
patches' persistent allocations (P38 K_full/V_full ~750 MiB at 185K
+ compile-safe sidecar ~138 MiB) leave too little headroom for the
GDN intermediate buffer at 30K+ token prefills on this variant.
P37 disabled on vision (was on for parity with long-text but P37's
MoE intermediate cache pool is no-op on dense Qwen3.6-27B and the
env gate doesn't free memory anyway).
Verification at the new ceilings:
long-text 185K + 0.975: verify-full 8/8 (MTP AL 2.66),
130K-char tool-prefill stress PASS
long-vision 140K + 0.95: verify-full 8/8 (MTP AL 3.27),
130K-char tool-prefill stress PASS
bounded-thinking 185K + 0.975: not re-booted in this final state
(config identical to long-text +
one --structured-outputs flag,
no memory delta expected)
Docs updated: SINGLE_CARD.md picker table + activation-budget +
per-variant blurbs; engines/VLLM.md TL;DR + KV cache table; engines/
LLAMA_CPP.md "when to use vLLM"; STRUCTURED_COT.md "When to pick
this over long-text"; models/qwen3.6-27b/README.md per-variant lines;
docs/CLIFFS.md "Update 2026-05-01 PM" with full bisection sweep and
final decision.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Setup script: pin updated 917519b → 64dd18b. Verified clean on tools-text
(75K + fp8 KV + MTP, PN8 enabled): verify-full.sh 8/8 + verify-stress.sh
tool-prefill OK. v7.64 release notes adopted: PN17 is Sandermage's anchored
version of our P104 sidecar (FA2 softmax_lse runtime clamp), PN19 sets
max_split_size_mb=20 during model load.
long-text.yml (218K + 0.985 + TQ3 + MTP):
- Genesis pin v7.64 (PN17 enabled, PN19 disabled — costs ~120 MiB KV pool
on Ampere consumer; the documented "200-500 MiB win on H100" is negative
on our hardware).
- Drops max-model-len 218000 → 205000 because PN17 reserves ~120 MiB of KV
pool space at boot ("estimated maximum model length is 206400" is the
engine pre-check failure mode otherwise).
- P104 sidecar (patch_fa_max_seqlen_clamp.py) kept mounted but env-disabled;
PN17 covers the same path via Sandermage's anchored fix. Flip the env
var back on if PN17 turns out not to cover turboquant_attn.py for some
config.
- Mounts new patch_pn12_compile_safe_custom_op.py — opaque torch.library
custom_op for the inductor-compiled forward_native FFN path that the
eager-mode forward_cuda sidecar can't reach (issue #16, mech B). Only
active when GENESIS_ENABLE_PN12_FFN_INTERMEDIATE_POOL=1; harmless when
off. Forward_native body simplified to a single static-guard branch on
module-level _PN12_ENABLED so Dynamo specializes at trace time instead
of compiling both branches (the else-branch's plain F.silu/mul lowers
to empty_strided_cuda, defeating the patch).
verify-stress.sh: fixed silent false-positive where check_tool_prefill
returned 0 even after fail() because rm -f cleanup clobbered $? — now
captures rc before cleanup and propagates.
CLIFFS.md: cross-reference Sandermage's broader 8-cliff catalog and add
"vLLM pin compatibility status" section documenting the v0.20 workspace-
lock regression (PR #39226) that blocks our config and is unrelated to
Cliff 1/2.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CLIFFS.md: replace single-config narrative with shipped-configs table
(long-text 218K, long-vision 198K) plus the bisection table that
established each ceiling.
CHANGELOG.md: add bisection summary to the 2026-04-30 PM entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Initial 'PN12 is partial / architectural wall' framing was wrong. PN12
was silently no-op'd on dev205+ — same anchor-drift bug class as P101.
Once a local sidecar repairs the anchor, Cliff 1 closes at 205K with
verify-full + verify-stress passing and MTP n=3 active.
Sandermage's PN12 design intent was correct; we don't need a
gate_up_proj pool extension. The anchor fix is the missing piece.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #12 (P101 anchor drift fix) is now open on Sandermage's repo.
P104 stays held back pending Sandermage's response on issue #11 to
avoid piling on his in-progress mechanism-B / FFN-pool work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tested PN12 + PN13 + P101(anchor-fixed) + P103 + P104 + 50-block-override
on long-text 205K + 0.98 + TQ3 + no-vision. Result: Cliff 1 STILL
fires at 138 MiB / 130 MiB free, same FFN buffer signature.
PN12 only pools SiluAndMul.forward_cuda output (step 3 of FFN forward).
The OOM site is gate_proj or up_proj output (steps 1/2), each shape
[max_num_batched_tokens, intermediate_size] = 138 MiB, fresh-allocated
per layer per step. PN12 cuts allocator churn from 4× per layer to 3×;
significant but insufficient.
Holding off on commenting at issue #11 while Sandermage is actively
shipping. Will share data when he resurfaces. Local Genesis branches
(P101 anchor fix, P104) ready when needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add "architectural wall" section documenting empirical conclusion from
the Codex agent's 5-hour build session: P101 + P103 + P104 + 50-block-
override on long-text fail at the same FFN buffer (138 MiB / 130.5 MiB
free) regardless of max_model_len in [175K, 205K]. max_num_batched_tokens
is pinned at 4128 by Mamba block_size, sizing the FFN intermediate
buffer at 138 MiB constant. This is the architectural wall for
TQ3 + single-card + MTP at 24GB.
Document what an actual complete fix would require (chunked FFN forward,
drop MTP, FA3/FlashInfer/FlashQLA Ampere path, dual-card TP=2, llama.cpp).
Document explicitly what P104 + P101 anchor fix DO unblock (Genesis
community + future variant optionality, not our currently shipped configs).
Revise recommended path forward: ship P104 + P101 anchor fix as Genesis
PRs (community benefit), keep current shipped composes correct, route
users to dual-card or llama.cpp for genuine cliff-free long-context.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5-hour Codex agent session shipped two Genesis contributions on a local
branch (club-3090-cliff1-prep in the Genesis clone), waiting for upstream
PR submission. Empirical finding: TQ3+MTP+single-card hits an architectural
wall — Cliff 1 mechanism A (FA2 softmax_lse) closes via P104; mechanism B
(FFN intermediate buffer) is bounded by max_num_batched_tokens which is
pinned at 4128 by Mamba block_size.
P101 anchor drift discovery: P101 was silently no-op'd on dev205+ for
anyone enabling GENESIS_ENABLE_P101=1. apply_all reported "applied"
misleadingly. Fix updates anchor to match upstream torch.arange form.
P104 (new): env-gated FA max_seqlen_k runtime clamp, follows Genesis
text-patch infra, ~260 lines. PR-ready.
Empirical: 205K + 50-block-override + P101+P103+P104 still hits 138 MiB
FFN buffer with 130.5 MiB free. 175K + same stack: identical signature.
max_model_len is not the dominant variable.
No shipped config changes — default 48K + tools-text 75K stay correct.
Updates limited to documentation (CLIFFS.md, UPSTREAM.md, CHANGELOG.md).
Branch cliff1-fa-clamp NOT merged to master — waiting on user review +
upstream PR decision before merging.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tested Sandermage's existing P101 (TQ continuation 64-token slicing)
+ P103 (FLA Cliff 2 chunked fwd) — both opt-in, default-OFF, never
enabled by us. Hypothesis was they'd close Cliff 1 directly.
Empirical result: they don't fully close Cliff 1. P101 reroutes
around the FA2 softmax_lse mechanism (the one ChatGPT/DeepSeek
pointed at), but Cliff 1 has a SECOND mechanism — the FFN
intermediate buffer (max_num_batched_tokens × intermediate_size
= 4128 × 17408 × 2 bytes = 138 MiB per chunk). With vision tower's
~500 MiB pressure, the FFN buffer dominates and fires Cliff 1
even with P101 active.
Three test runs (all P101+P103 enabled):
- 192K + 0.98 + vision: FFN buffer OOM (138 MiB / 130 MiB free)
- 175K + 0.97 + vision: FFN buffer OOM (138 MiB / 110 MiB free)
- 205K + 0.98 + no-vision: FA2 softmax_lse OOM (50 MiB / 50 MiB)
So the dominant Cliff 1 mechanism depends on whether vision is on.
The proposed FA2 clamp at issue #11 still useful for the tools-text
/ long-text-no-vision case but wouldn't fully unlock long-vision —
FFN buffer is downstream and architectural.
Updates:
- CLIFFS.md: revised "Root cause" section with dual mechanism, table
of which fires under which configs, what each mitigation closes
- CHANGELOG: documented the discovery
- Posted cross-rig data to Sandermage/genesis-vllm-patches#11 (the
pleasant surprise — confirms his patches work for what they do,
identifies the second mechanism we missed)
No shipped config changes. Default 48K + tools-text 75K stay correct.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three corrections + clarifications from ChatGPT consultation on the
proposed Cliff 1 clamp:
1. Clamp formula correction: use min(attn_metadata.max_seq_len,
actual_max_seq_len_for_this_batch) — NOT chunk size. Chunk size
is the Q dimension; softmax_lse pads on the K dimension which
spans accumulated prompt. Clamping to chunk size would break
continuation prefill.
2. Specific guards documented: FA2/Ampere only, runtime-not-capture,
never below max(seqused_k). Each guard maps to a concrete failure
mode if violated.
3. Implementation shape: env-gated (GENESIS_FA2_CLAMP_MAX_SEQLEN=1),
diagnostic logging at the call site (num_actual_tokens,
max_query_len, attn_metadata.max_seq_len, seq_lens.max()), and
test progression starting at 86K (known-fail) before pushing
higher.
4. Added "Don't pursue --max-num-batched-tokens=2048 as primary fix"
to dead-ends — touches Q dimension; cap leak is on K.
5. Tightened recommended path forward — keep default capped at 48K
UNTIL clamp verified, then cautiously re-open 75K/86K/128K.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single canonical reference for everything we know about Cliff 1
(FA2 softmax_lse cap-leak) and Cliff 2 (fla.ops GDN forward
intermediate buffer): TL;DR table, empirical bisection with stack
traces, root-cause walk-through, why earlier "FFN intermediate
buffer" framing was wrong, why mem-util doesn't help, why PN8
closes Cliff 1 on tools-text but not on TQ3 paths, why llama.cpp
dodges both structurally, alternative attention backends with
feasibility, who-can-fix-it landscape (Sandermage, Tri Dao, fla-org,
QwenLM, us at any difficulty), recommended path forward, and
re-test triggers.
Cross-linked from FAQ.md and README.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
