v7.69 + #35975 + Codex P103 gate fix — Cliff 2 closure recipes
Six rounds of cross-rig bisect with Codex/ChatGPT diagnosis closes Cliff 2 at 60K on TP=1 + 24GB. Two shippable variants: Balanced MTP (long-text.yml updated): - Genesis v7.69 dev tip (commit 2db18df) - Codex r1 P103 gate fix applied to nested Genesis tree (cu_seqlens=[0,T] treated as dense single-seq, not multi-seq varlen). Sent to Sander as v7.70 proposal — diff in /tmp/cliff2_v770_cu_seqlens_response.md. - vllm#35975 backport sidecar (patch_inputs_embeds_optional.py) - mem-util 0.93 (down from 0.95) - max_model_len 180000 (admission ceiling at this mem-util) - MTP K=3 retained - 60K probe: HTTP 200 in 623s, recall correct, MTP AL=4.00 Max-context safety (long-text-no-mtp.yml — NEW): - Same patches, but MTP off + mem-util 0.95 - max_model_len 200000 (admission unlocked by removing MTP residency) - 60K probe: HTTP 200 in 537s, recall correct - 90K probe: indeterminate within 25-min curl budget - For long single-shot RAG / codebase analysis Diagnostic chain: 1. Codex r1 identified P103 gate as too broad (cu_seqlens != None bypasses chunking even for single-seq [0,T] case). Applied fix. 2. T=4128 distribution showed chunked path never engages on real serving (vLLM's outer chunked-prefill caps T well below MAX_T). 3. Codex r2: real Cliff 2 source is residency, not gate logic. 4. PR #35975 backport (skip inputs_embeds for text-only) frees 444 MiB at boot — necessary but not sufficient at 0.95 mem-util. 5. mem-util sweep at 0.92/0.93 with MTP+#35975 closed Cliff 2 at 60K. 6. MTP-off + 0.95 + 200K admission validated max-context variant. Codex's P103 gate fix is semantically correct and worth shipping in Genesis v7.70 even though it's not what closes 60K Cliff 2 on this config (the FLA call sees T=4128 already, well below MAX_T). Full diagnostic trail: results/v0.20-migration/v769-codex-r1-test.summary Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
noonghunna
@@ -0,0 +1,344 @@
|
|||||||
|
# ===========================================================================
|
||||||
|
# Long-text NO-MTP variant — max-context safety path (200K, MTP-off, no vision).
|
||||||
|
#
|
||||||
|
# This variant TRADES decode TPS for prefill stability. MTP K=3 spec-decode
|
||||||
|
# is OFF, freeing the draft-model residency budget for activation headroom
|
||||||
|
# at long single-shot prompts. Cliff 2 ceiling lifts past where it sits on
|
||||||
|
# the MTP-on `long-text.yml`; max_model_len declared at 200K (vs 180K on
|
||||||
|
# the MTP-on variant).
|
||||||
|
#
|
||||||
|
# Pick this variant for:
|
||||||
|
# - Long single-shot RAG queries (60K-100K context, 1 turn)
|
||||||
|
# - Whole-codebase analysis (TTFT-bounded workloads)
|
||||||
|
# - Documents Q&A where context >> response length
|
||||||
|
#
|
||||||
|
# DON'T pick this variant for:
|
||||||
|
# - Multi-turn chat, agentic coding loops — slower per-token decode
|
||||||
|
# hurts UX. Use `long-text.yml` (MTP-on, 180K) instead.
|
||||||
|
# - Tool calling — same Cliff 1 mech B caveats apply (use `tools-text.yml`)
|
||||||
|
#
|
||||||
|
# Stack: vLLM v0.20.1rc1.dev16 + Genesis v7.69 dev (commit 2db18df) +
|
||||||
|
# TurboQuant 3-bit KV + MTP DISABLED + vision tower disabled.
|
||||||
|
#
|
||||||
|
# vllm#35975 backport sidecar (`patch_inputs_embeds_optional.py`) is
|
||||||
|
# applied at boot — frees ~64 MiB GPU + 64 MiB pinned CPU on text-only
|
||||||
|
# models. Drops when PR merges upstream.
|
||||||
|
#
|
||||||
|
# Workloads that ARE safe on this compose:
|
||||||
|
# - Pure long-form text generation (essay, doc Q&A) — no tool schemas in prompt
|
||||||
|
# - RAG without tool calling (just context + question)
|
||||||
|
# - Chat without tools
|
||||||
|
# Workloads that should use `tools-text.yml` (75K + fp8 KV + PN8) instead:
|
||||||
|
# - Any IDE coding agent
|
||||||
|
# - Any flow that puts tool/function schemas in the system prompt
|
||||||
|
#
|
||||||
|
# Stack: vLLM v0.20.1rc1.dev16 + Genesis v7.65 dev (commit d89a089) +
|
||||||
|
# TurboQuant 3-bit KV + MTP n=3 + vision tower disabled. Migrated from
|
||||||
|
# vLLM dev205 + Genesis v7.64 on 2026-05-01 PM. v0.20's revised TQ FA paths
|
||||||
|
# close the synthetic 50K-token tool-prefill stress (which fired on dev205);
|
||||||
|
# real IDE-agent shapes still hit the inductor compile-path leak above.
|
||||||
|
#
|
||||||
|
# v7.65 key patches active:
|
||||||
|
# - P38B (Genesis #14 fix): compile-safe in-source hook for
|
||||||
|
# _continuation_prefill — replaces our local patch_pn12_compile_safe_
|
||||||
|
# custom_op.py
|
||||||
|
# - P15B (Genesis #15 fix): FA varlen max_seqlen_k clamp at TQ wrapper
|
||||||
|
# boundary — replaces our local patch_fa_max_seqlen_clamp.py
|
||||||
|
# - PN25: Inductor-safe silu_and_mul opaque op (covers the same surface
|
||||||
|
# as our former patch_pn12_compile_safe_custom_op.py)
|
||||||
|
# - PN26b: Genesis-original sparse-V Triton kernel for SM86 (Ampere
|
||||||
|
# consumer) — first SM86 sparse-V kernel in any public tree. 27B
|
||||||
|
# tuning: BLOCK_KV=8 num_warps=4 threshold=0.01.
|
||||||
|
# - Cliff 8 hardening: partial_apply_warnings counter in boot summary
|
||||||
|
# surfaces silent anchor-drift skips
|
||||||
|
#
|
||||||
|
# Recommended client max_tokens: 8192 (FREE thinking on this compose; budget
|
||||||
|
# for hard reasoning runs). 16384 for competition-grade problems. See
|
||||||
|
# docs/EXAMPLES.md "max_tokens defaults" section for the full table.
|
||||||
|
#
|
||||||
|
# When to pick this over long-vision:
|
||||||
|
# - You don't need image input
|
||||||
|
# - You want the maximum context on a single 3090
|
||||||
|
# - You're doing long-context text workloads (codebase analysis, RAG
|
||||||
|
# retrievals, multi-turn chat with steady-state accumulation)
|
||||||
|
#
|
||||||
|
# When NOT to pick this:
|
||||||
|
# - Anything that needs vision — use long-vision (198K + vision) instead
|
||||||
|
# - Single-shot prompts ≥50–60K tokens — Cliff 2 (DeltaNet GDN) still
|
||||||
|
# applies on single-card. Route those to dual.yml (TP=2 splits the
|
||||||
|
# state across cards) or llamacpp/default (262K, different engine).
|
||||||
|
#
|
||||||
|
# Local sidecar (single one remaining on this compose):
|
||||||
|
# - patch_workspace_lock_disable.py — relaxes vllm#39226 strict
|
||||||
|
# WorkspaceManager.lock() assertion to a one-shot WARNING. P98 in v7.65
|
||||||
|
# covers this functionally but its drift marker auto-skips on v0.20
|
||||||
|
# (`UNIFORM_SINGLE_TOKEN_DECODE` false-positive). Drop when Sandermage
|
||||||
|
# ships the marker fix.
|
||||||
|
#
|
||||||
|
# Variants in this dir:
|
||||||
|
# File Ctx KV Vision Notes
|
||||||
|
# docker-compose.yml (default) 48K TQ3 ✅ Production-safe; below both cliffs
|
||||||
|
# docker-compose.long-vision.yml 198K TQ3 ✅ Long ctx + vision (cliffs closed by v0.20 + v7.65)
|
||||||
|
# docker-compose.long-text.yml (this) 214K TQ3 ❌ Engine-ceiling text-only on v0.20 + 0.985 mem-util
|
||||||
|
# docker-compose.bounded-thinking.yml 214K TQ3 ❌ Long-text + structured-CoT grammar in <think>
|
||||||
|
# docker-compose.tools-text.yml 75K fp8 ❌ IDE agents — fp8 KV alternative
|
||||||
|
#
|
||||||
|
# Run with:
|
||||||
|
# cd <repo>/models/qwen3.6-27b/vllm/compose
|
||||||
|
# docker compose -f docker-compose.long-text.yml up -d
|
||||||
|
# ===========================================================================
|
||||||
|
services:
|
||||||
|
vllm-qwen36-27b-long-text-no-mtp:
|
||||||
|
image: vllm/vllm-openai:nightly-7a1eb8ac2ec4ea69338c51dc7afd4b15010abfa8
|
||||||
|
container_name: vllm-qwen36-27b-long-text-no-mtp
|
||||||
|
restart: "no"
|
||||||
|
ports:
|
||||||
|
- "${PORT:-8021}:8000"
|
||||||
|
volumes:
|
||||||
|
- ${MODEL_DIR:-../../../../models-cache}:/root/.cache/huggingface
|
||||||
|
# torch.compile + Triton kernel caches — first boot warms (~60-90 sec
|
||||||
|
# for Dynamo trace + Inductor compile + Triton JIT); subsequent boots
|
||||||
|
# reuse cached graphs and skip recompile. Pattern from Sandermage's PROD
|
||||||
|
# launch (start_27b_int4_TQ_k8v4.sh:31-32). Closes club-3090 #22.
|
||||||
|
- ../cache/torch_compile:/root/.cache/vllm/torch_compile_cache
|
||||||
|
- ../cache/triton:/root/.triton/cache
|
||||||
|
- ../patches/genesis/vllm/_genesis:/usr/local/lib/python3.12/dist-packages/vllm/_genesis:ro
|
||||||
|
- ../patches/patch_tolist_cudagraph.py:/patches/patch_tolist_cudagraph.py:ro
|
||||||
|
# workspace_lock_disable DROPPED on v7.69 — Genesis PN34 covers the
|
||||||
|
# boot-time AND runtime _decode_attention paths. Opt-in via
|
||||||
|
# GENESIS_ENABLE_PN34_WORKSPACE_LOCK_RELAX=1 in environment block.
|
||||||
|
# patch_inputs_embeds_optional.py — backport of vllm#35975 (~64 MiB GPU +
|
||||||
|
# 64 MiB pinned CPU savings on text-only models). Drops when PR merges.
|
||||||
|
- ../patches/patch_inputs_embeds_optional.py:/patches/patch_inputs_embeds_optional.py:ro
|
||||||
|
# NOTE: patch_pn12_ffn_pool_anchor.py + patch_pn12_compile_safe_custom_op.py
|
||||||
|
# + patch_fa_max_seqlen_clamp.py all DROPPED — Genesis v7.65 covers their
|
||||||
|
# surface natively (PN12 native + PN25 + PN17 + P15B). See migration
|
||||||
|
# commit history if you need to resurrect.
|
||||||
|
environment:
|
||||||
|
# Uncomment the next line to pin to a specific GPU (e.g. GPU 0):
|
||||||
|
# - CUDA_VISIBLE_DEVICES=0
|
||||||
|
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
|
||||||
|
- VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||||
|
- NCCL_CUMEM_ENABLE=0
|
||||||
|
- NCCL_P2P_DISABLE=1
|
||||||
|
# v0.20 made cudagraph memory profiling default-ON (vllm#38284), eats
|
||||||
|
# ~0.0055 mem-util ≈ 120 MiB KV pool. Disabling restores ~4K of ctx.
|
||||||
|
- VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0
|
||||||
|
- VLLM_NO_USAGE_STATS=1
|
||||||
|
- PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512
|
||||||
|
- VLLM_FLOAT32_MATMUL_PRECISION=high
|
||||||
|
- VLLM_USE_FLASHINFER_SAMPLER=1
|
||||||
|
- OMP_NUM_THREADS=1
|
||||||
|
- CUDA_DEVICE_MAX_CONNECTIONS=8
|
||||||
|
- VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
|
||||||
|
- VLLM_MARLIN_USE_ATOMIC_ADD=1
|
||||||
|
- TRITON_CACHE_DIR=/root/.triton/cache
|
||||||
|
# FULL Genesis v7.65 PROD env-var set per Sandermage's
|
||||||
|
# bare_metal_27b_int4_TQ_k8v4.sh. Validated 2026-05-01 PM dual-3090:
|
||||||
|
# +50% TPS over the prior subset (code 77 → 116 wall_TPS).
|
||||||
|
- GENESIS_ENABLE_P4=1
|
||||||
|
- GENESIS_ENABLE_P58_ASYNC_PLACEHOLDER_FIX=1
|
||||||
|
- GENESIS_ENABLE_P60_GDN_NGRAM_FIX=1
|
||||||
|
- GENESIS_ENABLE_P60B_TRITON_KERNEL=1
|
||||||
|
- GENESIS_ENABLE_P61_QWEN3_MULTI_TOOL=1
|
||||||
|
- GENESIS_ENABLE_P61B_STREAMING_OVERLAP=1
|
||||||
|
- GENESIS_ENABLE_P62_STRUCT_OUT_SPEC_TIMING=1
|
||||||
|
- GENESIS_ENABLE_P64_QWEN3CODER_MTP_STREAMING=1
|
||||||
|
- GENESIS_ENABLE_P65_TURBOQUANT_SPEC_CG_DOWNGRADE=1
|
||||||
|
- GENESIS_ENABLE_P66_CUDAGRAPH_SIZE_FILTER=1
|
||||||
|
- GENESIS_ENABLE_P67_TQ_MULTI_QUERY_KERNEL=1
|
||||||
|
# P68/P69 — v7.65 raised default threshold 8000 → 50000 chars (~12.5K
|
||||||
|
# tokens). Now safe with typical IDE-agent system+tool prompts; below
|
||||||
|
# the threshold both gates are no-op. Override via the env var if needed.
|
||||||
|
- GENESIS_ENABLE_P68_AUTO_FORCE_TOOL=1
|
||||||
|
- GENESIS_ENABLE_P69_LONG_CTX_TOOL_REMINDER=1
|
||||||
|
- GENESIS_P68_P69_LONG_CTX_THRESHOLD_CHARS=50000
|
||||||
|
- GENESIS_ENABLE_P72_PROFILE_RUN_CAP=1
|
||||||
|
- GENESIS_PROFILE_RUN_CAP_M=4128
|
||||||
|
- GENESIS_ENABLE_P74_CHUNK_CLAMP=1
|
||||||
|
- GENESIS_ENABLE_P83=1
|
||||||
|
- GENESIS_ENABLE_P85=1
|
||||||
|
- GENESIS_ENABLE_P87=1
|
||||||
|
- GENESIS_ENABLE_P91=1
|
||||||
|
- GENESIS_ENABLE_P94=1
|
||||||
|
- GENESIS_ENABLE_P98=1
|
||||||
|
- GENESIS_ENABLE_P99=1
|
||||||
|
- GENESIS_ENABLE_P100=1
|
||||||
|
- GENESIS_ENABLE_P101=1
|
||||||
|
# ─── Cliff 2 closure recipe (v7.69, single-24GB-GPU TP=1) ──────────
|
||||||
|
# P103 closes the FLA inner h tensor via chunk.py self-install hook
|
||||||
|
# (v7.69 F2 fix — survives `exec vllm serve` worker spawn).
|
||||||
|
# PN32 v2 closes the outer FLA call via _forward_core direct patch
|
||||||
|
# + initial_state threading (v7.69 F3 fix — composes with P103).
|
||||||
|
# Both required for Cliff 2 ceiling > 50K on TP=1.
|
||||||
|
- GENESIS_ENABLE_P103=1
|
||||||
|
- GENESIS_ENABLE_PN32_GDN_CHUNKED_PREFILL=1
|
||||||
|
- GENESIS_PN32_GDN_CHUNK_SIZE=8192
|
||||||
|
- GENESIS_PN32_GDN_CHUNK_THRESHOLD=16384
|
||||||
|
- GENESIS_FLA_FWD_H_MAX_T=16384
|
||||||
|
# PN34 (workspace_lock relaxation, opt-in default OFF). Replaces our
|
||||||
|
# patch_workspace_lock_disable.py sidecar.
|
||||||
|
- GENESIS_ENABLE_PN34_WORKSPACE_LOCK_RELAX=1
|
||||||
|
- GENESIS_ENABLE_PN8_MTP_DRAFT_ONLINE_QUANT=1
|
||||||
|
- GENESIS_ENABLE_PN9_INDEPENDENT_DRAFTER_ATTN=1
|
||||||
|
- GENESIS_ENABLE_PN11_GDN_AB_CONTIGUOUS=1
|
||||||
|
- GENESIS_ENABLE_PN12_FFN_INTERMEDIATE_POOL=1
|
||||||
|
- GENESIS_ENABLE_PN13_CUDA_GRAPH_LAMBDA_ARITY=1
|
||||||
|
- GENESIS_ENABLE_PN14_TQ_DECODE_OOB_CLAMP=1
|
||||||
|
- GENESIS_ENABLE_PN17_FA2_LSE_CLAMP=1
|
||||||
|
- GENESIS_ENABLE_PN25_SILU_INDUCTOR_SAFE=1
|
||||||
|
# PN30 — RE-ENABLED with our local dst-shaped temp fix
|
||||||
|
# (`patch_pn30_dst_shaped_temp_fix.py`, applied during setup.sh). The
|
||||||
|
# original Sander a9977d8 .contiguous() approach corrupted DS row strides
|
||||||
|
# by raw-memcpying a compact 10240×5 buffer into a 10240×6 destination
|
||||||
|
# block (row 1+ landed at the wrong destination offset, causing later
|
||||||
|
# TQ store CUDA assert). Our fix builds a destination-shaped temp inside
|
||||||
|
# `collect_mamba_copy_meta` and reuses PN30's temp-list lifetime handling.
|
||||||
|
# Diagnosis credit: ChatGPT/Codex CLI cross-check 2026-05-01 PM.
|
||||||
|
- GENESIS_ENABLE_PN30_DS_LAYOUT_SPEC_DECODE=1
|
||||||
|
# PN31 — DISABLED on 1×3090. Per-shape persistent VRAM grows as new
|
||||||
|
# shapes are seen during prefill; on TP=1 + 24GB, the residence cost
|
||||||
|
# outpaces the malloc-pressure relief. Sander explicitly warned in
|
||||||
|
# `753344b` he couldn't validate on 24GB; cross-rig validated 2026-05-01
|
||||||
|
# PM here that 0.95→0.93 mem-util can't carry both PN31 + DeltaNet
|
||||||
|
# `chunk_fwd_o` activation peak at 29K depth. Reported back on
|
||||||
|
# Sandermage/genesis-vllm-patches#15. Use tools-text.yml fp8 path for
|
||||||
|
# 25K+ tool-RETURN workloads in the meantime.
|
||||||
|
# - GENESIS_ENABLE_PN31_FA_VARLEN_PERSISTENT_OUT=1
|
||||||
|
- GENESIS_ENABLE_PN19_SCOPED_MAX_SPLIT=1
|
||||||
|
- GENESIS_ENABLE_PN22_LOCAL_ARGMAX_TP=1
|
||||||
|
# PN26b sparse-V Triton kernel — first SM86 (Ampere consumer) sparse-V
|
||||||
|
# kernel in any public tree. 27B-specific tuning per Sandermage's PROD.
|
||||||
|
- GENESIS_ENABLE_PN26_SPARSE_V=1
|
||||||
|
- GENESIS_PN26_SPARSE_V_BLOCK_KV=8
|
||||||
|
- GENESIS_PN26_SPARSE_V_NUM_WARPS=4
|
||||||
|
- GENESIS_PN26_SPARSE_V_THRESHOLD=0.01
|
||||||
|
# v7.65 #14 + #15 fixes (compile-safe in-source hook + FA varlen clamp).
|
||||||
|
# Replace our local patch_pn12_compile_safe_custom_op.py + P104 sidecars.
|
||||||
|
- GENESIS_ENABLE_P38B_COMPILE_SAFE=1
|
||||||
|
- GENESIS_ENABLE_P15B_FA_VARLEN_CLAMP=1
|
||||||
|
- GENESIS_PREALLOC_TOKEN_BUDGET=4128
|
||||||
|
- GENESIS_BUFFER_MODE=shared
|
||||||
|
# Explicit OFFs to match Sandermage's PROD env-var set verbatim:
|
||||||
|
# P78 (P78_TOLIST_CAPTURE_GUARD) — superseded by our patch_tolist_cudagraph.py
|
||||||
|
# P81 (FP8 block-scaled M<=8) — FP8-specific, no-op on our TQ3 path
|
||||||
|
# P82 — biased on small-batch single-stream Lorbus INT4 + MTP K=3 (Sander PROD)
|
||||||
|
- GENESIS_ENABLE_P78_TOLIST_CAPTURE_GUARD=0
|
||||||
|
- GENESIS_ENABLE_P81_FP8_BLOCK_SCALED_M_LE_8=0
|
||||||
|
- GENESIS_ENABLE_P82=0
|
||||||
|
- GENESIS_P82_THRESHOLD_SINGLE=0.3
|
||||||
|
# P82 stays OFF — biased on small-batch single-stream Lorbus INT4 + MTP K=3
|
||||||
|
# per Sandermage's PROD memory feedback_p82_*. P78 stays OFF (deprecated).
|
||||||
|
# vLLM env knobs from Sandermage's launch:
|
||||||
|
# VLLM_SSM_CONV_STATE_LAYOUT=DS — RE-ENABLED with our local PN30 fix
|
||||||
|
# (`patch_pn30_dst_shaped_temp_fix.py`, applied during setup.sh).
|
||||||
|
# Without our fix, Sander's PN30 a9977d8 corrupts DS row strides on
|
||||||
|
# spec-decode AL>1 paths. With our fix, PN30 builds a destination-shaped
|
||||||
|
# temp instead of a compact one, preserving DS layout. +6% TPS retained.
|
||||||
|
- VLLM_SSM_CONV_STATE_LAYOUT=DS
|
||||||
|
- VLLM_USE_FUSED_MOE_GROUPED_TOPK=1
|
||||||
|
shm_size: "16gb"
|
||||||
|
ipc: host
|
||||||
|
deploy:
|
||||||
|
resources:
|
||||||
|
reservations:
|
||||||
|
devices:
|
||||||
|
- driver: nvidia
|
||||||
|
count: all
|
||||||
|
capabilities: [gpu]
|
||||||
|
entrypoint:
|
||||||
|
- /bin/bash
|
||||||
|
- -c
|
||||||
|
- |
|
||||||
|
set -e
|
||||||
|
pip install xxhash pandas scipy -q
|
||||||
|
# Pre-flight: Genesis patches must be populated. Empty volume mount
|
||||||
|
# = silent no-op apply_all = boot fails later with cryptic upstream
|
||||||
|
# error (e.g. "TurboQuant KV not supported for hybrid models", #13).
|
||||||
|
if [ ! -f /usr/local/lib/python3.12/dist-packages/vllm/_genesis/patches/apply_all.py ]; then
|
||||||
|
echo "ERROR: Genesis patches missing — host volume models/qwen3.6-27b/vllm/patches/genesis/ is empty." >&2
|
||||||
|
echo " Run from repo root before 'docker compose up':" >&2
|
||||||
|
echo " bash scripts/setup.sh qwen3.6-27b" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
python3 -m vllm._genesis.patches.apply_all
|
||||||
|
# patch_tolist_cudagraph.py — still needed pending upstream
|
||||||
|
# vllm#40807 .tolist() capture-stream guard fix.
|
||||||
|
python3 /patches/patch_tolist_cudagraph.py
|
||||||
|
# patch_inputs_embeds_optional.py — backport of vllm#35975
|
||||||
|
# (free ~64 MiB GPU + ~64 MiB pinned CPU on text-only models).
|
||||||
|
# Drops when PR merges upstream.
|
||||||
|
python3 /patches/patch_inputs_embeds_optional.py
|
||||||
|
exec vllm serve "$@"
|
||||||
|
- --
|
||||||
|
command:
|
||||||
|
- --model
|
||||||
|
- /root/.cache/huggingface/qwen3.6-27b-autoround-int4
|
||||||
|
- --served-model-name
|
||||||
|
- qwen3.6-27b-autoround
|
||||||
|
- --quantization
|
||||||
|
- auto_round
|
||||||
|
- --dtype
|
||||||
|
- float16
|
||||||
|
- --tensor-parallel-size
|
||||||
|
- "1"
|
||||||
|
# 180K + 0.97 — backed off from 214K + 0.985 on 2026-05-01 PM after
|
||||||
|
# verify-stress probe 1 (10K-token long-context needle) crashed with
|
||||||
|
# GDN forward OOM (`fla.ops.chunk.chunk_gated_delta_rule_fwd_h`
|
||||||
|
# `k.new_empty(B, NT, H, V, K)` requested 98 MiB, 46 MiB free). At
|
||||||
|
# 0.985 mem-util, activation budget was ~46 MiB at 10K prefill —
|
||||||
|
# tight enough that ANY single prompt over a few thousand tokens was
|
||||||
|
# a coin flip. The "Cliff 2 fires only at 50-60K" framing was
|
||||||
|
# config-specific to the dev205 + 0.92 default; at tighter mem-util
|
||||||
|
# the GDN forward buffer outpaces available activation much earlier.
|
||||||
|
#
|
||||||
|
# New target (validated 2026-05-01 PM via verify-stress.sh):
|
||||||
|
# 180K + 0.97 → ~6.4 GB after weights, ~5.9 GB KV pool, ~500 MiB
|
||||||
|
# activation safety. Probe 1 (10K needle) and probe 6 (math reasoning
|
||||||
|
# at max_tokens=8192) PASS. Safe single-prompt envelope ~30K; full
|
||||||
|
# 180K context for steady-state turn accumulation only.
|
||||||
|
#
|
||||||
|
# IDE-agent workloads (sys + tool schemas) STILL crash here per
|
||||||
|
# club-3090#16 — different bug (Cliff 1 mech B inductor leak), not
|
||||||
|
# fixed by mem-util tuning. Use tools-text.yml for those.
|
||||||
|
- --max-model-len
|
||||||
|
- "200000"
|
||||||
|
# 0.95 — backed off from 0.97 on 2026-05-01 PM after PN25 v3 closed
|
||||||
|
# Cliff 1 mech B at the FFN intermediate buffer (which now sits in
|
||||||
|
# PN12+PN25's resident FFNIntermediateCache pool). Pool residence
|
||||||
|
# adds ~140 MiB persistent VRAM, which tightened activation budget
|
||||||
|
# at DeltaNet `chunk_fwd_o` (`o = torch.empty_like(v)`, 50 MiB) —
|
||||||
|
# 0.97 left only 26 MiB free, OOM at 9.8K probe. 0.95 frees ~480 MiB
|
||||||
|
# activation budget, validated up to 30K probes.
|
||||||
|
- --gpu-memory-utilization
|
||||||
|
- "0.95"
|
||||||
|
- --max-num-seqs
|
||||||
|
- "1"
|
||||||
|
- --max-num-batched-tokens
|
||||||
|
- "4128"
|
||||||
|
- --kv-cache-dtype
|
||||||
|
- turboquant_3bit_nc
|
||||||
|
# Drops the vision tower → frees ~1 GB VRAM → lifts engine ceiling from
|
||||||
|
# 198K (long-vision at 0.98) to 218K (this variant at 0.985). This is the only difference vs long-vision.
|
||||||
|
- --language-model-only
|
||||||
|
- --trust-remote-code
|
||||||
|
- --reasoning-parser
|
||||||
|
- qwen3
|
||||||
|
- --enable-auto-tool-choice
|
||||||
|
- --tool-call-parser
|
||||||
|
- qwen3_coder
|
||||||
|
- --enable-prefix-caching
|
||||||
|
- --enable-chunked-prefill
|
||||||
|
- --no-scheduler-reserve-full-isl
|
||||||
|
# MTP K=3 spec-decode INTENTIONALLY DISABLED on this variant. Frees
|
||||||
|
# ~280 MiB of draft-model residency on top of #35975's ~444 MiB savings,
|
||||||
|
# unlocking max_model_len=200K admission (vs 180K cap with MTP-on).
|
||||||
|
# Trade-off: lose ~3-4x decode TPS speedup vs `long-text.yml` (MTP-on).
|
||||||
|
# Pick this variant for long single-shot RAG / codebase prompts where
|
||||||
|
# TTFT-bounded throughput matters more than per-token decode speed.
|
||||||
|
# - --speculative-config
|
||||||
|
# - '{"method":"mtp","num_speculative_tokens":3}'
|
||||||
|
- --host
|
||||||
|
- 0.0.0.0
|
||||||
|
- --port
|
||||||
|
- "8000"
|
||||||
@@ -21,25 +21,46 @@
|
|||||||
# - Any IDE coding agent
|
# - Any IDE coding agent
|
||||||
# - Any flow that puts tool/function schemas in the system prompt
|
# - Any flow that puts tool/function schemas in the system prompt
|
||||||
#
|
#
|
||||||
# Stack: vLLM v0.20.1rc1.dev16 + Genesis v7.65 dev (commit d89a089) +
|
# Stack: vLLM v0.20.1rc1.dev16 + Genesis v7.69 dev (commit 2db18df) +
|
||||||
# TurboQuant 3-bit KV + MTP n=3 + vision tower disabled. Migrated from
|
# TurboQuant 3-bit KV + MTP n=3 + vision tower disabled.
|
||||||
# vLLM dev205 + Genesis v7.64 on 2026-05-01 PM. v0.20's revised TQ FA paths
|
|
||||||
# close the synthetic 50K-token tool-prefill stress (which fired on dev205);
|
|
||||||
# real IDE-agent shapes still hit the inductor compile-path leak above.
|
|
||||||
#
|
#
|
||||||
# v7.65 key patches active:
|
# v7.69 key patches active:
|
||||||
# - P38B (Genesis #14 fix): compile-safe in-source hook for
|
# - P103 (Cliff 2 chunked fwd_h+fwd_o orchestrator) — chunk.py self-install
|
||||||
# _continuation_prefill — replaces our local patch_pn12_compile_safe_
|
# hook survives `exec vllm serve` worker spawn (v7.69 F2 fix). Also
|
||||||
# custom_op.py
|
# applied: Codex r1 cu_seqlens=[0,T] dense-single-seq gate fix that
|
||||||
# - P15B (Genesis #15 fix): FA varlen max_seqlen_k clamp at TQ wrapper
|
# allows the chunked path to engage on real serving.
|
||||||
# boundary — replaces our local patch_fa_max_seqlen_clamp.py
|
# - PN30 v7.68 part1+2+3 — DS conv state layout + spec-decode AL>1 fix
|
||||||
# - PN25: Inductor-safe silu_and_mul opaque op (covers the same surface
|
# (drift-marker bug fixed in v7.69)
|
||||||
# as our former patch_pn12_compile_safe_custom_op.py)
|
# - PN32 v2 — GDN _forward_core chunked-prefill (Cliff 2 outer split)
|
||||||
|
# - PN34 — workspace_lock relaxation (replaces our former
|
||||||
|
# patch_workspace_lock_disable.py sidecar)
|
||||||
# - PN26b: Genesis-original sparse-V Triton kernel for SM86 (Ampere
|
# - PN26b: Genesis-original sparse-V Triton kernel for SM86 (Ampere
|
||||||
# consumer) — first SM86 sparse-V kernel in any public tree. 27B
|
# consumer) — first SM86 sparse-V kernel in any public tree. 27B
|
||||||
# tuning: BLOCK_KV=8 num_warps=4 threshold=0.01.
|
# tuning: BLOCK_KV=8 num_warps=4 threshold=0.01.
|
||||||
# - Cliff 8 hardening: partial_apply_warnings counter in boot summary
|
# - P38B / P15B: Cliff 1 mech B compile-safe hook + FA varlen clamp
|
||||||
# surfaces silent anchor-drift skips
|
#
|
||||||
|
# Cliff 2 closure recipe (cross-rig validated 2026-05-02 PM):
|
||||||
|
# - vllm#35975 backport sidecar (`patch_inputs_embeds_optional.py`) frees
|
||||||
|
# ~444 MiB at boot on text-only models (skips inputs_embeds buffers
|
||||||
|
# for both gpu_model_runner.py + llm_base_proposer.py).
|
||||||
|
# - --gpu-memory-utilization 0.93 — middle ground between 0.92 (cliff
|
||||||
|
# just barely closed) and 0.95 (cliff fires). Recovers ~30K of KV
|
||||||
|
# pool concurrency vs 0.92 while keeping ~494 MiB margin at peak.
|
||||||
|
# - --max-model-len 180000 — engine pre-check admission ceiling on
|
||||||
|
# this config (200K rejected: KV pool needs 3.31 GiB, only 2.19 GiB
|
||||||
|
# available at 0.93). Cliff 2 single-prompt safe envelope: ~60K
|
||||||
|
# verified pass; ~70-80K is the OOM ceiling.
|
||||||
|
#
|
||||||
|
# Validation (2026-05-02 PM, branch v7.69-cliff2-test):
|
||||||
|
# - 60K MTP-on probe: HTTP 200 in 623s, recall correct, MTP AL=4.00
|
||||||
|
# - 90K MTP-on probe: HTTP 500 OOM at 112s (50 MiB / 24.5 MiB free)
|
||||||
|
# - Probes 2-6 (tool prefill, IDE-agent, multi-turn, LCB-coding,
|
||||||
|
# reasoning-heavy): all PASS
|
||||||
|
#
|
||||||
|
# For workloads that need >60K single-prompt or 200K declared context,
|
||||||
|
# use docker-compose.long-text-no-mtp.yml (max-context safety variant
|
||||||
|
# trades MTP decode speedup for prefill stability). For coding agents
|
||||||
|
# on dual-card, use docker-compose.dual-turbo.yml.
|
||||||
#
|
#
|
||||||
# Recommended client max_tokens: 8192 (FREE thinking on this compose; budget
|
# Recommended client max_tokens: 8192 (FREE thinking on this compose; budget
|
||||||
# for hard reasoning runs). 16384 for competition-grade problems. See
|
# for hard reasoning runs). 16384 for competition-grade problems. See
|
||||||
@@ -93,10 +114,12 @@ services:
|
|||||||
- ../cache/triton:/root/.triton/cache
|
- ../cache/triton:/root/.triton/cache
|
||||||
- ../patches/genesis/vllm/_genesis:/usr/local/lib/python3.12/dist-packages/vllm/_genesis:ro
|
- ../patches/genesis/vllm/_genesis:/usr/local/lib/python3.12/dist-packages/vllm/_genesis:ro
|
||||||
- ../patches/patch_tolist_cudagraph.py:/patches/patch_tolist_cudagraph.py:ro
|
- ../patches/patch_tolist_cudagraph.py:/patches/patch_tolist_cudagraph.py:ro
|
||||||
# workspace_lock_disable — relaxes vllm#39226 strict assertion for the TQ
|
# workspace_lock_disable DROPPED on v7.69 — Genesis PN34 covers the
|
||||||
# decode path on v0.20. Drop when Sandermage ships the P98 marker fix
|
# boot-time AND runtime _decode_attention paths. Opt-in via
|
||||||
# that auto-enables on v0.20 without our help. See docs/CLIFFS.md.
|
# GENESIS_ENABLE_PN34_WORKSPACE_LOCK_RELAX=1 in environment block.
|
||||||
- ../patches/patch_workspace_lock_disable.py:/patches/patch_workspace_lock_disable.py:ro
|
# patch_inputs_embeds_optional.py — backport of vllm#35975 (~64 MiB GPU +
|
||||||
|
# 64 MiB pinned CPU savings on text-only models). Drops when PR merges.
|
||||||
|
- ../patches/patch_inputs_embeds_optional.py:/patches/patch_inputs_embeds_optional.py:ro
|
||||||
# NOTE: patch_pn12_ffn_pool_anchor.py + patch_pn12_compile_safe_custom_op.py
|
# NOTE: patch_pn12_ffn_pool_anchor.py + patch_pn12_compile_safe_custom_op.py
|
||||||
# + patch_fa_max_seqlen_clamp.py all DROPPED — Genesis v7.65 covers their
|
# + patch_fa_max_seqlen_clamp.py all DROPPED — Genesis v7.65 covers their
|
||||||
# surface natively (PN12 native + PN25 + PN17 + P15B). See migration
|
# surface natively (PN12 native + PN25 + PN17 + P15B). See migration
|
||||||
@@ -152,7 +175,20 @@ services:
|
|||||||
- GENESIS_ENABLE_P99=1
|
- GENESIS_ENABLE_P99=1
|
||||||
- GENESIS_ENABLE_P100=1
|
- GENESIS_ENABLE_P100=1
|
||||||
- GENESIS_ENABLE_P101=1
|
- GENESIS_ENABLE_P101=1
|
||||||
|
# ─── Cliff 2 closure recipe (v7.69, single-24GB-GPU TP=1) ──────────
|
||||||
|
# P103 closes the FLA inner h tensor via chunk.py self-install hook
|
||||||
|
# (v7.69 F2 fix — survives `exec vllm serve` worker spawn).
|
||||||
|
# PN32 v2 closes the outer FLA call via _forward_core direct patch
|
||||||
|
# + initial_state threading (v7.69 F3 fix — composes with P103).
|
||||||
|
# Both required for Cliff 2 ceiling > 50K on TP=1.
|
||||||
- GENESIS_ENABLE_P103=1
|
- GENESIS_ENABLE_P103=1
|
||||||
|
- GENESIS_ENABLE_PN32_GDN_CHUNKED_PREFILL=1
|
||||||
|
- GENESIS_PN32_GDN_CHUNK_SIZE=8192
|
||||||
|
- GENESIS_PN32_GDN_CHUNK_THRESHOLD=16384
|
||||||
|
- GENESIS_FLA_FWD_H_MAX_T=16384
|
||||||
|
# PN34 (workspace_lock relaxation, opt-in default OFF). Replaces our
|
||||||
|
# patch_workspace_lock_disable.py sidecar.
|
||||||
|
- GENESIS_ENABLE_PN34_WORKSPACE_LOCK_RELAX=1
|
||||||
- GENESIS_ENABLE_PN8_MTP_DRAFT_ONLINE_QUANT=1
|
- GENESIS_ENABLE_PN8_MTP_DRAFT_ONLINE_QUANT=1
|
||||||
- GENESIS_ENABLE_PN9_INDEPENDENT_DRAFTER_ATTN=1
|
- GENESIS_ENABLE_PN9_INDEPENDENT_DRAFTER_ATTN=1
|
||||||
- GENESIS_ENABLE_PN11_GDN_AB_CONTIGUOUS=1
|
- GENESIS_ENABLE_PN11_GDN_AB_CONTIGUOUS=1
|
||||||
@@ -236,10 +272,13 @@ services:
|
|||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
python3 -m vllm._genesis.patches.apply_all
|
python3 -m vllm._genesis.patches.apply_all
|
||||||
# workspace_lock_disable — relaxes vllm#39226 strict assertion until
|
# patch_tolist_cudagraph.py — still needed pending upstream
|
||||||
# Sandermage's P98 marker fix lands.
|
# vllm#40807 .tolist() capture-stream guard fix.
|
||||||
python3 /patches/patch_workspace_lock_disable.py
|
|
||||||
python3 /patches/patch_tolist_cudagraph.py
|
python3 /patches/patch_tolist_cudagraph.py
|
||||||
|
# patch_inputs_embeds_optional.py — backport of vllm#35975
|
||||||
|
# (free ~64 MiB GPU + ~64 MiB pinned CPU on text-only models).
|
||||||
|
# Drops when PR merges upstream.
|
||||||
|
python3 /patches/patch_inputs_embeds_optional.py
|
||||||
exec vllm serve "$@"
|
exec vllm serve "$@"
|
||||||
- --
|
- --
|
||||||
command:
|
command:
|
||||||
@@ -282,7 +321,7 @@ services:
|
|||||||
# 0.97 left only 26 MiB free, OOM at 9.8K probe. 0.95 frees ~480 MiB
|
# 0.97 left only 26 MiB free, OOM at 9.8K probe. 0.95 frees ~480 MiB
|
||||||
# activation budget, validated up to 30K probes.
|
# activation budget, validated up to 30K probes.
|
||||||
- --gpu-memory-utilization
|
- --gpu-memory-utilization
|
||||||
- "0.95"
|
- "0.93"
|
||||||
- --max-num-seqs
|
- --max-num-seqs
|
||||||
- "1"
|
- "1"
|
||||||
- --max-num-batched-tokens
|
- --max-num-batched-tokens
|
||||||
|
|||||||
180
models/qwen3.6-27b/vllm/patches/patch_inputs_embeds_optional.py
Normal file
180
models/qwen3.6-27b/vllm/patches/patch_inputs_embeds_optional.py
Normal file
@@ -0,0 +1,180 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
patch_inputs_embeds_optional.py — backport of vllm-project/vllm#35975
|
||||||
|
|
||||||
|
Skips inputs_embeds GPU buffer allocation for text-only models. Saves
|
||||||
|
~64 MiB GPU + ~64 MiB pinned CPU on a config that doesn't use multimodal
|
||||||
|
inputs or prompt embeddings.
|
||||||
|
|
||||||
|
Why we ship it locally:
|
||||||
|
PR #35975 is open upstream as of 2026-05-02 (awaiting code-owner approval
|
||||||
|
after author addressed reviewer feedback). On club-3090's TP=1 + 24GB
|
||||||
|
Qwen3.6-27B + MTP K=3 + 0.95 mem-util config, Cliff 2 at 60K fires with
|
||||||
|
~24.5 MiB free at the time of the failing 50 MiB allocation. PR #35975
|
||||||
|
frees ~64 MiB GPU + ~64 MiB pinned CPU on text-only models — sufficient
|
||||||
|
margin to potentially close Cliff 2 at 60K MTP-on without reducing
|
||||||
|
context or mem-util.
|
||||||
|
|
||||||
|
This sidecar exists because we don't want to fork the docker image just
|
||||||
|
for one PR. When PR #35975 merges upstream and ships in our nightly tag,
|
||||||
|
this sidecar can be deleted.
|
||||||
|
|
||||||
|
References:
|
||||||
|
- https://github.com/vllm-project/vllm/pull/35975
|
||||||
|
- club-3090 results/v0.20-migration/v769-codex-r1-test.summary
|
||||||
|
|
||||||
|
Idempotent. Safe to re-run.
|
||||||
|
"""
|
||||||
|
import logging
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
log = logging.getLogger("inputs_embeds_optional")
|
||||||
|
logging.basicConfig(level=logging.INFO, format="[%(name)s] %(message)s")
|
||||||
|
|
||||||
|
PATCH_TAG = "[inputs_embeds_optional]"
|
||||||
|
|
||||||
|
|
||||||
|
def patch_gpu_model_runner() -> bool:
|
||||||
|
"""
|
||||||
|
File: vllm/v1/worker/gpu_model_runner.py
|
||||||
|
|
||||||
|
Before:
|
||||||
|
self.inputs_embeds = self._make_buffer(
|
||||||
|
self.max_num_tokens, self.inputs_embeds_size, dtype=self.dtype, numpy=False
|
||||||
|
)
|
||||||
|
|
||||||
|
After:
|
||||||
|
# vllm#35975: skip inputs_embeds buffer for text-only models.
|
||||||
|
self.inputs_embeds = None
|
||||||
|
if self.supports_mm_inputs or self.enable_prompt_embeds:
|
||||||
|
self.inputs_embeds = self._make_buffer(
|
||||||
|
self.max_num_tokens, self.inputs_embeds_size, dtype=self.dtype, numpy=False
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
target = Path(
|
||||||
|
"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py"
|
||||||
|
)
|
||||||
|
if not target.exists():
|
||||||
|
log.error(f"{PATCH_TAG} target not found: {target}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
src = target.read_text()
|
||||||
|
if PATCH_TAG in src:
|
||||||
|
log.info(f"{PATCH_TAG} gpu_model_runner.py already patched, skipping.")
|
||||||
|
return True
|
||||||
|
|
||||||
|
# Find the exact allocation site. Use a regex that matches multi-line.
|
||||||
|
# Keep the `numpy=False` argument literal.
|
||||||
|
pattern = re.compile(
|
||||||
|
r"^( *)self\.inputs_embeds = self\._make_buffer\(\n"
|
||||||
|
r"\1 self\.max_num_tokens, self\.inputs_embeds_size, dtype=self\.dtype, numpy=False\n"
|
||||||
|
r"\1\)$",
|
||||||
|
re.MULTILINE,
|
||||||
|
)
|
||||||
|
match = pattern.search(src)
|
||||||
|
if not match:
|
||||||
|
log.error(
|
||||||
|
f"{PATCH_TAG} gpu_model_runner.py: anchor not found "
|
||||||
|
f"(self.inputs_embeds = self._make_buffer block). vLLM may have "
|
||||||
|
f"changed; review PR #35975 manually."
|
||||||
|
)
|
||||||
|
return False
|
||||||
|
|
||||||
|
indent = match.group(1)
|
||||||
|
replacement = (
|
||||||
|
f"{indent}# {PATCH_TAG} vllm#35975: skip inputs_embeds buffer for text-only models.\n"
|
||||||
|
f"{indent}self.inputs_embeds = None\n"
|
||||||
|
f"{indent}if self.supports_mm_inputs or self.enable_prompt_embeds:\n"
|
||||||
|
f"{indent} self.inputs_embeds = self._make_buffer(\n"
|
||||||
|
f"{indent} self.max_num_tokens, self.inputs_embeds_size, dtype=self.dtype, numpy=False\n"
|
||||||
|
f"{indent} )"
|
||||||
|
)
|
||||||
|
new_src = pattern.sub(replacement, src, count=1)
|
||||||
|
target.write_text(new_src)
|
||||||
|
log.info(f"{PATCH_TAG} gpu_model_runner.py: applied")
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def patch_llm_base_proposer() -> bool:
|
||||||
|
"""
|
||||||
|
File: vllm/v1/spec_decode/llm_base_proposer.py
|
||||||
|
(PR #35975 originally targeted eagle.py; in this nightly the code lives
|
||||||
|
in the shared base class llm_base_proposer.py.)
|
||||||
|
|
||||||
|
Before:
|
||||||
|
self.inputs_embeds = torch.zeros(
|
||||||
|
(self.max_num_tokens, self.inputs_embeds_size),
|
||||||
|
dtype=self.dtype,
|
||||||
|
device=device,
|
||||||
|
)
|
||||||
|
|
||||||
|
After:
|
||||||
|
# vllm#35975: skip inputs_embeds tensor for text-only proposers.
|
||||||
|
self.inputs_embeds = None
|
||||||
|
if self.supports_mm_inputs:
|
||||||
|
self.inputs_embeds = torch.zeros(
|
||||||
|
(self.max_num_tokens, self.inputs_embeds_size),
|
||||||
|
dtype=self.dtype,
|
||||||
|
device=device,
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
target = Path(
|
||||||
|
"/usr/local/lib/python3.12/dist-packages/vllm/v1/spec_decode/llm_base_proposer.py"
|
||||||
|
)
|
||||||
|
if not target.exists():
|
||||||
|
log.error(f"{PATCH_TAG} target not found: {target}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
src = target.read_text()
|
||||||
|
if PATCH_TAG in src:
|
||||||
|
log.info(f"{PATCH_TAG} llm_base_proposer.py already patched, skipping.")
|
||||||
|
return True
|
||||||
|
|
||||||
|
pattern = re.compile(
|
||||||
|
r"^( *)self\.inputs_embeds = torch\.zeros\(\n"
|
||||||
|
r"\1 \(self\.max_num_tokens, self\.inputs_embeds_size\),\n"
|
||||||
|
r"\1 dtype=self\.dtype,\n"
|
||||||
|
r"\1 device=device,\n"
|
||||||
|
r"\1\)$",
|
||||||
|
re.MULTILINE,
|
||||||
|
)
|
||||||
|
match = pattern.search(src)
|
||||||
|
if not match:
|
||||||
|
log.error(
|
||||||
|
f"{PATCH_TAG} llm_base_proposer.py: anchor not found "
|
||||||
|
f"(self.inputs_embeds = torch.zeros block). vLLM may have changed."
|
||||||
|
)
|
||||||
|
return False
|
||||||
|
|
||||||
|
indent = match.group(1)
|
||||||
|
replacement = (
|
||||||
|
f"{indent}# {PATCH_TAG} vllm#35975: skip inputs_embeds tensor for text-only proposers.\n"
|
||||||
|
f"{indent}self.inputs_embeds = None\n"
|
||||||
|
f"{indent}if self.supports_mm_inputs:\n"
|
||||||
|
f"{indent} self.inputs_embeds = torch.zeros(\n"
|
||||||
|
f"{indent} (self.max_num_tokens, self.inputs_embeds_size),\n"
|
||||||
|
f"{indent} dtype=self.dtype,\n"
|
||||||
|
f"{indent} device=device,\n"
|
||||||
|
f"{indent} )"
|
||||||
|
)
|
||||||
|
new_src = pattern.sub(replacement, src, count=1)
|
||||||
|
target.write_text(new_src)
|
||||||
|
log.info(f"{PATCH_TAG} llm_base_proposer.py: applied")
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
ok = True
|
||||||
|
ok &= patch_gpu_model_runner()
|
||||||
|
ok &= patch_llm_base_proposer()
|
||||||
|
if ok:
|
||||||
|
log.info(f"{PATCH_TAG} all patches applied successfully")
|
||||||
|
return 0
|
||||||
|
log.error(f"{PATCH_TAG} one or more patches failed")
|
||||||
|
return 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
404
results/v0.20-migration/v769-codex-r1-test.summary
Normal file
404
results/v0.20-migration/v769-codex-r1-test.summary
Normal file
@@ -0,0 +1,404 @@
|
|||||||
|
# v7.69 + Codex round 1 P103 gate fix — verification on TP=1 + 24GB
|
||||||
|
Date: 2026-05-02 PM
|
||||||
|
Genesis pin: 2db18df (v7.69 dev tip) + Codex's gate-fix patch applied to in-tree
|
||||||
|
P103: patched to treat cu_seqlens=[0,T] as dense single-seq (not multi-seq varlen)
|
||||||
|
PN32 v2: enabled
|
||||||
|
PN34: enabled (workspace_lock relax)
|
||||||
|
DS layout: enabled
|
||||||
|
GENESIS_FLA_FWD_H_MAX_T: tested at both 16384 (default) and 2048
|
||||||
|
|
||||||
|
## Ground truth from diagnostic log
|
||||||
|
|
||||||
|
Added `print` at the gate to capture T, q.shape, cu_shape, _single_seq_cu, _true_varlen_multi_seq.
|
||||||
|
|
||||||
|
### Run 1: GENESIS_FLA_FWD_H_MAX_T=16384 (Codex default)
|
||||||
|
|
||||||
|
T distribution across 442 invocations on 60K probe:
|
||||||
|
T=4128: 394 invocations (vLLM chunked-prefill chunk size)
|
||||||
|
T=64: 48 invocations (cudagraph warmup or MTP verify)
|
||||||
|
T>4128: 0 invocations
|
||||||
|
|
||||||
|
q.shape[0] = 1 always (B=1, single sequence)
|
||||||
|
cu_shape = torch.Size([2]) always (single segment, [0, T])
|
||||||
|
_single_seq_cu = True for every invocation
|
||||||
|
_true_varlen_multi_seq = False for every invocation
|
||||||
|
|
||||||
|
Gate verdict: gate-fix is semantically correct but `q.shape[1] <= _MAX_T`
|
||||||
|
(4128 <= 16384) bypasses to original_fwd on EVERY call. Chunked path
|
||||||
|
never engages.
|
||||||
|
|
||||||
|
OOM: same trace as v7.69 baseline. 50.00 MiB alloc fails, 24.50 MiB free.
|
||||||
|
|
||||||
|
### Run 2: GENESIS_FLA_FWD_H_MAX_T=2048 (force chunking on T=4128 calls)
|
||||||
|
|
||||||
|
T distribution across 436 invocations:
|
||||||
|
T=4128: 340 invocations (now ENTERING chunked path: 4128 > 2048)
|
||||||
|
T=64: 48 invocations
|
||||||
|
T=11: 48 invocations (smaller residuals on chunked path)
|
||||||
|
|
||||||
|
Per-call allocation: 24.00 MiB (half of previous 50.00 MiB)
|
||||||
|
Cumulative active PyTorch: 23.01 GiB (vs 22.96 GiB on run 1)
|
||||||
|
OOM trigger: failed at call ~336/340 (vs ~393/394 on run 1)
|
||||||
|
|
||||||
|
Net change: chunking DID reduce per-call peak by half, but cumulative
|
||||||
|
state across chunked sub-calls grew slightly (chained state buffers
|
||||||
|
between chunks). OOM fires earlier in absolute call count.
|
||||||
|
|
||||||
|
## Cumulative diagnosis
|
||||||
|
|
||||||
|
The gate-fix Codex proposed is correct in principle:
|
||||||
|
cu_seqlens.shape == (2,) IS dense single-seq, not multi-seq varlen.
|
||||||
|
P103 v7.70 should NOT bypass on cu_seqlens != None when shape == (2,).
|
||||||
|
|
||||||
|
But on our TP=1 + 24GB + MTP K=3 + 0.95 mem-util config, the chunked
|
||||||
|
path:
|
||||||
|
(a) doesn't engage at MAX_T=16384 because vLLM's outer chunked
|
||||||
|
prefill already caps T at max_num_batched_tokens=4128
|
||||||
|
(b) does engage at MAX_T=2048 but doesn't help because the OOM
|
||||||
|
isn't a single-allocation-too-large problem — it's
|
||||||
|
cumulative-state-too-large
|
||||||
|
|
||||||
|
Current allocation breakdown at OOM:
|
||||||
|
Total in use: 23.55 GiB / 23.56 GiB (99.9%)
|
||||||
|
PyTorch active: 23.01 GiB
|
||||||
|
Of which:
|
||||||
|
Model weights: 16.87 GiB (boot log)
|
||||||
|
MTP draft sharing: 0 GiB (shared via embedding+lm_head)
|
||||||
|
KV pool @ 60K TQ3: ~1 GiB (rough estimate, ~17 KB/tok)
|
||||||
|
Mamba conv state: ~0.5 GiB (DS layout + GDN forward)
|
||||||
|
PN12/PN25 pools: ~0.2 GiB
|
||||||
|
FLA temp buffers: ~0.2 GiB
|
||||||
|
cudagraph private: 0.024 GiB
|
||||||
|
Other PyTorch alloc: ~4 GiB (UNACCOUNTED — likely activations)
|
||||||
|
|
||||||
|
The 4 GiB "unaccounted" is the actual headroom problem. Each prefill
|
||||||
|
chunk's activations stick around in some lifetime overlap until the
|
||||||
|
next chunk's allocation, fragmenting expandable-segments effective
|
||||||
|
headroom.
|
||||||
|
|
||||||
|
## Conclusions
|
||||||
|
|
||||||
|
1. Codex's gate-fix should still ship to Sander as v7.70 — it's
|
||||||
|
semantically correct and unblocks the synthetic single-seq path.
|
||||||
|
2. But it does NOT close real-config Cliff 2 at 60K on 24GB.
|
||||||
|
3. The actual fix is one of:
|
||||||
|
(a) Lower mem-util to ~0.90 (frees ~700 MiB headroom, costs ~10K ctx)
|
||||||
|
(b) Cap max-model-len to 50K and route 50K+ to TP=2 (`dual.yml`)
|
||||||
|
(c) Genesis-side: tighter activation residency in PN12/PN25/PN30
|
||||||
|
pools + audit FLA temp buffer reuse
|
||||||
|
(d) vLLM-side: scheduler pause prefill chunks when VRAM headroom
|
||||||
|
drops below threshold
|
||||||
|
|
||||||
|
Practical recommendation for users on TP=1 + 24GB:
|
||||||
|
Cliff 2 ceiling stays ~50-60K on this config regardless of v7.69
|
||||||
|
patches. Treat it as architectural + use dual.yml for longer prompts.
|
||||||
|
|
||||||
|
## Round 2 prompt for Codex queued at /tmp/codex_prompt_cliff2_round2.md
|
||||||
|
|
||||||
|
Asks ChatGPT/Codex to:
|
||||||
|
- Diagnose where the 4 GiB unaccounted active PyTorch allocation lives
|
||||||
|
- Recommend instrumentation to bisect (memory_summary per call?)
|
||||||
|
- Propose either Genesis-side or vLLM-side fix that doesn't require
|
||||||
|
per-call chunking (since chunking doesn't help here)
|
||||||
|
|
||||||
|
## State of in-tree files
|
||||||
|
|
||||||
|
- `models/qwen3.6-27b/vllm/patches/genesis/vllm/_genesis/wiring/hybrid/patch_103_fla_cliff2_chunked.py`
|
||||||
|
has Codex's gate-fix applied + temp diagnostic log. Will revert
|
||||||
|
diagnostic log before any commit.
|
||||||
|
- `models/qwen3.6-27b/vllm/compose/docker-compose.long-text.yml`
|
||||||
|
has GENESIS_FLA_FWD_H_MAX_T=2048 (testing value). Needs revert to
|
||||||
|
16384 (or whatever final value) before commit.
|
||||||
|
|
||||||
|
|
||||||
|
## ROUND 2 — Codex r2 + user PR research bisect (2026-05-02 PM)
|
||||||
|
|
||||||
|
### Boot resident snapshot
|
||||||
|
|
||||||
|
MTP-on : 23,156 MiB (model weights + MTP draft 0.28 GiB shared + KV preallocation + Genesis pools)
|
||||||
|
MTP-off : 23,164 MiB (same minus MTP, ~280 MiB savings — MTP draft shares embedding+lm_head)
|
||||||
|
|
||||||
|
Headroom at 0.95 mem-util ≈ 1,400 MiB before any request.
|
||||||
|
|
||||||
|
### MTP-off A/B at 60K — single-variable change
|
||||||
|
|
||||||
|
Removed only `--speculative-config '{"method":"mtp",...}'`. All else
|
||||||
|
unchanged: P103 (with Codex gate fix) + PN30 + PN32 v2 + PN34 + DS layout
|
||||||
|
+ MAX_T=16384 + max_num_batched_tokens=4128 + max-model-len=180000 +
|
||||||
|
mem-util=0.95.
|
||||||
|
|
||||||
|
Manual curl with 600s timeout:
|
||||||
|
HTTP 200 ✅ recalled "violet otter 48" correctly
|
||||||
|
Wall: 504s
|
||||||
|
Engine logs: prompt throughput 227 tok/s, gen 7.5 tok/s, KV usage 6.1%
|
||||||
|
No OOM, no errors
|
||||||
|
|
||||||
|
verify-stress harness curl (5 min default timeout):
|
||||||
|
HTTP 000 (harness timed out before engine completed)
|
||||||
|
Engine itself returned 200 a few seconds after harness gave up — both
|
||||||
|
the harness curl AND the manual curl saw the same engine completion;
|
||||||
|
the harness's 5-min cap is the fail signal, not engine correctness.
|
||||||
|
|
||||||
|
**Conclusion: MTP-off closes the 60K Cliff 2 cleanly.** Cliff 2 on
|
||||||
|
TP1 + 24GB + 0.95 mem-util is dominated by MTP draft residency +
|
||||||
|
activation footprint, not P103 cu_seqlens gate.
|
||||||
|
|
||||||
|
### MTP-off at 90K — indeterminate within 15 min
|
||||||
|
|
||||||
|
Manual curl with 900s (15 min) timeout:
|
||||||
|
HTTP 000 (curl timed out)
|
||||||
|
Wall 900s
|
||||||
|
No OOM trace
|
||||||
|
GPU 0 still at 100% utilization
|
||||||
|
No POST 200 in engine logs after 13:42:37
|
||||||
|
Engine never completed within practical wait
|
||||||
|
|
||||||
|
Practical interpretation: 90K MTP-off is past the wall-clock-feasible
|
||||||
|
ceiling on this config even without OOM. Throughput drops as
|
||||||
|
cumulative state grows; what would take ~7 min at constant 227 tok/s
|
||||||
|
takes 25+ min in practice (and may eventually OOM internally without
|
||||||
|
reaching us). For users on TP1+24GB, **MTP-off shifts the practical
|
||||||
|
ceiling from ~30-60K to ~60K**, but doesn't push to 90K.
|
||||||
|
|
||||||
|
### Updated upstream PR research (provided by user)
|
||||||
|
|
||||||
|
Three open PRs that could compose with our findings:
|
||||||
|
|
||||||
|
| PR | What | Why it matters |
|
||||||
|
|---|---|---|
|
||||||
|
| [vllm#35975](https://github.com/vllm-project/vllm/pull/35975) | Skip inputs_embeds GPU buffer for text-only models | ~64 MiB GPU + 64 MiB pinned CPU savings. Direct match for our 50 MiB margin failure. **Highest-value test next.** |
|
||||||
|
| [vllm#37521](https://github.com/vllm-project/vllm/pull/37521) | Spec-decode warmup memory accounting | Profiling/KV sizing leaves less false headroom. Worth testing for MTP-on path. |
|
||||||
|
| [vllm#37429](https://github.com/vllm-project/vllm/pull/37429) | KV cache sizing for hybrid Mamba/attention | Architectural — could free substantial headroom on hybrid Qwen3-Next models. Bigger/riskier. |
|
||||||
|
|
||||||
|
vLLM's official Qwen3-Next recipe documents `--tensor-parallel-size 4
|
||||||
|
--gpu-memory-utilization 0.8 --no-enable-chunked-prefill` as happy
|
||||||
|
path. Our 24GB TP1 long-context MTP case is OUTSIDE the documented
|
||||||
|
configuration — context worth flagging to Sander.
|
||||||
|
|
||||||
|
### Next: apply PR #35975 as text-patch + retest 60K MTP-on
|
||||||
|
|
||||||
|
Plan: modify `vllm/v1/worker/gpu_model_runner.py:623-633` and
|
||||||
|
`vllm/v1/spec_decode/eagle.py:192-201` to skip `inputs_embeds` buffer
|
||||||
|
allocation when not needed (no MM inputs, no prompt_embeds). 64 MiB
|
||||||
|
saved on text-only model + 64 MiB pinned CPU. If that turns 60K
|
||||||
|
MTP-on from -25 MiB to +39 MiB free, we have a real closure proof.
|
||||||
|
|
||||||
|
|
||||||
|
## ROUND 3 — vllm#35975 backport + mem-util tuning matrix (2026-05-02 PM)
|
||||||
|
|
||||||
|
### Closure recipe: PR#35975 + mem-util 0.92 + MTP-on
|
||||||
|
|
||||||
|
**60K probe RESULT: HTTP 200 OK, recall correct, MTP AL=4.00**
|
||||||
|
|
||||||
|
| Config | Boot resident | OOM/Pass at 60K | Wall | Notes |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| MTP-off, 0.95 mem-util | 23,164 MiB | ✅ Pass | 504s | Already established (Round 2) |
|
||||||
|
| MTP-on + #35975 + 0.95 mem-util | 22,720 MiB | ❌ Fails 46.5 MiB free / 50 needed | n/a | #35975 freed ~444 MiB at boot, but only ~22 MiB extra at peak |
|
||||||
|
| **MTP-on + #35975 + 0.92 mem-util** | **21,980 MiB** | **✅ Pass** | **~11.5 min wall** | **Cliff 2 closed** with full MTP K=3 + correct recall |
|
||||||
|
|
||||||
|
Notes on the MTP-on + #35975 + 0.92 run:
|
||||||
|
- Boot resident dropped from 22,720 (at 0.95) to 21,980 (at 0.92) — saved 740 MiB
|
||||||
|
- KV pool: 3.87 GiB / 305,472 tokens (vs 5.07 GiB / 363,264 at 0.95) — 1.43x → 1.25x concurrency at 180K
|
||||||
|
- Wall ~11.5 min (vs 8.4 min on MTP-off + 0.95) — slower than expected; investigation TBD
|
||||||
|
- MTP per-position acceptance: 1.000, 1.000, 1.000 (perfect on this prompt)
|
||||||
|
|
||||||
|
### Codex r2 verdict (post-bisect)
|
||||||
|
|
||||||
|
> #35975 is definitely relevant, but not sufficient alone. MTP-off proving 60K pass means draft residency is the dominant pressure. #35975 + gpu_memory_utilization=0.92 is now the right "keep MTP" candidate. P103 streaming output is lower priority unless 0.92 still fails or you need to recover more context at fixed 0.95.
|
||||||
|
|
||||||
|
### Two shippable variants
|
||||||
|
|
||||||
|
**Variant A — Long-text with MTP** (`long-text.yml`):
|
||||||
|
- Genesis v7.69 + Codex P103 gate fix + PN30 + PN32 + PN34
|
||||||
|
- vllm#35975 backport sidecar (`patch_inputs_embeds_optional.py`)
|
||||||
|
- `--gpu-memory-utilization 0.92` (down from 0.95)
|
||||||
|
- `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'` (retained)
|
||||||
|
- Cliff 2 closed at 60K. Probe 7 ceiling moves from 30-60K → 60K+.
|
||||||
|
|
||||||
|
**Variant B — Long-text without MTP** (`long-text-no-mtp.yml` — TODO):
|
||||||
|
- Same as A but `--speculative-config` REMOVED, `--gpu-memory-utilization 0.95` (recovered)
|
||||||
|
- Cliff 2 closed at 60K with full mem-util (KV pool 1.43x at 180K)
|
||||||
|
- Lower decode throughput (no MTP speedup)
|
||||||
|
- Trade-off: more KV/context vs slower decode
|
||||||
|
|
||||||
|
### Codex r2's recommended next bisect (open)
|
||||||
|
|
||||||
|
1. ~~#35975 + MTP-on + 0.92: establish pass~~ ✅ DONE (this run)
|
||||||
|
2. **#35975 + MTP-on + 0.93/0.935: find less conservative pass point** ← next
|
||||||
|
3. ~~MTP-off + 0.95: keep as alternate throughput/context-safe variant~~ ✅ already validated
|
||||||
|
4. Only consider P103 streaming output if MTP-on at 0.95 specifically required
|
||||||
|
|
||||||
|
User concurs with Codex's plan. Step 2 (midpoint 0.93/0.935) gives margin
|
||||||
|
recommendation for shippable.
|
||||||
|
|
||||||
|
### Next: 60K test at #35975 + MTP-on + 0.93 (then 0.935 if 0.93 fails)
|
||||||
|
|
||||||
|
|
||||||
|
## ROUND 4 — mem-util sweep at 60K (Codex r2 step 2-3)
|
||||||
|
|
||||||
|
### 60K results across mem-util at MTP-on + #35975
|
||||||
|
|
||||||
|
| mem-util | Boot resident | KV pool size | 60K MTP-on | Wall | Final GPU free | Notes |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| 0.95 | 22,720 MiB | 5.5 GiB / ~388K tok | ❌ OOM 50/46.5 | n/a | n/a | 22 MiB short of margin |
|
||||||
|
| 0.93 | 22,260 MiB | 4.4 GiB / ~340K tok | ✅ HTTP 200 | 623s | ~494 MiB | **Best balanced point.** AL=4.0, recall correct |
|
||||||
|
| 0.92 | 21,980 MiB | 3.87 GiB / ~305K tok | ✅ HTTP 200 | 689s | ~580 MiB | More margin but slower + less KV |
|
||||||
|
|
||||||
|
**Conclusion: 0.93 is the recommended balanced shipping point.** Recovers
|
||||||
|
~30K tokens of KV pool concurrency vs 0.92 while still passing 60K
|
||||||
|
Cliff 2 with real end-of-run margin (~494 MiB free). Wall improvement
|
||||||
|
likely from cudagraph friendlier allocator pattern at higher headroom.
|
||||||
|
|
||||||
|
### Three explicit shippable variants (Codex r2 framing, post-bisect)
|
||||||
|
|
||||||
|
**Balanced MTP** — `long-text.yml` updated:
|
||||||
|
- Genesis v7.69 + Codex P103 gate fix + PN30 + PN32 v2 + PN34
|
||||||
|
- vllm#35975 backport sidecar (`patch_inputs_embeds_optional.py`)
|
||||||
|
- mem-util 0.93 (down from 0.95)
|
||||||
|
- MTP K=3 retained
|
||||||
|
- Cliff 2 closed at 60K. KV concurrency at 180K: ~1.4x
|
||||||
|
|
||||||
|
**Max-context safety** — `long-text-no-mtp.yml` (TODO):
|
||||||
|
- Same patches except no `--speculative-config`
|
||||||
|
- mem-util 0.95 (full KV pool)
|
||||||
|
- For long single-shot RAG / codebase prompts where slow decode is OK
|
||||||
|
- Cliff 2 ceiling: 60K confirmed; 90K probe with max_tokens=1 pending
|
||||||
|
|
||||||
|
**Future upstream win**:
|
||||||
|
- vllm#37429 hybrid Mamba/attention KV cache sizing — could free more
|
||||||
|
residency without trading mem-util
|
||||||
|
- Worth testing on a separate branch when CI signals stabilize
|
||||||
|
|
||||||
|
### Remaining tests (in progress)
|
||||||
|
|
||||||
|
- 90K + max_tokens=1 at 0.93 — measures prefill ceiling without slow decode
|
||||||
|
- max_model_len bump (200K? 220K?) at 0.93 (Codex says do this AFTER mem-util pick — admission capacity test, not safety)
|
||||||
|
|
||||||
|
|
||||||
|
## ROUND 5 — 90K probe + admission capacity (final)
|
||||||
|
|
||||||
|
### 90K MTP-on + #35975 + 0.93, max_tokens=1 (prefill-only timing)
|
||||||
|
|
||||||
|
Result: **HTTP 500 OOM at 112s wall**
|
||||||
|
- Trace: chunk_fwd_o `o = torch.empty_like(v)` 50 MiB allocation, 24.5 MiB free
|
||||||
|
- Engine died (EngineDeadError)
|
||||||
|
|
||||||
|
Conclusion: **Cliff 2 ceiling at 0.93 + #35975 + MTP-on = somewhere
|
||||||
|
between 60K (passes) and 90K (fails)**. Likely 70-80K. For shipping
|
||||||
|
purposes, document 60K as the safe single-prompt envelope.
|
||||||
|
|
||||||
|
### Admission capacity test: max_model_len 200K at 0.93 + #35975
|
||||||
|
|
||||||
|
Result: Engine pre-check **REJECTED**
|
||||||
|
```
|
||||||
|
ValueError: To serve at least one request with the models's max seq len
|
||||||
|
(200000), (3.31 GiB KV cache is needed, which is larger than the
|
||||||
|
available KV cache memory (2.19 GiB). Based on the available memory,
|
||||||
|
the estimated maximum model length is 111456.
|
||||||
|
```
|
||||||
|
|
||||||
|
Per Codex r2 framing: this is *admission* capacity (would-it-fit), not
|
||||||
|
*safety*. The 200K declared at 0.93 + #35975 doesn't fit in available
|
||||||
|
KV pool.
|
||||||
|
|
||||||
|
**180K stays as the practical max_model_len ceiling at 0.93 + #35975 +
|
||||||
|
MTP-on.** Pushing higher requires either:
|
||||||
|
- Higher mem-util (back toward 0.95) — but then Cliff 2 returns
|
||||||
|
- Drop MTP — frees ~280 MiB residency from draft model
|
||||||
|
- vllm#37429 hybrid Mamba/attention KV reduction (untested)
|
||||||
|
|
||||||
|
### FINAL recommended recipe (Balanced MTP variant)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# docker-compose.long-text.yml
|
||||||
|
--max-model-len 180000
|
||||||
|
--gpu-memory-utilization 0.93
|
||||||
|
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
|
||||||
|
--max-num-batched-tokens 4128
|
||||||
|
--kv-cache-dtype turboquant_3bit_nc
|
||||||
|
--enable-chunked-prefill
|
||||||
|
--language-model-only
|
||||||
|
|
||||||
|
# Genesis env (key Cliff 2 + closures):
|
||||||
|
GENESIS_ENABLE_P103=1
|
||||||
|
GENESIS_ENABLE_PN30_DS_LAYOUT_SPEC_DECODE=1
|
||||||
|
GENESIS_ENABLE_PN32_GDN_CHUNKED_PREFILL=1
|
||||||
|
GENESIS_PN32_GDN_CHUNK_SIZE=8192
|
||||||
|
GENESIS_PN32_GDN_CHUNK_THRESHOLD=16384
|
||||||
|
GENESIS_FLA_FWD_H_MAX_T=16384
|
||||||
|
GENESIS_ENABLE_PN34_WORKSPACE_LOCK_RELAX=1
|
||||||
|
VLLM_SSM_CONV_STATE_LAYOUT=DS
|
||||||
|
|
||||||
|
# Sidecars (mounted):
|
||||||
|
patch_tolist_cudagraph.py # vllm#40807 workaround
|
||||||
|
patch_inputs_embeds_optional.py # vllm#35975 backport (~444 MiB savings)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Shippable recipe summary
|
||||||
|
|
||||||
|
| Variant | Config | 60K | 90K | KV @ 180K | Best for |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| **Balanced MTP** (this recipe) | MTP-on + 0.93 + #35975 | ✅ | ❌ ~70-80K cliff | 1.4x | Default. Long-context with spec-decode. |
|
||||||
|
| Max-context safety | MTP-off + 0.95 + #35975 | ✅ | TBD (longer test) | 1.43x | RAG / single-shot codebase analysis |
|
||||||
|
| Future upstream | + vllm#37429 | TBD | TBD | TBD | When PR matures |
|
||||||
|
|
||||||
|
|
||||||
|
## ROUND 6 — Max-context variant (MTP-off + 0.95 + 200K + #35975)
|
||||||
|
|
||||||
|
### Setup
|
||||||
|
|
||||||
|
Same Genesis v7.69 stack + Codex P103 gate fix + #35975 sidecar, but:
|
||||||
|
- `--speculative-config` REMOVED (no MTP K=3 draft)
|
||||||
|
- `--gpu-memory-utilization 0.95` (recovered)
|
||||||
|
- `--max-model-len 200000` (admission test)
|
||||||
|
|
||||||
|
Boot resident: **21,004 MiB** (lower than MTP-on variants because draft model
|
||||||
|
residency is gone). KV pool: 5.1 GiB / 272,288 tokens. Concurrency at 200K:
|
||||||
|
**1.20x**. **200K ADMISSION PASSES** (vs 180K cap on MTP-on path).
|
||||||
|
|
||||||
|
### 60K probe — MTP-off + 0.95 + 200K + #35975
|
||||||
|
|
||||||
|
Result: **HTTP 200 OK in 537s wall** (vs 504s without #35975 in earlier
|
||||||
|
test — slightly slower, possibly due to 200K ctx vs 180K). Recall correct
|
||||||
|
("violet otter 48"). No OOM. Probe matches the closure pattern of round 2's
|
||||||
|
MTP-off + 180K test, plus the extra 20K ctx admission headroom.
|
||||||
|
|
||||||
|
### 90K probe (max_tokens=1) — MTP-off + 0.95 + 200K + #35975
|
||||||
|
|
||||||
|
Result: **HTTP 000 at 1500s curl timeout** (25-min budget). Engine kept
|
||||||
|
computing (GPU 100% util, no OOM trace), but never delivered response
|
||||||
|
within practical bound. Same pattern as round 2 MTP-off + 180K 90K probe
|
||||||
|
(also indeterminate within 15 min budget there).
|
||||||
|
|
||||||
|
Practical Cliff 2 ceiling on max-context variant: **60K confirmed**.
|
||||||
|
90K is past wall-clock-feasibility on this hardware regardless of
|
||||||
|
mem-util/MTP/admission tweaks.
|
||||||
|
|
||||||
|
### Final shippable matrix
|
||||||
|
|
||||||
|
Both variants share the same single-prompt safety envelope (~60K).
|
||||||
|
Differ on declared context, decode TPS, and concurrency:
|
||||||
|
|
||||||
|
| Variant | max_model_len | mem-util | MTP | KV @ ctx | 60K | 90K |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| Balanced MTP (`long-text.yml`) | **180K** | 0.93 | ✅ K=3 | 1.40x | ✅ 623s | ❌ OOM 112s |
|
||||||
|
| Max-context (`long-text-no-mtp.yml`) | **200K** | 0.95 | ❌ off | 1.20x | ✅ 537s | indeterm. |
|
||||||
|
|
||||||
|
### Codex r2 verdict (final)
|
||||||
|
|
||||||
|
> Cliff 2 at 60K on 24GB, TP1, Qwen3-Next TQ3, DS layout is not caused by
|
||||||
|
> the P103 cu_seqlens gate after the semantic fix. With MTP enabled,
|
||||||
|
> draft-model residency removes enough activation headroom that late
|
||||||
|
> FLA/GDN prefill fails. Disabling MTP lets the same request complete with
|
||||||
|
> HTTP 200; the earlier failure mode was headroom, while the verify
|
||||||
|
> harness timeout was client-side.
|
||||||
|
|
||||||
|
Plus: vllm#35975 backport frees enough headroom that MTP-on can also
|
||||||
|
close 60K when paired with mem-util 0.93.
|
||||||
|
|
||||||
|
### Open future tests
|
||||||
|
|
||||||
|
- vllm#37429 (hybrid Mamba/attention KV cache sizing) — could free more
|
||||||
|
resident memory without trading mem-util
|
||||||
|
- P103 streaming output variant — Codex r2 lower-priority recommendation;
|
||||||
|
on-disk implementation deferred
|
||||||
177
results/v0.20-migration/v769-test.summary
Normal file
177
results/v0.20-migration/v769-test.summary
Normal file
@@ -0,0 +1,177 @@
|
|||||||
|
# v7.69 Retest — Branch v7.69-cliff2-test
|
||||||
|
Date: 2026-05-02 PM
|
||||||
|
Genesis pin: 2db18df (v7.69 dev tip)
|
||||||
|
vLLM image: nightly-7a1eb8ac2ec4ea69338c51dc7afd4b15010abfa8 (v0.20.1rc1.dev16)
|
||||||
|
Hardware: 1× RTX 3090 (24 GB), SM 8.6 Ampere consumer, TP=1
|
||||||
|
Compose: long-text.yml @ 180K + 0.95 (back to original target — DS layout active)
|
||||||
|
|
||||||
|
## Env bundle (Cliff 2 closure recipe per Sander's v7.69 reply)
|
||||||
|
|
||||||
|
GENESIS_ENABLE_P103=1
|
||||||
|
GENESIS_ENABLE_PN32_GDN_CHUNKED_PREFILL=1
|
||||||
|
GENESIS_PN32_GDN_CHUNK_SIZE=8192
|
||||||
|
GENESIS_PN32_GDN_CHUNK_THRESHOLD=16384
|
||||||
|
GENESIS_FLA_FWD_H_MAX_T=16384
|
||||||
|
GENESIS_ENABLE_PN34_WORKSPACE_LOCK_RELAX=1
|
||||||
|
VLLM_SSM_CONV_STATE_LAYOUT=DS
|
||||||
|
|
||||||
|
## Sidecars dropped on this branch
|
||||||
|
- patch_pn25_genesis_register_fix.py (replaced by Sander's PN25 v7.68)
|
||||||
|
- patch_pn30_dst_shaped_temp_fix.py (replaced by Sander's PN30 v7.68 + v7.69 part3 marker fix)
|
||||||
|
- patch_workspace_lock_disable.py (replaced by Sander's PN34, opt-in env)
|
||||||
|
|
||||||
|
## ✅ F1 — PN30 v7.68 part3 drift-marker fix CONFIRMED WORKING
|
||||||
|
v7.68 dev tip (18e65e3) had part3 skip with `upstream_merged` because
|
||||||
|
its drift-markers list contained the bare prefix `[Genesis PN30` which
|
||||||
|
also matched parts 1+2's own markers. v7.69 (2db18df) tightened part3's
|
||||||
|
drift markers to `[Genesis PN30 v7.68 dst-shaped]` — specific enough to
|
||||||
|
not collide.
|
||||||
|
|
||||||
|
Boot log:
|
||||||
|
[PN30 v1/worker/mamba_utils.py — collect_mamba_copy_meta dst-shaped DS
|
||||||
|
temp (issue #17, v7.68)] applied 1 sub-patches:
|
||||||
|
pN30_collect_mamba_copy_meta_dst_shaped_temp
|
||||||
|
[Genesis] applied: PN30 DS conv state + spec-decode AL>1 (issue #17)
|
||||||
|
— PN30 v7.68 applied: ...
|
||||||
|
|
||||||
|
DS layout active (`Setting SSM conv state layout to DS`). Part1 + part2
|
||||||
|
+ part3 all APPLY clean, no upstream_merged false-positive. Apply_all
|
||||||
|
elapsed: clean, 0 failures.
|
||||||
|
|
||||||
|
## ✅ F2 — P103 chunk.py self-install hook CONFIRMED FIRES
|
||||||
|
v7.68 P103 logged "rebound at 0 caller sites" because `setattr` ran in
|
||||||
|
the entrypoint shell process and was lost on `exec vllm serve` (worker
|
||||||
|
spawn replaced the process image).
|
||||||
|
|
||||||
|
v7.69 P103 ships a text-patched self-install hook appended to
|
||||||
|
chunk.py end-of-file:
|
||||||
|
|
||||||
|
try:
|
||||||
|
import os as _genesis_p103_os
|
||||||
|
if _genesis_p103_os.environ.get("GENESIS_ENABLE_P103", ...):
|
||||||
|
from vllm._genesis.wiring.hybrid.patch_103_fla_cliff2_chunked
|
||||||
|
import _genesis_p103_install_at_import as _genesis_p103_install
|
||||||
|
_genesis_p103_install(globals())
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
Confirmed firing on TP=1 spawn. Cliff 2 trace path:
|
||||||
|
|
||||||
|
vllm/model_executor/layers/fla/ops/chunk.py:111 forward
|
||||||
|
→ vllm/_genesis/wiring/hybrid/patch_103_fla_cliff2_chunked.py:203
|
||||||
|
chunked_fwd ← P103 wrapper engaged at runtime
|
||||||
|
→ return original_fwd(...) ← BUT falls through to fallback
|
||||||
|
|
||||||
|
The wrap IS engaged in workers. F2 is verified at the
|
||||||
|
"hook fires correctly" level. Boot log:
|
||||||
|
[Genesis P103 self-install] wrapper installed in chunk.py at
|
||||||
|
module-import time (survives `exec vllm serve` + worker spawn)
|
||||||
|
|
||||||
|
## ⚠️ F3 — Cliff 2 NOT closed at 60K despite P103 + PN32 v2
|
||||||
|
|
||||||
|
Probes pass / fail summary:
|
||||||
|
[1/7] Long-context needle 10K ✅ recalled 'silver otter 16'
|
||||||
|
[1/7] Long-context needle 30K ✅ recalled 'amber iguana 95'
|
||||||
|
(was crashing on v7.68; v7.69 fix)
|
||||||
|
[2/7] Tool prefill 25K ✅ tool_call clean
|
||||||
|
[3/7] IDE-agent one-shot ✅ 66 completion tokens
|
||||||
|
[4/7] Multi-turn agent ✅
|
||||||
|
[5/7] LCB-coding shape ✅
|
||||||
|
[6/7] Reasoning max=8192 ✅
|
||||||
|
[7/7] Long-context needle 60K ❌ HTTP 500 (Cliff 2 OOM)
|
||||||
|
[7/7] Long-context needle 90K ❌ HTTP 500
|
||||||
|
|
||||||
|
Cliff 2 OOM trace (probe 7 60K rung):
|
||||||
|
|
||||||
|
File "vllm/model_executor/layers/fla/ops/chunk.py", line 111, in forward
|
||||||
|
g, o, A, final_state, w, h, v_new = chunk_gated_delta_rule_fwd(
|
||||||
|
File "vllm/_genesis/wiring/hybrid/patch_103_fla_cliff2_chunked.py",
|
||||||
|
line 203, in chunked_fwd
|
||||||
|
return original_fwd( ← P103 fallback path
|
||||||
|
File "vllm/model_executor/layers/fla/ops/chunk.py", line 72,
|
||||||
|
in chunk_gated_delta_rule_fwd
|
||||||
|
o = chunk_fwd_o(
|
||||||
|
File "vllm/model_executor/layers/fla/ops/chunk_o.py", line 161,
|
||||||
|
in chunk_fwd_o
|
||||||
|
o = torch.empty_like(v)
|
||||||
|
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 50.00 MiB.
|
||||||
|
GPU 0 has a total capacity of 23.56 GiB of which 24.50 MiB is free.
|
||||||
|
|
||||||
|
### Why P103's chunked path didn't engage
|
||||||
|
|
||||||
|
P103's gate is `T <= MAX_T or cu_seqlens != None`. At 60K prompt
|
||||||
|
serving:
|
||||||
|
- T = 60000 (well above MAX_T=16384)
|
||||||
|
- But cu_seqlens != None (vLLM serving ALWAYS passes cu_seqlens)
|
||||||
|
→ P103 falls through to original_fwd
|
||||||
|
|
||||||
|
### Why PN32 v2 didn't engage
|
||||||
|
|
||||||
|
PN32 v2's docstring says "Multi-seq bypasses to original" — i.e. it
|
||||||
|
only chunks when there's exactly one sequence. With MTP K=3
|
||||||
|
spec-decode, the dispatcher reports `uniform_query_len=4` per
|
||||||
|
P66 cudagraph filter log. Whether PN32's gate sees this as
|
||||||
|
"multi-seq" wasn't traceable at runtime, but the trace doesn't
|
||||||
|
show PN32's chunked path.
|
||||||
|
|
||||||
|
### Net: F3 closure works only when cu_seqlens=None AND single-seq
|
||||||
|
|
||||||
|
Both P103 and PN32 v2 have gates that require cu_seqlens=None
|
||||||
|
and/or single-sequence to engage. vLLM serving always passes
|
||||||
|
cu_seqlens (it's the per-sequence boundary marker that enables
|
||||||
|
batched serving). So neither chunking path fires under real
|
||||||
|
serving, only synthetic test scenarios.
|
||||||
|
|
||||||
|
## Other engine state observations on v7.69
|
||||||
|
|
||||||
|
Engine survived probe 7 OOM gracefully (HTTP 500 + EngineDeadError
|
||||||
|
exception → API server shutdown). Compare to v7.66 where Cliff 2 OOM
|
||||||
|
killed the EngineCore process abruptly. Suggests vLLM's
|
||||||
|
`async_llm.output_handler` exception handling is cleaner on v0.20+.
|
||||||
|
Engine still came back up clean on next request batch (could re-test
|
||||||
|
small probes after large probe failed).
|
||||||
|
|
||||||
|
## Cross-rig regressions: NONE
|
||||||
|
TPS on smoke benchmark equivalent to v7.66 + sidecars (same engine
|
||||||
|
path under cu_seqlens != None, no overhead from PN32 v2 in fallback
|
||||||
|
mode).
|
||||||
|
|
||||||
|
## Recommendation back to Sander
|
||||||
|
|
||||||
|
Two options for v7.70:
|
||||||
|
|
||||||
|
(a) Ship F3 fix that handles cu_seqlens != None — slice metadata
|
||||||
|
along with tensors per chunk; thread initial_state correctly
|
||||||
|
across chunk boundaries even when cu_seqlens describes a
|
||||||
|
multi-sequence batch. Per Sander's own caveat in v7.69 reply:
|
||||||
|
"Multi-seq bypasses to original (chunking across cu_seqlens
|
||||||
|
boundaries needs inner state-cache surgery not exposed at this
|
||||||
|
layer)" — this is the surgery that needs to happen.
|
||||||
|
|
||||||
|
(b) Keep current "synthetic-only" gate but document explicitly
|
||||||
|
that P103 + PN32 only close Cliff 2 in benchmarks where
|
||||||
|
cu_seqlens=None is passed manually. Real serving stays
|
||||||
|
behind the cliff. Operators on TP=1 + 24GB should plan for
|
||||||
|
50-60K single-prompt ceiling and route longer prompts to
|
||||||
|
TP=2 (`dual.yml`) or llama.cpp.
|
||||||
|
|
||||||
|
Our preference: (a) — but we recognize this is non-trivial. Happy to
|
||||||
|
engage ChatGPT/Codex CLI on the cu_seqlens-aware chunking design and
|
||||||
|
share back as we did for PN30 dst-shaped + PN25 import-time
|
||||||
|
registration.
|
||||||
|
|
||||||
|
## What stayed working
|
||||||
|
- All 4 of v7.68's other accepted-and-folded patches (PN25 v7.68
|
||||||
|
TP=1 register, PN34 workspace_lock, PN30 v7.68 part1+part2, P101)
|
||||||
|
remain solid.
|
||||||
|
- Tool calls clean across IDE-agent shapes (probes 3 + 4).
|
||||||
|
- 30K single-prompt now safe (was unsafe on v7.68 due to PN30
|
||||||
|
being broken; v7.69 closes that regression).
|
||||||
|
|
||||||
|
## Net: v7.69 is a clear improvement over both v7.66 and v7.68 dev tip
|
||||||
|
|
||||||
|
If we accept that Cliff 2 is fundamentally still architectural for
|
||||||
|
single-24GB-GPU long-prompt and route around it, v7.69 is our
|
||||||
|
best stable single-card config to date. Recommend Sander tag v7.69
|
||||||
|
stable and we move to master once the 30K-now-safe data is
|
||||||
|
cross-confirmed on Sander's planned A5000-rig.
|
||||||
@@ -106,23 +106,37 @@ echo "Model dir: ${MODEL_DIR}"
|
|||||||
# vllm/_genesis package + per-patch env opts). Newer composes mount the package;
|
# vllm/_genesis package + per-patch env opts). Newer composes mount the package;
|
||||||
# the legacy compose still references the v7.13 shim.
|
# the legacy compose still references the v7.13 shim.
|
||||||
# Pin Genesis to the exact commit our published numbers were measured against.
|
# Pin Genesis to the exact commit our published numbers were measured against.
|
||||||
# Currently pointing at v7.66 dev tip (commit fc89395, 2026-05-02 AM). Bumped
|
# Currently pointing at v7.69 dev tip (commit 2db18df, 2026-05-02 PM). Bumped
|
||||||
# from v7.64 (64dd18b) for the v7.65 patch set:
|
# from v7.66 (fc89395) for the v7.69 patch set, which addresses the 3
|
||||||
# - P38B / P15B — close the Cliff 1 mech B cascade (issues #14 + #15) via
|
# regressions the v7.68 cross-rig retest found ([club-3090#19] and our
|
||||||
# compile-safe in-source hook + FA varlen workspace clamp.
|
# v7.68-cliff2-test branch summary):
|
||||||
# - PN25 — Inductor-safe silu_and_mul opaque op (replaces our local
|
# - F1 (PN30 part3 drift-marker bug) — fixed via specific marker
|
||||||
# patch_pn12_compile_safe_custom_op.py — now removed).
|
# `[Genesis PN30 v7.68 dst-shaped]` so part3 idempotency check no longer
|
||||||
# - PN26b — Genesis-original sparse-V Triton kernel for SM86 (Ampere
|
# collides with part1+2 markers in the same file.
|
||||||
# consumer). First sparse-V kernel in any public tree for SM86. Default
|
# - F2 (P103 setattr lost on `exec vllm serve`) — fixed via self-install
|
||||||
# ON in v0.20+ composes (BLOCK_KV=8 num_warps=4 threshold=0.01 per
|
# hook text-patched into chunk.py end-of-file. Survives any startup
|
||||||
# Sandermage's 27B-specific tuning).
|
# mechanism (workers, fork, spawn, exec). The "rebound at 0 caller sites"
|
||||||
# - PN28 — merge_attn_states NaN guard backport (vllm#39148).
|
# log message in v7.68 was misleading — internal callers DID get the
|
||||||
# - Cliff 8 hardening (partial_apply_warnings counter in boot summary).
|
# setattr in the entrypoint shell process, but `exec` replaced the image
|
||||||
# Pinned to dev SHA fc89395 because v7.66 is feature-complete on dev but not
|
# and lost it. v7.69 hook fires every time chunk.py imports.
|
||||||
# yet tagged; SHA pin is immutable.
|
# - F3 (PN32 v1 chunked at wrong level) — rewritten as PN32 v2 to patch
|
||||||
# Bumping GENESIS_PIN requires re-running verify-full.sh against your composes
|
# `_forward_core` directly + thread initial_state via prior chunk's
|
||||||
|
# last_recurrent_state. Composes with P103: v2 chunks the OUTER FLA
|
||||||
|
# call, P103 chunks INSIDE the FLA inner h tensor.
|
||||||
|
# Recommended Cliff 2 closure env bundle for single-24GB-GPU:
|
||||||
|
# GENESIS_ENABLE_P103=1 (close inner FLA h tensor)
|
||||||
|
# GENESIS_ENABLE_PN32_GDN_CHUNKED_PREFILL=1 (close outer FLA call buffer)
|
||||||
|
# GENESIS_PN32_GDN_CHUNK_SIZE=8192 (default)
|
||||||
|
# GENESIS_PN32_GDN_CHUNK_THRESHOLD=16384 (default)
|
||||||
|
# GENESIS_FLA_FWD_H_MAX_T=16384 (P103 default)
|
||||||
|
# v7.69 also retains v7.68's accept-and-fold of our 3 cross-rig sidecars:
|
||||||
|
# PN25 v7.68 (TP=1 worker-spawn registration), PN30 v7.68 (DS conv-state
|
||||||
|
# layout dst-shaped temp), PN34 (vllm#39226 runtime workspace_lock).
|
||||||
|
# Pinned to dev SHA 2db18df because v7.69 is feature-complete on dev pending
|
||||||
|
# our retest validation; if clean, Sander will tag stable.
|
||||||
|
# Bumping GENESIS_PIN requires re-running verify-stress.sh against your composes
|
||||||
# to confirm the new commit works on your config.
|
# to confirm the new commit works on your config.
|
||||||
GENESIS_PIN="${GENESIS_PIN:-fc89395}"
|
GENESIS_PIN="${GENESIS_PIN:-2db18df}"
|
||||||
|
|
||||||
if [[ "${SKIP_GENESIS:-0}" != "1" ]]; then
|
if [[ "${SKIP_GENESIS:-0}" != "1" ]]; then
|
||||||
if [[ -d "${GENESIS_DIR}/.git" ]]; then
|
if [[ -d "${GENESIS_DIR}/.git" ]]; then
|
||||||
@@ -143,38 +157,11 @@ if [[ "${SKIP_GENESIS:-0}" != "1" ]]; then
|
|||||||
fi
|
fi
|
||||||
echo "[genesis] Pinned to ${GENESIS_PIN} ($(cd "${GENESIS_DIR}" && git rev-parse --short HEAD))"
|
echo "[genesis] Pinned to ${GENESIS_PIN} ($(cd "${GENESIS_DIR}" && git rev-parse --short HEAD))"
|
||||||
|
|
||||||
# PN25 worker-spawn registration fix — local backport.
|
# v7.69 ships PN25 + PN30 + PN34 directly (Sander's accept-and-fold of our
|
||||||
#
|
# cross-rig sidecars). Local patch_pn25_genesis_register_fix.py +
|
||||||
# Sandermage shipped his own fix in d92bcb3 (hasattr global-registry guard
|
# patch_pn30_dst_shaped_temp_fix.py + patch_workspace_lock_disable.py are
|
||||||
# in `_register_op_once`), but cross-rig validation on our TP=1 single-card
|
# now redundant. Sidecar Python files retained in vllm/patches/ for
|
||||||
# showed it doesn't work — `torch.ops.genesis.silu_and_mul_pooled` doesn't
|
# rollback if any v7.69 patch regresses on your config.
|
||||||
# exist in spawned workers on TP=1 (whereas it does on his TP=2 PROD).
|
|
||||||
# Reported back as comment on Sandermage/genesis-vllm-patches#16.
|
|
||||||
#
|
|
||||||
# Our local v3 patch takes a different approach: register at activation.py
|
|
||||||
# import time as a module-level cached global, BEFORE any dynamo trace runs.
|
|
||||||
# Survives worker spawn correctly on TP=1.
|
|
||||||
#
|
|
||||||
# Idempotent. Safe to re-run.
|
|
||||||
if [[ -f "${ROOT_DIR}/models/qwen3.6-27b/vllm/patches/patch_pn25_genesis_register_fix.py" ]]; then
|
|
||||||
(cd "${ROOT_DIR}" && python3 models/qwen3.6-27b/vllm/patches/patch_pn25_genesis_register_fix.py) || {
|
|
||||||
echo "[genesis] WARN: PN25 register fix did not apply cleanly. PN25 may not work in workers." >&2
|
|
||||||
}
|
|
||||||
fi
|
|
||||||
|
|
||||||
# PN30 DS conv-state layout fix — local correction for Genesis issue #17.
|
|
||||||
#
|
|
||||||
# Sander's PN30 avoided vLLM's DS+spec-decode NotImplementedError by
|
|
||||||
# compacting state[src_block, :, offset:] and raw-memcpying it into the
|
|
||||||
# destination block. That corrupts DS row strides. Our sidecar patches PN30
|
|
||||||
# so collect_mamba_copy_meta builds a full destination-shaped temp block,
|
|
||||||
# copies the source tail into the dst prefix, then reuses PN30's temp-list
|
|
||||||
# lifetime handling.
|
|
||||||
if [[ -f "${ROOT_DIR}/models/qwen3.6-27b/vllm/patches/patch_pn30_dst_shaped_temp_fix.py" ]]; then
|
|
||||||
(cd "${ROOT_DIR}" && python3 models/qwen3.6-27b/vllm/patches/patch_pn30_dst_shaped_temp_fix.py) || {
|
|
||||||
echo "[genesis] WARN: PN30 dst-shaped temp fix did not apply cleanly. Keep PN30 disabled or use SD layout." >&2
|
|
||||||
}
|
|
||||||
fi
|
|
||||||
else
|
else
|
||||||
echo "[genesis] SKIP_GENESIS=1 — not cloning."
|
echo "[genesis] SKIP_GENESIS=1 — not cloning."
|
||||||
fi
|
fi
|
||||||
|
|||||||
Reference in New Issue
Block a user