5aa97a25d9
This branch migrates the entire vLLM stack from `dev205+g07351e088` + Genesis
v7.64 to `0.20.1rc1.dev16+g7a1eb8ac2` + Genesis v7.65 dev tip (commit
`d89a089`). v7.65 is on Sandermage's `dev` branch — explicitly the cross-rig
testing surface he requested in discussion #19; he'll merge dev→main once we
both confirm stable. Pin gates restated when that lands.
What changes
------------
Pin migration:
- vLLM image: nightly-07351e08... → nightly-7a1eb8ac2... (dev205 → v0.20.1rc1.dev16)
- Genesis: 64dd18b (v7.64) → d89a089 (v7.65 dev tip)
Sidecar churn:
- DROPPED: patch_pn12_ffn_pool_anchor.py (PN12 native on v0.20)
- DROPPED: patch_pn12_compile_safe_custom_op.py (Genesis P38B in-source hook)
- DROPPED: patch_fa_max_seqlen_clamp.py (Genesis PN17 + P15B)
- ADDED: patch_workspace_lock_disable.py (relaxes vllm#39226 strict assertion;
P98 covers same surface but auto-skips on v0.20 due to drift-marker false
positive — pending Sandermage marker fix)
Env-var alignment to Sandermage's PROD set (start_27b_int4_TQ_k8v4.sh@dev):
- FIXED naming bugs that silently no-op'd patches:
- PN9_INDEPENDENT_DRAFTER_ATT → _ATTN (was silently OFF)
- PN22 → PN22_LOCAL_ARGMAX_TP (was silently OFF)
- PN26_BLOCK_KV → PN26_SPARSE_V_BLOCK_KV (fell back to default 4, not 8)
- PN26_NUM_WARPS → PN26_SPARSE_V_NUM_WARPS
- PN26_THRESHOLD → PN26_SPARSE_V_THRESHOLD (fell back to default 0.001, not 0.01)
- ADDED explicit-OFFs to match Sander's PROD verbatim:
- P78_TOLIST_CAPTURE_GUARD=0 (we use our own patch_tolist_cudagraph.py)
- P81_FP8_BLOCK_SCALED_M_LE_8=0 (FP8-specific, no-op on TQ3)
- P82=0, P82_THRESHOLD_SINGLE=0.3
- Cap divergence (justified): PROFILE_RUN_CAP_M=4128 + PREALLOC_TOKEN_BUDGET=4128
(Sander uses 4096 — vLLM `interface.py:639` forces our config's Mamba
block_size to 4128 due to TQ3 + TP=1 page-size math; lower values
AssertionError at boot)
- Carry-forward (intentional): P4 (hybrid TQ required), P65 (TQ spec-CG
downgrade — pending v0.20 verification that #40880 closure makes it
redundant)
Cold-start cache mounts (closes #22):
- All 10 composes now mount torch_compile_cache + Triton cache from
`models/qwen3.6-27b/vllm/cache/`. First boot warms (~6 min); warm boot
drops to ~3.2 min (47% faster). Per-stage savings on long-text:
- Dynamo bytecode transform: 18s → 5s (-73%)
- torch.compile: 57s → 9s (-85%)
- Initial profiling/warmup: 51s → 7s (-87%)
Mamba block_size cap fix:
- v0.20 enforces `long_prefill_token_threshold >= block_size`; on hybrid
Mamba+TQ3, vLLM forces block_size=4128. Bumped GENESIS_PROFILE_RUN_CAP_M
and PREALLOC_TOKEN_BUDGET 4096→4128 across all 5 main composes.
Default 48K compose:
- Required workspace_lock_disable sidecar after initial v0.20 boot hit
vllm#39226 strict assertion. Caught during validation, fixed.
Context restored vs dev205 backoffs (validated 33K + 50K stress on v0.20):
- long-text: 185K → 214K (+16%)
- long-vision: 140K → 198K (+41%)
- bounded-thinking: 185K → 214K (+16%)
Bench results (n=5, results/v0.20-migration/):
- long-text 214K narr 49.74 / code 67.39 (CV 2.6/2.7%)
- long-vision 198K narr 50.32 / code 66.12 (CV 2.3/4.1%)
- bounded-thinking 214K narr 49.77 / code 65.80 (CV 1.4/2.3%)
- tools-text 75K (fp8) narr 53.32 / code 69.66 (CV 2.3/1.4%)
- dual-turbo 262K (TP=2) narr 58.33 / code 76.01 per-stream
269 TPS aggregate at n=4 streams (3.63x speedup)
- default 48K narr 48.82 / code 65.98 (n=3)
Validation: verify-full 8/8 on every variant. verify-stress 33K AND 50K
tool-prefill PASS on every variant — the cliff that fired on EVERY dev205
config no longer reproduces.
Docs + charts:
- README + SINGLE_CARD + DUAL_CARD + CLIFFS + EXAMPLES + STRUCTURED_COT
+ FAQ + UPSTREAM + 3 engine docs + model README + INTERNALS + CHANGELOG
all updated with new pin, ctx, TPS numbers, and "v0.20 unblock" section
- performance.{png,svg} + variants regenerated with measured TPS
- vram-budget.{png,svg} + variants regenerated with measured VRAM
- UPSTREAM tracker: 5 issues moved ✅ closed (PR #12, #13, #14, #15, P104
superseded by PN17 + P15B)
Issues addressed:
- #16 (Cliff 1 mech B leaks past PN12 on inductor-compiled FFN) — partial:
v0.20's revised TQ FA paths close the synthetic stress; PN25 (Sander's
proper compile-path opaque-op fix) is on dev but explicitly opt-in pending
worker-fork registration fix. Workarounds documented (tools-text fp8 path
/ --enforce-eager) until Sander ships PN25 default-on.
- #20 (launch.sh port + container-name mismatch) — already closed by
77ca576 (post-issue-filing).
- #22 (cold-start caching) — closed by cache mounts above.
Remaining caveats:
- Cliff 2 (DeltaNet GDN forward, single prompt ≥50-60K) unchanged —
architectural, applies to all single-card vLLM TQ3 paths. Mitigation:
dual-turbo TP=2 (state splits across cards) or llama.cpp 262K.
- Default 48K narr_TPS (48.82) slightly under chart's 55 reference —
bench variance + sample size n=3; not regression.
- Dual.yml / dual-dflash* not re-benched on v0.20; numbers carry forward
from dev205 (fp8 paths were not TPS-changed by the migration).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
46 lines
3.1 KiB
Plaintext
46 lines
3.1 KiB
Plaintext
|
|
========== NARRATIVE (prompt=65 chars, max_tokens=1000) ==========
|
|
=== warmups (3) ===
|
|
warm-1 wall= 23.06s ttft= 4096ms toks=1000 wall_TPS= 43.37 decode_TPS= 52.74
|
|
warm-2 wall= 17.50s ttft= 88ms toks=1000 wall_TPS= 57.14 decode_TPS= 57.43
|
|
warm-3 wall= 17.96s ttft= 86ms toks=1000 wall_TPS= 55.67 decode_TPS= 55.94
|
|
|
|
=== measured (5) ===
|
|
run-1 wall= 17.78s ttft= 119ms toks=1000 wall_TPS= 56.23 decode_TPS= 56.62
|
|
run-2 wall= 17.24s ttft= 88ms toks=1000 wall_TPS= 58.01 decode_TPS= 58.30
|
|
run-3 wall= 17.37s ttft= 118ms toks=1000 wall_TPS= 57.57 decode_TPS= 57.97
|
|
run-4 wall= 16.45s ttft= 116ms toks=1000 wall_TPS= 60.78 decode_TPS= 61.21
|
|
run-5 wall= 16.93s ttft= 117ms toks=1000 wall_TPS= 59.07 decode_TPS= 59.49
|
|
|
|
=== summary [narrative] (n=5) ===
|
|
wall_TPS mean= 58.33 std= 1.70 CV= 2.9% min=56.23 max=60.78
|
|
decode_TPS mean= 58.72 std= 1.73 CV= 2.9% min=56.62 max=61.21
|
|
TTFT mean= 112ms std= 13ms min=88ms max=119ms
|
|
|
|
========== CODE (prompt=78 chars, max_tokens=800) ==========
|
|
=== warmups (3) ===
|
|
warm-1 wall= 9.03s ttft= 116ms toks= 698 wall_TPS= 77.34 decode_TPS= 78.34
|
|
warm-2 wall= 10.48s ttft= 131ms toks= 800 wall_TPS= 76.34 decode_TPS= 77.30
|
|
warm-3 wall= 11.50s ttft= 116ms toks= 800 wall_TPS= 69.59 decode_TPS= 70.30
|
|
|
|
=== measured (5) ===
|
|
run-1 wall= 10.89s ttft= 87ms toks= 800 wall_TPS= 73.49 decode_TPS= 74.09
|
|
run-2 wall= 8.10s ttft= 86ms toks= 640 wall_TPS= 79.03 decode_TPS= 79.88
|
|
run-3 wall= 10.26s ttft= 117ms toks= 800 wall_TPS= 78.00 decode_TPS= 78.90
|
|
run-4 wall= 9.58s ttft= 123ms toks= 750 wall_TPS= 78.31 decode_TPS= 79.33
|
|
run-5 wall= 6.71s ttft= 135ms toks= 478 wall_TPS= 71.21 decode_TPS= 72.67
|
|
|
|
=== summary [code] (n=5) ===
|
|
wall_TPS mean= 76.01 std= 3.46 CV= 4.5% min=71.21 max=79.03
|
|
decode_TPS mean= 76.97 std= 3.34 CV= 4.3% min=72.67 max=79.88
|
|
TTFT mean= 110ms std= 22ms min=86ms max=135ms
|
|
|
|
=== GPU state ===
|
|
0, 60 %, 19816 MiB, 24576 MiB, 229.01 W, 68
|
|
1, 90 %, 19816 MiB, 24576 MiB, 227.53 W, 63
|
|
|
|
=== Last 3 SpecDecoding metrics ===
|
|
(APIServer pid=1) INFO 05-01 16:56:43 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.46, Accepted throughput: 55.80 tokens/s, Drafted throughput: 68.10 tokens/s, Accepted: 558 tokens, Drafted: 681 tokens, Per-position acceptance rate: 0.952, 0.841, 0.665, Avg Draft acceptance rate: 81.9%
|
|
(APIServer pid=1) INFO 05-01 16:56:53 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.51, Accepted throughput: 56.70 tokens/s, Drafted throughput: 67.80 tokens/s, Accepted: 567 tokens, Drafted: 678 tokens, Per-position acceptance rate: 0.951, 0.845, 0.712, Avg Draft acceptance rate: 83.6%
|
|
(APIServer pid=1) INFO 05-01 16:57:03 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.39, Accepted throughput: 53.50 tokens/s, Drafted throughput: 67.20 tokens/s, Accepted: 535 tokens, Drafted: 672 tokens, Per-position acceptance rate: 0.929, 0.808, 0.652, Avg Draft acceptance rate: 79.6%
|