5aa97a25d9
This branch migrates the entire vLLM stack from `dev205+g07351e088` + Genesis
v7.64 to `0.20.1rc1.dev16+g7a1eb8ac2` + Genesis v7.65 dev tip (commit
`d89a089`). v7.65 is on Sandermage's `dev` branch — explicitly the cross-rig
testing surface he requested in discussion #19; he'll merge dev→main once we
both confirm stable. Pin gates restated when that lands.
What changes
------------
Pin migration:
- vLLM image: nightly-07351e08... → nightly-7a1eb8ac2... (dev205 → v0.20.1rc1.dev16)
- Genesis: 64dd18b (v7.64) → d89a089 (v7.65 dev tip)
Sidecar churn:
- DROPPED: patch_pn12_ffn_pool_anchor.py (PN12 native on v0.20)
- DROPPED: patch_pn12_compile_safe_custom_op.py (Genesis P38B in-source hook)
- DROPPED: patch_fa_max_seqlen_clamp.py (Genesis PN17 + P15B)
- ADDED: patch_workspace_lock_disable.py (relaxes vllm#39226 strict assertion;
P98 covers same surface but auto-skips on v0.20 due to drift-marker false
positive — pending Sandermage marker fix)
Env-var alignment to Sandermage's PROD set (start_27b_int4_TQ_k8v4.sh@dev):
- FIXED naming bugs that silently no-op'd patches:
- PN9_INDEPENDENT_DRAFTER_ATT → _ATTN (was silently OFF)
- PN22 → PN22_LOCAL_ARGMAX_TP (was silently OFF)
- PN26_BLOCK_KV → PN26_SPARSE_V_BLOCK_KV (fell back to default 4, not 8)
- PN26_NUM_WARPS → PN26_SPARSE_V_NUM_WARPS
- PN26_THRESHOLD → PN26_SPARSE_V_THRESHOLD (fell back to default 0.001, not 0.01)
- ADDED explicit-OFFs to match Sander's PROD verbatim:
- P78_TOLIST_CAPTURE_GUARD=0 (we use our own patch_tolist_cudagraph.py)
- P81_FP8_BLOCK_SCALED_M_LE_8=0 (FP8-specific, no-op on TQ3)
- P82=0, P82_THRESHOLD_SINGLE=0.3
- Cap divergence (justified): PROFILE_RUN_CAP_M=4128 + PREALLOC_TOKEN_BUDGET=4128
(Sander uses 4096 — vLLM `interface.py:639` forces our config's Mamba
block_size to 4128 due to TQ3 + TP=1 page-size math; lower values
AssertionError at boot)
- Carry-forward (intentional): P4 (hybrid TQ required), P65 (TQ spec-CG
downgrade — pending v0.20 verification that #40880 closure makes it
redundant)
Cold-start cache mounts (closes #22):
- All 10 composes now mount torch_compile_cache + Triton cache from
`models/qwen3.6-27b/vllm/cache/`. First boot warms (~6 min); warm boot
drops to ~3.2 min (47% faster). Per-stage savings on long-text:
- Dynamo bytecode transform: 18s → 5s (-73%)
- torch.compile: 57s → 9s (-85%)
- Initial profiling/warmup: 51s → 7s (-87%)
Mamba block_size cap fix:
- v0.20 enforces `long_prefill_token_threshold >= block_size`; on hybrid
Mamba+TQ3, vLLM forces block_size=4128. Bumped GENESIS_PROFILE_RUN_CAP_M
and PREALLOC_TOKEN_BUDGET 4096→4128 across all 5 main composes.
Default 48K compose:
- Required workspace_lock_disable sidecar after initial v0.20 boot hit
vllm#39226 strict assertion. Caught during validation, fixed.
Context restored vs dev205 backoffs (validated 33K + 50K stress on v0.20):
- long-text: 185K → 214K (+16%)
- long-vision: 140K → 198K (+41%)
- bounded-thinking: 185K → 214K (+16%)
Bench results (n=5, results/v0.20-migration/):
- long-text 214K narr 49.74 / code 67.39 (CV 2.6/2.7%)
- long-vision 198K narr 50.32 / code 66.12 (CV 2.3/4.1%)
- bounded-thinking 214K narr 49.77 / code 65.80 (CV 1.4/2.3%)
- tools-text 75K (fp8) narr 53.32 / code 69.66 (CV 2.3/1.4%)
- dual-turbo 262K (TP=2) narr 58.33 / code 76.01 per-stream
269 TPS aggregate at n=4 streams (3.63x speedup)
- default 48K narr 48.82 / code 65.98 (n=3)
Validation: verify-full 8/8 on every variant. verify-stress 33K AND 50K
tool-prefill PASS on every variant — the cliff that fired on EVERY dev205
config no longer reproduces.
Docs + charts:
- README + SINGLE_CARD + DUAL_CARD + CLIFFS + EXAMPLES + STRUCTURED_COT
+ FAQ + UPSTREAM + 3 engine docs + model README + INTERNALS + CHANGELOG
all updated with new pin, ctx, TPS numbers, and "v0.20 unblock" section
- performance.{png,svg} + variants regenerated with measured TPS
- vram-budget.{png,svg} + variants regenerated with measured VRAM
- UPSTREAM tracker: 5 issues moved ✅ closed (PR #12, #13, #14, #15, P104
superseded by PN17 + P15B)
Issues addressed:
- #16 (Cliff 1 mech B leaks past PN12 on inductor-compiled FFN) — partial:
v0.20's revised TQ FA paths close the synthetic stress; PN25 (Sander's
proper compile-path opaque-op fix) is on dev but explicitly opt-in pending
worker-fork registration fix. Workarounds documented (tools-text fp8 path
/ --enforce-eager) until Sander ships PN25 default-on.
- #20 (launch.sh port + container-name mismatch) — already closed by
77ca576 (post-issue-filing).
- #22 (cold-start caching) — closed by cache mounts above.
Remaining caveats:
- Cliff 2 (DeltaNet GDN forward, single prompt ≥50-60K) unchanged —
architectural, applies to all single-card vLLM TQ3 paths. Mitigation:
dual-turbo TP=2 (state splits across cards) or llama.cpp 262K.
- Default 48K narr_TPS (48.82) slightly under chart's 55 reference —
bench variance + sample size n=3; not regression.
- Dual.yml / dual-dflash* not re-benched on v0.20; numbers carry forward
from dev205 (fp8 paths were not TPS-changed by the migration).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
42 lines
2.8 KiB
Plaintext
42 lines
2.8 KiB
Plaintext
|
|
========== NARRATIVE (prompt=65 chars, max_tokens=1000) ==========
|
|
=== warmups (3) ===
|
|
warm-1 wall= 25.85s ttft= 5813ms toks=1000 wall_TPS= 38.68 decode_TPS= 49.90
|
|
warm-2 wall= 20.28s ttft= 157ms toks=1000 wall_TPS= 49.31 decode_TPS= 49.70
|
|
warm-3 wall= 19.62s ttft= 166ms toks=1000 wall_TPS= 50.96 decode_TPS= 51.39
|
|
|
|
=== measured (3) ===
|
|
run-1 wall= 20.60s ttft= 159ms toks=1000 wall_TPS= 48.55 decode_TPS= 48.93
|
|
run-2 wall= 20.36s ttft= 160ms toks=1000 wall_TPS= 49.11 decode_TPS= 49.50
|
|
run-3 wall= 20.49s ttft= 156ms toks=1000 wall_TPS= 48.82 decode_TPS= 49.19
|
|
|
|
=== summary [narrative] (n=3) ===
|
|
wall_TPS mean= 48.82 std= 0.28 CV= 0.6% min=48.55 max=49.11
|
|
decode_TPS mean= 49.20 std= 0.28 CV= 0.6% min=48.93 max=49.50
|
|
TTFT mean= 158ms std= 2ms min=156ms max=160ms
|
|
|
|
========== CODE (prompt=78 chars, max_tokens=800) ==========
|
|
=== warmups (3) ===
|
|
warm-1 wall= 7.47s ttft= 157ms toks= 483 wall_TPS= 64.63 decode_TPS= 66.02
|
|
warm-2 wall= 9.92s ttft= 162ms toks= 641 wall_TPS= 64.63 decode_TPS= 65.70
|
|
warm-3 wall= 12.24s ttft= 157ms toks= 800 wall_TPS= 65.34 decode_TPS= 66.19
|
|
|
|
=== measured (3) ===
|
|
run-1 wall= 11.10s ttft= 155ms toks= 704 wall_TPS= 63.44 decode_TPS= 64.34
|
|
run-2 wall= 11.39s ttft= 156ms toks= 771 wall_TPS= 67.71 decode_TPS= 68.65
|
|
run-3 wall= 11.98s ttft= 161ms toks= 800 wall_TPS= 66.78 decode_TPS= 67.69
|
|
|
|
=== summary [code] (n=3) ===
|
|
wall_TPS mean= 65.98 std= 2.24 CV= 3.4% min=63.44 max=67.71
|
|
decode_TPS mean= 66.89 std= 2.26 CV= 3.4% min=64.34 max=68.65
|
|
TTFT mean= 157ms std= 4ms min=155ms max=161ms
|
|
|
|
=== GPU state ===
|
|
0, 92 %, 20826 MiB, 24576 MiB, 228.71 W, 65
|
|
1, 0 %, 4 MiB, 24576 MiB, 22.25 W, 45
|
|
|
|
=== Last 3 SpecDecoding metrics ===
|
|
(APIServer pid=1) INFO 05-01 17:41:54 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.34, Accepted throughput: 44.69 tokens/s, Drafted throughput: 57.29 tokens/s, Accepted: 447 tokens, Drafted: 573 tokens, Per-position acceptance rate: 0.916, 0.764, 0.660, Avg Draft acceptance rate: 78.0%
|
|
(APIServer pid=1) INFO 05-01 17:42:04 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.45, Accepted throughput: 46.70 tokens/s, Drafted throughput: 57.30 tokens/s, Accepted: 467 tokens, Drafted: 573 tokens, Per-position acceptance rate: 0.953, 0.832, 0.660, Avg Draft acceptance rate: 81.5%
|
|
(APIServer pid=1) INFO 05-01 17:42:14 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.47, Accepted throughput: 46.90 tokens/s, Drafted throughput: 57.00 tokens/s, Accepted: 469 tokens, Drafted: 570 tokens, Per-position acceptance rate: 0.958, 0.826, 0.684, Avg Draft acceptance rate: 82.3%
|