5aa97a25d9
This branch migrates the entire vLLM stack from `dev205+g07351e088` + Genesis
v7.64 to `0.20.1rc1.dev16+g7a1eb8ac2` + Genesis v7.65 dev tip (commit
`d89a089`). v7.65 is on Sandermage's `dev` branch — explicitly the cross-rig
testing surface he requested in discussion #19; he'll merge dev→main once we
both confirm stable. Pin gates restated when that lands.
What changes
------------
Pin migration:
- vLLM image: nightly-07351e08... → nightly-7a1eb8ac2... (dev205 → v0.20.1rc1.dev16)
- Genesis: 64dd18b (v7.64) → d89a089 (v7.65 dev tip)
Sidecar churn:
- DROPPED: patch_pn12_ffn_pool_anchor.py (PN12 native on v0.20)
- DROPPED: patch_pn12_compile_safe_custom_op.py (Genesis P38B in-source hook)
- DROPPED: patch_fa_max_seqlen_clamp.py (Genesis PN17 + P15B)
- ADDED: patch_workspace_lock_disable.py (relaxes vllm#39226 strict assertion;
P98 covers same surface but auto-skips on v0.20 due to drift-marker false
positive — pending Sandermage marker fix)
Env-var alignment to Sandermage's PROD set (start_27b_int4_TQ_k8v4.sh@dev):
- FIXED naming bugs that silently no-op'd patches:
- PN9_INDEPENDENT_DRAFTER_ATT → _ATTN (was silently OFF)
- PN22 → PN22_LOCAL_ARGMAX_TP (was silently OFF)
- PN26_BLOCK_KV → PN26_SPARSE_V_BLOCK_KV (fell back to default 4, not 8)
- PN26_NUM_WARPS → PN26_SPARSE_V_NUM_WARPS
- PN26_THRESHOLD → PN26_SPARSE_V_THRESHOLD (fell back to default 0.001, not 0.01)
- ADDED explicit-OFFs to match Sander's PROD verbatim:
- P78_TOLIST_CAPTURE_GUARD=0 (we use our own patch_tolist_cudagraph.py)
- P81_FP8_BLOCK_SCALED_M_LE_8=0 (FP8-specific, no-op on TQ3)
- P82=0, P82_THRESHOLD_SINGLE=0.3
- Cap divergence (justified): PROFILE_RUN_CAP_M=4128 + PREALLOC_TOKEN_BUDGET=4128
(Sander uses 4096 — vLLM `interface.py:639` forces our config's Mamba
block_size to 4128 due to TQ3 + TP=1 page-size math; lower values
AssertionError at boot)
- Carry-forward (intentional): P4 (hybrid TQ required), P65 (TQ spec-CG
downgrade — pending v0.20 verification that #40880 closure makes it
redundant)
Cold-start cache mounts (closes #22):
- All 10 composes now mount torch_compile_cache + Triton cache from
`models/qwen3.6-27b/vllm/cache/`. First boot warms (~6 min); warm boot
drops to ~3.2 min (47% faster). Per-stage savings on long-text:
- Dynamo bytecode transform: 18s → 5s (-73%)
- torch.compile: 57s → 9s (-85%)
- Initial profiling/warmup: 51s → 7s (-87%)
Mamba block_size cap fix:
- v0.20 enforces `long_prefill_token_threshold >= block_size`; on hybrid
Mamba+TQ3, vLLM forces block_size=4128. Bumped GENESIS_PROFILE_RUN_CAP_M
and PREALLOC_TOKEN_BUDGET 4096→4128 across all 5 main composes.
Default 48K compose:
- Required workspace_lock_disable sidecar after initial v0.20 boot hit
vllm#39226 strict assertion. Caught during validation, fixed.
Context restored vs dev205 backoffs (validated 33K + 50K stress on v0.20):
- long-text: 185K → 214K (+16%)
- long-vision: 140K → 198K (+41%)
- bounded-thinking: 185K → 214K (+16%)
Bench results (n=5, results/v0.20-migration/):
- long-text 214K narr 49.74 / code 67.39 (CV 2.6/2.7%)
- long-vision 198K narr 50.32 / code 66.12 (CV 2.3/4.1%)
- bounded-thinking 214K narr 49.77 / code 65.80 (CV 1.4/2.3%)
- tools-text 75K (fp8) narr 53.32 / code 69.66 (CV 2.3/1.4%)
- dual-turbo 262K (TP=2) narr 58.33 / code 76.01 per-stream
269 TPS aggregate at n=4 streams (3.63x speedup)
- default 48K narr 48.82 / code 65.98 (n=3)
Validation: verify-full 8/8 on every variant. verify-stress 33K AND 50K
tool-prefill PASS on every variant — the cliff that fired on EVERY dev205
config no longer reproduces.
Docs + charts:
- README + SINGLE_CARD + DUAL_CARD + CLIFFS + EXAMPLES + STRUCTURED_COT
+ FAQ + UPSTREAM + 3 engine docs + model README + INTERNALS + CHANGELOG
all updated with new pin, ctx, TPS numbers, and "v0.20 unblock" section
- performance.{png,svg} + variants regenerated with measured TPS
- vram-budget.{png,svg} + variants regenerated with measured VRAM
- UPSTREAM tracker: 5 issues moved ✅ closed (PR #12, #13, #14, #15, P104
superseded by PN17 + P15B)
Issues addressed:
- #16 (Cliff 1 mech B leaks past PN12 on inductor-compiled FFN) — partial:
v0.20's revised TQ FA paths close the synthetic stress; PN25 (Sander's
proper compile-path opaque-op fix) is on dev but explicitly opt-in pending
worker-fork registration fix. Workarounds documented (tools-text fp8 path
/ --enforce-eager) until Sander ships PN25 default-on.
- #20 (launch.sh port + container-name mismatch) — already closed by
77ca576 (post-issue-filing).
- #22 (cold-start caching) — closed by cache mounts above.
Remaining caveats:
- Cliff 2 (DeltaNet GDN forward, single prompt ≥50-60K) unchanged —
architectural, applies to all single-card vLLM TQ3 paths. Mitigation:
dual-turbo TP=2 (state splits across cards) or llama.cpp 262K.
- Default 48K narr_TPS (48.82) slightly under chart's 55 reference —
bench variance + sample size n=3; not regression.
- Dual.yml / dual-dflash* not re-benched on v0.20; numbers carry forward
from dev205 (fp8 paths were not TPS-changed by the migration).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
46 lines
3.1 KiB
Plaintext
46 lines
3.1 KiB
Plaintext
|
|
========== NARRATIVE (prompt=65 chars, max_tokens=1000) ==========
|
|
=== warmups (3) ===
|
|
warm-1 wall= 34.49s ttft= 14153ms toks=1000 wall_TPS= 28.99 decode_TPS= 49.17
|
|
warm-2 wall= 19.52s ttft= 152ms toks=1000 wall_TPS= 51.23 decode_TPS= 51.64
|
|
warm-3 wall= 19.46s ttft= 159ms toks=1000 wall_TPS= 51.38 decode_TPS= 51.80
|
|
|
|
=== measured (5) ===
|
|
run-1 wall= 20.56s ttft= 157ms toks=1000 wall_TPS= 48.64 decode_TPS= 49.01
|
|
run-2 wall= 19.92s ttft= 154ms toks=1000 wall_TPS= 50.21 decode_TPS= 50.60
|
|
run-3 wall= 19.34s ttft= 150ms toks=1000 wall_TPS= 51.70 decode_TPS= 52.10
|
|
run-4 wall= 20.20s ttft= 160ms toks=1000 wall_TPS= 49.50 decode_TPS= 49.90
|
|
run-5 wall= 20.56s ttft= 148ms toks=1000 wall_TPS= 48.65 decode_TPS= 49.00
|
|
|
|
=== summary [narrative] (n=5) ===
|
|
wall_TPS mean= 49.74 std= 1.28 CV= 2.6% min=48.64 max=51.70
|
|
decode_TPS mean= 50.12 std= 1.29 CV= 2.6% min=49.00 max=52.10
|
|
TTFT mean= 154ms std= 5ms min=148ms max=160ms
|
|
|
|
========== CODE (prompt=78 chars, max_tokens=800) ==========
|
|
=== warmups (3) ===
|
|
warm-1 wall= 12.50s ttft= 149ms toks= 800 wall_TPS= 64.02 decode_TPS= 64.79
|
|
warm-2 wall= 10.05s ttft= 152ms toks= 669 wall_TPS= 66.59 decode_TPS= 67.61
|
|
warm-3 wall= 8.75s ttft= 158ms toks= 564 wall_TPS= 64.43 decode_TPS= 65.61
|
|
|
|
=== measured (5) ===
|
|
run-1 wall= 11.73s ttft= 155ms toks= 800 wall_TPS= 68.22 decode_TPS= 69.13
|
|
run-2 wall= 10.54s ttft= 155ms toks= 734 wall_TPS= 69.65 decode_TPS= 70.69
|
|
run-3 wall= 10.18s ttft= 151ms toks= 693 wall_TPS= 68.09 decode_TPS= 69.12
|
|
run-4 wall= 12.22s ttft= 153ms toks= 797 wall_TPS= 65.20 decode_TPS= 66.03
|
|
run-5 wall= 12.16s ttft= 158ms toks= 800 wall_TPS= 65.80 decode_TPS= 66.67
|
|
|
|
=== summary [code] (n=5) ===
|
|
wall_TPS mean= 67.39 std= 1.85 CV= 2.7% min=65.20 max=69.65
|
|
decode_TPS mean= 68.33 std= 1.93 CV= 2.8% min=66.03 max=70.69
|
|
TTFT mean= 155ms std= 3ms min=151ms max=158ms
|
|
|
|
=== GPU state ===
|
|
0, 98 %, 23404 MiB, 24576 MiB, 229.10 W, 62
|
|
1, 0 %, 4 MiB, 24576 MiB, 21.92 W, 42
|
|
|
|
=== Last 3 SpecDecoding metrics ===
|
|
(APIServer pid=1) INFO 05-01 16:10:36 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.51, Accepted throughput: 48.40 tokens/s, Drafted throughput: 57.89 tokens/s, Accepted: 484 tokens, Drafted: 579 tokens, Per-position acceptance rate: 0.953, 0.865, 0.689, Avg Draft acceptance rate: 83.6%
|
|
(APIServer pid=1) INFO 05-01 16:10:46 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.42, Accepted throughput: 47.00 tokens/s, Drafted throughput: 58.20 tokens/s, Accepted: 470 tokens, Drafted: 582 tokens, Per-position acceptance rate: 0.933, 0.814, 0.675, Avg Draft acceptance rate: 80.8%
|
|
(APIServer pid=1) INFO 05-01 16:10:56 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.34, Accepted throughput: 45.29 tokens/s, Drafted throughput: 58.19 tokens/s, Accepted: 453 tokens, Drafted: 582 tokens, Per-position acceptance rate: 0.887, 0.799, 0.649, Avg Draft acceptance rate: 77.8%
|