5aa97a25d9
This branch migrates the entire vLLM stack from `dev205+g07351e088` + Genesis
v7.64 to `0.20.1rc1.dev16+g7a1eb8ac2` + Genesis v7.65 dev tip (commit
`d89a089`). v7.65 is on Sandermage's `dev` branch — explicitly the cross-rig
testing surface he requested in discussion #19; he'll merge dev→main once we
both confirm stable. Pin gates restated when that lands.
What changes
------------
Pin migration:
- vLLM image: nightly-07351e08... → nightly-7a1eb8ac2... (dev205 → v0.20.1rc1.dev16)
- Genesis: 64dd18b (v7.64) → d89a089 (v7.65 dev tip)
Sidecar churn:
- DROPPED: patch_pn12_ffn_pool_anchor.py (PN12 native on v0.20)
- DROPPED: patch_pn12_compile_safe_custom_op.py (Genesis P38B in-source hook)
- DROPPED: patch_fa_max_seqlen_clamp.py (Genesis PN17 + P15B)
- ADDED: patch_workspace_lock_disable.py (relaxes vllm#39226 strict assertion;
P98 covers same surface but auto-skips on v0.20 due to drift-marker false
positive — pending Sandermage marker fix)
Env-var alignment to Sandermage's PROD set (start_27b_int4_TQ_k8v4.sh@dev):
- FIXED naming bugs that silently no-op'd patches:
- PN9_INDEPENDENT_DRAFTER_ATT → _ATTN (was silently OFF)
- PN22 → PN22_LOCAL_ARGMAX_TP (was silently OFF)
- PN26_BLOCK_KV → PN26_SPARSE_V_BLOCK_KV (fell back to default 4, not 8)
- PN26_NUM_WARPS → PN26_SPARSE_V_NUM_WARPS
- PN26_THRESHOLD → PN26_SPARSE_V_THRESHOLD (fell back to default 0.001, not 0.01)
- ADDED explicit-OFFs to match Sander's PROD verbatim:
- P78_TOLIST_CAPTURE_GUARD=0 (we use our own patch_tolist_cudagraph.py)
- P81_FP8_BLOCK_SCALED_M_LE_8=0 (FP8-specific, no-op on TQ3)
- P82=0, P82_THRESHOLD_SINGLE=0.3
- Cap divergence (justified): PROFILE_RUN_CAP_M=4128 + PREALLOC_TOKEN_BUDGET=4128
(Sander uses 4096 — vLLM `interface.py:639` forces our config's Mamba
block_size to 4128 due to TQ3 + TP=1 page-size math; lower values
AssertionError at boot)
- Carry-forward (intentional): P4 (hybrid TQ required), P65 (TQ spec-CG
downgrade — pending v0.20 verification that #40880 closure makes it
redundant)
Cold-start cache mounts (closes #22):
- All 10 composes now mount torch_compile_cache + Triton cache from
`models/qwen3.6-27b/vllm/cache/`. First boot warms (~6 min); warm boot
drops to ~3.2 min (47% faster). Per-stage savings on long-text:
- Dynamo bytecode transform: 18s → 5s (-73%)
- torch.compile: 57s → 9s (-85%)
- Initial profiling/warmup: 51s → 7s (-87%)
Mamba block_size cap fix:
- v0.20 enforces `long_prefill_token_threshold >= block_size`; on hybrid
Mamba+TQ3, vLLM forces block_size=4128. Bumped GENESIS_PROFILE_RUN_CAP_M
and PREALLOC_TOKEN_BUDGET 4096→4128 across all 5 main composes.
Default 48K compose:
- Required workspace_lock_disable sidecar after initial v0.20 boot hit
vllm#39226 strict assertion. Caught during validation, fixed.
Context restored vs dev205 backoffs (validated 33K + 50K stress on v0.20):
- long-text: 185K → 214K (+16%)
- long-vision: 140K → 198K (+41%)
- bounded-thinking: 185K → 214K (+16%)
Bench results (n=5, results/v0.20-migration/):
- long-text 214K narr 49.74 / code 67.39 (CV 2.6/2.7%)
- long-vision 198K narr 50.32 / code 66.12 (CV 2.3/4.1%)
- bounded-thinking 214K narr 49.77 / code 65.80 (CV 1.4/2.3%)
- tools-text 75K (fp8) narr 53.32 / code 69.66 (CV 2.3/1.4%)
- dual-turbo 262K (TP=2) narr 58.33 / code 76.01 per-stream
269 TPS aggregate at n=4 streams (3.63x speedup)
- default 48K narr 48.82 / code 65.98 (n=3)
Validation: verify-full 8/8 on every variant. verify-stress 33K AND 50K
tool-prefill PASS on every variant — the cliff that fired on EVERY dev205
config no longer reproduces.
Docs + charts:
- README + SINGLE_CARD + DUAL_CARD + CLIFFS + EXAMPLES + STRUCTURED_COT
+ FAQ + UPSTREAM + 3 engine docs + model README + INTERNALS + CHANGELOG
all updated with new pin, ctx, TPS numbers, and "v0.20 unblock" section
- performance.{png,svg} + variants regenerated with measured TPS
- vram-budget.{png,svg} + variants regenerated with measured VRAM
- UPSTREAM tracker: 5 issues moved ✅ closed (PR #12, #13, #14, #15, P104
superseded by PN17 + P15B)
Issues addressed:
- #16 (Cliff 1 mech B leaks past PN12 on inductor-compiled FFN) — partial:
v0.20's revised TQ FA paths close the synthetic stress; PN25 (Sander's
proper compile-path opaque-op fix) is on dev but explicitly opt-in pending
worker-fork registration fix. Workarounds documented (tools-text fp8 path
/ --enforce-eager) until Sander ships PN25 default-on.
- #20 (launch.sh port + container-name mismatch) — already closed by
77ca576 (post-issue-filing).
- #22 (cold-start caching) — closed by cache mounts above.
Remaining caveats:
- Cliff 2 (DeltaNet GDN forward, single prompt ≥50-60K) unchanged —
architectural, applies to all single-card vLLM TQ3 paths. Mitigation:
dual-turbo TP=2 (state splits across cards) or llama.cpp 262K.
- Default 48K narr_TPS (48.82) slightly under chart's 55 reference —
bench variance + sample size n=3; not regression.
- Dual.yml / dual-dflash* not re-benched on v0.20; numbers carry forward
from dev205 (fp8 paths were not TPS-changed by the migration).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
46 lines
3.1 KiB
Plaintext
46 lines
3.1 KiB
Plaintext
|
|
========== NARRATIVE (prompt=65 chars, max_tokens=1000) ==========
|
|
=== warmups (3) ===
|
|
warm-1 wall= 24.02s ttft= 5437ms toks=1000 wall_TPS= 41.63 decode_TPS= 53.81
|
|
warm-2 wall= 18.45s ttft= 157ms toks=1000 wall_TPS= 54.20 decode_TPS= 54.67
|
|
warm-3 wall= 19.45s ttft= 156ms toks= 998 wall_TPS= 51.32 decode_TPS= 51.74
|
|
|
|
=== measured (5) ===
|
|
run-1 wall= 19.98s ttft= 159ms toks=1000 wall_TPS= 50.04 decode_TPS= 50.44
|
|
run-2 wall= 19.77s ttft= 154ms toks=1000 wall_TPS= 50.58 decode_TPS= 50.97
|
|
run-3 wall= 19.77s ttft= 160ms toks=1000 wall_TPS= 50.57 decode_TPS= 50.98
|
|
run-4 wall= 19.31s ttft= 159ms toks=1000 wall_TPS= 51.79 decode_TPS= 52.22
|
|
run-5 wall= 20.57s ttft= 165ms toks=1000 wall_TPS= 48.61 decode_TPS= 49.00
|
|
|
|
=== summary [narrative] (n=5) ===
|
|
wall_TPS mean= 50.32 std= 1.15 CV= 2.3% min=48.61 max=51.79
|
|
decode_TPS mean= 50.72 std= 1.16 CV= 2.3% min=49.00 max=52.22
|
|
TTFT mean= 159ms std= 4ms min=154ms max=165ms
|
|
|
|
========== CODE (prompt=78 chars, max_tokens=800) ==========
|
|
=== warmups (3) ===
|
|
warm-1 wall= 12.03s ttft= 160ms toks= 800 wall_TPS= 66.49 decode_TPS= 67.39
|
|
warm-2 wall= 9.56s ttft= 156ms toks= 639 wall_TPS= 66.83 decode_TPS= 67.93
|
|
warm-3 wall= 12.04s ttft= 162ms toks= 800 wall_TPS= 66.46 decode_TPS= 67.36
|
|
|
|
=== measured (5) ===
|
|
run-1 wall= 10.34s ttft= 163ms toks= 699 wall_TPS= 67.60 decode_TPS= 68.68
|
|
run-2 wall= 11.98s ttft= 156ms toks= 800 wall_TPS= 66.75 decode_TPS= 67.63
|
|
run-3 wall= 7.47s ttft= 157ms toks= 458 wall_TPS= 61.31 decode_TPS= 62.62
|
|
run-4 wall= 11.77s ttft= 156ms toks= 800 wall_TPS= 67.98 decode_TPS= 68.89
|
|
run-5 wall= 10.70s ttft= 159ms toks= 717 wall_TPS= 66.98 decode_TPS= 67.99
|
|
|
|
=== summary [code] (n=5) ===
|
|
wall_TPS mean= 66.12 std= 2.74 CV= 4.1% min=61.31 max=67.98
|
|
decode_TPS mean= 67.16 std= 2.59 CV= 3.9% min=62.62 max=68.89
|
|
TTFT mean= 158ms std= 3ms min=156ms max=163ms
|
|
|
|
=== GPU state ===
|
|
0, 98 %, 22326 MiB, 24576 MiB, 229.16 W, 62
|
|
1, 0 %, 4 MiB, 24576 MiB, 21.48 W, 42
|
|
|
|
=== Last 3 SpecDecoding metrics ===
|
|
(APIServer pid=1) INFO 05-01 16:27:27 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.45, Accepted throughput: 46.59 tokens/s, Drafted throughput: 56.99 tokens/s, Accepted: 466 tokens, Drafted: 570 tokens, Per-position acceptance rate: 0.942, 0.816, 0.695, Avg Draft acceptance rate: 81.8%
|
|
(APIServer pid=1) INFO 05-01 16:27:37 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.40, Accepted throughput: 45.80 tokens/s, Drafted throughput: 57.30 tokens/s, Accepted: 458 tokens, Drafted: 573 tokens, Per-position acceptance rate: 0.921, 0.822, 0.654, Avg Draft acceptance rate: 79.9%
|
|
(APIServer pid=1) INFO 05-01 16:27:47 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.56, Accepted throughput: 48.60 tokens/s, Drafted throughput: 57.00 tokens/s, Accepted: 486 tokens, Drafted: 570 tokens, Per-position acceptance rate: 0.937, 0.874, 0.747, Avg Draft acceptance rate: 85.3%
|