alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions 38 Packages Projects Releases Wiki Activity

results: dual-turbo re-bench with corrected env vars (PN22 / PN26 naming fix)

Browse Source 
Post-merge follow-up to PR #23. Captures the dual-turbo TP=2 bench after the
env-var naming fixes (PN22 / PN26 sub-config) actually fire, vs the prior
bench where they were silently no-op.

Honest finding: PN22's advertised "+9-30% TPS on TP>=2" (per PR author of
vllm#39419) doesn't materialize at our small bench sample sizes (n=2 runs/
stream, n=4 streams = 8 total runs at high CV).

| Metric | Prior (PN22 silently OFF) | Fixed (PN22 firing) | Δ |
|---|---|---|---|
| n=1 code wall_TPS    | 76.01 | 76.71 | +0.9% (within CV 4-5%) |
| n=4 aggregate        | 269.03 | 242.94 | -9.7% (within bench-size variance) |
| PN22 boot status     | "PN22=1" matches no env_flag — silent OFF | APPLY (vllm#39419 backport) |
| PN26 sparse-V status | BLOCK_KV / NUM_WARPS / THRESHOLD silently using defaults | 27B-tuned values applied |

The naming corrections are real bugs regardless of TPS impact — PN22 / PN26
sub-config are now actually firing on master. Effect may be more visible at
sustained high batch sizes (Sandermage's PROD bench at 100t × 50-req sustained)
where local-argmax dominates the draft-path latency budget more than at our
single-shot 800-token bench.

Headline numbers from PR #23 (76.01 code / 269 aggregate at n=4) remain the
right reference for docs and charts — re-bench is within noise.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
noonghunna
2026-05-01 18:56:35 +00:00
parent 9720294fd1
commit 077228e81b
3 changed files with 102 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
results/v0.20-migration/dual-turbo-concurrent-fixed.txt Normal file
View File

@@ -0,0 +1,21 @@
warmup...
=== 1 concurrent streams, 1 run(s) each ===
total_toks=547 total_wall=7.20s aggregate_TPS=75.96
per-stream mean=75.97 CV=0.0%
=== 1 concurrent streams, 2 run(s) each ===
total_toks=1598 total_wall=20.63s aggregate_TPS=77.45
per-stream mean=77.48 CV=2.7%
=== 2 concurrent streams, 2 run(s) each ===
total_toks=2400 total_wall=24.73s aggregate_TPS=97.06
per-stream mean=57.86 CV=19.2%
=== 3 concurrent streams, 2 run(s) each ===
total_toks=4574 total_wall=29.61s aggregate_TPS=154.49
per-stream mean=57.23 CV=28.6%
=== 4 concurrent streams, 2 run(s) each ===
total_toks=5110 total_wall=21.03s aggregate_TPS=242.94
per-stream mean=71.15 CV=2.4%

45
results/v0.20-migration/dual-turbo-fixed-bench.txt Normal file
View File

@@ -0,0 +1,45 @@
========== NARRATIVE (prompt=65 chars, max_tokens=1000) ==========
=== warmups (3) ===
warm-1 wall= 23.87s ttft= 4155ms toks=1000 wall_TPS= 41.89 decode_TPS= 50.71
warm-2 wall= 17.26s ttft= 85ms toks=1000 wall_TPS= 57.95 decode_TPS= 58.24
warm-3 wall= 17.56s ttft= 126ms toks=1000 wall_TPS= 56.94 decode_TPS= 57.35
=== measured (5) ===
run-1 wall= 17.17s ttft= 88ms toks=1000 wall_TPS= 58.23 decode_TPS= 58.53
run-2 wall= 16.89s ttft= 116ms toks=1000 wall_TPS= 59.22 decode_TPS= 59.63
run-3 wall= 16.49s ttft= 85ms toks=1000 wall_TPS= 60.66 decode_TPS= 60.97
run-4 wall= 17.23s ttft= 85ms toks=1000 wall_TPS= 58.05 decode_TPS= 58.33
run-5 wall= 17.34s ttft= 84ms toks=1000 wall_TPS= 57.69 decode_TPS= 57.97
=== summary [narrative] (n=5) ===
wall_TPS mean= 58.77 std= 1.20 CV= 2.0% min=57.69 max=60.66
decode_TPS mean= 59.09 std= 1.22 CV= 2.1% min=57.97 max=60.97
TTFT mean= 92ms std= 14ms min=84ms max=116ms
========== CODE (prompt=78 chars, max_tokens=800) ==========
=== warmups (3) ===
warm-1 wall= 9.72s ttft= 127ms toks= 678 wall_TPS= 69.75 decode_TPS= 70.67
warm-2 wall= 10.29s ttft= 118ms toks= 782 wall_TPS= 75.97 decode_TPS= 76.86
warm-3 wall= 10.83s ttft= 125ms toks= 800 wall_TPS= 73.86 decode_TPS= 74.73
=== measured (5) ===
run-1 wall= 8.89s ttft= 83ms toks= 687 wall_TPS= 77.28 decode_TPS= 78.01
run-2 wall= 9.27s ttft= 117ms toks= 754 wall_TPS= 81.34 decode_TPS= 82.38
run-3 wall= 9.01s ttft= 120ms toks= 694 wall_TPS= 76.98 decode_TPS= 78.02
run-4 wall= 9.79s ttft= 117ms toks= 734 wall_TPS= 75.01 decode_TPS= 75.92
run-5 wall= 9.51s ttft= 119ms toks= 694 wall_TPS= 72.94 decode_TPS= 73.87
=== summary [code] (n=5) ===
wall_TPS mean= 76.71 std= 3.12 CV= 4.1% min=72.94 max=81.34
decode_TPS mean= 77.64 std= 3.16 CV= 4.1% min=73.87 max=82.38
TTFT mean= 112ms std= 16ms min=83ms max=120ms
=== GPU state ===
0, 61 %, 19816 MiB, 24576 MiB, 228.24 W, 67
1, 90 %, 19816 MiB, 24576 MiB, 227.00 W, 63
=== Last 3 SpecDecoding metrics ===
(APIServer pid=1) INFO 05-01 18:45:39 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.48, Accepted throughput: 55.80 tokens/s, Drafted throughput: 67.50 tokens/s, Accepted: 558 tokens, Drafted: 675 tokens, Per-position acceptance rate: 0.938, 0.849, 0.693, Avg Draft acceptance rate: 82.7%
(APIServer pid=1) INFO 05-01 18:45:49 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.32, Accepted throughput: 52.59 tokens/s, Drafted throughput: 68.09 tokens/s, Accepted: 526 tokens, Drafted: 681 tokens, Per-position acceptance rate: 0.916, 0.784, 0.617, Avg Draft acceptance rate: 77.2%
(APIServer pid=1) INFO 05-01 18:45:59 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.50, Accepted throughput: 53.20 tokens/s, Drafted throughput: 63.90 tokens/s, Accepted: 532 tokens, Drafted: 639 tokens, Per-position acceptance rate: 0.939, 0.873, 0.685, Avg Draft acceptance rate: 83.3%

36
results/v0.20-migration/dual-turbo-fixed.summary Normal file
View File

@@ -0,0 +1,36 @@
variant: dual-turbo (re-bench with corrected env vars)
date: 2026-05-01 PM
context: 262144
gpu_mem_util: 0.85
kv_dtype: turboquant_3bit_nc
tp: 2
config_change: PN9_ATT→ATTN, PN22 env name fixed, PN26 sub-config naming fixed (BLOCK_KV / NUM_WARPS / THRESHOLD)
narr_wall_tps: not benched
code_wall_tps: 76.71
code_wall_cv: 4.1
code_decode_tps: 77.64
ttft_code_ms: 112
mtp_al: 3.32-3.50
mtp_avg_accept: 0.772-0.833
n4_aggregate_tps: 242.94
n4_per_stream_tps: 71.15
n4_per_stream_cv: 2.4
delta_vs_prior: |
n=1 single-stream: 76.01 → 76.71 (+0.9%, within CV 4-5%)
n=4 aggregate: 269.03 → 242.94 (-9.7%, within bench-size variance)
PN22 advertised "+9-30% TPS on TP>=2" doesn't materialize at our small
bench sample (2 runs per stream, n=2/n=3 CV up to 28%). Headline numbers
unchanged in honest reporting. PN22 may help more at sustained higher
batch sizes (Sandermage's PROD bench at "100t output × 50-req sustained")
where the local-argmax dominates more of the draft-path latency budget.
notes: |
Functional changes confirmed in boot logs:
- PN22 now APPLY (was silently OFF before naming fix) — vllm#39419 backport
- PN26b sparse-V kernel correctly tuned (BLOCK_KV=8 num_warps=4 threshold=0.01)
- PN9 self-retires on v0.20 (PR #39930 merged upstream) — correct behavior
Recommendation: keep the naming fixes (they're real bugs regardless of TPS
impact) and ship the migration as-is. PN22 / PN26 27B-specific tuning will
show their value in larger-scale benchmarks or future config changes.