077228e81b
Post-merge follow-up to PR #23. Captures the dual-turbo TP=2 bench after the env-var naming fixes (PN22 / PN26 sub-config) actually fire, vs the prior bench where they were silently no-op. Honest finding: PN22's advertised "+9-30% TPS on TP>=2" (per PR author of vllm#39419) doesn't materialize at our small bench sample sizes (n=2 runs/ stream, n=4 streams = 8 total runs at high CV). | Metric | Prior (PN22 silently OFF) | Fixed (PN22 firing) | Δ | |---|---|---|---| | n=1 code wall_TPS | 76.01 | 76.71 | +0.9% (within CV 4-5%) | | n=4 aggregate | 269.03 | 242.94 | -9.7% (within bench-size variance) | | PN22 boot status | "PN22=1" matches no env_flag — silent OFF | APPLY (vllm#39419 backport) | | PN26 sparse-V status | BLOCK_KV / NUM_WARPS / THRESHOLD silently using defaults | 27B-tuned values applied | The naming corrections are real bugs regardless of TPS impact — PN22 / PN26 sub-config are now actually firing on master. Effect may be more visible at sustained high batch sizes (Sandermage's PROD bench at 100t × 50-req sustained) where local-argmax dominates the draft-path latency budget more than at our single-shot 800-token bench. Headline numbers from PR #23 (76.01 code / 269 aggregate at n=4) remain the right reference for docs and charts — re-bench is within noise. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
37 lines
1.5 KiB
Plaintext
37 lines
1.5 KiB
Plaintext
variant: dual-turbo (re-bench with corrected env vars)
|
||
date: 2026-05-01 PM
|
||
context: 262144
|
||
gpu_mem_util: 0.85
|
||
kv_dtype: turboquant_3bit_nc
|
||
tp: 2
|
||
config_change: PN9_ATT→ATTN, PN22 env name fixed, PN26 sub-config naming fixed (BLOCK_KV / NUM_WARPS / THRESHOLD)
|
||
narr_wall_tps: not benched
|
||
code_wall_tps: 76.71
|
||
code_wall_cv: 4.1
|
||
code_decode_tps: 77.64
|
||
ttft_code_ms: 112
|
||
mtp_al: 3.32-3.50
|
||
mtp_avg_accept: 0.772-0.833
|
||
n4_aggregate_tps: 242.94
|
||
n4_per_stream_tps: 71.15
|
||
n4_per_stream_cv: 2.4
|
||
|
||
delta_vs_prior: |
|
||
n=1 single-stream: 76.01 → 76.71 (+0.9%, within CV 4-5%)
|
||
n=4 aggregate: 269.03 → 242.94 (-9.7%, within bench-size variance)
|
||
PN22 advertised "+9-30% TPS on TP>=2" doesn't materialize at our small
|
||
bench sample (2 runs per stream, n=2/n=3 CV up to 28%). Headline numbers
|
||
unchanged in honest reporting. PN22 may help more at sustained higher
|
||
batch sizes (Sandermage's PROD bench at "100t output × 50-req sustained")
|
||
where the local-argmax dominates more of the draft-path latency budget.
|
||
|
||
notes: |
|
||
Functional changes confirmed in boot logs:
|
||
- PN22 now APPLY (was silently OFF before naming fix) — vllm#39419 backport
|
||
- PN26b sparse-V kernel correctly tuned (BLOCK_KV=8 num_warps=4 threshold=0.01)
|
||
- PN9 self-retires on v0.20 (PR #39930 merged upstream) — correct behavior
|
||
|
||
Recommendation: keep the naming fixes (they're real bugs regardless of TPS
|
||
impact) and ship the migration as-is. PN22 / PN26 27B-specific tuning will
|
||
show their value in larger-scale benchmarks or future config changes.
|