077228e81b
Post-merge follow-up to PR #23. Captures the dual-turbo TP=2 bench after the env-var naming fixes (PN22 / PN26 sub-config) actually fire, vs the prior bench where they were silently no-op. Honest finding: PN22's advertised "+9-30% TPS on TP>=2" (per PR author of vllm#39419) doesn't materialize at our small bench sample sizes (n=2 runs/ stream, n=4 streams = 8 total runs at high CV). | Metric | Prior (PN22 silently OFF) | Fixed (PN22 firing) | Δ | |---|---|---|---| | n=1 code wall_TPS | 76.01 | 76.71 | +0.9% (within CV 4-5%) | | n=4 aggregate | 269.03 | 242.94 | -9.7% (within bench-size variance) | | PN22 boot status | "PN22=1" matches no env_flag — silent OFF | APPLY (vllm#39419 backport) | | PN26 sparse-V status | BLOCK_KV / NUM_WARPS / THRESHOLD silently using defaults | 27B-tuned values applied | The naming corrections are real bugs regardless of TPS impact — PN22 / PN26 sub-config are now actually firing on master. Effect may be more visible at sustained high batch sizes (Sandermage's PROD bench at 100t × 50-req sustained) where local-argmax dominates the draft-path latency budget more than at our single-shot 800-token bench. Headline numbers from PR #23 (76.01 code / 269 aggregate at n=4) remain the right reference for docs and charts — re-bench is within noise. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
22 lines
697 B
Plaintext
22 lines
697 B
Plaintext
warmup...
|
|
|
|
=== 1 concurrent streams, 1 run(s) each ===
|
|
total_toks=547 total_wall=7.20s aggregate_TPS=75.96
|
|
per-stream mean=75.97 CV=0.0%
|
|
|
|
=== 1 concurrent streams, 2 run(s) each ===
|
|
total_toks=1598 total_wall=20.63s aggregate_TPS=77.45
|
|
per-stream mean=77.48 CV=2.7%
|
|
|
|
=== 2 concurrent streams, 2 run(s) each ===
|
|
total_toks=2400 total_wall=24.73s aggregate_TPS=97.06
|
|
per-stream mean=57.86 CV=19.2%
|
|
|
|
=== 3 concurrent streams, 2 run(s) each ===
|
|
total_toks=4574 total_wall=29.61s aggregate_TPS=154.49
|
|
per-stream mean=57.23 CV=28.6%
|
|
|
|
=== 4 concurrent streams, 2 run(s) each ===
|
|
total_toks=5110 total_wall=21.03s aggregate_TPS=242.94
|
|
per-stream mean=71.15 CV=2.4%
|