ebca0c8921
Closes task #230. Measured PFlash NIAH compression at 16K-260K source contexts on 1× 24 GB / 3090 single-card. Result: PFlash works flawlessly up to 131K source. Compresses 131,068 tokens to 6,524 (5%) in 10.8s with NIAH key + answer both retained. Vanilla llama.cpp pp131072 takes ~257s per Luce's published numbers, so PFlash alone is ~24× faster at this context. End-to-end TTFT (PFlash + target prefill on 6.5K) would be ~12-13s vs ~257s = ~20×. Above 131K, drafter ephemeral forward-pass tensors (K_curr/V_curr/Q_last at full sequence length) exceed 24 GB. K-cache quantization (--pflash-k-type q8_0) doesn't help — the failing allocs are forward-pass not cache, confirmed by separate bench at 200K/260K with identical OOM at the same layer numbers. @weicj's PR #78 claim of 24K → 262K dual-GPU phase split is neither refuted nor reproduced. Their setup was 2× 22 GB Ti with target also loaded co-resident on one card; the "24K" was target+drafter combined. Our 131K is drafter-alone on 24 GB. Reproducing 262K specifically would require investigation of their drafter config (chunk_size, lookahead, BSA window) — drafter activation footprint at 200K+ is the binding constraint regardless of GPU count. Practical recommendation for 24 GB / 3090 single-card users: PFlash is shippable for source contexts ≤ 131K. The ~24× TTFT speedup is genuine and quality holds. Above 131K, fall back to vanilla llama.cpp prefill or wait for upstream drafter optimizations. Adds: - BENCHMARKS.md "PFlash long-context compression on 1× 3090" subsection with full per-context table + drafter ceiling explanation - results/lucebox-pflash-niah-20260504-150321/ (BF16 K cache run) - results/lucebox-pflash-niah-q8k-20260504-150600/ (q8_0 K cache run) This closes our active investigation of the Luce surface — three benches done (DFlash same-card 73.97 mean, K8V4 same-card 74.68 mean, PFlash compression ceiling 131K). Recommendation surface narrows to: PFlash at ≤131K is the one piece of Luce that beats vLLM dual.yml on TTFT for that workload class. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1.2 KiB
1.2 KiB
Dual-GPU PFlash Phase-Split Report
- PFlash GPU:
0 - PFlash daemon ready:
0.63 s - keep ratio:
0.05 - lookahead:
2 - PFlash K cache:
compute
Resource Peak
| gpu | samples | peak mem MiB | peak temp C | avg power W | peak power W | avg util % | peak util % |
|---|---|---|---|---|---|---|---|
| 0 | 20 | 21024.00 | 61.00 | 198.71 | 228.89 | 88.70 | 100.00 |
Cases
| case | source tokens | compressed tokens | ratio | PFlash s | PFlash tok/s | key retained | answer retained |
|---|---|---|---|---|---|---|---|
| niah_ctx16372 | 16372 | 788 | 0.0481 | 1.08 | 15117.23 | yes | yes |
| niah_ctx32764 | 32764 | 1628 | 0.0497 | 1.80 | 18204.51 | yes | yes |
| niah_ctx65524 | 65524 | 3252 | 0.0496 | 4.37 | 15009.40 | yes | yes |
| niah_ctx131068 | 131068 | 6524 | 0.0498 | 10.80 | 12134.52 | yes | yes |
| niah_ctx199996 | 199996 | 0 | 0.0000 | 0.10 | 1956471.24 | no | no |
| niah_ctx259996 | 259996 | 0 | 0.0000 | 0.09 | 2847516.82 | no | no |
Files:
- pflash:
/opt/ai/github/club-3090/results/lucebox-pflash-niah-20260504-150321/pflash_daemon.log - monitor:
/opt/ai/github/club-3090/results/lucebox-pflash-niah-20260504-150321/gpu_monitor.csv