ebca0c8921
Closes task #230. Measured PFlash NIAH compression at 16K-260K source contexts on 1× 24 GB / 3090 single-card. Result: PFlash works flawlessly up to 131K source. Compresses 131,068 tokens to 6,524 (5%) in 10.8s with NIAH key + answer both retained. Vanilla llama.cpp pp131072 takes ~257s per Luce's published numbers, so PFlash alone is ~24× faster at this context. End-to-end TTFT (PFlash + target prefill on 6.5K) would be ~12-13s vs ~257s = ~20×. Above 131K, drafter ephemeral forward-pass tensors (K_curr/V_curr/Q_last at full sequence length) exceed 24 GB. K-cache quantization (--pflash-k-type q8_0) doesn't help — the failing allocs are forward-pass not cache, confirmed by separate bench at 200K/260K with identical OOM at the same layer numbers. @weicj's PR #78 claim of 24K → 262K dual-GPU phase split is neither refuted nor reproduced. Their setup was 2× 22 GB Ti with target also loaded co-resident on one card; the "24K" was target+drafter combined. Our 131K is drafter-alone on 24 GB. Reproducing 262K specifically would require investigation of their drafter config (chunk_size, lookahead, BSA window) — drafter activation footprint at 200K+ is the binding constraint regardless of GPU count. Practical recommendation for 24 GB / 3090 single-card users: PFlash is shippable for source contexts ≤ 131K. The ~24× TTFT speedup is genuine and quality holds. Above 131K, fall back to vanilla llama.cpp prefill or wait for upstream drafter optimizations. Adds: - BENCHMARKS.md "PFlash long-context compression on 1× 3090" subsection with full per-context table + drafter ceiling explanation - results/lucebox-pflash-niah-20260504-150321/ (BF16 K cache run) - results/lucebox-pflash-niah-q8k-20260504-150600/ (q8_0 K cache run) This closes our active investigation of the Luce surface — three benches done (DFlash same-card 73.97 mean, K8V4 same-card 74.68 mean, PFlash compression ceiling 131K). Recommendation surface narrows to: PFlash at ≤131K is the one piece of Luce that beats vLLM dual.yml on TTFT for that workload class. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2.5 KiB
2.5 KiB
| 1 | ts | phase | index | temp_c | fan_pct | power_w | power_limit_w | mem_used_mib | mem_total_mib | util_pct |
|---|---|---|---|---|---|---|---|---|---|---|
| 2 | 1777907036.255 | pflash_load | 0 | 51 | 0 | 28.92 | 230.00 | 1 | 24576 | 0 |
| 3 | 1777907036.255 | pflash_load | 1 | 52 | 0 | 23.08 | 230.00 | 1 | 24576 | 0 |
| 4 | 1777907037.309 | niah_ctx16372 | 0 | 54 | 0 | 61.36 | 230.00 | 4348 | 24576 | 1 |
| 5 | 1777907037.309 | niah_ctx16372 | 1 | 52 | 0 | 25.83 | 230.00 | 1 | 24576 | 0 |
| 6 | 1777907038.355 | niah_ctx32764 | 0 | 58 | 57 | 210.91 | 230.00 | 6972 | 24576 | 100 |
| 7 | 1777907038.355 | niah_ctx32764 | 1 | 52 | 0 | 27.22 | 230.00 | 1 | 24576 | 0 |
| 8 | 1777907039.401 | niah_ctx32764 | 0 | 59 | 54 | 223.68 | 230.00 | 6984 | 24576 | 99 |
| 9 | 1777907039.401 | niah_ctx32764 | 1 | 52 | 0 | 23.30 | 230.00 | 1 | 24576 | 0 |
| 10 | 1777907040.447 | niah_ctx65524 | 0 | 58 | 56 | 211.54 | 230.00 | 11004 | 24576 | 100 |
| 11 | 1777907040.447 | niah_ctx65524 | 1 | 52 | 0 | 23.00 | 230.00 | 1 | 24576 | 0 |
| 12 | 1777907041.492 | niah_ctx65524 | 0 | 59 | 59 | 224.04 | 230.00 | 10954 | 24576 | 99 |
| 13 | 1777907041.492 | niah_ctx65524 | 1 | 52 | 0 | 22.92 | 230.00 | 1 | 24576 | 0 |
| 14 | 1777907042.536 | niah_ctx65524 | 0 | 60 | 62 | 227.43 | 230.00 | 10954 | 24576 | 99 |
| 15 | 1777907042.536 | niah_ctx65524 | 1 | 52 | 0 | 22.91 | 230.00 | 1 | 24576 | 0 |
| 16 | 1777907043.583 | niah_ctx65524 | 0 | 60 | 63 | 226.14 | 230.00 | 11482 | 24576 | 100 |
| 17 | 1777907043.583 | niah_ctx65524 | 1 | 52 | 0 | 22.81 | 230.00 | 1 | 24576 | 0 |
| 18 | 1777907044.715 | niah_ctx131068 | 0 | 59 | 64 | 190.93 | 230.00 | 18702 | 24576 | 100 |
| 19 | 1777907044.715 | niah_ctx131068 | 1 | 52 | 0 | 22.77 | 230.00 | 1 | 24576 | 0 |
| 20 | 1777907045.761 | niah_ctx131068 | 0 | 60 | 65 | 221.91 | 230.00 | 18944 | 24576 | 100 |
| 21 | 1777907045.761 | niah_ctx131068 | 1 | 52 | 0 | 22.94 | 230.00 | 1 | 24576 | 0 |
| 22 | 1777907046.805 | niah_ctx131068 | 0 | 60 | 65 | 227.38 | 230.00 | 19138 | 24576 | 99 |
| 23 | 1777907046.805 | niah_ctx131068 | 1 | 52 | 0 | 22.93 | 230.00 | 1 | 24576 | 0 |
| 24 | 1777907047.850 | niah_ctx131068 | 0 | 61 | 65 | 227.67 | 230.00 | 18944 | 24576 | 100 |
| 25 | 1777907047.850 | niah_ctx131068 | 1 | 52 | 0 | 22.81 | 230.00 | 1 | 24576 | 0 |
| 26 | 1777907048.894 | niah_ctx131068 | 0 | 61 | 66 | 227.86 | 230.00 | 19138 | 24576 | 99 |
| 27 | 1777907048.894 | niah_ctx131068 | 1 | 52 | 0 | 22.87 | 230.00 | 1 | 24576 | 0 |
| 28 | 1777907049.945 | niah_ctx131068 | 0 | 61 | 66 | 227.93 | 230.00 | 19138 | 24576 | 100 |
| 29 | 1777907049.945 | niah_ctx131068 | 1 | 52 | 0 | 22.92 | 230.00 | 1 | 24576 | 0 |
| 30 | 1777907051.001 | niah_ctx131068 | 0 | 61 | 66 | 227.80 | 230.00 | 18944 | 24576 | 100 |
| 31 | 1777907051.001 | niah_ctx131068 | 1 | 52 | 0 | 23.00 | 230.00 | 1 | 24576 | 0 |
| 32 | 1777907052.057 | niah_ctx131068 | 0 | 61 | 66 | 228.89 | 230.00 | 19138 | 24576 | 99 |
| 33 | 1777907052.057 | niah_ctx131068 | 1 | 51 | 0 | 22.97 | 230.00 | 1 | 24576 | 0 |
| 34 | 1777907053.102 | niah_ctx131068 | 0 | 60 | 67 | 219.70 | 230.00 | 21024 | 24576 | 94 |
| 35 | 1777907053.102 | niah_ctx131068 | 1 | 51 | 0 | 22.80 | 230.00 | 1 | 24576 | 0 |
| 36 | 1777907054.223 | niah_ctx131068 | 0 | 60 | 67 | 191.47 | 230.00 | 21024 | 24576 | 93 |
| 37 | 1777907054.223 | niah_ctx131068 | 1 | 51 | 0 | 22.93 | 230.00 | 1 | 24576 | 0 |
| 38 | 1777907055.359 | niah_ctx131068 | 0 | 59 | 67 | 190.35 | 230.00 | 2046 | 24576 | 92 |
| 39 | 1777907055.359 | niah_ctx131068 | 1 | 51 | 0 | 22.99 | 230.00 | 1 | 24576 | 0 |
| 40 | 1777907056.402 | cleanup | 0 | 59 | 67 | 178.23 | 230.00 | 4080 | 24576 | 100 |
| 41 | 1777907056.402 | cleanup | 1 | 51 | 0 | 22.87 | 230.00 | 1 | 24576 | 0 |