Closes task #230. Measured PFlash NIAH compression at 16K-260K source
contexts on 1× 24 GB / 3090 single-card.
Result: PFlash works flawlessly up to 131K source. Compresses 131,068
tokens to 6,524 (5%) in 10.8s with NIAH key + answer both retained.
Vanilla llama.cpp pp131072 takes ~257s per Luce's published numbers,
so PFlash alone is ~24× faster at this context. End-to-end TTFT
(PFlash + target prefill on 6.5K) would be ~12-13s vs ~257s = ~20×.
Above 131K, drafter ephemeral forward-pass tensors (K_curr/V_curr/Q_last
at full sequence length) exceed 24 GB. K-cache quantization
(--pflash-k-type q8_0) doesn't help — the failing allocs are
forward-pass not cache, confirmed by separate bench at 200K/260K with
identical OOM at the same layer numbers.
@weicj's PR #78 claim of 24K → 262K dual-GPU phase split is neither
refuted nor reproduced. Their setup was 2× 22 GB Ti with target also
loaded co-resident on one card; the "24K" was target+drafter
combined. Our 131K is drafter-alone on 24 GB. Reproducing 262K
specifically would require investigation of their drafter config
(chunk_size, lookahead, BSA window) — drafter activation footprint
at 200K+ is the binding constraint regardless of GPU count.
Practical recommendation for 24 GB / 3090 single-card users: PFlash
is shippable for source contexts ≤ 131K. The ~24× TTFT speedup is
genuine and quality holds. Above 131K, fall back to vanilla llama.cpp
prefill or wait for upstream drafter optimizations.
Adds:
- BENCHMARKS.md "PFlash long-context compression on 1× 3090" subsection
with full per-context table + drafter ceiling explanation
- results/lucebox-pflash-niah-20260504-150321/ (BF16 K cache run)
- results/lucebox-pflash-niah-q8k-20260504-150600/ (q8_0 K cache run)
This closes our active investigation of the Luce surface — three
benches done (DFlash same-card 73.97 mean, K8V4 same-card 74.68 mean,
PFlash compression ceiling 131K). Recommendation surface narrows to:
PFlash at ≤131K is the one piece of Luce that beats vLLM dual.yml on
TTFT for that workload class.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ebca0c8921