ebca0c8921
Closes task #230. Measured PFlash NIAH compression at 16K-260K source contexts on 1× 24 GB / 3090 single-card. Result: PFlash works flawlessly up to 131K source. Compresses 131,068 tokens to 6,524 (5%) in 10.8s with NIAH key + answer both retained. Vanilla llama.cpp pp131072 takes ~257s per Luce's published numbers, so PFlash alone is ~24× faster at this context. End-to-end TTFT (PFlash + target prefill on 6.5K) would be ~12-13s vs ~257s = ~20×. Above 131K, drafter ephemeral forward-pass tensors (K_curr/V_curr/Q_last at full sequence length) exceed 24 GB. K-cache quantization (--pflash-k-type q8_0) doesn't help — the failing allocs are forward-pass not cache, confirmed by separate bench at 200K/260K with identical OOM at the same layer numbers. @weicj's PR #78 claim of 24K → 262K dual-GPU phase split is neither refuted nor reproduced. Their setup was 2× 22 GB Ti with target also loaded co-resident on one card; the "24K" was target+drafter combined. Our 131K is drafter-alone on 24 GB. Reproducing 262K specifically would require investigation of their drafter config (chunk_size, lookahead, BSA window) — drafter activation footprint at 200K+ is the binding constraint regardless of GPU count. Practical recommendation for 24 GB / 3090 single-card users: PFlash is shippable for source contexts ≤ 131K. The ~24× TTFT speedup is genuine and quality holds. Above 131K, fall back to vanilla llama.cpp prefill or wait for upstream drafter optimizations. Adds: - BENCHMARKS.md "PFlash long-context compression on 1× 3090" subsection with full per-context table + drafter ceiling explanation - results/lucebox-pflash-niah-20260504-150321/ (BF16 K cache run) - results/lucebox-pflash-niah-q8k-20260504-150600/ (q8_0 K cache run) This closes our active investigation of the Luce surface — three benches done (DFlash same-card 73.97 mean, K8V4 same-card 74.68 mean, PFlash compression ceiling 131K). Recommendation surface narrows to: PFlash at ≤131K is the one piece of Luce that beats vLLM dual.yml on TTFT for that workload class. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
93 lines
3.1 KiB
Plaintext
93 lines
3.1 KiB
Plaintext
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
|
|
[transformers] Token indices sequence length is longer than the specified maximum sequence length for this model (251956 > 131072). Running this sequence through the model will result in indexing errors
|
|
[transformers] Token indices sequence length is longer than the specified maximum sequence length for this model (199996 > 131072). Running this sequence through the model will result in indexing errors
|
|
{
|
|
"date": "2026-05-04",
|
|
"mode": "dual_gpu_pflash_phase_split",
|
|
"pflash_gpu": 0,
|
|
"pflash_daemon_ready_s": 0.6323865110753104,
|
|
"pflash_drafter": "/mnt/models/gguf/qwen3-draft-0.6b/Qwen3-0.6B-BF16.gguf",
|
|
"tokenizer": "Qwen/Qwen3-0.6B",
|
|
"keep_ratio": 0.05,
|
|
"lookahead": 2,
|
|
"chunk_size": 32,
|
|
"pool_kernel": 13,
|
|
"pflash_k_type": "compute",
|
|
"cases": [
|
|
{
|
|
"name": "niah_ctx16372",
|
|
"source_tokens": 16372,
|
|
"compressed_tokens": 788,
|
|
"compress_wall_s": 1.0830025799805298,
|
|
"compress_tok_s": 15117.230838262947,
|
|
"compression_ratio": 0.04813095528951869,
|
|
"retained_key": true,
|
|
"retained_answer": true
|
|
},
|
|
{
|
|
"name": "niah_ctx32764",
|
|
"source_tokens": 32764,
|
|
"compressed_tokens": 1628,
|
|
"compress_wall_s": 1.799773955019191,
|
|
"compress_tok_s": 18204.50835430088,
|
|
"compression_ratio": 0.04968868270052497,
|
|
"retained_key": true,
|
|
"retained_answer": true
|
|
},
|
|
{
|
|
"name": "niah_ctx65524",
|
|
"source_tokens": 65524,
|
|
"compressed_tokens": 3252,
|
|
"compress_wall_s": 4.3655310838948935,
|
|
"compress_tok_s": 15009.399484458598,
|
|
"compression_ratio": 0.04963066967828582,
|
|
"retained_key": true,
|
|
"retained_answer": true
|
|
},
|
|
{
|
|
"name": "niah_ctx131068",
|
|
"source_tokens": 131068,
|
|
"compressed_tokens": 6524,
|
|
"compress_wall_s": 10.801249213982373,
|
|
"compress_tok_s": 12134.522350463923,
|
|
"compression_ratio": 0.04977568895535142,
|
|
"retained_key": true,
|
|
"retained_answer": true
|
|
},
|
|
{
|
|
"name": "niah_ctx199996",
|
|
"source_tokens": 199996,
|
|
"compressed_tokens": 0,
|
|
"compress_wall_s": 0.10222281608730555,
|
|
"compress_tok_s": 1956471.2424786773,
|
|
"compression_ratio": 0.0,
|
|
"retained_key": false,
|
|
"retained_answer": false
|
|
},
|
|
{
|
|
"name": "niah_ctx259996",
|
|
"source_tokens": 259996,
|
|
"compressed_tokens": 0,
|
|
"compress_wall_s": 0.09130622097291052,
|
|
"compress_tok_s": 2847516.820098575,
|
|
"compression_ratio": 0.0,
|
|
"retained_key": false,
|
|
"retained_answer": false
|
|
}
|
|
],
|
|
"resource_summary": {
|
|
"samples": 20,
|
|
"mem_max_mib": 21024.0,
|
|
"temp_max_c": 61.0,
|
|
"fan_max_pct": 67.0,
|
|
"power_avg_w": 198.707,
|
|
"power_max_w": 228.89,
|
|
"util_avg_pct": 88.7,
|
|
"util_max_pct": 100.0
|
|
},
|
|
"logs": {
|
|
"pflash": "/opt/ai/github/club-3090/results/lucebox-pflash-niah-20260504-150321/pflash_daemon.log",
|
|
"monitor": "/opt/ai/github/club-3090/results/lucebox-pflash-niah-20260504-150321/gpu_monitor.csv"
|
|
}
|
|
}
|