b03a134925
Reconciles the earlier L2=0 finding. With L2 on (production shape), a 60 GB L1 serves warm reads ~1.6x faster on average (~2x on the evicted half) than a 30 GB L1 for a ~50 GB working set: older sessions that spill past a 30 GB L1 fall to L2 disk (~3-5s) but stay in RAM at L1=60 (~1.3-2.4s). Two-part model: L2 disk decides *whether* a read is warm (retention); L1 RAM decides *how fast* (speed). L1 is a fast write-through front for L2, not a standalone store — hence L2=0 gave 0/8 yet L2=1+L1=60 is fastest. Decision: keep L1=60, it is not reclaimable headroom. - Add SALT env to lmcache-retention-test.sh (fresh keys per A/B leg, no L2 collision) - Add _local-lmcache-tuning/l1-ab-run.sh (two-leg orchestrator + comparison) - Record result + reconciliation in HANDOFF.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
111 lines
4.9 KiB
Plaintext
111 lines
4.9 KiB
Plaintext
== L1 A/B START 21:19:29 salt-stamp=1783970369 ==
|
|
====================================================================
|
|
LEG L1=30GB (L2 stays 1) — 21:19:29
|
|
====================================================================
|
|
[switch] 64s — Loading weights
|
|
[switch] 120s elapsed, still waiting...
|
|
[switch] 180s elapsed, still waiting...
|
|
[switch] 200s — Capturing CUDA graphs
|
|
[switch] 216s — Application startup complete
|
|
[switch] ✓ ready (216s)
|
|
[switch] ▶ API: http://192.168.1.98:8017/v1 (model: qwen3.6-27b · OpenAI-compatible · no auth)
|
|
[switch] done. Try: curl -s http://localhost:8017/v1/models | jq .
|
|
>> container LMCACHE_L1_GB=30 LMCACHE_L2=1 (want L1=30 L2=1)
|
|
LMCache L1 retention probe -> http://localhost:8017 model=qwen3.6-27b
|
|
sessions=8 target_tokens/session~=48000 warm_threshold=8.0s kv=131KB/tok
|
|
|
|
Building calibrated haystacks (one /tokenize probe each)...
|
|
|
|
Round 1 - cold insert (fills L1):
|
|
session 1: cold TTFT 53.20s (prompt_tokens=47559)
|
|
session 2: cold TTFT 52.73s (prompt_tokens=47991)
|
|
session 3: cold TTFT 53.78s (prompt_tokens=48207)
|
|
session 4: cold TTFT 51.33s (prompt_tokens=46318)
|
|
session 5: cold TTFT 52.83s (prompt_tokens=47015)
|
|
session 6: cold TTFT 54.30s (prompt_tokens=47676)
|
|
session 7: cold TTFT 55.95s (prompt_tokens=48343)
|
|
session 8: cold TTFT 56.34s (prompt_tokens=48098)
|
|
|
|
inserted ~381207 tokens ~= 49.9 GB of L1
|
|
capacity lines: L1=30 -> ~229K tok (~30 GB), L1=60 -> ~458K tok (~60 GB)
|
|
|
|
Round 2 - retention read (same prefixes; warm = retained):
|
|
session 1: TTFT 3.32s WARM retained
|
|
session 2: TTFT 4.35s WARM retained
|
|
session 3: TTFT 4.16s WARM retained
|
|
session 4: TTFT 3.58s WARM retained
|
|
session 5: TTFT 5.09s WARM retained
|
|
session 6: TTFT 1.09s WARM retained
|
|
session 7: TTFT 1.87s WARM retained
|
|
session 8: TTFT 1.48s WARM retained
|
|
|
|
retained warm: 8/8
|
|
|
|
PASS - all sessions retained. L1 held the full working set;
|
|
this set would have evicted its earliest sessions at L1=30.
|
|
>> leg L1=30 done, log=_local-lmcache-tuning/l1ab-L130.log
|
|
====================================================================
|
|
LEG L1=60GB (L2 stays 1) — 21:30:54
|
|
====================================================================
|
|
[switch] 68s — Loading weights
|
|
[switch] 120s elapsed, still waiting...
|
|
[switch] 180s elapsed, still waiting...
|
|
[switch] 208s — Capturing CUDA graphs
|
|
[switch] 224s — Application startup complete
|
|
[switch] ✓ ready (224s)
|
|
[switch] ▶ API: http://192.168.1.98:8017/v1 (model: qwen3.6-27b · OpenAI-compatible · no auth)
|
|
[switch] done. Try: curl -s http://localhost:8017/v1/models | jq .
|
|
>> container LMCACHE_L1_GB=60 LMCACHE_L2=1 (want L1=60 L2=1)
|
|
LMCache L1 retention probe -> http://localhost:8017 model=qwen3.6-27b
|
|
sessions=8 target_tokens/session~=48000 warm_threshold=8.0s kv=131KB/tok
|
|
|
|
Building calibrated haystacks (one /tokenize probe each)...
|
|
|
|
Round 1 - cold insert (fills L1):
|
|
session 1: cold TTFT 55.80s (prompt_tokens=48631)
|
|
session 2: cold TTFT 51.99s (prompt_tokens=46869)
|
|
session 3: cold TTFT 54.48s (prompt_tokens=48031)
|
|
session 4: cold TTFT 55.22s (prompt_tokens=48240)
|
|
session 5: cold TTFT 55.62s (prompt_tokens=47982)
|
|
session 6: cold TTFT 58.44s (prompt_tokens=49035)
|
|
session 7: cold TTFT 56.92s (prompt_tokens=47843)
|
|
session 8: cold TTFT 58.67s (prompt_tokens=48719)
|
|
|
|
inserted ~385350 tokens ~= 50.5 GB of L1
|
|
capacity lines: L1=30 -> ~229K tok (~30 GB), L1=60 -> ~458K tok (~60 GB)
|
|
|
|
Round 2 - retention read (same prefixes; warm = retained):
|
|
session 1: TTFT 2.39s WARM retained
|
|
session 2: TTFT 2.05s WARM retained
|
|
session 3: TTFT 1.42s WARM retained
|
|
session 4: TTFT 1.71s WARM retained
|
|
session 5: TTFT 1.27s WARM retained
|
|
session 6: TTFT 3.04s WARM retained
|
|
session 7: TTFT 1.05s WARM retained
|
|
session 8: TTFT 2.50s WARM retained
|
|
|
|
retained warm: 8/8
|
|
|
|
PASS - all sessions retained. L1 held the full working set;
|
|
this set would have evicted its earliest sessions at L1=30.
|
|
>> leg L1=60 done, log=_local-lmcache-tuning/l1ab-L160.log
|
|
>> FINAL production state: LMCACHE_L1_GB=60 LMCACHE_L2=1 (want 60 / 1)
|
|
====================================================================
|
|
COMPARISON — Round-2 (warm) TTFT per session, L1=30 vs L1=60
|
|
====================================================================
|
|
sess L1=30 s L1=60 s faster
|
|
1 3.32 2.39 L1=60
|
|
2 4.35 2.05 L1=60
|
|
3 4.16 1.42 L1=60
|
|
4 3.58 1.71 L1=60
|
|
5 5.09 1.27 L1=60
|
|
6 1.09 3.04 L1=30
|
|
7 1.87 1.05 L1=60
|
|
8 1.48 2.50 L1=30
|
|
|
|
L1=30 warm: n=8 mean=3.12s median=3.45s max=5.09s
|
|
L1=60 warm: n=8 mean=1.93s median=1.88s max=3.04s
|
|
|
|
VERDICT: L1=60 warm reads ~1.6x faster => the RAM IS doing work.
|
|
== L1 A/B DONE 21:42:31 ==
|