alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions 38 Packages Projects Releases Wiki Activity

Local: L2=0 result — retention is L2 disk, not L1 RAM (0/8 vs 8/8)

Browse Source 
Clean L2-off experiment: 50.6 GB working set, L2 on = 8/8 warm,
L2 off = 0/8 (all re-prefill 45-71s incl. MRU session 8). The warm
benefit is the disk tier; L1=60 RAM retained nothing standalone.
Corrects the earlier 'L1=60 retains the working set' claim.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Alex
2026-07-13 17:37:45 +02:00
parent ffab073ef6
commit 8269e79e54
3 changed files with 152 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
_local-lmcache-tuning/HANDOFF.md
View File

@@ -75,13 +75,30 @@ pgrep -af lmcache-retention-test.sh # still running?
the ~52 s true-cold cost. All 8 still warm; 2.0 s cut lands inside cache-hit
jitter. With L2 on there's no cold floor to separate L1 from L2, so no
threshold cleanly isolates L1. DO NOT report the rerun as a real failure.
- Clean L1-isolation experiment — IN FLIGHT (bg job bacnmj267, orchestrator
- **L2=0 RESULT (COMPLETE, 2026-07-13 07:34) — SURPRISE, corrects earlier claim:**
L2 off → **0/8 retained** (all re-prefilled 45–71 s, incl. just-inserted session 8).
L2 on → 8/8 warm. ⇒ **The warm-cache benefit is L2 DISK, not L1 RAM.** The 60 GB
L1 provisioning is NOT the active lever on this rig. Earlier "L1=60 retains the
working set" claim was WRONG — it was L2. Also contradicts #423's L1=0.51s
attribution (likely L1-fronting-L2, not standalone). Likely mechanism (UNPROVEN):
LMCache CPU tier only persists with a disk backend attached. Production (L2=1)
works great regardless — 52 s→<5 s on 50 GB. Full logs: _local-lmcache-tuning/run3-L2off.log.
Open: (a) is L1-standalone supposed to work? (b) if L2 carries it, can we reclaim
RAM by lowering L1? (c) disk/L2 is load-bearing → prune lmcache-kv + disk speed matters.
- Clean L1-isolation experiment — DONE (was bg job bacnmj267, orchestrator
`<scratchpad>/l2off-run.sh`). NOTE: L2 is a CONTAINER-level flag (not a script
env) — must RECREATE the container. Chain: recreate L2=0 (~4min) → run test
(Round1 true-cold ~52s) → restore L2=1 (~4min). Verifies each flip. Output:
`<scratchpad>/tasks/bacnmj267.output`. Warm count = sessions genuinely in fast
60 GB L1 RAM. Endpoint down during the 2 restarts; restored to L2=1 at end.
## NEXT PHASE (queued, gated on prod-ready)
- User wants a **big Hermes test** against qwen3.6-27b :8017 using the **"coldcase"
project on .96** (192.168.1.96, pings OK; NO coldcase refs in club-3090 repo).
- Gate: only after the L1=60+L2 config is validated prod-ready.
- BLOCKED ON INFO (asked user): what coldcase is, how to reach it on .96, what the
test should measure/produce. See memory `planned-hermes-coldcase-test`.
## KEY FACTS
- Endpoint: http://localhost:8017/v1 (LAN 192.168.1.98:8017), no auth, model id `qwen3.6-27b`.
- Container: `vllm-qwen36-27b-lmcache`.

77
_local-lmcache-tuning/run3-L2off.log Normal file
View File

@@ -0,0 +1,77 @@
====================================================================
STEP 1 — recreate with LMCACHE_L2=0 (07:10:56)
====================================================================
Container vllm-qwen36-27b-lmcache Starting
Container vllm-qwen36-27b-lmcache Started
[switch] waiting for http://localhost:8017/v1/models (container=vllm-qwen36-27b-lmcache, timeout 600s)...
[switch] 16s — Application startup complete
[switch] 28s — Resolved architecture: Qwen3_5ForConditionalGeneration
[switch] 36s — Resolved architecture: Qwen3_5MTP
[switch] 60s elapsed, still waiting...
[switch] 68s — Loading weights
[switch] 120s elapsed, still waiting...
[switch] 180s elapsed, still waiting...
[switch] 208s — Capturing CUDA graphs
[switch] 224s — Application startup complete
[switch] ✓ ready (224s)
[switch] ▶ API: http://192.168.1.98:8017/v1 (model: qwen3.6-27b · OpenAI-compatible · no auth)
[switch] done. Try: curl -s http://localhost:8017/v1/models | jq .
>> container LMCACHE_L2=0
====================================================================
STEP 2 — clean retention test, L2 OFF, fresh L1 (07:14:52)
(Round 1 = true cold ~52s; Round 2 warm<8s = held in L1, cold = evicted)
====================================================================
LMCache L1 retention probe -> http://localhost:8017 model=qwen3.6-27b
sessions=8 target_tokens/session~=48000 warm_threshold=8.0s kv=131KB/tok
Building calibrated haystacks (one /tokenize probe each)...
Round 1 - cold insert (fills L1):
session 1: cold TTFT 55.05s (prompt_tokens=48173)
session 2: cold TTFT 54.25s (prompt_tokens=48141)
session 3: cold TTFT 57.25s (prompt_tokens=49197)
session 4: cold TTFT 54.68s (prompt_tokens=46422)
session 5: cold TTFT 60.64s (prompt_tokens=49274)
session 6: cold TTFT 64.87s (prompt_tokens=50187)
session 7: cold TTFT 63.76s (prompt_tokens=48045)
session 8: cold TTFT 63.24s (prompt_tokens=46967)
inserted ~386406 tokens ~= 50.6 GB of L1
capacity lines: L1=30 -> ~229K tok (~30 GB), L1=60 -> ~458K tok (~60 GB)
Round 2 - retention read (same prefixes; warm = retained):
session 1: TTFT 66.52s COLD evicted / re-prefill
session 2: TTFT 47.73s COLD evicted / re-prefill
session 3: TTFT 69.82s COLD evicted / re-prefill
session 4: TTFT 45.63s COLD evicted / re-prefill
session 5: TTFT 71.24s COLD evicted / re-prefill
session 6: TTFT 49.44s COLD evicted / re-prefill
session 7: TTFT 69.20s COLD evicted / re-prefill
session 8: TTFT 48.29s COLD evicted / re-prefill
retained warm: 0/8
FAIL - 8 session(s) evicted. Working set exceeded the live L1,
or L1 is smaller than expected. Check LMCACHE_L1_GB and that
shm_size >= L1 (else the MP connector falls back to slow pickle).
>> test exit rc=1
====================================================================
STEP 3 — restore production (LMCACHE_L2=1) (07:30:37)
====================================================================
Container vllm-qwen36-27b-lmcache Starting
Container vllm-qwen36-27b-lmcache Started
[switch] waiting for http://localhost:8017/v1/models (container=vllm-qwen36-27b-lmcache, timeout 600s)...
[switch] 12s — Application startup complete
[switch] 28s — Resolved architecture: Qwen3_5ForConditionalGeneration
[switch] 36s — Resolved architecture: Qwen3_5MTP
[switch] 60s elapsed, still waiting...
[switch] 64s — Loading weights
[switch] 120s elapsed, still waiting...
[switch] 180s elapsed, still waiting...
[switch] 204s — Capturing CUDA graphs
[switch] 220s — Application startup complete
[switch] ✓ ready (220s)
[switch] ▶ API: http://192.168.1.98:8017/v1 (model: qwen3.6-27b · OpenAI-compatible · no auth)
[switch] done. Try: curl -s http://localhost:8017/v1/models | jq .
>> restored container LMCACHE_L2=1 (expect 1)
== DONE 07:34:27 ==

57
_local-lmcache-tuning/run3-L2off.partial.log
View File

@@ -1,3 +1,60 @@
====================================================================
STEP 1 — recreate with LMCACHE_L2=0 (07:10:56)
====================================================================
Container vllm-qwen36-27b-lmcache Starting
Container vllm-qwen36-27b-lmcache Started
[switch] waiting for http://localhost:8017/v1/models (container=vllm-qwen36-27b-lmcache, timeout 600s)...
[switch] 16s — Application startup complete
[switch] 28s — Resolved architecture: Qwen3_5ForConditionalGeneration
[switch] 36s — Resolved architecture: Qwen3_5MTP
[switch] 60s elapsed, still waiting...
[switch] 68s — Loading weights
[switch] 120s elapsed, still waiting...
[switch] 180s elapsed, still waiting...
[switch] 208s — Capturing CUDA graphs
[switch] 224s — Application startup complete
[switch] ✓ ready (224s)
[switch] ▶ API: http://192.168.1.98:8017/v1 (model: qwen3.6-27b · OpenAI-compatible · no auth)
[switch] done. Try: curl -s http://localhost:8017/v1/models | jq .
>> container LMCACHE_L2=0
====================================================================
STEP 2 — clean retention test, L2 OFF, fresh L1 (07:14:52)
(Round 1 = true cold ~52s; Round 2 warm<8s = held in L1, cold = evicted)
====================================================================
LMCache L1 retention probe -> http://localhost:8017 model=qwen3.6-27b
sessions=8 target_tokens/session~=48000 warm_threshold=8.0s kv=131KB/tok
Building calibrated haystacks (one /tokenize probe each)...
Round 1 - cold insert (fills L1):
session 1: cold TTFT 55.05s (prompt_tokens=48173)
session 2: cold TTFT 54.25s (prompt_tokens=48141)
session 3: cold TTFT 57.25s (prompt_tokens=49197)
session 4: cold TTFT 54.68s (prompt_tokens=46422)
session 5: cold TTFT 60.64s (prompt_tokens=49274)
session 6: cold TTFT 64.87s (prompt_tokens=50187)
session 7: cold TTFT 63.76s (prompt_tokens=48045)
session 8: cold TTFT 63.24s (prompt_tokens=46967)
inserted ~386406 tokens ~= 50.6 GB of L1
capacity lines: L1=30 -> ~229K tok (~30 GB), L1=60 -> ~458K tok (~60 GB)
Round 2 - retention read (same prefixes; warm = retained):
session 1: TTFT 66.52s COLD evicted / re-prefill
session 2: TTFT 47.73s COLD evicted / re-prefill
session 3: TTFT 69.82s COLD evicted / re-prefill
session 4: TTFT 45.63s COLD evicted / re-prefill
session 5: TTFT 71.24s COLD evicted / re-prefill
session 6: TTFT 49.44s COLD evicted / re-prefill
session 7: TTFT 69.20s COLD evicted / re-prefill
session 8: TTFT 48.29s COLD evicted / re-prefill
retained warm: 0/8
FAIL - 8 session(s) evicted. Working set exceeded the live L1,
or L1 is smaller than expected. Check LMCACHE_L1_GB and that
shm_size >= L1 (else the MP connector falls back to slow pickle).
>> test exit rc=1
====================================================================
STEP 3 — restore production (LMCACHE_L2=1) (07:30:37)
====================================================================