diff --git a/_local-lmcache-tuning/HANDOFF.md b/_local-lmcache-tuning/HANDOFF.md index 68e73137..57eea983 100644 --- a/_local-lmcache-tuning/HANDOFF.md +++ b/_local-lmcache-tuning/HANDOFF.md @@ -92,6 +92,44 @@ pgrep -af lmcache-retention-test.sh # still running? `/tasks/bacnmj267.output`. Warm count = sessions genuinely in fast 60 GB L1 RAM. Endpoint down during the 2 restarts; restored to L2=1 at end. +## L1=30 vs L1=60 A/B (COMPLETE, 2026-07-13 21:42) — RECONCILES the L2=0 result +- Question: with L2 ON (production shape), does 60 GB L1 give FASTER warm reads + than 30 GB L1 on a ~50 GB working set — or is the extra 30 GB pure headroom? +- Method: `_local-lmcache-tuning/l1-ab-run.sh` — recreate at L1=30 then L1=60 (L2 + stays 1), each leg a fresh SALT (new `SALT` env in the retention test → byte- + different prefixes, cannot hit the 49 GB already on L2 disk or the other leg). + L1=60 leg doubles as the prod restore. Compares raw Round-2 (warm) TTFTs, NOT + pass/fail (both legs pass 8/8 with L2 on). Logs: `l1ab-L130.log`, `l1ab-L160.log`, + `l1ab-run.log`. +- RESULT — **L1=60 warm reads ~1.6× faster on average; the RAM IS doing work:** + ``` + sess L1=30 s L1=60 s faster + 1 3.32 2.39 L1=60 + 2 4.35 2.05 L1=60 + 3 4.16 1.42 L1=60 + 4 3.58 1.71 L1=60 + 5 5.09 1.27 L1=60 + 6 1.09 3.04 L1=30 (noise; MRU-resident either way) + 7 1.87 1.05 L1=60 + 8 1.48 2.50 L1=30 (noise; MRU-resident either way) + L1=30 warm: mean 3.12s median 3.45s max 5.09s + L1=60 warm: mean 1.93s median 1.88s max 3.04s + ``` + Pattern fits theory: the OLDER half (sess 1–5) spills past a 30 GB L1 to L2 disk + (~3–5 s) but stays in RAM at L1=60 (~1.3–2.4 s) — ~2× on exactly the sessions that + get evicted. The newest ~3 (6–8) are L1-resident in BOTH legs → jitter-level flips. +- **RECONCILIATION of the two findings (both true, different questions):** + - L2 disk decides *whether* a read is warm at all (L2=0 → 0/8; L2=1 → 8/8). RETENTION = L2. + - L1 RAM decides *how fast* a warm read is: bigger L1 keeps more sessions in RAM, + ~2× faster than falling to L2 disk. WARM-READ SPEED for large working sets = L1. + - So L1 works as a **fast front for L2**, not a standalone retention store — which + is exactly why L2=0 gave 0/8 (no backend to persist to) yet L2=1+L1=60 is fastest. +- DECISION: **keep L1=60** — it is NOT reclaimable headroom; it earns ~1.6–2× on warm + reads once the working set exceeds ~30 GB (multi-session chat). Production restored + and verified: LMCACHE_L1_GB=60, LMCACHE_L2=1, container up. Config is prod-ready. +- Caveat: n=1 per leg; warm TTFT is jittery (see sess 6/8). Signal is the consistent + ~2× on the evicted half, not the modest 1.6× aggregate. Good enough to decide "keep RAM". + ## NEXT PHASE (queued, gated on prod-ready) - User wants a **big Hermes test** against qwen3.6-27b :8017 using the **"coldcase" project on .96** (192.168.1.96, pings OK; NO coldcase refs in club-3090 repo). diff --git a/_local-lmcache-tuning/l1-ab-run.sh b/_local-lmcache-tuning/l1-ab-run.sh new file mode 100755 index 00000000..5ab2f5a4 --- /dev/null +++ b/_local-lmcache-tuning/l1-ab-run.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash +# =========================================================================== +# l1-ab-run.sh — does the 60 GB of L1 RAM actually buy anything, or is it +# pure headroom the L2 disk makes moot? +# --------------------------------------------------------------------------- +# The L2=0 experiment already showed L2 disk (not L1 RAM) delivers RETENTION. +# This asks the sharper question with L2 left ON (production shape): at a +# 50 GB working set, does a 60 GB L1 give FASTER warm reads than a 30 GB L1? +# +# - If L1 RAM serves reads: L1=60 holds all 8 sessions -> every Round-2 read +# is L1-fast; L1=30 spills the oldest ~3 to L2 -> those read slower. +# - If L1 read is inert (the L2=0=0/8 hypothesis): both legs read everything +# from L2 -> identical Round-2 TTFTs -> the extra 30 GB is reclaimable. +# +# Method: for L1 in {30, 60}, recreate the container (L2 stays 1), run the +# retention test with a UNIQUE salt (fresh cache keys — cannot hit the 49 GB +# already on L2 disk, nor the other leg's blocks), capture the Round-2 curve. +# The L1=60 leg doubles as the restore to production. Compares raw Round-2 +# TTFTs (NOT pass/fail — with L2 on both legs "pass" the 8 s threshold). +# =========================================================================== +set -uo pipefail +cd /home/alex/club-3090 +C=vllm-qwen36-27b-lmcache +SLUG=vllm/qwen-27b-dual-lmcache +DIR=_local-lmcache-tuning +STAMP=$(date +%s) + +run_leg() { + local L1="$1" + local log="$DIR/l1ab-L1${L1}.log" + echo "====================================================================" + echo "LEG L1=${L1}GB (L2 stays 1) — $(date '+%H:%M:%S')" + echo "====================================================================" + # inline prefix: only THIS switch.sh sees the override; L2 comes from .env (=1). + LMCACHE_L1_GB="$L1" bash scripts/switch.sh "$SLUG" --force 2>&1 | tail -8 + local got_l1 got_l2 + got_l1=$(docker exec "$C" printenv LMCACHE_L1_GB 2>/dev/null || echo MISSING) + got_l2=$(docker exec "$C" printenv LMCACHE_L2 2>/dev/null || echo MISSING) + echo ">> container LMCACHE_L1_GB=$got_l1 LMCACHE_L2=$got_l2 (want L1=$L1 L2=1)" + if [ "$got_l1" != "$L1" ] || [ "$got_l2" != "1" ]; then + echo "!! ABORT leg: env did not match. Skipping test for L1=$L1." | tee "$log" + return 1 + fi + # Unique salt per leg => byte-different prefixes => guaranteed-cold Round 1, + # no contamination from L2 disk or the other leg. + SALT="abL1${L1}-${STAMP}" stdbuf -oL -eL \ + bash scripts/lmcache-retention-test.sh 2>&1 | tee "$log" + echo ">> leg L1=$L1 done, log=$log" +} + +echo "== L1 A/B START $(date '+%H:%M:%S') salt-stamp=$STAMP ==" +run_leg 30 +run_leg 60 # <- also the restore to production (.env L1=60) + +# Safety net: the RAM gate reads MemAvailable once (no settle-retry); if a slow +# reclaim after the L1=30 teardown tripped it, production could be left DOWN. +# Retry the plain (.env L1=60) restore up to twice before giving up. +for attempt in 1 2; do + FL1=$(docker exec "$C" printenv LMCACHE_L1_GB 2>/dev/null || echo MISSING) + up=$(docker ps --filter "name=$C" --format '{{.Names}}') + [ "$FL1" = "60" ] && [ -n "$up" ] && break + echo "!! production not at L1=60/up (got L1=$FL1, up='$up') — restore retry $attempt after 20s" + sleep 20 + bash scripts/switch.sh "$SLUG" --force 2>&1 | tail -6 +done + +# Final production sanity. +FL1=$(docker exec "$C" printenv LMCACHE_L1_GB 2>/dev/null || echo MISSING) +FL2=$(docker exec "$C" printenv LMCACHE_L2 2>/dev/null || echo MISSING) +echo ">> FINAL production state: LMCACHE_L1_GB=$FL1 LMCACHE_L2=$FL2 (want 60 / 1)" + +# ---- Compare the two Round-2 curves -------------------------------------- +echo "====================================================================" +echo "COMPARISON — Round-2 (warm) TTFT per session, L1=30 vs L1=60" +echo "====================================================================" +python3 - "$DIR/l1ab-L130.log" "$DIR/l1ab-L160.log" <<'PY' +import re, sys, statistics as st +def curve(path): + # Round-2 lines look like: " session 1: TTFT 2.13s WARM retained" + out=[] + try: + txt=open(path,encoding="utf-8").read() + except FileNotFoundError: + return out + seg=txt.split("Round 2",1) + body=seg[1] if len(seg)>1 else "" + for m in re.finditer(r"session\s+(\d+):\s+TTFT\s+([\d.]+)s", body): + out.append((int(m.group(1)), float(m.group(2)))) + return out +a=curve(sys.argv[1]); b=curve(sys.argv[2]) +da=dict(a); db=dict(b) +print(f"{'sess':>4} {'L1=30 s':>9} {'L1=60 s':>9} {'faster':>8}") +for s in sorted(set(da)|set(db)): + x=da.get(s); y=db.get(s) + xs=f"{x:.2f}" if x is not None else " --" + ys=f"{y:.2f}" if y is not None else " --" + fast = "" + if x is not None and y is not None: + fast = "L1=60" if y < x*0.85 else ("L1=30" if x < y*0.85 else "~tie") + print(f"{s:>4} {xs:>9} {ys:>9} {fast:>8}") +def summ(v): + xs=[t for _,t in v] + return f"n={len(xs)} mean={st.mean(xs):.2f}s median={st.median(xs):.2f}s max={max(xs):.2f}s" if xs else "no data" +print("\nL1=30 warm:", summ(a)) +print("L1=60 warm:", summ(b)) +if a and b: + ma,mb=st.mean(t for _,t in a),st.mean(t for _,t in b) + if mb < ma*0.85: + print(f"\nVERDICT: L1=60 warm reads ~{ma/mb:.1f}x faster => the RAM IS doing work.") + elif ma < mb*0.85: + print(f"\nVERDICT: L1=30 faster (unexpected) — investigate.") + else: + print("\nVERDICT: warm reads ~identical => extra 30 GB L1 is HEADROOM (reclaimable);" + " L2 disk is serving the reads either way.") +PY +echo "== L1 A/B DONE $(date '+%H:%M:%S') ==" diff --git a/_local-lmcache-tuning/l1ab-L130.log b/_local-lmcache-tuning/l1ab-L130.log new file mode 100644 index 00000000..a2bcaac6 --- /dev/null +++ b/_local-lmcache-tuning/l1ab-L130.log @@ -0,0 +1,32 @@ +LMCache L1 retention probe -> http://localhost:8017 model=qwen3.6-27b + sessions=8 target_tokens/session~=48000 warm_threshold=8.0s kv=131KB/tok + +Building calibrated haystacks (one /tokenize probe each)... + +Round 1 - cold insert (fills L1): + session 1: cold TTFT 53.20s (prompt_tokens=47559) + session 2: cold TTFT 52.73s (prompt_tokens=47991) + session 3: cold TTFT 53.78s (prompt_tokens=48207) + session 4: cold TTFT 51.33s (prompt_tokens=46318) + session 5: cold TTFT 52.83s (prompt_tokens=47015) + session 6: cold TTFT 54.30s (prompt_tokens=47676) + session 7: cold TTFT 55.95s (prompt_tokens=48343) + session 8: cold TTFT 56.34s (prompt_tokens=48098) + + inserted ~381207 tokens ~= 49.9 GB of L1 + capacity lines: L1=30 -> ~229K tok (~30 GB), L1=60 -> ~458K tok (~60 GB) + +Round 2 - retention read (same prefixes; warm = retained): + session 1: TTFT 3.32s WARM retained + session 2: TTFT 4.35s WARM retained + session 3: TTFT 4.16s WARM retained + session 4: TTFT 3.58s WARM retained + session 5: TTFT 5.09s WARM retained + session 6: TTFT 1.09s WARM retained + session 7: TTFT 1.87s WARM retained + session 8: TTFT 1.48s WARM retained + + retained warm: 8/8 + + PASS - all sessions retained. L1 held the full working set; + this set would have evicted its earliest sessions at L1=30. diff --git a/_local-lmcache-tuning/l1ab-L160.log b/_local-lmcache-tuning/l1ab-L160.log new file mode 100644 index 00000000..e2b2e90e --- /dev/null +++ b/_local-lmcache-tuning/l1ab-L160.log @@ -0,0 +1,32 @@ +LMCache L1 retention probe -> http://localhost:8017 model=qwen3.6-27b + sessions=8 target_tokens/session~=48000 warm_threshold=8.0s kv=131KB/tok + +Building calibrated haystacks (one /tokenize probe each)... + +Round 1 - cold insert (fills L1): + session 1: cold TTFT 55.80s (prompt_tokens=48631) + session 2: cold TTFT 51.99s (prompt_tokens=46869) + session 3: cold TTFT 54.48s (prompt_tokens=48031) + session 4: cold TTFT 55.22s (prompt_tokens=48240) + session 5: cold TTFT 55.62s (prompt_tokens=47982) + session 6: cold TTFT 58.44s (prompt_tokens=49035) + session 7: cold TTFT 56.92s (prompt_tokens=47843) + session 8: cold TTFT 58.67s (prompt_tokens=48719) + + inserted ~385350 tokens ~= 50.5 GB of L1 + capacity lines: L1=30 -> ~229K tok (~30 GB), L1=60 -> ~458K tok (~60 GB) + +Round 2 - retention read (same prefixes; warm = retained): + session 1: TTFT 2.39s WARM retained + session 2: TTFT 2.05s WARM retained + session 3: TTFT 1.42s WARM retained + session 4: TTFT 1.71s WARM retained + session 5: TTFT 1.27s WARM retained + session 6: TTFT 3.04s WARM retained + session 7: TTFT 1.05s WARM retained + session 8: TTFT 2.50s WARM retained + + retained warm: 8/8 + + PASS - all sessions retained. L1 held the full working set; + this set would have evicted its earliest sessions at L1=30. diff --git a/_local-lmcache-tuning/l1ab-run.log b/_local-lmcache-tuning/l1ab-run.log new file mode 100644 index 00000000..bbd3adf0 --- /dev/null +++ b/_local-lmcache-tuning/l1ab-run.log @@ -0,0 +1,110 @@ +== L1 A/B START 21:19:29 salt-stamp=1783970369 == +==================================================================== +LEG L1=30GB (L2 stays 1) — 21:19:29 +==================================================================== +[switch] 64s — Loading weights +[switch] 120s elapsed, still waiting... +[switch] 180s elapsed, still waiting... +[switch] 200s — Capturing CUDA graphs +[switch] 216s — Application startup complete +[switch] ✓ ready (216s) +[switch] ▶ API: http://192.168.1.98:8017/v1 (model: qwen3.6-27b · OpenAI-compatible · no auth) +[switch] done. Try: curl -s http://localhost:8017/v1/models | jq . +>> container LMCACHE_L1_GB=30 LMCACHE_L2=1 (want L1=30 L2=1) +LMCache L1 retention probe -> http://localhost:8017 model=qwen3.6-27b + sessions=8 target_tokens/session~=48000 warm_threshold=8.0s kv=131KB/tok + +Building calibrated haystacks (one /tokenize probe each)... + +Round 1 - cold insert (fills L1): + session 1: cold TTFT 53.20s (prompt_tokens=47559) + session 2: cold TTFT 52.73s (prompt_tokens=47991) + session 3: cold TTFT 53.78s (prompt_tokens=48207) + session 4: cold TTFT 51.33s (prompt_tokens=46318) + session 5: cold TTFT 52.83s (prompt_tokens=47015) + session 6: cold TTFT 54.30s (prompt_tokens=47676) + session 7: cold TTFT 55.95s (prompt_tokens=48343) + session 8: cold TTFT 56.34s (prompt_tokens=48098) + + inserted ~381207 tokens ~= 49.9 GB of L1 + capacity lines: L1=30 -> ~229K tok (~30 GB), L1=60 -> ~458K tok (~60 GB) + +Round 2 - retention read (same prefixes; warm = retained): + session 1: TTFT 3.32s WARM retained + session 2: TTFT 4.35s WARM retained + session 3: TTFT 4.16s WARM retained + session 4: TTFT 3.58s WARM retained + session 5: TTFT 5.09s WARM retained + session 6: TTFT 1.09s WARM retained + session 7: TTFT 1.87s WARM retained + session 8: TTFT 1.48s WARM retained + + retained warm: 8/8 + + PASS - all sessions retained. L1 held the full working set; + this set would have evicted its earliest sessions at L1=30. +>> leg L1=30 done, log=_local-lmcache-tuning/l1ab-L130.log +==================================================================== +LEG L1=60GB (L2 stays 1) — 21:30:54 +==================================================================== +[switch] 68s — Loading weights +[switch] 120s elapsed, still waiting... +[switch] 180s elapsed, still waiting... +[switch] 208s — Capturing CUDA graphs +[switch] 224s — Application startup complete +[switch] ✓ ready (224s) +[switch] ▶ API: http://192.168.1.98:8017/v1 (model: qwen3.6-27b · OpenAI-compatible · no auth) +[switch] done. Try: curl -s http://localhost:8017/v1/models | jq . +>> container LMCACHE_L1_GB=60 LMCACHE_L2=1 (want L1=60 L2=1) +LMCache L1 retention probe -> http://localhost:8017 model=qwen3.6-27b + sessions=8 target_tokens/session~=48000 warm_threshold=8.0s kv=131KB/tok + +Building calibrated haystacks (one /tokenize probe each)... + +Round 1 - cold insert (fills L1): + session 1: cold TTFT 55.80s (prompt_tokens=48631) + session 2: cold TTFT 51.99s (prompt_tokens=46869) + session 3: cold TTFT 54.48s (prompt_tokens=48031) + session 4: cold TTFT 55.22s (prompt_tokens=48240) + session 5: cold TTFT 55.62s (prompt_tokens=47982) + session 6: cold TTFT 58.44s (prompt_tokens=49035) + session 7: cold TTFT 56.92s (prompt_tokens=47843) + session 8: cold TTFT 58.67s (prompt_tokens=48719) + + inserted ~385350 tokens ~= 50.5 GB of L1 + capacity lines: L1=30 -> ~229K tok (~30 GB), L1=60 -> ~458K tok (~60 GB) + +Round 2 - retention read (same prefixes; warm = retained): + session 1: TTFT 2.39s WARM retained + session 2: TTFT 2.05s WARM retained + session 3: TTFT 1.42s WARM retained + session 4: TTFT 1.71s WARM retained + session 5: TTFT 1.27s WARM retained + session 6: TTFT 3.04s WARM retained + session 7: TTFT 1.05s WARM retained + session 8: TTFT 2.50s WARM retained + + retained warm: 8/8 + + PASS - all sessions retained. L1 held the full working set; + this set would have evicted its earliest sessions at L1=30. +>> leg L1=60 done, log=_local-lmcache-tuning/l1ab-L160.log +>> FINAL production state: LMCACHE_L1_GB=60 LMCACHE_L2=1 (want 60 / 1) +==================================================================== +COMPARISON — Round-2 (warm) TTFT per session, L1=30 vs L1=60 +==================================================================== +sess L1=30 s L1=60 s faster + 1 3.32 2.39 L1=60 + 2 4.35 2.05 L1=60 + 3 4.16 1.42 L1=60 + 4 3.58 1.71 L1=60 + 5 5.09 1.27 L1=60 + 6 1.09 3.04 L1=30 + 7 1.87 1.05 L1=60 + 8 1.48 2.50 L1=30 + +L1=30 warm: n=8 mean=3.12s median=3.45s max=5.09s +L1=60 warm: n=8 mean=1.93s median=1.88s max=3.04s + +VERDICT: L1=60 warm reads ~1.6x faster => the RAM IS doing work. +== L1 A/B DONE 21:42:31 == diff --git a/scripts/lmcache-retention-test.sh b/scripts/lmcache-retention-test.sh index 1a4e9484..4862ff34 100755 --- a/scripts/lmcache-retention-test.sh +++ b/scripts/lmcache-retention-test.sh @@ -41,6 +41,11 @@ # re-prefill is ~35-45 s — 8 s cleanly separates them) # KV_KB_PER_TOKEN LMCache cache rate for the GB math. Default: 131 (measured) # PROBE_MAX_TOKENS Output cap per request (just enough to complete). Default: 8 +# SALT Mixes into every prefix's seed + anchor so a re-run inserts +# BYTE-DIFFERENT prefixes -> fresh cache keys that cannot hit +# KV left on L2 disk by a prior run. Default: "" (the original +# deterministic prefixes). Give each A/B leg a distinct SALT so +# the two legs never read each other's (or old) cached blocks. # # Exit 0 = all sessions retained (PASS). Exit 1 = one or more evicted. # =========================================================================== @@ -54,18 +59,22 @@ SESSION_TOKENS="${SESSION_TOKENS:-48000}" WARM_THRESHOLD_S="${WARM_THRESHOLD_S:-8.0}" KV_KB_PER_TOKEN="${KV_KB_PER_TOKEN:-131}" PROBE_MAX_TOKENS="${PROBE_MAX_TOKENS:-8}" +SALT="${SALT:-}" command -v python3 >/dev/null || { echo "Fix: python3 not found on PATH." >&2; exit 2; } python3 - "$URL" "$MODEL" "$NUM_SESSIONS" "$SESSION_TOKENS" \ - "$WARM_THRESHOLD_S" "$KV_KB_PER_TOKEN" "$PROBE_MAX_TOKENS" <<'PY' -import json, random, string, sys, time, urllib.request + "$WARM_THRESHOLD_S" "$KV_KB_PER_TOKEN" "$PROBE_MAX_TOKENS" "$SALT" <<'PY' +import json, random, string, sys, time, urllib.request, zlib # Community rigs run non-UTF-8 locales; a piped stdout defaults to ASCII (repo # convention: pin utf-8 on both read and write). sys.stdout.reconfigure(encoding="utf-8") -URL, MODEL, N, STOK, WARM, KVKB, PMAX = sys.argv[1:] +URL, MODEL, N, STOK, WARM, KVKB, PMAX, SALT = sys.argv[1:] N = int(N); STOK = int(STOK); WARM = float(WARM); KVKB = float(KVKB); PMAX = int(PMAX) +# Stable integer offset derived from SALT (crc32 is process-independent, unlike +# builtin hash() under PYTHONHASHSEED). Empty SALT -> 0 -> original prefixes. +SALT_OFF = (zlib.crc32(SALT.encode()) % 1_000_000) if SALT else 0 def tokenize_count(text): @@ -90,14 +99,16 @@ def rand_words(rng, n): def make_prefix(idx, target_tokens): - """Deterministic per idx -> byte-identical across rounds -> real cache hit. + """Deterministic per (idx, SALT) -> byte-identical across rounds within a run + (real cache hit) but byte-DIFFERENT across salts (fresh key vs prior runs). Calibrated against the live tokenizer so token count is ~on target.""" - rng = random.Random(idx * 7919 + 13) - sample = " ".join(rand_words(random.Random(idx), 400)) + key = idx + SALT_OFF + rng = random.Random(key * 7919 + 13) + sample = " ".join(rand_words(random.Random(key), 400)) c = tokenize_count(sample) tpw = (c / 400.0) if c else 1.4 # tokens-per-word for this stream n_words = max(64, int(target_tokens / tpw)) - header = f"SESSION-{idx:03d}-anchor-{idx * 7919}\n" + header = f"SESSION-{idx:03d}-salt-{SALT_OFF}-anchor-{key * 7919}\n" body = header + " ".join(rand_words(rng, n_words)) return body + "\n\nReply with only: OK"