ff4bbbc09d
* LMCache: in-repo L2 default + LMCACHE_L2=1 toggle, preflight disk-check, corrected sizing - LMCACHE_L2=1 enables the L2 disk tier at a gitignored repo-root lmcache-kv/ (zero config); LMCACHE_L2_ADAPTER still fully overrides. L2 stays OFF by default. - preflight_lmcache_ram now also soft-warns on low L2 disk space at the L2 host dir. - CORRECTED capacity (measured > estimated): the LMCache offload cache is ~131 KB/token MEASURED (lmcache_mp_l1_memory_usage_bytes 4.93 GB + 4.6 GB on L2 disk, 36,808-token session) — ~7x the GPU's 18.9 int8-PTH rate, which only sets max servable context. Prior docs used 18.9 for capacity and overstated it ~7x: --l1-size-gb 30 holds ~4 x 50K sessions (not ~33), <1 full 262K. - Added a RAM/disk-vs-context sizing table to INTERNALS.md. - Measured L2 rehydrate: 4.8s vs 43s cold re-prefill (~9x), cross-restart persistence confirmed; use the fs adapter (not nixl_store — this image's NIXL is broken). Refs #133. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * LMCache: document fp8 serde is broken on this model (no reduced-size L2 yet) Measured 2026-06-17: LMCache's fp8 serde shape-errors on this model's KV ('[2,16,1584,260]' invalid — wrong layout for the hybrid GDN/head_dim-256/int8-PTH source), 60 serialize fails, L2 stays empty. CacheGen-MP is 'coming soon'. So L2 is uncompressed (~131 KB/token); the fp8-serde toggle was NOT wired (it doesn't work). Re-test trigger noted. Refs #133. --------- Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
6 lines
266 B
Plaintext
6 lines
266 B
Plaintext
# LMCache L2 disk tier (gitignored runtime cache — KV chunks, multi-GB).
|
|
# Default base_path for the vllm/qwen-27b-dual-lmcache compose when LMCACHE_L2=1.
|
|
# Override to a faster/larger SSD via LMCACHE_L2_ADAPTER. See models/qwen3.6-27b/INTERNALS.md.
|
|
*
|
|
!.gitignore
|