Local: session report + L1 A/B artifacts + coldcase test notes
Adds SESSION-REPORT.md (LMCache tuning, coldcase big-model test, Hermes agentic audio_type fix) and the L1 A/B run logs. Branch-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Alex
72
_local-lmcache-tuning/SESSION-REPORT.md
Normal file
72
_local-lmcache-tuning/SESSION-REPORT.md
Normal file
@@ -0,0 +1,72 @@
|
|||||||
|
# Session report — LMCache tuning + coldcase big-model test + Hermes agentic fix
|
||||||
|
**Date:** 2026-07-13 (into 07-14) · **Rig:** .98 (club-3090, 2×3090) · **Branch:** `rig-local-tuning`
|
||||||
|
|
||||||
|
## TL;DR
|
||||||
|
- **LMCache L1=60 config validated prod-ready** on `:8017`. Your spare RAM *is* doing
|
||||||
|
work (~1.6–2× faster warm reads), and the retention itself comes from the L2 disk tier.
|
||||||
|
- **Coldcase "big Hermes test" run** against `:8017`: the local model serves the real app
|
||||||
|
correctly (38% external prefix-cache hits). LMCache helps the short chat-style flows
|
||||||
|
(evaluate ~1.3×), not the decode-bound bulk generation.
|
||||||
|
- **A real coldcase bug found and fixed by the local model itself** (Hermes agentic),
|
||||||
|
then applied to the live service.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. LMCache tuning (branch commits)
|
||||||
|
- Deployed `vllm/qwen-27b-dual-lmcache` on `:8017` (qwen3.6-27b fp8, 262K ctx), **L1=60 / L2=1**,
|
||||||
|
`shm_size 32→64` (commit c88c1883), power cap fixed to 250 W both cards.
|
||||||
|
- **L2=0 vs L2=1 experiment:** retention (whether a read is warm at all) is delivered by the
|
||||||
|
**L2 disk** tier, not L1 RAM — L2 off → 0/8 retained; L2 on → 8/8 (52 s cold → <5 s warm).
|
||||||
|
- **L1=30 vs L1=60 A/B (L2 on):** the RAM tier controls warm-read *speed* — L1=60 warm reads
|
||||||
|
~1.6× faster mean (~2× on the sessions that spill past a 30 GB L1). **Decision: keep L1=60**
|
||||||
|
— it is not reclaimable headroom. (commit b03a1349)
|
||||||
|
- Coherent two-part model: **L2 disk = whether warm; L1 RAM = how fast.** LMCache is a
|
||||||
|
chat-*reuse* optimization, not a throughput one.
|
||||||
|
|
||||||
|
## 2. Coldcase big-model test (.96 → :8017)
|
||||||
|
Coldcase = Danish cold-case murder-mystery web app on `.96` (live `coldcase.service`, :3099).
|
||||||
|
Temporarily repointed it (and Hermes) from `:8010` autoround to the `:8017` LMCache model.
|
||||||
|
|
||||||
|
| Flow | Result |
|
||||||
|
|---|---|
|
||||||
|
| Mystery generation (`generateMystery`) | ✅ full coherent mystery (20 docs / 7 chars / 7 locs), but **decode-bound ~31 min** → LMCache barely helps |
|
||||||
|
| Room evaluation (`evaluateAnswer`) | ✅ coherent Danish feedback; **cold 9.1 s → warm 7.0 s (~1.3× LMCache win)** — the flow where LMCache pays off |
|
||||||
|
| Server-side proof | ✅ `External prefix cache hit rate 38.2%` on `:8017` |
|
||||||
|
|
||||||
|
## 3. Bug found + fixed by the local model
|
||||||
|
- 🐛 **`audio_type` enum mismatch:** generation completes, then the DB save dies with
|
||||||
|
*"Data truncated for column 'audio_type'"* — the model emits audio labels outside coldcase's
|
||||||
|
strict `ENUM(emergency_call…eavesdropped)`; no coerce layer. (Bug note:
|
||||||
|
`~/coldcase-temp/BUG-audio_type-enum.md`.)
|
||||||
|
- 🤖 **Hermes (local qwen3.6-27b, headless `hermes -z --yolo`)** implemented the fix: a
|
||||||
|
`normalizeAudioType()` helper (enum list matching the DB exactly + EN/Danish synonym map +
|
||||||
|
fallback `recorded_conversation`), applied at the INSERT + upstream plan sites. Reviewed &
|
||||||
|
verified: `VALID_AUDIO_TYPES` matches the DB enum exactly, zero new build errors.
|
||||||
|
- ✅ **Applied the verified fix to the LIVE `/opt/coldcase`** (targeted edits, backups
|
||||||
|
`*.bak-audiofix-*`): `provider.ts` (+helper), `generator.ts` (:115), `multi_stage.ts`
|
||||||
|
(:769/:777). tsc clean on the new code; live service healthy after `tsx` hot-reload.
|
||||||
|
|
||||||
|
## 4. Housekeeping
|
||||||
|
- **Pruned `lmcache-kv`** (unbounded L2, host `/home/alex/club-3090/lmcache-kv`): had hit
|
||||||
|
**210 GB, disk at 97%** → cleared → **disk 74% (240 GB free)**, `:8017` restarted fresh at
|
||||||
|
L1=60/L2=1. Prune this periodically.
|
||||||
|
|
||||||
|
## 5. End-to-end save confirmation
|
||||||
|
- Full generation re-run against the fixed live service to confirm Phase-2 save now succeeds.
|
||||||
|
- **RESULT: _(appended on completion below)_**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Rig state at shutdown / notes for next boot
|
||||||
|
- **`:8017` LMCache was launched manually** via `switch.sh`, not a systemd autostart. After
|
||||||
|
reboot it will NOT come back on its own — re-launch with:
|
||||||
|
`cd ~/club-3090 && bash scripts/switch.sh vllm/qwen-27b-dual-lmcache --force`
|
||||||
|
- **`:8010` autoround** (`club3090-vllm-dual.service`) is currently inactive (the two TP=2
|
||||||
|
27Bs can't share the 2×3090s). On reboot systemd may bring it up instead — only one of
|
||||||
|
`:8010`/`:8017` can run at a time.
|
||||||
|
- **Coldcase (.96) is pointed at `:8017`** (`.env` + `provider.ts`, backups `*.bak-lmtest-*`)
|
||||||
|
and carries the audio_type fix. If you don't re-launch `:8017` after boot, coldcase
|
||||||
|
generation will fail — either re-launch `:8017`, or revert coldcase to `:8010` from the
|
||||||
|
backups (keep the audio fix). **Hermes** config also points at `:8017` (backup `*.bak-lmtest-*`).
|
||||||
|
- Branch `rig-local-tuning`: commits c88c1883 (shm), 99cbd81a (retention test), b03a1349
|
||||||
|
(L1 A/B), plus this report + artifacts under `_local-lmcache-tuning/`. Master untouched.
|
||||||
Reference in New Issue
Block a user