alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions 38 Packages Projects Releases Wiki Activity
Files
525f5a00cb40669b47e641e28102b2e1c7de24a6
club-3090/tools/residency-instrument
History
noonghunna eaa7a8c1ad docs+scripts: finish <quant>/ path migration across full repo sweep 
Repo-wide follow-up to the compose quant-layer move (9821c94). The
mechanical move + registry/launch/test rewire covered the launchable
surface; a full-tree sweep found compose-path refs the move invalidated
in docs, two functional scripts, and one half-migrated mapping.

Functional fixes:
- bench-row-formatter.sh infer_compose_path(): 6 dual entries
  (int8-tq3, tq3-mtp-genesis, tq3-nomtp, tq3-mtp, int8, bf16) were left
  at bare dual/<file>.yml while the rest were migrated -> would emit
  dead compose paths into BENCHMARKS rows.
- residency-instrument/run-instrumented-soak.sh: case->COMPOSE_FILE
  paths (long-text, long-text-no-mtp, tools-text, dual default) now
  resolve under <quant>/.

Docs: README layout line + tree, engine/model READMEs, patch-README
quick-recipes, diagnostics, FAQ/CLIFFS/KV_MATH/DTYPE/MULTI/SINGLE/
STRUCTURED_COT/TQ3/UPSTREAM, issue template, sglang cross-refs
(-> vllm prod path). Per-model targets: qwen-vllm->autoround-int4,
llama-cpp->unsloth-q4km, gemma defaults (bf16-mtp/fp8-mtp),
carnice->own slug dir.

Intentionally left as historical/append-only records: CHANGELOG x2,
BENCHMARKS row-labels (live paths already correct in row bodies),
calibration source: provenance citations, switch.sh/parity history
comments. Separate follow-ups: gpu-mode.sh (#417 deprecated-repo
repoint), bench-row-formatter compose_display() docker-compose.yml
branch (PR-B). Flagged pre-existing-stale: dual/int8-tq3.yml in
pr40798/pr40914 READMEs (predate this refactor; ambiguous target).

Guard tests (registry-disk, mounts-resolve, switch-parity,
launch-compat) all PASS post-edit. Leak-clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:50:13 +00:00
..
instrument.py
chore(tools): commit residency-instrument as research tool with framing README (#41, #217)
2026-05-04 13:29:17 +00:00
README.md
chore(tools): commit residency-instrument as research tool with framing README (#41, #217)
2026-05-04 13:29:17 +00:00
run-instrumented-soak.sh
docs+scripts: finish <quant>/ path migration across full repo sweep
2026-05-26 16:50:13 +00:00
sitecustomize.py
chore(tools): commit residency-instrument as research tool with framing README (#41, #217)
2026-05-04 13:29:17 +00:00

README.md

residency-instrument — Cliff 2b activation-residency probe

Research-grade instrumentation harness used to investigate the multi-turn accumulating-context cliff (Cliff 2b) on single-card vLLM. Not a stable user-facing tool — preserved for archeological value and to let cross-rig contributors reproduce the residency findings underlying docs/CLIFFS.md Cliff 2b.

Status

  • Built for: club-3090#41 (Cliff 2b investigation, 2026-05-03)
  • Origin: Codex-authored against docs/diagnostics/cliff2-followup-residency-instrumentation-brief.md (gitignored)
  • SLA: none. May need adaptation for future PyTorch / vLLM / Genesis pin combinations. If it breaks, file an issue and we'll triage on best-effort — this is research-grade tooling, not a shipped feature.

What it does

Instruments three layers of the vLLM stack via a sitecustomize.py hook (auto-loaded by Python on container start):

  1. PyTorch caching allocator — periodic snapshots of pool sizes, fragmentation, free segments
  2. Engine step boundaries — request/response transitions, KV pool deltas across turns
  3. DeltaNet GDN forward call sites — vllm.attention.backends.turboquant_attn allocation patterns at the activation peak

Output is CSV per request boundary; default lands in results/residency-<timestamp>/ (gitignored). Joining instrumentation rows to soak-test turn rows produces the residency-vs-time picture that fed the Cliff 2b root-cause analysis ("fragmentation-dominated, not pool growth — PN12 stays flat at 137 MiB across the cliff fire").

The harness is observational only — it monkey-patches request/engine/worker boundaries to write snapshots, but does not change scheduling, memory policy, or kernels.

How to run

# Defaults target the known Cliff 2b reproducer (long-text + 2-session × 5-turn soak)
bash tools/residency-instrument/run-instrumented-soak.sh

# Override target compose / soak shape:
VARIANT=vllm/bounded-thinking \
SOAK_SESSIONS=5 SOAK_TURNS=5 \
bash tools/residency-instrument/run-instrumented-soak.sh

The script:

  1. Boots the target compose with tools/residency-instrument/sitecustomize.py mounted as a Python entry point via PYTHONPATH
  2. Runs SOAK_MODE=continuous bash scripts/soak-test.sh against the booted endpoint
  3. Joins raw instrumentation rows to soak turn rows
  4. Writes the merged output to results/residency-<timestamp>/

Output is local-only

results/residency-* is gitignored by repo policy. We don't auto-publish diagnostic outputs because they're large, rig-specific, and most of the analytical value lives in the issue thread + commit log + memory entries that distill what was learned. If you investigate Cliff 2b on your own rig and want to share findings, the right surface is a comment on the relevant issue (e.g. #41) — paste the summary table, not the raw CSV.

Cross-links

Don't run on production composes

The sitecustomize.py mount changes Python's startup behavior inside the container. It's safe for investigation runs but slows boot and adds CPU overhead per request. Use a dedicated compose for instrumentation; tear down + re-launch the regular variant when you're done.