docs(agents): fix stale guidance + add user-rig framing to AGENTS.md

Audit of the agent guide against current repo state; ten fixes:

- Hardware truths: reframe as "reference rig" — repo supports 10 hardware
  classes (profiles/hardware/*.yml); Ampere-only constraints (no native FP8
  compute) no longer stated as universal; fp8 composes are 5090-safe via the
  launcher's arch-conditional VLLM_USE_DEEP_GEMM pass-through; MTP +
  beellama-DFlash noted as the working spec-dec paths.
- beellama default-resolver text: it HAS live DEFAULTS entries and IS the
  single-card default for qwen3.6-27b + gemma-4-31b (text said "no entries
  yet, blocked on upstream image").
- Engine-pin table: vLLM pins are release tags (never nightly-* — upstream
  purges them); beellama row added (digest-pinned); patches.yml named as
  patch source of truth over a hand-listed set.
- New convention: encoding="utf-8" on every Python read of repo files —
  non-UTF-8-locale community rigs crash on unicode compose headers (#599);
  incl. the repro recipe + the no-blind-2>/dev/null corollary.
- Tests: rebench-full.sh documented as the canonical eval orchestrator;
  MODEL=<served-name> 404-gotcha called out.
- New subsection: tools/serve-cockpit (c3) — own venv/pytest suite, fast vs
  slow test split, registry-emit --json contract, VS16-emoji width rule.
- Full-suite scoping: catalog-shape changes only; targeted guards otherwise.
- Orphan-compose list (all 3 files gone) replaced with the compose/_archive/
  convention (archive vs deprecate distinction).
- Genesis section: status line — no shipped compose enables Genesis;
  guidance retained for reintroduction.
- Symlink self-description fixed: AGENTS.md is canonical, CLAUDE.md links it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
This commit is contained in:
noonghunna
2026-07-06 20:21:35 +00:00
parent 7fbccd1936
commit 196d18a213

View File

@@ -2,7 +2,7 @@
Guidance for AI coding agents (Claude Code, Cursor, Copilot, Continue, etc.) working in this repo. Focused — only conventions an agent wouldn't infer from the code itself.
> **One file, two names:** this is `CLAUDE.md` (canonical); **`AGENTS.md` is a symlink → it.** Edit `CLAUDE.md` — both names resolve to the same guide so any agent that looks for either finds it, and the two can't drift.
> **One file, two names:** this is `AGENTS.md` (canonical); **`CLAUDE.md` is a symlink → it.** Edit either — both names resolve to the same guide so any agent that looks for either finds it, and the two can't drift.
## Read first
@@ -19,10 +19,13 @@ Before making non-trivial changes:
## Hardware truths
- 2× RTX 3090 Ampere SM 8.6, PCIe-only, **no NVLink** (and we won't add it).
- Custom all-reduce must be disabled in vLLM/SGLang configs (PCIe topology breaks NVLink-assumed paths).
- No native FP8 compute; FP8 KV is a storage optimization only.
- Speculative decoding using EAGLE / DFlash is blocked on Qwen3-Next family (DeltaNet rollback). MTP works. See [`docs/UPSTREAM.md`](docs/UPSTREAM.md) — vllm#39931.
**The reference rig** (the maintainer's, where all first-party numbers come from): 2× RTX 3090 Ampere SM 8.6, PCIe-only, **no NVLink** (and we won't add it). Its constraints:
- Custom all-reduce must be disabled in vLLM/SGLang configs on PCIe-only multi-GPU topologies (NVLink-assumed paths break).
- **Ampere (sm_86) has no native FP8 compute** — there, FP8 KV is a storage optimization only. On Ada/Hopper/Blackwell, FP8 compute paths are real (fp8-weights composes are 5090-safe via the launcher's `VLLM_USE_DEEP_GEMM` pass-through — an arch-conditional guard, not a blanket "everything works on 5090").
- Speculative decoding using EAGLE / DFlash **inside vLLM** is blocked on Qwen3-Next family on every arch (DeltaNet rollback). MTP works, and DFlash works via the beellama engine (external drafter). See [`docs/UPSTREAM.md`](docs/UPSTREAM.md) — vllm#39931.
**The rig you're running on may not be the reference rig.** Supported hardware classes live in `scripts/lib/profiles/hardware/*.yml` (3060 → 5090, A5000, A100, H100, DGX Spark, …); the profile-catalog compat layer and launchers key on the *detected* class and inject arch-aware env. Before assuming any constraint above applies, check which class you're on (`nvidia-smi` + the matching hardware YAML) — and never hand-copy a reference-rig workaround (e.g. disabling all-reduce) onto an NVLink-equipped or non-Ampere rig without checking it's still warranted.
## Upstream issues — single source of truth
@@ -41,6 +44,8 @@ When filing a fresh upstream issue from this work:
3 warm + 5 measured runs. Canonical prompts: 800-word essay (narrative, max_tokens=1000) + quicksort code (max_tokens=800). `temperature=0.6, top_p=0.95, top_k=20`. Capture both wall-time TPS and engine-internal `gen throughput` from logs. **Always capture per-card peak VRAM** alongside TPS.
### Genesis opt-in env vars
**Status (2026-07-06): no shipped compose currently enables Genesis** — the Genesis-pinned production paths were retired (their composes live under `compose/_archive/`). The guidance below stays because it applies verbatim if a Genesis-pinned compose is reintroduced, and the incident it encodes is the canonical example of why behavioral patches need repro-gating.
Genesis ships ~50 env-gated patches. Some are **targeted bugfixes** (P64 streaming, PN8 memory savings, P3/P5/P6 KV); others are **behavioral mitigations** that silently rewrite the request (P68 = `tool_choice → required`, P69 = inject "must use tool" reminder). Behavioral mitigations need a streaming + large-prompt repro before shipping default-on. We learned this the hard way on 2026-04-29 — see [`docs/UPSTREAM.md`](docs/UPSTREAM.md) → Genesis #9 row + the [club-3090 #2 thread](https://github.com/noonghunna/club-3090/issues/2#issuecomment-4346740245).
If you're considering enabling a new Genesis env var by default in a shipped compose:
@@ -56,13 +61,19 @@ Pin engine images only when we vendor patches into the running container. Otherw
| Engine | Patches we vendor | Tag policy |
|---|---|---|
| `llama.cpp` | none | rolling `ghcr.io/ggml-org/llama.cpp:server-cuda` |
| `vLLM` | Genesis sidecars, Marlin pad, INT8 PTH, DFlash overlays | pinned to a specific nightly digest |
| `vLLM` | Marlin pad, KV/loader overlays — `scripts/lib/profiles/patches.yml` is the source of truth | pinned **release tag** (e.g. `v0.22.0`, `v0.24.0`) — **never `nightly-*`/`latest`**: upstream purges nightly tags, orphaning the pin |
| `beellama.cpp` | none (the fork *is* the delta) | **digest-pinned** to Anbeeld's official image (`engines/beellama-local.yml` `install.spec`) |
| `SGLang` | per-compose decision (pin if we vendor a patch, rolling otherwise) | per-compose |
When adding the first vendored patch to a previously-rolling engine: pin in the same commit. When dropping the last patch: unpin in the same commit. Bump pins via PR with a `verify-full.sh` + `bench.sh` re-run, never silently.
**Delivery model (vLLM):** patches reach the container by **volume-mounting into the pinned *stock* `vllm/vllm-openai` image** (python sidecars / site-package overlays / install scripts — see `delivery_mechanism` in `scripts/lib/profiles/patches.yml`), **not** by baking a custom image. The older baked-image path (`ghcr.io/noonghunna/vllm-club3090`, which shipped the release images through `club-v0.8.3`) is **retired** — no compose or engine-pin references it, and the `dockerfile_bake` `delivery:` block in `patches.yml` is legacy/test-only. The GHCR package is kept as historical release artifacts (users pinned to a `club-v0.8.x` tag can still pull); it is not deleted and not produced by anything in-repo.
### File encoding in scripts — always `encoding="utf-8"`
Any Python read of a repo source file (compose YAML, profile YAML, `baselines.yml`, …) — including python heredocs inside shell scripts — MUST pass `encoding="utf-8"`. `Path.read_text()` / `open()` default to the **locale** encoding, and community rigs run non-UTF-8 locales (minimal VMs / containers with `LC_ALL=C``ANSI_X3.4-1968`). Repo files are full of unicode (`— × → ⚠` in compose headers), so a bare read that works on the dev machine crashes `switch.sh`/`launch.sh` on those rigs (#599). Two corollaries:
- **Repro before claiming fixed:** modern Python coerces `C``C.UTF-8`, so plain `LC_ALL=C` won't reproduce — use `PYTHONUTF8=0 PYTHONCOERCECLOCALE=0 LC_ALL=C`.
- **Don't blind-`2>/dev/null` launcher derive paths** — swallowing the traceback hid this exact class for months; capture stderr and surface it on failure instead.
### CHANGELOG
- `CHANGELOG.md` (cross-cutting) and `models/<name>/CHANGELOG.md` (per-model) are **append-only history**. Don't rewrite past entries even when a finding is superseded — add a new entry. The historical trail is load-bearing for "why did we do X."
- Old entries can reference files / patches that no longer exist. That's fine — leave them.
@@ -89,7 +100,7 @@ The directory hierarchy encodes model, engine, topology, and the weights artifac
**Default-resolver knobs** (maintainer-owned, next to `DEFAULTS` in `compose_registry.py`):
- `DEFAULTS[(model, engine, topology)] → slug` — the `<engine>/default` map (club-3090's recommended config per engine; reason can evolve, edited by PR).
- `ENGINE_PREFERENCE[topology] → [engine, …]` — the curated `<model>/default` policy. The resolver walks this list and picks the first engine with a **functional** (`status ∉ {experimental, preview, upstream-gated, deprecated}`) `DEFAULTS` entry. **Reorder a row to change a recommendation — no code change, any topology.** single = `[beellama, ik-llama, llamacpp, vllm]`; dual/multi = `[vllm, ik-llama, llamacpp, beellama]`. `beellama` is ranked but has no entries yet (blocked on upstream image → `docs/UPSTREAM.md`); the resolver skips it and it auto-promotes to single-default on catalog.
- `ENGINE_PREFERENCE[topology] → [engine, …]` — the curated `<model>/default` policy. The resolver walks this list and picks the first engine with a **functional** (`status ∉ {experimental, preview, upstream-gated, deprecated}`) `DEFAULTS` entry. **Reorder a row to change a recommendation — no code change, any topology.** single = `[beellama, ik-llama, llamacpp, vllm]`; dual/multi = `[vllm, ik-llama, llamacpp, beellama]`. `beellama` leads the single ranking **and has live `DEFAULTS` entries** — it IS the shipped single-card default for `qwen3.6-27b` (`beellama/dflash`) and `gemma-4-31b` (`beellama/gemma-dflash`).
- `RECOMMENDED_DEFAULT_MODELS` — a **short opt-in shortlist** (`["qwen3.6-27b", "gemma-4-31b"]`) of models eligible to be the *bare-`launch.sh`* default (first installed → its `<model>/default`). **NOT** an exhaustive ranking; absent models are runnable by name but never auto-default; **new models are NOT auto-added** — promote one explicitly.
- The shared resolver `model_default_target(root, model, topology)` (in `registry-emit.sh`) is the single injection point for both launchers. Precedence: `--variant` → user `.env` pin (`CLUB3090_DEFAULT_<MODELID, non-alnum→_>`) → community seam (`community_default_target``None` today) → curated walk → degradation (nearest-lower topology, else "pick explicitly"). `X/default` dispatch: `X ∈ engine-set` → engine rec; `X ∈ model-set` → model default; else error. Users pin/clear via `switch.sh --set-default <slug>` / `--clear-default <model>`.
@@ -180,7 +191,7 @@ Read that doc before catalog work; the at-a-glance for agents:
- **Just serving, not cataloging?** Safetensors → `scripts/pull.sh <org/Model> --profile-like vllm/minimal`; a self-grabbed **GGUF** → copy an ik/llama compose and point `--model` at it (no registry/profile needed — see ADDING_MODELS "Run a local GGUF without the catalog"). The steps below are only for promoting a model into the **curated catalog**.
- **Catalog steps the compose alone doesn't cover:** (1) `scripts/lib/profiles/models/<id>.yml``weights:` is a **map keyed by quant-slug**, not a list; (2) a `compose_registry.py` entry (`weights_variant`=slug · `kvcalc_key` — vLLM `"<model>:<profile>"`, ik/llama `"SKIP"` · `default_port` == the compose's `${PORT:-NNNN}`); (3) launchers **auto-derive** from the registry — never edit `launch.sh`/`switch.sh`; promote a default via the `DEFAULTS` map.
- **Profile-catalog compatibility (easy to miss — hotfix #236):** the new `(model, engine, KV-format)` combo must validate or `test-profiles-compat` / `diagnose-profile` go red. Add the model's `family` to the engine's `supported_model_families` (`scripts/lib/profiles/engines/*.yml`), the KV format to the hardware profiles' `supported_kv_formats` (`scripts/lib/profiles/hardware/*.yml`); register any vendored chat-template in `scripts/lib/profiles/patches.yml` (with the symmetric-protocol `drift_guard`); bump the `test-compose-registry-disk` size-count.
- **Run the FULL catalog test suite**, not just the serving tests in [Tests](#tests): `for t in scripts/tests/*.sh; do bash "$t"; done`. Key gates: `test-compose-registry-disk`, `test-compose-mounts-resolve` (the `../` depth), `test-model-weights-registry`, `test-switch-registry-parity` + `test-launch-registry-parity`, `test-profiles-compat`, `test-patch-attribution`, plus `tools/kv-calc.py --calibration`. A narrow subset shipped a model with two real catalog gaps (#236) — and some failures are pre-existing/env, so **baseline against the last release tag** before treating one as a blocker.
- **Run the FULL catalog test suite for catalog-shape changes** (new model / engine / slug / profile-schema change), not just the serving tests in [Tests](#tests): `for t in scripts/tests/*.sh; do bash "$t"; done`. Key gates: `test-compose-registry-disk`, `test-compose-mounts-resolve` (the `../` depth), `test-model-weights-registry`, `test-switch-registry-parity` + `test-launch-registry-parity`, `test-profiles-compat`, `test-patch-attribution`, plus `tools/kv-calc.py --calibration`. A narrow subset shipped a model with two real catalog gaps (#236) — and some failures are pre-existing/env, so **baseline against the last release tag** before treating one as a blocker. **Scoping rule:** the full sweep is for catalog-shape changes only — for a scoped change (one compose's flags, a doc, a single script) run just the guards that touch what you changed; most gates are irrelevant to it and the sweep wastes the signal.
#### Where do experimental / unvalidated composes live?
@@ -195,10 +206,9 @@ Workflow:
For **entirely new models** under validation (e.g. "let's try MiniMax-M2.7"): keep the whole `models/<new-model>/` directory untracked until at least one compose validates. Avoid pushing `models/<new-model>/README.md` etc. before there's a working compose to back it up — empty model directories on master signal capability we don't actually have.
References (orphan composes / patches sitting in this state as of 2026-05-09):
- `models/gemma-4-31b/vllm/compose/dual/awq/bf16-mtp.yml` — AWQ-4bit weights variant
- `models/gemma-4-31b/vllm/compose/dual/autoround-int4/dflash-int8.yml` — DFlash + INT8 PTH variant
- `models/qwen3.6-27b/vllm/compose/dual/qwopus-bf16mtp/bf16-mtp.yml` — Qwopus fine-tune preview path
(The 2026-05-09 orphan set has since shipped or been removed — don't expect specific untracked files; the `git status` `??` gap is the live signal.)
**Retired composes → `compose/_archive/`.** A compose that's fully superseded but still referenced by history (CHANGELOG entries, learnings, old discussions) moves to `models/<model>/<engine>/compose/_archive/<topology>/...` instead of being deleted: it keeps old links resolving while staying **out of the registry** (no slug, not launchable, invisible to `switch.sh --list`). This is one step beyond `🗑️ Deprecated` (which keeps the registry entry — see the Status enum): deprecate when users may still reference the slug; archive when nothing but history points at it.
### Documentation
- Don't create new docs proactively. Most non-obvious things belong in `INTERNALS.md`, `FAQ.md`, `SINGLE_CARD.md`, or `DUAL_CARD.md`. New top-level files only when there's a recurring search miss.
@@ -212,9 +222,18 @@ References (orphan composes / patches sitting in this state as of 2026-05-09):
- `bench.sh` — canonical TPS bench (~3-5 min). Run when you change anything that could move TPS (compose flags, Genesis env vars, vLLM pin).
- `quality-test.sh` — behavioral quality (~10-30 min depending on `--quick` / `--medium` / `--full`). Wraps [`benchlocal-cli`](https://github.com/noonghunna/benchlocal-cli) — runs verifier-backed bench packs (ToolCall-15, InstructFollow-15, StructOutput-15, etc) against the running endpoint. Catches what operational tests miss: a compose can pass verify + stress + bench + soak and still ship with degraded tool-call accuracy or instruction-follow drift from quantization or Genesis env-flips. Run before promoting `Status: ✅ Production` and before any pin bump that could shift behavior. **Run via this wrapper, not raw `benchlocal-cli`** — it auto-detects endpoint/model and, for localhost URLs, sets `BENCHLOCAL_HERMES_RESOLVE_LOCALHOST=1` so the sandboxed HermesAgent can reach the host model (direct `benchlocal-cli` skips that → hermes silently scores ~0/20). **Live per-scenario `[N/M]` progress is on by default** (the wrapper forwards `--progress` to benchlocal-cli) so long runs don't go dark; pass `--no-progress` only for CI / log-volume contexts. **Timeout sizing:** the wrapper auto-scales per-scenario timeouts (startup decode-TPS probe × thinking-token multiplier), deliberately over-budgeting — the fix for the thinking-on spurious-timeout class (benchlocal-cli #54/#59). **Don't hand-set `--timeout-per-case` to "fix" a slow run unless you've confirmed the probe measured wrong.** A planned opt-in tier will size from a soak-derived per-depth TPS curve (#114). Full precedence + flags → [`docs/QUALITY_TEST.md`](docs/QUALITY_TEST.md) "Per-scenario timeouts".
- `soak-test.sh` — stability (30-60 min). Run before shipping config / Genesis / memory-policy changes — catches Cliff 2b.
- `rebench-full.sh`**the canonical full-eval orchestrator: use this instead of hand-sequencing the scripts above.** verify-full preflight (fail-fast) + 5 measured steps in the `docs/QUALITY_TEST.md` pipeline order, with the recurring manual-run mistakes guarded (wrong cwd, missing `--save-json`, forgotten `MODEL=` override, missing hermes localhost env, wrong port) and `--resume` idempotency so an interrupt doesn't redo the matrix.
**Gotcha (all serving tests):** `verify-*`/`bench.sh` default `MODEL=qwen3.6-27b-autoround` — against any other served model that's a silent HTTP 404. Pass `MODEL=<served-name>` explicitly (or use `rebench-full.sh`, which autodetects).
The pipeline is layered: each script has a different question it answers ("does it serve / work / survive / fast / behave correctly / stay healthy"). Skipping any layer can mask regressions.
### serve-cockpit (c3)
`tools/serve-cockpit/` is the Textual TUI cockpit — a separate Python app with its **own venv and pytest suite**, NOT covered by `scripts/tests/*.sh`. See its `README.md`. For agents:
- Run tests with `tools/serve-cockpit/.venv/bin/python -m pytest tools/serve-cockpit/tests/ -q`. `test_services.py` + `test_registry_parser.py` are fast — run them on every c3 change; `test_app_headless.py` boots the full app and is slow — prefer targeted `-k` selection while iterating.
- c3 consumes the `registry-emit.sh --json` contract. Adding a field to the emit means threading it through `services.py` (`_variant_row_from_dict`) and, if displayed, `app.py` — and emit changes also need the `test-switch-registry-parity` / `test-launch-registry-parity` guards green.
- DataTable cells render in terminals: avoid U+FE0F variation-selector emoji (`⚠️ 👁️ ⏸️ 🗑️`) in fixed-width columns — Rich reserves 2 cells but many terminals draw 1, misaligning every column after it. Use `Emoji_Presentation=Yes` glyphs (see `_STATUS_GLYPH` in `app.py`).
### Commits
- New commit per logical change. Don't amend published commits.
- Commit messages: subject ≤72 chars, imperative ("Disable P68/P69..." not "Disabled..."), optional body for "why."