diff --git a/AGENTS.md b/AGENTS.md index 11c23609..b7f2ab78 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -54,18 +54,20 @@ If you're considering enabling a new Genesis env var by default in a shipped com 3. Pure bugfixes (no behavioral override) are fine to ship default-on once they pass `verify-full.sh`. ### Engine image pinning -Pin engine images only when we vendor patches into the running container. Otherwise track upstream's rolling tag and accept that the compose YAML may need maintenance when upstream changes flags. +Pin engine images when **either** trigger holds: (1) we vendor patches into the running container, or (2) the engine's rolling tag has burned us with a regression (a crash-loop or behavior break shipped under the same tag we'd validated — llama.cpp earned this pin via #187). Otherwise track upstream's rolling tag and accept that the compose YAML may need maintenance when upstream changes flags. -**Why:** patches hook into specific upstream code paths — a silent upstream change drifts those hooks and breaks the patched container in production. Pinning ensures the bytes we tested against are the bytes users get. Unpatched engines have no such hook, so upstream changes are upstream's problem to fix (or the YAML's, which is cheap to maintain). +**Why:** patches hook into specific upstream code paths — a silent upstream change drifts those hooks and breaks the patched container in production. Pinning ensures the bytes we tested against are the bytes users get. Unpatched engines have no such hook, so upstream changes are upstream's problem to fix (or the YAML's, which is cheap to maintain) — *unless* the tag itself has proven mutable-under-validation, which is trigger (2). + +**Pins are a maintenance liability — bump them on a cadence.** A stale stability-pin silently costs quality, not just features: the b9246→b9967 llama.cpp A/B (2026-07-11, #680) was think-OFF-neutral but **+4 think-ON** with 3 scenario-level flips. `scripts/engine-pin-bump.sh --check` does the mechanical half (spec + compose defaults + a bucketed report of fixtures/claims needing hands); the judgment half (re-validation, live boot) stays with the pin-bump checklist. | Engine | Patches we vendor | Tag policy | |---|---|---| -| `llama.cpp` | none | rolling `ghcr.io/ggml-org/llama.cpp:server-cuda` | +| `llama.cpp` | none | pinned **build tag** (e.g. `server-cuda-b9967`) — trigger (2): stability-pinned after the #187 rolling-tag crash-loop; zero patches, so bumps are cheap (`engine-pin-bump.sh llama-cpp-local `) | | `vLLM` | Marlin pad, KV/loader overlays — `scripts/lib/profiles/patches.yml` is the source of truth | pinned **release tag** (e.g. `v0.22.0`, `v0.24.0`) — **never `nightly-*`/`latest`**: upstream purges nightly tags, orphaning the pin | | `beellama.cpp` | none (the fork *is* the delta) | **digest-pinned** to Anbeeld's official image (`engines/beellama-local.yml` `install.spec`) | | `SGLang` | per-compose decision (pin if we vendor a patch, rolling otherwise) | per-compose | -When adding the first vendored patch to a previously-rolling engine: pin in the same commit. When dropping the last patch: unpin in the same commit. Bump pins via PR with a `verify-full.sh` + `bench.sh` re-run, never silently. +When adding the first vendored patch to a previously-rolling engine: pin in the same commit. When dropping the last patch: unpin in the same commit — unless the engine holds a trigger-(2) stability pin, which outlives its patches. Bump pins via PR with a `verify-full.sh` + `bench.sh` re-run, never silently. **Delivery model (vLLM):** patches reach the container by **volume-mounting into the pinned *stock* `vllm/vllm-openai` image** (python sidecars / site-package overlays / install scripts — see `delivery_mechanism` in `scripts/lib/profiles/patches.yml`), **not** by baking a custom image. The older baked-image path (`ghcr.io/noonghunna/vllm-club3090`, which shipped the release images through `club-v0.8.3`) is **retired** — no compose or engine-pin references it, and the `dockerfile_bake` `delivery:` block in `patches.yml` is legacy/test-only. The GHCR package is kept as historical release artifacts (users pinned to a `club-v0.8.x` tag can still pull); it is not deleted and not produced by anything in-repo. diff --git a/docs/UPSTREAM.md b/docs/UPSTREAM.md index e83cb175..79bbdbf6 100644 --- a/docs/UPSTREAM.md +++ b/docs/UPSTREAM.md @@ -68,9 +68,9 @@ purged (e.g. `VLLM_IMAGE=vllm/vllm-openai:latest`). | `scripts/lib/profiles/engines/vllm-nightly-mtp.yml` → `vllm/vllm-openai:nightly-01d4d1ad...` (PURGED) | 🗑️ **DEPRECATED 2026-06-05 (#254)** — Genesis/TQ3 composes archived to `compose/_archive/`, zero registry users | Genesis MTP nightly; image 404 on Docker Hub + Genesis path on hold upstream. Retained as the `genesis_equipped` test anchor (`required_genesis:true`). | Do not bump; revive only if Genesis re-anchors on a post-#42521 nightly. | | `scripts/lib/profiles/engines/vllm-nightly-dflash.yml` → `vllm/vllm-openai:nightly-e47c98ef...` | DFlash vLLM composes | DFlash overlay baseline. | Bump this YAML after DFlash overlay drift is revalidated. | | `scripts/lib/profiles/engines/vllm-nightly-full.yml` → `vllm/vllm-openai:nightly-e47c98ef...` (PURGED) | 🗑️ **DEPRECATED 2026-06-05 (#254)** — `vllm/dual-int8` archived, zero registry users | Experimental full-overlay nightly; image 404 on Docker Hub. | Do not bump; retained for revival. | -| `scripts/lib/profiles/engines/vllm-stable.yml` → `vllm/vllm-openai:v0.22.0` (overlay-free) | Qwen3-Next vLLM: `vllm/minimal`, `vllm/dual` (27B), `vllm/qwen-35b-a3b-dual` (35B-A3B); generic `dense`; uncurated derived-emission bases | Stock immutable image, **no vendored overlays** — the load-bearing distinction (CONTRACT-5) that lets it be a derived-emission base. Successor to the purged `vllm-nightly-clean` (#254). | Bump `install.spec` only; never the compose literals (drift-guarded by `test-compose-image-drift`). | +| `scripts/lib/profiles/engines/vllm-stable.yml` → `vllm/vllm-openai:v0.24.0` (overlay-free; bumped from v0.22.0 2026-06-30, see engine-pin row above; v0.25.0 staged as hold-for-arm draft #682) | Qwen3-Next vLLM: `vllm/minimal`, `vllm/dual` (27B), `vllm/qwen-35b-a3b-dual` (35B-A3B); generic `dense`; uncurated derived-emission bases | Stock immutable image, **no vendored overlays** — the load-bearing distinction (CONTRACT-5) that lets it be a derived-emission base. Successor to the purged `vllm-nightly-clean` (#254). | Bump `install.spec` only; never the compose literals (drift-guarded by `test-compose-image-drift`). | | `scripts/lib/profiles/engines/vllm-gemma-stable.yml` → `vllm/vllm-openai:v0.22.0` (overlay-carrying; split off `vllm-stable` in #324) | Gemma 4 31B vLLM: `vllm/gemma-bf16-mtp` (**v0.22.0**, #42006 tool-parser overlay only, 131K) + `vllm/gemma-int8-mtp` (**v0.22.0** as of 2026-05-31, #40391 + #42006 overlays, 262K); `vllm/gemma-mtp-tp1` **DEPRECATED 2026-05-31** (fp8 dead on Ampere) | Both gemma duals now on immutable `v0.22.0`, both carry the #42006 streaming-multi-tool fix (lean diff-apply); `gemma-int8-mtp` ALSO carries the #40391 per-head-KV overlay. **#40391 DID need a real rebase onto v0.22.0** — the old v0.21.0-era full-module copies `ImportError`'d on v0.22.0 (`cannot import get_kv_cache_spec_kind` — v0.22.0 added that symbol; the stale copies lacked it). Rebased = v0.22.0-stock + the ~240-line #40391 delta (2 club-3090 resolutions: hand-merged `attn_utils` `is_mla()` branch onto v0.22.0's refactored `_reshape_kv_cache`; kept `kv_cache_utils`'s `replace` import that the PR removed but v0.22.0 still uses). Re-delivered LEAN (boot-time diff-apply `install_script`, not 7 full-module mounts). Live-validated 2026-05-31: boots, `int8_per_token_head` KV inits, KV pool 447K @ 262K, bench 95.7/125.8, coherent generation. The 2 other overlays int8 used to carry (#41800, #41991) are in stock v0.22.0; #42006 is vendored on both duals (see its row). | Drop the #40391 overlay entirely when [PR #40391](https://github.com/vllm-project/vllm/pull/40391) merges + lands in a release. **UPDATE 2026-07-02 (v0.24.0 consolidation):** the 31b DEFAULT moved to overlay-free **bf16** (`vllm/gemma-31b-dual`, stock v0.24.0, ~224K). On v0.24.0 int8-PTH allocates 262K but SILENTLY craters recall past ~32K — #40391 is NOT native there (both cyankiwi + w4a16 crater identically; the SAME cyankiwi weights recall clean on v0.22.0+#40391). The #40391 + #42006 overlays and `vllm-gemma-stable` v0.22.0 **STAY** — still used by the 26b-a4b-single (`gemma-a4b` copy) + the now-**deprecated** 31b composes (`gemma-int8-mtp` / `gemma-bf16-mtp` / `qat-w4a16`). The 262K int8-PTH 31b path returns overlay-free when #40391 merges. | -| `ghcr.io/ggml-org/llama.cpp:server-cuda` | 2 (Qwen 3.6-27B llama-cpp) | Stable tag, no hash drift on upstream side. No patches mounted. | Not a retirement candidate — drift-free. Capture digest if reproducibility matters. | +| `ghcr.io/ggml-org/llama.cpp:server-cuda-b9967` (pinned build tag; was rolling `server-cuda`, then pinned b9246) | 8 llama-cpp composes (Qwen 3.6-27B, Tess-4, Deckard, gemma-12b, pi-reasoning, vibethinker) | **Stability-pinned** (rolling-tag crash-loop, #187) despite zero vendored patches — AGENTS.md pinning trigger (2). Bumped b9246→b9967 2026-07-11 (#680): think-OFF-neutral, **+4 think-ON** (3 scenario flips) — stale build pins silently cost thinking-leg quality. | Not a retirement candidate. Bump on a cadence via `scripts/engine-pin-bump.sh llama-cpp-local ` + per-model spot re-validation; don't let the pin age 50+ days again. | **Retirement workflow:** see [`NIGHTLY_BUMP_RUNBOOK.md`](./NIGHTLY_BUMP_RUNBOOK.md).