AGENTS.md said llama.cpp tracks the rolling server-cuda tag, but it has
been build-pinned since the #187 crash-loop (b9246, now b9967 via #680)
with zero vendored patches. Codify that as pinning trigger (2): a
rolling tag that shipped a regression under a validated name earns a
stability pin, which outlives patches. Add the bump-on-a-cadence rule
(stale pins cost think-ON quality: b9246→b9967 +4) and point at
engine-pin-bump.sh for the mechanical half.
UPSTREAM.md: llama.cpp image row rewritten to the pinned reality
(consumers 2→8, #680 bump recorded); vllm-stable inventory row bumped
v0.22.0→v0.24.0 (stale since 2026-06-30) with the #682 hold-for-arm
staging noted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Piping a multi-hour quality run through tail buffers everything until
exit: no live progress, no partial results, interrupt loses the run.
Redirect to a file and summarize from it. Learned on the 2026-07-11
template A/B; benchlocal-cli#82 tracks the upstream resume half.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
ADDING_MODELS.md: the canonical Step-4 and worked-example registry
entries pinned the deprecated vllm-nightly-mtp engine (purged image,
0 registry users) and the archived TQ3 KV path; the Step-6 calibration
schema taught vllm-nightly-<sha> + genesis_pin: v7.72.2 (both retired);
the worked-example weights block used the old list format the doc's own
schema section forbids. All examples now match current practice
(vllm-stable, release-tag pins, genesis_pin: null, map-format weights).
Added "The lightweight path" — the new-quant/new-slug checklist the doc
previously punted to a design-doc reference (full suite, diagnose-
profile, boot-the-actual-compose are the bolded skip-at-your-peril
items — each shipped a real gap this week).
AGENTS.md: at-a-glance routing line to the new section.
BRING_YOUR_OWN.md: pointer to the c3 Bring & Validate producer funnel
(same flow, guided) + --engine enum fixed to the documented
vllm|llama-cpp|sglang|other set.
Audit basis: full staleness sweep 2026-07-11 (every script/flag/path/
cross-link in both docs verified; BYO otherwise current).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Agent-facing version of the eval guidance shipped user-facing in
ANNOUNCEMENT_TEMPLATE.md §7 (da063d7): behavioral quality (8-pack, both
reasoning modes, benchlocal + sandbox one-time setup, REASONING=on for
the thinking-ON leg) + operational health (report.sh --full), the
non-overlap guarantee, and the don't-pair-rebench-full-with-report-full
rule with the pick-by-goal split.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
ryan's Proxmox VM (no python3-yaml) fails switch.sh with
`ModuleNotFoundError: No module named 'yaml'` — surfaced by #599's
error-unswallowing. The launcher table path only used PyYAML for one
thing: pulling container_name out of each compose. Three-layer fix:
1. PyYAML is now OPTIONAL on the table path: a regex container_name
fallback (first non-comment `container_name:` line + the existing
${VAR:-default} unwrap) kicks in when yaml is absent.
CLUB3090_EMIT_NO_YAML=1 forces the fallback so CI can exercise it.
2. The --json contract path (c3 / baselines join) legitimately requires
PyYAML — it now checks FIRST (before load_profiles, which also
imports yaml) and fails with an actionable Fix: line
(`sudo apt install python3-yaml`), not a bare traceback.
3. Output side of the #599 locale class: a PIPED stdout under LC_ALL=C
defaults to ASCII → UnicodeEncodeError printing the unicode in
status notes. Both emit blocks now pin
sys.stdout/stderr.reconfigure(encoding="utf-8").
New guard test-registry-emit-no-yaml.sh asserts byte-identical table
output across {baseline, no-yaml, C-locale, no-yaml+C-locale (the
literal #584 rig)} and the actionable --json refusal. Live-validated:
`CLUB3090_EMIT_NO_YAML=1 LC_ALL=C PYTHONUTF8=0 PYTHONCOERCECLOCALE=0
bash scripts/switch.sh --list` renders all 48 variants. AGENTS.md
encoding section extended with the stdlib-only + write-side rules.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Audit of the agent guide against current repo state; ten fixes:
- Hardware truths: reframe as "reference rig" — repo supports 10 hardware
classes (profiles/hardware/*.yml); Ampere-only constraints (no native FP8
compute) no longer stated as universal; fp8 composes are 5090-safe via the
launcher's arch-conditional VLLM_USE_DEEP_GEMM pass-through; MTP +
beellama-DFlash noted as the working spec-dec paths.
- beellama default-resolver text: it HAS live DEFAULTS entries and IS the
single-card default for qwen3.6-27b + gemma-4-31b (text said "no entries
yet, blocked on upstream image").
- Engine-pin table: vLLM pins are release tags (never nightly-* — upstream
purges them); beellama row added (digest-pinned); patches.yml named as
patch source of truth over a hand-listed set.
- New convention: encoding="utf-8" on every Python read of repo files —
non-UTF-8-locale community rigs crash on unicode compose headers (#599);
incl. the repro recipe + the no-blind-2>/dev/null corollary.
- Tests: rebench-full.sh documented as the canonical eval orchestrator;
MODEL=<served-name> 404-gotcha called out.
- New subsection: tools/serve-cockpit (c3) — own venv/pytest suite, fast vs
slow test split, registry-emit --json contract, VS16-emoji width rule.
- Full-suite scoping: catalog-shape changes only; targeted guards otherwise.
- Orphan-compose list (all 3 files gone) replaced with the compose/_archive/
convention (archive vs deprecate distinction).
- Genesis section: status line — no shipped compose enables Genesis;
guidance retained for reintroduction.
- Symlink self-description fixed: AGENTS.md is canonical, CLAUDE.md links it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Introduce a new pre-experimental status tier, `incubating`, for composes that
work but aren't ready for the actionable list — niche specialists or models
that fail the standard functional gate by design. Incubating composes are
HIDDEN from `switch.sh --list` by default (revealed by `--list --all`) and
launch-gated behind `--force` (non-functional), so a half-validated model is
catalogued and discoverable without cluttering the recommended set.
Tier wiring (reuses the existing status plumbing — no new emit column):
- compose_registry.py: `incubating` in STATUS_VALUES + `🐣` in COMPOSE_STATUS_EMOJI
(stays out of FUNCTIONAL_STATUSES → --force-gated, never auto-defaulted)
- switch.sh: both --list filters skip incubating unless --all, with a
`(+N incubating hidden — --all)` header note
- AGENTS.md: Status enum row + Caveats-required references
- docs/ADDING_MODELS.md: new rule — NEW MODELS START at 🐣 Incubating, promote
up the enum (🐣 → 🧪 → ⚠️/✅) as they earn the actionable list
First occupant — VibeThinker-3B (WeiboAI, Qwen2 dense reasoning fine-tune):
- `vllm/vibethinker-3b-single` — bf16 weights + fp8_e5m2 KV, single 3090,
full 131072 ctx, mem_util 0.40 (~9.8 GB, single-concurrency sized),
--reasoning-parser qwen3, no tool-calling
- Live-validated 2026-06-16: serves clean correct reasoning/code (~110 TPS),
qwen3 parser splits <think>. fp8 WEIGHTS rejected (break the quant-sensitive
3B: non-terminating empty output). Always-reasoning + no-tools → fails
verify-full's fixed-small-budget checks (5/9) by design → incubating, not gate-passing.
Guard suite 47/47 green (nex-n2-mini parked aside; it's an unrelated local experiment).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ADDING_MODELS.md gets the full rule + AGENTS.md (CLAUDE.md) the short form:
filename = serving-feature delta from a plain boot — <drafter>[-<kv>][-vision].yml,
suffix order drafter→KV→vision. base.yml = engine-default KV + no drafter; don't
name the default KV (no bf16.yml when bf16 is the default — that's base.yml), only
non-default KV (int8/fp8/tq3). Workload-tuned variants (long-text, tools-text,
bounded-thinking, minimal) keep a descriptive name — recognized exception. Never
encode the weights-quant (that's the <quant>/ dir) or topology (the path).
Existing filenames grandfathered (renaming re-paths registry compose_path).
Resolves the base.yml-vs-bf16.yml-vs-int8.yml spread. New composes only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every vLLM compose + engine-pin now defaults to stock vllm/vllm-openai
(nightly-SHA / v0.21.0 / v0.22.0); nothing builds or pulls the baked
vllm-club3090 image. Repoint the test-preflight-compose-deps fixture off the
retired club image to a stock tag (the image is incidental — the test asserts
on missing model weights). Document the vLLM delivery model in AGENTS.md:
patches are volume-mounted into the pinned stock image, not baked; the
vllm-club3090 GHCR package is retired-by-disuse (kept as historical release
artifacts, not deleted). Leaves the legacy dockerfile_bake delivery block +
its patch_attribution handler (test-covered, marked read-only) untouched.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(switch): model-default resolver + user-pinnable defaults
Add a two-layer default scheme on top of the existing <engine>/default map.
<engine>/default stays the maintainer's recommendation (read-only to users);
the new <model>/default is the user's preference — their .env pin if set,
else a curated pick for the detected topology.
compose_registry.py: two maintainer knobs next to DEFAULTS —
RECOMMENDED_DEFAULT_MODELS (short opt-in shortlist, not an exhaustive
ranking; new models are NOT auto-added) and ENGINE_PREFERENCE (per-topology
engine order). Plus resolver helpers (curated_default_target,
community_default_target stub, model_default_pin_key, engine_set/model_set,
slug_topology, model_of_slug).
registry-emit.sh: shared resolver model_default_target(root, model, topology)
— the single injection point for both launchers. Precedence ladder: --variant
(caller) -> .env pin -> community seam (None today) -> curated ENGINE_PREFERENCE
walk (skips non-functional (NA) slugs) -> degradation (notice + nearest-lower
topology, else a clear "pick explicitly" message; never crashes). Plus
x_default_dispatch: X/default with X in engine-set -> engine rec; X in
model-set -> model default; else error (engines + model-ids are disjoint).
switch.sh: <model>/default token; --set-default <slug> / --clear-default
<model> (round-trip the .env pin CLUB3090_DEFAULT_<MODELID>); a Defaults view
appended to --list (also standalone via --defaults) marking user-pin vs
curated. PR-A's grouping/markers/counts and the --force gate preserved.
launch.sh: bare invocation -> first installed shortlist model -> its
<model>/default (no full wizard); a pinned fast-path ("Launch your default
<slug>? [Y/n]"); a post-boot offer ("Make <slug> your default? [y/N]"). Any
narrowing flag keeps the explicit wizard path. Also load CLUB3090_DEFAULT_*
pin keys from .env even when MODEL_DIR is exported in the shell (the existing
.env loader is MODEL_DIR-gated, which would otherwise hide the pin).
Pin validation is warn + fall back, never blocking: unknown slug / wrong
model / topology-mismatch / (NA) status -> notice + curated default.
Tests: new test-model-default-resolver.sh (curated walk, (NA) skip,
degradation, X/default dispatch, pin override + all validation paths,
community seam skipped, .env round-trip); test-default-resolver.sh extended
with <model>/default launch.sh dispatch. Full suite green (only the
pre-existing test-submit-bench.sh fixture failure remains); 45 entries
unchanged; kv-calc calibration 17/17.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: document the model-default resolver + user pins
Ship the user/contributor docs alongside the resolver code (repo convention —
update existing docs, no new top-level doc).
- README: single-card realign (ik-llama = fastest blessed single default,
llama.cpp = cliff-immune alternative — matches ENGINE_PREFERENCE single
order); add "pin your default" + <model>/default to Quick start.
- CLAUDE.md (= AGENTS.md): document RECOMMENDED_DEFAULT_MODELS +
ENGINE_PREFERENCE + the shared resolver as maintainer knobs next to DEFAULTS.
- FAQ: extend "switch to a different model" with <model>/default; new "How do
I set my own default config?" Q (two-layer model, --set-default/--clear-
default/--defaults, .env key, warn+fallback validation).
- SINGLE_CARD / DUAL_CARD: the resolver + the per-topology engine order; pin
hint.
- GETTING_STARTED: first-run uses <model>/default + the in-flow "set as
default?" prompt.
- ADDING_MODELS: a new model resolves via ENGINE_PREFERENCE; add a DEFAULTS row
per engine; RECOMMENDED_DEFAULT_MODELS is not auto-grown.
- UPSTREAM: beellama Docker-image row (gates beellama onboarding; the resolver
skips it today and it auto-promotes to single default on catalog).
- models/qwen3.6-27b/CHANGELOG: dated PR-B entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The four dual/autoround-int4/nvlink-*.yml composes (nvlink-fp8-mtp,
nvlink-turbo, nvlink-dflash, nvlink-dflash-noviz) and their
vllm/dual-nvlink* launch slugs are removed. They were thin Docker Compose
`extends:` stubs whose ONLY override was NVLINK_MODE=force_on — fully
redundant since every base dual compose now auto-detects NVLink at boot
via detect_nvlink.sh (NVLINK_MODE=auto): it probes nvidia-smi and flips on
NCCL_P2P_LEVEL=NVL + custom-all-reduce when a bridge is present, else
NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce. NVLink rigs get the
identical fast path from the base dual compose with no separate slug;
force it explicitly with `NVLINK_MODE=force_on scripts/switch.sh vllm/dual`
if auto-detect ever misses.
Scope (prune only; no engine-image change — the v0.22.0 stable-engine
consolidation is a separate follow-up):
- delete 4 nvlink-*.yml composes
- compose_registry.py: drop the 4 _entry blocks (50 -> 46 entries)
- test-compose-registry-disk.sh: count guards 50/51 -> 46/47
- profile_runtime.yml + patches.yml: drop the 4 nvlink blocks/slug refs
- test-profiles-compat.sh: retire the C13/E3 NVLink-required-compose
scenarios (the estate NVLink-gating code is now dormant; cleanup tracked
as a follow-up, code retained)
- docs: switch.sh help, AGENTS.md, HARDWARE.md, UPSTREAM.md, patches/README.md,
fp8-mtp/dflash/dflash-noviz compose headers, per-model CHANGELOG removal
entry; add a docs/README.md pointer to `switch.sh --list` as the
authoritative registry-derived compose x slug matrix
Historical NVLink bench rows (JusefPol #31, danbedford #74/#92/#96) are
preserved in BENCHMARKS.md. Full test suite green (the lone
test-submit-bench failure is a worktree-isolation artifact — it needs
gitignored results/rebench/ fixtures absent from a fresh worktree; passes
on the working tree).
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
QUALITY_TEST.md gains a 'Per-scenario timeouts' section (precedence ladder,
over-budget rationale, the planned opt-in soak-derived per-depth tier #114) +
points to the benchlocal-cli README for the canonical formula/flags. CLAUDE.md
(agent guide) gains a concise pointer + the 'don't hand-set --timeout-per-case'
convention. Timeout-sizing was previously undocumented in both.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The benchlocal-cli `--progress` flag emits per-scenario `[N/M]` lines
to stderr — critical for the long-running modes (--full ~30-40 min,
--reasoning ~similar, --pack aider-polyglot-30 ~25-30 min) which
otherwise go dark for the whole duration. The wrapper never forwarded
the flag, so users (including agents) ran blind unless they bypassed
the wrapper.
Make --progress on by default, --no-progress as the opt-out for CI /
log-volume-sensitive contexts. Mirror PROGRESS=0/1 env var. Document
the convention in AGENTS.md so agents see it as policy, not just
private/personal memory.
Convention surfaced after wasting 25 min on a blind --full run during
autonomous validation: the wrapper logged its own progress line and
nothing else for the entire 30+ min window.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Symlink direction: repo convention is AGENTS.md canonical, CLAUDE.md a
symlink -> AGENTS.md (opposite of the maintainer system, where AGENTS.md ->
CLAUDE.md). The repo had it backwards (CLAUDE.md real, AGENTS.md -> CLAUDE.md);
flip so AGENTS.md is the real file and CLAUDE.md symlinks to it.
Localhost gotcha: HermesAgent-20 runs its agent INSIDE the Docker sandbox and
calls the model over the network, so a localhost endpoint is the container's
own loopback, not the host. scripts/quality-test.sh already auto-sets
BENCHLOCAL_HERMES_RESOLVE_LOCALHOST=1 for localhost URLs, but invoking
benchlocal-cli directly bypasses that guard and hermes silently scores ~0/20.
- AGENTS.md (now canonical): "run quality via the wrapper, not raw benchlocal-cli."
- docs/QUALITY_TEST.md: new Limitations item + annotate the direct-CLI example
(its localhost example was the exact trap). Failure signature is uniform
~timeout-length latencies + flat GPU, not turn_count (0 for hermes regardless).
Surfaced 2026-05-27: Gemma-4 8-pack hermes 1/20 (artifact) -> 13/20 with the var set.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The maintainer's stack uses agents.md -> CLAUDE.md; the repo now mirrors it.
CLAUDE.md is the canonical agent guide (moved from AGENTS.md — git history
preserved via the rename) and AGENTS.md is a symlink -> CLAUDE.md. Replaces the
earlier thin-pointer CLAUDE.md: one real file, the two names can't drift, and any
agent that looks for either name resolves to the same guide.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
AGENTS.md 'Read first' + the new CLAUDE.md now point at docs/README.md (the
canonical user/contributor docs index) + ARCHITECTURE.md/GETTING_STARTED/
BENCHMARKS/FAQ/KV_MATH, so a user's AI agent landing on either entry point can
navigate the whole doc set instead of just the add-a-model path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ADDING_MODELS.md was stale (pre-refactor) and AGENTS.md lacked the full catalog
flow; neither covered the profile-catalog compatibility class that hotfix #236
exposed. Refresh both for the <quant>/ layout + registry-as-single-source-of-truth,
and add a thin repo CLAUDE.md so users' AI agents discover the workflow.
- docs/ADDING_MODELS.md:
- "Three paths" intro (serve safetensors via pull.sh · run a local GGUF · catalog)
+ a new "Run a local GGUF without the catalog" 3-step recipe (the pull.sh gap).
- Fix the stale weights schema (a MAP keyed by quant-slug, not a list) + add
kvcalc_key + default_port==PORT to the registry example.
- New "Step 4b — Profile-catalog compatibility" (the #236 class: engine
supported_model_families, hardware supported_kv_formats, canonical-scenario fit,
patches.yml chat-template, catalog-size guard, +1 ../ mount depth, registry-
derived launchers).
- Rewrite Step 7 to run the FULL guard suite (table of what each gate guards) +
the baseline-vs-last-tag rule. Update diagram + checklist. De-link the
/opt/ai/CLAUDE.md reference (path leak + 404) -> point at AGENTS.md.
- AGENTS.md: new "Adding a model — full workflow" at-a-glance block linking
docs/ADDING_MODELS.md, with the catalog steps + #236-compat + the catalog guard
tests (the in-repo entry point for AI agents working in a clone).
- CLAUDE.md (new): thin pointer to AGENTS.md so Claude-Code agents pick up the same
guidance without the two large files drifting.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codifies the policy we already apply: rolling tag for unpatched engines
(llama.cpp :server-cuda), pinned nightly digest for engines where we vendor
overlays (vLLM Genesis sidecars / Marlin pad / INT8 PTH / DFlash). New
table per engine + rule for adding/dropping patches in the same commit
as the pin/unpin.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lands the Gemma 4 31B work that was in flight last week + Qwopus3.6-27B
preview compose. Captures four distinct serving paths for Gemma 4:
- dual/awq.yml — AWQ-4bit (text-only, simplest path)
- dual/dflash-int8.yml — DFlash + INT8 PTH KV (262K ctx, full pipeline)
- vllm-gemma4-dflash-int8/ — vendored patches stacking DFlash spec-decode
+ INT8 PTH KV across model_executor + v1/spec_decode + v1/attention +
v1/worker (~13 patched files; PR #42102 + #40391-rebased + tool-parser
fixes#42006 + #41991 stacked)
- vllm-gemma4-fp8-ampere/ — earlier Phase 2 attempt before INT8 PTH
reframe (kept for forensics); Ampere has no native FP8 tensor cores
- vllm-perheadkv-hybridpage-fix/ — hybrid-page bug fix surfaced during
Phase 3
- vllm-pr40391-perheadkv/ — PR #40391 vendored at the tree level
(separate from the rebased variant under refs/jianc99-dflash-gemma4)
Plus:
- models/qwen3.6-27b/vllm/compose/dual/qwopus-bf16mtp.yml — preview compose
for Carnice AutoRound Recipe D output (port 8071, NOT production —
see club-3090-todo.md for known gaps + cheap A/Bs).
- AGENTS.md — codify compose naming + profile-schema + experimental-compose
conventions that the new files follow.
- docs/QUALITY_TEST.md — runbook for `quality-test.sh` + the benchlocal-cli
packs it wraps.
CHANGELOG.md narrative entries for these are added separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous "Status optional, only when not production" convention
left readers guessing whether absence-of-status meant "validated
production" or "author forgot to fill it in." Making Status required +
enumerated removes that ambiguity.
Schema enhancement (codified in AGENTS.md):
- Status: now REQUIRED, exactly one of:
✅ Production — verify-full + stress + bench + soak PASS
⚠️ Production w/ caveats — works under documented constraints
🧪 Experimental — under active validation
👁️ Preview — known quality issues, not production
⏸️ Upstream-gated — blocked by external action (PR/driver)
🗑️ Deprecated — kept for historical reference
- Caveats: REQUIRED if Status is ⚠️/👁️/⏸️/🗑️, OMITTED for ✅/🧪.
Single-line summary or short bullet list with issue/PR links.
Status applied to all 27 composes (vLLM + llama-cpp, all topologies):
✅ Production: 20 (canonical Qwen + Gemma + llama.cpp + gemma-awq)
⚠️ Production w/ caveats: 4 (long-text*, long-vision, carnice-bf16mtp)
⏸️ Upstream-gated: 2 (gemma single boot-OOM, gemma dflash-int8 needs #42102)
👁️ Preview: 1 (qwopus-bf16mtp)
🧪 Experimental: 0
🗑️ Deprecated: 0
llama-cpp composes also gained `Genesis: N/A — llama.cpp engine` for
parity with the vLLM schema (Genesis is Qwen3-Next-specific and
vLLM-only; explicit N/A prevents future readers from looking for
patches that don't exist).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three rationale points for why models/<m>/<engine>/patches/ and cache/
sit parallel to compose/ rather than nested under each topology:
1. Patches are reused across topologies (e.g. vllm-marlin-pad/ is
mounted by dual/, multi4/, and every dual/nvlink-*.yml). Topology
subdir would force symlinks or duplication.
2. Patches are scoped by (model, engine), not topology — a vLLM source
override doesn't change based on TP value; it's engine-internal.
3. Caches (torch_compile/, triton/) warm-start across composes — sharing
at engine level means switching from single/default to single/long-text
reuses JIT'd kernels.
Documents the relative-path convention (../../patches/, ../../cache/
from compose/<topology>/<file>.yml) and the rule for genuinely
topology-specific patches if any ever land: keep at engine level,
document the constraint in the patch's README.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Compose files now live under `<model>/<engine>/compose/<topology>/<file>.yml`,
with topology as a folder rather than a filename prefix. Solves all 7
inconsistencies surfaced in the post-rename audit (single-card composes
without `single-` prefix, unsuffixed `docker-compose.yml` ambiguity,
fine-tunes encoding model name in filename, etc.) by making the directory
hierarchy enforce the convention.
Layout:
models/<model>/<engine>/compose/<topology>/<feature>.yml
Where:
- <model>: qwen3.6-27b, gemma-4-31b
- <engine>: vllm, llama-cpp, sglang
- <topology>: single, dual, multi3, multi4, multi8
- <feature>: docker-compose.yml (default) | turbo.yml | dflash.yml | etc.
Each topology subdir has a `docker-compose.yml` for the recommended
starter — bare `cd <topology> && docker compose up` works because
docker compose finds that filename automatically. Variants drop the
`docker-compose.` prefix since they're invoked via `-f` flag.
27 compose file moves total:
- 18 Qwen vLLM composes redistributed across single/dual/multi4
- 2 Qwen llama-cpp composes into single/
- 6 Gemma vLLM composes redistributed across single/dual
- 1 untracked qwopus-bf16mtp moved to dual/
Inside each compose: relative paths to `../patches/` and `../cache/`
bumped to `../../patches/` / `../../cache/`, and `../../../../models-cache`
to `../../../../../models-cache` (one extra `..` for the new depth).
Reference updates across 148 files (BENCHMARKS, all docs, CHANGELOGs,
sibling-table cross-references in compose headers, scripts, patch
READMEs, .github issue templates, tools/residency-instrument).
scripts/switch.sh VARIANTS map updated; tags themselves unchanged
(`vllm/dual` → `dual/docker-compose.yml`, `vllm/dual4` → `multi4/docker-compose.yml`,
`vllm/gemma-mtp` → `gemma-4-31b/.../dual/docker-compose.yml`, etc.).
AGENTS.md "Compose layout" section rewritten to describe the new
hierarchy, with concrete examples and the fine-tune exception
(`dual/carnice-bf16mtp.yml` carries the fine-tune name as a filename
prefix until the fine-tune graduates to its own model directory).
All switch.sh paths verified to resolve to actual files post-move.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Documentation distinguishes 1 / 2 / 3+ GPUs as `single` / `dual` /
`multi`, with separate doc pages (SINGLE_CARD.md, DUAL_CARD.md,
MULTI_CARD.md). Compose filename topology prefix should match.
`dual4.yml` was an awkward outlier — it had `dual` in the prefix
but actually meant TP=4 (4-card config). The clean shape: GPU
count is implicit when there's no ambiguity (`single` always 1,
`dual` always 2), and explicit when there is (`multi3` / `multi4`
/ `multi8`).
Renames:
- docker-compose.dual4.yml → docker-compose.multi4.yml
- docker-compose.dual4-dflash.yml → docker-compose.multi4-dflash.yml
Registry tags `vllm/dual4` and `vllm/dual4-dflash` in scripts/switch.sh
keep their existing names (backward compat for users running
`bash scripts/switch.sh vllm/dual4`); only the file paths in the
VARIANTS map are updated.
References updated: BENCHMARKS.md, docs/MULTI_CARD.md, docs/UPSTREAM.md,
models/qwen3.6-27b/CHANGELOG.md, models/qwen3.6-27b/vllm/patches/README.md,
sibling-table cross-references in 7 other compose headers.
AGENTS.md "Topology prefix" row tightened: examples now read
`single · dual · multi3 · multi4 · multi8` (dropped `dual-nvlink`
which is actually an interconnect feature suffix; dropped `quad`
which doesn't exist as a name in our convention).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three subsections under "Compose variants" documenting the convention
established by the 2026-05-09 Gemma 4 alignment pass:
1. Compose filename convention — <topology>-<feature>.yml, model
implied by parent directory. Examples: dual.yml, dual-turbo.yml,
dual-int8.yml, single.yml. Filename collisions across model
directories are fine (path disambiguates). Registry tags in
scripts/switch.sh decouple from filenames; rename file paths
while keeping tags backward-compatible.
2. Profile schema header — every compose declares (Model, Topology,
Drafter, KV, Vision, Max-ctx, Genesis, Best-for) in a structured
block at the top, before any free-form description. Schema
forces explicit declaration; catches drift between header
description and actual config.
3. Where experimental / unvalidated composes live — same directory
as shipped composes, untracked until verify-full + verify-stress
+ bench + soak validation passes. Mark with `Status: ⚠️
EXPERIMENTAL` or `⚠️ PREVIEW` so readers know. Don't create a
separate experimental/ subdirectory — relative paths to
../patches and ../cache would need re-pathing on promotion.
Future Claude sessions and any human contributor cloning this repo
now get this convention loaded by default via AGENTS.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
We had upstream issue / PR links scattered across CHANGELOG, INTERNALS,
FAQ, per-compose comments — drifting independently. Centralizing in
one place with a status convention so the tracker stays current.
- docs/UPSTREAM.md — single source of truth, categorized by upstream
(vLLM / Genesis / fla-org / FlashQLA / llama.cpp / transformers /
SGLang), with status emoji + what unblocks for us + workaround.
- AGENTS.md (repo root) — AI-coding-agent guidance. The rule:
before filing or referencing an upstream issue, check + update
docs/UPSTREAM.md. Also captures today's Genesis-opt-in vetting
lesson (behavioral mitigations need streaming + large-prompt repro
before shipping default-on).
- README, CONTRIBUTING, INTERNALS "See also" sections cross-link the
two new files.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
