The shared served-model-name `qwen3.6-27b-autoround` (from #490) mislabels the
non-autoround 27b scenes (fp8 dual-max, lmcache): /v1/models advertises
"autoround" while `root` points at qwen3.6-27b-fp8. Same class on gemma-4-31b —
after the v0.24.0 consolidation the default is cyankiwi qat-AWQ-INT4 (bf16 KV),
yet the LiteLLM route still targeted `gemma-4-31b-autoround` (a latent #482 drift).
Fix WITHOUT breaking anything, via vLLM multi-served-name:
- Every 27b scene now serves `qwen3.6-27b <its-quant-name>`; every gemma-31b
scene serves `gemma-4-31b <its-quant-name>`. The neutral name is PRIMARY
(honest /v1/models id); the quant-specific name is retained as a live ALIAS.
- LiteLLM: add `qwen3.6-27b` / `gemma-4-31b` canonical public routes; keep the
`-autoround` routes as back-compat aliases (same upstream). Repairs the gemma drift.
- Migrate our own MODEL= defaults + docs (bench/verify/quality/launch/setup, c3,
tui-core, EXAMPLES, ...) to the neutral name. Weights slugs (`-autoround-int4`)
untouched; CHANGELOG + results/ history left as-is.
Retiring the `-autoround` alias entirely is a deliberate later step once nothing
still asks for it.
Live-verified on-rig (single/minimal, stock v0.24.0): /v1/models lists BOTH names
(root=...-autoround-int4); chat to `qwen3.6-27b` AND `qwen3.6-27b-autoround` both
return 200; `qwen3.6-27b-fp8` correctly 404s. Full shell gate 59/59 (1 = known
worktree-fixture); c3 pytest 41 passed; served-name arg-order + YAML validated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The CLI Quick-start had 7 steps; c3 had one buried blockquote. Expand it into a proper subsection — install (uv + pip), launch, first-run Settings (S → Model Dir + HF token → Ctrl+S → r), modes (--lean), and the keep-current note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
c3 was install-broken for a fresh checkout and its README described a months-old
skeleton. Fixed so users can actually clone → install → run:
- pyproject: wire the in-repo `club3090-tui-core` (tools/tui-core, not on PyPI) via
`[tool.uv.sources]` so `uv pip install -e tools/serve-cockpit` resolves it from the
checkout. Validated in a clean uv venv: single command → `c3` present, imports OK
(single-step previously failed: "club3090-tui-core was not found in the package registry").
- README (serve-cockpit): rewritten to match the actual app — the two modes (Run & Operate ·
Bring & Validate), the real install (uv one-liner + plain-pip two-step), live keybindings,
and the working/guarded status — replacing the "Phase 1 walking skeleton / 🧪 Experimental"
text that no longer described anything.
- Top-level README: a pointer to the cockpit so a cloner discovers it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
35B-A3B row (#390 follow-up): drop NEW v0.7.3 tag (we're at v0.8.7; matches
the other established production rows that carry no tag), add the byteshape-iq4xs
single-card path (113/129 TPS @262K, 110/150 8-pack, PR #293) alongside apex-fit.
Also drop the same stale NEW v0.7.3 tag from the Gemma 4 26B-A4B row.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
vLLM dual was promoted preview→✅ Production on 2026-05-30 (vllm/qwen-35b-a3b-dual,
v0.22.0 stable, 262K + vision, 178/174 TPS) per BENCHMARKS.md. README still showed
'Preview (vLLM dual)' / 'vLLM ✅ (preview)' / '182/177 at 16K (no MTP/TQ3/Genesis)'.
Corrected Status + Engines + Highlights to the production numbers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
deucebucket (#390) ran qwen3.6-35b-a3b on mainline llama.cpp single-3090
(141/144 TPS, 15.1GB @131K). The README support table marked llama.cpp ❌
for this MoE, conflating 'no shipped catalog compose' with 'unsupported' —
contradicted by our own docs/HARDWARE.md mainline power-cap curves on this
exact model and the llama-cpp-mainline engine profile (qwen3-next-moe).
Note added that ik_llama is the shipped single-card path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Groups all Club 3090 AI Studio docs under docs/ai-studio/ and fills the
gaps (no front-door overview; audio undocumented; image doc stale).
- NEW docs/ai-studio/README.md — the high-level overview: the director-driven
flow, the full 8-lane matrix, lanes-vs-modes, the shared substrate
(director/gallery/orchestrator/shim/tts/step-voice/gpu-mode), the VRAM
truth, productization framing.
- NEW docs/ai-studio/audio.md — voices (Kokoro narration), premium voice
(Step-Audio-EditX clone+edit), music (ACE-Step), SFX (Stable Audio). Notes
the future Understand pillar (ASR/diarize/separate) kept in private design.
- docs/ai-studio/image.md (was IMAGE_STUDIO.md) — REFRESHED to the current
lanes: HiDream-O1 + Ideogram-4 + Chroma + the native-button shim (the old
doc still described the retired gemma-12b image-studio mode).
- docs/ai-studio/video.md (was VIDEO_STUDIO.md) — TRIMMED to video-only;
image/audio sections moved to their own docs; cross-links added.
- Rewrote every reference across the repo (~13 files: root README, FAQ, docs
index, gpu-mode.sh, the pipe builder + studio service composes/py) to the
new paths. docs/README.md index gains the overview + audio entries.
LOCAL_AI_PRIMER.md stays at docs/ (general). The realtime voice-agent design
stays private until built.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
- docs/IMAGE_STUDIO.md (new): the bundle — quickstart, two-front-ends, gpu-mode
modes, measured Ideogram-4 VRAM-by-resolution, chat-model options, the
PersistentConfig "image button missing" fix, single-GPU fallback.
- docs/README.md: IMAGE_STUDIO.md under the user track.
- README.md: one-line image-gen pointer.
- docs/FAQ.md: point the existing image-gen answer at the turnkey bundle.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gemma single-card had NO functional default (vLLM single is upstream-gated at
head_dim=512; ik-llama walls ~24K; stock llama.cpp ~12 TPS). beellama is the
only viable fast single-card Gemma-4 path, so promote it to fill the gap.
Registry: beellama/gemma-dflash experimental -> caveats + new
DEFAULTS[(gemma-4-31b, beellama, single)]. Resolver now returns it for Gemma
single (beellama is #1 in ENGINE_PREFERENCE[single]); Gemma dual unchanged.
Validated #441: 131-200K ctx, 47/88 TPS, 8-pack 109/150 think-off /
114/150 think-on (reasoning net-positive on Gemma). Reuses the multi-arch
image from #271 (sm_86/89/120; sm_89/120 compiled-not-validated).
Docs: compose header (caveats + Quality + ctx ceiling), README Gemma row,
BENCHMARKS row marker + intro, UPSTREAM.md row. Resolver guard test flipped
from a degradation assertion to beellama/gemma-dflash; full suite 38/38.
Re-test trigger: re-point the default to the no-fork mainline path when
llama.cpp#23398 (Gemma-4 MTP) merges (docs/UPSTREAM.md).
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make `beellama/dflash` the single-GPU default for Qwen3.6-27B, served via
an unofficial multi-arch image, and demote ik-llama to the balanced alt.
Decision basis (single-3090, 370 W; bench.sh n=5 + quality-test --full):
- Code TPS ~100 vs ik 69 — fastest single-card 3090 code path.
- 8-pack 107/150 (71%) think-off vs ik 99/150 (66%); 113/150 think-on
(same-session). DFlash is output-lossless, so quality == the Q5_K_S target.
- Context ceiling ladder (measured): 130K comfortable, 160K usable
(115K-tok prefill OK, 0.8 GB headroom), 200K boots but OOMs on prefill.
Ships 102K default; raise to 160K via CTX_SIZE.
- ik still wins narrative TPS (63), VRAM, and shipped vision — kept as the
balanced alt (`--variant ik-llama/iq4ks-mtp`).
Registry: `beellama/dflash` experimental -> caveats + a new
DEFAULTS[(qwen3.6-27b, beellama, single)] row. beellama is already #1 in
ENGINE_PREFERENCE[single], so the resolver now returns it for Qwen single.
Image: unofficial multi-arch ghcr build
`beellama-cpp:multiarch-b9459-07ac3ce` (sm_86/89/120 = RTX 3090/4090/5090),
built from Anbeeld/beellama.cpp .devops/cuda.Dockerfile with
CUDA_DOCKER_ARCH="86;89;120" + GGML_CUDA_FA_ALL_QUANTS=ON. Engine reports
`ARCHS = 860,890,1200`; cuobjdump confirms all 3 SASS. CAVEAT: sm_89/sm_120
are compiled but UNVALIDATED — only sm_86/3090 is verified on-rig.
Also bumps the gemma beellama compose to the same multi-arch image (stays
experimental — not promoted). Docs: README single-card framing,
INFERENCE_ENGINES.md, UPSTREAM.md row, BENCHMARKS.md Qwen beellama row.
Resolver guard tests flipped to the new default; full suite 38/38.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`switch.sh --list` showed every slug regardless of topology, so a
single-GPU box listed dual/multi configs it can't actually run — noise.
Filter `--list` to the topologies the detected GPU count supports:
1 GPU -> single only, 2 -> single+dual, 4+ -> everything. Detection
reuses switch_topology_from_gpus (CUDA/NVIDIA_VISIBLE_DEVICES, then
nvidia-smi) mapped to a topology rank.
- `--all` (and the `--list-all` alias) bypass the filter for
discoverability; --list is deferred until args are parsed so
`--list --all` works in either order.
- Don't silently hide: print a one-line note naming the detected GPU
count and exactly which topologies were hidden, plus a `(+N … hidden
— --all)` tally in the header. No note under --all / when nothing is
hidden.
- Fail-open: with no detection signal (no selector, no nvidia-smi) show
ALL — never hide off a guessed "single" fallback.
- Counts (header + per-model) reflect the visible set; PR-A health
markers/grouping and PR-B Defaults view unchanged.
New test scripts/tests/test-list-topology-filter.sh covers 1/2-GPU
filtering, --all, --list-all, order independence, the --all-without-list
guard, and fail-open. Docs: --help usage, FAQ model-switch entry, README.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(switch): model-default resolver + user-pinnable defaults
Add a two-layer default scheme on top of the existing <engine>/default map.
<engine>/default stays the maintainer's recommendation (read-only to users);
the new <model>/default is the user's preference — their .env pin if set,
else a curated pick for the detected topology.
compose_registry.py: two maintainer knobs next to DEFAULTS —
RECOMMENDED_DEFAULT_MODELS (short opt-in shortlist, not an exhaustive
ranking; new models are NOT auto-added) and ENGINE_PREFERENCE (per-topology
engine order). Plus resolver helpers (curated_default_target,
community_default_target stub, model_default_pin_key, engine_set/model_set,
slug_topology, model_of_slug).
registry-emit.sh: shared resolver model_default_target(root, model, topology)
— the single injection point for both launchers. Precedence ladder: --variant
(caller) -> .env pin -> community seam (None today) -> curated ENGINE_PREFERENCE
walk (skips non-functional (NA) slugs) -> degradation (notice + nearest-lower
topology, else a clear "pick explicitly" message; never crashes). Plus
x_default_dispatch: X/default with X in engine-set -> engine rec; X in
model-set -> model default; else error (engines + model-ids are disjoint).
switch.sh: <model>/default token; --set-default <slug> / --clear-default
<model> (round-trip the .env pin CLUB3090_DEFAULT_<MODELID>); a Defaults view
appended to --list (also standalone via --defaults) marking user-pin vs
curated. PR-A's grouping/markers/counts and the --force gate preserved.
launch.sh: bare invocation -> first installed shortlist model -> its
<model>/default (no full wizard); a pinned fast-path ("Launch your default
<slug>? [Y/n]"); a post-boot offer ("Make <slug> your default? [y/N]"). Any
narrowing flag keeps the explicit wizard path. Also load CLUB3090_DEFAULT_*
pin keys from .env even when MODEL_DIR is exported in the shell (the existing
.env loader is MODEL_DIR-gated, which would otherwise hide the pin).
Pin validation is warn + fall back, never blocking: unknown slug / wrong
model / topology-mismatch / (NA) status -> notice + curated default.
Tests: new test-model-default-resolver.sh (curated walk, (NA) skip,
degradation, X/default dispatch, pin override + all validation paths,
community seam skipped, .env round-trip); test-default-resolver.sh extended
with <model>/default launch.sh dispatch. Full suite green (only the
pre-existing test-submit-bench.sh fixture failure remains); 45 entries
unchanged; kv-calc calibration 17/17.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: document the model-default resolver + user pins
Ship the user/contributor docs alongside the resolver code (repo convention —
update existing docs, no new top-level doc).
- README: single-card realign (ik-llama = fastest blessed single default,
llama.cpp = cliff-immune alternative — matches ENGINE_PREFERENCE single
order); add "pin your default" + <model>/default to Quick start.
- CLAUDE.md (= AGENTS.md): document RECOMMENDED_DEFAULT_MODELS +
ENGINE_PREFERENCE + the shared resolver as maintainer knobs next to DEFAULTS.
- FAQ: extend "switch to a different model" with <model>/default; new "How do
I set my own default config?" Q (two-layer model, --set-default/--clear-
default/--defaults, .env key, warn+fallback validation).
- SINGLE_CARD / DUAL_CARD: the resolver + the per-topology engine order; pin
hint.
- GETTING_STARTED: first-run uses <model>/default + the in-flow "set as
default?" prompt.
- ADDING_MODELS: a new model resolves via ENGINE_PREFERENCE; add a DEFAULTS row
per engine; RECOMMENDED_DEFAULT_MODELS is not auto-grown.
- UPSTREAM: beellama Docker-image row (gates beellama onboarding; the resolver
skips it today and it auto-promotes to single default on catalog).
- models/qwen3.6-27b/CHANGELOG: dated PR-B entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gemma 4 31B: was "2× 3090 only" with footnote about single-card OOMs.
We now ship beellama.cpp single-card path validated 2026-05-27 in
discussion #239 — 47/88 TPS, 100–150K ctx, 109/114 8-pack. Row +
footnote refreshed.
Qwen3.6 35B-A3B: was "preview / 2× 3090" only. PR #243 landed the
ik_llama --fit q8/q5 single-card variant — 103/149 TPS at 196K with
hermes 11/20, aider 12/30, cli 12/40 measured post-fix verifiers.
Status upgraded, row + card-count + engines refreshed.
Gemma 4 26B-A4B: left unchanged per a single-card test not yet run on
Ampere; added a small footnote (²) to disclose that explicitly rather
than imply "dual only because Ampere can't fit."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Currently the README's tagline + welcome paragraph both say "RTX 3090s"
explicitly — a 4090 / 5090 visitor scanning the top of the page has no
signal that the composes run on their card. They bounce thinking
"3090-only" before reaching the Hardware FAQ.
Adds a single callout (mirroring the existing 🪟 WSL2 pattern from
Quick start) right under the welcome paragraph: links straight to the
two FAQ entries with measured cross-rig numbers + per-class gotchas.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Repo-wide follow-up to the compose quant-layer move (9821c94). The
mechanical move + registry/launch/test rewire covered the launchable
surface; a full-tree sweep found compose-path refs the move invalidated
in docs, two functional scripts, and one half-migrated mapping.
Functional fixes:
- bench-row-formatter.sh infer_compose_path(): 6 dual entries
(int8-tq3, tq3-mtp-genesis, tq3-nomtp, tq3-mtp, int8, bf16) were left
at bare dual/<file>.yml while the rest were migrated -> would emit
dead compose paths into BENCHMARKS rows.
- residency-instrument/run-instrumented-soak.sh: case->COMPOSE_FILE
paths (long-text, long-text-no-mtp, tools-text, dual default) now
resolve under <quant>/.
Docs: README layout line + tree, engine/model READMEs, patch-README
quick-recipes, diagnostics, FAQ/CLIFFS/KV_MATH/DTYPE/MULTI/SINGLE/
STRUCTURED_COT/TQ3/UPSTREAM, issue template, sglang cross-refs
(-> vllm prod path). Per-model targets: qwen-vllm->autoround-int4,
llama-cpp->unsloth-q4km, gemma defaults (bf16-mtp/fp8-mtp),
carnice->own slug dir.
Intentionally left as historical/append-only records: CHANGELOG x2,
BENCHMARKS row-labels (live paths already correct in row bodies),
calibration source: provenance citations, switch.sh/parity history
comments. Separate follow-ups: gpu-mode.sh (#417 deprecated-repo
repoint), bench-row-formatter compose_display() docker-compose.yml
branch (PR-B). Flagged pre-existing-stale: dual/int8-tq3.yml in
pr40798/pr40914 READMEs (predate this refactor; ambiguous target).
Guard tests (registry-disk, mounts-resolve, switch-parity,
launch-compat) all PASS post-edit. Leak-clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Diagnostics section:
- New "If launch.sh / switch.sh won't boot" subsection: --force escalation
(skip preflight only) then full bypass — `MODEL_DIR=… docker compose -f
<compose> up -d` with the three common compose paths (llama.cpp single,
ik_llama single, vLLM dual), the verify curl, and `down`. Covers the
Windows CRLF/.env, missing-PyYAML, and false-preflight failure modes.
- Capture-to-file example now uses --full (the complete cross-rig pass);
note says drop --full for the ~2 s hardware-only capture.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The only WSL pointers were ~185-235 lines below Quick start (docs tree +
"What you'll need") — a Windows user hits the Linux/Docker/bash commands
first and fails before discovering WSL_SETUP.md. Add a one-line callout at
the top of Quick start (before the command block), the standard place for
OS prerequisites.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- README: drop the redundant standalone quality-test block left by #215 and
fold `quality-test.sh --reasoning` into the existing Benchmarks section.
- WSL_SETUP.md: add a "Native llama.cpp in WSL (no Docker)" section (Step 2
passthrough is all native needs; lift llama-server flags from the compose),
and a "Shrink the overhead" block in Step 8 — headless AI GPU, close GPU
apps, current driver, WSLg off, dual-boot — distinguishing the unavoidable
WSL2 paravirt context from the reclaimable display/WDDM portion.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New docs/WSL_SETUP.md walks a fresh Windows box to a running club-3090:
WSL2 + Ubuntu install, Windows NVIDIA driver / GPU passthrough, .wslconfig
RAM sizing (the silent host-RAM-starvation OOM, #32), Docker + container
toolkit, and the two gotchas not documented anywhere yet — cloning onto
ext4 (not /mnt/c) and keeping .env/scripts LF (not CRLF). Weights via
WEIGHTS=gguf + MODEL_DIR, then the normal launch.
Cross-links the existing runtime tuning (VRAM budget, TDR, expandable_segments)
in FAQ.md + HARDWARE.md rather than duplicating it; adds discovery links from
README ("What you'll need" + docs tree) and the FAQ Windows/WSL2 answer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two new sections in the operational/reference half so users find the
benchmark + triage-report commands without spelunking scripts/:
- Benchmarks: placed right after "Measured TPS at a glance" (reproduce
those numbers) — bench.sh (TPS), quality-test.sh (modes + --pack),
rebench-full.sh (canonical 5-step pipeline + endpoint-first mode).
- Diagnostics: report.sh (default + --verify/--stress/--soak/--bench/
--full/--no-redact) with the redaction + Cliff-2b/soak rationale.
Reasoning-suite (quality-test.sh --reasoning) intentionally omitted —
not on master yet (lands with the benchlocal reasoning packs).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SGLang is blocked on Ampere (watch-list, not a working engine), so listing it
in the README's multi-engine tagline / engine-picker / supported-models table
overstated it as a supported route. Removed from:
- tagline ("Multi-engine (vLLM, llama.cpp, ik_llama)")
- engine-picker inline list
- all 4 supported-models "Engines" cells (kept llama.cpp ❌ — it IS a supported
engine with model-specific status; SGLang is not a supported engine at all)
Kept a one-line "SGLang was evaluated — currently blocked on Ampere, see
docs/engines/SGLANG.md" pointer in the multi-engine bullet + the docs-tree
annotation, so the evaluation context isn't erased (full detail stays in
docs/engines/SGLANG.md + INFERENCE_ENGINES.md).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The quick-start listed `vllm/default` as "single-card chat (recommended)", but
that path is on the purged Genesis-MTP pin (#167) — it won't boot — and is
Cliff-2-prone for agentic/accumulated-context use, so newcomers were handed a
dead command. Per the #209 decision (llamacpp/default = simplest default,
ik-llama = fastest):
- "(recommended)" → llamacpp/default (cliff-immune, 200K, ~51/60 TPS)
- ik-llama/iq4ks-mtp kept as the FASTEST single-card pick
- dropped the #167-blocked vllm/default line (still documented in SINGLE_CARD.md
with strikethrough for when the pin lands) + the redundant bare llamacpp/mtp
(same compose as llamacpp/default)
- vllm/dual stays (clean nightly, boots) with a note that vLLM single-card is #167-blocked
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same boots≠fills correction as the merged ctx sweep — this 'Supported models'
highlights cell was missed (sweep excluded it). Single-card llama.cpp is 200K
max-safe, not 262K (262K is dual vLLM turbo). Also tidied ik TPS ~62→~60/69 to
match BENCHMARKS + tagged it 'fastest single-card'. Dual-card 262K refs (L31/67/
103) and the LLAMA_CPP.md '262K recipe' pointer (L139, pending fill-test) left.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ik-llama/iq4ks-mtp is the fastest single-card path (~18-20% faster decode +
leanest VRAM, #184) but was absent from the README quick-start and the registry
ctx values had drifted from the 200K shipped default. Keeping llamacpp/default =
mainline (the simplest / clean-upstream-image pick) — surfacing ik, not renaming.
- README quick-start: add `ik-llama/iq4ks-mtp` (fastest single-card) alongside
the llamacpp/* variants.
- docs/SINGLE_CARD.md: one-line "simplest (llamacpp/default) vs fastest
(ik-llama/iq4ks-mtp)" steer atop the config table (ik rows were already
present + ⭐-marked).
- compose_registry.py: fix stale wizard-projection max_ctx to the 200K default —
llamacpp/default + llamacpp/mtp were 131072 (too low), ik-llama/iq4ks-mtp was
262144 (boots-not-fills); all → 200000. Vision entries (49152 / 163840) already
correct. Reworded the "262K via -ub 512" comment (that was the boots≠fills
false ceiling). NOTE: launch.sh doesn't pass max_ctx to runtime, so this is
wizard-projection accuracy only — runtime ctx still comes from the compose
CTX_SIZE=200000 default.
Validated: registry imports; test-launch-compat, test-switch-registry-parity
(45 composes, parity), test-profiles-compat all pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The single-card MTP defaults were bumped 262144 -> 200000 (PR #201 + the ik
commit) after the "boots != fills" finding: 262K boots and passes verify-stress
but only fills ~125K before the FA-scratch-at-fill OOM; 200K fills ~183K with
~1.1 GB margin (see docs/CLIFFS.md). The docs lagged.
Corrected to 200K (the shipped, fillable default) across the Q4_K_M-MTP +
ik IQ4_KS single-card paths:
- README.md (quick-start + TL;DR robustness pitch)
- docs/SINGLE_CARD.md (cliff-warning row, config table, llamacpp/default + ik
narratives, fallback table, Cliff 2 explainer)
- docs/engines/IK_LLAMA.md (defaults + max-context override)
- models/qwen3.6-27b/llama-cpp/README.md (single/mtp.yml heading + workhorse line)
- scripts/switch.sh (--list/--help comment)
Also reconciled an older drift: llamacpp/default = llamacpp/mtp alias (collapsed
2026-05-22), Q4_K_M MTP, no vision, ~52/61 TPS -- several spots still described
it as the retired Q3_K_XL @ ~21 TPS + vision.
Left for a separate fill-test (smaller-quant / no-MTP headroom may genuinely
exceed 200K): the "going-to-262K" max-ctx sections in docs/engines/LLAMA_CPP.md
+ VLLM.md, and the Q3_K_XL legacy rows in models/qwen3.6-27b/llama-cpp/README.md.
Dual-card vLLM 262K, the model's 262144 max_position_embeddings, and the CLIFFS
"boots!=fills" explainer are correct as-is and untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
We don't distribute a club-3090 vLLM image to users, so the
build-vllm-image.yml workflow + its surrounding docs are dead weight. The
workflow had also been failing on every tag push and the weekly cron because
its base nightly (nightly-1acd67a7) was purged from Docker Hub (#167/#407) —
that red CI on the v0.8.4 tag is what surfaced this.
Removed:
- .github/workflows/build-vllm-image.yml (the GHCR image builder)
- docker/vllm-club3090/Dockerfile (its build recipe)
- docs/CI_RUNNER_SETUP.md (build/distribution doc)
- README + UPSTREAM references to ghcr.io/noonghunna/vllm-club3090
- docs/README index link to the deleted CI doc
Kept: the generic VLLM_IMAGE override (now documented against
vllm/vllm-openai:latest — also the #167 workaround). The launch-compat test
still exercises that override, just with an upstream image.
Recoverable from history if we ever want to ship an image again.
NOT touched: patch_attribution.py still references the dockerfile_bake
delivery mode — left for a separate decision (internal, harmless).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(llamacpp): collapse single-card composes 3→2 (default = mtp alias)
Mainline Q4_K_M reaches the full 262K with -ub 512 (validated: boot + serve +
verify-stress 7/7 + soak PASS), so the old vanilla Q3_K_XL docker-compose.yml no
longer earns a separate profile. Collapse the llama.cpp single-card tree to two
compose files:
- `llamacpp/default` is now an ALIAS for `llamacpp/mtp` (mtp.yml) — launch.sh +
compose_registry repointed. All references (estate-CLI default, tests, docs)
keep resolving; the variant name survives.
- Retired models/qwen3.6-27b/llama-cpp/compose/single/docker-compose.yml.
- mtp.yml: document the -ub 512 -> 262K recipe in the header.
- mtp-vision.yml: align with the ik vision profile — default 160K (was 49K) via
-ub 512, and pin --image-min-tokens 1024 / --image-max-tokens 4096 (mainline
previously left image bounds at the model default). Validated: full-res 2048^2
image @ ~22.2 GB / 24 (~2.4 GB headroom).
- switch.sh / README / SINGLE_CARD.md: descriptions + the dead docker-compose.yml
link repointed.
Note: on mainline llama.cpp the VRAM lever is -ub (not -b — mainline honors -ub
independently, unlike ik_llama which forces n_ubatch=n_batch).
Docs follow-up (deferred): SINGLE_CARD.md prose still describes `default` as the
old Q3_K_XL path in places; mtp.yml's "froggeric suppresses --reasoning off" note
is now disproven (froggeric v19 + --reasoning off works on mainline b9246).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(engines): ik_llama coverage + MTP-merged-on-mainline corrections
- Correct stale "llama.cpp MTP via community PR / not merged" across
INFERENCE_ENGINES.md, LLAMA_CPP.md, engines/README.md — PR #22673 merged
on mainline 2026-05-16 (the b9246 image we bench is post-merge).
- Add ik_llama.cpp coverage that was missing: engine list / repo tree /
supported-models table (README), workload picker (SINGLE_CARD), choose
table (engines/README), model engines line (qwen3.6-27b/README).
- Document new ik_llama features: two-stage ngram+MTP (PR #1789), -vhad,
--recurrent-ckpt-mode, -ctk-first (IK_LLAMA.md, INFERENCE_ENGINES.md).
- IK_LLAMA gotcha: native template won the 8-pack A/B (103 vs 99, toolcall
tied) so the ik composes default native; froggeric stays vLLM-only.
- New compose iq4ks-two-stage.yml (ngram-mod + MTP via --spec-stage). Boots
+ two-stage spec-dec engages on cu13-server (validated 2026-05-22); perf
bench pending.
Review fixes on top of the drafted changes: repo-tree └→├, a 131K↔262K
self-contradiction in llama-cpp/README, a now-stale "mainline MTP still
open PR" bullet, and softened/attributed the unverified IQ5_KS PPL figures.
External PR links verified resolving; no internal paths leaked.
Docs drafted via the Qwen coding agent; reviewed, corrected, and the new
compose boot-validated by Claude.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(composes): promote native template default + bump llama-cpp context ceilings
llama-cpp/mtp.yml: default ctx 131K → 262K (full native), -ub 512
llama-cpp/mtp-vision.yml: default ctx 49K → 160K, -b/-ub 1024
ik-llama/mtp{,-vision}.yml: native template (froggeric v19 A/B'd, native
won 8-pack 103 vs 99), +'-vhad' (V-cache Hadamard), +'--recurrent-ckpt-
mode auto' (hybrid-aware MTP rejection), removed ipc:host (not needed)
Retire stale artifacts:
ik-llama/patches/froggeric-chat-template/ — native won, no longer mounted
llama-cpp/recipes/single-card-*.sh — host-binary scripts superseded by
compose files (canonical path since v0.8.0)
8-pack ship-gate: 99/150 (vs prior native 103/150) — within noise,
6/8 packs within ±1. Delta is cli-40 variance, not a flag effect
(-vhad and --recurrent-ckpt-mode are not output-quality levers).
* docs(composes): guard -np 1 with hardware-conditional rationale
Add inline ⚠ comment to all 5 single-card composes explaining why
-np 1 is intentional on a single 24 GB card (compute-bound, not
memory-bound — extra slots divide throughput, don't multiply it).
Also expose NP as an env var on the two mainline composes (was
hardcoded; ik composes already had ${NP:-1}).
The 5090/multi-GPU carve-out is explicit: 'on a higher-throughput
card or multi-GPU the trade may flip — re-validate before raising.'
Prevents future agents from blindly parallelizing slots on Ampere.
* fix(composes): move -np guard comment out of folded scalar
The ⚠ -np 1 rationale comment was inside 'command: >-' in all 5
single-card composes. YAML folded scalars treat # lines as content,
not comments — docker compose config failed on all 5 files.
Move the comment block to YAML-level (between volumes: and command:).
All 5 now pass 'docker compose -f <file> config'.
Lesson learned: always validate compose edits with
'docker compose -f <file> config >/dev/null' before committing.
* docs(FAQ): expand WSL2 section with GPU overhead guidance
The existing FAQ entry only mentioned TDR and expandable_segments
gotchas. Expanded to explain the ~1.3 GiB invisible WSL2 overhead,
how it affects each engine path differently (dual-card: noise,
single-card vLLM: one env var, single-card llama.cpp: lower ctx),
and provide a concrete VRAM budget table showing which composes
OOM on WSL2 vs which fit at defaults.
Key finding: mainline llama.cpp mtp.yml at 262K leaves only ~1.5 GB
headroom on Linux — WSL2 eats that to ~0.2 GB (instant OOM). Fix is
CTX_SIZE=131072 (drops to 20 GB, ~2.7 GB WSL2 headroom). ik_llama
composes (IQ4_KS, smaller weights) fit at defaults on WSL2.
No compose file changes — this is FAQ docs only. The llama.cpp
composes already expose CTX_SIZE as an env var override.
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Conceptual narrative (hardware → engine → model size → quant → templates)
that the doc set lacked: GETTING_STARTED is commands-only, GLOSSARY is
isolated definitions, FAQ is troubleshooting. Cross-linked from README,
GETTING_STARTED, and GLOSSARY. Carries a scope banner (stack is
NVIDIA/3090-tested; other vendors are context, not a support commitment)
and an RTX 3090 FP8 caveat (no FP8 weight compute on Ampere; AWQ/GPTQ
for weights, fp8_e5m2 KV only). Links to PULL.md / KV_MATH.md for the
stack's actual fit-math path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ARCHITECTURE.md described the v0.8.0 pull→gate→emit→boot→loop pipeline but
not v0.8.2's user surfaces. Adds: the --recommend verdict UX, the
--submit-last/--submit failure on-ramp, the broadened arch registry
(native built-ins reach a clean serve verdict, per-repo remote-code still
fail-closed), and the optional non-NVIDIA hwdetect subprocess; scripts-tree
pull.sh line annotated. README universal-pull bullet refreshed
(v0.8.0; extended in v0.8.2 — recommend + on-ramp + wider coverage).
Docs only; no behaviour change.
Two new rows in the Supported Models table:
* **Qwen 3.6 35B-A3B** ⭐ NEW v0.7.3 — preview track (production-track
blocked on Genesis v7.73.x). MoE 256 experts × 8 active (~3 B active
params), upstream native loader via vLLM PR #42521. Preview dual:
182/177 wall TPS at 16K — ~2× the Qwen 3.6-27B dense baseline.
* **Gemma 4 26B-A4B** ⭐ NEW v0.7.3 — production via AWQ path. MoE
128 experts × 8 active (~4 B active params). Intel AutoRound INT4
variants are Ampere-blocked (Marlin K-dim alignment); the cyankiwi
AWQ-4bit weights work on Ampere via vendored vLLM PR #40886
(compressed-tensors MoE key remapping). AWQ dual: 139/139 wall TPS
at 32K, CV 0.2% / 0.0%.
Both rows surface the key trade-offs (Genesis-pending for Qwen,
AutoRound-blocked for Gemma on Ampere) so users picking a model see
the constraints up front.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- README.md: example block now shows model + GPUs flow with new --model /
--gpus / --tp / --pp flag examples; scripts/ layout description updated to
"model → GPUs → KV projection".
- docs/FAQ.md: rewrite the "how do I pick model + variant?" answer to
describe the new wizard (model picker → GPU picker → TP/PP auto-pick →
kv-calc projection → boot). Replace CUDA_VISIBLE_DEVICES card-selection
guidance with --gpus flag, preserving the env-var path for compatibility.
- docs/SINGLE_CARD.md, docs/DUAL_CARD.md: fix inline wizard descriptions
("asks engine + workload" / "asks GPU count + workload" → "asks model +
GPUs, projects VRAM budget").
All --variant <name> back-compat invocations across the docs continue to
work; no flag-renaming churn surfaced to users.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@tomByrer asked on disc #125 about pointing the scripts at a model
dir on a second drive. The support has always been there (MODEL_DIR
env var with 4-tier resolution: shell export, .env file, interactive
prompt, silent fallback) but the docs were too terse to discover
without running setup.sh interactively.
Two changes:
- docs/FAQ.md: expand the "where does it go?" entry from a one-liner
to a full explanation covering all 4 resolution paths, HF env-var
integration (we mount host dir INTO container cache, so HF_HOME
isn't directly respected but layout is byte-compatible), and
Windows/WSL2 path handling.
- README.md: add a 3-line parenthetical to step 2 of the quickstart
pointing at MODEL_DIR. Users who copy-paste the quickstart now see
the option without having to dig.
Both surfaces now point at the same canonical answer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename the 4 shipped Gemma composes to drop the redundant `gemma-` prefix
(model is already named by the parent directory `models/gemma-4-31b/`):
- gemma-mtp.yml → dual.yml (recommended default, MTP implicit
— matches Qwen's `dual.yml` shape)
- gemma-mtp-int8.yml → dual-int8.yml (MTP + INT8 PTH KV variant)
- gemma-mtp-tp1.yml → single.yml (TP=1 single-card variant — boot
OOMs on Ampere 24 GB; for 32 GB+ only)
- gemma-dflash.yml → dual-dflash.yml (DFlash drafter variant — same shape
as Qwen's `dual-dflash.yml`)
Convention going forward (matches what Qwen has done since launch):
- File name: `<topology>-<feature>.yml`, model implied by parent directory
- Topology: `single` | `dual` | `dual-nvlink` | `dual4` | etc.
- Feature suffix: `-turbo` | `-dflash` | `-int8` | `-awq` | etc.
Registry tags in `scripts/switch.sh` keep their existing names
(`vllm/gemma-mtp`, `vllm/gemma-mtp-tp1`, `vllm/gemma-dflash`) — only the
file paths in the VARIANTS map are updated. This preserves backward
compat for users running `bash scripts/switch.sh vllm/gemma-mtp` etc.
Container names (`vllm-gemma-4-31b-mtp` etc.) are unchanged — they already
include the model name in the `vllm-<model>-<feature>` form, so no rename
needed there.
Plus continuing the at-a-glance profile schema rollout from 4d7356a:
profile blocks added to 9 more Qwen composes (bounded-thinking,
carnice-bf16mtp, dual4, dual4-dflash, dual-nvlink, dual-nvlink-turbo,
dual-nvlink-dflash, dual-nvlink-dflash-noviz, long-text). Each declares
Model / Topology / Drafter / KV / Vision / Max ctx / Genesis / Best-for.
References updated across BENCHMARKS.md, README.md, DUAL_CARD.md,
UPSTREAM.md, scripts/setup.sh, scripts/switch.sh, all 6 Gemma compose
cross-references, model patch READMEs, and codex-brief-dflash-int8.md.
Refs: noonghunna/club-3090#67
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Listed as a community pointer per disc #108 announcement. Repo is AGPL-3.0,
provides browser admin panel + OpenAI-compatible proxy + multi-instance GPU
orchestration on top of club-3090. Marked "not yet officially adopted" — the
intent is a non-binding pointer until VykosX's project converges on a
stable surface area. Open invitation in the section text for other community
projects to be linked similarly.
Refs: noonghunna/club-3090#108
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
