- docs/IMAGE_STUDIO.md (new): the bundle — quickstart, two-front-ends, gpu-mode
modes, measured Ideogram-4 VRAM-by-resolution, chat-model options, the
PersistentConfig "image button missing" fix, single-GPU fallback.
- docs/README.md: IMAGE_STUDIO.md under the user track.
- README.md: one-line image-gen pointer.
- docs/FAQ.md: point the existing image-gen answer at the turnkey bundle.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Gemma single-card had NO functional default (vLLM single is upstream-gated at
head_dim=512; ik-llama walls ~24K; stock llama.cpp ~12 TPS). beellama is the
only viable fast single-card Gemma-4 path, so promote it to fill the gap.
Registry: beellama/gemma-dflash experimental -> caveats + new
DEFAULTS[(gemma-4-31b, beellama, single)]. Resolver now returns it for Gemma
single (beellama is #1 in ENGINE_PREFERENCE[single]); Gemma dual unchanged.
Validated #441: 131-200K ctx, 47/88 TPS, 8-pack 109/150 think-off /
114/150 think-on (reasoning net-positive on Gemma). Reuses the multi-arch
image from #271 (sm_86/89/120; sm_89/120 compiled-not-validated).
Docs: compose header (caveats + Quality + ctx ceiling), README Gemma row,
BENCHMARKS row marker + intro, UPSTREAM.md row. Resolver guard test flipped
from a degradation assertion to beellama/gemma-dflash; full suite 38/38.
Re-test trigger: re-point the default to the no-fork mainline path when
llama.cpp#23398 (Gemma-4 MTP) merges (docs/UPSTREAM.md).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Make `beellama/dflash` the single-GPU default for Qwen3.6-27B, served via
an unofficial multi-arch image, and demote ik-llama to the balanced alt.
Decision basis (single-3090, 370 W; bench.sh n=5 + quality-test --full):
- Code TPS ~100 vs ik 69 — fastest single-card 3090 code path.
- 8-pack 107/150 (71%) think-off vs ik 99/150 (66%); 113/150 think-on
(same-session). DFlash is output-lossless, so quality == the Q5_K_S target.
- Context ceiling ladder (measured): 130K comfortable, 160K usable
(115K-tok prefill OK, 0.8 GB headroom), 200K boots but OOMs on prefill.
Ships 102K default; raise to 160K via CTX_SIZE.
- ik still wins narrative TPS (63), VRAM, and shipped vision — kept as the
balanced alt (`--variant ik-llama/iq4ks-mtp`).
Registry: `beellama/dflash` experimental -> caveats + a new
DEFAULTS[(qwen3.6-27b, beellama, single)] row. beellama is already #1 in
ENGINE_PREFERENCE[single], so the resolver now returns it for Qwen single.
Image: unofficial multi-arch ghcr build
`beellama-cpp:multiarch-b9459-07ac3ce` (sm_86/89/120 = RTX 3090/4090/5090),
built from Anbeeld/beellama.cpp .devops/cuda.Dockerfile with
CUDA_DOCKER_ARCH="86;89;120" + GGML_CUDA_FA_ALL_QUANTS=ON. Engine reports
`ARCHS = 860,890,1200`; cuobjdump confirms all 3 SASS. CAVEAT: sm_89/sm_120
are compiled but UNVALIDATED — only sm_86/3090 is verified on-rig.
Also bumps the gemma beellama compose to the same multi-arch image (stays
experimental — not promoted). Docs: README single-card framing,
INFERENCE_ENGINES.md, UPSTREAM.md row, BENCHMARKS.md Qwen beellama row.
Resolver guard tests flipped to the new default; full suite 38/38.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
`switch.sh --list` showed every slug regardless of topology, so a
single-GPU box listed dual/multi configs it can't actually run — noise.
Filter `--list` to the topologies the detected GPU count supports:
1 GPU -> single only, 2 -> single+dual, 4+ -> everything. Detection
reuses switch_topology_from_gpus (CUDA/NVIDIA_VISIBLE_DEVICES, then
nvidia-smi) mapped to a topology rank.
- `--all` (and the `--list-all` alias) bypass the filter for
discoverability; --list is deferred until args are parsed so
`--list --all` works in either order.
- Don't silently hide: print a one-line note naming the detected GPU
count and exactly which topologies were hidden, plus a `(+N … hidden
— --all)` tally in the header. No note under --all / when nothing is
hidden.
- Fail-open: with no detection signal (no selector, no nvidia-smi) show
ALL — never hide off a guessed "single" fallback.
- Counts (header + per-model) reflect the visible set; PR-A health
markers/grouping and PR-B Defaults view unchanged.
New test scripts/tests/test-list-topology-filter.sh covers 1/2-GPU
filtering, --all, --list-all, order independence, the --all-without-list
guard, and fail-open. Docs: --help usage, FAQ model-switch entry, README.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
* feat(switch): model-default resolver + user-pinnable defaults
Add a two-layer default scheme on top of the existing <engine>/default map.
<engine>/default stays the maintainer's recommendation (read-only to users);
the new <model>/default is the user's preference — their .env pin if set,
else a curated pick for the detected topology.
compose_registry.py: two maintainer knobs next to DEFAULTS —
RECOMMENDED_DEFAULT_MODELS (short opt-in shortlist, not an exhaustive
ranking; new models are NOT auto-added) and ENGINE_PREFERENCE (per-topology
engine order). Plus resolver helpers (curated_default_target,
community_default_target stub, model_default_pin_key, engine_set/model_set,
slug_topology, model_of_slug).
registry-emit.sh: shared resolver model_default_target(root, model, topology)
— the single injection point for both launchers. Precedence ladder: --variant
(caller) -> .env pin -> community seam (None today) -> curated ENGINE_PREFERENCE
walk (skips non-functional (NA) slugs) -> degradation (notice + nearest-lower
topology, else a clear "pick explicitly" message; never crashes). Plus
x_default_dispatch: X/default with X in engine-set -> engine rec; X in
model-set -> model default; else error (engines + model-ids are disjoint).
switch.sh: <model>/default token; --set-default <slug> / --clear-default
<model> (round-trip the .env pin CLUB3090_DEFAULT_<MODELID>); a Defaults view
appended to --list (also standalone via --defaults) marking user-pin vs
curated. PR-A's grouping/markers/counts and the --force gate preserved.
launch.sh: bare invocation -> first installed shortlist model -> its
<model>/default (no full wizard); a pinned fast-path ("Launch your default
<slug>? [Y/n]"); a post-boot offer ("Make <slug> your default? [y/N]"). Any
narrowing flag keeps the explicit wizard path. Also load CLUB3090_DEFAULT_*
pin keys from .env even when MODEL_DIR is exported in the shell (the existing
.env loader is MODEL_DIR-gated, which would otherwise hide the pin).
Pin validation is warn + fall back, never blocking: unknown slug / wrong
model / topology-mismatch / (NA) status -> notice + curated default.
Tests: new test-model-default-resolver.sh (curated walk, (NA) skip,
degradation, X/default dispatch, pin override + all validation paths,
community seam skipped, .env round-trip); test-default-resolver.sh extended
with <model>/default launch.sh dispatch. Full suite green (only the
pre-existing test-submit-bench.sh fixture failure remains); 45 entries
unchanged; kv-calc calibration 17/17.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* docs: document the model-default resolver + user pins
Ship the user/contributor docs alongside the resolver code (repo convention —
update existing docs, no new top-level doc).
- README: single-card realign (ik-llama = fastest blessed single default,
llama.cpp = cliff-immune alternative — matches ENGINE_PREFERENCE single
order); add "pin your default" + <model>/default to Quick start.
- CLAUDE.md (= AGENTS.md): document RECOMMENDED_DEFAULT_MODELS +
ENGINE_PREFERENCE + the shared resolver as maintainer knobs next to DEFAULTS.
- FAQ: extend "switch to a different model" with <model>/default; new "How do
I set my own default config?" Q (two-layer model, --set-default/--clear-
default/--defaults, .env key, warn+fallback validation).
- SINGLE_CARD / DUAL_CARD: the resolver + the per-topology engine order; pin
hint.
- GETTING_STARTED: first-run uses <model>/default + the in-flow "set as
default?" prompt.
- ADDING_MODELS: a new model resolves via ENGINE_PREFERENCE; add a DEFAULTS row
per engine; RECOMMENDED_DEFAULT_MODELS is not auto-grown.
- UPSTREAM: beellama Docker-image row (gates beellama onboarding; the resolver
skips it today and it auto-promotes to single default on catalog).
- models/qwen3.6-27b/CHANGELOG: dated PR-B entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Gemma 4 31B: was "2× 3090 only" with footnote about single-card OOMs.
We now ship beellama.cpp single-card path validated 2026-05-27 in
discussion #239 — 47/88 TPS, 100–150K ctx, 109/114 8-pack. Row +
footnote refreshed.
Qwen3.6 35B-A3B: was "preview / 2× 3090" only. PR #243 landed the
ik_llama --fit q8/q5 single-card variant — 103/149 TPS at 196K with
hermes 11/20, aider 12/30, cli 12/40 measured post-fix verifiers.
Status upgraded, row + card-count + engines refreshed.
Gemma 4 26B-A4B: left unchanged per a single-card test not yet run on
Ampere; added a small footnote (²) to disclose that explicitly rather
than imply "dual only because Ampere can't fit."
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Currently the README's tagline + welcome paragraph both say "RTX 3090s"
explicitly — a 4090 / 5090 visitor scanning the top of the page has no
signal that the composes run on their card. They bounce thinking
"3090-only" before reaching the Hardware FAQ.
Adds a single callout (mirroring the existing 🪟 WSL2 pattern from
Quick start) right under the welcome paragraph: links straight to the
two FAQ entries with measured cross-rig numbers + per-class gotchas.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Diagnostics section:
- New "If launch.sh / switch.sh won't boot" subsection: --force escalation
(skip preflight only) then full bypass — `MODEL_DIR=… docker compose -f
<compose> up -d` with the three common compose paths (llama.cpp single,
ik_llama single, vLLM dual), the verify curl, and `down`. Covers the
Windows CRLF/.env, missing-PyYAML, and false-preflight failure modes.
- Capture-to-file example now uses --full (the complete cross-rig pass);
note says drop --full for the ~2 s hardware-only capture.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The only WSL pointers were ~185-235 lines below Quick start (docs tree +
"What you'll need") — a Windows user hits the Linux/Docker/bash commands
first and fails before discovering WSL_SETUP.md. Add a one-line callout at
the top of Quick start (before the command block), the standard place for
OS prerequisites.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
- README: drop the redundant standalone quality-test block left by #215 and
fold `quality-test.sh --reasoning` into the existing Benchmarks section.
- WSL_SETUP.md: add a "Native llama.cpp in WSL (no Docker)" section (Step 2
passthrough is all native needs; lift llama-server flags from the compose),
and a "Shrink the overhead" block in Step 8 — headless AI GPU, close GPU
apps, current driver, WSLg off, dual-boot — distinguishing the unavoidable
WSL2 paravirt context from the reclaimable display/WDDM portion.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
New docs/WSL_SETUP.md walks a fresh Windows box to a running club-3090:
WSL2 + Ubuntu install, Windows NVIDIA driver / GPU passthrough, .wslconfig
RAM sizing (the silent host-RAM-starvation OOM, #32), Docker + container
toolkit, and the two gotchas not documented anywhere yet — cloning onto
ext4 (not /mnt/c) and keeping .env/scripts LF (not CRLF). Weights via
WEIGHTS=gguf + MODEL_DIR, then the normal launch.
Cross-links the existing runtime tuning (VRAM budget, TDR, expandable_segments)
in FAQ.md + HARDWARE.md rather than duplicating it; adds discovery links from
README ("What you'll need" + docs tree) and the FAQ Windows/WSL2 answer.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Two new sections in the operational/reference half so users find the
benchmark + triage-report commands without spelunking scripts/:
- Benchmarks: placed right after "Measured TPS at a glance" (reproduce
those numbers) — bench.sh (TPS), quality-test.sh (modes + --pack),
rebench-full.sh (canonical 5-step pipeline + endpoint-first mode).
- Diagnostics: report.sh (default + --verify/--stress/--soak/--bench/
--full/--no-redact) with the redaction + Cliff-2b/soak rationale.
Reasoning-suite (quality-test.sh --reasoning) intentionally omitted —
not on master yet (lands with the benchlocal reasoning packs).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
SGLang is blocked on Ampere (watch-list, not a working engine), so listing it
in the README's multi-engine tagline / engine-picker / supported-models table
overstated it as a supported route. Removed from:
- tagline ("Multi-engine (vLLM, llama.cpp, ik_llama)")
- engine-picker inline list
- all 4 supported-models "Engines" cells (kept llama.cpp ❌ — it IS a supported
engine with model-specific status; SGLang is not a supported engine at all)
Kept a one-line "SGLang was evaluated — currently blocked on Ampere, see
docs/engines/SGLANG.md" pointer in the multi-engine bullet + the docs-tree
annotation, so the evaluation context isn't erased (full detail stays in
docs/engines/SGLANG.md + INFERENCE_ENGINES.md).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The quick-start listed `vllm/default` as "single-card chat (recommended)", but
that path is on the purged Genesis-MTP pin (#167) — it won't boot — and is
Cliff-2-prone for agentic/accumulated-context use, so newcomers were handed a
dead command. Per the #209 decision (llamacpp/default = simplest default,
ik-llama = fastest):
- "(recommended)" → llamacpp/default (cliff-immune, 200K, ~51/60 TPS)
- ik-llama/iq4ks-mtp kept as the FASTEST single-card pick
- dropped the #167-blocked vllm/default line (still documented in SINGLE_CARD.md
with strikethrough for when the pin lands) + the redundant bare llamacpp/mtp
(same compose as llamacpp/default)
- vllm/dual stays (clean nightly, boots) with a note that vLLM single-card is #167-blocked
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Same boots≠fills correction as the merged ctx sweep — this 'Supported models'
highlights cell was missed (sweep excluded it). Single-card llama.cpp is 200K
max-safe, not 262K (262K is dual vLLM turbo). Also tidied ik TPS ~62→~60/69 to
match BENCHMARKS + tagged it 'fastest single-card'. Dual-card 262K refs (L31/67/
103) and the LLAMA_CPP.md '262K recipe' pointer (L139, pending fill-test) left.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
ik-llama/iq4ks-mtp is the fastest single-card path (~18-20% faster decode +
leanest VRAM, #184) but was absent from the README quick-start and the registry
ctx values had drifted from the 200K shipped default. Keeping llamacpp/default =
mainline (the simplest / clean-upstream-image pick) — surfacing ik, not renaming.
- README quick-start: add `ik-llama/iq4ks-mtp` (fastest single-card) alongside
the llamacpp/* variants.
- docs/SINGLE_CARD.md: one-line "simplest (llamacpp/default) vs fastest
(ik-llama/iq4ks-mtp)" steer atop the config table (ik rows were already
present + ⭐-marked).
- compose_registry.py: fix stale wizard-projection max_ctx to the 200K default —
llamacpp/default + llamacpp/mtp were 131072 (too low), ik-llama/iq4ks-mtp was
262144 (boots-not-fills); all → 200000. Vision entries (49152 / 163840) already
correct. Reworded the "262K via -ub 512" comment (that was the boots≠fills
false ceiling). NOTE: launch.sh doesn't pass max_ctx to runtime, so this is
wizard-projection accuracy only — runtime ctx still comes from the compose
CTX_SIZE=200000 default.
Validated: registry imports; test-launch-compat, test-switch-registry-parity
(45 composes, parity), test-profiles-compat all pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The single-card MTP defaults were bumped 262144 -> 200000 (PR #201 + the ik
commit) after the "boots != fills" finding: 262K boots and passes verify-stress
but only fills ~125K before the FA-scratch-at-fill OOM; 200K fills ~183K with
~1.1 GB margin (see docs/CLIFFS.md). The docs lagged.
Corrected to 200K (the shipped, fillable default) across the Q4_K_M-MTP +
ik IQ4_KS single-card paths:
- README.md (quick-start + TL;DR robustness pitch)
- docs/SINGLE_CARD.md (cliff-warning row, config table, llamacpp/default + ik
narratives, fallback table, Cliff 2 explainer)
- docs/engines/IK_LLAMA.md (defaults + max-context override)
- models/qwen3.6-27b/llama-cpp/README.md (single/mtp.yml heading + workhorse line)
- scripts/switch.sh (--list/--help comment)
Also reconciled an older drift: llamacpp/default = llamacpp/mtp alias (collapsed
2026-05-22), Q4_K_M MTP, no vision, ~52/61 TPS -- several spots still described
it as the retired Q3_K_XL @ ~21 TPS + vision.
Left for a separate fill-test (smaller-quant / no-MTP headroom may genuinely
exceed 200K): the "going-to-262K" max-ctx sections in docs/engines/LLAMA_CPP.md
+ VLLM.md, and the Q3_K_XL legacy rows in models/qwen3.6-27b/llama-cpp/README.md.
Dual-card vLLM 262K, the model's 262144 max_position_embeddings, and the CLIFFS
"boots!=fills" explainer are correct as-is and untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
We don't distribute a club-3090 vLLM image to users, so the
build-vllm-image.yml workflow + its surrounding docs are dead weight. The
workflow had also been failing on every tag push and the weekly cron because
its base nightly (nightly-1acd67a7) was purged from Docker Hub (#167/#407) —
that red CI on the v0.8.4 tag is what surfaced this.
Removed:
- .github/workflows/build-vllm-image.yml (the GHCR image builder)
- docker/vllm-club3090/Dockerfile (its build recipe)
- docs/CI_RUNNER_SETUP.md (build/distribution doc)
- README + UPSTREAM references to ghcr.io/noonghunna/vllm-club3090
- docs/README index link to the deleted CI doc
Kept: the generic VLLM_IMAGE override (now documented against
vllm/vllm-openai:latest — also the #167 workaround). The launch-compat test
still exercises that override, just with an upstream image.
Recoverable from history if we ever want to ship an image again.
NOT touched: patch_attribution.py still references the dockerfile_bake
delivery mode — left for a separate decision (internal, harmless).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* refactor(llamacpp): collapse single-card composes 3→2 (default = mtp alias)
Mainline Q4_K_M reaches the full 262K with -ub 512 (validated: boot + serve +
verify-stress 7/7 + soak PASS), so the old vanilla Q3_K_XL docker-compose.yml no
longer earns a separate profile. Collapse the llama.cpp single-card tree to two
compose files:
- `llamacpp/default` is now an ALIAS for `llamacpp/mtp` (mtp.yml) — launch.sh +
compose_registry repointed. All references (estate-CLI default, tests, docs)
keep resolving; the variant name survives.
- Retired models/qwen3.6-27b/llama-cpp/compose/single/docker-compose.yml.
- mtp.yml: document the -ub 512 -> 262K recipe in the header.
- mtp-vision.yml: align with the ik vision profile — default 160K (was 49K) via
-ub 512, and pin --image-min-tokens 1024 / --image-max-tokens 4096 (mainline
previously left image bounds at the model default). Validated: full-res 2048^2
image @ ~22.2 GB / 24 (~2.4 GB headroom).
- switch.sh / README / SINGLE_CARD.md: descriptions + the dead docker-compose.yml
link repointed.
Note: on mainline llama.cpp the VRAM lever is -ub (not -b — mainline honors -ub
independently, unlike ik_llama which forces n_ubatch=n_batch).
Docs follow-up (deferred): SINGLE_CARD.md prose still describes `default` as the
old Q3_K_XL path in places; mtp.yml's "froggeric suppresses --reasoning off" note
is now disproven (froggeric v19 + --reasoning off works on mainline b9246).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* docs(engines): ik_llama coverage + MTP-merged-on-mainline corrections
- Correct stale "llama.cpp MTP via community PR / not merged" across
INFERENCE_ENGINES.md, LLAMA_CPP.md, engines/README.md — PR #22673 merged
on mainline 2026-05-16 (the b9246 image we bench is post-merge).
- Add ik_llama.cpp coverage that was missing: engine list / repo tree /
supported-models table (README), workload picker (SINGLE_CARD), choose
table (engines/README), model engines line (qwen3.6-27b/README).
- Document new ik_llama features: two-stage ngram+MTP (PR #1789), -vhad,
--recurrent-ckpt-mode, -ctk-first (IK_LLAMA.md, INFERENCE_ENGINES.md).
- IK_LLAMA gotcha: native template won the 8-pack A/B (103 vs 99, toolcall
tied) so the ik composes default native; froggeric stays vLLM-only.
- New compose iq4ks-two-stage.yml (ngram-mod + MTP via --spec-stage). Boots
+ two-stage spec-dec engages on cu13-server (validated 2026-05-22); perf
bench pending.
Review fixes on top of the drafted changes: repo-tree └→├, a 131K↔262K
self-contradiction in llama-cpp/README, a now-stale "mainline MTP still
open PR" bullet, and softened/attributed the unverified IQ5_KS PPL figures.
External PR links verified resolving; no internal paths leaked.
Docs drafted via the Qwen coding agent; reviewed, corrected, and the new
compose boot-validated by Claude.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* refactor(composes): promote native template default + bump llama-cpp context ceilings
llama-cpp/mtp.yml: default ctx 131K → 262K (full native), -ub 512
llama-cpp/mtp-vision.yml: default ctx 49K → 160K, -b/-ub 1024
ik-llama/mtp{,-vision}.yml: native template (froggeric v19 A/B'd, native
won 8-pack 103 vs 99), +'-vhad' (V-cache Hadamard), +'--recurrent-ckpt-
mode auto' (hybrid-aware MTP rejection), removed ipc:host (not needed)
Retire stale artifacts:
ik-llama/patches/froggeric-chat-template/ — native won, no longer mounted
llama-cpp/recipes/single-card-*.sh — host-binary scripts superseded by
compose files (canonical path since v0.8.0)
8-pack ship-gate: 99/150 (vs prior native 103/150) — within noise,
6/8 packs within ±1. Delta is cli-40 variance, not a flag effect
(-vhad and --recurrent-ckpt-mode are not output-quality levers).
* docs(composes): guard -np 1 with hardware-conditional rationale
Add inline ⚠ comment to all 5 single-card composes explaining why
-np 1 is intentional on a single 24 GB card (compute-bound, not
memory-bound — extra slots divide throughput, don't multiply it).
Also expose NP as an env var on the two mainline composes (was
hardcoded; ik composes already had ${NP:-1}).
The 5090/multi-GPU carve-out is explicit: 'on a higher-throughput
card or multi-GPU the trade may flip — re-validate before raising.'
Prevents future agents from blindly parallelizing slots on Ampere.
* fix(composes): move -np guard comment out of folded scalar
The ⚠ -np 1 rationale comment was inside 'command: >-' in all 5
single-card composes. YAML folded scalars treat # lines as content,
not comments — docker compose config failed on all 5 files.
Move the comment block to YAML-level (between volumes: and command:).
All 5 now pass 'docker compose -f <file> config'.
Lesson learned: always validate compose edits with
'docker compose -f <file> config >/dev/null' before committing.
* docs(FAQ): expand WSL2 section with GPU overhead guidance
The existing FAQ entry only mentioned TDR and expandable_segments
gotchas. Expanded to explain the ~1.3 GiB invisible WSL2 overhead,
how it affects each engine path differently (dual-card: noise,
single-card vLLM: one env var, single-card llama.cpp: lower ctx),
and provide a concrete VRAM budget table showing which composes
OOM on WSL2 vs which fit at defaults.
Key finding: mainline llama.cpp mtp.yml at 262K leaves only ~1.5 GB
headroom on Linux — WSL2 eats that to ~0.2 GB (instant OOM). Fix is
CTX_SIZE=131072 (drops to 20 GB, ~2.7 GB WSL2 headroom). ik_llama
composes (IQ4_KS, smaller weights) fit at defaults on WSL2.
No compose file changes — this is FAQ docs only. The llama.cpp
composes already expose CTX_SIZE as an env var override.
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Conceptual narrative (hardware → engine → model size → quant → templates)
that the doc set lacked: GETTING_STARTED is commands-only, GLOSSARY is
isolated definitions, FAQ is troubleshooting. Cross-linked from README,
GETTING_STARTED, and GLOSSARY. Carries a scope banner (stack is
NVIDIA/3090-tested; other vendors are context, not a support commitment)
and an RTX 3090 FP8 caveat (no FP8 weight compute on Ampere; AWQ/GPTQ
for weights, fp8_e5m2 KV only). Links to PULL.md / KV_MATH.md for the
stack's actual fit-math path.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
ARCHITECTURE.md described the v0.8.0 pull→gate→emit→boot→loop pipeline but
not v0.8.2's user surfaces. Adds: the --recommend verdict UX, the
--submit-last/--submit failure on-ramp, the broadened arch registry
(native built-ins reach a clean serve verdict, per-repo remote-code still
fail-closed), and the optional non-NVIDIA hwdetect subprocess; scripts-tree
pull.sh line annotated. README universal-pull bullet refreshed
(v0.8.0; extended in v0.8.2 — recommend + on-ramp + wider coverage).
Docs only; no behaviour change.
Two new rows in the Supported Models table:
* **Qwen 3.6 35B-A3B** ⭐ NEW v0.7.3 — preview track (production-track
blocked on Genesis v7.73.x). MoE 256 experts × 8 active (~3 B active
params), upstream native loader via vLLM PR #42521. Preview dual:
182/177 wall TPS at 16K — ~2× the Qwen 3.6-27B dense baseline.
* **Gemma 4 26B-A4B** ⭐ NEW v0.7.3 — production via AWQ path. MoE
128 experts × 8 active (~4 B active params). Intel AutoRound INT4
variants are Ampere-blocked (Marlin K-dim alignment); the cyankiwi
AWQ-4bit weights work on Ampere via vendored vLLM PR #40886
(compressed-tensors MoE key remapping). AWQ dual: 139/139 wall TPS
at 32K, CV 0.2% / 0.0%.
Both rows surface the key trade-offs (Genesis-pending for Qwen,
AutoRound-blocked for Gemma on Ampere) so users picking a model see
the constraints up front.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
- README.md: example block now shows model + GPUs flow with new --model /
--gpus / --tp / --pp flag examples; scripts/ layout description updated to
"model → GPUs → KV projection".
- docs/FAQ.md: rewrite the "how do I pick model + variant?" answer to
describe the new wizard (model picker → GPU picker → TP/PP auto-pick →
kv-calc projection → boot). Replace CUDA_VISIBLE_DEVICES card-selection
guidance with --gpus flag, preserving the env-var path for compatibility.
- docs/SINGLE_CARD.md, docs/DUAL_CARD.md: fix inline wizard descriptions
("asks engine + workload" / "asks GPU count + workload" → "asks model +
GPUs, projects VRAM budget").
All --variant <name> back-compat invocations across the docs continue to
work; no flag-renaming churn surfaced to users.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@tomByrer asked on disc #125 about pointing the scripts at a model
dir on a second drive. The support has always been there (MODEL_DIR
env var with 4-tier resolution: shell export, .env file, interactive
prompt, silent fallback) but the docs were too terse to discover
without running setup.sh interactively.
Two changes:
- docs/FAQ.md: expand the "where does it go?" entry from a one-liner
to a full explanation covering all 4 resolution paths, HF env-var
integration (we mount host dir INTO container cache, so HF_HOME
isn't directly respected but layout is byte-compatible), and
Windows/WSL2 path handling.
- README.md: add a 3-line parenthetical to step 2 of the quickstart
pointing at MODEL_DIR. Users who copy-paste the quickstart now see
the option without having to dig.
Both surfaces now point at the same canonical answer.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Rename the 4 shipped Gemma composes to drop the redundant `gemma-` prefix
(model is already named by the parent directory `models/gemma-4-31b/`):
- gemma-mtp.yml → dual.yml (recommended default, MTP implicit
— matches Qwen's `dual.yml` shape)
- gemma-mtp-int8.yml → dual-int8.yml (MTP + INT8 PTH KV variant)
- gemma-mtp-tp1.yml → single.yml (TP=1 single-card variant — boot
OOMs on Ampere 24 GB; for 32 GB+ only)
- gemma-dflash.yml → dual-dflash.yml (DFlash drafter variant — same shape
as Qwen's `dual-dflash.yml`)
Convention going forward (matches what Qwen has done since launch):
- File name: `<topology>-<feature>.yml`, model implied by parent directory
- Topology: `single` | `dual` | `dual-nvlink` | `dual4` | etc.
- Feature suffix: `-turbo` | `-dflash` | `-int8` | `-awq` | etc.
Registry tags in `scripts/switch.sh` keep their existing names
(`vllm/gemma-mtp`, `vllm/gemma-mtp-tp1`, `vllm/gemma-dflash`) — only the
file paths in the VARIANTS map are updated. This preserves backward
compat for users running `bash scripts/switch.sh vllm/gemma-mtp` etc.
Container names (`vllm-gemma-4-31b-mtp` etc.) are unchanged — they already
include the model name in the `vllm-<model>-<feature>` form, so no rename
needed there.
Plus continuing the at-a-glance profile schema rollout from 4d7356a:
profile blocks added to 9 more Qwen composes (bounded-thinking,
carnice-bf16mtp, dual4, dual4-dflash, dual-nvlink, dual-nvlink-turbo,
dual-nvlink-dflash, dual-nvlink-dflash-noviz, long-text). Each declares
Model / Topology / Drafter / KV / Vision / Max ctx / Genesis / Best-for.
References updated across BENCHMARKS.md, README.md, DUAL_CARD.md,
UPSTREAM.md, scripts/setup.sh, scripts/switch.sh, all 6 Gemma compose
cross-references, model patch READMEs, and codex-brief-dflash-int8.md.
Refs: noonghunna/club-3090#67
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Listed as a community pointer per disc #108 announcement. Repo is AGPL-3.0,
provides browser admin panel + OpenAI-compatible proxy + multi-instance GPU
orchestration on top of club-3090. Marked "not yet officially adopted" — the
intent is a non-binding pointer until VykosX's project converges on a
stable surface area. Open invitation in the section text for other community
projects to be linked similarly.
Refs: noonghunna/club-3090#108
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Two minimal pointers added now that scripts are engine-agnostic:
- README.md: host-build invocation pattern + link to disc #88
- CONTRIBUTING.md: same + new bullet for power-cap-sweep.sh ask
(anchors for cards we don't have yet — A5000/A6000, 4080, 5060
Ti/5080, modded variants)
No structural changes — keeps the existing flow intact for vLLM-stack
users; just opens the door for non-Docker contributors who would have
been silently turned away before #87 / #88.
scripts/report.sh now has four optional sections:
--verify verify-full.sh (~1-2 min)
--stress verify-stress.sh (~5-10 min, 7/7 incl. Cliff 2 needles)
--soak SOAK_MODE=continuous + summary.md embed (~25 min, catches Cliff 2b)
--bench bench.sh (~3 min, canonical TPS)
--full all four (~35 min total, the canonical cross-rig pass)
Why soak as its own flag: verify-full + verify-stress + bench all PASS on
configs that FAIL the multi-turn continuous soak (Cliff 2b at ~25K accumulated
tokens). Until upstream lands a fix, soak is the only test that catches the
agentic-workload failure mode that bit issues #41 / #42 / #43 / #45.
Propagated --full as the recommended single-command path through:
- README.md (bug/bench callout)
- CONTRIBUTING.md (Numbers from your rig + new compose variant gate)
- .github/PULL_REQUEST_TEMPLATE.md (one checkbox covers verify+stress+soak+bench)
- .github/ISSUE_TEMPLATE/numbers-from-your-rig.yml (single paste includes soak)
- BENCHMARKS.md ("How to add a row for your rig")
Backward compatible: existing --verify and --bench flags unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Companion docs update for 43fe2a4. Three places where users would expect
to find this:
- README.md — adds step 7 to quick-start ("Keep your install up-to-date")
with the upgrade flow + the soft-warn note. Updates the directory tree
to list update.sh and refresh the preflight.sh one-line summary.
- docs/FAQ.md — new "How do I keep my install up-to-date?" entry under
Setup, covering the dirty-tree refusal, --dry-run / --force flags, and
the cross-link to the existing Genesis-pin warning. Also tweaked the
"How do I bump Genesis" entry to point at update.sh as the normal path
and frame manual bumps as the testing-only escape hatch.
- CHANGELOG.md — 2026-05-03 entry describing both pieces (preflight_repo_drift
+ scripts/update.sh) and the JusefPol dual-nvlink variant landing.
Maintainer rig is 2x 3090. Users with 3+ GPUs (4x 3090, 8x A6000,
mixed setups) have been asking "does club-3090 apply?" — yes, but we
can't ship pre-baked configs for hardware we can't validate.
New `docs/MULTI_CARD.md` explains:
- **What scales** going TP=2 → TP=4 → TP=8: per-card weights drop
proportionally, KV pool grows linearly, Cliff 2 disappears entirely
on TP=4+ (DeltaNet GDN forward state splits across cards).
- **What doesn't scale**: per-stream decode TPS without NVLink. PCIe
NCCL all-reduce overhead grows with TP count — TP=4 PCIe per-stream
may be lower than TP=2. Aggregate concurrent throughput still scales.
- **Valid TP values for Qwen3.6-27B**: 1, 2, 4, 5, 8, 10. Must divide
both 80 attention heads AND 5 KV heads cleanly. TP=3, 6, 7, 9 do NOT
work — vLLM errors at boot. Awkward GPU counts (3, 6, 7) need to use
the next-lower valid TP with idle cards.
- **Derivation recipe**: copy `dual.yml`, change three lines
(--tensor-parallel-size, --max-num-seqs, --max-num-batched-tokens),
pick a distinct container_name + port. Marlin pad-sub-tile-n patch
stays mounted (more relevant at higher TP, not less).
- **What to expect on TP=4** (4x 3090 PCIe): per-card peak ~16-18 GB
(vs 23.6 GB on TP=2), Cliff 2 doesn't apply, per-stream TPS likely
drops to ~50-65 narr / ~70-80 code from PCIe NCCL overhead, but KV
pool 2x larger means more concurrent streams fit.
- **What to expect on TP=8** (server-class): per-card pressure
essentially disappears. With NVLink fabric, per-stream TPS could
approach 1.6-1.8x single-card vs the ~1.0x we see on PCIe TP=2.
- **Cross-rig data ask**: TP=4 PCIe, TP=4 with NVLink, TP=8 server-
class, TP=4 mixed cards. Each is a coverage gap we'd love filled.
Honest disclaimer at the top: nothing in this doc is locally measured,
all derived from documented vLLM TP behavior + our TP=2 baseline +
Marlin pad math. Asks community contributors with 3+ card hardware to
share `bash scripts/report.sh --bench > my-rig.md` results.
Why no pre-baked composes:
1. Can't hardware-test them.
2. Hardware combinations explode (different VRAM, NVLink topology,
power profiles, allreduce characteristics — no single quad.yml
optimal for all).
3. Users at this scale are typically experienced — they need
methodology + constraints + dial, not hand-held tested compose.
If a community member contributes a tested compose for their topology
(verify-stress passing + bench numbers), we ship it under
models/qwen3.6-27b/vllm/compose/ with credit.
Cross-links: README.md "Pick your path" table now includes 3+ GPU row;
DUAL_CARD.md gets a header pointer to MULTI_CARD.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Bug report and numbers-from-your-rig issue templates previously asked for
6+ separate fields each (docker logs, verify-full, nvidia-smi, GPU config,
compose variant, commit, etc.). The new scripts/report.sh captures all
of that in one paste-ready dump, plus the data we kept asking individually
(power caps + default vs current, NVLink topology, OS, system RAM, idle
GPU VRAM detection, container runtime versions, cached vLLM image SHAs).
Templates restructured to lead with `bash scripts/report.sh > my-rig.md`:
- bug-report.yml: 6 required + 2 optional fields → 3 required (what
happened, repro, rig report) + 1 optional (extras / fallback). Net -38%
lines, but more importantly: one command instead of six manual asks.
- numbers-from-your-rig.yml: 5 required + 2 optional fields → 1 required
(rig + bench report via `--bench`) + 1 optional (notes / fallback). Net
-38% lines.
Both templates retain manual-fallback guidance for users who can't run
report.sh (no shell access, different rig environment, etc.).
CONTRIBUTING.md "Numbers from your rig" + "Bug reports" entries updated
to reference report.sh as the primary path. README.md repo layout adds
report.sh to scripts/ list, plus a one-liner in the docs-and-extras
paragraph pointing affected users at the script.
Net effect: future cross-rig contributors share more standardized data
with less effort. Triage threads stop bouncing on "could you also send
me X?" follow-ups for the surface report.sh covers.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
