The shared served-model-name `qwen3.6-27b-autoround` (from #490) mislabels the
non-autoround 27b scenes (fp8 dual-max, lmcache): /v1/models advertises
"autoround" while `root` points at qwen3.6-27b-fp8. Same class on gemma-4-31b —
after the v0.24.0 consolidation the default is cyankiwi qat-AWQ-INT4 (bf16 KV),
yet the LiteLLM route still targeted `gemma-4-31b-autoround` (a latent #482 drift).
Fix WITHOUT breaking anything, via vLLM multi-served-name:
- Every 27b scene now serves `qwen3.6-27b <its-quant-name>`; every gemma-31b
scene serves `gemma-4-31b <its-quant-name>`. The neutral name is PRIMARY
(honest /v1/models id); the quant-specific name is retained as a live ALIAS.
- LiteLLM: add `qwen3.6-27b` / `gemma-4-31b` canonical public routes; keep the
`-autoround` routes as back-compat aliases (same upstream). Repairs the gemma drift.
- Migrate our own MODEL= defaults + docs (bench/verify/quality/launch/setup, c3,
tui-core, EXAMPLES, ...) to the neutral name. Weights slugs (`-autoround-int4`)
untouched; CHANGELOG + results/ history left as-is.
Retiring the `-autoround` alias entirely is a deliberate later step once nothing
still asks for it.
Live-verified on-rig (single/minimal, stock v0.24.0): /v1/models lists BOTH names
(root=...-autoround-int4); chat to `qwen3.6-27b` AND `qwen3.6-27b-autoround` both
return 200; `qwen3.6-27b-fp8` correctly 404s. Full shell gate 59/59 (1 = known
worktree-fixture); c3 pytest 41 passed; served-name arg-order + YAML validated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
New dedicated guide for getting the most out of a PCIe-only multi-GPU
rig: reading nvidia-smi topo -m (why PHB not PIX), NUMA/NPS1, triple-slot
placement + M.2 lane-sharing, BIOS (Above 4G / ReBAR / ACS / IOMMU),
enabling consumer P2P via the patched driver + NVLINK_MODE=pcie_p2p,
realistic measured expectations (cross-rig #91/#95/#77), verifying
engagement via report.sh's new field, and troubleshooting. Generic
guidance with ROMED8-2T as a labeled example board.
Cross-referenced from HARDWARE.md (NVLink section), DUAL_CARD.md (NVLink
auto-detection), FAQ.md (Do I need NVLink?), and the docs index.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Sweep the studio docs to match the consolidation: `image-studio` /
`video-studio` / `comfyui` scene refs → the single `ai-studio` scene, drop the
gemma-12b chat-on-the-other-card framing, and document the new pieces.
- README: 8→9 lanes (+10Eros); "Lanes vs modes" → "One scene, lanes inside it".
- video.md: model manifest LoRA `-384`→`-384-1.1`, +10Eros unet row, lane matrix
+🔓 10Eros, VRAM note → `ai-studio`.
- audio.md: step-voice ⊕-video mutex bullet (c3 guards it) + start-from-c3 note.
- image.md / FAQ.md / services/studio/README.md: scene-name + bring-up fixups.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Groups all Club 3090 AI Studio docs under docs/ai-studio/ and fills the
gaps (no front-door overview; audio undocumented; image doc stale).
- NEW docs/ai-studio/README.md — the high-level overview: the director-driven
flow, the full 8-lane matrix, lanes-vs-modes, the shared substrate
(director/gallery/orchestrator/shim/tts/step-voice/gpu-mode), the VRAM
truth, productization framing.
- NEW docs/ai-studio/audio.md — voices (Kokoro narration), premium voice
(Step-Audio-EditX clone+edit), music (ACE-Step), SFX (Stable Audio). Notes
the future Understand pillar (ASR/diarize/separate) kept in private design.
- docs/ai-studio/image.md (was IMAGE_STUDIO.md) — REFRESHED to the current
lanes: HiDream-O1 + Ideogram-4 + Chroma + the native-button shim (the old
doc still described the retired gemma-12b image-studio mode).
- docs/ai-studio/video.md (was VIDEO_STUDIO.md) — TRIMMED to video-only;
image/audio sections moved to their own docs; cross-links added.
- Rewrote every reference across the repo (~13 files: root README, FAQ, docs
index, gpu-mode.sh, the pipe builder + studio service composes/py) to the
new paths. docs/README.md index gains the overview + audio entries.
LOCAL_AI_PRIMER.md stays at docs/ (general). The realtime voice-agent design
stays private until built.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
- docs/IMAGE_STUDIO.md (new): the bundle — quickstart, two-front-ends, gpu-mode
modes, measured Ideogram-4 VRAM-by-resolution, chat-model options, the
PersistentConfig "image button missing" fix, single-GPU fallback.
- docs/README.md: IMAGE_STUDIO.md under the user track.
- README.md: one-line image-gen pointer.
- docs/FAQ.md: point the existing image-gen answer at the turnkey bundle.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Capture the quant/KV findings so others don't re-hit them:
DTYPE_MATRIX.md — new "KV-quant × checkpoint compatibility — the two Ampere traps":
(1) fp8 KV is rejected for compressed-tensors checkpoints (AWQ/FP8/INT8 weights),
flag-independent; auto_round/GPTQ unaffected. (2) int8_per_token_head is NATIVE
in stock v0.22.0 for uniform-head-dim models — #40391 is the Gemma-4-only
(interleaved 256/512 head-dim → page-size unification) adapter, don't copy it.
+ an Ampere KV-dtype picker table; tag the INT8-PTH row native/overlay status.
QUANTIZATION.md — new §4a "Picking a quant by fidelity (KLD) — and where QAT fits":
Phaelon74 KLD ranking (INT8 0.009 < FP8 0.023 < AWQ-BF16-INT4 0.042 < AWQ-INT4
0.051 < AutoRound 0.063); KLD is weights-only (KV-quant adds separate error);
QAT only out-earns PTQ at <=4-bit (8-bit PTQ already near-lossless); dual=fidelity /
single=fit tiering. + int8_per_token_head row + the fp8-guard caveat in §5.
FIX the stale §4 FP8 line: FP8 *weights* DO run on Ampere via Marlin W8A16 (not
"emulated / KV-only") and are a top-fidelity option.
FAQ.md — new Q: "My AWQ/FP8 model errors on --kv-cache-dtype fp8" → use int8-PTH.
Gate 42/42.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add HF_HUB_OFFLINE + TRANSFORMERS_OFFLINE passthrough (default off) to all
31 vLLM composes, driven by one OFFLINE flag. Even with local weights and a
local --model path, vLLM/transformers resolve config/tokenizer metadata
against HF unless these are set, so air-gapped boots hang/fail (#318).
OFFLINE=1 (or either var) makes vLLM use local files only.
Compose only injects env vars it explicitly lists, so a host-shell
HF_HUB_OFFLINE never reached the container before this. Default 0 = unchanged
behavior online. Documents the air-gap flow in docs/FAQ.md (incl. the gated-
drafter gotcha for *-mtp composes).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
FAQ/DUAL_CARD/HARDWARE told users to `git clone vLLM to /opt/ai/engines/vllm/primary/`
and said the dual composes mount from there — both stale. The marlin-pad patch
(vllm#40361) is now vendored in-repo (models/qwen3.6-27b/vllm/patches/vllm-marlin-pad/
{marlin.py,MPLinearKernel.py}) and overlaid into the stock vLLM image automatically by
each dual compose (delivery_mechanism: site_package_overlay). No user clone, no rig path.
Genericizes the internal path out of the public docs AND fixes the no-longer-needed
clone step in one go. (setup.sh already notes the clone is "previous design".)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Promote the cross-cutting findings out of the omni README so all clubbers
find them:
- FAQ: new "Image & video generation" section — use ComfyUI (not the LLM
stack) on a free card; Open WebUI -> ComfyUI for a unified UI; open-weight
model shortlist; + "why does my image model OOM with a small transformer?"
(the ~8-24GB text-encoder hidden cost; GGUF Q5/Q6 near-lossless).
- DUAL_CARD: "multimodal & image/video models" heads-up — size the full
pipeline (encoder, not just transformer); fp8/int8 KV reaches full context
single-card before reaching for TP/PP on PCIe; ComfyUI for image/video.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A user-track front door for testing your own model/compose without the
catalog: serve (pull.sh for safetensors / copy-a-compose for GGUF, any
engine, single or dual) → tune with the fast scripts (verify-full /
verify-stress / bench / quality-test --full|--medium) → validate with the
full rebench-full gate (optional for self-use, mandatory for a registry
contribution). Includes a tuning guide: real context ceiling, NIAH, KV
quant (asymmetric K-high/V-low), MTP/DFlash n-sweeps, batch/ubatch, A/B
discipline. Indexed in docs/README user track + a FAQ pointer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Copilot entry recommended `vllm/tools-text`, deprecated 2026-05-31 with the
rest of the Genesis single-card vLLM family. Rewrote to current reality:
- 2× 3090 → vllm/dual (262K); 1× 3090 → no dedicated ≥48K vLLM tools compose
remains, raise vllm/minimal's MAX_MODEL_LEN cautiously (single-card KV is tight,
cf. #35).
- Genesis P68/P69 greeting-stall paragraph reframed as moot — those patches
aren't in the stable v0.22.0 composes (vllm/dual, vllm/minimal).
- Added the qwen3_coder streaming-tool-call caveat (#145, open) since Copilot
streams, with the qwen3_xml workaround.
- Kept the low-max_tokens client caveat; cross-linked the new temperature entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a FAQ entry for the #232 symptom (Hermes/Cline/OpenHands returns a
one-char/empty reply + finish_reason: stop mid-tool-task). Root cause is
sampling temperature 1.0 (Qwen3.6's model-card default); our composes default
to 0.6 server-side, but a client-sent temperature wins in vLLM, so agent
harnesses inheriting 1.0 must set it in their own provider config. Captures the
fix where a confused user will look instead of reopening #232.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`switch.sh --list` showed every slug regardless of topology, so a
single-GPU box listed dual/multi configs it can't actually run — noise.
Filter `--list` to the topologies the detected GPU count supports:
1 GPU -> single only, 2 -> single+dual, 4+ -> everything. Detection
reuses switch_topology_from_gpus (CUDA/NVIDIA_VISIBLE_DEVICES, then
nvidia-smi) mapped to a topology rank.
- `--all` (and the `--list-all` alias) bypass the filter for
discoverability; --list is deferred until args are parsed so
`--list --all` works in either order.
- Don't silently hide: print a one-line note naming the detected GPU
count and exactly which topologies were hidden, plus a `(+N … hidden
— --all)` tally in the header. No note under --all / when nothing is
hidden.
- Fail-open: with no detection signal (no selector, no nvidia-smi) show
ALL — never hide off a guessed "single" fallback.
- Counts (header + per-model) reflect the visible set; PR-A health
markers/grouping and PR-B Defaults view unchanged.
New test scripts/tests/test-list-topology-filter.sh covers 1/2-GPU
filtering, --all, --list-all, order independence, the --all-without-list
guard, and fail-open. Docs: --help usage, FAQ model-switch entry, README.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(switch): model-default resolver + user-pinnable defaults
Add a two-layer default scheme on top of the existing <engine>/default map.
<engine>/default stays the maintainer's recommendation (read-only to users);
the new <model>/default is the user's preference — their .env pin if set,
else a curated pick for the detected topology.
compose_registry.py: two maintainer knobs next to DEFAULTS —
RECOMMENDED_DEFAULT_MODELS (short opt-in shortlist, not an exhaustive
ranking; new models are NOT auto-added) and ENGINE_PREFERENCE (per-topology
engine order). Plus resolver helpers (curated_default_target,
community_default_target stub, model_default_pin_key, engine_set/model_set,
slug_topology, model_of_slug).
registry-emit.sh: shared resolver model_default_target(root, model, topology)
— the single injection point for both launchers. Precedence ladder: --variant
(caller) -> .env pin -> community seam (None today) -> curated ENGINE_PREFERENCE
walk (skips non-functional (NA) slugs) -> degradation (notice + nearest-lower
topology, else a clear "pick explicitly" message; never crashes). Plus
x_default_dispatch: X/default with X in engine-set -> engine rec; X in
model-set -> model default; else error (engines + model-ids are disjoint).
switch.sh: <model>/default token; --set-default <slug> / --clear-default
<model> (round-trip the .env pin CLUB3090_DEFAULT_<MODELID>); a Defaults view
appended to --list (also standalone via --defaults) marking user-pin vs
curated. PR-A's grouping/markers/counts and the --force gate preserved.
launch.sh: bare invocation -> first installed shortlist model -> its
<model>/default (no full wizard); a pinned fast-path ("Launch your default
<slug>? [Y/n]"); a post-boot offer ("Make <slug> your default? [y/N]"). Any
narrowing flag keeps the explicit wizard path. Also load CLUB3090_DEFAULT_*
pin keys from .env even when MODEL_DIR is exported in the shell (the existing
.env loader is MODEL_DIR-gated, which would otherwise hide the pin).
Pin validation is warn + fall back, never blocking: unknown slug / wrong
model / topology-mismatch / (NA) status -> notice + curated default.
Tests: new test-model-default-resolver.sh (curated walk, (NA) skip,
degradation, X/default dispatch, pin override + all validation paths,
community seam skipped, .env round-trip); test-default-resolver.sh extended
with <model>/default launch.sh dispatch. Full suite green (only the
pre-existing test-submit-bench.sh fixture failure remains); 45 entries
unchanged; kv-calc calibration 17/17.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: document the model-default resolver + user pins
Ship the user/contributor docs alongside the resolver code (repo convention —
update existing docs, no new top-level doc).
- README: single-card realign (ik-llama = fastest blessed single default,
llama.cpp = cliff-immune alternative — matches ENGINE_PREFERENCE single
order); add "pin your default" + <model>/default to Quick start.
- CLAUDE.md (= AGENTS.md): document RECOMMENDED_DEFAULT_MODELS +
ENGINE_PREFERENCE + the shared resolver as maintainer knobs next to DEFAULTS.
- FAQ: extend "switch to a different model" with <model>/default; new "How do
I set my own default config?" Q (two-layer model, --set-default/--clear-
default/--defaults, .env key, warn+fallback validation).
- SINGLE_CARD / DUAL_CARD: the resolver + the per-topology engine order; pin
hint.
- GETTING_STARTED: first-run uses <model>/default + the in-flow "set as
default?" prompt.
- ADDING_MODELS: a new model resolves via ENGINE_PREFERENCE; add a DEFAULTS row
per engine; RECOMMENDED_DEFAULT_MODELS is not auto-grown.
- UPSTREAM: beellama Docker-image row (gates beellama onboarding; the resolver
skips it today and it auto-promotes to single default on catalog).
- models/qwen3.6-27b/CHANGELOG: dated PR-B entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tighten the 'try a different model' entry: pull.sh gates on arch support +
emittability + VRAM fit (not VRAM fit alone), and is safetensors+vLLM only.
Add an explicit Scope note — these scripts are not a universal 'run any
weights/quant' runner; GGUF/.bin and unsupported archs are evaluated/refused,
not silently run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Covers the registry list (switch.sh --list), launch/switch by slug, and the
deliberate no-auto-download behavior (setup.sh for catalog, pull.sh fit-gated
for arbitrary HF). Answers a recurring Discord question; cross-links the
existing pick-variant and pull-gate entries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the speculative "should work, untested" entries with real
numbers from cross-rig contributors:
4090: @laurimyllari Qwen3.6-35B-A3B ik --fit 205/256, 27B two-stage 82/121
5090: @apnar Gemma 4 31B forced TP=1 160/215, @efschu 27B DFlash 127/200
Adds the 4090 context-derate gotcha (~15-20% lower ceiling than headless
3090 from idle desktop/driver VRAM — observed across multiple composes).
Adds the 5090 32 GB headroom story (clears Ampere boot OOMs + boots
SM86-blocked composes like Gemma-4 26B-A4B Intel AutoRound INT4).
Removes a stale internal-path reference (/opt/ai/engines/vllm/primary)
from the 5090 entry.
Links both entries to the in-flight #246 (arch-aware launcher injection).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Links Anbeeld's KV-quant long-context benchmarks (Qwen3.6-27B / RTX 3090 —
our exact setup), with the tail-precision (99.9%-pct KLD) framing, the
quality ladder, the asymmetric-K/V takeaway, and the turbo3-not-quality-
neutral correction. Honest about our q4_0 default + the real KV_TYPE=q5_0
override for coding/agent traffic; cross-links the CLIFFS NIAH caveat.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Repo-wide follow-up to the compose quant-layer move (9821c94). The
mechanical move + registry/launch/test rewire covered the launchable
surface; a full-tree sweep found compose-path refs the move invalidated
in docs, two functional scripts, and one half-migrated mapping.
Functional fixes:
- bench-row-formatter.sh infer_compose_path(): 6 dual entries
(int8-tq3, tq3-mtp-genesis, tq3-nomtp, tq3-mtp, int8, bf16) were left
at bare dual/<file>.yml while the rest were migrated -> would emit
dead compose paths into BENCHMARKS rows.
- residency-instrument/run-instrumented-soak.sh: case->COMPOSE_FILE
paths (long-text, long-text-no-mtp, tools-text, dual default) now
resolve under <quant>/.
Docs: README layout line + tree, engine/model READMEs, patch-README
quick-recipes, diagnostics, FAQ/CLIFFS/KV_MATH/DTYPE/MULTI/SINGLE/
STRUCTURED_COT/TQ3/UPSTREAM, issue template, sglang cross-refs
(-> vllm prod path). Per-model targets: qwen-vllm->autoround-int4,
llama-cpp->unsloth-q4km, gemma defaults (bf16-mtp/fp8-mtp),
carnice->own slug dir.
Intentionally left as historical/append-only records: CHANGELOG x2,
BENCHMARKS row-labels (live paths already correct in row bodies),
calibration source: provenance citations, switch.sh/parity history
comments. Separate follow-ups: gpu-mode.sh (#417 deprecated-repo
repoint), bench-row-formatter compose_display() docker-compose.yml
branch (PR-B). Flagged pre-existing-stale: dual/int8-tq3.yml in
pr40798/pr40914 READMEs (predate this refactor; ambiguous target).
Guard tests (registry-disk, mounts-resolve, switch-parity,
launch-compat) all PASS post-edit. Leak-clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "What works on Windows" table implied club-3090 runs on native Windows
via a native llama.cpp build. Corrected: only the *upstream* llama.cpp engine
runs natively — club-3090's bash/Docker/Linux-path tooling (setup/launch/
switch, composes, bench/verify) requires WSL2 (or native Linux). Same fix
applied to the matching FAQ line.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New docs/WSL_SETUP.md walks a fresh Windows box to a running club-3090:
WSL2 + Ubuntu install, Windows NVIDIA driver / GPU passthrough, .wslconfig
RAM sizing (the silent host-RAM-starvation OOM, #32), Docker + container
toolkit, and the two gotchas not documented anywhere yet — cloning onto
ext4 (not /mnt/c) and keeping .env/scripts LF (not CRLF). Weights via
WEIGHTS=gguf + MODEL_DIR, then the normal launch.
Cross-links the existing runtime tuning (VRAM budget, TDR, expandable_segments)
in FAQ.md + HARDWARE.md rather than duplicating it; adds discovery links from
README ("What you'll need" + docs tree) and the FAQ Windows/WSL2 answer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(llamacpp): collapse single-card composes 3→2 (default = mtp alias)
Mainline Q4_K_M reaches the full 262K with -ub 512 (validated: boot + serve +
verify-stress 7/7 + soak PASS), so the old vanilla Q3_K_XL docker-compose.yml no
longer earns a separate profile. Collapse the llama.cpp single-card tree to two
compose files:
- `llamacpp/default` is now an ALIAS for `llamacpp/mtp` (mtp.yml) — launch.sh +
compose_registry repointed. All references (estate-CLI default, tests, docs)
keep resolving; the variant name survives.
- Retired models/qwen3.6-27b/llama-cpp/compose/single/docker-compose.yml.
- mtp.yml: document the -ub 512 -> 262K recipe in the header.
- mtp-vision.yml: align with the ik vision profile — default 160K (was 49K) via
-ub 512, and pin --image-min-tokens 1024 / --image-max-tokens 4096 (mainline
previously left image bounds at the model default). Validated: full-res 2048^2
image @ ~22.2 GB / 24 (~2.4 GB headroom).
- switch.sh / README / SINGLE_CARD.md: descriptions + the dead docker-compose.yml
link repointed.
Note: on mainline llama.cpp the VRAM lever is -ub (not -b — mainline honors -ub
independently, unlike ik_llama which forces n_ubatch=n_batch).
Docs follow-up (deferred): SINGLE_CARD.md prose still describes `default` as the
old Q3_K_XL path in places; mtp.yml's "froggeric suppresses --reasoning off" note
is now disproven (froggeric v19 + --reasoning off works on mainline b9246).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(engines): ik_llama coverage + MTP-merged-on-mainline corrections
- Correct stale "llama.cpp MTP via community PR / not merged" across
INFERENCE_ENGINES.md, LLAMA_CPP.md, engines/README.md — PR #22673 merged
on mainline 2026-05-16 (the b9246 image we bench is post-merge).
- Add ik_llama.cpp coverage that was missing: engine list / repo tree /
supported-models table (README), workload picker (SINGLE_CARD), choose
table (engines/README), model engines line (qwen3.6-27b/README).
- Document new ik_llama features: two-stage ngram+MTP (PR #1789), -vhad,
--recurrent-ckpt-mode, -ctk-first (IK_LLAMA.md, INFERENCE_ENGINES.md).
- IK_LLAMA gotcha: native template won the 8-pack A/B (103 vs 99, toolcall
tied) so the ik composes default native; froggeric stays vLLM-only.
- New compose iq4ks-two-stage.yml (ngram-mod + MTP via --spec-stage). Boots
+ two-stage spec-dec engages on cu13-server (validated 2026-05-22); perf
bench pending.
Review fixes on top of the drafted changes: repo-tree └→├, a 131K↔262K
self-contradiction in llama-cpp/README, a now-stale "mainline MTP still
open PR" bullet, and softened/attributed the unverified IQ5_KS PPL figures.
External PR links verified resolving; no internal paths leaked.
Docs drafted via the Qwen coding agent; reviewed, corrected, and the new
compose boot-validated by Claude.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(composes): promote native template default + bump llama-cpp context ceilings
llama-cpp/mtp.yml: default ctx 131K → 262K (full native), -ub 512
llama-cpp/mtp-vision.yml: default ctx 49K → 160K, -b/-ub 1024
ik-llama/mtp{,-vision}.yml: native template (froggeric v19 A/B'd, native
won 8-pack 103 vs 99), +'-vhad' (V-cache Hadamard), +'--recurrent-ckpt-
mode auto' (hybrid-aware MTP rejection), removed ipc:host (not needed)
Retire stale artifacts:
ik-llama/patches/froggeric-chat-template/ — native won, no longer mounted
llama-cpp/recipes/single-card-*.sh — host-binary scripts superseded by
compose files (canonical path since v0.8.0)
8-pack ship-gate: 99/150 (vs prior native 103/150) — within noise,
6/8 packs within ±1. Delta is cli-40 variance, not a flag effect
(-vhad and --recurrent-ckpt-mode are not output-quality levers).
* docs(composes): guard -np 1 with hardware-conditional rationale
Add inline ⚠ comment to all 5 single-card composes explaining why
-np 1 is intentional on a single 24 GB card (compute-bound, not
memory-bound — extra slots divide throughput, don't multiply it).
Also expose NP as an env var on the two mainline composes (was
hardcoded; ik composes already had ${NP:-1}).
The 5090/multi-GPU carve-out is explicit: 'on a higher-throughput
card or multi-GPU the trade may flip — re-validate before raising.'
Prevents future agents from blindly parallelizing slots on Ampere.
* fix(composes): move -np guard comment out of folded scalar
The ⚠ -np 1 rationale comment was inside 'command: >-' in all 5
single-card composes. YAML folded scalars treat # lines as content,
not comments — docker compose config failed on all 5 files.
Move the comment block to YAML-level (between volumes: and command:).
All 5 now pass 'docker compose -f <file> config'.
Lesson learned: always validate compose edits with
'docker compose -f <file> config >/dev/null' before committing.
* docs(FAQ): expand WSL2 section with GPU overhead guidance
The existing FAQ entry only mentioned TDR and expandable_segments
gotchas. Expanded to explain the ~1.3 GiB invisible WSL2 overhead,
how it affects each engine path differently (dual-card: noise,
single-card vLLM: one env var, single-card llama.cpp: lower ctx),
and provide a concrete VRAM budget table showing which composes
OOM on WSL2 vs which fit at defaults.
Key finding: mainline llama.cpp mtp.yml at 262K leaves only ~1.5 GB
headroom on Linux — WSL2 eats that to ~0.2 GB (instant OOM). Fix is
CTX_SIZE=131072 (drops to 20 GB, ~2.7 GB WSL2 headroom). ik_llama
composes (IQ4_KS, smaller weights) fit at defaults on WSL2.
No compose file changes — this is FAQ docs only. The llama.cpp
composes already expose CTX_SIZE as an env var override.
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
AMD hosts (Threadripper/Ryzen/EPYC) in the default AMD-Vi "Translated" IOMMU
mode can drive both GPUs into Xid 154 fatal MMU faults under sustained TP=2
DMA — surfacing as misleading tokenizers Rust segfaults / Triton "unspecified
launch failure". Fix: iommu=pt (IOMMU stays on for isolation, DMA bypasses
page-table translation). Adds a HARDWARE.md AMD-platforms section + a FAQ
troubleshooting Q with the dmesg triage line. Diagnosed by @mgabor3141.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a clean front-door to rebench-full.sh so users running OpenAI-compatible
endpoints outside our pre-baked Docker composes (llama-swap, ramalama, host-
build llama-server, ik_llama.cpp, raw vLLM) can run the canonical 5-phase
suite without script edits or env-soup.
Today the entry chain assumes a `vllm-qwen36-27b`-style container exists and
calls `docker logs` / `docker inspect` for ENGINE_KIND detection + spec-decode
metric scraping. The infra to bypass this was already 80% there (CONTAINER=none
host mode already in soak-test.sh + verify-stress.sh, docker-inspect guards
already in bench.sh line 82), but there was no top-level flag, and the silent
preflight pipefail bug below meant users hit a confusing no-op exit instead of
a useful error.
Changes:
* rebench-full.sh: add --url/--model/--engine. When --url is set we export
PREFLIGHT_NO_AUTODETECT=1 + CONTAINER=none so the chained scripts skip the
Docker scrapes (they no-op cleanly).
* rebench-full.sh: container snapshot block now wrapped in
`[[ "$CONTAINER" != "none" ]] && command -v docker` so it doesn't try to
inspect the wrong (or no) container in endpoint-first mode.
* rebench-full.sh: endpoint-not-responding error now distinguishes the --url
case (check host/port) from the default case (start a compose or pass --url).
* bench.sh: same CONTAINER=none guard for the trailing `docker logs ... grep
SpecDecoding metrics` block.
* preflight.sh: fix a pre-existing silent-exit when no container matches the
autodetect regex. `grep -E` returns 1, which under `set -euo pipefail` in
the caller killed rebench-full.sh before it reached its own endpoint-error
path. Adding `|| true` to the pipe restores the intended fall-through.
* preflight.sh: regex now also recognises sglang-qwen36-27b containers + port
30000 (added with v0.7.x sglang experimental tree, not yet auto-detected).
* docs/FAQ.md: document the new --url flag under the "Numbers from your rig"
section.
Tested:
- bash -n on all three scripts: OK
- --help renders the new endpoint-first block
- Default no-flag mode now exits cleanly with a helpful error (was silent
exit 1 pre-fix)
- --url FAKE skips autodetect and emits the external-endpoint-specific hint
- --url + non-existent host: clean error, no Docker side-effects
Motivated by ampersandru's external eval in discussion #152 — running
ik_llama via llama-swap forced him to set PREFLIGHT_NO_AUTODETECT=1 manually
and edit scripts to bypass docker assumptions. With this PR the invocation
becomes:
bash scripts/rebench-full.sh \
--url http://192.168.29.235:8887 \
--model 'Qwen3.6-27B MTP ik_llama:instruct' \
--engine llama-cpp
— and the whole 5-phase chain (bench + verify-stress + quality-full + soak
+ aider-polyglot) works against any OpenAI-API endpoint.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to commit 1b2a76c which added the speed-vs-context section to
models/qwen3.6-27b/llama-cpp/README.md. The new learning (smaller -ub
buys more ctx, especially when mmproj F16 is loaded) needs to be
discoverable from the high-level user-facing docs too — not just buried
in the model-specific README.
- SINGLE_CARD.md: corrected the "49K is the safe-headroom max with mmproj"
claim (it's the speed-optimal max; ub=512 lever pushes to 192K) +
prominent override snippet right under the profile description.
- CLIFFS.md: added a 2026-05-20 refinement note pointing out that `-ub`
is doing two jobs (cliff-survival + KV-budget), and the optimal value
is configuration-conditional on vision-on/off.
- FAQ.md: inline note in the canonical switch.sh recipe for mtp-vision
pointing users at the override.
All three back-reference the README's full sweep data as the canonical
source; surfaced by @JensJN in #170.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
From the v0.8.0 docs-fidelity test (#369) — align docs with shipped CLI:
- PULL.md Quickstart + FAQ: first-run heads-up that common archs
(Qwen2ForCausalLM &c) hard-block at needs-trust-remote-code-ack even
with --dry-run; add --trust-remote-code (after vetting the code) to
clear it. (Was a silent new-user wall.)
- PULL.md exit-codes: documented honestly — argparse usage/arg errors
exit 2 (shared with honest hard-stop); 64 is reserved, arg-parser
errors do not currently reach it (tracked CLI follow-up, #370).
- FAQ GGUF claim: "clear message" → accurate "aborts as
unsupported-format (generic message; clearer GGUF message is a
tracked v0.8.1 follow-up), not a crash".
Additive, leak-clean, links resolve, curated path untouched. Docs-only
(triggers no CI). Follows the (b) cross-link pass afe56f7.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Post-release additive pass — the pre-existing high-traffic guides didn't
know `pull` exists. All edits additive, curated path untouched (same
discipline as the README migration nudge):
- FAQ.md: new Q "model not in the supported list — can I still run it?";
GGUF Q gets a v0.8.0 note (safetensors-only eval, GGUF→llama.cpp stays
curated/manual, cross-engine deferred); launch.sh answer points
non-catalog models at `pull`.
- SINGLE_CARD / DUAL_CARD / MULTI_CARD: one blockquote cross-link each to
docs/PULL.md ("not in the configs / any HF safetensors repo — both
paths work").
- ADDING_MODELS.md: reframed catalog-onboarding vs just-run-a-model
(`pull`); the doc is the heavier calibration-catalog promotion task,
not a prerequisite for serving.
- GLOSSARY.md: new "Universal pull (v0.8.0)" table (pull, dry-run,
confidence tier, boot-fit≠runtime, calibration backbone).
Leak-clean; all links resolve on master; CommonMark structure verified
(blockquotes/headings blank-line separated). Docs-only — triggers no CI
(only tags do); lands as post-v0.8.0 polish on master.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- README.md: example block now shows model + GPUs flow with new --model /
--gpus / --tp / --pp flag examples; scripts/ layout description updated to
"model → GPUs → KV projection".
- docs/FAQ.md: rewrite the "how do I pick model + variant?" answer to
describe the new wizard (model picker → GPU picker → TP/PP auto-pick →
kv-calc projection → boot). Replace CUDA_VISIBLE_DEVICES card-selection
guidance with --gpus flag, preserving the env-var path for compatibility.
- docs/SINGLE_CARD.md, docs/DUAL_CARD.md: fix inline wizard descriptions
("asks engine + workload" / "asks GPU count + workload" → "asks model +
GPUs, projects VRAM budget").
All --variant <name> back-compat invocations across the docs continue to
work; no flag-renaming churn surfaced to users.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@tomByrer asked on disc #125 about pointing the scripts at a model
dir on a second drive. The support has always been there (MODEL_DIR
env var with 4-tier resolution: shell export, .env file, interactive
prompt, silent fallback) but the docs were too terse to discover
without running setup.sh interactively.
Two changes:
- docs/FAQ.md: expand the "where does it go?" entry from a one-liner
to a full explanation covering all 4 resolution paths, HF env-var
integration (we mount host dir INTO container cache, so HF_HOME
isn't directly respected but layout is byte-compatible), and
Windows/WSL2 path handling.
- README.md: add a 3-line parenthetical to step 2 of the quickstart
pointing at MODEL_DIR. Users who copy-paste the quickstart now see
the option without having to dig.
Both surfaces now point at the same canonical answer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds docs/DTYPE_MATRIX.md — a reference table mapping NVIDIA GPU
architectures (Pascal → Blackwell DC) to native vs emulated Tensor Core
support for each dtype + quant scheme.
Sections:
- At-a-glance compute-dtype matrix (9 archs × 10 dtypes, ✓/SW/✗ marks)
- Weight-quantization schemes (GPTQ / AWQ / AutoRound / NF4 / SmoothQuant
/ FP8 weights / MXFP8 / MXFP4 / NVFP4 / GGUF / HQQ-AQLM-SqueezeLLM)
with storage-vs-compute paths
- Weight-only vs weight+activation axis (W4A16 vs W8A8 vs W4A8)
- KV-cache dtype support (FP16 / FP8 / INT8 PTH / TQ3 / TQ4 / k8v4)
- Per-arch compose recommendations (which compose to ship for which
GPU class)
- Runtime detection (points at Genesis guards.py)
- Corner cases (Ada FP8 vs Hopper FP8, Blackwell consumer vs DC, NVFP4
vs MXFP4 block-size differences, MX* family overview, FP6)
- References (NVIDIA whitepapers, Marlin, Genesis, BENCHMARKS)
Cross-linked from:
- HARDWARE.md GPU-compat table
- GLOSSARY.md Quantization section
- FAQ.md as a new "What dtype/quant should I pick for my GPU?" Q under
Hardware
This sets the foundation for future per-arch compose optimization —
detecting compute capability at boot and picking the right KV dtype /
weight quant scheme automatically based on what the hardware actually
accelerates rather than what's nominally supported.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surfaces the canonical int8_per_token_head anti-scaling trade-off so the
next dual-3090 user who hits this finds the answer via search rather than
having to reason from the head-to-head matrix.
Frames it as:
- not a bug — per-(token, head) scale serializes at concurrency, fp8's
single global scale doesn't
- pick by workload: INT8 PTH for single-stream throughput, fp8 for
aggregate concurrency, Genesis-backed TQ3+MTP if you want both
- includes the diagnostic checklist for matching baseline numbers
(power cap, vLLM nightly, Genesis on/off, MTP n, prompt shape) so users
can self-troubleshoot a cross-rig gap before posting
Triggered by a Discord question on a dual-3090 + NVLink rig that observed
the canonical INT8 anti-scaling pattern (150 TPS single / flat at concurrency)
vs fp8 (70-100 TPS single / 400+ at concurrency).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five-format matrix across TQ3/TQ4/k8v4 + MTP all fail long-context needle
with first-word repetition, regardless of cudagraph state or PR #40914 K+1
dispatch. TQ3 with MTP disabled passes 7/7 verify-stress, so this is an
MTP × TurboQuant multi-query interaction (Sander's P67/P67b territory),
not TQ precision.
PR #40914 actively makes things worse on our stack: dropping it lifts
verify-stress from 3/7 to 5/7. Per vllm.ai/blog/turboquant the upstream
position is that hybrid-attention models (which Qwen3-Next is) are not yet
supported by TurboQuant — matches our matrix.
Updates anchor the conclusion in tree:
- docs/UPSTREAM.md: #40914 reframed as OPEN/negative on this stack;
P67/P67b called out as the only known working multi-query fix.
- docs/FAQ.md: #40914 answer updated with the 2026-05-11 local result.
- models/qwen3.6-27b/INTERNALS.md: P67 → P67/P67b, #40914 status flipped.
- vllm/patches/vllm-pr40914-k1-only/README.md: prepended "negative result"
status section; round-2/3 MTP-skip experiment annotated as ineffective.
- vllm/patches/vllm-pr40914-k1-only/.../turboquant_attn.py: epilogue now
writes through the caller's output buffer (Codex fix; left for the
re-test artifact, not deployed).
- vllm/compose/dual/tq3-mtp.yml: re-tombstoned to point at the upstream
feature-compat gap rather than a 5-PR landing list.
Deployable Genesis-free path remains dual/tq3-nomtp.yml; TQ + MTP path
remains dual/tq3-mtp-genesis.yml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The dev rig had grown across three home dirs (`/opt/ai/compose/`, `/opt/ai/github/`,
`/home/wasif/`) and three repos (single-3090, dual-3090, club-3090). Disk-out
on `/` (97% used) forced a cleanup; rather than just prune, we consolidated
the layout across the whole stack while we were at it. This commit captures
what landed inside this repo.
Services consolidation:
- services/{ollama,openwebui,litellm,qdrant,searxng}/ migrated in from
/opt/ai/compose/<svc>/ (zero functional change — same docker-compose.yml).
- services/litellm/config.yaml rewritten: explicit routes for current
primaries (qwen3.6-27b-autoround → :8010, gemma-4-31b-autoround → :8030).
Removed `* → ollama/*` wildcard.
- services/comfyui/ migrated in (was /opt/ai/compose/comfyui/) — wired into
gpu-mode with full mutex against vLLM/SGLang.
scripts/gpu-mode.sh under git:
- Was a loose /opt/ai/gpu-mode.sh outside any repo. Symlinked at
/usr/local/bin/gpu-mode.
- Five Gemma 4 31B modes added: gemma, gemma-dflash, gemma-int8,
gemma-dflash-int8, gemma-awq.
- One ComfyUI mode (mutex with all LLM serving).
- prune / prune-all subcommands (safe image prune; aggressive variant adds
build cache --keep-storage 5GB + dangling networks).
- gpu-mode status now shows Docker disk + /var/lib/docker + /tmp sizes.
- compose_at() passes --env-file <repo>/.env so MODEL_DIR resolves
regardless of which compose dir gpu-mode cd's into. Fixes the recurring
"MODEL_DIR not set, defaulting to ../../../../../models-cache" warning.
- stderr no longer swallowed by compose_at() (real errors surface).
- Cross-model VRAM mutex: every Qwen mode stop_all_gemma + stop_comfyui
and vice-versa.
scripts/maintenance/ — new hygiene-tools subdir:
- list-image-pins.sh: engine-agnostic pin auditor. Scans every compose's
`image:` line, groups by `<repo>:<tag>`, flags pin-drift (multiple tags
per repo), ranks composes by patch surface.
Pin tracking:
- docs/UPSTREAM.md gains a "Pinned images" section: table of every pinned
image, why each pin exists, retirement candidate criteria.
- docs/NIGHTLY_BUMP_RUNBOOK.md (new): 7-step procedure for bumping pinned
engine images (scope → branch → patch survival → boot → verify-full +
verify-stress → bench delta → land → retire). Engine-specific notes
for vLLM nightly hashes, llama.cpp digest pinning, SGLang variants.
Path updates from the engine + model dir consolidation:
- /opt/ai/vllm-src/ → /opt/ai/engines/vllm/primary/
(in setup.sh, INTERNALS.md, several patch READMEs, docs/HARDWARE.md,
docs/FAQ.md, docs/DUAL_CARD.md, docs/UPSTREAM.md, models/qwen3.6-27b/
CHANGELOG.md)
- /mnt/models/gguf/qwen3.6-27b/ → /mnt/models/huggingface/qwen3.6-27b-gguf/
(in models/qwen3.6-27b/llama-cpp/{compose/single/*.yml, recipes/*.sh,
README.md}, docs/engines/LLAMA_CPP.md)
CHANGELOG.md narrative gap fill:
- 2026-05-10 entry for this reorg.
- 2026-05-09 entry for compose convention formalization (topology
promoted to dir level, profile schema, Status enum + Caveats, cliff
CI swap, Discord launch).
- 2026-05-08 entry for Gemma 4 INT8 PTH unblock + 262K validation.
- 2026-05-07 entry for power-cap-sweep campaign + HARDWARE.md cross-rig
charts + cross-rig benchmark rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A single-card RTX 3090 Ti rig on WSL2 (driver 596.36) hits
RuntimeError: CUDA driver error: device not ready from gptq_marlin_repack
immediately after weight load on the v7.72.2-uplift nightly pin.
Bisect ruled out Genesis, spec-decode, TQ3 KV, async-residual error,
and TDR (registry already extended + Windows rebooted).
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False resolves it. Same
env-var workaround as JusefPol's NVLink boot-crash (PR #31), already
hardcoded in the dual-nvlink*.yml composes.
Replaces the hardcoded PYTORCH_CUDA_ALLOC_CONF line in 14 single-card
and PCIe dual-card composes with a ${PYTORCH_CUDA_ALLOC_CONF:-...}
override (defaults preserved). Pattern matches existing MAX_MODEL_LEN /
GPU_MEMORY_UTILIZATION overrides from #79. The two dual-nvlink*.yml
composes are unchanged — their existing JusefPol-driven default already
has expandable_segments off.
Documentation:
- docs/HARDWARE.md: new "disable PyTorch expandable_segments" subsection
alongside the TDR fix, with stack trace, what was ruled out, override
recipe, and a single uncontrolled observation about weight-load time
(32 sec → 13 sec).
- docs/FAQ.md: WSL2 question now cross-links both the TDR and
expandable_segments fix subsections.
- .env.example: documents the override under "vLLM tuning knobs".
- CHANGELOG.md (top-level + per-model): dated 2026-05-06 entries.
The exact failing call hasn't been isolated. The cuMemMap virtual-memory
API used by expandable_segments:True is the suspected culprit since
both known occurrences respond to the same workaround, but no specific
call has been proven to return cudaErrorNotReady.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After shipping report.sh --full / --soak / --stress flags in 8a29b95, the
bug report flow + troubleshooting guidance still pointed at older patterns.
Updated:
- .github/ISSUE_TEMPLATE/bug-report.yml — replaced single-flag suggestion
with a "pick the flag that matches your bug" decision table:
- boot crash / wrong output / tool-call regression → --verify (~2 min)
- OOM mid-conversation / agentic cliff → --soak (~25 min) — only test
that catches Cliff 2b
- TPS regression / cross-rig perf → --bench (~5 min)
- not sure / capture everything → --full (~35 min)
- docs/FAQ.md "Found a bug — what should I include?" — replaced the generic
"verify-full.sh output" note with the same flag decision table; updated
the troubleshooting-ladder pointers to use --verify (boot path) and
--full (multi-card TQ3+Genesis intersection bugs).
- docs/MULTI_CARD.md cross-rig contribution callouts (×2) — recommended
command upgraded from --bench to --full (or --bench fallback if soak
time-budget is tight, with explicit caveat that --bench skips Cliff 2b).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Today's full validation matrix exposed Cliff 2b — DeltaNet GDN forward
fires under accumulated multi-turn context (~21-26K), not just at single
prompts >50-60K. All 6 single-card vLLM variants FAIL v2 continuous;
only vllm/dual (TP=2) and llamacpp/default survive cleanly. Three issues
filed today (#41, #42, #43) all map to the same class. Docs needed to
reflect this so users don't keep landing on long-text/long-vision/
tools-text and burning time tuning what won't fix.
docs/SINGLE_CARD.md
- New ⚠️ section at top: single-card vLLM unsafe for hermes/openhands/
OpenCode/Cline/OpenClaw/Aider/Cursor with retained context. Routing
tree to dual.yml or llamacpp/default.
- Existing "One limitation" split into Cliff 2a (single-prompt, mostly
closed v7.69) and Cliff 2b (multi-turn, NOT closed). Reasons each
can/can't be tuned at config layer.
docs/CLIFFS.md
- TL;DR table extended from 2 cliffs to 3 (Cliff 2a + 2b separated).
- New section "Why TP=2 escapes" — per-card head sharding halves the
GDN live-tensor sizes (48 MiB → 24 MiB on v/u/o etc., 97 → 49 on h).
Concrete byte math; total per-card live FLA set drops from ~500 MiB
to ~250 MiB. Validated 0 MiB growth on dual.yml v2 continuous.
- New section "Why llama.cpp escapes" — three concrete differences:
(1) different GDN kernel (own CUDA, smaller per-step working buffers
vs FLA Triton), (2) ggml manual allocator (no PyTorch caching layer
fragmentation), (3) no JIT/Triton autotune (pre-compiled, static
layout from boot). Trade ~3× decode speed for cliff-immunity.
docs/FAQ.md
- New troubleshooting entry: "My hermes / openhands / OpenCode / Cline /
OpenClaw / Cursor session OOMs after a few turns. What do I do?"
Lists every dead-end we tested today so users don't repeat them:
mem-util tuning, MTP-off, max-num-batched-tokens (Mamba block_size
floor blocks <4128), TRITON_CACHE_AUTOTUNING (Blackwell-only recovery),
expandable_segments (already on), empty_cache (reclaims but cliff
fires next turn). Routing answer + soak-test repro command.
CHANGELOG.md
- Dated entry above the soak-test v2 entry summarizing the docs sweep.
Codex residency pilot data backs every claim:
results/residency-20260503-codex-pilot3/ (initial)
results/residency-20260503-empty-cache-idle/ (empty_cache experiment)
Investigation memo: docs/diagnostics/cliff2-handoff-results.md (gitignored).
Companion docs update for 43fe2a4. Three places where users would expect
to find this:
- README.md — adds step 7 to quick-start ("Keep your install up-to-date")
with the upgrade flow + the soft-warn note. Updates the directory tree
to list update.sh and refresh the preflight.sh one-line summary.
- docs/FAQ.md — new "How do I keep my install up-to-date?" entry under
Setup, covering the dirty-tree refusal, --dry-run / --force flags, and
the cross-link to the existing Genesis-pin warning. Also tweaked the
"How do I bump Genesis" entry to point at update.sh as the normal path
and frame manual bumps as the testing-only escape hatch.
- CHANGELOG.md — 2026-05-03 entry describing both pieces (preflight_repo_drift
+ scripts/update.sh) and the JusefPol dual-nvlink variant landing.
[@RossNE99]'s case in #32 surfaced a failure mode that wasn't in the
FAQ: vLLM throws a misleading "GPU OOM, tried to allocate 44 MiB"
error at model load when the actual problem is WSL2's default RAM
allocation (50% of Windows host) being insufficient for the 17.69 GiB
checkpoint.
The smoking-gun log line is `[weight_utils.py:934] Auto-prefetch is
disabled because ... checkpoint size (17.69 GiB) exceeds 90% of
available RAM` — surfaced automatically by report.sh now, but not
otherwise visible in standard triage output.
Adds a troubleshooting entry pointing at the .wslconfig fix
(memory=24GB minimum, swap=8GB), with the diagnostic log line so users
can self-identify if they suspect this. Also cross-links the original
RossNE99 repro thread for context.
Was invisible until report.sh captured the boot log highlights — good
worked example of why the standardized rig dump is more useful than
ad-hoc nvidia-smi pastes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related improvements after [@noonghunna]'s observation that the
ladder we just added (commit 9560efd) was structurally OK but easy to
miss:
1. **FAQ ladder gets an at-a-glance table** at the top of
"Before symptom-matching" section. Five rows, one per step, showing
variant name + what each step adds + what it tests. Plus a one-line
path-finder ("if single-card, run 1-3; if dual, layer-by-layer
tells you which intersection breaks"). Readers no longer have to
scroll through 80 lines before they see the full ladder shape.
2. **Bug report template leads with the ladder** instead of jumping
straight to the report.sh ask. New intro:
"Before filing — try the 5-step triage ladder first"
Links into the FAQ section anchor. Acknowledges that "a lot of
'should I file a bug' questions resolve at step 1 or 2 (often
re-running setup.sh is the fix)." Worth 15 min before opening an
issue. The report.sh paste field stays — for users who've done the
ladder and have a real bug to file.
3. **Issue chooser config gets a third option** above
"General Q&A / discussion":
"Troubleshooting — try the 5-step triage ladder first"
Clicking "Open an issue" → users now see Troubleshooting / Q&A /
Bug-report / Bench-contribution as four distinct paths, with the
ladder explicitly named as a self-help option BEFORE the bug-report
template appears.
Result: users with budget / boot / MTP-class issues get pointed at
the ladder twice (once in the chooser, once in the bug template intro)
before they ever fill out the form. Reduces the "filed a bug that
turned out to be a partial-pull / setup-not-rerun / config-too-tight
issue" pattern that's eaten the last few triage rounds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When users hit boot OOMs, MTP weirdness, or TQ3/long-context budget
issues, our previous troubleshooting flow jumped straight to
symptom-matching. That misses the systematic narrowing pass: validate
the simplest stack works first, then add one variable per step until
the failing layer is isolated.
New leading section in Troubleshooting:
- Step 1 — `vllm/minimal` (32K + fp8, no Genesis, no spec-decode):
validates hardware, driver, Docker, NVIDIA Container Toolkit, model
files, base vLLM. Strips out every layer that could be the cause.
- Step 2 — `vllm/tools-text` (75K + fp8 + MTP + Genesis): adds Genesis
+ MTP K=3. Still fp8 KV (no TQ3 yet). Most common failure here is
GENESIS_PIN-vs-tree mismatch — re-run setup.sh.
- Step 3 — `vllm/long-text` (180K + TQ3 + MTP + full Genesis): adds
TurboQuant 3-bit KV + long-context. The production-target single-card
config. Failure here narrows to TQ3 setup, GDN >60K hardware wall,
or Cliff 1 mech B (closed since v7.69 PN25).
- Step 4 — `vllm/dual` (262K + fp8 + TP=2 + 2 streams, Genesis-less):
adds TP=2 NCCL + multi-GPU memory split. Crucially removes Genesis,
since dual.yml is intentionally Genesis-less. Failure here despite
step 3 working narrows to TP=2 NCCL specifically. WSL2 is the most
common trigger (its vGPU layer adds memory accounting wrinkles that
bare-metal Linux doesn't have).
- Step 5 — `vllm/dual-turbo` (262K + TQ3 + TP=2 + Genesis): the full
multi-card stack. Failure here despite step 4 narrows to the
TQ3-on-TP=2-with-Genesis intersection.
The ladder works for both single and dual-card users because steps 1-3
isolate stack layers regardless of GPU count, and steps 4-5 add TP=2
surface separately. A dual-card user hitting issues should still run
steps 1-3 on a single card first — it's the only way to tell apart
"single-card stack issue that also breaks dual" vs "TP=2 NCCL specific."
Existing symptom-pattern-matching list demoted to "Quick recognition
guide" sub-section, kept verbatim — still useful for users who already
know the failure surface and want to skip the ladder.
Going forward we'll point to this ladder explicitly in triage replies
on issues / discussions, replacing the ad-hoc "have you tried X?" pattern.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes / addresses 3 reported issues + adds requested feature:
#7 vid (PORT not honored, MODEL_DIR vs MODELS_DIR confusion):
- All 8 vLLM compose files now use "${PORT:-XXXX}:8000" so .env PORT
flows through. Defaults preserved per-variant (8020 single, 8010-8013
dual). llama.cpp composes already had this pattern.
- scripts/switch.sh: load .env early; per-variant default-port table;
new resolve_ready_url() picks PORT > variant default for the readiness
probe.
- scripts/launch.sh: same default-port table; final endpoint URL printed
to user reflects actual mapped port.
- .env.example: ⚠ box callout that variable names are CASE-SENSITIVE
(MODEL_DIR singular, NOT MODELS_DIR plural — silently ignored).
New PORT section documenting per-variant defaults.
#4 timxx (tools-text.yml fails "Free memory ... less than desired"):
- docs/FAQ.md: new entry "Container fails to start: Free memory..."
explaining the vLLM startup check, the two workarounds (free VRAM /
lower mem-util), and which configs hit it most often (0.97+ mem-util).
- Compose defaults unchanged (0.97 stays the right design target on
headless rigs); the FAQ documents the workaround for users with X11.
#1 fabriciomalta (per-config VRAM column):
- docs/SINGLE_CARD.md: TL;DR table now has VRAM column with mem-util.
- docs/DUAL_CARD.md: TL;DR table same + footnote explaining per-card
semantics and which dual configs would/wouldn't fit on 2× 20 GB cards
(relevant to fabriciomalta's 2× 3080-20GB use case).
#2 tenitram (empty responses) — fixed in master via aab8ff4
(P68/P69 disabled). Closed with reply pointing at the fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
