Adds a FAQ entry for the #232 symptom (Hermes/Cline/OpenHands returns a
one-char/empty reply + finish_reason: stop mid-tool-task). Root cause is
sampling temperature 1.0 (Qwen3.6's model-card default); our composes default
to 0.6 server-side, but a client-sent temperature wins in vLLM, so agent
harnesses inheriting 1.0 must set it in their own provider config. Captures the
fix where a confused user will look instead of reopening #232.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`switch.sh --list` showed every slug regardless of topology, so a
single-GPU box listed dual/multi configs it can't actually run — noise.
Filter `--list` to the topologies the detected GPU count supports:
1 GPU -> single only, 2 -> single+dual, 4+ -> everything. Detection
reuses switch_topology_from_gpus (CUDA/NVIDIA_VISIBLE_DEVICES, then
nvidia-smi) mapped to a topology rank.
- `--all` (and the `--list-all` alias) bypass the filter for
discoverability; --list is deferred until args are parsed so
`--list --all` works in either order.
- Don't silently hide: print a one-line note naming the detected GPU
count and exactly which topologies were hidden, plus a `(+N … hidden
— --all)` tally in the header. No note under --all / when nothing is
hidden.
- Fail-open: with no detection signal (no selector, no nvidia-smi) show
ALL — never hide off a guessed "single" fallback.
- Counts (header + per-model) reflect the visible set; PR-A health
markers/grouping and PR-B Defaults view unchanged.
New test scripts/tests/test-list-topology-filter.sh covers 1/2-GPU
filtering, --all, --list-all, order independence, the --all-without-list
guard, and fail-open. Docs: --help usage, FAQ model-switch entry, README.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(switch): model-default resolver + user-pinnable defaults
Add a two-layer default scheme on top of the existing <engine>/default map.
<engine>/default stays the maintainer's recommendation (read-only to users);
the new <model>/default is the user's preference — their .env pin if set,
else a curated pick for the detected topology.
compose_registry.py: two maintainer knobs next to DEFAULTS —
RECOMMENDED_DEFAULT_MODELS (short opt-in shortlist, not an exhaustive
ranking; new models are NOT auto-added) and ENGINE_PREFERENCE (per-topology
engine order). Plus resolver helpers (curated_default_target,
community_default_target stub, model_default_pin_key, engine_set/model_set,
slug_topology, model_of_slug).
registry-emit.sh: shared resolver model_default_target(root, model, topology)
— the single injection point for both launchers. Precedence ladder: --variant
(caller) -> .env pin -> community seam (None today) -> curated ENGINE_PREFERENCE
walk (skips non-functional (NA) slugs) -> degradation (notice + nearest-lower
topology, else a clear "pick explicitly" message; never crashes). Plus
x_default_dispatch: X/default with X in engine-set -> engine rec; X in
model-set -> model default; else error (engines + model-ids are disjoint).
switch.sh: <model>/default token; --set-default <slug> / --clear-default
<model> (round-trip the .env pin CLUB3090_DEFAULT_<MODELID>); a Defaults view
appended to --list (also standalone via --defaults) marking user-pin vs
curated. PR-A's grouping/markers/counts and the --force gate preserved.
launch.sh: bare invocation -> first installed shortlist model -> its
<model>/default (no full wizard); a pinned fast-path ("Launch your default
<slug>? [Y/n]"); a post-boot offer ("Make <slug> your default? [y/N]"). Any
narrowing flag keeps the explicit wizard path. Also load CLUB3090_DEFAULT_*
pin keys from .env even when MODEL_DIR is exported in the shell (the existing
.env loader is MODEL_DIR-gated, which would otherwise hide the pin).
Pin validation is warn + fall back, never blocking: unknown slug / wrong
model / topology-mismatch / (NA) status -> notice + curated default.
Tests: new test-model-default-resolver.sh (curated walk, (NA) skip,
degradation, X/default dispatch, pin override + all validation paths,
community seam skipped, .env round-trip); test-default-resolver.sh extended
with <model>/default launch.sh dispatch. Full suite green (only the
pre-existing test-submit-bench.sh fixture failure remains); 45 entries
unchanged; kv-calc calibration 17/17.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: document the model-default resolver + user pins
Ship the user/contributor docs alongside the resolver code (repo convention —
update existing docs, no new top-level doc).
- README: single-card realign (ik-llama = fastest blessed single default,
llama.cpp = cliff-immune alternative — matches ENGINE_PREFERENCE single
order); add "pin your default" + <model>/default to Quick start.
- CLAUDE.md (= AGENTS.md): document RECOMMENDED_DEFAULT_MODELS +
ENGINE_PREFERENCE + the shared resolver as maintainer knobs next to DEFAULTS.
- FAQ: extend "switch to a different model" with <model>/default; new "How do
I set my own default config?" Q (two-layer model, --set-default/--clear-
default/--defaults, .env key, warn+fallback validation).
- SINGLE_CARD / DUAL_CARD: the resolver + the per-topology engine order; pin
hint.
- GETTING_STARTED: first-run uses <model>/default + the in-flow "set as
default?" prompt.
- ADDING_MODELS: a new model resolves via ENGINE_PREFERENCE; add a DEFAULTS row
per engine; RECOMMENDED_DEFAULT_MODELS is not auto-grown.
- UPSTREAM: beellama Docker-image row (gates beellama onboarding; the resolver
skips it today and it auto-promotes to single default on catalog).
- models/qwen3.6-27b/CHANGELOG: dated PR-B entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tighten the 'try a different model' entry: pull.sh gates on arch support +
emittability + VRAM fit (not VRAM fit alone), and is safetensors+vLLM only.
Add an explicit Scope note — these scripts are not a universal 'run any
weights/quant' runner; GGUF/.bin and unsupported archs are evaluated/refused,
not silently run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Covers the registry list (switch.sh --list), launch/switch by slug, and the
deliberate no-auto-download behavior (setup.sh for catalog, pull.sh fit-gated
for arbitrary HF). Answers a recurring Discord question; cross-links the
existing pick-variant and pull-gate entries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the speculative "should work, untested" entries with real
numbers from cross-rig contributors:
4090: @laurimyllari Qwen3.6-35B-A3B ik --fit 205/256, 27B two-stage 82/121
5090: @apnar Gemma 4 31B forced TP=1 160/215, @efschu 27B DFlash 127/200
Adds the 4090 context-derate gotcha (~15-20% lower ceiling than headless
3090 from idle desktop/driver VRAM — observed across multiple composes).
Adds the 5090 32 GB headroom story (clears Ampere boot OOMs + boots
SM86-blocked composes like Gemma-4 26B-A4B Intel AutoRound INT4).
Removes a stale internal-path reference (/opt/ai/engines/vllm/primary)
from the 5090 entry.
Links both entries to the in-flight #246 (arch-aware launcher injection).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Links Anbeeld's KV-quant long-context benchmarks (Qwen3.6-27B / RTX 3090 —
our exact setup), with the tail-precision (99.9%-pct KLD) framing, the
quality ladder, the asymmetric-K/V takeaway, and the turbo3-not-quality-
neutral correction. Honest about our q4_0 default + the real KV_TYPE=q5_0
override for coding/agent traffic; cross-links the CLIFFS NIAH caveat.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Repo-wide follow-up to the compose quant-layer move (9821c94). The
mechanical move + registry/launch/test rewire covered the launchable
surface; a full-tree sweep found compose-path refs the move invalidated
in docs, two functional scripts, and one half-migrated mapping.
Functional fixes:
- bench-row-formatter.sh infer_compose_path(): 6 dual entries
(int8-tq3, tq3-mtp-genesis, tq3-nomtp, tq3-mtp, int8, bf16) were left
at bare dual/<file>.yml while the rest were migrated -> would emit
dead compose paths into BENCHMARKS rows.
- residency-instrument/run-instrumented-soak.sh: case->COMPOSE_FILE
paths (long-text, long-text-no-mtp, tools-text, dual default) now
resolve under <quant>/.
Docs: README layout line + tree, engine/model READMEs, patch-README
quick-recipes, diagnostics, FAQ/CLIFFS/KV_MATH/DTYPE/MULTI/SINGLE/
STRUCTURED_COT/TQ3/UPSTREAM, issue template, sglang cross-refs
(-> vllm prod path). Per-model targets: qwen-vllm->autoround-int4,
llama-cpp->unsloth-q4km, gemma defaults (bf16-mtp/fp8-mtp),
carnice->own slug dir.
Intentionally left as historical/append-only records: CHANGELOG x2,
BENCHMARKS row-labels (live paths already correct in row bodies),
calibration source: provenance citations, switch.sh/parity history
comments. Separate follow-ups: gpu-mode.sh (#417 deprecated-repo
repoint), bench-row-formatter compose_display() docker-compose.yml
branch (PR-B). Flagged pre-existing-stale: dual/int8-tq3.yml in
pr40798/pr40914 READMEs (predate this refactor; ambiguous target).
Guard tests (registry-disk, mounts-resolve, switch-parity,
launch-compat) all PASS post-edit. Leak-clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "What works on Windows" table implied club-3090 runs on native Windows
via a native llama.cpp build. Corrected: only the *upstream* llama.cpp engine
runs natively — club-3090's bash/Docker/Linux-path tooling (setup/launch/
switch, composes, bench/verify) requires WSL2 (or native Linux). Same fix
applied to the matching FAQ line.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New docs/WSL_SETUP.md walks a fresh Windows box to a running club-3090:
WSL2 + Ubuntu install, Windows NVIDIA driver / GPU passthrough, .wslconfig
RAM sizing (the silent host-RAM-starvation OOM, #32), Docker + container
toolkit, and the two gotchas not documented anywhere yet — cloning onto
ext4 (not /mnt/c) and keeping .env/scripts LF (not CRLF). Weights via
WEIGHTS=gguf + MODEL_DIR, then the normal launch.
Cross-links the existing runtime tuning (VRAM budget, TDR, expandable_segments)
in FAQ.md + HARDWARE.md rather than duplicating it; adds discovery links from
README ("What you'll need" + docs tree) and the FAQ Windows/WSL2 answer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(llamacpp): collapse single-card composes 3→2 (default = mtp alias)
Mainline Q4_K_M reaches the full 262K with -ub 512 (validated: boot + serve +
verify-stress 7/7 + soak PASS), so the old vanilla Q3_K_XL docker-compose.yml no
longer earns a separate profile. Collapse the llama.cpp single-card tree to two
compose files:
- `llamacpp/default` is now an ALIAS for `llamacpp/mtp` (mtp.yml) — launch.sh +
compose_registry repointed. All references (estate-CLI default, tests, docs)
keep resolving; the variant name survives.
- Retired models/qwen3.6-27b/llama-cpp/compose/single/docker-compose.yml.
- mtp.yml: document the -ub 512 -> 262K recipe in the header.
- mtp-vision.yml: align with the ik vision profile — default 160K (was 49K) via
-ub 512, and pin --image-min-tokens 1024 / --image-max-tokens 4096 (mainline
previously left image bounds at the model default). Validated: full-res 2048^2
image @ ~22.2 GB / 24 (~2.4 GB headroom).
- switch.sh / README / SINGLE_CARD.md: descriptions + the dead docker-compose.yml
link repointed.
Note: on mainline llama.cpp the VRAM lever is -ub (not -b — mainline honors -ub
independently, unlike ik_llama which forces n_ubatch=n_batch).
Docs follow-up (deferred): SINGLE_CARD.md prose still describes `default` as the
old Q3_K_XL path in places; mtp.yml's "froggeric suppresses --reasoning off" note
is now disproven (froggeric v19 + --reasoning off works on mainline b9246).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(engines): ik_llama coverage + MTP-merged-on-mainline corrections
- Correct stale "llama.cpp MTP via community PR / not merged" across
INFERENCE_ENGINES.md, LLAMA_CPP.md, engines/README.md — PR #22673 merged
on mainline 2026-05-16 (the b9246 image we bench is post-merge).
- Add ik_llama.cpp coverage that was missing: engine list / repo tree /
supported-models table (README), workload picker (SINGLE_CARD), choose
table (engines/README), model engines line (qwen3.6-27b/README).
- Document new ik_llama features: two-stage ngram+MTP (PR #1789), -vhad,
--recurrent-ckpt-mode, -ctk-first (IK_LLAMA.md, INFERENCE_ENGINES.md).
- IK_LLAMA gotcha: native template won the 8-pack A/B (103 vs 99, toolcall
tied) so the ik composes default native; froggeric stays vLLM-only.
- New compose iq4ks-two-stage.yml (ngram-mod + MTP via --spec-stage). Boots
+ two-stage spec-dec engages on cu13-server (validated 2026-05-22); perf
bench pending.
Review fixes on top of the drafted changes: repo-tree └→├, a 131K↔262K
self-contradiction in llama-cpp/README, a now-stale "mainline MTP still
open PR" bullet, and softened/attributed the unverified IQ5_KS PPL figures.
External PR links verified resolving; no internal paths leaked.
Docs drafted via the Qwen coding agent; reviewed, corrected, and the new
compose boot-validated by Claude.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(composes): promote native template default + bump llama-cpp context ceilings
llama-cpp/mtp.yml: default ctx 131K → 262K (full native), -ub 512
llama-cpp/mtp-vision.yml: default ctx 49K → 160K, -b/-ub 1024
ik-llama/mtp{,-vision}.yml: native template (froggeric v19 A/B'd, native
won 8-pack 103 vs 99), +'-vhad' (V-cache Hadamard), +'--recurrent-ckpt-
mode auto' (hybrid-aware MTP rejection), removed ipc:host (not needed)
Retire stale artifacts:
ik-llama/patches/froggeric-chat-template/ — native won, no longer mounted
llama-cpp/recipes/single-card-*.sh — host-binary scripts superseded by
compose files (canonical path since v0.8.0)
8-pack ship-gate: 99/150 (vs prior native 103/150) — within noise,
6/8 packs within ±1. Delta is cli-40 variance, not a flag effect
(-vhad and --recurrent-ckpt-mode are not output-quality levers).
* docs(composes): guard -np 1 with hardware-conditional rationale
Add inline ⚠ comment to all 5 single-card composes explaining why
-np 1 is intentional on a single 24 GB card (compute-bound, not
memory-bound — extra slots divide throughput, don't multiply it).
Also expose NP as an env var on the two mainline composes (was
hardcoded; ik composes already had ${NP:-1}).
The 5090/multi-GPU carve-out is explicit: 'on a higher-throughput
card or multi-GPU the trade may flip — re-validate before raising.'
Prevents future agents from blindly parallelizing slots on Ampere.
* fix(composes): move -np guard comment out of folded scalar
The ⚠ -np 1 rationale comment was inside 'command: >-' in all 5
single-card composes. YAML folded scalars treat # lines as content,
not comments — docker compose config failed on all 5 files.
Move the comment block to YAML-level (between volumes: and command:).
All 5 now pass 'docker compose -f <file> config'.
Lesson learned: always validate compose edits with
'docker compose -f <file> config >/dev/null' before committing.
* docs(FAQ): expand WSL2 section with GPU overhead guidance
The existing FAQ entry only mentioned TDR and expandable_segments
gotchas. Expanded to explain the ~1.3 GiB invisible WSL2 overhead,
how it affects each engine path differently (dual-card: noise,
single-card vLLM: one env var, single-card llama.cpp: lower ctx),
and provide a concrete VRAM budget table showing which composes
OOM on WSL2 vs which fit at defaults.
Key finding: mainline llama.cpp mtp.yml at 262K leaves only ~1.5 GB
headroom on Linux — WSL2 eats that to ~0.2 GB (instant OOM). Fix is
CTX_SIZE=131072 (drops to 20 GB, ~2.7 GB WSL2 headroom). ik_llama
composes (IQ4_KS, smaller weights) fit at defaults on WSL2.
No compose file changes — this is FAQ docs only. The llama.cpp
composes already expose CTX_SIZE as an env var override.
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
AMD hosts (Threadripper/Ryzen/EPYC) in the default AMD-Vi "Translated" IOMMU
mode can drive both GPUs into Xid 154 fatal MMU faults under sustained TP=2
DMA — surfacing as misleading tokenizers Rust segfaults / Triton "unspecified
launch failure". Fix: iommu=pt (IOMMU stays on for isolation, DMA bypasses
page-table translation). Adds a HARDWARE.md AMD-platforms section + a FAQ
troubleshooting Q with the dmesg triage line. Diagnosed by @mgabor3141.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a clean front-door to rebench-full.sh so users running OpenAI-compatible
endpoints outside our pre-baked Docker composes (llama-swap, ramalama, host-
build llama-server, ik_llama.cpp, raw vLLM) can run the canonical 5-phase
suite without script edits or env-soup.
Today the entry chain assumes a `vllm-qwen36-27b`-style container exists and
calls `docker logs` / `docker inspect` for ENGINE_KIND detection + spec-decode
metric scraping. The infra to bypass this was already 80% there (CONTAINER=none
host mode already in soak-test.sh + verify-stress.sh, docker-inspect guards
already in bench.sh line 82), but there was no top-level flag, and the silent
preflight pipefail bug below meant users hit a confusing no-op exit instead of
a useful error.
Changes:
* rebench-full.sh: add --url/--model/--engine. When --url is set we export
PREFLIGHT_NO_AUTODETECT=1 + CONTAINER=none so the chained scripts skip the
Docker scrapes (they no-op cleanly).
* rebench-full.sh: container snapshot block now wrapped in
`[[ "$CONTAINER" != "none" ]] && command -v docker` so it doesn't try to
inspect the wrong (or no) container in endpoint-first mode.
* rebench-full.sh: endpoint-not-responding error now distinguishes the --url
case (check host/port) from the default case (start a compose or pass --url).
* bench.sh: same CONTAINER=none guard for the trailing `docker logs ... grep
SpecDecoding metrics` block.
* preflight.sh: fix a pre-existing silent-exit when no container matches the
autodetect regex. `grep -E` returns 1, which under `set -euo pipefail` in
the caller killed rebench-full.sh before it reached its own endpoint-error
path. Adding `|| true` to the pipe restores the intended fall-through.
* preflight.sh: regex now also recognises sglang-qwen36-27b containers + port
30000 (added with v0.7.x sglang experimental tree, not yet auto-detected).
* docs/FAQ.md: document the new --url flag under the "Numbers from your rig"
section.
Tested:
- bash -n on all three scripts: OK
- --help renders the new endpoint-first block
- Default no-flag mode now exits cleanly with a helpful error (was silent
exit 1 pre-fix)
- --url FAKE skips autodetect and emits the external-endpoint-specific hint
- --url + non-existent host: clean error, no Docker side-effects
Motivated by ampersandru's external eval in discussion #152 — running
ik_llama via llama-swap forced him to set PREFLIGHT_NO_AUTODETECT=1 manually
and edit scripts to bypass docker assumptions. With this PR the invocation
becomes:
bash scripts/rebench-full.sh \
--url http://192.168.29.235:8887 \
--model 'Qwen3.6-27B MTP ik_llama:instruct' \
--engine llama-cpp
— and the whole 5-phase chain (bench + verify-stress + quality-full + soak
+ aider-polyglot) works against any OpenAI-API endpoint.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to commit 1b2a76c which added the speed-vs-context section to
models/qwen3.6-27b/llama-cpp/README.md. The new learning (smaller -ub
buys more ctx, especially when mmproj F16 is loaded) needs to be
discoverable from the high-level user-facing docs too — not just buried
in the model-specific README.
- SINGLE_CARD.md: corrected the "49K is the safe-headroom max with mmproj"
claim (it's the speed-optimal max; ub=512 lever pushes to 192K) +
prominent override snippet right under the profile description.
- CLIFFS.md: added a 2026-05-20 refinement note pointing out that `-ub`
is doing two jobs (cliff-survival + KV-budget), and the optimal value
is configuration-conditional on vision-on/off.
- FAQ.md: inline note in the canonical switch.sh recipe for mtp-vision
pointing users at the override.
All three back-reference the README's full sweep data as the canonical
source; surfaced by @JensJN in #170.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
From the v0.8.0 docs-fidelity test (#369) — align docs with shipped CLI:
- PULL.md Quickstart + FAQ: first-run heads-up that common archs
(Qwen2ForCausalLM &c) hard-block at needs-trust-remote-code-ack even
with --dry-run; add --trust-remote-code (after vetting the code) to
clear it. (Was a silent new-user wall.)
- PULL.md exit-codes: documented honestly — argparse usage/arg errors
exit 2 (shared with honest hard-stop); 64 is reserved, arg-parser
errors do not currently reach it (tracked CLI follow-up, #370).
- FAQ GGUF claim: "clear message" → accurate "aborts as
unsupported-format (generic message; clearer GGUF message is a
tracked v0.8.1 follow-up), not a crash".
Additive, leak-clean, links resolve, curated path untouched. Docs-only
(triggers no CI). Follows the (b) cross-link pass afe56f7.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Post-release additive pass — the pre-existing high-traffic guides didn't
know `pull` exists. All edits additive, curated path untouched (same
discipline as the README migration nudge):
- FAQ.md: new Q "model not in the supported list — can I still run it?";
GGUF Q gets a v0.8.0 note (safetensors-only eval, GGUF→llama.cpp stays
curated/manual, cross-engine deferred); launch.sh answer points
non-catalog models at `pull`.
- SINGLE_CARD / DUAL_CARD / MULTI_CARD: one blockquote cross-link each to
docs/PULL.md ("not in the configs / any HF safetensors repo — both
paths work").
- ADDING_MODELS.md: reframed catalog-onboarding vs just-run-a-model
(`pull`); the doc is the heavier calibration-catalog promotion task,
not a prerequisite for serving.
- GLOSSARY.md: new "Universal pull (v0.8.0)" table (pull, dry-run,
confidence tier, boot-fit≠runtime, calibration backbone).
Leak-clean; all links resolve on master; CommonMark structure verified
(blockquotes/headings blank-line separated). Docs-only — triggers no CI
(only tags do); lands as post-v0.8.0 polish on master.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- README.md: example block now shows model + GPUs flow with new --model /
--gpus / --tp / --pp flag examples; scripts/ layout description updated to
"model → GPUs → KV projection".
- docs/FAQ.md: rewrite the "how do I pick model + variant?" answer to
describe the new wizard (model picker → GPU picker → TP/PP auto-pick →
kv-calc projection → boot). Replace CUDA_VISIBLE_DEVICES card-selection
guidance with --gpus flag, preserving the env-var path for compatibility.
- docs/SINGLE_CARD.md, docs/DUAL_CARD.md: fix inline wizard descriptions
("asks engine + workload" / "asks GPU count + workload" → "asks model +
GPUs, projects VRAM budget").
All --variant <name> back-compat invocations across the docs continue to
work; no flag-renaming churn surfaced to users.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@tomByrer asked on disc #125 about pointing the scripts at a model
dir on a second drive. The support has always been there (MODEL_DIR
env var with 4-tier resolution: shell export, .env file, interactive
prompt, silent fallback) but the docs were too terse to discover
without running setup.sh interactively.
Two changes:
- docs/FAQ.md: expand the "where does it go?" entry from a one-liner
to a full explanation covering all 4 resolution paths, HF env-var
integration (we mount host dir INTO container cache, so HF_HOME
isn't directly respected but layout is byte-compatible), and
Windows/WSL2 path handling.
- README.md: add a 3-line parenthetical to step 2 of the quickstart
pointing at MODEL_DIR. Users who copy-paste the quickstart now see
the option without having to dig.
Both surfaces now point at the same canonical answer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds docs/DTYPE_MATRIX.md — a reference table mapping NVIDIA GPU
architectures (Pascal → Blackwell DC) to native vs emulated Tensor Core
support for each dtype + quant scheme.
Sections:
- At-a-glance compute-dtype matrix (9 archs × 10 dtypes, ✓/SW/✗ marks)
- Weight-quantization schemes (GPTQ / AWQ / AutoRound / NF4 / SmoothQuant
/ FP8 weights / MXFP8 / MXFP4 / NVFP4 / GGUF / HQQ-AQLM-SqueezeLLM)
with storage-vs-compute paths
- Weight-only vs weight+activation axis (W4A16 vs W8A8 vs W4A8)
- KV-cache dtype support (FP16 / FP8 / INT8 PTH / TQ3 / TQ4 / k8v4)
- Per-arch compose recommendations (which compose to ship for which
GPU class)
- Runtime detection (points at Genesis guards.py)
- Corner cases (Ada FP8 vs Hopper FP8, Blackwell consumer vs DC, NVFP4
vs MXFP4 block-size differences, MX* family overview, FP6)
- References (NVIDIA whitepapers, Marlin, Genesis, BENCHMARKS)
Cross-linked from:
- HARDWARE.md GPU-compat table
- GLOSSARY.md Quantization section
- FAQ.md as a new "What dtype/quant should I pick for my GPU?" Q under
Hardware
This sets the foundation for future per-arch compose optimization —
detecting compute capability at boot and picking the right KV dtype /
weight quant scheme automatically based on what the hardware actually
accelerates rather than what's nominally supported.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surfaces the canonical int8_per_token_head anti-scaling trade-off so the
next dual-3090 user who hits this finds the answer via search rather than
having to reason from the head-to-head matrix.
Frames it as:
- not a bug — per-(token, head) scale serializes at concurrency, fp8's
single global scale doesn't
- pick by workload: INT8 PTH for single-stream throughput, fp8 for
aggregate concurrency, Genesis-backed TQ3+MTP if you want both
- includes the diagnostic checklist for matching baseline numbers
(power cap, vLLM nightly, Genesis on/off, MTP n, prompt shape) so users
can self-troubleshoot a cross-rig gap before posting
Triggered by a Discord question on a dual-3090 + NVLink rig that observed
the canonical INT8 anti-scaling pattern (150 TPS single / flat at concurrency)
vs fp8 (70-100 TPS single / 400+ at concurrency).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five-format matrix across TQ3/TQ4/k8v4 + MTP all fail long-context needle
with first-word repetition, regardless of cudagraph state or PR #40914 K+1
dispatch. TQ3 with MTP disabled passes 7/7 verify-stress, so this is an
MTP × TurboQuant multi-query interaction (Sander's P67/P67b territory),
not TQ precision.
PR #40914 actively makes things worse on our stack: dropping it lifts
verify-stress from 3/7 to 5/7. Per vllm.ai/blog/turboquant the upstream
position is that hybrid-attention models (which Qwen3-Next is) are not yet
supported by TurboQuant — matches our matrix.
Updates anchor the conclusion in tree:
- docs/UPSTREAM.md: #40914 reframed as OPEN/negative on this stack;
P67/P67b called out as the only known working multi-query fix.
- docs/FAQ.md: #40914 answer updated with the 2026-05-11 local result.
- models/qwen3.6-27b/INTERNALS.md: P67 → P67/P67b, #40914 status flipped.
- vllm/patches/vllm-pr40914-k1-only/README.md: prepended "negative result"
status section; round-2/3 MTP-skip experiment annotated as ineffective.
- vllm/patches/vllm-pr40914-k1-only/.../turboquant_attn.py: epilogue now
writes through the caller's output buffer (Codex fix; left for the
re-test artifact, not deployed).
- vllm/compose/dual/tq3-mtp.yml: re-tombstoned to point at the upstream
feature-compat gap rather than a 5-PR landing list.
Deployable Genesis-free path remains dual/tq3-nomtp.yml; TQ + MTP path
remains dual/tq3-mtp-genesis.yml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The dev rig had grown across three home dirs (`/opt/ai/compose/`, `/opt/ai/github/`,
`/home/wasif/`) and three repos (single-3090, dual-3090, club-3090). Disk-out
on `/` (97% used) forced a cleanup; rather than just prune, we consolidated
the layout across the whole stack while we were at it. This commit captures
what landed inside this repo.
Services consolidation:
- services/{ollama,openwebui,litellm,qdrant,searxng}/ migrated in from
/opt/ai/compose/<svc>/ (zero functional change — same docker-compose.yml).
- services/litellm/config.yaml rewritten: explicit routes for current
primaries (qwen3.6-27b-autoround → :8010, gemma-4-31b-autoround → :8030).
Removed `* → ollama/*` wildcard.
- services/comfyui/ migrated in (was /opt/ai/compose/comfyui/) — wired into
gpu-mode with full mutex against vLLM/SGLang.
scripts/gpu-mode.sh under git:
- Was a loose /opt/ai/gpu-mode.sh outside any repo. Symlinked at
/usr/local/bin/gpu-mode.
- Five Gemma 4 31B modes added: gemma, gemma-dflash, gemma-int8,
gemma-dflash-int8, gemma-awq.
- One ComfyUI mode (mutex with all LLM serving).
- prune / prune-all subcommands (safe image prune; aggressive variant adds
build cache --keep-storage 5GB + dangling networks).
- gpu-mode status now shows Docker disk + /var/lib/docker + /tmp sizes.
- compose_at() passes --env-file <repo>/.env so MODEL_DIR resolves
regardless of which compose dir gpu-mode cd's into. Fixes the recurring
"MODEL_DIR not set, defaulting to ../../../../../models-cache" warning.
- stderr no longer swallowed by compose_at() (real errors surface).
- Cross-model VRAM mutex: every Qwen mode stop_all_gemma + stop_comfyui
and vice-versa.
scripts/maintenance/ — new hygiene-tools subdir:
- list-image-pins.sh: engine-agnostic pin auditor. Scans every compose's
`image:` line, groups by `<repo>:<tag>`, flags pin-drift (multiple tags
per repo), ranks composes by patch surface.
Pin tracking:
- docs/UPSTREAM.md gains a "Pinned images" section: table of every pinned
image, why each pin exists, retirement candidate criteria.
- docs/NIGHTLY_BUMP_RUNBOOK.md (new): 7-step procedure for bumping pinned
engine images (scope → branch → patch survival → boot → verify-full +
verify-stress → bench delta → land → retire). Engine-specific notes
for vLLM nightly hashes, llama.cpp digest pinning, SGLang variants.
Path updates from the engine + model dir consolidation:
- /opt/ai/vllm-src/ → /opt/ai/engines/vllm/primary/
(in setup.sh, INTERNALS.md, several patch READMEs, docs/HARDWARE.md,
docs/FAQ.md, docs/DUAL_CARD.md, docs/UPSTREAM.md, models/qwen3.6-27b/
CHANGELOG.md)
- /mnt/models/gguf/qwen3.6-27b/ → /mnt/models/huggingface/qwen3.6-27b-gguf/
(in models/qwen3.6-27b/llama-cpp/{compose/single/*.yml, recipes/*.sh,
README.md}, docs/engines/LLAMA_CPP.md)
CHANGELOG.md narrative gap fill:
- 2026-05-10 entry for this reorg.
- 2026-05-09 entry for compose convention formalization (topology
promoted to dir level, profile schema, Status enum + Caveats, cliff
CI swap, Discord launch).
- 2026-05-08 entry for Gemma 4 INT8 PTH unblock + 262K validation.
- 2026-05-07 entry for power-cap-sweep campaign + HARDWARE.md cross-rig
charts + cross-rig benchmark rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A single-card RTX 3090 Ti rig on WSL2 (driver 596.36) hits
RuntimeError: CUDA driver error: device not ready from gptq_marlin_repack
immediately after weight load on the v7.72.2-uplift nightly pin.
Bisect ruled out Genesis, spec-decode, TQ3 KV, async-residual error,
and TDR (registry already extended + Windows rebooted).
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False resolves it. Same
env-var workaround as JusefPol's NVLink boot-crash (PR #31), already
hardcoded in the dual-nvlink*.yml composes.
Replaces the hardcoded PYTORCH_CUDA_ALLOC_CONF line in 14 single-card
and PCIe dual-card composes with a ${PYTORCH_CUDA_ALLOC_CONF:-...}
override (defaults preserved). Pattern matches existing MAX_MODEL_LEN /
GPU_MEMORY_UTILIZATION overrides from #79. The two dual-nvlink*.yml
composes are unchanged — their existing JusefPol-driven default already
has expandable_segments off.
Documentation:
- docs/HARDWARE.md: new "disable PyTorch expandable_segments" subsection
alongside the TDR fix, with stack trace, what was ruled out, override
recipe, and a single uncontrolled observation about weight-load time
(32 sec → 13 sec).
- docs/FAQ.md: WSL2 question now cross-links both the TDR and
expandable_segments fix subsections.
- .env.example: documents the override under "vLLM tuning knobs".
- CHANGELOG.md (top-level + per-model): dated 2026-05-06 entries.
The exact failing call hasn't been isolated. The cuMemMap virtual-memory
API used by expandable_segments:True is the suspected culprit since
both known occurrences respond to the same workaround, but no specific
call has been proven to return cudaErrorNotReady.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After shipping report.sh --full / --soak / --stress flags in 8a29b95, the
bug report flow + troubleshooting guidance still pointed at older patterns.
Updated:
- .github/ISSUE_TEMPLATE/bug-report.yml — replaced single-flag suggestion
with a "pick the flag that matches your bug" decision table:
- boot crash / wrong output / tool-call regression → --verify (~2 min)
- OOM mid-conversation / agentic cliff → --soak (~25 min) — only test
that catches Cliff 2b
- TPS regression / cross-rig perf → --bench (~5 min)
- not sure / capture everything → --full (~35 min)
- docs/FAQ.md "Found a bug — what should I include?" — replaced the generic
"verify-full.sh output" note with the same flag decision table; updated
the troubleshooting-ladder pointers to use --verify (boot path) and
--full (multi-card TQ3+Genesis intersection bugs).
- docs/MULTI_CARD.md cross-rig contribution callouts (×2) — recommended
command upgraded from --bench to --full (or --bench fallback if soak
time-budget is tight, with explicit caveat that --bench skips Cliff 2b).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Today's full validation matrix exposed Cliff 2b — DeltaNet GDN forward
fires under accumulated multi-turn context (~21-26K), not just at single
prompts >50-60K. All 6 single-card vLLM variants FAIL v2 continuous;
only vllm/dual (TP=2) and llamacpp/default survive cleanly. Three issues
filed today (#41, #42, #43) all map to the same class. Docs needed to
reflect this so users don't keep landing on long-text/long-vision/
tools-text and burning time tuning what won't fix.
docs/SINGLE_CARD.md
- New ⚠️ section at top: single-card vLLM unsafe for hermes/openhands/
OpenCode/Cline/OpenClaw/Aider/Cursor with retained context. Routing
tree to dual.yml or llamacpp/default.
- Existing "One limitation" split into Cliff 2a (single-prompt, mostly
closed v7.69) and Cliff 2b (multi-turn, NOT closed). Reasons each
can/can't be tuned at config layer.
docs/CLIFFS.md
- TL;DR table extended from 2 cliffs to 3 (Cliff 2a + 2b separated).
- New section "Why TP=2 escapes" — per-card head sharding halves the
GDN live-tensor sizes (48 MiB → 24 MiB on v/u/o etc., 97 → 49 on h).
Concrete byte math; total per-card live FLA set drops from ~500 MiB
to ~250 MiB. Validated 0 MiB growth on dual.yml v2 continuous.
- New section "Why llama.cpp escapes" — three concrete differences:
(1) different GDN kernel (own CUDA, smaller per-step working buffers
vs FLA Triton), (2) ggml manual allocator (no PyTorch caching layer
fragmentation), (3) no JIT/Triton autotune (pre-compiled, static
layout from boot). Trade ~3× decode speed for cliff-immunity.
docs/FAQ.md
- New troubleshooting entry: "My hermes / openhands / OpenCode / Cline /
OpenClaw / Cursor session OOMs after a few turns. What do I do?"
Lists every dead-end we tested today so users don't repeat them:
mem-util tuning, MTP-off, max-num-batched-tokens (Mamba block_size
floor blocks <4128), TRITON_CACHE_AUTOTUNING (Blackwell-only recovery),
expandable_segments (already on), empty_cache (reclaims but cliff
fires next turn). Routing answer + soak-test repro command.
CHANGELOG.md
- Dated entry above the soak-test v2 entry summarizing the docs sweep.
Codex residency pilot data backs every claim:
results/residency-20260503-codex-pilot3/ (initial)
results/residency-20260503-empty-cache-idle/ (empty_cache experiment)
Investigation memo: docs/diagnostics/cliff2-handoff-results.md (gitignored).
Companion docs update for 43fe2a4. Three places where users would expect
to find this:
- README.md — adds step 7 to quick-start ("Keep your install up-to-date")
with the upgrade flow + the soft-warn note. Updates the directory tree
to list update.sh and refresh the preflight.sh one-line summary.
- docs/FAQ.md — new "How do I keep my install up-to-date?" entry under
Setup, covering the dirty-tree refusal, --dry-run / --force flags, and
the cross-link to the existing Genesis-pin warning. Also tweaked the
"How do I bump Genesis" entry to point at update.sh as the normal path
and frame manual bumps as the testing-only escape hatch.
- CHANGELOG.md — 2026-05-03 entry describing both pieces (preflight_repo_drift
+ scripts/update.sh) and the JusefPol dual-nvlink variant landing.
[@RossNE99]'s case in #32 surfaced a failure mode that wasn't in the
FAQ: vLLM throws a misleading "GPU OOM, tried to allocate 44 MiB"
error at model load when the actual problem is WSL2's default RAM
allocation (50% of Windows host) being insufficient for the 17.69 GiB
checkpoint.
The smoking-gun log line is `[weight_utils.py:934] Auto-prefetch is
disabled because ... checkpoint size (17.69 GiB) exceeds 90% of
available RAM` — surfaced automatically by report.sh now, but not
otherwise visible in standard triage output.
Adds a troubleshooting entry pointing at the .wslconfig fix
(memory=24GB minimum, swap=8GB), with the diagnostic log line so users
can self-identify if they suspect this. Also cross-links the original
RossNE99 repro thread for context.
Was invisible until report.sh captured the boot log highlights — good
worked example of why the standardized rig dump is more useful than
ad-hoc nvidia-smi pastes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related improvements after [@noonghunna]'s observation that the
ladder we just added (commit 9560efd) was structurally OK but easy to
miss:
1. **FAQ ladder gets an at-a-glance table** at the top of
"Before symptom-matching" section. Five rows, one per step, showing
variant name + what each step adds + what it tests. Plus a one-line
path-finder ("if single-card, run 1-3; if dual, layer-by-layer
tells you which intersection breaks"). Readers no longer have to
scroll through 80 lines before they see the full ladder shape.
2. **Bug report template leads with the ladder** instead of jumping
straight to the report.sh ask. New intro:
"Before filing — try the 5-step triage ladder first"
Links into the FAQ section anchor. Acknowledges that "a lot of
'should I file a bug' questions resolve at step 1 or 2 (often
re-running setup.sh is the fix)." Worth 15 min before opening an
issue. The report.sh paste field stays — for users who've done the
ladder and have a real bug to file.
3. **Issue chooser config gets a third option** above
"General Q&A / discussion":
"Troubleshooting — try the 5-step triage ladder first"
Clicking "Open an issue" → users now see Troubleshooting / Q&A /
Bug-report / Bench-contribution as four distinct paths, with the
ladder explicitly named as a self-help option BEFORE the bug-report
template appears.
Result: users with budget / boot / MTP-class issues get pointed at
the ladder twice (once in the chooser, once in the bug template intro)
before they ever fill out the form. Reduces the "filed a bug that
turned out to be a partial-pull / setup-not-rerun / config-too-tight
issue" pattern that's eaten the last few triage rounds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When users hit boot OOMs, MTP weirdness, or TQ3/long-context budget
issues, our previous troubleshooting flow jumped straight to
symptom-matching. That misses the systematic narrowing pass: validate
the simplest stack works first, then add one variable per step until
the failing layer is isolated.
New leading section in Troubleshooting:
- Step 1 — `vllm/minimal` (32K + fp8, no Genesis, no spec-decode):
validates hardware, driver, Docker, NVIDIA Container Toolkit, model
files, base vLLM. Strips out every layer that could be the cause.
- Step 2 — `vllm/tools-text` (75K + fp8 + MTP + Genesis): adds Genesis
+ MTP K=3. Still fp8 KV (no TQ3 yet). Most common failure here is
GENESIS_PIN-vs-tree mismatch — re-run setup.sh.
- Step 3 — `vllm/long-text` (180K + TQ3 + MTP + full Genesis): adds
TurboQuant 3-bit KV + long-context. The production-target single-card
config. Failure here narrows to TQ3 setup, GDN >60K hardware wall,
or Cliff 1 mech B (closed since v7.69 PN25).
- Step 4 — `vllm/dual` (262K + fp8 + TP=2 + 2 streams, Genesis-less):
adds TP=2 NCCL + multi-GPU memory split. Crucially removes Genesis,
since dual.yml is intentionally Genesis-less. Failure here despite
step 3 working narrows to TP=2 NCCL specifically. WSL2 is the most
common trigger (its vGPU layer adds memory accounting wrinkles that
bare-metal Linux doesn't have).
- Step 5 — `vllm/dual-turbo` (262K + TQ3 + TP=2 + Genesis): the full
multi-card stack. Failure here despite step 4 narrows to the
TQ3-on-TP=2-with-Genesis intersection.
The ladder works for both single and dual-card users because steps 1-3
isolate stack layers regardless of GPU count, and steps 4-5 add TP=2
surface separately. A dual-card user hitting issues should still run
steps 1-3 on a single card first — it's the only way to tell apart
"single-card stack issue that also breaks dual" vs "TP=2 NCCL specific."
Existing symptom-pattern-matching list demoted to "Quick recognition
guide" sub-section, kept verbatim — still useful for users who already
know the failure surface and want to skip the ladder.
Going forward we'll point to this ladder explicitly in triage replies
on issues / discussions, replacing the ad-hoc "have you tried X?" pattern.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes / addresses 3 reported issues + adds requested feature:
#7 vid (PORT not honored, MODEL_DIR vs MODELS_DIR confusion):
- All 8 vLLM compose files now use "${PORT:-XXXX}:8000" so .env PORT
flows through. Defaults preserved per-variant (8020 single, 8010-8013
dual). llama.cpp composes already had this pattern.
- scripts/switch.sh: load .env early; per-variant default-port table;
new resolve_ready_url() picks PORT > variant default for the readiness
probe.
- scripts/launch.sh: same default-port table; final endpoint URL printed
to user reflects actual mapped port.
- .env.example: ⚠ box callout that variable names are CASE-SENSITIVE
(MODEL_DIR singular, NOT MODELS_DIR plural — silently ignored).
New PORT section documenting per-variant defaults.
#4 timxx (tools-text.yml fails "Free memory ... less than desired"):
- docs/FAQ.md: new entry "Container fails to start: Free memory..."
explaining the vLLM startup check, the two workarounds (free VRAM /
lower mem-util), and which configs hit it most often (0.97+ mem-util).
- Compose defaults unchanged (0.97 stays the right design target on
headless rigs); the FAQ documents the workaround for users with X11.
#1 fabriciomalta (per-config VRAM column):
- docs/SINGLE_CARD.md: TL;DR table now has VRAM column with mem-util.
- docs/DUAL_CARD.md: TL;DR table same + footnote explaining per-card
semantics and which dual configs would/wouldn't fit on 2× 20 GB cards
(relevant to fabriciomalta's 2× 3080-20GB use case).
#2 tenitram (empty responses) — fixed in master via aab8ff4
(P68/P69 disabled). Closed with reply pointing at the fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single canonical reference for everything we know about Cliff 1
(FA2 softmax_lse cap-leak) and Cliff 2 (fla.ops GDN forward
intermediate buffer): TL;DR table, empirical bisection with stack
traces, root-cause walk-through, why earlier "FFN intermediate
buffer" framing was wrong, why mem-util doesn't help, why PN8
closes Cliff 1 on tools-text but not on TQ3 paths, why llama.cpp
dodges both structurally, alternative attention backends with
feasibility, who-can-fix-it landscape (Sandermage, Tri Dao, fla-org,
QwenLM, us at any difficulty), recommended path forward, and
re-test triggers.
Cross-linked from FAQ.md and README.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After bisecting long-vision config space (192K/128K/96K/86K at 0.98
and 0.92 mem-util) and second-opinion synthesis from ChatGPT +
DeepSeek + vLLM source review, the actual root cause is:
softmax_lse in flash_attn_varlen_func is allocated as
[num_seqs, num_heads, max_seqlen] — sized by the max_seqlen
parameter, NOT the actual cu_seqlens.
vLLM passes attn_metadata.max_seq_len; during cudagraph capture
that's set to max_model_len. So a 25K-token tool prefill at
max-model-len=192K allocates softmax_lse for 192K, eating the
activation headroom. The 50-138 MiB OOMs we'd been observing
are downstream of this leak.
Empirical OOM site (verified in our docker logs): _vllm_fa2_C.varlen_fwd
in flash_attn_varlen_func. Upstream root cause: Dao-AILab/flash-
attention#1011 (open since 2024). vLLM cap-leak path: vllm#40961.
Earlier "FFN intermediate buffer" characterization was wrong.
Updates:
- UPSTREAM.md: new FA2 section (Dao-AILab/flash-attention#1011);
added vllm#40961 (cudagraph capture max_seq_len pattern), vllm#40069
(TurboQuant follow-ups tracker), and vllm#25543 (V0 deprecation
removed max_seq_len_to_capture, so commonly-suggested mitigation
doesn't apply on V1 nightly)
- FAQ.md: corrected Cliff 1 explanation
- SINGLE_CARD.md: corrected "Cliff 1 still fires" caveat
- CHANGELOG: documented bisection + revision
- memory/qwen36_27b_prefill_cliffs.md: revised Cliff 1 mechanism;
noted Cliff 2 likely shares the same architectural pattern
Practical implication: no new variant ships. Default 48K + 0.92 +
TQ3 + vision stays the prefill-safe ceiling — pushing higher requires
upstream fix at FA repo, not config tuning. tools-text.yml (75K + FP8
+ PN8 closes Cliff 1) remains the IDE-agent path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fast-chat (20K, fp8, vision) and default docker-compose.yml (48K, TQ3,
vision) had effectively the same TPS post-PN8. fast-chat's only
remaining differentiator was "smaller context = ~3s faster boot," and
20K is actively bad for IDE-agent users (Copilot tool-schema preamble
alone hits 20K). Net negative — removed.
Default compose was missed in the previous P68/P69 fix — it had the
same env vars enabled and the same silent-stop bug above 8000 chars.
Both now disabled with the same explanatory comment.
Updated:
- scripts/switch.sh, scripts/launch.sh — drop the variant
- docs/SINGLE_CARD.md, FAQ.md, engines/VLLM.md, model + vllm + patches
READMEs — references removed or pointed to default/tools-text
- All sibling compose YAML "see also" tables — fast-chat row removed,
tools-text row repurposed for IDE-agent guidance
- CHANGELOG entry; old historical entries kept as-is (append-only)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
P68 (auto force tool_choice=required) and P69 (inject "must use a
tool" reminder) silently fired on prompts > 8000 chars — every IDE
agent (Cline, Cursor, OpenCode, Copilot Gateway) blew past that
threshold instantly and got silent finish_reason=stop with no
content + no tool_calls on greetings or clarifying questions.
Bisection on club-3090#2 (HoodOG1 + tenitram):
state A (P64+P68+P69+PN8): broken
state B (P64+P69+PN8, P68 off): still broken — model loops on
"I cannot respond with plain text" then stops mid-reasoning
state D (P64+PN8, P68+P69 off): clean — greeting → plain-text
reply; tool request → clean read_file call
P64 (qwen3coder MTP streaming early-return fix) and PN8 (FP8+MTP
draft online-quant memory savings) stay enabled — real bugfixes,
no user-intent override.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User feedback: navigating to relevant docs was cumbersome. The natural
first decision is "1 GPU or 2 GPUs?" and the existing docs mixed model-
specific reference with deployment guidance.
New navigation:
- README.md adds a "Pick your path" pivot pointing at hardware-axis pages
- docs/SINGLE_CARD.md — 1× 3090 deployment menu (workload → compose →
TPS) with all single-card configs (vLLM + llama.cpp), VRAM budget,
prefill cliffs explained operationally, what single-card can't do
- docs/DUAL_CARD.md — 2× 3090 mirror (4 dual variants + TP=2 explainer +
what dual unlocks vs single + Marlin pad fork dependency)
Slimming:
- models/qwen3.6-27b/README.md: dropped duplicated variant tables
(now in GPU-count pages); kept model-specific content (quants, Genesis
patch surface table, what's working / not, VRAM diagram)
- models/qwen3.6-27b/USE_CASES.md: deleted. Per-workload content
absorbed into the GPU-count pages (deduplicated). Troubleshooting
list moved to docs/FAQ.md as a new "Troubleshooting" subsection.
Image-token cost / vision specifics absorbed into SINGLE_CARD.md.
Reference updates: 8 files updated (engines/VLLM.md, engines/README.md,
COMPARISONS.md, ARCHITECTURE.md, EXAMPLES.md, FAQ.md, INTERNALS.md,
top-level README) — all USE_CASES.md links re-pointed to SINGLE_CARD/
DUAL_CARD where appropriate.
Net delta: -167 lines (was 197 in USE_CASES + duplicated tables in
model README; now 342 lines split between SINGLE_CARD + DUAL_CARD with
content deduplicated against each other).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Documents the two compatibility issues users will hit with VS Code's
Copilot LLM Gateway:
1. Tool-schema preamble is ~20K tokens — fast-chat.yml's 20K cap is
too small. Recommended pick is tools-text.yml (75K + fp8 + PN8,
Cliff 1 closed since Genesis v7.62.x).
2. Copilot probe-style requests with max_tokens=64 truncate tool-call
JSON mid-string. With tool_choice: required + minItems: 1 in their
structured-outputs schema, the model must emit a tool call that
takes real arguments — won't fit in 64 tokens. Manifests as
"empty response" client-side. Server-side correct.
Background + debug-log analysis from tenitram on club-3090 #2.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/setup.sh — GENESIS_PIN bumped from bf667c7 (v7.54) to 917519b
(v7.62.x release, 2026-04-29). New patches: PN8 (MTP draft online-quant
propagation, backport of vllm#40849), PN11 (Quentin-M streaming tool-call
IndexError fix vllm#41142), per-GPU profile auto-rec, k8v4 unlock on
hybrid GDN via P4+P98.
PN8 enabled on FP8 paths only:
- tools-text.yml: -900 MiB at boot, Cliff 1 25K tool prefill closes,
-7% code TPS. Net win — production-safe for tool-using agents.
- fast-chat.yml: -800 MiB at boot, no cliff to test at 20K, -4.7% code
TPS. Free VRAM is useful for tighter mem-util configs.
PN8 not enabled on TQ3 paths (default 48K, long-vision, long-text) or
dual configs:
- default 48K: PN8 is no-op on TQ3 + 0.92 (plenty of headroom already)
- long-vision: PN8 grows KV pool 230 MiB and lifts engine ceiling 192K
→ 198K, but does NOT close Cliff 1 — the 138 MiB allocate is an FFN
intermediate-buffer activation peak (intermediate_size × max-num-
batched-tokens), not a draft-model footprint
- long-text: engine ceiling at 206K is gated by attention-block-size
divisor, not KV; PN8 has nothing to give
- dual.yml: deliberately Genesis-less by design; not worth restructuring
Verify-full passes on default 48K + v7.62.x without PN8 (8/8). Verify-
stress on tools-text + PN8 passes all checks including the 25K tool
prefill that was the launch-tweet headline caveat.
Cross-rig data shared with Sandermage:
https://github.com/noonghunna/qwen36-27b-single-3090/issues/1#issuecomment-4343317153
Docs updated: cross-cutting CHANGELOG, per-model CHANGELOG, USE_CASES.md
(Cliff 1 closure note on tools-text), FAQ.md (Cliff 1 entry + new PN8
entry).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
