The pinned cu13 ik-llama digest has a CUDA 13.2 runtime. On a driver whose
supported CUDA < 13.2 (e.g. 580.159 = CUDA 13.0, which satisfies the README's
"580.x+" but NOT this pin) the forward-compat path fails on GeForce (CUDA error
804) -> silent CPU fallback -> segfault crash-loop, with launch.sh just timing
out after 600 s and no actionable hint.
- preflight.sh: preflight_ik_llama_image() detects the driver's supported CUDA
(_driver_cuda_version) and, for ik-llama variants on a driver < 13.2 with
IK_LLAMA_IMAGE unset, auto-selects the cu12 sibling build (same build 4574,
CUDA 12.6, backward-compatible, DelspoN-validated ~68 TPS on 580.159) + prints
a ⚠ hint. >=13.2 keeps cu13; a user IK_LLAMA_IMAGE pin always wins (echoed).
_cuda_ge() does the major.minor compare. Verified: 13.3/13.2 GE, 13.0/12.6 LT.
- switch.sh: call it right after export_variant_engine_pin, before compose up
(the single up path — launch.sh delegates here). preflight.sh is already
sourced in that block.
- docs: qualify the driver requirement in the ik-llama compose header + HARDWARE.md
("580.x+" is insufficient; needs CUDA >= 13.2; auto-fallback documented).
Note: this rig is CUDA 13.3, so the <13.2 boot path is verified by logic +
DelspoN's cu12-server-4574 validation, not a local boot. Rolling cu12 tags are
NOT safe (cu12-server-100 rejects the compose's --spec-type flag, #633) — pinned
to the build-number tag matching the cu13 digest's build.
Closes#633.
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per maintainer call: "cluster" conventionally means multiple networked
machines (a non-goal here — LiteLLM fronts multi-host) and is best reserved
for a future enterprise/multi-node meaning. "Pod" is the accurate analogy
for what this is — one model on a GPU subset on ONE host (k8s/RunPod sense).
The capability is unchanged; only the name.
Scoped rename (cluster→pod, case-aware) across the pod feature ONLY:
- scripts/cluster.sh → scripts/pod.sh; test-cluster-cli.sh → test-pod-cli.sh;
docs/CLUSTERS.md → docs/PODS.md
- estate_cli.py verbs + wording; app.py (ClusterCreateScreen→PodCreateScreen,
action_new_cluster→new_pod, _populate_clusters→_populate_pods, #cluster-view
→#pod-view, the [N] help/empty-state text); services.py cluster_create_plan
→pod_create_plan; data.py kind cluster_create→pod_create; tests + doc
pointers (HARDWARE/MULTI_CARD/README/c3-README)
- UNTOUCHED (unrelated "cluster"): compat.py + test-profiles-compat.sh (the
VRAM-topology classifier), services.py:1863 / test_services.py (the scene-
table "cluster by group" verb), older docs, .venv
Also folds in the [N] discoverability fix (n was already bound to
serving_switch — moved to N; empty-estate now shows a "no pods — [N] new
pod" affordance + a help entry) and a heterogeneous-rig section in PODS.md:
one homogeneous pod per card family (2×3090 · GB10 · 6000 Pro) is the clean
pattern — mixing families in one TP pod makes NCCL wait on the slowest +
wastes VRAM; a worked 2-pod lifecycle walkthrough.
Verified: test-pod-cli + estate/gpu/profiles guards green; 25 c3 pod/binding
+ 239 fast tests green; pod.sh live create/list/D1-reject on 2×3090; zero
stray "cluster" in pod files (scene-verb preserved); no CLUSTERS.md links
left; PODS.md leak-clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Clusters (cluster.sh + the c3 view/wizard + the estate file + CDI pinning)
had grown into a subsystem scattered across HARDWARE.md and the c3 README.
Consolidate into a dedicated docs/CLUSTERS.md, sitting alongside
SINGLE/DUAL/MULTI_CARD as the multi-model-per-host workload axis:
- concept (cluster = model + GPU set + port = estate instance) + cluster-vs-TP
- a Defaults section answering "what happens before you make any cluster":
no default cluster / empty estate; a plain launch boots ONE model whose
GPU count = the compose's TP (prompts both/all on multi-GPU), NOT the
cluster machinery — clusters are opt-in
- CLI quickstart (create/list/status/up/down/rm) + c3 quickstart ([n] wizard,
cluster view)
- D1 fit-vs-set rules, placement verification, the estate file schema, the
command reference
HARDWARE.md keeps the GPU-pinning mechanics (UUID / CDI-NixOS — a hardware
topic) and points to CLUSTERS for management; c3 README, MULTI_CARD, and the
docs index now link CLUSTERS. All cross-links verified; public-safe.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
The CLI seam the c3 cluster UX (Phase C) will consume, and a full headless/
SSH cluster capability on its own. A "cluster" = a named model on a chosen
GPU set + port (an estate instance).
- estate_cli.py gains create / list / status / rm verbs (D2: NEW verbs go
IN estate_cli.py, which owns the schema + validate_estate + boot/down —
ONE validation path shared with hand-written estate files and the wizard).
up/down reuse `boot`/`down --only <name>`.
- create runs the D1 fit-vs-set logic (#610 addendum 3): kv-calc --card is
single-card + registry TP, so count != compose TP is a HARD REJECT, a
heterogeneous set is estimated against its min-VRAM card (+ note), and the
whole set is re-validated (validate_estate: GPU collision, port collision,
per-instance fits) before append. GPU indices stay index-based in the
estate file; UUIDs resolve at boot (Phase A).
- status carries the {requested, actual, placement} verdict per cluster
(the shape the c3 badge reads); list/status take --json.
- scripts/cluster.sh: the ergonomic bash front (create/list/status/up/down/
rm), a thin wrapper over estate_cli.py.
- test-cluster-cli.sh (NEW): full lifecycle hardware-free via CLUB3090_FAKE_
GPUS — create/D1-reject/collision-reject/list/status/rm.
LIVE-VERIFIED (2x3090): create chat (fit-clean ~20.2 GiB) → up → estate boot
→ "✓ placement verified" → status "serving ✓ placement=ok", GPU0 idle /
GPU1 loaded → down → rm. D1 count!=TP + GPU-collision rejects confirmed.
Part of #610 (Phase A′). Next: Phase C (c3 cluster view + wizard).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Extends the #611 primitive to the estate (multi-cluster) path and adds a
post-boot placement check — the foundation Phase A′ (cluster.sh) and C
(c3 cluster UX) sit on.
- scripts/lib/gpu-select.sh (NEW, shared lib): factors the #611 inline
resolver out of launch.sh into gpu_select_indices_to_uuids /
gpu_select_export, plus gpu_select_container_uuids +
gpu_select_assert_placement. launch.sh sources it (no re-inlined drift)
and runs the placement assertion after verify-full.
- estate_cli.py: resolve_gpu_uuids() (python twin of the bash resolver) +
compose_env / compose_override_doc now UUID-pin CUDA_/NVIDIA_VISIBLE_
DEVICES (ESTATE_GPUS stays index-based — the compose device_ids read the
host view). assert_placement[_quiet]() runs after each instance is ready
(sequential → stderr, parallel → per-instance boot log), returning the
{requested, actual, placement: ok|mismatch|unknown} verdict — the shape
cluster.sh status + the c3 badge will read.
- Placement uses --query-compute-apps=gpu_uuid (NOT --query-gpu): under CDI
the container sees all cards but RUNS on the CUDA-masked set, so
compute-apps is the runtime-agnostic ground truth.
- test-gpu-select (NEW): asserts the bash + python resolvers agree on real
hardware and both fall back identically; test-compose-gpu-mask-passthrough
updated to follow the resolver into the lib.
LIVE-VERIFIED (2x3090, classic runtime): `launch.sh --gpus 1` → UUID-pinned,
verify-full 8/8, "✓ placement verified", host GPU0 = 1 MiB / GPU1 = 20.8 GB.
estate compose_env confirmed UUID-izing while ESTATE_GPUS stays index-based.
CDI leg still awaits mog. Relevant guard sweep (11) green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
--gpus selection was NVIDIA_VISIBLE_DEVICES-only, which CDI runtimes
(NixOS hardware.nvidia-container-toolkit, nvidia-ctk cdi) IGNORE — mog's
3x3090 rig got all cards exposed and vLLM took the wrong two. Index-based
CUDA_VISIBLE_DEVICES can't fix it either: the classic runtime renumbers
the exposed set in-container.
UUIDs solve both at once:
- launch.sh --gpus N,M resolves indices -> GPU UUIDs (nvidia-smi -i) and
exports BOTH NVIDIA_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES as UUIDs.
Classic runtime: exposes-by-UUID; CUDA mask agrees (renumbering-proof).
CDI: exposure comes from device_ids (nvidia.com/gpu=all) and the CUDA
UUID mask is what pins the cards. Index fallback if the query fails.
- 26 composes gain a bare `- CUDA_VISIBLE_DEVICES` passthrough (unset ->
absent -> zero change for existing users); stale commented hints removed.
- preflight: selector consumers are host-index-based -> new
_preflight_selector_normalize maps GPU-xxxx tokens back to indices at
the single choke point (the first live boot caught this: preflight
rejected the UUID selector).
- new guard test-compose-gpu-mask-passthrough.sh: every compose with the
NVIDIA env line must pass CUDA_VISIBLE_DEVICES through + launch.sh must
keep the UUID resolution.
- docs/HARDWARE.md: "Pinning specific GPUs (and CDI / NixOS runtimes)" —
the runtime table, the CDI deploy-block swap, the in-container
renumbering gotcha, manual recipes.
LIVE-VERIFIED on this rig (classic runtime, 2x3090):
launch.sh --variant vllm/minimal --gpus 1
-> host GPU0 = 1 MiB (empty), GPU1 = 20.7 GB (model), both env vars
UUID-set in-container. CDI leg validation goes to mog (NixOS reporter).
Follow-up (tracked on #610): estate ESTATE_GPUS UUID-ization for CDI
multi-instance rigs; post-boot visible-GPU-count sanity check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Consolidates the validated findings from the #246 A/B + #571 volunteer
data + vLLM/TRT-LLM source review. New DTYPE_MATRIX section "Having the
Tensor Cores ≠ using them" makes the two load-bearing distinctions
explicit:
- Axis 1 — weights vs KV are gated SEPARATELY. FP8/FP4 *weights* work
wherever the TCs exist (sm_89+ FP8, all Blackwell FP4). KV is
different because it rides the attention (FMHA) kernel.
- Axis 2 — KV compute vs KV storage. Native FP8/FP4 *attention* compute
needs FA3 (Hopper sm_90 only) or trtllm-gen FMHA (datacenter Blackwell
sm_100/103 only). On EVERY consumer card — Ada 4090 (sm_89), consumer
Blackwell 5090/PRO-6000 (sm_120), DGX Spark GB10 (sm_121) — FP8 KV is
storage-only and nvfp4 KV doesn't work.
Consequences documented (both empirically confirmed): e4m3 ≡ e5m2 in
speed on consumer cards (86.73 vs 86.66 on a 5090, disc #571 — a
precision choice, not a perf lever); nvfp4 KV crashes on consumer
Blackwell (#43562). DGX Spark = same sm_12x family as the 5090.
Corrected now-wrong claims: DTYPE_MATRIX "Ada/Blackwell get a real win
on FP8 KV" (false — Hopper/DC-only) + the per-arch Ada/Blackwell-consumer
rows; QUANTIZATION + HARDWARE "native FP8 compute on sm_89+" → storage-
only + precision framing. Compute-win KV on consumer cards = INT8-PTH.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Two 5090 owners hit `--kv-cache-dtype nvfp4 requires sm100f` crashing
mid-boot on the #246 A/B (disc #571). Root-caused from vLLM #43562 /
TRT-LLM #10241: nvfp4 KV forces the trtllm-gen FP4 FMHA, built ONLY for
datacenter Blackwell sm_100/sm_103. Consumer Blackwell (sm_120/121 —
RTX 5090 / PRO 6000 Blackwell) is a HIGHER cc number but a different
family with no FMHA build. NVFP4 *weights* work there; only the KV path
doesn't. Our #246 gate had used a numeric ">=10.0" floor that wrongly
passed sm_120 — a floor can't express "sm_100/103 but not the
numerically-higher sm_120".
- gates.py: new `_ARCH_KERNEL_SM_FAMILY` allowlist ({nvfp4: sm_100/103});
dropped nvfp4 from the numeric `_ARCH_KERNEL_SM` floor; family-membership
reject with the FMHA reason + fp8_e4m3 fallback.
- arch-ab.sh: nvfp4 arm now refuses on consumer Blackwell (not just
<sm_10), naming the FMHA gap + the fp8_e4m3 path; dropped nvfp4 from the
recommended arms in help.
- hardware profiles: removed nvfp4 from rtx-5090 / rtx-6000-pro-blackwell
KV lists (both sm_120) + added a why-not note.
- docs (DTYPE_MATRIX / HARDWARE / KV_MATH / QUANTIZATION): corrected the
"Blackwell sm >= 10.0" framing to "datacenter sm_100/103 only".
- UPSTREAM.md: #43562 / TRT-LLM #10241 row + re-test trigger.
- test-arch-ab: nvfp4 refuses on sm_86 AND sm_120, allowed on sm_100;
the dual-5090 all-arms test drops nvfp4.
Full scripts gate 66/66.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
launch.sh/switch.sh now detect GPU arch and export KV_CACHE_DTYPE=
fp8_e4m3 (native FP8 compute) on sm_89+ cards for the pilot slugs
(vllm/dual, vllm/minimal). The injected value is the hardware
profiles' dormant kv_format_default.balanced -- one source of truth
shared with the pull gates; the Ampere no-op is data equality
(3090-class balanced = fp8_e5m2 = compose default -> nothing emitted),
not a code branch.
Injection guards (all load-bearing):
- pilot allowlist only; expansion gated on the #246 cross-rig A/B
- per-variant: registry kv_format == fp8_e5m2 only (int8-PTH/TQ/bf16
slugs never touched -- compressed-tensors weights reject fp8 KV)
- vLLM-family variants only; explicit user KV_CACHE_DTYPE= wins;
unmapped cards / heterogeneous rigs / no nvidia-smi -> no injection
- direct `docker compose up` keeps Ampere-safe compose defaults
Rides the existing resolve-variant-pin export seam (new optional
--gpu-spec); VLLM_ATTENTION_BACKEND is whitelisted but ships no value
(vLLM auto-detect stays the default until measured). Preflight banner
names the detected arch class.
Consistency fixes the injection exposed:
- gates.py _ARCH_KERNEL_SM: fp8_e4m3 9.0 -> 8.9 (vLLM's real floor is
SM89+; we'd otherwise inject e4m3 on 4090s our own pull-gate calls
unloadable) + new nvfp4: 10.0 row (v0.24.0 literal had NO gate --
a 3090 pull of an nvfp4 config wouldn't have been rejected)
- Blackwell hardware profiles: nvfp4 declared as CANDIDATE capability
(engine list unchanged until validated -- gates take the intersection)
- kv-calc: projected nvfp4 rows (bytes/elem + activation coefs),
calibration unchanged
Docs: HARDWARE.md new section, KV_MATH/QUANTIZATION/DTYPE_MATRIX rows
(incl. retiring the Genesis-era "e4m3 undertuned per #51" advisory in
favor of the A/B).
Validated: 8-case injection matrix in test-launch-compat; live no-op
on the real 2x3090 (spec built, nothing injected); faked 4090 through
the real bash detection path emits e4m3; compose interpolation both
ways; kv-calc --calibration green; full scripts gate 64/64.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
New dedicated guide for getting the most out of a PCIe-only multi-GPU
rig: reading nvidia-smi topo -m (why PHB not PIX), NUMA/NPS1, triple-slot
placement + M.2 lane-sharing, BIOS (Above 4G / ReBAR / ACS / IOMMU),
enabling consumer P2P via the patched driver + NVLINK_MODE=pcie_p2p,
realistic measured expectations (cross-rig #91/#95/#77), verifying
engagement via report.sh's new field, and troubleshooting. Generic
guidance with ROMED8-2T as a labeled example board.
Cross-referenced from HARDWARE.md (NVLink section), DUAL_CARD.md (NVLink
auto-detection), FAQ.md (Do I need NVLink?), and the docs index.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
FAQ/DUAL_CARD/HARDWARE told users to `git clone vLLM to /opt/ai/engines/vllm/primary/`
and said the dual composes mount from there — both stale. The marlin-pad patch
(vllm#40361) is now vendored in-repo (models/qwen3.6-27b/vllm/patches/vllm-marlin-pad/
{marlin.py,MPLinearKernel.py}) and overlaid into the stock vLLM image automatically by
each dual compose (delivery_mechanism: site_package_overlay). No user clone, no rig path.
Genericizes the internal path out of the public docs AND fixes the no-longer-needed
clone step in one go. (setup.sh already notes the clone is "previous design".)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The four dual/autoround-int4/nvlink-*.yml composes (nvlink-fp8-mtp,
nvlink-turbo, nvlink-dflash, nvlink-dflash-noviz) and their
vllm/dual-nvlink* launch slugs are removed. They were thin Docker Compose
`extends:` stubs whose ONLY override was NVLINK_MODE=force_on — fully
redundant since every base dual compose now auto-detects NVLink at boot
via detect_nvlink.sh (NVLINK_MODE=auto): it probes nvidia-smi and flips on
NCCL_P2P_LEVEL=NVL + custom-all-reduce when a bridge is present, else
NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce. NVLink rigs get the
identical fast path from the base dual compose with no separate slug;
force it explicitly with `NVLINK_MODE=force_on scripts/switch.sh vllm/dual`
if auto-detect ever misses.
Scope (prune only; no engine-image change — the v0.22.0 stable-engine
consolidation is a separate follow-up):
- delete 4 nvlink-*.yml composes
- compose_registry.py: drop the 4 _entry blocks (50 -> 46 entries)
- test-compose-registry-disk.sh: count guards 50/51 -> 46/47
- profile_runtime.yml + patches.yml: drop the 4 nvlink blocks/slug refs
- test-profiles-compat.sh: retire the C13/E3 NVLink-required-compose
scenarios (the estate NVLink-gating code is now dormant; cleanup tracked
as a follow-up, code retained)
- docs: switch.sh help, AGENTS.md, HARDWARE.md, UPSTREAM.md, patches/README.md,
fp8-mtp/dflash/dflash-noviz compose headers, per-model CHANGELOG removal
entry; add a docs/README.md pointer to `switch.sh --list` as the
authoritative registry-derived compose x slug matrix
Historical NVLink bench rows (JusefPol #31, danbedford #74/#92/#96) are
preserved in BENCHMARKS.md. Full test suite green (the lone
test-submit-bench failure is a worktree-isolation artifact — it needs
gitignored results/rebench/ fixtures absent from a fresh worktree; passes
on the working tree).
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The TurboQuant paper's "near-optimal distortion at 3.5 bits" is a perplexity/
average claim; per Anbeeld's KV-quant benchmarks on our exact Qwen3.6-27B/3090,
turbo3_tcq is ~82% 99.9th-pct KLD tail precision (visible loss on the worst 0.1%
of positions: JSON keys, closing braces, tool calls). Reframe TQ3 as a
context/concurrency trade, not quality-neutral; prefer fp8/q5_0 for code/JSON/
agent workloads. Cross-links FAQ + CLIFFS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
BENCHMARKS: new dual.yml row for the X399 + Threadripper 1950X (Zen 1) +
2× 3090 Gen 3 ×16+×16 rig (58.98 / 79.19, fp8, soak PASS) — same-rig PCIe
upgrade from the Z77 + 3770K Gen 2 ×4 row (+8% narr / +15% code), validates
the v0.7.3 Gen-3 prediction band. Placed above the Z77 row for the A/B.
HARDWARE: extend the older-platforms note — pre-Zen2 (Zen 1 / Skylake-X) is
a measurable additional cut, and CPU IPC can outrank cross-card PCIe in this
class. Worked datapoint, same vLLM SHA: mgabor3141's Gen 3 ×16+×16 + Zen 1
lands 17-19% below OVDEN13's Gen 4 ×4+×8 + Zen 3 (#142) — better interconnect,
slower CPU, lower TPS.
Credit: @mgabor3141 (#178).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
AMD hosts (Threadripper/Ryzen/EPYC) in the default AMD-Vi "Translated" IOMMU
mode can drive both GPUs into Xid 154 fatal MMU faults under sustained TP=2
DMA — surfacing as misleading tokenizers Rust segfaults / Triton "unspecified
launch failure". Fix: iommu=pt (IOMMU stays on for isolation, DMA bypasses
page-table translation). Adds a HARDWARE.md AMD-platforms section + a FAQ
troubleshooting Q with the dmesg triage line. Diagnosed by @mgabor3141.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sethbrasile's controlled 9-run matrix on a headless 3090 + driver
595.71.05 / CUDA 13.2 shows the same env-override pattern as the 4090
display-overhead case: the newer driver's vLLM activation-profile
reserve measured ~2.87 GiB vs ~1.5 GiB on the bare-metal reference rig,
shrinking the KV pool and capping long-text.yml at MAX_MODEL_LEN=105000
(vs 180K default). Added as the 3090 sibling anchor next to the
@laurimyllari 4090 -> 90000 data point so newer-driver 3090 users start
from the right number. Tuning-data contribution, not a bug
(corroborates the known Cliff 2a-under-v7.72.2 / genesis#22 picture).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New "Note for older host platforms" section between the SM86 long-ctx
note and the WSL2 section. Documents the 30-40% TP=2 throughput hit
that PCIe Gen 3 + pre-Zen3 / pre-2018 Xeon rigs see vs the reference
Gen 4 / Ryzen 5950X / EPYC rigs in BENCHMARKS.md.
Covers:
- Two compounding causes (PCIe bandwidth halved, weaker CPU per-core).
- Symptom (asymmetric GPU util during decode — communication-starved
TP=2).
- Three concrete mitigations: enable persistence mode, prefer single-
card paths over dual.yml, raise host RAM allocation on VMs.
- Worked example linking to issue #137 (Xeon Gold 6138 + Gen 3) and
@lolren disc #18 (Ryzen 5950X + Gen 4) for the side-by-side.
Follows from comment on #137 that promised this addition.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per @Mateleo's feedback in disc #129: the power-cap table rows showing
370W stock / 390W max for @noonghunna's 3090 are specific to a high-OC
SKU. Reference / Founders Edition 3090s top out at 350W stock.
Efficiency-knee findings (290W decode, 250W prefill) hold across SKUs —
those are SM 8.6 properties, not card-model-dependent. Added clarification
note at the top of the cross-rig power-cap section.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#128.
@JohnTheNerd's contribution — discussed in #98. Net -639 lines across 22 files.
What lands:
- scripts/detect_nvlink.sh — runtime NVLink detection at compose boot via
nvidia-smi topo -m. NVLINK_MODE=auto|force_on|force_off override.
- 4 Qwen NVLink composes (nvlink/turbo/dflash/dflash-noviz) collapsed
from 871 lines into 32-line stubs that 'extends:' the unified compose
with NVLINK_MODE=force_on. Back-compat preserved (same container_name,
same ports — switch.sh routes unchanged).
- 4 Qwen non-NVLink composes (docker-compose, turbo, dflash, dflash-noviz)
gain the unified entrypoint that sources detect_nvlink.sh and selects
--disable-custom-all-reduce based on detection.
- Same pattern extended to all 7 Gemma dual composes.
- launch.sh + switch.sh: doc-only updates noting the new env var + that
nvlink-* variants are stubs.
- docs/HARDWARE.md NVLink section rewritten.
Validated by author on their rig (full report.sh --full attached to PR).
Local smoke on canonical 2× 3090 PCIe (no NVLink bridge): unified dual.yml
boots clean, detect_nvlink.sh correctly reports 'PCIe topology (PHB) →
using PCIe mode', verify-full 8/8 PASS including MTP AL 2.67.
v0.6.1/v0.6.2 metadata preserved across all touched composes: Engine-profile
headers, ${TP:-N}/${PP:-1} env overrides, Requires-min-* fit metadata.
Co-Authored-By: John Karabudak <JohnTheNerd>
- HARDWARE.md: +2 rows in power-cap table — @eddietheengineer's 3090 air
(vLLM dual MTP, knee 210W / 0.144 TPS/W) and 3090 Ti air (knee 200W /
0.154 TPS/W, FIRST 3090 Ti data point on the matrix).
- BENCHMARKS.md: +1 multi4 row — @alanspires #127 on 6× 3090
VFIO-passthrough + AMD EPYC 7313 host. TP=4 (Qwen num_kv_heads=4 doesn't
divide 6 → GPUs 4-5 free). 74.93 / 92.85 TPS. First VFIO + virtualized
data point; scripts (verify-full / verify-stress / soak) all PASS
without modification.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
As discussed in #98, collapse 8 Qwen dual-card compose files into 4.
NVLink presence is now detected at startup instead of requiring
a separate compose per interconnect. The 4 nvlink-*.yml files become
deprecated stubs that extend the unified compose with NVLINK_MODE=force_on.
Add entrypoint conditionals and env-var substitution so the same compose
selects the correct NCCL settings and --disable-custom-all-reduce flag based
on detected topology. Extend the same pattern to all 7 Gemma dual composes.
Adds docs/DTYPE_MATRIX.md — a reference table mapping NVIDIA GPU
architectures (Pascal → Blackwell DC) to native vs emulated Tensor Core
support for each dtype + quant scheme.
Sections:
- At-a-glance compute-dtype matrix (9 archs × 10 dtypes, ✓/SW/✗ marks)
- Weight-quantization schemes (GPTQ / AWQ / AutoRound / NF4 / SmoothQuant
/ FP8 weights / MXFP8 / MXFP4 / NVFP4 / GGUF / HQQ-AQLM-SqueezeLLM)
with storage-vs-compute paths
- Weight-only vs weight+activation axis (W4A16 vs W8A8 vs W4A8)
- KV-cache dtype support (FP16 / FP8 / INT8 PTH / TQ3 / TQ4 / k8v4)
- Per-arch compose recommendations (which compose to ship for which
GPU class)
- Runtime detection (points at Genesis guards.py)
- Corner cases (Ada FP8 vs Hopper FP8, Blackwell consumer vs DC, NVFP4
vs MXFP4 block-size differences, MX* family overview, FP6)
- References (NVIDIA whitepapers, Marlin, Genesis, BENCHMARKS)
Cross-linked from:
- HARDWARE.md GPU-compat table
- GLOSSARY.md Quantization section
- FAQ.md as a new "What dtype/quant should I pick for my GPU?" Q under
Hardware
This sets the foundation for future per-arch compose optimization —
detecting compute capability at boot and picking the right KV dtype /
weight quant scheme automatically based on what the hardware actually
accelerates rather than what's nominally supported.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The dev rig had grown across three home dirs (`/opt/ai/compose/`, `/opt/ai/github/`,
`/home/wasif/`) and three repos (single-3090, dual-3090, club-3090). Disk-out
on `/` (97% used) forced a cleanup; rather than just prune, we consolidated
the layout across the whole stack while we were at it. This commit captures
what landed inside this repo.
Services consolidation:
- services/{ollama,openwebui,litellm,qdrant,searxng}/ migrated in from
/opt/ai/compose/<svc>/ (zero functional change — same docker-compose.yml).
- services/litellm/config.yaml rewritten: explicit routes for current
primaries (qwen3.6-27b-autoround → :8010, gemma-4-31b-autoround → :8030).
Removed `* → ollama/*` wildcard.
- services/comfyui/ migrated in (was /opt/ai/compose/comfyui/) — wired into
gpu-mode with full mutex against vLLM/SGLang.
scripts/gpu-mode.sh under git:
- Was a loose /opt/ai/gpu-mode.sh outside any repo. Symlinked at
/usr/local/bin/gpu-mode.
- Five Gemma 4 31B modes added: gemma, gemma-dflash, gemma-int8,
gemma-dflash-int8, gemma-awq.
- One ComfyUI mode (mutex with all LLM serving).
- prune / prune-all subcommands (safe image prune; aggressive variant adds
build cache --keep-storage 5GB + dangling networks).
- gpu-mode status now shows Docker disk + /var/lib/docker + /tmp sizes.
- compose_at() passes --env-file <repo>/.env so MODEL_DIR resolves
regardless of which compose dir gpu-mode cd's into. Fixes the recurring
"MODEL_DIR not set, defaulting to ../../../../../models-cache" warning.
- stderr no longer swallowed by compose_at() (real errors surface).
- Cross-model VRAM mutex: every Qwen mode stop_all_gemma + stop_comfyui
and vice-versa.
scripts/maintenance/ — new hygiene-tools subdir:
- list-image-pins.sh: engine-agnostic pin auditor. Scans every compose's
`image:` line, groups by `<repo>:<tag>`, flags pin-drift (multiple tags
per repo), ranks composes by patch surface.
Pin tracking:
- docs/UPSTREAM.md gains a "Pinned images" section: table of every pinned
image, why each pin exists, retirement candidate criteria.
- docs/NIGHTLY_BUMP_RUNBOOK.md (new): 7-step procedure for bumping pinned
engine images (scope → branch → patch survival → boot → verify-full +
verify-stress → bench delta → land → retire). Engine-specific notes
for vLLM nightly hashes, llama.cpp digest pinning, SGLang variants.
Path updates from the engine + model dir consolidation:
- /opt/ai/vllm-src/ → /opt/ai/engines/vllm/primary/
(in setup.sh, INTERNALS.md, several patch READMEs, docs/HARDWARE.md,
docs/FAQ.md, docs/DUAL_CARD.md, docs/UPSTREAM.md, models/qwen3.6-27b/
CHANGELOG.md)
- /mnt/models/gguf/qwen3.6-27b/ → /mnt/models/huggingface/qwen3.6-27b-gguf/
(in models/qwen3.6-27b/llama-cpp/{compose/single/*.yml, recipes/*.sh,
README.md}, docs/engines/LLAMA_CPP.md)
CHANGELOG.md narrative gap fill:
- 2026-05-10 entry for this reorg.
- 2026-05-09 entry for compose convention formalization (topology
promoted to dir level, profile schema, Status enum + Caveats, cliff
CI swap, Discord launch).
- 2026-05-08 entry for Gemma 4 INT8 PTH unblock + 262K validation.
- 2026-05-07 entry for power-cap-sweep campaign + HARDWARE.md cross-rig
charts + cross-rig benchmark rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two findings from @easel's deep cross-rig writeup on issue #102 (5090
Laptop WSL2):
1. **HARDWARE.md** — new "GPU memory budget on WSL2" subsection. WSL2
container CUDA-context consumes ~1.31 GiB before vLLM profiler runs.
`gpu_memory_utilization=0.95` crashes at boot; 0.944 works. Validated
on 2 machines. Formula: `(vram_total - 1.31) / vram_total`.
Recommendation: `GPU_MEMORY_UTILIZATION=0.94` in `.env` on WSL2.
2. **CLIFFS.md** — new section "Cliff 3 — DeltaNet SSM state is not
prefix-cacheable (the prefill cliff)". This is a structural finding
that explains a class of failures we'd been describing without
naming. @easel's warm-cache run (68.7% KV-block hit, turn 10 at
35.6K tokens took 577s — 2.3× the cold-start 254s) is the smoking
gun: prefix cache helps attention but DeltaNet's recurrent state
`h_t = f(h_{t-1}, x_t)` must be recomputed from scratch every turn.
PN32 fixes OOM stability; nothing fixes the O(n) prefill scaling
because the architecture itself is sequential.
Practical ceiling on single-card vLLM (any 24 GB Qwen3-Next config):
sub-30s TTFT only below 5K accumulated tokens. 22-35K = 3-4 min/turn.
~74K = 10+ min client timeout.
Elevates "for single-card agentic Qwen3-Next, use llama.cpp" from
implied to explicit in the docs. Dual-card extends the envelope to
25-30K accumulated; deep sessions (50K+) still need llama.cpp on
either topology.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
His earlier 15-cap sweep (260-400W, 16832066) is superseded by a
38-cap sweep at 10W resolution from 230-600W (16854218) — strictly
richer envelope. Two findings worth surfacing:
- Firmware boost-clock plateau at SM 2610 MHz: caps 400-600W produce
identical 51.96 TPS at 393W actual draw. Auto-plateau detector in
power-cap-sweep.sh caught both sub-plateaus cleanly (400-470W and
480-600W).
- decode-concurrent N=4 plateau lower than decode-single (46 vs 52
TPS) — on a 4090 + 27B Q3_K_XL the GPU is under-load at c=4 even
at full TDP, so single-stream wins. Useful cross-rig signal.
HARDWARE.md table + chart caption updated; chart regenerated with
both load-mode curves overlaid + plateau zone shaded.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three documentation/script follow-ups from @easel's #102 re-bench
on RTX 5090 Laptop:
- HARDWARE.md: new "Laptop GPUs — EC-managed power" subsection.
nvidia-smi -pl returns N/A on laptop-class GPUs (EC owns the
envelope, not the OS). Documents clock-lock as the only
software characterization path on laptops.
- CLIFFS.md: new "naming trap" callout in the KV-format section.
fp8_e5m2 is 8 bits/token; turboquant_3bit_nc packs 3 bits.
At 180K on 24GB, TQ3 fits where fp8 OOMs (4.36 GiB available
vs 6.64 GiB needed for fp8). Pin: TQ3 = long-context KV;
fp8 = short-context throughput.
- verify-stress.sh: auto-detect VLLM_ENFORCE_EAGER=1 in the
running container's env via docker inspect; when set, bump
STRESS_LONGCTX_TIMEOUT_S 300→600s and STRESS_TOOL_PREFILL_-
TIMEOUT_S 240→480s. Eager-mode prefill at 60K-140K runs
200-290s and was false-positiving as HTTP 000 (curl timeout)
in @easel's run. Both env vars also exposed for manual override.
Refs: noonghunna/club-3090#102
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same 3090 (GPU 0, air-cooled), same engine (mainline llama.cpp), same
Q4-class quant (Q4_K_XL). Only the model changes: dense Qwen3.6-27B vs
A3B 35B (3B active per token).
Findings:
1. MoE shifts decode sweet spot 80W lower:
- Dense decode: 290W → 0.111 TPS/W, SM 1380 MHz at sweet spot
- A3B decode: 210W → 0.546 TPS/W, SM 1290 MHz at sweet spot
Each token only activates 3B of 35B params on MoE → less compute per
token → bandwidth-bound knee fires at lower power.
2. Prefill sweet spot is workload-determined, NOT model-determined:
- Dense prefill: 250W → 3.633 TPS/W
- A3B prefill: 250W → 9.865 TPS/W
Both converge to same cap because prefill is compute-bound regardless
of MoE routing.
3. Boost-clock plateau depends on workload AND model:
- Dense decode: PLATEAU at 340-370W (SM 1560 MHz lock)
- A3B decode: NO PLATEAU (SM climbs smoothly 1875→1890→1890→1905)
- Dense prefill: PLATEAU at 330-370W (SM 1605-1620)
- A3B prefill: PLATEAU at 340-370W (SM 1680-1710)
Plateau auto-detection correctly flagged dense decode but not A3B
decode — confirming firmware operating-point selection responds to
compute pressure, not just to cap value.
Adds:
- docs/img/power-cap-3090-a3b-decode.py + .png
- docs/img/power-cap-3090-a3b-prefill.py + .png
- HARDWARE.md cross-rig table rows for both A3B sweeps
- HARDWARE.md "Same hardware, MoE workload" subsection with comparison
table + practical recommendation (A3B users → 210W cap, vs 290W for
dense Qwen)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@apnar found a Pareto improvement over our prior 5090 power-cap sweet
spot by switching from power-cap to clock-lock methodology. The 5090
has a 400W minimum power cap (nvidia-smi -pl rejects below 400W on this
card), so the power-cap-sweep methodology is blind to the entire <400W
envelope. Clock-locking via nvidia-smi -lgc + -lmc has no minimum-power
floor and surfaces operating points the power-cap sweep can't reach.
Headline findings (1× 5090 air, Gemma-4-31B-AutoRound + MTP K=3,
decode-concurrent N=6):
Power-cap sweet spot (prior):
400W cap → 571 narr TPS, ~400W draw, 1.43 TPS/W
Freq-cap peak efficiency (new):
7001 mem / 1635 GPU MHz → 428 narr TPS, 211W draw, 2.025 TPS/W
→ 1.42× more efficient at 47% less power, for 25% lower TPS
Freq-cap Pareto point (new):
14001 mem / 2122 GPU MHz → 602 narr TPS, 314W draw, 1.92 TPS/W
→ strictly better than 400W power-cap: +5% more TPS at -22% less power
The 35-point sweep proves memory clock is the dominant TPS variable on
Blackwell decode — at 405 MHz mem (lowest tested), TPS caps at ~53
regardless of GPU clock; at 14001 MHz mem (stock max), TPS scales
through 800+ at GPU 3090 MHz.
Adds:
- docs/img/freq-cap-5090-gemma4.py + .png
- docs/HARDWARE.md > Power > "Clock-locking on Blackwell" subsection
with the chart, per-workload operating-point recommendations table,
apply-via-nvidia-smi snippet, and caveats (Ampere not portable,
air-cooled-only, hand-rolled methodology pending freq-cap-sweep.sh
companion script).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related additions:
1. Plateau auto-detection. At end of sweep, scans for 3+ consecutive caps with
identical draw (±2W) and TPS (±1%) — the firmware boost-clock plateau pattern
we documented on this 3090 (caps 340-370W decode → SM 1560 MHz lock; caps
330-370W prefill → SM 1605-1620 MHz lock). When detected:
- [plateau detected] line emitted to stdout during sweep finalization
- "Detected boost-clock plateau(s)" section appended to summary file with
cap range, SM clock (single value or range), draw, TPS, and a directive:
"raise past Nw to step to the next firmware operating point"
Smoke-tested on this rig with --caps 340,350,360,370,380 — detected plateau
at 350-370W, SM 1575-1605 MHz, 333.57W, 34.96 TPS. Caps inside the plateau
are functionally equivalent — pick the LOWEST to save power for free TPS.
2. Recommended sweep chain docs. HARDWARE.md > Power section now has a
"Recommended sweep chain" subsection explaining when to run each load mode:
- decode-single: chat / IDE-agent (~8 min on 3090)
- prefill-heavy: RAG / long-context (~6 min)
- decode-concurrent --concurrency auto: multi-tenant (~8 min)
Sweet spots can differ across modes (we measured 290W decode vs 250W prefill
on the same 3090) — a single-mode sweep can mislead you about your card's
full operating envelope. For mixed workloads, take the min cap across the
modes you care about.
Same recommendation echoed in the script header comment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Re-ran both 21-cap sweeps with the new sampling fields (commit ab2796d).
The boost-state plateau hypothesized in the prior captions is now directly
evidenced by the SM-clock data:
decode-single (~8m sweep):
Caps 340/350/360/370W → SM 1560 MHz, draw 334W, TPS 34.66 — identical.
Cap 380W → SM 1635 MHz, draw 361W, TPS 35.56 (plateau escape).
prefill-heavy (~6m sweep):
Caps 330-370W → SM 1605-1620 MHz, draw 327W, TPS 1050 — identical.
Cap 380W → SM 1665 MHz, draw 355W, TPS 1080 (plateau escape).
Throttle stays at 100% across the plateau in both modes — firmware *is*
power-capping, but the cap it enforces is its own internal voltage/clock
setpoint, not the user-set software cap. This makes the prior "we think
this is a firmware plateau" framing concrete: it's a boost-clock lock,
released only at the next cap step.
Chart annotations updated to reference SM-clock evidence directly.
HARDWARE.md cross-rig table rows for @noonghunna's rig refreshed with
new TPS numbers + SM clock annotation. Sweet spots unchanged (290W
decode / 250W prefill); the new data refines the explanation rather
than the conclusion.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #99 (@easel) added the env-var hook to 8 PN59/GDN-class composes that
already had bash entrypoint wrappers. The 3 missing composes (dual.yml,
dual-nvlink.yml, dual4.yml) used direct command: arrays with no bash
wrapper, so they're brought into parity here by adding a minimal
entrypoint that does only the optional --enforce-eager interpolation.
Also documents the third WSL2 / laptop workaround in HARDWARE.md
alongside the TDR fix and expandable_segments:False pattern, plus a
combined three-variable .env template (GPU_MEMORY_UTILIZATION,
PYTORCH_CUDA_ALLOC_CONF, VLLM_ENFORCE_EAGER) that's the typical landing
point for WSL2 / laptop GPU rigs with reduced VRAM headroom.
Validation: docker compose config parses cleanly on all 3 modified
composes; bash interpolation behaves correctly (empty when unset,
--enforce-eager prepended when set).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "230W is the sweet spot" lore was stale — it traces to coarse 3-cap-resolution
data. Dense 10W-resolution sweeps on this rig now show:
- 290W: actual air-cooled decode sweet spot (0.111 TPS/W)
- 330W: water-cooled sweet spot (per @syangsao 3-cap data)
- 230W: NOT a sweet spot — costs ~16% efficiency vs 290W; just a low-power cap
Add decode-concurrent N=4 row to cross-rig table (290W also peaks here on this rig,
matching decode-single — concurrency doesn't move the knee). Add per-workload-class
table showing decode at 290W vs prefill at 250W on the same card. Update vLLM engine
doc to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends the time-bounded streaming-bench architecture (commit 7877c04, which
was decode-single only) to the other two load modes.
prefill-heavy:
- Adaptive prompt sizing: at sweep start, run a 1000-repeat-filler probe at
the highest sweep cap, measure TPS, calculate prompt size = TPS × N seconds
(TARGET_PREFILL_SECONDS=10 default). Use that prompt size across the sweep.
- At low caps the same prompt takes longer (TPS scales down with throttle),
but never underloads at high caps — solves the "10K-prompt at 390W finishes
in 9 sec, sampler under-served" issue from earlier today.
- Smoke on 3090 (190-390W, 21 caps): 5m36s total wall, vs 50+ min for old
token-bounded with default 50K filler.
decode-concurrent:
- Time-bounded streaming with N concurrent curls (same pattern as decode-single,
just N parallel streams).
- BENCH_RUNS=1 default (was 2) — single batch for sweep shape, multi-batch
via --bench-runs N for anchor data.
- Warns when --bench-runs > 1 since wall scales linearly.
- Smoke projection on 3090 with concurrency=4: ~8.4 min for 21 caps.
Codex Round 2 — applies their prior decode-single architecture wins to the
remaining load modes, after we caught that they hadn't been optimized when
running apnar's prefill comparison data and seeing 50+ min sweep projections.
Cross-card extrapolation (prefill-heavy + 21-31 cap range):
- 3090: ~6 min ✓ (measured)
- 5090: ~8-10 min (estimated)
- 4090: ~10-12 min (estimated)
3090 prefill-heavy chart added (docs/img/power-cap-3090-prefill.png) showing:
- Sweet spot at 250W (3.617 TPS/W) — different from decode's 290W
- Boost-state plateau 340-370W (326W actual draw) — same firmware behavior
as decode but different absolute draw level
- TPS rises 543 → 1097 across 190-390W cap range, smooth efficiency decline
HARDWARE.md adds 3 prefill-heavy anchor rows + chart embed with explicit
cross-workload comparison call-out (3090 has different sweet spot for prefill
vs decode on same rig, while 5090 has a much more dramatic difference where
decode is bandwidth-bound at ~91% cap-respect and prefill saturates 100%).
Note: the "boost-state plateau then escape" pattern reproduces across decode
and prefill on the 3090 — it's a firmware behavior of the card, independent
of workload class. Different from 5090 where the workload class itself is
the bottleneck.
Headline finding from @apnar's 4th + 5th sweeps in one day (disc #86):
- Decode N=8 (Gemma 4 + MTP): tops at 551W actual draw (vs 547W at N=4)
→ confirms decode is memory-bandwidth bound, not concurrency-limited
- Prefill-heavy (Qwen3.6 long-text): hits 599.98W actual draw at 600W cap
→ 99.997% cap-respect, full TDP saturation
Per-workload-class power ceiling on the 5090:
- Decode workloads: ~547-551W max (memory bandwidth limit, regardless of cap)
- Prefill workloads: ~600W (compute-bound, scales with TDP)
Both workload classes share efficiency knee at 400W cap (67% of stock TDP) —
that "60-85% of stock TDP" cross-rig pattern holds across workloads. But
absolute throughput behavior differs: decode at 600W gives only 4% more TPS
than at 400W (bandwidth ceiling); prefill at 600W gives 19% more TPS (compute
actually uses the watts).
Practical implication for 5090 deployments:
- Chat/IDE-agent: cap at 400W (huge efficiency win, ~5% TPS loss)
- RAG/long-context: leave at stock 600W (compute-bound, watts buy throughput)
- Mixed: 400W if chat-dominant; pure prefill loads suffer
New chart: docs/img/power-cap-5090-qwen36-prefill.png shows the curve with
the "599.98W actual at 600W cap" callout.
HARDWARE.md additions:
- 2 new BENCHMARKS rows for prefill-heavy 400W (sweet spot) + 600W (saturation)
- Inline embed of the prefill chart after the decode (Gemma 4 + MTP) chart
- New "Per-workload-class power ceilings" subsection with the cross-workload
bottleneck table
Note on apnar's contribution: he ran 5 sweeps total today (50W resolution
v1, 50W after calibration fix v2, 10W canonical anchor v3, decode N=8
headroom probe v4, prefill-heavy compute-saturation test v5) — all on the
old token-bounded bench architecture before today's time-bounded redesign
shipped (his sweeps at 13-14 UTC, my time-bounded redesign at 18:15 UTC).
The cross-validation depth is rare — most cross-rig contributions stop at
one sweep.
@noonghunna's rig is air-cooled (peak 76°C at 390W is air-typical;
water would be ~50-60°C). Earlier rows + chart caption were mislabeled
as water-cooled — that was an inference error on my part.
Updates:
- HARDWARE.md table rows: 3090 cooling water → air (3 rows)
- Chart caption: water-cooled → air-cooled, added GPU temp note
- Chart script subtitle/comments: water-cooled → air-cooled
- Refreshed PNG with corrected subtitle
Cross-rig context: we now have TWO air-cooled 1× 3090 anchor points
(@lamentofhighborne MTP build vs @noonghunna mainline) instead of one
air + one water. The "cooling-class delta" cross-rig comparison angle
is moot here — the build-class delta (MTP vs no-MTP) is what differs.
syangsao's 3-cap sparse data remains the only water-cooled 3090 anchor.
Replace token-bounded bench.sh per-cap with inline time-bounded streaming
curl. Per-cap wall is now constant ~23s regardless of cap or card class,
fixing the cross-card portability issue where token-counted benches took
2-4× longer at low caps than high caps.
Implementation:
- New flag --target-cap-seconds (default 10): seconds of streaming per
direction (narrative + code). Total per-cap = 2 × N + ~3s settle.
- Inline curl with stream=true, max_tokens=99999, --max-time=N. Curl exits
28 when wall budget hits — that's the bench boundary, not an error.
- TPS = streamed token-chunks / wall_seconds, with fallback to
usage.completion_tokens if engine emits final usage before timeout.
- Per-cap UTC start/end timestamps + wall-time line in output.
- Updated runtime estimate to use TARGET_CAP_SECONDS math for decode-single.
Validated 2026-05-07 on @noonghunna's 3090 water-cooled rig:
- 21-cap sweep (190-390W): 8m12s total wall (target was ≤12 min)
- Per-cap wall: 23.4s consistent (target was 15-45s)
- Power sampler: 35-37 util>50% samples per cap = 17-18.5s of data
(target was ≥10s)
- TPS values reproduce prior data within ~1% (290W: 32.16 narr / 0.111
TPS/W matches our prior 32.08 / 0.111 measurement)
Cross-card extrapolation (linear in cap count, not throttle):
- 5090 (300-600W, 31 caps): ~12 min
- 4090 (230-600W, 38 caps): ~15 min
- 3090 (190-390W, 21 caps): ~8 min ✓
Removed env-var token overrides for decode-single (BENCH_MAX_TOKENS_*
no longer apply since we don't pass token counts). BENCH_WARMUPS and
BENCH_RUNS_PER_CAP also moot for decode-single now. Other load modes
(decode-concurrent, prefill-heavy) unchanged.
Refreshed 3090 chart with clean 21-cap data (no more cold-cache anomaly
on cap 220W since time-bounded approach has no warmup/measurement gap).
HARDWARE.md canonical-command section updated to recommend decode-single
as the default and document the time-bounded methodology.
Implementation by Codex via mcp__codex__codex per brief at
docs/diagnostics/power-cap-sweep-cross-card-codex-brief.md.
First-party data: water-cooled dual-3090 rig (GPU 0 used), 18-cap
sweep 200-390W on mainline llama.cpp + Qwen3.6-27B Q3_K_XL.
Findings:
- Sweet spot at 290W cap (0.111 TPS/W, 32.1 narr / 31.7 code)
- 290W is 78% of stock 370W TDP — fits cross-rig "60-85% of stock"
pattern alongside 4090 (58%) and 5090 (67%) sweet spots
- Firmware boost-state plateau: caps 340-370W all draw identical
~334W actual (looks like a hardware ceiling but isn't)
- At 380W cap, actual draw escapes to 361W; at 390W cap, 388W
- Card temp peaked 75°C at 390W cap — water cooling has headroom
Mixed bench shapes during sweep:
- 200-210W: full bench (1+2 / 500+400 tokens) for higher-fidelity
data at the throttled-cap region
- 220W: cold-cache narr biased (skipped), code retained
- 230-240W: quick bench (0+1 / 250+200, no warmup, cache warm
from previous cap)
- 250-390W: quick bench with warmup=1 (1+1 / 250+200) for clean
per-cap measurements
TPS values are wall-time throughput so directly comparable across
bench shapes. Caveat noted in chart caption.
BENCHMARKS row added for sweet spot + stock + max caps. Now have
charts for 3090 / 4090 / 5090 — completes the consumer-class
cross-rig anchor set.
Visualizes @laurimyllari's 15-cap 10W-resolution sweep from disc #62:
- Sweet spot at 260W cap (0.186 TPS/W)
- Workload-saturated: +8% TPS for +54% wattage going from 260W to 400W
- 4090's 450W stock TDP is well above the efficiency knee on this workload
Source script (docs/img/power-cap-4090-qwen36.py) parses raw data from
laurimyllari's attached log for reproducibility / future re-rendering.
Now have charts for 5090 + 4090. 3090 chart pending — will use first-party
data (water-cooled rig) once driver mismatch is resolved.
Visualizes @apnar's 21-cap 10W-resolution sweep from disc #86 with
three signals on one chart:
- Narrative TPS (peaks 619 at 510W cap)
- Code TPS (peaks 773 at 490W cap)
- Efficiency TPS/W (monotonic 1.43 → 1.10 from 400W → 600W)
Sweet-spot annotation at 400W (1.43 TPS/W). Red-shaded zone 530-600W
shows workload-saturation (~547W max actual draw regardless of cap,
not thermal throttle — GPU temp peaked 66°C).
Includes the source script (docs/img/power-cap-5090-gemma4.py) so
future contributors can regenerate or adapt the chart for other rigs.
Generated via uvx --from matplotlib --with numpy.
Embedded in HARDWARE.md right after the cross-rig power-cap data table.
Pairs with --concurrency auto. After plateau-detect picks N, adds N more
streams to probe whether the card has compute headroom unused by the
plateau-safe pick.
Motivation from disc #86: apnar's 5090 + Gemma 4 + MTP sweep showed
actual draw plateau at 547W against a 600W cap — only 91% cap-respect
even with N=4 at 99% util. That 53W gap could be:
(a) memory-bandwidth-bound decode (typical, fundamental)
(b) N=4 not exercising compute (concurrency-contention artifact)
(c) firmware/voltage cap below spec'd TDP
Distinguishing them previously required either editing --concurrency
manually or switching to --load-mode prefill-heavy. The new flag lets
contributors probe (b) cleanly:
sudo bash scripts/power-cap-sweep.sh \
--cooling air \
--load-mode decode-concurrent \
--concurrency auto \
--concurrency-stretch 4 \
--bench-runs 3
Validation:
- Rejects negative integers and non-numbers ("[error] --concurrency-stretch
must be a non-negative integer")
- Requires --concurrency auto ("[error] --concurrency-stretch only applies
with --concurrency auto")
- Default 0 (no behavior change for existing users)
docs/HARDWARE.md gets a new "Interpreting 'draw plateaued below cap'
sweeps" subsection — covers four common patterns (memory-bandwidth-bound,
thermal throttle, undersized workload, firmware cap) with what-to-try-next
for each. Surfaces both --concurrency-stretch and --load-mode prefill-heavy
as the diagnostic tools.
Apnar isn't asked to re-run; current data is already a headline anchor.
The flag exists so future cross-rig contributors who hit the same pattern
have a one-flag debugging path instead of needing to manually override
--concurrency or switch load modes blindly.
Three operating points from @apnar's full 21-cap 10W sweep:
- 400W (efficiency winner): 571 narr / 701 code, 1.429 TPS/W
- 510W (narr peak): 619 narr / 724 code, 1.215 TPS/W
- 600W (stock baseline): 601 narr / 757 code, 1.103 TPS/W
Cross-workload pattern emerges combining apnar's two 5090 sweeps:
both Qwen3.6-27B AutoRound and Gemma 4 31B + MTP land at the same
~400W efficiency sweet spot despite ~5× different absolute TPS scales.
Updated 5090 compute-saturation note to reflect this is workload-
independent on consumer-air-cooled 5090.
Hardware-physical ceiling for Gemma 4 + MTP at concurrency=4:
~547W actual draw, no thermal throttle (66°C peak). Above 530W
cap = wasted budget.
Validates the calibration fix shipped at 29e7de5: at 600W cap with
new logic (N=4 plateau-detected), TPS jumps from 499/616 (old N=6)
to 600/757 — pure calibration win, +20-25% same-cap TPS.
Calibration fix in 29e7de5 + apnar's re-run on disc #86 surfaced that
the canonical anchor-data invocation needs to be more explicit. Two
risks: (1) default --step-size 10 is right but coarse-step suggestions
(--step-size 50) leave too few data points to find the knee, (2) without
--bench-runs 3, single-batch variance can produce 10-30% adjacent-cap
deltas that are noise rather than signal.
HARDWARE.md: adds full canonical command with all four flags (cooling,
load-mode, concurrency auto, bench-runs 3) plus rationale for each.
Explicitly notes --step-size 10 is the default and not to override
unless you know why.
CONTRIBUTING.md: bumps the inline example to the same canonical form.
A single-card RTX 3090 Ti rig on WSL2 (driver 596.36) hits
RuntimeError: CUDA driver error: device not ready from gptq_marlin_repack
immediately after weight load on the v7.72.2-uplift nightly pin.
Bisect ruled out Genesis, spec-decode, TQ3 KV, async-residual error,
and TDR (registry already extended + Windows rebooted).
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False resolves it. Same
env-var workaround as JusefPol's NVLink boot-crash (PR #31), already
hardcoded in the dual-nvlink*.yml composes.
Replaces the hardcoded PYTORCH_CUDA_ALLOC_CONF line in 14 single-card
and PCIe dual-card composes with a ${PYTORCH_CUDA_ALLOC_CONF:-...}
override (defaults preserved). Pattern matches existing MAX_MODEL_LEN /
GPU_MEMORY_UTILIZATION overrides from #79. The two dual-nvlink*.yml
composes are unchanged — their existing JusefPol-driven default already
has expandable_segments off.
Documentation:
- docs/HARDWARE.md: new "disable PyTorch expandable_segments" subsection
alongside the TDR fix, with stack trace, what was ruled out, override
recipe, and a single uncontrolled observation about weight-load time
(32 sec → 13 sec).
- docs/FAQ.md: WSL2 question now cross-links both the TDR and
expandable_segments fix subsections.
- .env.example: documents the override under "vLLM tuning knobs".
- CHANGELOG.md (top-level + per-model): dated 2026-05-06 entries.
The exact failing call hasn't been isolated. The cuMemMap virtual-memory
API used by expandable_segments:True is the suspected culprit since
both known occurrences respond to the same workaround, but no specific
call has been proven to return cudaErrorNotReady.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@apnar's 5090 (air-cooled, vLLM + Qwen3.6-27B-AutoRound) data on
disc #62 shows two anchors — 400W (peak efficiency on this workload)
and 575W (near-stock comparison). Critical finding: card maxes at
~430W actual draw regardless of cap because the workload is
compute-saturated for the 5090's compute envelope. Implication: the
knee will shift higher on larger models that actually use the 5090.
Replaces "5090 anchor pending" with the actual data + a note pointing
to the new dedicated discussion thread (disc #86 — Cross-rig power-cap
efficiency matrix) where future sweeps will land.
Wraps bench.sh in a sudo-aware loop that:
- auto-detects running container + URL + model (no env-var fiddling)
- iterates user-specified power caps via nvidia-smi -pl
- captures wall TPS narr/code, actual power draw, GPU temp, TPS/W per cap
- emits paste-ready markdown summary at /tmp/power-cap-summary.md
Required flag: --cooling air|water|aio (or omit for "unspecified" with a
warning). Cooling class is essential context — air-cooled cards
thermal-throttle at 80-83 °C and effectively cap below the software limit;
water-cooled cards sustain full board power. Same software cap on
different cooling produces different curves.
Required because @syangsao's three-cap data on issue #58 set the 330W
production-default recommendation, and @laurimyllari's 4090 sweep on
disc #62 surfaced a 260-280W knee on Ada — both done by hand. The script
makes future cross-rig sweeps reproducible without each contributor
re-deriving the bash invocations.
Smoke-tested 2026-05-06 on 2× RTX 3090 + Gemma 4 31B MTP: --caps 280,330
boot + bench + reset cycle clean, summary header captures GPU/cooling/
model/engine/endpoint/date plus the cross-rig comparison-fairness note.
Updates docs/HARDWARE.md power section:
- adds @laurimyllari's 4090 anchor rows alongside @syangsao's 3090 rows
- documents the "knee at 60-85% of stock TDP" cross-rig pattern
- references the script as the canonical way to add new anchors
- placeholder for 5090 anchor (cross-rig ask pending @efschu / @apnar)
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Air-cooled 3090s thermal-throttle to ~310-340W effective under sustained
decode load regardless of software cap, so the 388W→330W gap mostly
disappears for them. The 330W cap recommendation mainly benefits
liquid-cooled rigs that can actually sustain full board power.
Per follow-up to @syangsao thread — his 38 TPS at 388W is achievable
because of the Alphacool Eiswolf 2 360mm AIO; air-cooled users would
likely already be at 330W-equivalent without any cap.
@syangsao's three-point sweep (230W/330W/388W stock) on 1× water-cooled 3090
+ llama.cpp + Qwen3.6 27B Q3_K_XL revealed 230W costs ~34% TPS (25 vs 38)
on this engine path — far larger than the "<10%" framing in prior docs. The
chunked_gated_delta_rule kernel is genuinely compute-bound on GDN-attention
models, so power cap throttles SM clocks ~linearly.
330W is the actual sweet spot: peak TPS/W efficiency, only ~5% TPS loss vs
388W stock, and 388W is *less* efficient than 330W on this kernel mix.
Updates docs/HARDWARE.md power section with the cross-rig data table and
flips the recommended default. docs/engines/VLLM.md gets a smaller note
flagging the engine-specific difference.
Source: https://github.com/noonghunna/club-3090/issues/58#issuecomment-4388766174
* composes: env-override knobs MAX_MODEL_LEN + GPU_MEMORY_UTILIZATION
Two cross-rig users hit the same friction within hours of each other:
- @laurimyllari (4090, disc #62 / issue #71): default max_model_len=180000
on long-text.yml exceeded his rig's KV-cache budget; had to drop to 90K
- @PiotrZadka (disc #66): wants to run vLLM alongside a desktop session
on the same GPU; needs to reserve some VRAM for X server / browser etc.
Both root causes are the same: composes were calibrated for headless 3090
with no other VRAM consumers, and there's no clean override path short of
hand-editing the YAML.
Add env-substitution for the two knobs with the highest "shrink to fit"
elasticity:
--max-model-len ${MAX_MODEL_LEN:-<existing default>}
--gpu-memory-utilization ${GPU_MEMORY_UTILIZATION:-<existing default>}
Existing defaults preserved verbatim — zero behavior change for users who
don't set the env. Pattern matches existing ${MODEL_DIR:-...},
${PORT:-...}, ${HF_TOKEN:-...} substitutions in our composes.
Usage:
# Default (unchanged):
bash scripts/switch.sh vllm/long-text # max_model_len=180000
# Override for desktop-coexist or sub-24 GB VRAM:
MAX_MODEL_LEN=32768 GPU_MEMORY_UTILIZATION=0.80 \\
bash scripts/switch.sh vllm/long-text
Validated end-to-end on long-text.yml:
$ MAX_MODEL_LEN=32768 GPU_MEMORY_UTILIZATION=0.80 \\
bash scripts/switch.sh vllm/long-text
$ docker exec vllm-qwen36-27b-long-text ps aux | grep vllm
... --max-model-len 32768 --gpu-memory-utilization 0.80 ...
Default boot also unaffected:
$ docker compose -f .../docker-compose.long-text.yml config | \\
grep -A1 "max-model-len\|gpu-memory-utilization"
- --max-model-len
- "180000" # original default preserved
- --gpu-memory-utilization
- "0.93" # original default preserved
18 vLLM composes touched. Mechanical find/replace; one YAML pattern,
no logic changes. Doesn't touch llama-cpp composes (different flag
shape; out of scope for this PR).
Closes the friction reported on disc #62 + disc #66.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: env-override knobs for desktop-coexist + sub-24 GB usable VRAM
Document MAX_MODEL_LEN + GPU_MEMORY_UTILIZATION env overrides shipped in
the composes. Two surfaces:
- docs/SINGLE_CARD.md "Running alongside a desktop / sub-24 GB usable
VRAM" — explains the override pattern, when to drop which knob, safe
ranges. Calls out empirically that GPU_MEMORY_UTILIZATION=0.80 is
too aggressive for TQ3 KV paths (vLLM profiling overhead consumes
more than the saved 0.05 budget; engine init reports
'No available memory for the cache blocks').
- docs/HARDWARE.md "Note for sub-24 GB cards" — adds a 4090-with-display
paragraph above the existing 20 GB modded-3080 note. Cites
@laurimyllari's MAX_MODEL_LEN=90000 fit on 4090 long-text.yml.
Validated end-to-end during the env-override implementation:
- ps aux inside container confirms override values reach vLLM CLI
- Default boot unchanged (env unset → original values reach CLI)
- 0.80 mem-util on long-text dies cleanly with the documented error,
which is what surfaced the "stay 0.85-0.92 for TQ3" guidance
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
