Adds `gpu-mode dgemma` (alias `diffusiongemma`): stops all other GPU models
(it's TP=2, both cards — shared 8199 slot with deckard) then launches the
DiffusionGemma compose (base.yml, PORT=8199) + wires it into Open WebUI.
Mirrors mode_deckard. Also stops it in mode_off.
Dual-card, so before switching to another model run `gpu-mode off` first
(single-card modes don't individually stop it — noted in usage). The mode
launches whatever base.yml is on the tree (the official :gemma image once
the dgemma image-adoption PR lands).
Validated live on 2x RTX 3090: `gpu-mode dgemma` stops peers, recreates the
container, starts services, owui-registers :8199, serves. bash -n clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
`gpu-mode` is a fixed mode dispatcher; there was no way to launch the
newly-promoted Deckard-40B through it. Add a `deckard` mode mirroring the
27b/gemma pattern:
- DECKARD_DIR + start_deckard/stop_deckard helpers (compose_at on the
piehsoft-q6k/mtp.yml compose; MODEL_DIR resolves from .env via --env-file,
same as the vLLM modes).
- mode_deckard: stops all other GPU models (Deckard layer-splits across both
cards), boots Deckard on :8199, brings up the support services, and wires it
into Open WebUI via the switch.sh --owui helper (best-effort; Deckard isn't
in the LiteLLM gateway config).
- stop_deckard added after every `stop_all_27b` call site (all 10 GPU-exclusive
modes) so switching AWAY from Deckard frees both cards first.
- usage() + dispatch case arm.
Live-validated: `gpu-mode deckard` boots the piehsoft-q6k project, container
healthy on :8199 serving deckard-40b, mount /mnt/models/huggingface, OWUI
connection recognized. bash -n clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Both 3090s normally run capped at 230W (quiet/cool), enforced at boot by
the enabled nvidia-power-cap.service oneshot. The cap suppresses benchmark
TPS — a documented artifact (e.g. an ik_llama "+18-20%" result that was
really just 230W vs 370W) — so maintainers need a quick way to uncap for a
true-TPS bench, then re-cap.
Adds `gpu-mode power-cap <on|off|status>` (alias: `powercap`):
- on: restart nvidia-power-cap.service (reusing the service as the single
source of truth for 230W). restart, NOT start: the unit is already
active from boot, and `start` on a RemainAfterExit oneshot is a no-op
that would silently fail to re-apply. Falls back to direct
`nvidia-smi -pl 230` if the service is missing.
- off: read each card's Default Power Limit from nvidia-smi and apply it.
Defaults differ per card here (GPU0 370W, GPU1 420W), so the value is
read, never hardcoded; nvidia-smi has no reset flag. Session-scoped:
a reboot or driver reload re-caps via the service (never disabled).
- status: print per-GPU enforced/default/min/max limits.
Also surfaces a one-line per-card cap state in `gpu-mode status` and echoes
the resulting enforced limit after on/off. Errors loudly (no nvidia-smi, no
sudo) in the script's existing style.
Live-tested on the rig: status -> off (370/420) -> status -> on (230/230),
verified each transition with `nvidia-smi -q -d POWER`; left at 230W.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Match gpu-mode.sh to the #279 dual prune: remove the 27b-turbo/27b-dflash/27b-dflash-noviz modes (they launched deprecated composes; defensive stop_ helpers kept), and repoint Gemma so gemma -> vllm/gemma-int8 default (:8032, 262K+vision), new gemma-mtp -> bf16 32K fallback (:8030), gemma-int8 kept as alias. bash -n clean; suite 38/38.
Every gemma-4-31b vLLM compose pinned a purged Docker Hub nightly
(e47c98ef / bf610c2f), un-bootable for fresh users (#250, #167 class).
Prune the 9-variant set to 3 and repoint survivors onto the immutable
stable release tag vllm/vllm-openai:v0.21.0 (never :latest).
Survivors: vllm/gemma-mtp (bf16 dual, default), vllm/gemma-int8 (int8-PTH
dual), vllm/gemma-mtp-tp1 (fp8 single). Dropped: gemma-dflash,
gemma-dflash-int8, gemma-int8-tq3, gemma-bf16, gemma-awq; the 262K path
folds into gemma-int8 via a CTX env override. Gemma is decoupled from the
shared Qwen nightly profiles via a new vllm-gemma-stable engine profile;
Qwen nightly profiles are untouched.
Diagnose decoupling: the pruned gemma dflash patch dirs doubled as the
diagnose tool's cross-model disk-source proxies for two Qwen overlays
(vllm-pr41703-dflash, vllm-pr42102-dflash-kv-quant). Those capabilities
are image-baked in the pinned nightly, not mountable files, so mark them
image_baked and have diagnose skip the disk-source check for image-baked
overlays. Runtime-neutral (no Qwen compose mounts them).
switch.sh launcher: handle the new VLLM_IMAGE engine-pin export (the twin
loop in launch.sh had it; switch.sh did not -> "unexpected engine pin
export"), and guard the VLLM_NIGHTLY_SHA echo for the image-only stable
profile (was unbound under set -u).
Validated on 2x RTX 3090 (Ampere, v0.21.0): full scripts/tests/*.sh suite
35/0; live boots of gemma-mtp (bf16) and gemma-int8 serve coherent output,
qwen vllm/dual regression-clean. gemma-mtp-tp1 (fp8_e4m3) is correctly
SM-gated (required_sm=9.0) -- confirmed Ampere-incompatible (Triton
"fp8e4nv not supported on sm_86"), a documented 32 GB+/Hopper variant;
beellama remains the single-card gemma path on Ampere.
Refs #451, #250, #167.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
PR-A (#231) inserted the <quant>/ layer; two compose-path references
outside the doc sweep were left stale:
- scripts/gpu-mode.sh (the rig mode-switcher, symlinked into
/usr/local/bin) cd'd into the dual/ topology dir and -f'd bare
filenames (turbo.yml, docker-compose.yml, ...) that PR-A moved under
<quant>/. Repointed DUAL_27B_DIR / GEMMA_DUAL_DIR at the autoround-int4
quant dir (mount-safe — same cd-into-compose-dir invocation switch.sh
uses), added GEMMA_DUAL_AWQ_DIR for the awq mode, and renamed the two
ex-docker-compose.yml defaults (qwen -> fp8-mtp.yml, gemma -> bf16-mtp.yml).
All 13 mode targets verified to resolve; bash -n clean.
- vllm-pr40798 / vllm-pr40914 patch READMEs cited dual/int8-tq3.yml (the
pre-PR-A name). Commit 6182922 split int8-tq3 into the tq3-* family; the
READMEs describe "TQ3 KV + MTP n=3" = tq3-mtp.yml (whose tombstone header
cites #40914) -> dual/autoround-int4/tq3-mtp.yml.
Surfaced (not fixed here — PR-B's registry-derivation eliminates it):
launch.sh LAUNCH_VARIANT_COMPOSE[vllm/gemma-mtp] points at a non-existent
dual/autoround-int4/fp8-mtp.yml; gemma dual default is bf16-mtp.yml (the
registry is correct). test-launch-compat doesn't sweep launch.sh paths.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
All llama.cpp + ik single composes shared one default container_name, so a
second variant hit a docker "name in use" conflict (ESTATE_CONTAINER override
existed but was undocumented). Give each non-base variant a distinct default:
- llamacpp/mtp → llama-cpp-qwen36-27b (base, unchanged)
- llamacpp/mtp-vision → llama-cpp-qwen36-27b-vision
- ik/iq4ks-mtp → ik-llama-qwen36-27b (base, unchanged)
- ik/iq4ks-mtp-vision → ik-llama-qwen36-27b-vision
- ik/iq4ks-two-stage → ik-llama-qwen36-27b-two-stage
The base names (mtp / iq4ks-mtp = the default workhorses) are kept so existing
hardcoded refs and benchlocal `docker logs <base>` keep working.
launch.sh: track the two renamed variants in LAUNCH_DEFAULT_CONTAINER, and add
the missing ik-llama/iq4ks-two-stage entry (it fell back to the wrong default
container before). gpu-mode.sh: comment-only update for the new naming.
Audit (all clear): the autodetect globs in preflight/soak/switch/health/update
are prefix-anchored (`^llama-cpp-qwen36-27b`, no end anchor) and gpu-mode uses
`== llama-cpp-*`, so the suffixed names still match — verified.
Ports still default to 8020 (concurrency uses ESTATE_PORT, per the issue's own
example) — documented with a "run two variants at once" snippet in SINGLE_CARD.
Validated: `docker compose config` resolves each distinct name; all names still
match the autodetect glob; bash -n + test-launch-compat pass.
Closes#169.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Post-v0.8.3 the llama.cpp profile family lands containers on :8020
(llamacpp/default/mtp) and :8030 (llamacpp/mtp-vision). The status block was
missing :8020 entirely and hardcoded :8030 as "gemma-mtp / Gemma 4 31B".
Add :8020 probe, and on :8030 detect the active container so the label tracks
whichever engine (vLLM Gemma vs llama.cpp Qwen) is actually serving.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Before this fix, gpu-mode off only stopped services it knew about
(stop_all_27b vLLM variants, stop_all_gemma vLLM variants, stop_comfyui,
and the SERVICES array). Instances booted via launch.sh --estate or
--estate-file persist via Docker `restart: unless-stopped` and so kept
running after gpu-mode off — appearing to "auto-restart". The user had
to know about launch.sh --down-estate to clean them up.
New stop_estate function:
- Reads ~/.club3090/estate.yml (constant ESTATE_YAML)
- No-ops if file missing, launch.sh unavailable, or estate list empty
- Otherwise calls launch.sh --down-estate <yaml> which cleanly removes
each estate-managed container + its docker network
Called from mode_off only — keeps blast radius small. Other mode
switchers (mode_27b, mode_gemma, etc.) aren't touched on purpose:
estate instances and gpu-mode classic services can in principle share
a rig at TP=1 + small contexts, so the user should choose explicitly
whether to keep them up when switching gpu-mode targets. mode_off is
the "stop everything" intent — clearly should include estate.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The dev rig had grown across three home dirs (`/opt/ai/compose/`, `/opt/ai/github/`,
`/home/wasif/`) and three repos (single-3090, dual-3090, club-3090). Disk-out
on `/` (97% used) forced a cleanup; rather than just prune, we consolidated
the layout across the whole stack while we were at it. This commit captures
what landed inside this repo.
Services consolidation:
- services/{ollama,openwebui,litellm,qdrant,searxng}/ migrated in from
/opt/ai/compose/<svc>/ (zero functional change — same docker-compose.yml).
- services/litellm/config.yaml rewritten: explicit routes for current
primaries (qwen3.6-27b-autoround → :8010, gemma-4-31b-autoround → :8030).
Removed `* → ollama/*` wildcard.
- services/comfyui/ migrated in (was /opt/ai/compose/comfyui/) — wired into
gpu-mode with full mutex against vLLM/SGLang.
scripts/gpu-mode.sh under git:
- Was a loose /opt/ai/gpu-mode.sh outside any repo. Symlinked at
/usr/local/bin/gpu-mode.
- Five Gemma 4 31B modes added: gemma, gemma-dflash, gemma-int8,
gemma-dflash-int8, gemma-awq.
- One ComfyUI mode (mutex with all LLM serving).
- prune / prune-all subcommands (safe image prune; aggressive variant adds
build cache --keep-storage 5GB + dangling networks).
- gpu-mode status now shows Docker disk + /var/lib/docker + /tmp sizes.
- compose_at() passes --env-file <repo>/.env so MODEL_DIR resolves
regardless of which compose dir gpu-mode cd's into. Fixes the recurring
"MODEL_DIR not set, defaulting to ../../../../../models-cache" warning.
- stderr no longer swallowed by compose_at() (real errors surface).
- Cross-model VRAM mutex: every Qwen mode stop_all_gemma + stop_comfyui
and vice-versa.
scripts/maintenance/ — new hygiene-tools subdir:
- list-image-pins.sh: engine-agnostic pin auditor. Scans every compose's
`image:` line, groups by `<repo>:<tag>`, flags pin-drift (multiple tags
per repo), ranks composes by patch surface.
Pin tracking:
- docs/UPSTREAM.md gains a "Pinned images" section: table of every pinned
image, why each pin exists, retirement candidate criteria.
- docs/NIGHTLY_BUMP_RUNBOOK.md (new): 7-step procedure for bumping pinned
engine images (scope → branch → patch survival → boot → verify-full +
verify-stress → bench delta → land → retire). Engine-specific notes
for vLLM nightly hashes, llama.cpp digest pinning, SGLang variants.
Path updates from the engine + model dir consolidation:
- /opt/ai/vllm-src/ → /opt/ai/engines/vllm/primary/
(in setup.sh, INTERNALS.md, several patch READMEs, docs/HARDWARE.md,
docs/FAQ.md, docs/DUAL_CARD.md, docs/UPSTREAM.md, models/qwen3.6-27b/
CHANGELOG.md)
- /mnt/models/gguf/qwen3.6-27b/ → /mnt/models/huggingface/qwen3.6-27b-gguf/
(in models/qwen3.6-27b/llama-cpp/{compose/single/*.yml, recipes/*.sh,
README.md}, docs/engines/LLAMA_CPP.md)
CHANGELOG.md narrative gap fill:
- 2026-05-10 entry for this reorg.
- 2026-05-09 entry for compose convention formalization (topology
promoted to dir level, profile schema, Status enum + Caveats, cliff
CI swap, Discord launch).
- 2026-05-08 entry for Gemma 4 INT8 PTH unblock + 262K validation.
- 2026-05-07 entry for power-cap-sweep campaign + HARDWARE.md cross-rig
charts + cross-rig benchmark rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
