510a76c885
--gpus selection was NVIDIA_VISIBLE_DEVICES-only, which CDI runtimes (NixOS hardware.nvidia-container-toolkit, nvidia-ctk cdi) IGNORE — mog's 3x3090 rig got all cards exposed and vLLM took the wrong two. Index-based CUDA_VISIBLE_DEVICES can't fix it either: the classic runtime renumbers the exposed set in-container. UUIDs solve both at once: - launch.sh --gpus N,M resolves indices -> GPU UUIDs (nvidia-smi -i) and exports BOTH NVIDIA_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES as UUIDs. Classic runtime: exposes-by-UUID; CUDA mask agrees (renumbering-proof). CDI: exposure comes from device_ids (nvidia.com/gpu=all) and the CUDA UUID mask is what pins the cards. Index fallback if the query fails. - 26 composes gain a bare `- CUDA_VISIBLE_DEVICES` passthrough (unset -> absent -> zero change for existing users); stale commented hints removed. - preflight: selector consumers are host-index-based -> new _preflight_selector_normalize maps GPU-xxxx tokens back to indices at the single choke point (the first live boot caught this: preflight rejected the UUID selector). - new guard test-compose-gpu-mask-passthrough.sh: every compose with the NVIDIA env line must pass CUDA_VISIBLE_DEVICES through + launch.sh must keep the UUID resolution. - docs/HARDWARE.md: "Pinning specific GPUs (and CDI / NixOS runtimes)" — the runtime table, the CDI deploy-block swap, the in-container renumbering gotcha, manual recipes. LIVE-VERIFIED on this rig (classic runtime, 2x3090): launch.sh --variant vllm/minimal --gpus 1 -> host GPU0 = 1 MiB (empty), GPU1 = 20.7 GB (model), both env vars UUID-set in-container. CDI leg validation goes to mog (NixOS reporter). Follow-up (tracked on #610): estate ESTATE_GPUS UUID-ization for CDI multi-instance rigs; post-boot visible-GPU-count sanity check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Gemma 4 26B-A4B MoE — on 2× RTX 3090
Run Gemma 4 26B-A4B — a 128-expert MoE with ~4B active params, vision, and tool calling — on 2× RTX 3090s.
⭐ v0.7.3 onboarding target. AWQ production path validated; Intel AutoRound INT4 blocked on Ampere (Marlin K-dim alignment).
Deployment
See docs/DUAL_CARD.md for workload-driven config picks. TL;DR:
| Config | Max ctx | Narr / Code TPS | Best for |
|---|---|---|---|
vllm/gemma4-26b-a4b-tp2 |
32K | 139 / 139 | General-purpose, vision + tools |
Run via:
bash scripts/launch.sh --variant vllm/gemma4-26b-a4b-tp2
Models
- Target:
Intel/gemma-4-26B-A4B-it-int4-mixed-AutoRound(~16 GB, MoE expert layers quant-mixed) - AWQ alternative: cyankiwi AWQ-4bit weights (validated production path on Ampere)
- Draft: TBD (Google's
gemma-4-26B-A4B-it-assistantavailable, compose pending)
Key details
| Aspect | Notes |
|---|---|
| Arch | MoE — 128 experts × 8 active, ~4B active params |
| Quants | AWQ-4bit (production), Intel AutoRound INT4 (Ampere-blocked for now) |
| KV | bfloat16 |
| Vision | ✅ Yes (off by default in base compose for first-boot validation) |
| Tools | ✅ --tool-call-parser gemma4 |
| NVLink | Auto-detected via NVLINK_MODE env var |
Upstream tracker
- vLLM PR #40886 — compressed-tensors MoE key remapping (vendored for AWQ path)
- Discussion #67 — first Ampere consumer cross-rig data thread