--gpus selection was NVIDIA_VISIBLE_DEVICES-only, which CDI runtimes
(NixOS hardware.nvidia-container-toolkit, nvidia-ctk cdi) IGNORE — mog's
3x3090 rig got all cards exposed and vLLM took the wrong two. Index-based
CUDA_VISIBLE_DEVICES can't fix it either: the classic runtime renumbers
the exposed set in-container.
UUIDs solve both at once:
- launch.sh --gpus N,M resolves indices -> GPU UUIDs (nvidia-smi -i) and
exports BOTH NVIDIA_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES as UUIDs.
Classic runtime: exposes-by-UUID; CUDA mask agrees (renumbering-proof).
CDI: exposure comes from device_ids (nvidia.com/gpu=all) and the CUDA
UUID mask is what pins the cards. Index fallback if the query fails.
- 26 composes gain a bare `- CUDA_VISIBLE_DEVICES` passthrough (unset ->
absent -> zero change for existing users); stale commented hints removed.
- preflight: selector consumers are host-index-based -> new
_preflight_selector_normalize maps GPU-xxxx tokens back to indices at
the single choke point (the first live boot caught this: preflight
rejected the UUID selector).
- new guard test-compose-gpu-mask-passthrough.sh: every compose with the
NVIDIA env line must pass CUDA_VISIBLE_DEVICES through + launch.sh must
keep the UUID resolution.
- docs/HARDWARE.md: "Pinning specific GPUs (and CDI / NixOS runtimes)" —
the runtime table, the CDI deploy-block swap, the in-container
renumbering gotcha, manual recipes.
LIVE-VERIFIED on this rig (classic runtime, 2x3090):
launch.sh --variant vllm/minimal --gpus 1
-> host GPU0 = 1 MiB (empty), GPU1 = 20.7 GB (model), both env vars
UUID-set in-container. CDI leg validation goes to mog (NixOS reporter).
Follow-up (tracked on #610): estate ESTATE_GPUS UUID-ization for CDI
multi-instance rigs; post-boot visible-GPU-count sanity check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
510a76c885