noonghunnaandClaude Opus 4.8
3970c2d7c5
Extends the #611 primitive to the estate (multi-cluster) path and adds a post-boot placement check — the foundation Phase A′ (cluster.sh) and C (c3 cluster UX) sit on. - scripts/lib/gpu-select.sh (NEW, shared lib): factors the #611 inline resolver out of launch.sh into gpu_select_indices_to_uuids / gpu_select_export, plus gpu_select_container_uuids + gpu_select_assert_placement. launch.sh sources it (no re-inlined drift) and runs the placement assertion after verify-full. - estate_cli.py: resolve_gpu_uuids() (python twin of the bash resolver) + compose_env / compose_override_doc now UUID-pin CUDA_/NVIDIA_VISIBLE_ DEVICES (ESTATE_GPUS stays index-based — the compose device_ids read the host view). assert_placement[_quiet]() runs after each instance is ready (sequential → stderr, parallel → per-instance boot log), returning the {requested, actual, placement: ok|mismatch|unknown} verdict — the shape cluster.sh status + the c3 badge will read. - Placement uses --query-compute-apps=gpu_uuid (NOT --query-gpu): under CDI the container sees all cards but RUNS on the CUDA-masked set, so compute-apps is the runtime-agnostic ground truth. - test-gpu-select (NEW): asserts the bash + python resolvers agree on real hardware and both fall back identically; test-compose-gpu-mask-passthrough updated to follow the resolver into the lib. LIVE-VERIFIED (2x3090, classic runtime): `launch.sh --gpus 1` → UUID-pinned, verify-full 8/8, "✓ placement verified", host GPU0 = 1 MiB / GPU1 = 20.8 GB. estate compose_env confirmed UUID-izing while ESTATE_GPUS stays index-based. CDI leg still awaits mog. Relevant guard sweep (11) green. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
70 lines
3.1 KiB
Bash
Executable File
70 lines
3.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# test-gpu-select.sh — the shared GPU-UUID resolver (#610 Phase A) has TWO
|
|
# implementations that MUST agree: scripts/lib/gpu-select.sh (bash, for
|
|
# launch.sh) and resolve_gpu_uuids() in estate_cli.py (python, for the estate
|
|
# boot path). This guard asserts:
|
|
# 1. both resolve the SAME indices to the SAME UUID csv (real nvidia-smi);
|
|
# 2. both fall back to empty/None identically when a resolver can't run
|
|
# (fake-GPU test mode / no nvidia-smi);
|
|
# 3. gpu_select_export sets both CUDA_ and NVIDIA_VISIBLE_DEVICES;
|
|
# 4. the launcher sources the lib (no re-inlined resolver drift).
|
|
set -euo pipefail
|
|
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
|
cd "$ROOT"
|
|
# shellcheck source=../lib/gpu-select.sh
|
|
source "$ROOT/scripts/lib/gpu-select.sh"
|
|
|
|
fail() { echo "FAIL: $*" >&2; exit 1; }
|
|
|
|
# (4) launch.sh sources the lib rather than re-inlining the resolver.
|
|
grep -q 'scripts/lib/gpu-select.sh' scripts/launch.sh \
|
|
|| fail "launch.sh no longer sources gpu-select.sh (resolver drift risk)"
|
|
grep -q 'gpu_select_export' scripts/launch.sh \
|
|
|| fail "launch.sh no longer calls gpu_select_export"
|
|
|
|
# (2) fallback parity: with no real GPUs to resolve, both yield empty/None.
|
|
if ! command -v nvidia-smi >/dev/null 2>&1; then
|
|
bash_out="$(gpu_select_indices_to_uuids "0,1")"
|
|
[[ -z "$bash_out" ]] || fail "bash resolver should be empty without nvidia-smi (got '$bash_out')"
|
|
fi
|
|
# python resolver under the fake-GPU seam must return None (empty print).
|
|
py_fake="$(CLUB3090_FAKE_GPUS='0:RTX 3090:24576:8.6' python3 - <<'PY'
|
|
import sys
|
|
sys.path.insert(0, ".")
|
|
from scripts.lib.profiles.estate_cli import resolve_gpu_uuids
|
|
v = resolve_gpu_uuids([0, 1])
|
|
print("" if v is None else v)
|
|
PY
|
|
)"
|
|
[[ -z "$py_fake" ]] || fail "python resolver should be None under CLUB3090_FAKE_GPUS (got '$py_fake')"
|
|
|
|
# (3) gpu_select_export exports both masks (UUID when resolvable, else index).
|
|
( gpu_select_export "0" "test" >/dev/null 2>&1
|
|
[[ -n "${CUDA_VISIBLE_DEVICES:-}" && -n "${NVIDIA_VISIBLE_DEVICES:-}" ]] \
|
|
|| { echo "gpu_select_export did not set both masks" >&2; exit 1; }
|
|
[[ "${CUDA_VISIBLE_DEVICES}" == "${NVIDIA_VISIBLE_DEVICES}" ]] \
|
|
|| { echo "masks disagree: cuda=${CUDA_VISIBLE_DEVICES} nvidia=${NVIDIA_VISIBLE_DEVICES}" >&2; exit 1; }
|
|
) || fail "gpu_select_export contract"
|
|
|
|
# (1) real-hardware parity: only when nvidia-smi is present with >=1 GPU.
|
|
if command -v nvidia-smi >/dev/null 2>&1 && nvidia-smi -L >/dev/null 2>&1; then
|
|
n="$(nvidia-smi -L | grep -c '^GPU ' || echo 0)"
|
|
if [[ "$n" -ge 1 ]]; then
|
|
bash_uuids="$(gpu_select_indices_to_uuids "0")"
|
|
py_uuids="$(python3 - <<'PY'
|
|
import sys
|
|
sys.path.insert(0, ".")
|
|
from scripts.lib.profiles.estate_cli import resolve_gpu_uuids
|
|
v = resolve_gpu_uuids([0])
|
|
print("" if v is None else v)
|
|
PY
|
|
)"
|
|
[[ "$bash_uuids" == "$py_uuids" ]] \
|
|
|| fail "bash/python resolver disagree for GPU 0: bash='$bash_uuids' python='$py_uuids'"
|
|
[[ "$bash_uuids" == GPU-* ]] \
|
|
|| fail "resolver returned a non-UUID for GPU 0: '$bash_uuids'"
|
|
fi
|
|
fi
|
|
|
|
echo "PASS: gpu-select.sh + estate_cli resolver agree; launcher sources the lib; export sets both masks"
|