club-3090/docs/PODS.md

Pods — running multiple models on one host

A pod is one model pinned to a chosen GPU set + port — an estate instance. On a multi-GPU box you compose several pods (e.g. a chat model on GPU 0, a coder on GPUs 1+2), each its own endpoint, side by side. This page is the home for creating and managing them; for the raw GPU-pinning mechanics (UUIDs, CDI/NixOS runtimes) see HARDWARE.md → "Pinning specific GPUs".

Pod vs TP=N. A single model split across cards (tensor-parallel, vllm/dual = TP=2) is one endpoint on N GPUs — see DUAL_CARD.md / MULTI_CARD.md. A pod is the other axis: many models, each on its own subset of the cards. A pod's GPU count still equals its compose's TP (a TP=2 slug claims 2 GPUs).

Non-goal: pods are single-host. Routing across multiple boxes is the LiteLLM gateway's job — these tools don't do multi-node.

---

Defaults — before you make any pod

There is no default pod, and no pod claims "all GPUs" for you. A fresh setup has an empty estate — pod.sh list shows no pods and every GPU free. Pods are entirely opt-in; you only get them when you pod.sh create (or use the c3 [N] wizard).

What happens without pods: a plain bash scripts/launch.sh (or switch.sh <slug>) boots one model, and how many cards it uses is the compose's tensor-parallel size, not "all your GPUs":

• 1 GPU → that card.
• 2 GPUs → it asks — Use GPU 0, GPU 1, or both? [both]. The both default runs one model across both cards (TP=2), e.g. vllm/dual — a single endpoint, not two pods.
• 3+ GPUs → it asks — Which GPU(s)? [all]. "all" runs one model across all of them (TP=N).

So the default is one model that spans the cards you pick — which you can loosely think of as "one pod on all GPUs," but it's a single TP=N endpoint, not the pod/estate machinery. The moment you want several models at once — one on GPU 0, another on GPUs 1+2 — that's when you create pods. Pin the single-model case to specific cards without any pod setup via bash scripts/launch.sh --gpus <a,b>.

---

Quickstart (CLI)

# create — fit-checked against the SELECTED GPUs, written to the estate file
bash scripts/pod.sh create chat  --gpus 0   --slug beellama/dflash   # TP=1 on GPU 0
bash scripts/pod.sh create coder --gpus 1,2 --slug vllm/dual         # TP=2 on GPUs 1+2

bash scripts/pod.sh up coder      # boot just this pod (UUID-pinned + placement-asserted)
bash scripts/pod.sh list          # pods + free GPUs      (--json for tooling)
bash scripts/pod.sh status        # live serving state + placement per pod
bash scripts/pod.sh down coder    # stop just this pod
bash scripts/pod.sh rm coder      # remove from the estate file (refuses while running)

up / down / rm act on one pod; to boot the whole file at once use bash scripts/launch.sh --estate-file <path> (add --parallel to boot instances concurrently).

Quickstart (c3 cockpit)

The Operate · Orchestration tab shows a live pod view — each pod with its GPUs stacked beneath its header and a placement health badge — and:

• [N] — open the New-pod modal: name · slug (catalog picker) · GPU set (prefilled with the free GPUs). It runs the same fit-checked, gated pod.sh create.
• [o] — stop the whole estate (gated).

CLI and cockpit share one estate file and one validation path — a pod you make in either shows up in the other.

---

Two or more pods — a worked example

Say you have a 4-GPU box and want a fast chat model and a bigger coder running at once:

# define two pods (writes the estate file; nothing boots yet)
bash scripts/pod.sh create chat  --gpus 0   --slug beellama/dflash   # TP=1 on GPU 0
bash scripts/pod.sh create coder --gpus 1,2 --slug vllm/dual         # TP=2 on GPUs 1+2

bash scripts/pod.sh list         # → 2 pods; GPU 3 free
bash scripts/pod.sh up chat      # boot each (or `launch.sh --estate-file <path> --parallel` for all)
bash scripts/pod.sh up coder
bash scripts/pod.sh status       # both serving, ✓ placed, on the GPUs you picked

# later: free up the coder's cards without touching chat
bash scripts/pod.sh down coder
bash scripts/pod.sh rm coder     # (or leave it defined and `up` it again later)

Each pod is its own endpoint (chat on :8080, coder on :8010) — point different clients at different ports, or front them with the LiteLLM gateway.

Heterogeneous rigs — one pod per card family

Local rigs increasingly mix families (2× 3090 + a GB10 + an RTX 6000 Pro). The clean pattern is one homogeneous pod per family, each sized to what that hardware does well:

bash scripts/pod.sh create qwen  --gpus 0,1 --slug vllm/dual                  # the two 3090s (TP=2)
bash scripts/pod.sh create big   --gpus 2   --slug vllm/qwen-27b-single-nvfp4  # the GB10 (sm ≥ 9)
bash scripts/pod.sh create vision --gpus 3  --slug <a-6000-pro-slug>          # the 6000 Pro

Keep card families in separate pods, not mixed inside one TP group: tensor-parallel across mismatched cards makes NCCL wait on the slowest card and wastes the bigger card's VRAM. create lets you build a mixed-family pod (it estimates fit against the smallest card in the set), but a same-family pod is faster and its fit estimate is exact. pod.sh create also enforces per-slug required_sm — an NVFP4 slug won't land on your 3090s.

---

How create decides fit (D1)

create prices the slug against the GPU set you selected (via kv-calc --card) before writing anything:

• GPU count must equal the compose's tensor-parallel size — a TP=2 slug needs exactly 2 GPUs. A mismatch is a hard reject with a clear message (no silent drop).
• Homogeneous set (all same card) → priced against that card.
• Heterogeneous set (e.g. a 3090 + a 5090) → estimated against the smallest card in the set (a conservative floor; a note says which card was used). True per-card heterogeneous modelling is a deferred kv-calc enhancement.
• The whole estate is then re-validated (validate_estate) for GPU collisions (two pods claiming the same card) and port collisions before the new pod is appended.

create only writes the plan — no GPU is claimed until up. In c3 the create routes through the standard confirm gate; a bad set is refused by the same checks.

Placement verification

Pinning specific GPUs is only trustworthy if you can confirm it worked. After any pod boots, a placement assertion compares where the model actually ran (nvidia-smi --query-compute-apps=gpu_uuid — runtime-agnostic ground truth) against the GPUs you requested:

• CLI up prints ✓ placement verified or a loud ⚠ PLACEMENT MISMATCH.
• pod.sh status / the c3 pod view carry the verdict per pod (✓ placed / ⚠ PLACEMENT MISMATCH).

A mismatch on a CDI/NixOS rig usually means CUDA_VISIBLE_DEVICES didn't reach the container — see HARDWARE.md for the CDI deploy-block recipe. The view never shows a requested-but-not-actual placement.

The estate file

Pods live in a YAML estate file (default scripts/lib/profiles/estate.yml; override with --file). GPU indices are stored index-based; they're resolved to UUIDs at boot so pods land on the right cards on both container runtimes. Hand-written files pass the same validation as pod.sh create.

schema_version: 1
estate:
  - name: chat
    compose: beellama/dflash    # a registry slug (see `switch.sh --list`)
    gpus: [0]                    # host GPU indices — count must equal the slug's TP
    port: 8080
  - name: coder
    compose: vllm/dual
    gpus: [1, 2]
    port: 8010

---

Reference

Command	What
pod.sh create <name> --gpus <a,b> --slug <slug> [--port N] [--file P]	fit-check + validate + append a pod
pod.sh list [--json]	pods + membership + free GPUs
pod.sh status [--json]	live serving state + placement verdict per pod
pod.sh up <name> / down <name>	boot / stop one pod
pod.sh rm <name>	remove from the estate file (refuses while running)
launch.sh --estate-file <P> [--parallel]	boot the whole estate file

Related: HARDWARE.md (GPU pinning / CDI runtimes) · MULTI_CARD.md (TP scaling for a single model across cards) · ARCHITECTURE.md (services, ports).