Files

noonghunna 47ac38385a Director placement lever: CPU / GPU0 / GPU1 (c3 Settings) (#473 )

* Director placement lever: env-driven CPU / GPU0 / GPU1 (backend)

The studio director's GPU/CPU placement is now a single lever,
STUDIO_DIRECTOR_DEVICE (gpu0 | gpu1 | cpu, default gpu0), read from the rig
.env by gpu-mode's start_studio_director and translated into the compose
env (-ngl + CUDA_VISIBLE_DEVICES + device_ids):

- gpu0 (default): -ngl 99, GPU0 — fast craft (~50-100 tok/s), ~4.6 GB,
  coexists with the image lanes. Unchanged from before.
- gpu1: -ngl 99, GPU1 — only when GPU1 has room (NOT during a video render;
  GPU1 is the DisTorch DiT donor).
- cpu: -ngl 0, CUDA_VISIBLE_DEVICES="" — frees ~4.6 GB off GPU0 (lifts the
  single-card Wan window 121→161 frames) at ~single-digit tok/s craft.

Compose now reads ${DIRECTOR_NGL:-99} + ${STUDIO_DIRECTOR_CUDA-0} (no-colon
so an explicit empty value = CPU survives). Default (no override) preserves
current GPU0 behaviour exactly.

Live-validated: CPU mode starts with GPU0 full (gemma12b), adds 0 MiB VRAM
to GPU0, serves on :8090, generates (~5 tok/s CPU). The c3 Settings field
that writes STUDIO_DIRECTOR_DEVICE follows in the next commit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm

* c3 Settings: Director placement (CPU / GPU0 / GPU1) + CPU thread cap

Exposes the director-placement lever in the c3 Settings modal so users pick
GPU0 (fast, default) / GPU1 / CPU without hand-editing the .env.

- services.py: director_device() reads STUDIO_DIRECTOR_DEVICE from the repo
  .env (default gpu0, validates the value); set_repo_env_var() upserts a key
  in place (preserves other lines, no duplicates, creates the file if absent).
- app.py: SettingsScreen gains a "Director placement" Select; apply_settings
  persists the choice to the repo .env (the SHARED config gpu-mode reads —
  distinct from c3-settings.json for MODEL_DIR/HF_TOKEN). Applies on the next
  ai-studio start.
- compose: CPU thread cap — -t ${DIRECTOR_THREADS:-8} bounds CPU use so the
  director doesn't starve OWUI's embedder/reranker (also CPU). The ~2.6 GB
  GGUF loads into system RAM (mmap'd; resident in page cache, not run from SSD).
- tests: +6 data-layer (TestDirectorPlacement) + 1 headless apply-settings
  round-trip (persists STUDIO_DIRECTOR_DEVICE, idempotent re-apply). Full
  suite green (728), settings/director subset 13/13.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm

* tests: fix stale scene names in gpu-mode-list EXPECT

The scene catalog renamed its dispatch keywords to qwen27b / gemma-31b,
but the test's EXPECT spot-check map still referenced the old 27b / gemma
short names — so the JSON-shape assertion had been red on master. Point
EXPECT at the canonical names the catalog now emits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm

* studio: run director in chat scene as the catalog-support layer

Bring the uncensored director up in the `chat` scene (honoring the
STUDIO_DIRECTOR_DEVICE placement knob), framing chat as the supporting-
infra home — OWUI + LiteLLM + Qdrant + SearXNG + director — for ad-hoc
Catalog models launched via `switch.sh --owui`.

A CPU-placed director uses no GPU, so it's the always-on path: it survives
scene switches and stays live in OWUI. New _director_evict_if_gpu helper
frees only a GPU-resident director when a dual-card LLM scene claims the
cards; mode_off stops it outright. Also brings mode_gemma_int8 in line with
its dual-card siblings (it was missing the studio teardown entirely).

Docs: requirements.md gains a "Chat scene — the Catalog-support layer"
section + reframes director placement around the unified knob / c3 Setting;
video.md note synced.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm

* studio: disable director thinking on CPU only (latency)

The uncensored director is an "Aggressive" reasoning fine-tune that emits
a full <think> trace before every answer. On GPU that's cheap and the
trace lands in reasoning_content (content stays clean), so leave it on.
On CPU (~14 tok/s) the trace dominates latency, so gpu-mode now passes
`--jinja --reasoning off` for the cpu placement only — forcing the
template's enable_thinking=false (this fine-tune ignores /no_think and
--reasoning-budget 0, but honors --reasoning off).

Wired via a new DIRECTOR_THINK_ARGS compose param (empty on GPU). Live:
CPU director now answers in one pass, no reasoning trace, craft quality
intact (full cinematic spec, finish=stop).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm

* c3: Containers-pane director start honors placement + enumerate sidecars

Two consistency fixes for the Containers tab:

1. Starting studio-director from Containers now honors STUDIO_DIRECTOR_DEVICE
   instead of falling back to the GPU0/thinking-on compose default. New
   director_compose_env() mirrors gpu-mode's start_studio_director translation
   (NGL/CUDA/GPU/THINK_ARGS), injected as an `env K=V …` prefix on the compose
   up cmd (process env wins over --env-file). cpu → -ngl 0 + --reasoning off.

2. The nested studio sidecars (director/gallery/orchestrator/image-shim/
   step-voice/tts) now enumerate when STOPPED, so they're startable rows — not
   only visible while running. New STUDIO_SIDECARS map is the single SoT for
   resolving the container-name → services/studio/<sub>/ project (fixing the
   director↔enhancer name mismatch that previously returned None → docker
   restart, which fails on a fresh install).

+8 tests (director_compose_env cpu/gpu, director resolves to enhancer with the
env prefix, sidecar enumeration). Live: c3 service_start plan starts the director
CPU + no-think (argv -ngl 0 --reasoning off, GPU0 free, clean generation).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm

---------

Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

2026-06-25 09:01:46 +05:00

7.9 KiB

Raw Permalink Blame History

AI Studio — requirements

What you need to self-host Club 3090 AI Studio. Everything is open-weight and runs locally — no cloud APIs, no accounts. Values are minimum / recommended; the reference rig is a 2× RTX 3090 workstation, but nothing here is specific to it.

TL;DR

2× 24 GB NVIDIA GPUs — Ampere (sm_86) or newer (RTX 3090 / 4090 / A5000-class). PCIe is fine, no NVLink required.
Linux + Docker with the NVIDIA Container Toolkit; a driver new enough for the container CUDA (12.4+).
~120 GB disk for the full model roster (less if you skip lanes); SSD recommended.
32 GB+ system RAM.

A subset runs on a single 24 GB card — see Single-GPU below.

GPU

Lane group	Models	VRAM	Card(s)
Director	Qwen3.5-4B-Uncensored (prompt crafter, llama.cpp)	~4.5 GB	one card; or CPU / the idle second card (configurable — see note)
Image	Ideogram-4 (~18.5 GB) · HiDream-O1 (~15 GB) · Chroma (~9 GB) · Z-Image (~7 GB)	up to ~18.5 GB	single card (GPU0), coexists with the director
Video	LTX-2.3 22B · Sulphur / 10Eros 22B · Wan2.2 14B	~22 GB weights + ~7–14 GB compute	both cards — DisTorch donates the DiT weights to GPU1, compute runs on GPU0
Music / SFX	ACE-Step (~8 GB) · Stable Audio	single card	GPU0
Premium voice	Step-Audio-EditX	~14 GB	a free card, on-demand (⊕ mutually exclusive with an active video render)

Why two cards: the video DiTs (22B LTX / 14B Wan) plus their compute exceed one 24 GB card, so they split across both via DisTorch — a VRAM split (weights stored on the second card, compute on the first), not a compute split, so it works on PCIe with no NVLink. Image / audio / voice each fit one card.

Director placement (a VRAM lever). The director is a small, latency-tolerant helper — ~4.5 GB, ~1.4 s to craft a prompt (measured, GPU), then idle. Default: GPU0, which coexists with every shipped default lane. It's also the swing factor on the single-card Wan ceiling: its 4.5 GB on GPU0 caps the single-card 480p window at ~121 frames; freeing it lifts that to 161 (measured).

One knob drives it: STUDIO_DIRECTOR_DEVICE=gpu0|gpu1|cpu in the rig .env — set it from c3 → Settings → "Director placement", or edit the line directly. gpu-mode reads it on every scene launch and translates it to the container's CUDA_VISIBLE_DEVICES + -ngl:

gpu0 (default) → GPU0, snappy refine-by-reply (~1.4 s craft).

gpu1 → the second card. Safe only when GPU1 has room — the image lanes and the Wan video lane (18 GB donor + 4.5 GB = 22.5 GB fits). NOT the LTX / Sulphur / 10Eros lanes: they use GPU1 as their ~22 GB DisTorch donor, so a director there OOMs them.

cpu → universally safe, frees the GPU entirely (model mmaps into ~5 GB system RAM, SSD-backed), at a craft-latency cost (~14 tok/s for a 4B on CPU vs ~1.4 s on GPU — noticeable before a fast image lane, invisible before a multi-minute video). Cap its cores with DIRECTOR_THREADS (default 8). Thinking is auto-disabled on CPU only (gpu-mode sets --reasoning off): a full <think> trace would dominate the slow decode, so CPU craft is a direct one-pass answer. On GPU thinking stays on (fast there, and the trace lands in reasoning_content, leaving content clean either way). A CPU director is the always-on path: it uses no GPU, so it survives scene switches and stays live as the uncensored model in Open WebUI — gpu-mode only evicts a GPU-placed director when a dual-card LLM scene needs the cards.

The chat scene also starts the director (honoring the same knob) as the supporting-infra home for Catalog models — see the chat scene + Catalog note below.

Keep it on GPU0 for the snappy refine-by-reply UX in AI-Studio; choose cpu for an always-on chat companion, or gpu1 to unlock a GPU0 edge case (Ideogram 2048², single-window Wan >121 frames).

Single-GPU (1× 24 GB): image + music + SFX + the director run comfortably; video is the constraint — a 22B DiT won't fit one card at full resolution. Treat dual-card as recommended and single-card as "image + audio studio, video best-effort (short / low-res)."

CPU + RAM

CPU: a modern multi-core (8+ cores). Drives the Kokoro narration TTS (ONNX, CPU), the orchestrator (ffmpeg long-clip concat / mux), the image-shim proxy, and — optionally — the director when CPU-hosted.
RAM: 32 GB minimum, 64 GB+ comfortable. Add ~5 GB if the director runs on CPU.

Disk

~120 GB for the full open-weight roster (GGUF / fp8). SSD recommended — the 18–22 GB video GGUFs load faster. Per modality:

Modality	Models	Disk
Video	LTX-2.3 + Sulphur + 10Eros (22.8 GB each) + Wan2.2 (18.7 GB)	~87 GB
Image	Ideogram-4 (9 GB) + Z-Image (6 GB) + HiDream-O1 + Chroma	~25–35 GB
Audio	ACE-Step (7.7 GB) + Stable Audio (4.9 GB) + Kokoro (0.3 GB)	~13 GB
Director	Qwen3.5-4B-Uncensored GGUF	~2.5 GB
Shared	text encoders (umt5, qwen3-4b, t5) + VAEs	~15 GB

Skip lanes you don't want — scripts/lib/studio-models.tsv is the manifest, and each lane's weights are an independent download (services/comfyui/download_*.sh). bash services/comfyui/download_studio_models.sh fetches the whole roster (idempotent — only what's missing).

Software

OS: Linux (the images are CUDA Linux containers).
Docker + NVIDIA Container Toolkit (GPU passthrough). No host CUDA toolkit needed — CUDA lives in the images.
NVIDIA driver: recent enough for the container CUDA (12.4+; the reference rig runs a CUDA-13 driver).
Images (pulled / built on first bring-up): ComfyUI (custom build, pinned commit + custom nodes for HiDream-O1, GGUF, and DisTorch multi-GPU), llama.cpp (the director), an isolated Step-Audio-EditX container (pinned transformers==4.53.3), nginx (gallery), Open WebUI.
No cloud APIs / accounts — content capability lives in the open weights; the infra is content-neutral.

Bring it up

bash services/comfyui/download_studio_models.sh   # fetch the roster (~120 GB, idempotent)
gpu-mode ai-studio                                 # ComfyUI (both cards) + director + sidecars + OWUI

Then open Open WebUI and pick a lane. Full per-lane detail in image.md / video.md / audio.md; the service bundle is in services/studio/README.md.

Chat scene — the Catalog-support layer

The studio's front-of-house services aren't studio-only. The chat scene (gpu-mode chat) brings up the same supporting infra — Open WebUI + LiteLLM + Qdrant (vector DB / document RAG) + SearXNG (web search) + the uncensored director (:8090) — without a scene GPU model. It's the home base for Catalog models: anything you launch ad-hoc outside a scene.

gpu-mode chat                                  # supporting infra + director, no scene LLM
bash scripts/switch.sh --owui <variant>        # launch any catalog model AND register it into OWUI

switch.sh --owui (also reachable from c3 → Catalog → Serve) starts the chosen variant and wires it into Open WebUI as a direct connection, so it appears in the model picker alongside the director, with web search and document RAG already attached. You get a full chat workstation for any model in the catalog without authoring a scene for it.

The director's placement here honors the same STUDIO_DIRECTOR_DEVICE knob (c3 → Settings → Director placement). Set it to cpu to keep the director always-on: it uses no GPU, so it persists across scene switches and stays available in OWUI even while a dual-card LLM scene owns both cards.

7.9 KiB Raw Permalink Blame History Unescape Escape