81181fc4fa
Folds the current-design rationale from the parked production-agent design doc into the public architecture page (leaving out roadmap/staging/target arch): - "Why this shape — creative within bounds": serial dissolves the VRAM problem on a PCIe rig (the hard part is creative orchestration); the model proposes, the executor disposes (never trusted to sequence GPU ops); creative in the film-making layer, never the control plane. - Cross-shot consistency framed as the quality ceiling the Character Bible + continuity modes attack. - "Transport-success ≠ real success" added to Key decisions — the executor ffprobe-validates the actual output (duration/audio/placeholder), not just "a file appeared". - Reproducible productions/<job_id>/ + typed manifest (seeds, workflow versions, exact ffmpeg cmd, validator results) added to the troubleshooting section. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Club 3090 AI Studio
A chat-driven, open-weight creative studio for image, video, and audio generation, running on a 2× RTX 3090 workstation — all behind one consistent flow in Open WebUI:
casual prompt → a "director" LLM crafts it → ComfyUI / a service renders → gallery link → reply to refine.
Fully self-hosted, no cloud APIs. Uncensored lanes where the model allows. One director, one gallery, one refine-by-reply UX across the creative modalities.
Scope. AI Studio is the creative-generation umbrella — image · video · audio. Text (chat / agentic LLM serving — the model catalog, engines, composes, KV, topology) is the core rig stack, documented separately in the global architecture, not a Studio modality. The qwen "director" used here is a small prompt-crafting helper service, not a chat lane — Open WebUI is just the shared front-end for both.
| Deep-dive | Covers |
|---|---|
| agents-architecture.md | How the director thinks — the one 4B behind every lane, the two agent shapes, the Production Director decision flow, the per-lane comparison + the system-prompt map |
| requirements.md | Can I run this? — GPU / CPU / RAM / disk / software, single-vs-dual-card, the director-placement VRAM lever |
| image.md | HiDream-O1 (top quality) · Ideogram-4 (design/logo/text) · Chroma + Z-Image (uncensored) · Krea 2 (aesthetic) · the native-button shim |
| video.md | LTX-2.3 (video+audio) · Sulphur / 10Eros (uncensored) · Wan2.2 (uncensored T2V) · 60 s+ chaining · the single-stage rule |
| audio.md | Step-Audio-EditX (premium voice clone+edit) · Kokoro (narration) · ACE-Step (music) · Stable Audio (SFX) |
The 12 lanes
Pick a lane in the OWUI model picker; the director crafts the right prompt shape for it.
They all live in the single ai-studio scene — gpu-mode ai-studio brings the whole
creative surface up; you switch lanes in OWUI, not gpu-mode modes.
| Lane | Model | Modality | License |
|---|---|---|---|
🎬 Studio · Video (LTX-2.3) |
LTX-2.3 distilled 22B | video + synced audio | open |
🔓 Studio · Video (Sulphur) |
Sulphur (LTX-2.3 dev FT) | video (uncensored) | open |
🔓 Studio · Video (10Eros) |
10Eros (LTX-2.3 dev FT) | video (uncensored) | open |
🔓 Studio · Video (Wan2.2) |
Wan2.2-Rapid-AllInOne Mega v10 | video (uncensored, text→video) | Apache |
✨ Studio · Image (HiDream-O1) |
HiDream-O1-Image-Dev-2604 | image — top-quality / photoreal (AA #1 single-model open-weight) | MIT |
🖼️ Studio · Image |
Ideogram-4 fp8 | image — design / logo / text | open |
🔓 Studio · Image (Chroma) |
Chroma1-HD fp8 | image (uncensored) | open |
🔓 Studio · Image (Z-Image) |
Z-Image-Turbo fp8 | image — uncensored, fast (~25 s) | Apache |
🎨 Studio · Image (Krea 2) |
Krea 2 Turbo fp8 | image — aesthetic / stylized (~40 s) | Krea |
🎵 Studio · Music |
ACE-Step v1 3.5B | music — songs + instrumentals | open |
🔊 Studio · SFX |
Stable Audio Open 1.0 | sound effects / ambience | open |
🎙️ Studio · Voice |
Step-Audio-EditX 3B | premium voice — clone + emotion/style edit | Apache |
Video lanes can also mix a Kokoro voiceover onto the clip (a directive in the message; see audio.md). (Text chat / agentic serving is the core rig stack, not a Studio lane — the Studio only borrows a small qwen "director" to craft prompts.)
Architecture
Browser — Open WebUI :8080
│ pick a lane · type an idea · reply to refine
▼
┌──────────────────────────────────────────┐
│ Studio pipe (OWUI Function) │
│ routes the lane · returns gallery links │
└───────┬──────────────────────────┬───────┘
craft (1)│ render (2)│
▼ ▼
┌──────────────────────┐ ┌──────────────────────────────────────┐
│ Director :8090 │ │ Renderers │
│ qwen3.5-4b · GPU0 │ │ • ComfyUI :8188 — image · video · │
│ idea → crafted prompt│ │ music · SFX (GPU0; video uses │
│ (JSON / prose / │ │ both GPUs via DisTorch) │
│ tags / sound) │ │ • step-voice :8193 — premium voice │
└──────────────────────┘ │ (isolated, transformers 4.53.3) │
┌──────────────────────┐ │ • studio-tts :8192 — Kokoro (CPU) │
│ image-shim :8191 │ │ voiceover ducked onto a clip │
│ proxy for OWUI's 🖼️ │ └──────────────────┬───────────────────┘
│ button → JSON caption│ long video >15s → orchestrator :8190
└──────────────────────┘ (chains ~10s segments + mux)
▼
┌───────────────────────────┐
│ Gallery :8189 (nginx) │
│ /output — survives ComfyUI │
│ down; ▶️/🖼️/🎧 links in chat │
└───────────────────────────┘
The qwen director crafts the right prompt shape per lane; ComfyUI renders image/video/music/SFX; the step-voice and studio-tts services handle premium + narration voice; the orchestrator chains long videos; SearXNG (:8088) lets the experimental 🎬 Production director ground documentaries in real web facts (see agents-architecture.md); everything lands in the always-on gallery. Text/LLM chat is the separate core stack (this is image/video/audio only).
One scene, lanes inside it
There's a single
ai-studiogpu-mode scene now (it replaced the old separateimage-studio/video-studiomodes).gpu-mode ai-studiobrings up ComfyUI on both GPUs + the director + all the sidecars; you pick image / video / audio / voice as a lane in OWUI — no gpu-mode switching between modalities. Same trade-off as switching tools in a DAW/NLE on one box, but it's all one workspace.
ComfyUI runs one workflow at a time, so the lanes time-share the cards:
- GPU0 lanes (coexist with the director): all 5 image lanes, music, SFX — single-device.
- Both-GPU lane: video (the 22B DiT splits across both 3090s via DisTorch).
- GPU1 ⊕ video: premium voice (Step-Audio-EditX, ~14 GB on GPU1) is on-demand and mutually exclusive with an active video render (both want GPU1) — c3 guards this.
The hardware truth (measured): during a video render GPU1 holds the 22B DiT (~22 GB donor) and
GPU0 does compute (~7–14 GB) + the ~4.6 GB director — so a ≤1024² image lane also fits on
GPU0 in ai-studio with no switch. Heavy modalities time-share (one ComfyUI queue), not
simultaneous — a workstation reality, framed like switching tools in a creative suite.
Shared substrate (services)
| Service | Port | Role |
|---|---|---|
| ComfyUI | 8188 | the renderer (image/video/music/SFX lanes) |
Director (enhancer/) |
8090 | qwen3.5-4b-uncensored — casual idea → crafted prompt; always-on, GPU0 ~4.6 GB |
Gallery (gallery/) |
8189 | always-on nginx over the output dir — links survive ComfyUI down |
Orchestrator (orchestrator/) |
8190 | long-clip chaining + ffmpeg mux (host-side, no GPU) |
Image shim (image-shim/) |
8191 | ComfyUI reverse-proxy — crafts Ideogram JSON for the native 🖼️ button |
Studio TTS (tts/) |
8192 | Kokoro-82M (CPU) voiceover + layer-aware ffmpeg mixdown |
Step-Voice (step-voice/) |
8193 | Step-Audio-EditX premium voice (isolated, transformers 4.53.3, GPU, on-demand) |
gpu-mode |
— | the mode switcher (ai-studio / chat / off) |
The OWUI Studio pipe (services/studio/build_studio_pipe.py → studio_pipe.py) routes each lane
to the right backend and returns a gallery link. Install it once: Admin → Functions → +, paste
studio_pipe.py, enable.
Why this is interesting
- Fully open-weight + self-hosted — no API, no cloud, no per-call cost; your data stays local.
- Uncensored lanes where the model allows — Sulphur (video), Chroma (image), the uncensored director — capability lives in the weights; the infrastructure is content-neutral.
- One consistent director-driven UX across image / video / audio.
- Honest constraint as a feature: heavy modalities are mode-switched, not simultaneous — lightweight combos (chat + a ≤1024² image + a voice) coexist.
On the uncensored models
The Sulphur DiT, Chroma, and the director are uncensored fine-tunes — chosen so the creative lanes
don't refuse or sanitize. That capability is in the model weights; the infra is content-neutral. To
craft prompts through an aligned model instead, point the pipe's chat_model valve at e.g.
gemma-4-12b — the uncensored DiTs still render, only the prompt-writing changes.
Bring it up
One command (fresh clone → generating) — builds the ComfyUI image, downloads the full roster (~120 GB), brings the scene up, and installs the OWUI Studio pipe:
bash scripts/setup-ai-studio.sh # add --yes to skip the confirm; SKIP_BUILD / SKIP_DOWNLOAD / SKIP_DISK_CHECK / SKIP_PIPE to trim
Where the models land: everything goes under your MODEL_DIR (set in repo-root .env, e.g. via c3 Settings) — the HF/GGUF weights at $MODEL_DIR, and the ComfyUI tree at a comfyui sibling of it (e.g. MODEL_DIR=/home/me/models → ComfyUI at /home/me/comfyui). Override COMFYUI_ROOT / COMFYUI_MODELS_DIR to decouple them. Resuming a download that already pushed you under the free-space threshold? SKIP_DISK_CHECK=1 bypasses the preflight (the roster pull is idempotent and only fetches what's missing).
Already set up? Just bring the scene up (or do it from c3 → Operate):
bash scripts/gpu-mode.sh ai-studio # ComfyUI (both cards) + director + gallery + orchestrator + shim + tts + OWUI
# premium voice (on demand): docker compose -f services/studio/step-voice/docker-compose.yml up -d
Then open Open WebUI at http://<your-host>:8080, sign up (first account = admin), set the pipe's
browser_base valve to your host's LAN IP (http://<your-host>:8189), and pick a lane. (If you signed
up after setup-ai-studio.sh ran, install the pipe with bash services/studio/push-pipe-to-owui.sh.)
Per-modality setup + model manifests are in image.md / video.md /
audio.md; requirements in requirements.md. The service bundle itself is
documented in services/studio/README.md.