docs: ai-studio consolidation pass (one scene, lanes inside it)
Sweep the studio docs to match the consolidation: `image-studio` / `video-studio` / `comfyui` scene refs → the single `ai-studio` scene, drop the gemma-12b chat-on-the-other-card framing, and document the new pieces. - README: 8→9 lanes (+10Eros); "Lanes vs modes" → "One scene, lanes inside it". - video.md: model manifest LoRA `-384`→`-384-1.1`, +10Eros unet row, lane matrix +🔓 10Eros, VRAM note → `ai-studio`. - audio.md: step-voice ⊕-video mutex bullet (c3 guards it) + start-from-c3 note. - image.md / FAQ.md / services/studio/README.md: scene-name + bring-up fixups. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
This commit is contained in:
@@ -163,7 +163,7 @@ Yes — but **not through the LLM stack.** The text models (Qwen3.6 / Gemma) and
|
||||
|
||||
Open-weight models that fit one 3090 (run in ComfyUI): **FLUX.1-dev** (Q8 — aesthetic benchmark), **Qwen-Image** (best text-in-image), **FLUX.2-klein-4B** / **Z-Image-Turbo** (lighter/faster), **HiDream-I1**, **Ideogram-4** (top open quality, ~whole card). Video: **Cosmos3-Nano** / **LTX-2** are feasible on one card; **Wan2.2** / **HunyuanVideo-1.5** are tight. Worked-out shortlist + VRAM sizes: [`models/qwen3-omni-30b-a3b/vllm-omni/README.md`](../models/qwen3-omni-30b-a3b/vllm-omni/README.md).
|
||||
|
||||
**Turnkey path:** the **[Image Studio bundle](ai-studio/image.md)** wires this up for you — `bash scripts/setup-image-studio.sh` (or `gpu-mode image-studio`) brings up Ideogram-4 image gen on one card + a gemma-4-12b chat model on the other + Open WebUI as the front end, so chat and image gen coexist on a 2-GPU box. See [`ai-studio/image.md`](ai-studio/image.md).
|
||||
**Turnkey path:** the **[AI Studio](ai-studio/README.md)** bundle wires this up for you — `gpu-mode ai-studio` (or `bash scripts/setup-image-studio.sh`) brings up ComfyUI (both cards) + the qwen director + Open WebUI as the front end. It's **one scene** with image / video / audio **lanes** you pick in OWUI — image/music/SFX render on GPU0, video splits across both cards. See [`ai-studio/README.md`](ai-studio/README.md).
|
||||
|
||||
### Why does my image model OOM even though the transformer quant is small?
|
||||
|
||||
|
||||
@@ -22,14 +22,17 @@ gallery, one refine-by-reply UX across the creative modalities.
|
||||
|
||||
---
|
||||
|
||||
## The 8 lanes
|
||||
## The 9 lanes
|
||||
|
||||
Pick a lane in the OWUI model picker; the director crafts the right prompt shape for it.
|
||||
They all live in the single **`ai-studio`** scene — `gpu-mode ai-studio` brings the whole
|
||||
creative surface up; you switch *lanes* in OWUI, not gpu-mode *modes*.
|
||||
|
||||
| Lane | Model | Modality | License |
|
||||
|---|---|---|---|
|
||||
| 🎬 `Studio · LTX-2.3` | LTX-2.3 distilled 22B | video + synced audio | open |
|
||||
| 🔓 `Studio · Sulphur` | Sulphur (LTX-2.3 dev FT) | video (uncensored) | open |
|
||||
| 🔓 `Studio · 10Eros` | 10Eros (LTX-2.3 dev FT) | video (uncensored) | open |
|
||||
| ✨ `Studio · Image (HiDream-O1)` | HiDream-O1-Image-Dev-2604 | image — **top-quality / photoreal** (AA #1 single-model open-weight) | MIT |
|
||||
| 🖼️ `Studio · Image` | Ideogram-4 fp8 | image — design / logo / text | open |
|
||||
| 🔓 `Studio · Image (Chroma)` | Chroma1-HD fp8 | image (uncensored) | open |
|
||||
@@ -75,21 +78,25 @@ Studio only borrows a small qwen "director" to craft prompts.)_
|
||||
|
||||
The qwen **director** crafts the right prompt shape per lane; **ComfyUI** renders image/video/music/SFX; the **step-voice** and **studio-tts** services handle premium + narration voice; the **orchestrator** chains long videos; everything lands in the always-on **gallery**. Text/LLM chat is the separate core stack (this is image/video/audio only).
|
||||
|
||||
### Lanes vs. modes
|
||||
### One scene, lanes inside it
|
||||
|
||||
> A **lane** is anything light enough to coexist with the director on **GPU0** — it's just a pipe
|
||||
> route, no GPU-mode switch. A **mode** (`gpu-mode <name>`) is for anything that needs **both
|
||||
> GPUs** (video) or evicts the LLMs. Same trade-off as switching tools in a DAW/NLE on one box.
|
||||
> There's a **single** `ai-studio` gpu-mode scene now (it replaced the old separate
|
||||
> `image-studio`/`video-studio` modes). `gpu-mode ai-studio` brings up ComfyUI on **both
|
||||
> GPUs** + the director + all the sidecars; you pick image / video / audio / voice **as a
|
||||
> lane in OWUI** — no gpu-mode switching between modalities. Same trade-off as switching
|
||||
> tools in a DAW/NLE on one box, but it's all one workspace.
|
||||
|
||||
- **Lanes (coexist on GPU0):** all 3 image lanes, music, SFX — single-device, run in either mode.
|
||||
- **Modes (need both cards / evict):** **video** (22B DiT split across both 3090s via DisTorch).
|
||||
- **Premium voice** (Step-Audio-EditX, ~14 GB) and the future realtime voice agent run as their
|
||||
own services, brought up on demand.
|
||||
ComfyUI runs **one workflow at a time**, so the lanes time-share the cards:
|
||||
|
||||
- **GPU0 lanes (coexist with the director):** all 3 image lanes, music, SFX — single-device.
|
||||
- **Both-GPU lane:** **video** (the 22B DiT splits across both 3090s via DisTorch).
|
||||
- **GPU1 ⊕ video:** **premium voice** (Step-Audio-EditX, ~14 GB on GPU1) is on-demand and
|
||||
**mutually exclusive with an active video render** (both want GPU1) — c3 guards this.
|
||||
|
||||
**The hardware truth (measured):** during a video render GPU1 holds the 22B DiT (~22 GB donor) and
|
||||
GPU0 does compute (~7–14 GB) **+** the ~4.6 GB director — so a ≤1024² image lane *also* fits on
|
||||
GPU0 in `video-studio` with no switch. Heavy modalities are mode-switched, not simultaneous — a
|
||||
workstation reality, framed like switching tools in a creative suite.
|
||||
GPU0 in `ai-studio` with no switch. Heavy modalities time-share (one ComfyUI queue), not
|
||||
simultaneous — a workstation reality, framed like switching tools in a creative suite.
|
||||
|
||||
## Shared substrate (services)
|
||||
|
||||
@@ -102,7 +109,7 @@ workstation reality, framed like switching tools in a creative suite.
|
||||
| **Image shim** (`image-shim/`) | 8191 | ComfyUI reverse-proxy — crafts Ideogram JSON for the native 🖼️ button |
|
||||
| **Studio TTS** (`tts/`) | 8192 | Kokoro-82M (CPU) voiceover + layer-aware ffmpeg mixdown |
|
||||
| **Step-Voice** (`step-voice/`) | 8193 | Step-Audio-EditX premium voice (isolated, transformers 4.53.3, GPU, on-demand) |
|
||||
| **`gpu-mode`** | — | the mode switcher (`video-studio` / `image-studio` / chat / off) |
|
||||
| **`gpu-mode`** | — | the mode switcher (`ai-studio` / chat / off) |
|
||||
|
||||
The OWUI Studio pipe (`services/studio/build_studio_pipe.py` → `studio_pipe.py`) routes each lane
|
||||
to the right backend and returns a gallery link. Install it once: **Admin → Functions → +**, paste
|
||||
@@ -127,8 +134,7 @@ gemma-4-12b — the uncensored DiTs still render, only the prompt-writing change
|
||||
## Bring it up
|
||||
|
||||
```bash
|
||||
bash scripts/gpu-mode.sh video-studio # ComfyUI (both cards) + director + gallery + orchestrator + shim + tts + OWUI
|
||||
# image-only: bash scripts/gpu-mode.sh image-studio
|
||||
bash scripts/gpu-mode.sh ai-studio # ComfyUI (both cards) + director + gallery + orchestrator + shim + tts + OWUI
|
||||
# premium voice (on demand): docker compose -f services/studio/step-voice/docker-compose.yml up -d
|
||||
```
|
||||
|
||||
|
||||
@@ -69,8 +69,13 @@ S2-Pro and Higgs v3 are research/non-commercial).
|
||||
correct, zero conflict. The pipe POSTs `/clone`; the service writes a 24 kHz `.wav` to the gallery.
|
||||
- **VRAM / serving:** ~14 GB bf16 on a pinned card (GPU1 by default), **on-demand** (not
|
||||
always-on — bring it up with `docker compose -f services/studio/step-voice/docker-compose.yml
|
||||
up -d`). An AWQ-4bit build (~3–4 GB) is the future light-deploy option. Validated: ~30 s to
|
||||
load, then a clip in seconds.
|
||||
up -d`, or start it from **c3 → Operate → Containers**). An AWQ-4bit build (~3–4 GB) is the
|
||||
future light-deploy option. Validated: ~30 s to load, then a clip in seconds.
|
||||
- **⊕ Mutually exclusive with an active video render.** In `ai-studio`, video uses *both* 3090s
|
||||
(the 22B DiT donates ~22 GB to GPU1 via DisTorch), and premium voice wants ~14 GB on that same
|
||||
GPU1 — they can't both be resident. **c3 guards this**: starting `step-voice` while a video
|
||||
render holds GPU1 is blocked with a "GPU1 busy (video)" notice; let the render finish (or stop
|
||||
ComfyUI's video lane) first. Music/SFX/image lanes are GPU0 and don't conflict with voice.
|
||||
|
||||
> Step-Audio-EditX is **generate-only** — it clones + edits speech, it does **not** diarize or
|
||||
> separate multi-speaker recordings (that's the Understand pillar).
|
||||
|
||||
@@ -116,16 +116,16 @@ OWUI's `COMFYUI_BASE_URL` points at it (`imagegen.env`), with OWUI's image-promp
|
||||
for a rich Ideogram-4 JSON caption, and rewrites the node before forwarding. The escaping is done in
|
||||
**Python** (reliable). Blast radius = image generation only — title/tag task-generation is untouched.
|
||||
|
||||
`gpu-mode video-studio` and `image-studio` start the shim (and, in image-studio, the director). If
|
||||
the shim is down, point `COMFYUI_BASE_URL` back at `:8188` (plain text then hits the placeholder) or
|
||||
use the **Studio · Image lane**.
|
||||
`gpu-mode ai-studio` starts the shim and the director. If the shim is down, point
|
||||
`COMFYUI_BASE_URL` back at `:8188` (plain text then hits the placeholder) or use the
|
||||
**Studio · Image lane**.
|
||||
|
||||
## VRAM / GPU split
|
||||
|
||||
All three image lanes render on **GPU0** and coexist with the ~4.6 GB director. Ideogram ~18.5 GB
|
||||
@1024² + director ≈ 23 GB (fits; 2048² would OOM with the director resident). HiDream is fixed at
|
||||
2048² (~15 GB) + director ≈ 20 GB. Because ComfyUI holds both cards in `video-studio`, you can do
|
||||
**video and a ≤1024² image in the same mode with no switch**.
|
||||
2048² (~15 GB) + director ≈ 20 GB. Because ComfyUI holds both cards in `ai-studio`, you can do
|
||||
**video and a ≤1024² image in the same scene with no switch**.
|
||||
|
||||
## Models (obtain separately → `/mnt/models/comfyui/models/...`)
|
||||
|
||||
|
||||
@@ -66,7 +66,7 @@ manifest below); the director GGUF → `/mnt/models/huggingface/qwen3.5-4b-gguf/
|
||||
**2. Bring the stack up:**
|
||||
|
||||
```bash
|
||||
bash scripts/gpu-mode.sh video-studio
|
||||
bash scripts/gpu-mode.sh ai-studio
|
||||
```
|
||||
|
||||
Stops the GPU LLMs and starts ComfyUI (both cards) + director (`:8090`) + gallery (`:8189`) +
|
||||
@@ -148,7 +148,7 @@ scene-cuts inside one segment; exact object counts.
|
||||
| **Audio** | yes — LTX-2.3 generates synced ambient audio; optional Kokoro voiceover ([audio.md](audio.md)) |
|
||||
| **Resolution** | Sulphur 1280×720 · LTX 768×512 (set in the workflow) |
|
||||
| **Length** | default ~10 s; see the ceiling below |
|
||||
| **Video lanes** | `🎬 LTX-2.3` (video+audio) · `🔓 Sulphur` (uncensored) — image/audio lanes in [image.md](image.md) / [audio.md](audio.md), full matrix in [README.md](README.md) |
|
||||
| **Video lanes** | `🎬 LTX-2.3` (video+audio) · `🔓 Sulphur` · `🔓 10Eros` (uncensored) — image/audio lanes in [image.md](image.md) / [audio.md](audio.md), full matrix in [README.md](README.md) |
|
||||
|
||||
### Length ceiling (measured on 2× 3090, 1280×720, frames = 24·seconds + 1)
|
||||
|
||||
@@ -204,8 +204,8 @@ LoRA onto the base sampler, 8 steps, cfg 1, no upscaler. Clean output. The workf
|
||||
|
||||
Video and the dual-card LLMs are **mutually exclusive** (both want the GPUs). In video mode: GPU1
|
||||
holds the 22B DiT weights (~22 GB, DisTorch donor); GPU0 does compute (~7–14 GB) **and** hosts the
|
||||
~4 GB director — they coexist on one card. Because ComfyUI holds both cards in `video-studio`, you
|
||||
can also run a ≤1024² **image** lane in the same mode with no switch (it fits on GPU0 beside the
|
||||
~4 GB director — they coexist on one card. Because ComfyUI holds both cards in `ai-studio`, you
|
||||
can also run a ≤1024² **image** lane in the same scene with no switch (it fits on GPU0 beside the
|
||||
director). Full per-lane VRAM in [image.md](image.md) / [audio.md](audio.md) / [README.md](README.md).
|
||||
|
||||
## On the uncensored models
|
||||
@@ -223,9 +223,10 @@ meaningful censorship lever, so it's not abliterated.)
|
||||
|---|---|---|
|
||||
| `ltx-2.3-22b-distilled-1.1-Q8_0.gguf` | `unet/ltx2.3/distilled-1.1/` | LTX |
|
||||
| `sulphur-2/sulphur_dev-Q8_0.gguf` | `unet/` | Sulphur |
|
||||
| `ltx-2.3-22b-distilled-lora-384.safetensors` | `loras/` | both (single-stage splice) |
|
||||
| `ltx-2.3-22b-{distilled,dev}_{audio,video}_vae.safetensors` | `vae/` | LTX / Sulphur |
|
||||
| `ltx-2.3-22b-{distilled,dev}_embeddings_connectors.safetensors` | `text_encoders/` | LTX / Sulphur |
|
||||
| `10eros/10Eros_v1-Q8_0.gguf` | `unet/` | 10Eros |
|
||||
| `ltx-2.3-22b-distilled-lora-384-1.1.safetensors` | `loras/` | all dev lanes (single-stage splice) |
|
||||
| `ltx-2.3-22b-{distilled,dev}_{audio,video}_vae.safetensors` | `vae/` | LTX / Sulphur / 10Eros |
|
||||
| `ltx-2.3-22b-{distilled,dev}_embeddings_connectors.safetensors` | `text_encoders/` | LTX / Sulphur / 10Eros |
|
||||
|
||||
Director GGUF (`Qwen3.5-4B-Uncensored-…`) → `/mnt/models/huggingface/qwen3.5-4b-gguf/…`. Image +
|
||||
audio model manifests are in [image.md](image.md) / [audio.md](audio.md).
|
||||
|
||||
@@ -34,10 +34,11 @@ python3 build_studio_pipe.py # writes studio_pipe.py
|
||||
```
|
||||
|
||||
Then in Open WebUI: **Admin → Functions → +**, paste the contents of `studio_pipe.py`,
|
||||
save, enable. Eight models appear in the picker:
|
||||
save, enable. Nine models appear in the picker:
|
||||
|
||||
- `🎬 Studio · LTX-2.3` — video + audio (stock model)
|
||||
- `🔓 Studio · Sulphur` — uncensored video lane
|
||||
- `🔓 Studio · Sulphur` — uncensored video lane (LTX-2.3-22B-dev fine-tune)
|
||||
- `🔓 Studio · 10Eros` — uncensored video lane (LTX-2.3-native dev fine-tune; A/B vs Sulphur)
|
||||
- `✨ Studio · Image (HiDream-O1)` — top-quality / photoreal stills (natural-language prompt)
|
||||
- `🖼️ Studio · Image` — Ideogram-4 (graphic design / logo / photo / text)
|
||||
- `🔓 Studio · Image (Chroma)` — uncensored stills (natural-language prompt)
|
||||
@@ -66,13 +67,13 @@ Set the pipe's **Valves** (gear icon on the function):
|
||||
|
||||
## Bring it up
|
||||
|
||||
`bash scripts/gpu-mode.sh video-studio` brings up ComfyUI (both GPUs) + the director +
|
||||
`bash scripts/gpu-mode.sh ai-studio` brings up ComfyUI (both GPUs) + the director +
|
||||
the gallery + Open WebUI as a unit. Or start pieces individually:
|
||||
|
||||
```bash
|
||||
docker compose -f services/studio/gallery/docker-compose.yml up -d # always-on gallery
|
||||
docker compose -f services/studio/enhancer/docker-compose.yml up -d # director :8090
|
||||
bash scripts/gpu-mode.sh comfyui # ComfyUI :8188
|
||||
docker compose -f services/comfyui/docker-compose.yml up -d # ComfyUI :8188
|
||||
```
|
||||
|
||||
## Use
|
||||
|
||||
Reference in New Issue
Block a user