diff --git a/docs/FAQ.md b/docs/FAQ.md index 67959fad..c250a1c4 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -163,7 +163,7 @@ Yes — but **not through the LLM stack.** The text models (Qwen3.6 / Gemma) and Open-weight models that fit one 3090 (run in ComfyUI): **FLUX.1-dev** (Q8 — aesthetic benchmark), **Qwen-Image** (best text-in-image), **FLUX.2-klein-4B** / **Z-Image-Turbo** (lighter/faster), **HiDream-I1**, **Ideogram-4** (top open quality, ~whole card). Video: **Cosmos3-Nano** / **LTX-2** are feasible on one card; **Wan2.2** / **HunyuanVideo-1.5** are tight. Worked-out shortlist + VRAM sizes: [`models/qwen3-omni-30b-a3b/vllm-omni/README.md`](../models/qwen3-omni-30b-a3b/vllm-omni/README.md). -**Turnkey path:** the **[Image Studio bundle](ai-studio/image.md)** wires this up for you — `bash scripts/setup-image-studio.sh` (or `gpu-mode image-studio`) brings up Ideogram-4 image gen on one card + a gemma-4-12b chat model on the other + Open WebUI as the front end, so chat and image gen coexist on a 2-GPU box. See [`ai-studio/image.md`](ai-studio/image.md). +**Turnkey path:** the **[AI Studio](ai-studio/README.md)** bundle wires this up for you — `gpu-mode ai-studio` (or `bash scripts/setup-image-studio.sh`) brings up ComfyUI (both cards) + the qwen director + Open WebUI as the front end. It's **one scene** with image / video / audio **lanes** you pick in OWUI — image/music/SFX render on GPU0, video splits across both cards. See [`ai-studio/README.md`](ai-studio/README.md). ### Why does my image model OOM even though the transformer quant is small? diff --git a/docs/ai-studio/README.md b/docs/ai-studio/README.md index 948a8a19..9952e2f2 100644 --- a/docs/ai-studio/README.md +++ b/docs/ai-studio/README.md @@ -22,14 +22,17 @@ gallery, one refine-by-reply UX across the creative modalities. --- -## The 8 lanes +## The 9 lanes Pick a lane in the OWUI model picker; the director crafts the right prompt shape for it. +They all live in the single **`ai-studio`** scene — `gpu-mode ai-studio` brings the whole +creative surface up; you switch *lanes* in OWUI, not gpu-mode *modes*. | Lane | Model | Modality | License | |---|---|---|---| | 🎬 `Studio · LTX-2.3` | LTX-2.3 distilled 22B | video + synced audio | open | | 🔓 `Studio · Sulphur` | Sulphur (LTX-2.3 dev FT) | video (uncensored) | open | +| 🔓 `Studio · 10Eros` | 10Eros (LTX-2.3 dev FT) | video (uncensored) | open | | ✨ `Studio · Image (HiDream-O1)` | HiDream-O1-Image-Dev-2604 | image — **top-quality / photoreal** (AA #1 single-model open-weight) | MIT | | 🖼️ `Studio · Image` | Ideogram-4 fp8 | image — design / logo / text | open | | 🔓 `Studio · Image (Chroma)` | Chroma1-HD fp8 | image (uncensored) | open | @@ -75,21 +78,25 @@ Studio only borrows a small qwen "director" to craft prompts.)_ The qwen **director** crafts the right prompt shape per lane; **ComfyUI** renders image/video/music/SFX; the **step-voice** and **studio-tts** services handle premium + narration voice; the **orchestrator** chains long videos; everything lands in the always-on **gallery**. Text/LLM chat is the separate core stack (this is image/video/audio only). -### Lanes vs. modes +### One scene, lanes inside it -> A **lane** is anything light enough to coexist with the director on **GPU0** — it's just a pipe -> route, no GPU-mode switch. A **mode** (`gpu-mode `) is for anything that needs **both -> GPUs** (video) or evicts the LLMs. Same trade-off as switching tools in a DAW/NLE on one box. +> There's a **single** `ai-studio` gpu-mode scene now (it replaced the old separate +> `image-studio`/`video-studio` modes). `gpu-mode ai-studio` brings up ComfyUI on **both +> GPUs** + the director + all the sidecars; you pick image / video / audio / voice **as a +> lane in OWUI** — no gpu-mode switching between modalities. Same trade-off as switching +> tools in a DAW/NLE on one box, but it's all one workspace. -- **Lanes (coexist on GPU0):** all 3 image lanes, music, SFX — single-device, run in either mode. -- **Modes (need both cards / evict):** **video** (22B DiT split across both 3090s via DisTorch). -- **Premium voice** (Step-Audio-EditX, ~14 GB) and the future realtime voice agent run as their - own services, brought up on demand. +ComfyUI runs **one workflow at a time**, so the lanes time-share the cards: + +- **GPU0 lanes (coexist with the director):** all 3 image lanes, music, SFX — single-device. +- **Both-GPU lane:** **video** (the 22B DiT splits across both 3090s via DisTorch). +- **GPU1 ⊕ video:** **premium voice** (Step-Audio-EditX, ~14 GB on GPU1) is on-demand and + **mutually exclusive with an active video render** (both want GPU1) — c3 guards this. **The hardware truth (measured):** during a video render GPU1 holds the 22B DiT (~22 GB donor) and GPU0 does compute (~7–14 GB) **+** the ~4.6 GB director — so a ≤1024² image lane *also* fits on -GPU0 in `video-studio` with no switch. Heavy modalities are mode-switched, not simultaneous — a -workstation reality, framed like switching tools in a creative suite. +GPU0 in `ai-studio` with no switch. Heavy modalities time-share (one ComfyUI queue), not +simultaneous — a workstation reality, framed like switching tools in a creative suite. ## Shared substrate (services) @@ -102,7 +109,7 @@ workstation reality, framed like switching tools in a creative suite. | **Image shim** (`image-shim/`) | 8191 | ComfyUI reverse-proxy — crafts Ideogram JSON for the native 🖼️ button | | **Studio TTS** (`tts/`) | 8192 | Kokoro-82M (CPU) voiceover + layer-aware ffmpeg mixdown | | **Step-Voice** (`step-voice/`) | 8193 | Step-Audio-EditX premium voice (isolated, transformers 4.53.3, GPU, on-demand) | -| **`gpu-mode`** | — | the mode switcher (`video-studio` / `image-studio` / chat / off) | +| **`gpu-mode`** | — | the mode switcher (`ai-studio` / chat / off) | The OWUI Studio pipe (`services/studio/build_studio_pipe.py` → `studio_pipe.py`) routes each lane to the right backend and returns a gallery link. Install it once: **Admin → Functions → +**, paste @@ -127,8 +134,7 @@ gemma-4-12b — the uncensored DiTs still render, only the prompt-writing change ## Bring it up ```bash -bash scripts/gpu-mode.sh video-studio # ComfyUI (both cards) + director + gallery + orchestrator + shim + tts + OWUI -# image-only: bash scripts/gpu-mode.sh image-studio +bash scripts/gpu-mode.sh ai-studio # ComfyUI (both cards) + director + gallery + orchestrator + shim + tts + OWUI # premium voice (on demand): docker compose -f services/studio/step-voice/docker-compose.yml up -d ``` diff --git a/docs/ai-studio/audio.md b/docs/ai-studio/audio.md index 1856797a..9a75ae77 100644 --- a/docs/ai-studio/audio.md +++ b/docs/ai-studio/audio.md @@ -69,8 +69,13 @@ S2-Pro and Higgs v3 are research/non-commercial). correct, zero conflict. The pipe POSTs `/clone`; the service writes a 24 kHz `.wav` to the gallery. - **VRAM / serving:** ~14 GB bf16 on a pinned card (GPU1 by default), **on-demand** (not always-on — bring it up with `docker compose -f services/studio/step-voice/docker-compose.yml - up -d`). An AWQ-4bit build (~3–4 GB) is the future light-deploy option. Validated: ~30 s to - load, then a clip in seconds. + up -d`, or start it from **c3 → Operate → Containers**). An AWQ-4bit build (~3–4 GB) is the + future light-deploy option. Validated: ~30 s to load, then a clip in seconds. +- **⊕ Mutually exclusive with an active video render.** In `ai-studio`, video uses *both* 3090s + (the 22B DiT donates ~22 GB to GPU1 via DisTorch), and premium voice wants ~14 GB on that same + GPU1 — they can't both be resident. **c3 guards this**: starting `step-voice` while a video + render holds GPU1 is blocked with a "GPU1 busy (video)" notice; let the render finish (or stop + ComfyUI's video lane) first. Music/SFX/image lanes are GPU0 and don't conflict with voice. > Step-Audio-EditX is **generate-only** — it clones + edits speech, it does **not** diarize or > separate multi-speaker recordings (that's the Understand pillar). diff --git a/docs/ai-studio/image.md b/docs/ai-studio/image.md index 66adb099..65930bb0 100644 --- a/docs/ai-studio/image.md +++ b/docs/ai-studio/image.md @@ -116,16 +116,16 @@ OWUI's `COMFYUI_BASE_URL` points at it (`imagegen.env`), with OWUI's image-promp for a rich Ideogram-4 JSON caption, and rewrites the node before forwarding. The escaping is done in **Python** (reliable). Blast radius = image generation only — title/tag task-generation is untouched. -`gpu-mode video-studio` and `image-studio` start the shim (and, in image-studio, the director). If -the shim is down, point `COMFYUI_BASE_URL` back at `:8188` (plain text then hits the placeholder) or -use the **Studio · Image lane**. +`gpu-mode ai-studio` starts the shim and the director. If the shim is down, point +`COMFYUI_BASE_URL` back at `:8188` (plain text then hits the placeholder) or use the +**Studio · Image lane**. ## VRAM / GPU split All three image lanes render on **GPU0** and coexist with the ~4.6 GB director. Ideogram ~18.5 GB @1024² + director ≈ 23 GB (fits; 2048² would OOM with the director resident). HiDream is fixed at -2048² (~15 GB) + director ≈ 20 GB. Because ComfyUI holds both cards in `video-studio`, you can do -**video and a ≤1024² image in the same mode with no switch**. +2048² (~15 GB) + director ≈ 20 GB. Because ComfyUI holds both cards in `ai-studio`, you can do +**video and a ≤1024² image in the same scene with no switch**. ## Models (obtain separately → `/mnt/models/comfyui/models/...`) diff --git a/docs/ai-studio/video.md b/docs/ai-studio/video.md index 19ddbc98..a55b0cf3 100644 --- a/docs/ai-studio/video.md +++ b/docs/ai-studio/video.md @@ -66,7 +66,7 @@ manifest below); the director GGUF → `/mnt/models/huggingface/qwen3.5-4b-gguf/ **2. Bring the stack up:** ```bash -bash scripts/gpu-mode.sh video-studio +bash scripts/gpu-mode.sh ai-studio ``` Stops the GPU LLMs and starts ComfyUI (both cards) + director (`:8090`) + gallery (`:8189`) + @@ -148,7 +148,7 @@ scene-cuts inside one segment; exact object counts. | **Audio** | yes — LTX-2.3 generates synced ambient audio; optional Kokoro voiceover ([audio.md](audio.md)) | | **Resolution** | Sulphur 1280×720 · LTX 768×512 (set in the workflow) | | **Length** | default ~10 s; see the ceiling below | -| **Video lanes** | `🎬 LTX-2.3` (video+audio) · `🔓 Sulphur` (uncensored) — image/audio lanes in [image.md](image.md) / [audio.md](audio.md), full matrix in [README.md](README.md) | +| **Video lanes** | `🎬 LTX-2.3` (video+audio) · `🔓 Sulphur` · `🔓 10Eros` (uncensored) — image/audio lanes in [image.md](image.md) / [audio.md](audio.md), full matrix in [README.md](README.md) | ### Length ceiling (measured on 2× 3090, 1280×720, frames = 24·seconds + 1) @@ -204,8 +204,8 @@ LoRA onto the base sampler, 8 steps, cfg 1, no upscaler. Clean output. The workf Video and the dual-card LLMs are **mutually exclusive** (both want the GPUs). In video mode: GPU1 holds the 22B DiT weights (~22 GB, DisTorch donor); GPU0 does compute (~7–14 GB) **and** hosts the -~4 GB director — they coexist on one card. Because ComfyUI holds both cards in `video-studio`, you -can also run a ≤1024² **image** lane in the same mode with no switch (it fits on GPU0 beside the +~4 GB director — they coexist on one card. Because ComfyUI holds both cards in `ai-studio`, you +can also run a ≤1024² **image** lane in the same scene with no switch (it fits on GPU0 beside the director). Full per-lane VRAM in [image.md](image.md) / [audio.md](audio.md) / [README.md](README.md). ## On the uncensored models @@ -223,9 +223,10 @@ meaningful censorship lever, so it's not abliterated.) |---|---|---| | `ltx-2.3-22b-distilled-1.1-Q8_0.gguf` | `unet/ltx2.3/distilled-1.1/` | LTX | | `sulphur-2/sulphur_dev-Q8_0.gguf` | `unet/` | Sulphur | -| `ltx-2.3-22b-distilled-lora-384.safetensors` | `loras/` | both (single-stage splice) | -| `ltx-2.3-22b-{distilled,dev}_{audio,video}_vae.safetensors` | `vae/` | LTX / Sulphur | -| `ltx-2.3-22b-{distilled,dev}_embeddings_connectors.safetensors` | `text_encoders/` | LTX / Sulphur | +| `10eros/10Eros_v1-Q8_0.gguf` | `unet/` | 10Eros | +| `ltx-2.3-22b-distilled-lora-384-1.1.safetensors` | `loras/` | all dev lanes (single-stage splice) | +| `ltx-2.3-22b-{distilled,dev}_{audio,video}_vae.safetensors` | `vae/` | LTX / Sulphur / 10Eros | +| `ltx-2.3-22b-{distilled,dev}_embeddings_connectors.safetensors` | `text_encoders/` | LTX / Sulphur / 10Eros | Director GGUF (`Qwen3.5-4B-Uncensored-…`) → `/mnt/models/huggingface/qwen3.5-4b-gguf/…`. Image + audio model manifests are in [image.md](image.md) / [audio.md](audio.md). diff --git a/services/studio/README.md b/services/studio/README.md index 8b409b43..7d218ca9 100644 --- a/services/studio/README.md +++ b/services/studio/README.md @@ -34,10 +34,11 @@ python3 build_studio_pipe.py # writes studio_pipe.py ``` Then in Open WebUI: **Admin → Functions → +**, paste the contents of `studio_pipe.py`, -save, enable. Eight models appear in the picker: +save, enable. Nine models appear in the picker: - `🎬 Studio · LTX-2.3` — video + audio (stock model) -- `🔓 Studio · Sulphur` — uncensored video lane +- `🔓 Studio · Sulphur` — uncensored video lane (LTX-2.3-22B-dev fine-tune) +- `🔓 Studio · 10Eros` — uncensored video lane (LTX-2.3-native dev fine-tune; A/B vs Sulphur) - `✨ Studio · Image (HiDream-O1)` — top-quality / photoreal stills (natural-language prompt) - `🖼️ Studio · Image` — Ideogram-4 (graphic design / logo / photo / text) - `🔓 Studio · Image (Chroma)` — uncensored stills (natural-language prompt) @@ -66,13 +67,13 @@ Set the pipe's **Valves** (gear icon on the function): ## Bring it up -`bash scripts/gpu-mode.sh video-studio` brings up ComfyUI (both GPUs) + the director + +`bash scripts/gpu-mode.sh ai-studio` brings up ComfyUI (both GPUs) + the director + the gallery + Open WebUI as a unit. Or start pieces individually: ```bash docker compose -f services/studio/gallery/docker-compose.yml up -d # always-on gallery docker compose -f services/studio/enhancer/docker-compose.yml up -d # director :8090 -bash scripts/gpu-mode.sh comfyui # ComfyUI :8188 +docker compose -f services/comfyui/docker-compose.yml up -d # ComfyUI :8188 ``` ## Use