# Audio Studio — voices · music · SFX (chat-driven, on 2× 3090)

The audio side of [Club 3090 AI Studio](README.md). Four **generate** lanes share the pipe + the
director, output `.mp3`/`.wav` into the gallery, and refine by reply like every other lane:

| Lane | Model | What | Where |
|---|---|---|---|
| `🎙️ Voice` | **Step-Audio-EditX** (3B, Apache) | premium **zero-shot voice clone** + emotion/style **editing** | isolated `step-voice` service `:8193` (GPU, on-demand) |
| *(narration)* | **Kokoro-82M** (Apache) | voiceover **mixed onto a video** clip | `studio-tts` service `:8192` (CPU) |
| `🎵 Music` | **ACE-Step v1 3.5B** | songs + instrumentals | ComfyUI lane, GPU0 |
| `🔊 SFX` | **Stable Audio Open 1.0** | sound effects / ambience / textures | ComfyUI lane, GPU0 |

> **Two pillars.** This doc covers the **Generate** pillar (synthesize audio). The mirror
> **Understand** pillar — transcribe (Whisper), diarize who-spoke-when (pyannote/WhisperX),
> separate overlapping voices (SepFormer) — feeds the future realtime voice-agent (calling /
> bookings) and is tracked in the private design notes, not yet built.

## Architecture

```
                          Browser
                             │  "a lofi beat" · "rain on a tin roof" · "say: welcome to the show"
                             ▼
              ┌──────────────────────────────────────────────────┐
              │  Open WebUI   :8080   (the front-end)             │
              │  audio lanes: 🎵 Music · 🔊 SFX · 🎙️ Voice          │
              └───────┬──────────────────────────────────┬───────┘
                  (1) │ craft the spec                (2) │ render
                      ▼                                   ▼
        ┌──────────────────────────┐   ┌───────────────────────────────────┐
        │ Director   :8090         │   │ Renderer (per lane)               │
        │ qwen3.5-4b · GPU0        │   │  🎵 Music · 🔊 SFX → ComfyUI :8188 │
        │ idea → tags + lyrics     │   │     ACE-Step / Stable Audio (GPU0) │
        │   (music) · sound prompt │   │  🎙️ Voice → step-voice :8193       │
        │   (SFX) · voice speaks   │   │     Step-Audio-EditX (isolated,    │
        │   the text verbatim      │   │     transformers 4.53.3, GPU)      │
        └──────────────────────────┘   └─────────────┬─────────────────────┘
                                                      ▼
        voiceover-on-video:                ┌───────────────────────────┐
        studio-tts :8192 (Kokoro, CPU)     │ Gallery   :8189            │
        ducks a voice onto the clip's      │ nginx over /output         │
        native bed via ffmpeg              │ (.mp3 / .wav)              │
                                           └─────────────┬─────────────┘
                                                         ▼  🎧 link back in chat
                                                      Browser  (reply "more upbeat" to refine)
```

Music/SFX are ComfyUI lanes on GPU0; the **premium voice** runs in its own isolated `step-voice` service; **Kokoro narration** is the over-video path (CPU). See [README.md](README.md) for the full substrate.

---

## 🎙️ Premium voice — Step-Audio-EditX (clone + edit)

The **`🎙️ Studio · Voice`** lane is the quality voice tier: **zero-shot voice cloning** (no
training — a 10–30 s reference clip + its transcript → speak any text in that voice) plus
iterative **audio editing** (emotion, speaking style, paralinguistics, speed). Step-Audio-EditX
is **3B, Apache-2.0** — the only commercially-licensed one of the top open TTS models (Fish
S2-Pro and Higgs v3 are research/non-commercial).

- **Ask for it:** pick the 🎙️ lane and type what you want spoken. It's cloned in the
  `voice_reference` voice (a bundled sample — `Narrator.wav` / `Narrator-UK.wav` / `Pirates.wav`
  — by default). Point `voice_reference` at your own clean clip to clone **your** voice.
- **No transcript needed:** if you don't give the reference's transcript, the service
  auto-transcribes it with Whisper first.
- **Why a separate service, not a ComfyUI lane:** Step-Audio-EditX hard-pins
  **`transformers==4.53.3`** (4.54+ produces *silent audio*, per the authors), which conflicts
  with ComfyUI's transformers 5.x (HiDream needs 5.x). So it runs in its **own isolated
  container** (`services/studio/step-voice/`, `:8193`) pinned to the exact version — guaranteed
  correct, zero conflict. The pipe POSTs `/clone`; the service writes a 24 kHz `.wav` to the gallery.
- **VRAM / serving:** ~14 GB bf16 on a pinned card (GPU1 by default), **on-demand** (not
  always-on — bring it up with `docker compose -f services/studio/step-voice/docker-compose.yml
  up -d`, or start it from **c3 → Operate → Containers**). An AWQ-4bit build (~3–4 GB) is the
  future light-deploy option. Validated: ~30 s to load, then a clip in seconds.
- **⊕ Mutually exclusive with an active video render.** In `ai-studio`, video uses *both* 3090s
  (the 22B DiT donates ~22 GB to GPU1 via DisTorch), and premium voice wants ~14 GB on that same
  GPU1 — they can't both be resident. **c3 guards this**: starting `step-voice` while a video
  render holds GPU1 is blocked with a "GPU1 busy (video)" notice; let the render finish (or stop
  ComfyUI's video lane) first. Music/SFX/image lanes are GPU0 and don't conflict with voice.

> Step-Audio-EditX is **generate-only** — it clones + edits speech, it does **not** diarize or
> separate multi-speaker recordings (that's the Understand pillar).

## Integrated audio — voices for video (Kokoro)

The video lanes already render a clip *with* native ambient audio. The Studio can add a
**voiceover/narration** on top: include a directive in your message and the pipe generates a
voice and **mixes it over the clip**.

- **Ask for it:** *"a fox padding through a neon alley at night, **voiceover: the city never
  sleeps, and neither do we**"* — or `narration: "..."`, or `say: ...`. The pipe pulls the spoken
  line out (so it doesn't pollute the video prompt), renders the clip, then narrates.
- **Engine:** **`studio-tts`** (`:8192`) — **Kokoro-82M** on **CPU** (never touches the GPUs),
  so it adds no VRAM pressure and runs after the render (voice ≈ 1–2 s of compute). Pick a voice
  with the `narrate_voice` valve (`af_heart`, `am_adam`, `bf_emma`, …).
- **Layer-aware mixdown** (ffmpeg): the Kokoro voice is mixed over the clip's native audio, the
  **bed is ducked** under the voice (`sidechaincompress`), and the master is **loudness-normalized**
  (`loudnorm`, −16 LUFS); output is capped to the clip length. The mix stage is structured to
  accept more layers (generated music / SFX) later without a rewrite.

> Kokoro is the **fast, zero-VRAM, low-latency** voice (and the right TTS for a future realtime
> voice agent); Step-Audio-EditX is the **premium / cloned / editable** voice. Two tiers, like
> the image lanes.

## 🎵 Music — ACE-Step

The **`🎵 Studio · Music`** lane generates **songs + instrumentals** from a text idea on
**ACE-Step v1 (3.5B)** — a standalone audio track (not muxed onto video; that's the voiceover
path above).

- **Ask for it:** pick the 🎵 lane and describe the music — *"upbeat synthwave instrumental"*,
  *"a melancholic piano ballad about the sea"*, *"a lofi hip-hop beat to study to"*. Add a length
  (*"a 30-second…"*) or it defaults to ~60 s.
- **Director → tags + lyrics:** ACE-Step takes **tags** (genre / mood / instruments / tempo /
  vocal type) plus **lyrics** (`[verse]`/`[chorus]` structure) or `[instrumental]`. The director
  turns your idea into both — real singable lyrics for a song, or `[instrumental]` for a
  beat/score. Refine like everything else (*"more upbeat"*, *"add a sax solo"*, *"make it instrumental"*).
- **Single-device GPU0** (~8 GB, 50-step euler, cfg 5) — light enough to be a **lane** (coexists
  with the director on GPU0), not a separate mutex mode. ~6–16 s for a short clip. Output is an
  `.mp3` in the gallery. Uses the bundled ACE-Step ComfyUI nodes.

## 🔊 SFX — Stable Audio

The **`🔊 Studio · SFX`** lane generates **sound effects, ambiences, and textures** from a text
description on **Stable Audio Open 1.0** — distinct from the Music lane (songs/instrumentals).

- **Ask for it:** pick the 🔊 lane and describe a sound — *"rain on a tin roof"*, *"sci-fi door
  whoosh"*, *"forest ambience with distant birds"*. Add a length (*"a 5-second…"*) or it defaults
  to ~10 s. **Capped at 47 s** (the model's max).
- **Director → sound prompt:** the director turns your idea into a concrete sound description
  (source, materials, acoustic space, motion). Refine like the rest (*"more distant"*, *"add
  reverb"*, *"heavier rain"*).
- **Single-device GPU0** — a lane like music. Output is an `.mp3` in the gallery. Reuses the
  bundled Stable Audio ComfyUI nodes (Stable Audio Open 1.0 + a T5-base encoder).

## Models (obtain separately → `/mnt/models/comfyui/models/...`)

| File | Dir | Lane |
|---|---|---|
| `kokoro-v1.0.onnx` + `voices-v1.0.bin` ([onnx-community/Kokoro-82M-v1.0-ONNX](https://huggingface.co/onnx-community/Kokoro-82M-v1.0-ONNX)) | `tts/kokoro/` | integrated voices (CPU) |
| `Step-Audio-EditX/` + `Step-Audio-Tokenizer/` (stepfun-ai) | `Step-Audio/` | premium voice (`step-voice` service) |
| `ace_step_v1_3.5b.safetensors` (ACE-Step v1 3.5B) | `checkpoints/ace-step-1.5/all_in_one/` | music |
| `stable-audio-open-1.0.safetensors` (Comfy-Org repackaged) + `t5-base.safetensors` | `checkpoints/`, `text_encoders/` | SFX |
