# Qwen3-Omni-30B-A3B on vLLM-Omni — 2× RTX 3090

Run Alibaba's **Qwen3-Omni-30B-A3B-Instruct** (omni-modal: text / image / audio / video **in**, text **+ speech out**) on two RTX 3090s via the **vLLM-Omni** framework.

> **Status: 🧪 Experimental.** The **text** path is validated on this rig (full 65 K context, needle-in-haystack recall clean at 60 K, coherent output). **Audio/speech output works in principle but is unvalidated here.** This model is **not** in the central registry (`compose_registry.py`) — launch it with **direct `docker compose`**, not `switch.sh` / `launch.sh`.

Compose: [`compose/dual/autoround-int4/omni.yml`](compose/dual/autoround-int4/omni.yml) (+ its mounted `qwen3_omni_3090.yaml` deploy-config).

---

## What you get (and what you don't)

| | |
|---|---|
| **Generates** | text ✅, speech ✅ (real-time TTS of its own reply) |
| **Understands (input)** | text, image, audio, video ✅ |
| **Does NOT generate** | images ❌, video ❌ — it's a conversation model, not a diffusion model |

Architecture = Qwen's 3-stage **Thinker → Talker → Code2Wav** pipeline:
- **Stage 0 — Thinker** (the 30B-A3B MoE text LLM): understands the prompt, generates the **text** reply. *This is all you need for text.*
- **Stage 1 — Talker**: turns the reply into audio codec tokens.
- **Stage 2 — Code2Wav**: vocoder, codec tokens → speech waveform.

On 2× 3090 the stages are **stage-parallel** (one engine per role), **not** tensor-parallel:
`thinker → GPU0` (~23 GB), `talker + code2wav → GPU1` (~16 GB).

---

## Why this exact image (don't "upgrade" it)

Use **`vllm/vllm-omni:v0.20.0`** (the **stable** release: omni 0.20.0 + its matching vLLM 0.20.0). The newer tags are **internally version-skewed and crash on load**:

| Image | Result |
|---|---|
| `vllm/vllm-omni:v0.20.0` | ✅ self-consistent — **use this** |
| `:latest` == `:v0.21.0rc1` | ❌ omni 0.21 + vLLM 0.20 → `ImportError: split_routed_experts` |
| `:v0.22.0rc1` (build) | ❌ needs symbols no released vLLM has → unbuildable |
| stock `vllm/vllm-openai` (any tag) | ❌ wrong tool — crashes profiling the omni MM-encoder (`cu_seqlens must be on CUDA`) |

Re-check only when vLLM-Omni publishes a **correctly-built 0.22 stable** image.

---

## Setup (step by step)

### 1. Get the weights (~25 GB, ungated)
```bash
hf download Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound \
  --local-dir "$MODEL_DIR/qwen3-omni-30b-a3b-instruct-int4-autoround"
```
`$MODEL_DIR` is your models root (the dir mounted as `/models`). int4 AutoRound is mandatory — bf16 is ~60 GB and won't fit 2× 24 GB.

### 2. Pull the engine image
```bash
docker pull vllm/vllm-omni:v0.20.0
```

### 3. Launch (both GPUs must be free)
```bash
cd models/qwen3-omni-30b-a3b/vllm-omni/compose/dual/autoround-int4
MODEL_DIR=/your/models/dir docker compose -f omni.yml up -d
```
First boot takes ~3 min (loads 3 stages + cudagraph capture). Watch it:
```bash
docker logs -f vllm-omni-qwen3-omni-30b   # wait for "Application startup complete"
curl http://localhost:8042/v1/models
```

### 4. Text request — **always pass `"modalities": ["text"]`**
```bash
curl localhost:8042/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "/models/qwen3-omni-30b-a3b-instruct-int4-autoround",
  "modalities": ["text"],
  "messages": [{"role":"user","content":"In two sentences, what is an MoE model?"}],
  "max_tokens": 256
}'
```
> **`"modalities":["text"]` is not optional.** Without it the prompt is routed through the Talker (audio) stage, which hits a prefill shape bug and **takes the whole engine down**.

---

## Performance (measured on this rig)

- **Context:** full **65,536** (native max) on the single-card thinker via fp8 KV. NIAH recall PASS at 60 K (50 % & 90 % depth).
- **Decode (text, `modalities:["text"]`):** **~164 tok/s** single-stream (3 runs, rock-steady) — the thinker's true text rate.
- **Decode (full speech pipeline, no `modalities:text`):** ~12 tok/s — vocoder-bound (Code2Wav, `enforce_eager`). This is the *audio-synthesis* rate, **not** a text number.
- **Prefill:** ~1,800 tok/s (a 60 K prompt ≈ 34 s).
- **VRAM:** GPU0 ~23 GB (thinker), GPU1 ~16 GB (talker + code2wav).

At ~164 tok/s + full 65 K context, Qwen3-Omni is a genuinely **fast text / understanding** model on this rig (in line with our A3B-class models) — and it adds vision/audio input + speech output on top.

---

## Speech / audio output (experimental, unvalidated here)

The talker + code2wav stages are loaded and *can* produce speech, but we only smoke-tested them.
- **fp8 KV (the default) breaks the Code2Wav vocoder** (`snake_activation` dtype error). For audio output you must run **bf16 KV**:
  ```bash
  KV_CACHE_DTYPE=auto MODEL_DIR=/your/models/dir docker compose -f omni.yml up -d
  ```
  (bf16 KV → less context headroom on the thinker.)
- Omit `"modalities":["text"]` (or request audio) to exercise the speech path — expect to debug; the talker prefill path was fragile in our testing.

---

## Environment overrides

| Var | Default | Purpose |
|---|---|---|
| `MODEL_DIR` | `../../../../../../models-cache` | host models root → `/models` |
| `MODEL_SUBDIR` | `qwen3-omni-30b-a3b-instruct-int4-autoround` | weights subdir under `/models` |
| `PORT` | `8042` | host port (container serves on 8091) |
| `KV_CACHE_DTYPE` | `fp8` | `fp8` = full ctx text; `auto` = bf16 (needed for audio out) |
| `VLLM_OMNI_IMAGE` | `vllm/vllm-omni:v0.20.0` | engine image (keep pinned — see above) |

Tune per-stage VRAM/context in the mounted **`qwen3_omni_3090.yaml`** (thinker on GPU0, talker+code2wav on GPU1). If you raise the thinker's `max_model_len`, raise the **talker's too** — the prompt flows through every stage.

---

## Troubleshooting

| Symptom | Cause → Fix |
|---|---|
| `ImportError: split_routed_experts` / `_resolve_module_name` | Wrong image. Use `vllm/vllm-omni:v0.20.0`. |
| `cu_seqlens_q must be on CUDA` | You're on stock `vllm/vllm-openai`, not vLLM-Omni. Use the omni image. |
| Engine dies on a chat request; `_get_talker_assistant_parts` shape error | Missing `"modalities":["text"]` → request hit the talker. Add it. |
| `No available memory for the cache blocks` | Thinker gpu-util too low, or `max_model_len` too high for the KV pool. Keep thinker `gpu_memory_utilization ≥ 0.85`. |
| `snake_activation: expected fp32 got bf16` | fp8 KV + audio. Set `KV_CACHE_DTYPE=auto` for the speech path. |
| OOM on a long prompt | Both stages need `max_model_len ≥ prompt length` (prompt flows through all stages). |

---

## Using GPU1's spare VRAM for image / video generation (future — not done yet)

We looked at co-locating an image-gen model on GPU1 alongside the audio stages and **shelved it.** The binding constraint: every quality image model bundles a **large text encoder** (~8 GB+) that blows the ~8–11 GB co-located budget — even though the *transformer* quantizes small:

| Image model | text encoder | full-pipeline VRAM |
|---|---|---|
| FLUX.1-dev | T5-XXL ~5–8 GB | ~14 GB (Q8) |
| FLUX.2-klein / Z-Image-Turbo | Qwen3-4B ~8 GB | ~10–15 GB |
| FLUX.2-dev | **Mistral-3-24B** | huge — multi-GPU |
| Ideogram-4 | Qwen3-VL-8B (+ dual transformer) | ~18–20 GB (NF4) |

**So don't co-locate — time-share the card.** Since chat and image gen are rarely simultaneous: run omni for conversation, and when you want images either stop omni or run it **thinker-only** to free the full 24 GB of GPU1 for a dedicated image model.

**Best open-weight image models for a freed 24 GB GPU1** (quality-ranked, all run in ComfyUI):

| Model | ~VRAM (quant) | Why |
|---|---|---|
| **FLUX.1-dev** (Q8 GGUF) | ~14 GB | aesthetic/photoreal benchmark; most mature in ComfyUI |
| **Qwen-Image** (Q4 GGUF) | ~14 GB | best **text-in-image** + prompt adherence |
| **HiDream-I1** (Q5) | ~12 GB | top open quality (tight) |
| **FLUX.2-klein-4B** (Q8) | ~8 GB | newer-gen, smaller/faster; lighter ceiling |
| **Z-Image-Turbo** (Q6) | ~10 GB | 6B turbo (sub-second), photoreal; also vLLM-Omni-tested |
| **Ideogram-4** (NF4) | ~18–20 GB | #1 open on DesignArena, but eats the whole card; diffusers/ComfyUI only |
| SD-3.5-medium / Sana | ~5 GB | budget / fast |

**Video** (VRAM-hungry — full card minimum): **Cosmos3-Nano**, **LTX-2 / 2.3** are the feasible-on-one-3090 options; **Wan2.2** (14B) and **HunyuanVideo-1.5** want quant + a full card and are tight.

## UI recommendations

- **Image / video generation → use [ComfyUI](https://github.com/comfyanonymous/ComfyUI), NOT this omni model.** Qwen3-Omni only generates text + speech; image/video are separate models, and ComfyUI is the mature runtime for them (GGUF/NF4/fp8 quants, LoRA, ControlNet, day-0 model support, node-graph workflows).
- **Chat UI → [Open WebUI](https://github.com/open-webui/open-webui).** A clean OpenAI-compatible front-end — point it at this omni endpoint (and the rig's other LLMs) for conversation. It can *also* trigger image generation via a **ComfyUI backend**, so the tidy combined setup is: **Open WebUI as the front door (chat + image requests) → ComfyUI as the image/video engine.**

## Lessons for 2× 3090 (from this deploy — worth knowing)

- **The text encoder is the hidden VRAM cost.** A "4B"/"6B" image model often drags an 8–24 GB encoder. Always size the *full pipeline*, not the transformer.
- **On PCIe-no-NVLink, reach full context with fp8/int8 KV on a single card before reaching for TP/PP.** fp8 KV got the omni thinker to its full 65 K on one card with zero cross-card all-reduce — strictly better than tensor/pipeline-parallel here.
- **Pin pre-release engine images by exact tag + verify on boot.** `latest`/`rc` tags are frequently internally version-skewed (vLLM-Omni `latest` == `v0.21.0rc1` shipped a mismatched vLLM → ImportError). A `docker pull latest` is not reproducible.
- **Request-level flags can swing perf 10×.** This model's text rate went **12 → 164 tok/s** just by adding `"modalities":["text"]`. Bench the path you'll actually use.
- **Quantized diffusion: GGUF Q5/Q6 ≈ near-lossless** and FLUX-class tolerates Q4 well — but GGUF shrinks only the transformer; the encoder needs separate quant or CPU offload.

For general 2× 3090 model guidance see the repo's `docs/DUAL_CARD.md` and `docs/FAQ.md`.

## Notes for maintainers

- **Not registry-wired on purpose** — custom-engine (vLLM-Omni) exploratory deploy. No `compose_registry.py` entry, no `switch.sh`/`launch.sh`; direct `docker compose` only.
