club-3090/models/qwen3-omni-30b-a3b/vllm-omni/README.md

Qwen3-Omni-30B-A3B on vLLM-Omni — 2× RTX 3090

Run Alibaba's Qwen3-Omni-30B-A3B-Instruct (omni-modal: text / image / audio / video in, text + speech out) on two RTX 3090s via the vLLM-Omni framework.

Status: 🧪 Experimental. The text path is validated on this rig (full 65 K context, needle-in-haystack recall clean at 60 K, coherent output). Audio/speech output works in principle but is unvalidated here. This model is not in the central registry (compose_registry.py) — launch it with direct docker compose, not switch.sh / launch.sh.

Compose: compose/dual/autoround-int4/omni.yml (+ its mounted qwen3_omni_3090.yaml deploy-config).

---

What you get (and what you don't)

Generates	text ✅, speech ✅ (real-time TTS of its own reply)
Understands (input)	text, image, audio, video ✅
Does NOT generate	images ❌, video ❌ — it's a conversation model, not a diffusion model

Architecture = Qwen's 3-stage Thinker → Talker → Code2Wav pipeline:

• Stage 0 — Thinker (the 30B-A3B MoE text LLM): understands the prompt, generates the text reply. This is all you need for text.
• Stage 1 — Talker: turns the reply into audio codec tokens.
• Stage 2 — Code2Wav: vocoder, codec tokens → speech waveform.

On 2× 3090 the stages are stage-parallel (one engine per role), not tensor-parallel: thinker → GPU0 (~23 GB), talker + code2wav → GPU1 (~16 GB).

---

Why this exact image (don't "upgrade" it)

Use vllm/vllm-omni:v0.20.0 (the stable release: omni 0.20.0 + its matching vLLM 0.20.0). The newer tags are internally version-skewed and crash on load:

Image	Result
vllm/vllm-omni:v0.20.0	✅ self-consistent — use this
:latest == :v0.21.0rc1	❌ omni 0.21 + vLLM 0.20 → ImportError: split_routed_experts
:v0.22.0rc1 (build)	❌ needs symbols no released vLLM has → unbuildable
stock vllm/vllm-openai (any tag)	❌ wrong tool — crashes profiling the omni MM-encoder (cu_seqlens must be on CUDA)

Re-check only when vLLM-Omni publishes a correctly-built 0.22 stable image.

---

Setup (step by step)

1. Get the weights (~25 GB, ungated)

hf download Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound \
  --local-dir "$MODEL_DIR/qwen3-omni-30b-a3b-instruct-int4-autoround"

$MODEL_DIR is your models root (the dir mounted as /models). int4 AutoRound is mandatory — bf16 is ~60 GB and won't fit 2× 24 GB.

2. Pull the engine image

docker pull vllm/vllm-omni:v0.20.0

3. Launch (both GPUs must be free)

cd models/qwen3-omni-30b-a3b/vllm-omni/compose/dual/autoround-int4
MODEL_DIR=/your/models/dir docker compose -f omni.yml up -d

First boot takes ~3 min (loads 3 stages + cudagraph capture). Watch it:

docker logs -f vllm-omni-qwen3-omni-30b   # wait for "Application startup complete"
curl http://localhost:8042/v1/models

4. Text request — always pass "modalities": ["text"]

curl localhost:8042/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "/models/qwen3-omni-30b-a3b-instruct-int4-autoround",
  "modalities": ["text"],
  "messages": [{"role":"user","content":"In two sentences, what is an MoE model?"}],
  "max_tokens": 256
}'

"modalities":["text"] is not optional. Without it the prompt is routed through the Talker (audio) stage, which hits a prefill shape bug and takes the whole engine down.

---

Performance (measured on this rig)

• Context: full 65,536 (native max) on the single-card thinker via fp8 KV. NIAH recall PASS at 60 K (50 % & 90 % depth).
• Decode (text, modalities:["text"]): ~164 tok/s single-stream (3 runs, rock-steady) — the thinker's true text rate.
• Decode (full speech pipeline, no modalities:text): ~12 tok/s — vocoder-bound (Code2Wav, enforce_eager). This is the audio-synthesis rate, not a text number.
• Prefill: ~1,800 tok/s (a 60 K prompt ≈ 34 s).
• VRAM: GPU0 ~23 GB (thinker), GPU1 ~16 GB (talker + code2wav).

At ~164 tok/s + full 65 K context, Qwen3-Omni is a genuinely fast text / understanding model on this rig (in line with our A3B-class models) — and it adds vision/audio input + speech output on top.

---

Speech / audio output (experimental, unvalidated here)

The talker + code2wav stages are loaded and can produce speech, but we only smoke-tested them.

• fp8 KV (the default) breaks the Code2Wav vocoder (snake_activation dtype error). For audio output you must run bf16 KV:

  KV_CACHE_DTYPE=auto MODEL_DIR=/your/models/dir docker compose -f omni.yml up -d
  

  (bf16 KV → less context headroom on the thinker.)
• Omit "modalities":["text"] (or request audio) to exercise the speech path — expect to debug; the talker prefill path was fragile in our testing.

---

Environment overrides

Var	Default	Purpose
MODEL_DIR	../../../../../../models-cache	host models root → /models
MODEL_SUBDIR	qwen3-omni-30b-a3b-instruct-int4-autoround	weights subdir under /models
PORT	8042	host port (container serves on 8091)
KV_CACHE_DTYPE	fp8	fp8 = full ctx text; auto = bf16 (needed for audio out)
VLLM_OMNI_IMAGE	vllm/vllm-omni:v0.20.0	engine image (keep pinned — see above)

Tune per-stage VRAM/context in the mounted qwen3_omni_3090.yaml (thinker on GPU0, talker+code2wav on GPU1). If you raise the thinker's max_model_len, raise the talker's too — the prompt flows through every stage.

---

Troubleshooting

Symptom	Cause → Fix
ImportError: split_routed_experts / _resolve_module_name	Wrong image. Use vllm/vllm-omni:v0.20.0.
cu_seqlens_q must be on CUDA	You're on stock vllm/vllm-openai, not vLLM-Omni. Use the omni image.
Engine dies on a chat request; _get_talker_assistant_parts shape error	Missing "modalities":["text"] → request hit the talker. Add it.
No available memory for the cache blocks	Thinker gpu-util too low, or max_model_len too high for the KV pool. Keep thinker gpu_memory_utilization ≥ 0.85.
snake_activation: expected fp32 got bf16	fp8 KV + audio. Set KV_CACHE_DTYPE=auto for the speech path.
OOM on a long prompt	Both stages need max_model_len ≥ prompt length (prompt flows through all stages).

---

Using GPU1's spare VRAM for image / video generation (future — not done yet)

We looked at co-locating an image-gen model on GPU1 alongside the audio stages and shelved it. The binding constraint: every quality image model bundles a large text encoder (~8 GB+) that blows the ~8–11 GB co-located budget — even though the transformer quantizes small:

Image model	text encoder	full-pipeline VRAM
FLUX.1-dev	T5-XXL ~5–8 GB	~14 GB (Q8)
FLUX.2-klein / Z-Image-Turbo	Qwen3-4B ~8 GB	~10–15 GB
FLUX.2-dev	Mistral-3-24B	huge — multi-GPU
Ideogram-4	Qwen3-VL-8B (+ dual transformer)	~18–20 GB (NF4)

So don't co-locate — time-share the card. Since chat and image gen are rarely simultaneous: run omni for conversation, and when you want images either stop omni or run it thinker-only to free the full 24 GB of GPU1 for a dedicated image model.

Best open-weight image models for a freed 24 GB GPU1 (quality-ranked, all run in ComfyUI):

Model	~VRAM (quant)	Why
FLUX.1-dev (Q8 GGUF)	~14 GB	aesthetic/photoreal benchmark; most mature in ComfyUI
Qwen-Image (Q4 GGUF)	~14 GB	best text-in-image + prompt adherence
HiDream-I1 (Q5)	~12 GB	top open quality (tight)
FLUX.2-klein-4B (Q8)	~8 GB	newer-gen, smaller/faster; lighter ceiling
Z-Image-Turbo (Q6)	~10 GB	6B turbo (sub-second), photoreal; also vLLM-Omni-tested
Ideogram-4 (NF4)	~18–20 GB	#1 open on DesignArena, but eats the whole card; diffusers/ComfyUI only
SD-3.5-medium / Sana	~5 GB	budget / fast

Video (VRAM-hungry — full card minimum): Cosmos3-Nano, LTX-2 / 2.3 are the feasible-on-one-3090 options; Wan2.2 (14B) and HunyuanVideo-1.5 want quant + a full card and are tight.

UI recommendations

• Image / video generation → use ComfyUI, NOT this omni model. Qwen3-Omni only generates text + speech; image/video are separate models, and ComfyUI is the mature runtime for them (GGUF/NF4/fp8 quants, LoRA, ControlNet, day-0 model support, node-graph workflows).
• Chat UI → Open WebUI. A clean OpenAI-compatible front-end — point it at this omni endpoint (and the rig's other LLMs) for conversation. It can also trigger image generation via a ComfyUI backend, so the tidy combined setup is: Open WebUI as the front door (chat + image requests) → ComfyUI as the image/video engine.

Lessons for 2× 3090 (from this deploy — worth knowing)

• The text encoder is the hidden VRAM cost. A "4B"/"6B" image model often drags an 8–24 GB encoder. Always size the full pipeline, not the transformer.
• On PCIe-no-NVLink, reach full context with fp8/int8 KV on a single card before reaching for TP/PP. fp8 KV got the omni thinker to its full 65 K on one card with zero cross-card all-reduce — strictly better than tensor/pipeline-parallel here.
• Pin pre-release engine images by exact tag + verify on boot. latest/rc tags are frequently internally version-skewed (vLLM-Omni latest == v0.21.0rc1 shipped a mismatched vLLM → ImportError). A docker pull latest is not reproducible.
• Request-level flags can swing perf 10×. This model's text rate went 12 → 164 tok/s just by adding "modalities":["text"]. Bench the path you'll actually use.
• Quantized diffusion: GGUF Q5/Q6 ≈ near-lossless and FLUX-class tolerates Q4 well — but GGUF shrinks only the transformer; the encoder needs separate quant or CPU offload.

For general 2× 3090 model guidance see the repo's docs/DUAL_CARD.md and docs/FAQ.md.

Notes for maintainers

• Not registry-wired on purpose — custom-engine (vLLM-Omni) exploratory deploy. No compose_registry.py entry, no switch.sh/launch.sh; direct docker compose only.