composes: env-override knobs MAX_MODEL_LEN + GPU_MEMORY_UTILIZATION (#79)

* composes: env-override knobs MAX_MODEL_LEN + GPU_MEMORY_UTILIZATION Two cross-rig users hit the same friction within hours of each other: - @laurimyllari (4090, disc #62 / issue #71): default max_model_len=180000 on long-text.yml exceeded his rig's KV-cache budget; had to drop to 90K - @PiotrZadka (disc #66): wants to run vLLM alongside a desktop session on the same GPU; needs to reserve some VRAM for X server / browser etc. Both root causes are the same: composes were calibrated for headless 3090 with no other VRAM consumers, and there's no clean override path short of hand-editing the YAML. Add env-substitution for the two knobs with the highest "shrink to fit" elasticity: --max-model-len ${MAX_MODEL_LEN:-<existing default>} --gpu-memory-utilization ${GPU_MEMORY_UTILIZATION:-<existing default>} Existing defaults preserved verbatim — zero behavior change for users who don't set the env. Pattern matches existing ${MODEL_DIR:-...}, ${PORT:-...}, ${HF_TOKEN:-...} substitutions in our composes. Usage: # Default (unchanged): bash scripts/switch.sh vllm/long-text # max_model_len=180000 # Override for desktop-coexist or sub-24 GB VRAM: MAX_MODEL_LEN=32768 GPU_MEMORY_UTILIZATION=0.80 \\ bash scripts/switch.sh vllm/long-text Validated end-to-end on long-text.yml: $ MAX_MODEL_LEN=32768 GPU_MEMORY_UTILIZATION=0.80 \\ bash scripts/switch.sh vllm/long-text $ docker exec vllm-qwen36-27b-long-text ps aux | grep vllm ... --max-model-len 32768 --gpu-memory-utilization 0.80 ... Default boot also unaffected: $ docker compose -f .../docker-compose.long-text.yml config | \\ grep -A1 "max-model-len\|gpu-memory-utilization" - --max-model-len - "180000" # original default preserved - --gpu-memory-utilization - "0.93" # original default preserved 18 vLLM composes touched. Mechanical find/replace; one YAML pattern, no logic changes. Doesn't touch llama-cpp composes (different flag shape; out of scope for this PR). Closes the friction reported on disc #62 + disc #66. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: env-override knobs for desktop-coexist + sub-24 GB usable VRAM Document MAX_MODEL_LEN + GPU_MEMORY_UTILIZATION env overrides shipped in the composes. Two surfaces: - docs/SINGLE_CARD.md "Running alongside a desktop / sub-24 GB usable VRAM" — explains the override pattern, when to drop which knob, safe ranges. Calls out empirically that GPU_MEMORY_UTILIZATION=0.80 is too aggressive for TQ3 KV paths (vLLM profiling overhead consumes more than the saved 0.05 budget; engine init reports 'No available memory for the cache blocks'). - docs/HARDWARE.md "Note for sub-24 GB cards" — adds a 4090-with-display paragraph above the existing 20 GB modded-3080 note. Cites @laurimyllari's MAX_MODEL_LEN=90000 fit on 4090 long-text.yml. Validated end-to-end during the env-override implementation: - ps aux inside container confirms override values reach vLLM CLI - Default boot unchanged (env unset → original values reach CLI) - 0.80 mem-util on long-text dies cleanly with the documented error, which is what surfaced the "stay 0.85-0.92 for TQ3" guidance Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 15:25:10 +05:00
parent 461c4d4d3d
commit da0c7e77d0
20 changed files with 61 additions and 36 deletions
--- a/docs/HARDWARE.md
+++ b/docs/HARDWARE.md
@@ -35,6 +35,14 @@ The recipes are written against 3090 specifically but should work on:

 On 20 GB cards (modded 3080) the cudagraph-profiling overhead is a meaningful slice of available VRAM. Drop `--gpu-memory-utilization` to **0.82** (vs shipped 0.95 for 24 GB). vLLM nightly's `gpu_worker.py` reports the equivalent effective KV size in the boot log; tune to keep activation headroom for the ~15K tool-prefill peak (verify-full check 8). Credit: [@troymroberts](https://github.com/troymroberts).

+**4090s with attached display — env-override the compose defaults.** Some 4090 rigs land at ~23.5 GB usable VRAM with X server + driver overhead, vs the headless 3090s the composes are calibrated for. Boot may fail with `No available memory for the cache blocks` at default `max-model-len`. Cross-rig data: @laurimyllari's 4090 single-card on `long-text.yml` needed `MAX_MODEL_LEN=90000` (down from 180K default) to fit cleanly ([disc #62](../../../noonghunna/club-3090/discussions/62) / [issue #71](../../../noonghunna/club-3090/issues/71)). Pattern:
+
+```bash
+MAX_MODEL_LEN=90000 bash scripts/switch.sh vllm/long-text
+```
+
+Same `MAX_MODEL_LEN` / `GPU_MEMORY_UTILIZATION` env overrides apply for any setup running vLLM alongside other GPU consumers on the same card. See [SINGLE_CARD.md "Running alongside a desktop"](SINGLE_CARD.md#running-alongside-a-desktop--sub-24-gb-usable-vram) for safe ranges.
+
 **`dual-turbo.yml` on 20 GB Ampere — swap TQ3 KV → fp8_e5m2.** The shipped `dual-turbo.yml` uses `--kv-cache-dtype turboquant_3bit_nc` (the technique from [TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate](https://arxiv.org/abs/2504.19874), ICLR 2026 — random rotation + scalar quantizers + 1-bit QJL transform on the residual; the paper claims absolute quality neutrality at 3.5 bits/channel). It's the right pick on 24 GB / 3090: smaller KV pool → more concurrency, and the 24 GB budget absorbs the dequant activation cost during the DeltaNet GDN forward. On 20 GB cards the trade flips: TQ3's activation peak (~1 GB/card more pressure than fp8 during the materialized block — see [PerfMamba arxiv 2511.22849](https://arxiv.org/html/2511.22849) for the underlying Mamba-2 block-state-materialization mechanism the GDN forward inherits) exceeds the per-card budget after TP=2 split, and Cliff 2 fires at 90K. **Override to `--kv-cache-dtype fp8_e5m2`** and you get the full 262K context working with verify-stress 7/7 PASS including 91K needles. Validated 2026-05-04 by [@efschu](https://github.com/noonghunna/club-3090/issues/47) on 2× 3080 modded 20 GB at 0.82 mem-util: bench 82.4 narr / 107.9 code TPS, full 257K-token auto-discovery needle PASS at 90% depth. Trade-off: fp8 KV is roomier per cached token but each token's KV state is larger, so concurrency at full ctx drops vs TQ3. Single-stream long-ctx works cleanly.

 ---
--- a/docs/SINGLE_CARD.md
+++ b/docs/SINGLE_CARD.md
@@ -179,6 +179,23 @@ What still keeps it off the recommended list:

 vLLM ships this off by default. Our composes set `--tool-call-parser qwen3_coder` + `--enable-auto-tool-choice`. If you're rolling your own compose, both are required.

+### Running alongside a desktop / sub-24 GB usable VRAM
+
+The compose defaults are calibrated for **headless** 3090 (no display server, no other GPU consumers). If you're running on a workstation where the same GPU also drives a desktop session — or your card has slightly less effective VRAM (e.g., some 4090s land at ~23.5 GB usable with X server overhead) — the default `max-model-len` may exceed your KV-cache budget at default `gpu-memory-utilization`.
+
+Two env-override knobs available on every vLLM compose (defaults preserved if unset):
+
+```bash
+MAX_MODEL_LEN=32768 \
+GPU_MEMORY_UTILIZATION=0.80 \
+  bash scripts/switch.sh vllm/long-text
+```
+
+- `MAX_MODEL_LEN` — shrink the KV-cache budget; trades long-context for fit. `90000` is a safe value for `long-text.yml` on rigs with ~1 GB of overhead (4090 with display, etc.). Validated on @laurimyllari's 4090 ([disc #62](../../../noonghunna/club-3090/discussions/62)).
+- `GPU_MEMORY_UTILIZATION` — reserve a percentage of VRAM for non-vLLM GPU consumers. **Stay in `0.85-0.92` range** for TQ3 KV paths on 24 GB Ampere. `0.80` is generally too aggressive — vLLM's profiling phase eats more than the saved 0.05 budget and reports `No available memory for the cache blocks` at engine init. fp8 KV paths (`tools-text.yml`, `dual.yml`) tolerate `0.80` better.
+
+Generally prefer **dropping `MAX_MODEL_LEN` first** (clean KV budget reduction, predictable behavior) over `GPU_MEMORY_UTILIZATION` (interacts with profiling overhead in non-obvious ways). Drop both if your envelope is really tight.
+
 ---

 ## Quick start
--- a/models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp-tp1.yml
+++ b/models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp-tp1.yml
@@ -104,9 +104,9 @@ services:
      # to reclaim ~1.5 GB and skip the mm-token-budget assertion.
      # Drop max-model-len 32K → 16K for headroom.
      - --max-model-len
-      - "8192"
+      - "${MAX_MODEL_LEN:-8192}"
      - --gpu-memory-utilization
-      - "0.95"
+      - "${GPU_MEMORY_UTILIZATION:-0.95}"
      - --max-num-seqs
      - "256"
      # max_tokens_per_mm_item=2496 must fit in batched tokens — even with
--- a/models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp.yml
+++ b/models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp.yml
@@ -101,9 +101,9 @@ services:
      - "2"
      - --disable-custom-all-reduce
      - --max-model-len
-      - "32768"
+      - "${MAX_MODEL_LEN:-32768}"
      - --gpu-memory-utilization
-      - "0.92"
+      - "${GPU_MEMORY_UTILIZATION:-0.92}"
      - --max-num-seqs
      - "4"
      # Vision tower: max_tokens_per_mm_item=2496 must fit in batched tokens.
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml
@@ -257,9 +257,9 @@ services:
      # on 2026-05-02 to give activation headroom for the PN12+PN25 FFN pool
      # residence + DeltaNet GDN buffer. See long-text.yml for full rationale.
      - --max-model-len
-      - "180000"
+      - "${MAX_MODEL_LEN:-180000}"
      - --gpu-memory-utilization
-      - "0.95"
+      - "${GPU_MEMORY_UTILIZATION:-0.95}"
      - --max-num-seqs
      - "1"
      - --max-num-batched-tokens
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.carnice-bf16mtp.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.carnice-bf16mtp.yml
@@ -91,9 +91,9 @@ services:
      - "2"
      - --disable-custom-all-reduce
      - --max-model-len
-      - "262144"
+      - "${MAX_MODEL_LEN:-262144}"
      - --gpu-memory-utilization
-      - "0.92"
+      - "${GPU_MEMORY_UTILIZATION:-0.92}"
      - --max-num-seqs
      - "2"
      - --max-num-batched-tokens
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash-noviz.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash-noviz.yml
@@ -79,9 +79,9 @@ services:
      - "2"
      - --disable-custom-all-reduce
      - --max-model-len
-      - "200000"
+      - "${MAX_MODEL_LEN:-200000}"
      - --gpu-memory-utilization
-      - "0.95"
+      - "${GPU_MEMORY_UTILIZATION:-0.95}"
      - --max-num-seqs
      - "1"
      - --max-num-batched-tokens
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash.yml
@@ -103,9 +103,9 @@ services:
      - "2"
      - --disable-custom-all-reduce
      - --max-model-len
-      - "185000"
+      - "${MAX_MODEL_LEN:-185000}"
      - --gpu-memory-utilization
-      - "0.95"
+      - "${GPU_MEMORY_UTILIZATION:-0.95}"
      - --max-num-seqs
      - "1"
      - --max-num-batched-tokens
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual-nvlink-turbo.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual-nvlink-turbo.yml
@@ -212,9 +212,9 @@ services:
      # makes vLLM's custom kernel a win. dual-turbo.yml disables it because
      # PCIe P2P bandwidth makes the NCCL fallback faster there.
      - --max-model-len
-      - "262144"
+      - "${MAX_MODEL_LEN:-262144}"
      - --gpu-memory-utilization
-      - "0.85"
+      - "${GPU_MEMORY_UTILIZATION:-0.85}"
      - --max-num-seqs
      - "4"
      - --max-num-batched-tokens
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual-nvlink.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual-nvlink.yml
@@ -104,9 +104,9 @@ services:
      # makes vLLM's custom kernel a win. dual.yml disables it because PCIe
      # P2P bandwidth makes the NCCL fallback faster there.
      - --max-model-len
-      - "262144"
+      - "${MAX_MODEL_LEN:-262144}"
      - --gpu-memory-utilization
-      - "0.92"
+      - "${GPU_MEMORY_UTILIZATION:-0.92}"
      - --max-num-seqs
      - "2"
      - --max-num-batched-tokens
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual-turbo.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual-turbo.yml
@@ -189,9 +189,9 @@ services:
      - "2"
      - --disable-custom-all-reduce
      - --max-model-len
-      - "262144"
+      - "${MAX_MODEL_LEN:-262144}"
      - --gpu-memory-utilization
-      - "0.85"
+      - "${GPU_MEMORY_UTILIZATION:-0.85}"
      - --max-num-seqs
      - "4"
      - --max-num-batched-tokens
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual.yml
@@ -90,9 +90,9 @@ services:
      - "2"
      - --disable-custom-all-reduce
      - --max-model-len
-      - "262144"
+      - "${MAX_MODEL_LEN:-262144}"
      - --gpu-memory-utilization
-      - "0.92"
+      - "${GPU_MEMORY_UTILIZATION:-0.92}"
      - --max-num-seqs
      - "2"
      - --max-num-batched-tokens
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual4-dflash.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual4-dflash.yml
@@ -98,9 +98,9 @@ services:
      - "4"
      - --disable-custom-all-reduce
      - --max-model-len
-      - "262144"
+      - "${MAX_MODEL_LEN:-262144}"
      - --gpu-memory-utilization
-      - "0.95"
+      - "${GPU_MEMORY_UTILIZATION:-0.95}"
      - --max-num-seqs
      - "2"
      - --max-num-batched-tokens
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual4.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual4.yml
@@ -102,9 +102,9 @@ services:
      - "4"
      - --disable-custom-all-reduce
      - --max-model-len
-      - "262144"
+      - "${MAX_MODEL_LEN:-262144}"
      - --gpu-memory-utilization
-      - "0.92"
+      - "${GPU_MEMORY_UTILIZATION:-0.92}"
      - --max-num-seqs
      - "4"
      - --max-num-batched-tokens
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.long-text-no-mtp.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.long-text-no-mtp.yml
@@ -308,7 +308,7 @@ services:
      # club-3090#16 — different bug (Cliff 1 mech B inductor leak), not
      # fixed by mem-util tuning. Use tools-text.yml for those.
      - --max-model-len
-      - "200000"
+      - "${MAX_MODEL_LEN:-200000}"
      # 0.95 — backed off from 0.97 on 2026-05-01 PM after PN25 v3 closed
      # Cliff 1 mech B at the FFN intermediate buffer (which now sits in
      # PN12+PN25's resident FFNIntermediateCache pool). Pool residence
@@ -317,7 +317,7 @@ services:
      # 0.97 left only 26 MiB free, OOM at 9.8K probe. 0.95 frees ~480 MiB
      # activation budget, validated up to 30K probes.
      - --gpu-memory-utilization
-      - "0.95"
+      - "${GPU_MEMORY_UTILIZATION:-0.95}"
      - --max-num-seqs
      - "1"
      - --max-num-batched-tokens
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.long-text.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.long-text.yml
@@ -325,7 +325,7 @@ services:
      # club-3090#16 — different bug (Cliff 1 mech B inductor leak), not
      # fixed by mem-util tuning. Use tools-text.yml for those.
      - --max-model-len
-      - "180000"
+      - "${MAX_MODEL_LEN:-180000}"
      # 0.95 — backed off from 0.97 on 2026-05-01 PM after PN25 v3 closed
      # Cliff 1 mech B at the FFN intermediate buffer (which now sits in
      # PN12+PN25's resident FFNIntermediateCache pool). Pool residence
@@ -334,7 +334,7 @@ services:
      # 0.97 left only 26 MiB free, OOM at 9.8K probe. 0.95 frees ~480 MiB
      # activation budget, validated up to 30K probes.
      - --gpu-memory-utilization
-      - "0.93"
+      - "${GPU_MEMORY_UTILIZATION:-0.93}"
      - --max-num-seqs
      - "1"
      - --max-num-batched-tokens
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.long-vision.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.long-vision.yml
@@ -231,9 +231,9 @@ services:
      # Engine pre-check at 175K + 0.95 returned `estimated maximum model length
      # is 148608` — falls back to 145K for safety margin.
      - --max-model-len
-      - "145000"
+      - "${MAX_MODEL_LEN:-145000}"
      - --gpu-memory-utilization
-      - "0.95"
+      - "${GPU_MEMORY_UTILIZATION:-0.95}"
      - --max-num-seqs
      - "1"
      - --max-num-batched-tokens
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.minimal.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.minimal.yml
@@ -64,9 +64,9 @@ services:
      - --tensor-parallel-size
      - "1"
      - --max-model-len
-      - "32768"
+      - "${MAX_MODEL_LEN:-32768}"
      - --gpu-memory-utilization
-      - "0.92"
+      - "${GPU_MEMORY_UTILIZATION:-0.92}"
      - --max-num-seqs
      - "1"
      - --kv-cache-dtype
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.tools-text.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.tools-text.yml
@@ -114,9 +114,9 @@ services:
      - --tensor-parallel-size
      - "1"
      - --max-model-len
-      - "75000"
+      - "${MAX_MODEL_LEN:-75000}"
      - --gpu-memory-utilization
-      - "0.97"
+      - "${GPU_MEMORY_UTILIZATION:-0.97}"
      - --max-num-seqs
      - "1"
      - --max-num-batched-tokens
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.yml
@@ -194,9 +194,9 @@ services:
      #   - long-vision.yml   → 198K + vision + sidecars (Cliff 1 closed; Cliff 2 caveat)
      #   - long-text.yml     → 218K text-only + sidecars (same Cliff 2 caveat)
      - --max-model-len
-      - "48000"
+      - "${MAX_MODEL_LEN:-48000}"
      - --gpu-memory-utilization
-      - "0.92"
+      - "${GPU_MEMORY_UTILIZATION:-0.92}"
      - --max-num-seqs
      - "1"
      - --max-num-batched-tokens