alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(hardware): newer-driver 3090 caps long-text.yml at MAX_MODEL_LEN=105000 (#149)

Browse Source 
@sethbrasile's controlled 9-run matrix on a headless 3090 + driver
595.71.05 / CUDA 13.2 shows the same env-override pattern as the 4090
display-overhead case: the newer driver's vLLM activation-profile
reserve measured ~2.87 GiB vs ~1.5 GiB on the bare-metal reference rig,
shrinking the KV pool and capping long-text.yml at MAX_MODEL_LEN=105000
(vs 180K default). Added as the 3090 sibling anchor next to the
@laurimyllari 4090 -> 90000 data point so newer-driver 3090 users start
from the right number. Tuning-data contribution, not a bug
(corroborates the known Cliff 2a-under-v7.72.2 / genesis#22 picture).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
noonghunna
2026-05-17 21:14:58 +00:00
parent 820eb3845c
commit b0774f953b
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/HARDWARE.md
View File

@@ -53,7 +53,7 @@ Use `--force` only when you are intentionally testing an unsupported combo. Exam
On 20 GB cards (modded 3080) the cudagraph-profiling overhead is a meaningful slice of available VRAM. Drop `--gpu-memory-utilization` to **0.82** (vs shipped 0.95 for 24 GB). vLLM nightly's `gpu_worker.py` reports the equivalent effective KV size in the boot log; tune to keep activation headroom for the ~15K tool-prefill peak (verify-full check 8). Credit: [@troymroberts](https://github.com/troymroberts).
**4090s with attached display — env-override the compose defaults.** Some 4090 rigs land at ~23.5 GB usable VRAM with X server + driver overhead, vs the headless 3090s the composes are calibrated for. Boot may fail with `No available memory for the cache blocks` at default `max-model-len`. Cross-rig data: @laurimyllari's 4090 single-card on `long-text.yml` needed `MAX_MODEL_LEN=90000` (down from 180K default) to fit cleanly ([disc #62](../../../noonghunna/club-3090/discussions/62) / [issue #71](../../../noonghunna/club-3090/issues/71)). Pattern:
**4090s with attached display — env-override the compose defaults.** Some 4090 rigs land at ~23.5 GB usable VRAM with X server + driver overhead, vs the headless 3090s the composes are calibrated for. Boot may fail with `No available memory for the cache blocks` at default `max-model-len`. Cross-rig data: @laurimyllari's 4090 single-card on `long-text.yml` needed `MAX_MODEL_LEN=90000` (down from 180K default) to fit cleanly ([disc #62](../../../noonghunna/club-3090/discussions/62) / [issue #71](../../../noonghunna/club-3090/issues/71)). **Newer driver shrinks the budget the same way even on a headless 3090:** @sethbrasile's controlled 9-run matrix on a headless 3090 with driver 595.71.05 / CUDA 13.2 capped `long-text.yml` at `MAX_MODEL_LEN=105000` — the newer driver's activation-profile reserve measured ~2.87 GiB vs ~1.5 GiB on the bare-metal reference rig, shrinking the KV pool by the difference ([issue #149](../../../noonghunna/club-3090/issues/149)). On a newer-driver 3090, start at `MAX_MODEL_LEN=105000` rather than the 180K default. Pattern:
```bash
MAX_MODEL_LEN=90000 bash scripts/switch.sh vllm/long-text