Set dual-nvlink-dflash-noviz --max-model-len default to 188000
Extensive testing (9 runs across 7 different lengths) found the passing threshold between 188000 and 189000. We are calling 188000 as the practical default here: 189000 was flaky (1 pass out of 3 attempts, including a failure on a freshly rebooted system), and going lower would lose the point of the noviz variant entirely — dropping to 185000 (the dual-nvlink-dflash with-vision default) still fails all stress checks, so the noviz gain is only about 3000 tokens of headroom. Any further reduction just to add buffer would erase that advantage. At 188000: all 7 verify-stress checks passed including Cliff 2 needle recall at 91K tokens, 25K tool prefill, and continuous soak with 0 errors. Soak baseline VRAM: 47944 MiB. TODO: find the actual context length boundary for dual-nvlink-dflash (with vision) to determine the real delta between the two variants' passing thresholds and quantify exactly what noviz buys in terms of extra context.
This commit is contained in:
@@ -120,7 +120,7 @@ services:
|
||||
# makes vLLM's custom kernel a win. dual-dflash-noviz.yml disables it
|
||||
# because PCIe P2P bandwidth makes the NCCL fallback faster there.
|
||||
- --max-model-len
|
||||
- "${MAX_MODEL_LEN:-200000}"
|
||||
- "${MAX_MODEL_LEN:-188000}"
|
||||
- --gpu-memory-utilization
|
||||
- "${GPU_MEMORY_UTILIZATION:-0.95}"
|
||||
- --max-num-seqs
|
||||
|
||||
Reference in New Issue
Block a user