Set dual-nvlink-dflash-noviz --max-model-len default to 188000

Extensive testing (9 runs across 7 different lengths) found the passing
threshold between 188000 and 189000. We are calling 188000 as the
practical default here: 189000 was flaky (1 pass out of 3 attempts,
including a failure on a freshly rebooted system), and going lower
would lose the point of the noviz variant entirely — dropping to
185000 (the dual-nvlink-dflash with-vision default) still fails all
stress checks, so the noviz gain is only about 3000 tokens of
headroom. Any further reduction just to add buffer would erase that
advantage.

At 188000: all 7 verify-stress checks passed including Cliff 2 needle
recall at 91K tokens, 25K tool prefill, and continuous soak with 0
errors. Soak baseline VRAM: 47944 MiB.

TODO: find the actual context length boundary for dual-nvlink-dflash
(with vision) to determine the real delta between the two variants'
passing thresholds and quantify exactly what noviz buys in terms of
extra context.
This commit is contained in:
Dan
2026-05-07 02:22:28 -04:00
committed by noonghunna
parent 63ab224c57
commit 89c686288e

View File

@@ -120,7 +120,7 @@ services:
# makes vLLM's custom kernel a win. dual-dflash-noviz.yml disables it
# because PCIe P2P bandwidth makes the NCCL fallback faster there.
- --max-model-len
- "${MAX_MODEL_LEN:-200000}"
- "${MAX_MODEL_LEN:-188000}"
- --gpu-memory-utilization
- "${GPU_MEMORY_UTILIZATION:-0.95}"
- --max-num-seqs