After the user flagged "are you validating all composer files" — ran a
full dry-run audit of all 9 composes via docker compose config, extracted
key flags (TP, max_len, mem_util, KV dtype, spec-decode), and found
several doc-vs-code mismatches inherited from the predecessor repos.
Compose header fixes:
- docker-compose.dual.yml — header described it as inheriting from
"single-card project's default", said "fp8 is plenty for 64K"
(stale — file actually does 262K). Updated to reflect: this IS the
dual-card default, fp8 is plenty for full 262K, plus a variant matrix
showing all 4 dual files with their actual TPS / streams / KV / vision.
- docker-compose.dual-turbo.yml — header claimed kv-cache-dtype was
`turboquant_3bit_nc` but the file actually ships `turboquant_k8v4`.
This mismatch was in the predecessor too; we kept the file (not the
header) since k8v4 is what was tested. Updated header to reflect
reality + noted the predecessor doc claim for archaeology.
- docker-compose.dual-dflash.yml — header said max_model_len "drops
from 262K to 16K" (stale dev-cycle comment); actual is 185K. Fixed.
Also added: KV cache is FP16 (DFlash + head_size=256 + non-causal
has no fp8/turbo Ampere backend), the bfloat16 dtype workaround for
vllm#40334, and clear positioning vs the noviz variant.
- docker-compose.dual-dflash-noviz.yml — minor: file path in "to run"
pointed at the old compose/ dir; updated to new layout path.
patches/README.md — was framed as dual-card-only ("we don't run
Genesis here") but the patches dir is now shared across single and
dual variants. Rewrote with a per-patch + per-variant matrix:
- patch_tolist_cudagraph.py: single-default + dual-turbo
- patch_pr40798_workspace.py: research artifact, no compose mounts
- genesis/: single-default + tools-text + dual-turbo
- Marlin pad fork (external /opt/ai/vllm-src/): all 4 dual composes
Added a Genesis env-opts table showing per-patch toggles and which
composes enable each.
scripts/setup.sh — final-output Next-steps block referenced the OLD
relative path `cd compose && docker compose up -d`, which would fail
in the new layout. Updated to:
cd models/<model>/vllm/compose && docker compose up -d
Plus added a clear note about the Marlin pad fork dependency for
dual-card composes (with the git-clone command users need to run
once before booting any dual-card variant).
YAML validation: `docker compose config` passes for all 9 composes
with MODEL_DIR set. Volume paths resolve, env vars substitute, no
syntax errors. Single-card default smoke-tested earlier (10/10
verify-full.sh checks pass); dual-card composes pass YAML validation
but require a 2× 3090 rig to actually boot — left for cross-rig users
to confirm.
noonghunna
0f33561b6b