alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
48238e4cbbe85b2ef2e2ff8cc335a8de9136c23b
club-3090/models/qwen3.6-27b/vllm
History
noonghunna deb58a5f55 fp8w on Blackwell: auto-disable DeepGEMM + --force the arch-ab arm 
Two fixes from guybrush01's fp8-weights 5090 run (disc #571), which needed both
to boot the fp8w arm on consumer Blackwell.

1) VLLM_USE_DEEP_GEMM: vLLM's DeepGEMM fp8-GEMM path is built for Hopper (sm_90)
   + datacenter Blackwell (sm_100/103). Consumer Blackwell (5090 / PRO 6000 /
   GB10, sm_120/121) has no recipe and hard-fails "recipe not found" at boot.
   Add _deepgemm_env: for fp8-weights slugs, inject VLLM_USE_DEEP_GEMM=0 on the
   consumer SMs (sm_120/121 confirmed-broken; sm_89 Ada added proactively — it
   routes fp8 via Marlin/CUTLASS so disabling is a harmless no-op that pre-empts
   the same wall for 4090 owners). Hopper/datacenter untouched. dual/fp8/mtp.yml
   gains a pass-through env; both launchers whitelist the export.

2) arch-ab.sh fp8w arm: add --force. vllm/qwen-27b-dual-max is status=experimental
   so switch.sh gates it without --force.

test-launch-compat locks 5090/Ada-down / Hopper-keep / non-fp8-skip / user-pin.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
2026-07-05 07:21:34 +00:00
..
cache
…
compose
fp8w on Blackwell: auto-disable DeepGEMM + --force the arch-ab arm
2026-07-05 07:21:34 +00:00
diagnostics
serve: neutral primary model name (qwen3.6-27b / gemma-4-31b), keep -autoround alias
2026-07-02 11:28:31 +00:00
patches
chore(vllm): remove redundant nvlink-* dual composes (NVLink auto-detected) (#257)
2026-05-30 20:14:02 +05:00
README.md
…

README.md

Qwen3.6-27B on vLLM

The recommended path for this model. Full features, validated end-to-end via verify-full.sh.

What's here

  • compose/ — docker-compose files for single-card and dual-card configs
  • patches/ — engine-specific patches (CUDA graph capture fix, research artifacts, Marlin pad notes)

Pick a compose

# Single-card default — 48K + Genesis v7.14 + TurboQuant 3-bit + vision (recommended for ≥20K + tool agents)
cd compose && docker compose up -d

# Single-card frontier 145K — vision on, Cliff 1 closed, Cliff 2 60K closed via v7.69 + #35975
cd compose && docker compose -f single/autoround-int4/long-vision.yml up -d

# Single-card 180K text-only — Balanced MTP, 60K single-prompt PASS (default IDE-agent recommendation)
cd compose && docker compose -f single/autoround-int4/long-text.yml up -d

# Single-card 200K text-only — Max-context, MTP off, 60K single-prompt PASS, more KV pool, slower decode
cd compose && docker compose -f single/autoround-int4/long-text-no-mtp.yml up -d

# Single-card bounded-thinking — 180K + structured-CoT grammar in reasoning (~30× cheaper think on coding)
cd compose && docker compose -f single/autoround-int4/bounded-thinking.yml up -d

# Single-card IDE-agent / long-prompt — 75K + fp8 + no vision (Cline / Cursor / Copilot / RAG)
cd compose && docker compose -f single/autoround-int4/tools-text.yml up -d

# Dual-card default — 262K + fp8 + MTP + vision (best general dual-card)
cd compose && docker compose -f dual/autoround-int4/fp8-mtp.yml up -d

# Dual-card multi-tenant — 4 streams at 262K
cd compose && docker compose -f dual/autoround-int4/turbo.yml up -d

# Dual-card peak code TPS — DFlash N=5 (78/128 narr/code)
cd compose && docker compose -f dual/autoround-int4/dflash.yml up -d

# Dual-card Hermes agentic — Carnice fine-tune + BF16 MTP (72/80 narr/code, Hermes tool format)
cd compose && docker compose -f dual/carnice-bf16mtp/bf16-mtp.yml up -d

See the model README's compose table for the full matrix with TPS numbers and use cases.

Patches

vLLM doesn't ship cleanly out of the box for this model + this hardware combination. We mount three layers of patches at container boot:

  1. patches/patch_tolist_cudagraph.py — fixes a CUDA graph capture crash in TurboQuant continuation prefill (single-card stacks). Auto-applied by container entrypoint.
  2. patches/genesis/ (gitignored, fetched by setup.sh) — Sandermage's Genesis patch tree. Loads at boot via python3 -m vllm._genesis.patches.apply_all.
  3. patches/vllm-marlin-pad/ — our vllm#40361 patched files (vendored in-repo), volume-mounted for dual-card composes only. No host filesystem dependency. See patches/vllm-marlin-pad/README.md for provenance + sync procedure. Drops out as a dependency when our PR lands upstream.

Tuning

See /docs/engines/VLLM.md for engine-general tuning (mem-util, KV type, spec-decode config, power cap, Genesis env-opts).

For model-specific quirks (the prefill cliffs, why we pick AutoRound, etc.), see ../INTERNALS.md.