noonghunna
|
3fa33332ce
|
Initial commit — club-3090: model-agnostic LLM serving recipes for RTX 3090
Consolidates and supersedes:
- noonghunna/qwen36-27b-single-3090
- noonghunna/qwen36-dual-3090
The two predecessor repos partitioned by card count (1× vs 2×). This
repo partitions by engine instead, which matches how users actually
decide ("vLLM or llama.cpp?" before "1 card or 2"). Card count becomes
a config variant within each engine.
Structure (model-agnostic from day 1):
docs/ cross-model engine + hardware docs
engines/ vLLM / llama.cpp / SGLang comparison + per-engine deep dives
HARDWARE.md Ampere SM 8.6+, NVLink, power, VRAM ceilings
GLOSSARY.md plain-language definitions
img/ illustrations (vram-budget.svg)
ARCHITECTURE.md how this stack thinks about LLM serving on 24 GB
models/<model-name>/ everything specific to a model
qwen3.6-27b/ today's only model
README.md / INTERNALS.md / USE_CASES.md / CHANGELOG.md
vllm/ vLLM-specific configs for this model
compose/ docker-compose files (single + dual variants)
patches/ tolist_cudagraph + Marlin pad notes
llama-cpp/ llama.cpp recipes for this model
recipes/ shell scripts (single-card default + 262K max-ctx)
sglang/ SGLang status (currently blocked)
scripts/ shared, model-aware
setup.sh bash setup.sh <model> → downloads + verifies
verify.sh / verify-full.sh smoke + functional tests
bench.sh canonical TPS bench
vLLM compose variants (all under models/qwen3.6-27b/vllm/compose/):
Single-card:
docker-compose.yml ⭐ DEFAULT — TQ3 + Genesis P65, 48K, 51/68 TPS
docker-compose.fast-chat.yml fp8 + 20K, 55/70 TPS — fastest at small ctx
docker-compose.tools-text.yml fp8 + 75K, 53/70 TPS — best for long single prompts
docker-compose.no-genesis-mtp.yml control variant
docker-compose.minimal.yml no spec-decode
Dual-card:
docker-compose.dual.yml ⭐ fp8 + 262K + MTP + vision, 71/89 TPS
docker-compose.dual-turbo.yml TQ3 + Genesis v7.14 — 4-stream concurrency
docker-compose.dual-dflash.yml DFlash N=5 + 185K + vision — 78/128 TPS
docker-compose.dual-dflash-noviz.yml DFlash + 200K text-only
llama.cpp recipes (under models/qwen3.6-27b/llama-cpp/recipes/):
single-card-default.sh Q4_K_M + 65K
single-card-max-ctx.sh Q4_K_M + q4_0 KV at full 262K — the standout recipe
Old repos remain readable for issue history + external links (Medium,
Reddit, Twitter, Sandermage's PR threads). New issues should be filed
here.
Credits in README. Apache 2.0.
|
2026-04-28 10:24:14 +00:00 |
|