# =========================================================================== # Profile (at-a-glance): # Model: Qwen3.6-27B (Unsloth MTP-enabled Q4_K_M GGUF) # Engine: llama.cpp (local MTP-enabled build, build 9235) # Topology: Single 3090 (TP=1) # Drafter: MTP n=2 (--spec-type draft-mtp, sweet spot — see BENCHMARKS.md) # KV: q4_0 K + q4_0 V (densest mainline, Ampere-fast) # Vision: no (mmproj NOT mounted — for vision, see mtp-vision.yml) # Template: native (GGUF-embedded) — froggeric A/B'd here, regressed 8-pack 102→95 # Max ctx: 200000 default (@ -ub 512, fills ~183K w/ ~1.1 GB margin); 262144 boots but # walls ~125K (FA scratch at fill — see CTX_SIZE note); 131072 = faster prefill @ -ub 1024 # Genesis: N/A — llama.cpp engine; Genesis is vLLM/Qwen3-Next-specific # Status: ✅ Production # Engine-profile: llama-cpp-local # Best for: IDE agents, opencode, Hermes, long-multi-turn agentic — the # speed + ctx workhorse. ~51 narr / ~60 code TPS, 7/7 verify-stress # (incl. 60K + 91K needle), 102/150 quality (68%) on the 8-pack. # --------------------------------------------------------------------------- # Qwen3.6-27B on llama.cpp — single 3090, MTP, 200K ctx, no vision. # # This is one of TWO named single-card profiles for this model (collapsed from # three on 2026-05-22 — the old vanilla Q3_K_XL `docker-compose.yml` was retired # once Q4_K_M was shown to cover long context too; see the CTX_SIZE note below): # - mtp.yml → MTP n=2 + no vision (THIS file). `llamacpp/default` # is now an ALIAS for this profile. # - mtp-vision.yml → MTP n=2 + vision (multimodal) # # Why no vision here: mmproj F16 costs ~0.8 GB. Without it the MTP-safe ctx # ceiling jumps from ~49K to 131K (sweep-verified 2026-05-19 on build 9235). # If you don't need image input, this is the better MTP profile. # # Template: native (GGUF-embedded) + `--reasoning off` + `--reasoning-format # deepseek` — the stack-wide thinking-off lever. We A/B'd froggeric v19 here # (it loads + honours --reasoning off fine on b9246 — the old "froggeric # suppresses --reasoning off" lore is stale), but it REGRESSED the 8-pack # 102 → 95 on mainline llama.cpp. froggeric helps the vLLM/Qwen3-Next path, # not this engine — so native is the default. Closes opencode hang (#97). # # Cliff-survival: -ub 1024 (lowered from the older 2048 default). The 25K # tool-prefill check in verify-stress fails at -ub 2048 on tight single-card # headroom; -ub 1024 halves the per-pass activation peak and the boundary # matrix goes 5/7 → 7/7. The "Cliff 2 single-prompt at 50–60K is # architectural" narrative is partially superseded by this config: at # -ub 1024 + 131K + MTP n=2 + thinking-off, verify-stress recalls needles # cleanly at 58K and 91K. # # VRAM budget on 24 GB (Q4_K_M): # weights (Q4_K_M): ~17.0 GB # KV at 131K (q4_0 K+V): ~5.0 GB # MTP draft head + overhead: ~0.5 GB # total: ~22.5 GB # headroom: ~1.6 GB for prompt + activation peaks # # Quick start: # 1. Get the MTP-enabled GGUF: # hf download unsloth/Qwen3.6-27B-MTP-GGUF Qwen3.6-27B-Q4_K_M.gguf \ # --local-dir $MODEL_DIR/qwen3.6-27b-gguf/unsloth-mtp-q4km # 2. From this directory: # MODEL_DIR=/your/models/dir docker compose -f mtp.yml up -d # 3. curl http://localhost:8020/v1/models → should list the model. # # Override defaults via .env or shell: # MODEL_DIR host dir to mount as /models (default: ../../../../../../models-cache) # GGUF_FILE path under /models (default: qwen3.6-27b-gguf/unsloth-mtp-q4km/Qwen3.6-27B-Q4_K_M.gguf) # CTX_SIZE KV pool size (default: 200000 @ -ub 512 — fills ~183K with # ~1.1 GB free, the max-safe single-card value. 262144 boots ~23 GB, # MTP active, passes verify-stress(91K) + soak — but only *fills* to # ~125K before OOMing on the FA scratch at high fill: a false ceiling. # See docs/CLIFFS.md (2026-05-23). Faster prefill: CTX_SIZE=131072 UBATCH_SIZE=1024.) # BATCH_SIZE llama.cpp -b (default: 4096 — on mainline -b does NOT drive VRAM) # UBATCH_SIZE llama.cpp -ub (default: 512 — funds the 200K default; raise to 1024 at 131K) # KV_TYPE K and V quant type (default: q4_0) # NP parallel slots (default: 1 — see ⚠ below) # MTP_DRAFT_N_MAX MTP draft tokens (default: 2 — sweet spot per BENCHMARKS.md) # REASONING thinking gate (default: off — stack-wide policy) # REASONING_FORMAT reasoning routing (default: deepseek — hygiene) # PORT host port (default: 8020) # CUDA_VISIBLE_DEVICES which GPU to use (default: 0) services: llama-cpp-qwen36-27b-mtp: # Pinned to b9246 (validated 2026-05-20). The rolling `:server-cuda` tag # regressed at b9282 (broken lib packaging: `libllama-common.so.0: cannot # open shared object file` → crash loop) — see #187. Bump via env when a # newer build is validated: LLAMACPP_IMAGE=ghcr.io/ggml-org/llama.cpp:server-cuda-bXXXX image: ${LLAMACPP_IMAGE:-ghcr.io/ggml-org/llama.cpp:server-cuda-b9967} container_name: "${ESTATE_CONTAINER:-llama-cpp-qwen36-27b}" restart: unless-stopped ports: - "${ESTATE_PORT:-${PORT:-8020}}:8080" volumes: - "${MODEL_DIR:-../../../../../../models-cache}:/models:ro" # ⚠ -np 1 is intentional on a single 24 GB card — do NOT raise it to # "parallelize." One GPU is compute-bound: extra slots divide its # throughput, they don't multiply it. At -np 4 each slot fell to # ~14 tok/s here — slow enough to trip agentic clients' per-request # timeouts (aider ran 1/30) — and -np>1 also auto-disables MTP and # can OOM the spec-context buffer. On a higher-throughput card (e.g. # 5090) or multi-GPU the trade may flip — re-validate before raising. command: >- --host 0.0.0.0 --port 8080 -m /models/${GGUF_FILE:-qwen3.6-27b-gguf/unsloth-mtp-q4km/Qwen3.6-27B-Q4_K_M.gguf} -c ${CTX_SIZE:-200000} -b ${BATCH_SIZE:-4096} -ub ${UBATCH_SIZE:-512} -ngl 99 -fa on --cache-type-k ${KV_TYPE:-q4_0} --cache-type-v ${KV_TYPE:-q4_0} -np ${NP:-1} --spec-type draft-mtp --spec-draft-n-max ${MTP_DRAFT_N_MAX:-2} --jinja --reasoning ${REASONING:-off} --reasoning-format ${REASONING_FORMAT:-deepseek} --temp ${TEMP:-${TEMPERATURE:-0.6}} --top-p ${TOP_P:-0.95} --top-k ${TOP_K:-20} --min-p ${MIN_P:-0.0} --repeat-penalty ${REPEAT_PENALTY:-1.0} deploy: resources: reservations: devices: - driver: nvidia device_ids: ["${ESTATE_GPUS:-${CUDA_VISIBLE_DEVICES:-0}}"] capabilities: [compute, utility]