alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions 38 Packages Projects Releases Wiki Activity

docs: add ik_llama engine page + QUANTIZATION primer; surface IQK quants

Browse Source 
- docs/engines/IK_LLAMA.md: ik_llama as the advanced-quant track (IQK imatrix,
  -khad/--merge-qkv/-ser, 262K single-card, MTP, cu13-server image)
- docs/QUANTIZATION.md: community quant primer (k-quants -> imatrix i-quants ->
  IQK; vLLM AWQ/GPTQ/AutoRound; KV-cache quants; engine x quant matrix)
- INFERENCE_ENGINES.md: correct stale "no official Docker image" claim + IQK pointer
- engines/README.md: ik_llama At-a-glance row + IQ4_KS quant row + per-engine link
- README.md: engine list + quant-names pointer

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
noonghunna
2026-05-21 13:56:58 +00:00
parent bc6e20bb8e
commit 554b85b8ea
5 changed files with 270 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
README.md
View File

@@ -73,7 +73,8 @@ bash scripts/update.sh
**New to local AI itself?** → [`docs/LOCAL_AI_PRIMER.md`](docs/LOCAL_AI_PRIMER.md) — plain-English: how hardware / engines / model sizes / quants fit together.
**New here?** → [`docs/GETTING_STARTED.md`](docs/GETTING_STARTED.md) — 5-minute clone-to-curl path.
**Already running, want to compare engines?** → [docs/engines/](docs/engines/)
**Picking an engine** (vLLM / llama.cpp / SGLang)? → [docs/INFERENCE_ENGINES.md](docs/INFERENCE_ENGINES.md)
**Picking an engine** (vLLM / llama.cpp / SGLang / ik_llama)? → [docs/INFERENCE_ENGINES.md](docs/INFERENCE_ENGINES.md)
**Confused by quant names** (Q4_K_M vs IQ4_KS vs AWQ)? → [docs/QUANTIZATION.md](docs/QUANTIZATION.md)
**Hardware questions** (4090, NVLink, power caps)? → [docs/HARDWARE.md](docs/HARDWARE.md)
**Don't know what TPS / KV / MTP mean?** → [docs/GLOSSARY.md](docs/GLOSSARY.md)

2
docs/INFERENCE_ENGINES.md
View File

@@ -292,7 +292,7 @@ Q1: Does the model fit your VRAM at desired quant?
### ik_llama.cpp
- **Smaller community than mainline llama.cpp** — bugs take longer to surface, fewer cross-rig data points.
- **No tagged releases** — rolls on main; no version pinning story for production users.
- **No official Docker image** — same friction as ktransformers (would need to containerize ourselves).
- ~~No official Docker image~~ — **corrected 2026-05-21:** an official image ships (`ghcr.io/ikawrakow/ik-llama-cpp:cu13-server`) and this stack now uses it for the **[advanced-quant track](engines/IK_LLAMA.md)** — fork-exclusive **IQK imatrix quants** (`IQ4_KS`), the best quality-per-bit in the GGUF world. See [QUANTIZATION.md](QUANTIZATION.md).
- **Diverging quant naming from mainline** — IQ_K series flags differ; cross-engine GGUF compatibility caveats.
- **Spec-decode coverage narrow** — MTP merged but no EAGLE3 / DFlash; lags mainline on those research paths.
- **Smaller maintainer surface** — primarily Iwan Kawrakow + a handful of contributors. Not the right pick for production where you need >1 person to debug a kernel issue.

124
docs/QUANTIZATION.md Normal file
View File

@@ -0,0 +1,124 @@
# Quantization — a field guide for the club-3090 community
Quantization is how a 27B model that would need ~54 GB at FP16 fits in 24 GB of VRAM. This page explains the **quant families** you'll see in the wild, what actually differs between them, and which ones this stack ships — including the **IQK imatrix quants** that exist only in [ik_llama.cpp](engines/IK_LLAMA.md).
> **The one idea to take away:** at the same *bits-per-weight*, not all quants are equal. The two levers that separate good from bad are (1) **non-linear levels** that match the weight distribution and (2) **calibration** (an "importance matrix") that spends bits where they matter. The best quants use both.
See also: [GLOSSARY.md](GLOSSARY.md) · [DTYPE_MATRIX.md](DTYPE_MATRIX.md) (KV/compute dtypes) · [engines/IK_LLAMA.md](engines/IK_LLAMA.md).
---
## 1. The vocabulary
- **bpw (bits per weight):** the headline number. FP16 = 16 bpw. A "4-bit" quant is ~4-4.5 bpw once you count the per-block scale/zero-point overhead. Lower bpw = smaller file = more context room, but more quality risk.
- **Block / group:** quants don't store one scale for the whole tensor — they chunk weights into blocks (e.g. 32 weights) and store a scale per block. Smaller blocks = finer = more accurate, but more overhead.
- **imatrix (importance matrix):** a calibration pass over real text that records *which weights matter most* for the model's outputs, so the quantizer protects those and compresses the rest harder. "i-quant" / "IQ" prefixes signal imatrix use.
- **Weight quant vs KV-cache quant:** two independent knobs. One shrinks the *model*; the other shrinks the *context* (see §5). You pick both.
---
## 2. The GGUF ladder (llama.cpp + ik_llama.cpp)
GGUF is the llama.cpp-family weight format. Roughly in order of quality-per-bit (worst → best at a given bpw):
| Family | Examples | Calibrated? | Where | Notes |
|---|---|---|---|---|
| **Legacy** | `Q4_0`, `Q4_1`, `Q5_0`, `Q8_0` | ❌ | mainline | Simple round-to-nearest, one scale/block. `Q8_0` is still a great near-lossless choice; the low-bit legacy ones are superseded. |
| **K-quants** | `Q3_K_M`, **`Q4_K_M`**, `Q5_K_M`, `Q6_K` | ❌ (data-free) | mainline | Mixed precision per tensor-type + 2-level block scales. The mainstream default. **`Q4_K_M` is what our shipped `llamacpp/mtp` runs.** Good, but data-free — no calibration. |
| **i-quants** | `IQ2_XXS` … `IQ3_M`, `IQ4_XS` | ✅ imatrix | mainline | Non-linear lattice codebooks + importance matrix. Clearly better quality-per-bit than k-quants, *especially below 4 bpw*. Slightly slower dequant than k-quants. |
| **IQK quants** ⭐ | **`IQ4_KS`**, `IQ5_KS`, `IQ4_K`, `IQ2_K` … | ✅ imatrix | **[ik_llama.cpp](engines/IK_LLAMA.md) only** | Refined grids + imatrix + **kernels co-designed for those grids**. Best quality-per-bit in the GGUF world *and* fast (the dequant path is hand-tuned). Fork-exclusive. |
**The progression that matters:** `Q4_K_M` (data-free) → `IQ4_XS` (imatrix, mainline) → `IQ4_KS` (imatrix + co-designed kernels, ik fork). Each step is better quality at similar bpw. Our shipped `llamacpp/mtp` is at the *first* rung (`Q4_K_M`); the [ik_llama track](engines/IK_LLAMA.md) is at the *last* (`IQ4_KS`).
---
## 3. What "imatrix" actually buys you
A data-free quant treats every weight as equally important and rounds uniformly. But in a trained model, a small fraction of weights carry most of the signal. An **importance matrix** is computed by running calibration text through the model and measuring how much each weight influences activations. The quantizer then:
- protects high-importance weights (more bits / closer grid points), and
- compresses low-importance weights harder.
Result: at 4 bpw, an imatrix quant loses noticeably less quality than a data-free one — and the gap *widens* as you go lower (at 2-3 bpw, imatrix is the difference between usable and broken). The cost is a one-time calibration step when *building* the quant; inference is the same speed.
> **Calibration corpus matters.** An imatrix calibrated on chat+code+tool-calling preserves those skills; one calibrated on Wikipedia may quietly drop tool-call formatting. This is exactly the kind of thing that shows up in our 8-pack quality tests (see [QUALITY_TEST.md](QUALITY_TEST.md)).
---
## 4. The vLLM / safetensors side (not GGUF)
vLLM and SGLang don't use GGUF — they load **safetensors** with these quant schemes:
| Quant | Bits | Calibrated? | Notes |
|---|---|---|---|
| **AutoRound** ⭐ | INT4 | ✅ (sign-gradient) | Intel's method; **what our shipped `vllm/dual` runs** (`qwen3.6-27b-autoround-int4`). Strong 4-bit quality. |
| **AWQ** | INT4 | ✅ (activation-aware) | Protects salient channels by inspecting activations. Widely available. |
| **GPTQ** | INT3/4/8 | ✅ (second-order) | Older, well-supported; AWQ/AutoRound usually edge it at 4-bit. |
| **FP8 (e4m3/e5m2)** | 8 | ❌ | Native on Hopper+; on Ampere it's emulated. We use **fp8 mostly for the KV cache**, not weights. |
| **bitsandbytes** | 4/8 | ❌ | Easy/on-the-fly; lower quality-per-bit than AWQ/AutoRound. |
These are conceptually the same idea as imatrix i-quants (calibrate, protect what matters) in a different ecosystem. There is **no GGUF↔safetensors interchange** — a quant is tied to its engine family.
---
## 5. KV-cache quantization (a separate knob)
Independent of the weight quant, you can quantize the **KV cache** — this is what sets your max context, not your model quality:
| KV type | Bits | Engine | Notes |
|---|---|---|---|
| `f16` | 16 | all | Lossless, biggest. Rarely needed. |
| `q8_0` | 8 | llama.cpp / ik | Near-lossless; good default when context is moderate. |
| `q4_0` | 4 | llama.cpp / ik | Halves KV vs q8_0 → enables **262K on one 3090** (ik IQ4_KS). Tiny quality cost. |
| `fp8_e5m2` | 8 | vLLM | Our `vllm/dual` default. |
| **TQ3 (TurboQuant)** | 3 | vLLM (Genesis) | 3-bit KV — beats fp8 on long-context memory; powers our `dual-turbo`. See [TQ3_MTP_GENESIS.md](TQ3_MTP_GENESIS.md) + [CLIFFS.md](CLIFFS.md). |
| `-khad` (modifier) | — | **ik only** | Hadamard transform on the K-cache → recovers accuracy lost to KV quantization, so you keep quality at q4_0/q8_0. |
---
## 6. Engine × quant support
| Quant family | vLLM | mainline llama.cpp | ik_llama.cpp | SGLang |
|---|---|---|---|---|
| K-quants (`Q4_K_M`…) | ❌ | ✅ | ✅ | ❌ |
| i-quants (`IQ4_XS`…) | ❌ | ✅ | ✅ | ❌ |
| **IQK (`IQ4_KS`…)** | ❌ | ❌ | ✅ **only** | ❌ |
| AutoRound / AWQ / GPTQ | ✅ | ❌ | ❌ | ✅ |
| FP8 weights | ✅ | ❌ | ❌ | ✅ |
---
## 7. Why doesn't every GGUF repo ship IQK?
If IQK is the best quality-per-bit, why are most community GGUFs still `Q4_K_M`?
1. **It's fork-locked.** IQK quants run *only* on ik_llama.cpp. A `Q4_K_M` runs on mainline llama.cpp, Ollama, LM Studio, LocalAI, Jan — everything. Quant authors optimize for reach.
2. **Kernel co-design.** IQK's quality comes partly from kernels written *for* its grids. Porting that to mainline isn't a small patch, and upstreaming has been slow.
3. **Inertia + tooling.** `Q4_K_M` is the well-trodden default; build pipelines, docs, and "recommended download" buttons all point at it.
So IQK is a deliberate "I'll run the fork to get the better quant" choice — which is exactly the niche the [ik_llama track](engines/IK_LLAMA.md) fills on this stack.
---
## 8. What this stack ships (and why)
| Path | Quant (weights) | KV | Rationale |
|---|---|---|---|
| `vllm/dual` | AutoRound INT4 | fp8_e5m2 | Production dual-card; deepest Qwen3-Next feature support |
| `vllm/dual-turbo` | AutoRound INT4 | **TQ3** | Max throughput + long context (3-bit KV) |
| `llamacpp/mtp` | **Q4_K_M** | q4_0 | Conservative, mainline image, cliff-immune single-card |
| `ik-llama/iq4ks-mtp` ⭐ | **IQ4_KS** (imatrix) | q4_0 + `-khad` | Advanced-quant track: best quality-per-bit + 262K single-card |
**Rule of thumb for your own rig:**
- Tightest VRAM / lowest bpw → reach for an **imatrix quant** (`IQ4_XS` mainline, or `IQ4_KS` on ik_llama), not a data-free `Q4_K_M`.
- Want maximum quality-per-bit and willing to run the fork → **ik_llama + IQK**.
- Multi-tenant / vision / tools at scale → **vLLM + AutoRound**.
- "Just works everywhere, no fork" → mainline **llama.cpp + Q4_K_M**.
---
## See also
- [engines/IK_LLAMA.md](engines/IK_LLAMA.md) — the engine that unlocks IQK
- [INFERENCE_ENGINES.md](INFERENCE_ENGINES.md) — engine comparison
- [DTYPE_MATRIX.md](DTYPE_MATRIX.md) — compute/KV dtype matrix
- [CLIFFS.md](CLIFFS.md) + [TQ3_MTP_GENESIS.md](TQ3_MTP_GENESIS.md) — KV-cache quant deep-dives
- [BENCHMARKS.md](../BENCHMARKS.md) — measured quality + TPS per quant/engine

139
docs/engines/IK_LLAMA.md Normal file
View File

@@ -0,0 +1,139 @@
# ik_llama.cpp — the advanced-quant engine
**Role on this stack:** the engine you reach for when you want **newer, higher-quality-per-bit quants** than mainline llama.cpp ships — specifically the **IQK imatrix family** (`IQ4_KS`, `IQ5_KS`, …) that exists *only* in this fork. It's a llama.cpp fork (ikawrakow), so it inherits llama.cpp's cliff-immune memory model and broad hardware support, then adds a co-designed quant + kernel stack on top.
> **In one line:** llama.cpp's robustness + fork-exclusive IQK quants + fused CUDA kernels → on our rig, **~62 narr / ~69 code TPS** single-3090 (decode ~64 / ~72) with **MTP**, clean to **262K context on one card** — ~+18-20% over the shipped `llamacpp/mtp` Q4_K_M path at equal-or-better quality.
For *what the quants actually are* and how IQK compares to k-quants / i-quants / AWQ, see **[../QUANTIZATION.md](../QUANTIZATION.md)**. For the cross-engine overview see **[../INFERENCE_ENGINES.md](../INFERENCE_ENGINES.md)**.
---
## TL;DR
- **Image:** `ghcr.io/ikawrakow/ik-llama-cpp:cu13-server` (cu13 = CUDA 13.x; matches our 13.2 host driver). Official, digest-pinnable. A `cu12` tag exists for older drivers.
- **Compose:** `models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp.yml` (text) + `iq4ks-mtp-vision.yml` (vision).
- **Model:** ubergarm `Qwen3.6-27B-MTP-IQ4_KS.gguf` (IQK imatrix quant, built-in MTP head).
- **Interface:** same `--jinja` + `--reasoning on|off` server contract as mainline llama.cpp — so the stack-wide thinking-off policy works unchanged.
```bash
MODEL_DIR=/your/models docker compose \
-f models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp.yml up -d
curl http://localhost:8020/v1/models
```
---
## Why pick ik_llama.cpp over mainline llama.cpp?
Both are cliff-immune (same ggml flat allocator — see [LLAMA_CPP.md](LLAMA_CPP.md) "Why llama.cpp doesn't hit the prefill cliffs"). ik_llama adds, on top:
1. **IQK imatrix quants (fork-exclusive).** `IQ4_KS` / `IQ5_KS` use refined non-linear grids + an importance matrix + **kernels co-designed for those grids**. Net: better quality-per-bit than mainline `Q4_K_M`, and *faster* than mainline i-quants because the dequant path is hand-tuned. `IQ4_KS` is ~15.1 GB vs `Q4_K_M`'s ~17 GB on Qwen3.6-27B — smaller weights leave room for **262K context** on a single 24 GB card.
2. **Fused CUDA kernels** — `--merge-qkv` (fused QKV projection), `--merge-up-gate-experts` (MoE, no-op on dense), and a fast IQK dequant path.
3. **`-khad` (Hadamard K-cache)** — a Hadamard transform on the K-cache that improves quantized-KV accuracy, so you keep more quality at `q4_0`/`q8_0` KV.
4. **MoE-on-consumer tooling** — `-ser` (smart expert reduction) + on-the-fly MLA tensors make big-MoE-over-VRAM (DeepSeek/Kimi-class) practical, an alternative to ktransformers.
5. **Qwen3.x MTP on `main`** — MTP merged on the fork's main branch (mainline still has it as open [PR #22673](https://github.com/ggml-org/llama.cpp/pull/22673)), so no PR-branch building.
The cost: it's a **fork** with no tagged releases (rolling `main`, smaller community) — a second engine image to track vs the clean `ggml-org/llama.cpp:server-cuda`. That's the trade for the IQK quants. Use mainline `llamacpp/mtp` for the conservative production path; use ik_llama when you want the cutting-edge quant.
---
## Pros
- **IQK imatrix quants** — fork-exclusive, best quality-per-bit available in the GGUF world.
- **Cliff-immune** (inherits llama.cpp's ggml allocator — no Cliff 1/2 GDN OOM).
- **262K context on a single 3090** with MTP (q4_0 KV ~5 GB; verified, verify-stress 7/7 incl. 91K needle).
- **Broad hardware** — CUDA (incl. CC 7.0 Volta), ROCm, Apple Metal, Intel, CPU (same as mainline).
- **MoE-over-VRAM** path (`-ser` + MLA) for models bigger than your cards.
- Same `--jinja` / `--reasoning` server contract as mainline → froggeric chat template + thinking-off both work (validated on this fork; see "Gotchas").
## Cons
- **A fork to track** — rolling `main`, no tags, smaller community; pin by image digest.
- **GGUF-only** — no safetensors / vLLM-class continuous batching (single-stream `-np 1` is the sweet spot here).
- **Tooling lag** — some mainline llama.cpp server flags differ or arrive later (e.g. `--alias`, `--no-mmproj-offload` are in source builds but not every published image tag).
- Not the production default — that's vLLM (dual, max TPS) and mainline llama.cpp (single, conservative).
---
## Quick recipe
### 1. Pull the engine image
```bash
docker pull ghcr.io/ikawrakow/ik-llama-cpp:cu13-server # cu12-server for CUDA 12 drivers
```
### 2. Get an IQK GGUF
```bash
hf download ubergarm/Qwen3.6-27B-GGUF Qwen3.6-27B-MTP-IQ4_KS.gguf \
--local-dir $MODEL_DIR/qwen3.6-27b-gguf/ubergarm-mtp-iq4ks
# Always SHA256-verify multi-GB GGUFs after download.
```
### 3. Launch (via our compose)
```bash
MODEL_DIR=$MODEL_DIR docker compose \
-f models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp.yml up -d
```
Defaults: q4_0 KV, 131K ctx, MTP n=2, froggeric v19 template, thinking-off. Overrides:
- **Max context (262K):** `UBATCH_SIZE=512 CTX_SIZE=262144` (q4_0 KV is the default; ~21.5 GB / 24).
- **Higher KV fidelity:** `KV_TYPE=q8_0` (caps ~131-200K — q8_0 KV @262K OOMs).
- **Reasoning on:** `REASONING=on` (pair with `MTP_DRAFT_N_MAX=5 DRAFT_P_MIN=0.5` — reasoning text drafts deeper).
---
## Tuning levers (ik-specific)
| Flag | What it does | Default here |
|---|---|---|
| `-ctk` / `-ctv` | KV cache quant (`q4_0` / `q8_0` / `f16`) — biggest VRAM lever | `q4_0` |
| `-khad` / `--k-cache-hadamard` | Hadamard transform on K-cache → better quantized-KV accuracy | on |
| `--merge-qkv` (`-mqkv`) | Fused QKV projection | on |
| `--merge-up-gate-experts` (`-muge`) | Fused MoE up/gate (no-op on dense models) | — |
| `-ser N,f` | Smart expert reduction (big-MoE) | — |
| `--multi-token-prediction` + `--draft-max` / `--draft-p-min` | Built-in MTP spec-decode | n=2 / p-min 0.0 |
| `-b` / `-ub` | Batch / micro-batch — `-ub 512` unlocks higher ctx at ~2-4% TPS | 4096 / 1024 |
---
## ik_llama-specific gotchas
- **froggeric chat template works here** (unlike mainline). The mainline `llama.cpp` note that froggeric "silently suppresses `--reasoning off`" is a *mainline* issue — on ik_llama, froggeric v19 + `--reasoning off` suppresses thinking cleanly **and** renders tool-calls correctly (validated 2026-05-21). So the ik composes ship froggeric v19; mainline composes stay native.
- **The published image lags source on a few flags.** `--alias` and `--no-mmproj-offload` exist in from-source builds but not the `cu13-server` tag we pull — don't copy a from-source config verbatim. Check `llama-server --help` in the container.
- **Single-stream is the regime.** `-np 1` — this isn't a continuous-batching server. For multi-tenant, use vLLM.
- **MoE flags are no-ops on dense models** (`--merge-up-gate-experts`, `-ser` do nothing on dense Qwen3.6-27B; they matter for the MoE catalog).
---
## Measured on this stack (Qwen3.6-27B, IQ4_KS + MTP, single 3090)
| Metric | Value | vs shipped `llamacpp/mtp` (Q4_K_M) |
|---|---|---|
| Narrative TPS (wall / decode) | ~62.8 / 64.4 | +18-20% |
| Code TPS (wall / decode) | ~68.7 / 71.9 | |
| Max context (1× 3090) | **262K** (q4_0 KV) | vs 131K |
| verify-stress | 7/7 (incl. 91K Cliff 2 needle) | parity |
| Quality 8-pack | ≈ parity with the vLLM autoround path | — |
| toolcall-15 | 60% — **no regression** (froggeric recovers the formatting) | — |
> Bench: canonical prompt, 3 warmup + measured runs, q4_0 KV / 131K / MTP n=2 / thinking-off, 370 W cap. See [../../BENCHMARKS.md](../../BENCHMARKS.md).
---
## When ik_llama.cpp is the right pick
- You want the **best quality-per-bit GGUF** (IQK imatrix) on a single card.
- You want **262K context on one 3090** with MTP decode speed.
- You're running a **big MoE that doesn't fit VRAM** and want `-ser`/MLA instead of ktransformers.
- You're **experimenting with newer quants** generally — this is the stack's advanced-quant track.
## When to use something else
- **Production multi-tenant / max dual-card TPS** → vLLM (`dual` / `dual-turbo`).
- **Conservative single-card, mainline image, no fork** → llama.cpp (`llamacpp/mtp`).
- **Apple Silicon** → either llama.cpp (Metal) or ik_llama (also Metal).
## See also
- [../QUANTIZATION.md](../QUANTIZATION.md) — what IQK / imatrix / k-quants actually are
- [LLAMA_CPP.md](LLAMA_CPP.md) — the mainline sibling (shared cliff-immunity)
- [../INFERENCE_ENGINES.md](../INFERENCE_ENGINES.md) — full cross-engine comparison
- [../../BENCHMARKS.md](../../BENCHMARKS.md) — measured TPS across engines/configs

5
docs/engines/README.md
View File

@@ -12,6 +12,7 @@ This repo's main path is **vLLM** because it has the deepest support for Qwen3-N
|---|---|---|---|---|---|---|---|
| **[vLLM](VLLM.md)** ⭐ | **Validated, production-grade** (this repo) | 50-53 narr / 66-70 code | 48K default · 75K IDE-agent · **198K vision · 214K text-only** | ✅ | ✅ | ✅ MTP n=3 | ✅ Full |
| **[llama.cpp](LLAMA_CPP.md)** | Works mainline + [Luce DFlash fork](https://github.com/Luce-Org/lucebox-hub) for spec-decode | 35-60 (varies by quant + KV type) | **262K** (Q4_K_M + q4_0 KV) | ✅ (via mmproj) | ⚠️ Limited (no auto-tool-choice in server) | ✅ DFlash N=5 in fork | ⚠️ Partial |
| **[ik_llama.cpp](IK_LLAMA.md)** ⭐ *advanced quants* | **Shipped — advanced-quant track** | ~62 narr / ~69 code (IQ4_KS+MTP) | **262K** (IQ4_KS + q4_0 KV) | ✅ (mmproj) | ✅ (template + parser) | ✅ MTP n=2 | ⚠️ Partial (llama.cpp-class) |
| **[SGLang](../../models/qwen3.6-27b/sglang/README.md)** | **Re-test pending** (May 2026). Historical block partially out-of-date — DFlash + MTP have landed natively on SGLang mainline; Marlin pad-sub-tile-n fix status unknown. See sglang README for re-test plan. | n/a (untested) | n/a | ✅ | ✅ | ✅ DFlash + MTP native upstream (untested here) | ✅ Full |
---
@@ -100,7 +101,7 @@ Full plan in [models/qwen3.6-27b/sglang/README.md](../../models/qwen3.6-27b/sgla
## Quant choice (orthogonal to engine choice)
The model itself comes in several quant formats. Engine-quant compatibility:
The model itself comes in several quant formats. Engine-quant compatibility (full primer: **[QUANTIZATION.md](../QUANTIZATION.md)**):
| Quant | Disk size | Engine fit | Notes |
|---|---|---|---|
@@ -110,6 +111,7 @@ The model itself comes in several quant formats. Engine-quant compatibility:
| GGUF Q4_K_M | ~16.8 GB | llama.cpp ✅ · vLLM ⚠️ experimental · SGLang ❌ | The default GGUF mid-range quant. Strong quality, broad ecosystem (Ollama, LM Studio, etc). |
| GGUF UD-Q3_K_XL ([Unsloth](https://huggingface.co/unsloth/Qwen3.6-27B-GGUF)) | **~14.5 GB** | llama.cpp ✅ | Smaller than 4-bit options. Quality cost is small on Qwen3.6 (quantization-friendly), buys substantial KV cache room. |
| GGUF Q3_K_M | ~13.6 GB | llama.cpp ✅ | More aggressive 3-bit; quality cost real but acceptable for many workloads. |
| **GGUF IQ4_KS (imatrix)** ⭐ ([ubergarm](https://huggingface.co/ubergarm/Qwen3.6-27B-GGUF)) | **~15.1 GB** | **[ik_llama.cpp](IK_LLAMA.md) only** · llama.cpp ❌ · vLLM ❌ | Best quality-per-bit GGUF (imatrix + kernels co-designed for IQK grids). Smaller than Q4_K_M → **262K single-card**. Fork-exclusive — see [QUANTIZATION.md](../QUANTIZATION.md). |
### AutoRound vs GPTQ vs AWQ (within vLLM)
@@ -136,6 +138,7 @@ If MTP isn't a priority for your workload, GPTQ or AWQ are equally valid.
- **[VLLM.md](VLLM.md)** — current setup (what this repo ships). Brief recap + tuning levers.
- **[LLAMA_CPP.md](LLAMA_CPP.md)** — quick GGUF recipe, vision via mmproj, Luce DFlash fork pointer for spec-decode, gotchas around server feature parity.
- **[SGLANG.md](SGLANG.md)** — current blocked state, what would unblock, when to revisit. TBD recipe placeholder until either Marlin pad lands upstream or DeltaNet rollback lands.
- **[IK_LLAMA.md](IK_LLAMA.md)** ⭐ — the advanced-quant engine: fork-exclusive IQK imatrix quants (`IQ4_KS`), 262K single-card, MTP, `-khad` / `--merge-qkv`. Pairs with [QUANTIZATION.md](../QUANTIZATION.md).
---