diff --git a/README.md b/README.md index b3cedf41..67663744 100644 --- a/README.md +++ b/README.md @@ -73,7 +73,8 @@ bash scripts/update.sh **New to local AI itself?** → [`docs/LOCAL_AI_PRIMER.md`](docs/LOCAL_AI_PRIMER.md) — plain-English: how hardware / engines / model sizes / quants fit together. **New here?** → [`docs/GETTING_STARTED.md`](docs/GETTING_STARTED.md) — 5-minute clone-to-curl path. **Already running, want to compare engines?** → [docs/engines/](docs/engines/) -**Picking an engine** (vLLM / llama.cpp / SGLang)? → [docs/INFERENCE_ENGINES.md](docs/INFERENCE_ENGINES.md) +**Picking an engine** (vLLM / llama.cpp / SGLang / ik_llama)? → [docs/INFERENCE_ENGINES.md](docs/INFERENCE_ENGINES.md) +**Confused by quant names** (Q4_K_M vs IQ4_KS vs AWQ)? → [docs/QUANTIZATION.md](docs/QUANTIZATION.md) **Hardware questions** (4090, NVLink, power caps)? → [docs/HARDWARE.md](docs/HARDWARE.md) **Don't know what TPS / KV / MTP mean?** → [docs/GLOSSARY.md](docs/GLOSSARY.md) diff --git a/docs/INFERENCE_ENGINES.md b/docs/INFERENCE_ENGINES.md index 3c7d1e99..82c6bf3e 100644 --- a/docs/INFERENCE_ENGINES.md +++ b/docs/INFERENCE_ENGINES.md @@ -292,7 +292,7 @@ Q1: Does the model fit your VRAM at desired quant? ### ik_llama.cpp - **Smaller community than mainline llama.cpp** — bugs take longer to surface, fewer cross-rig data points. - **No tagged releases** — rolls on main; no version pinning story for production users. -- **No official Docker image** — same friction as ktransformers (would need to containerize ourselves). +- ~~No official Docker image~~ — **corrected 2026-05-21:** an official image ships (`ghcr.io/ikawrakow/ik-llama-cpp:cu13-server`) and this stack now uses it for the **[advanced-quant track](engines/IK_LLAMA.md)** — fork-exclusive **IQK imatrix quants** (`IQ4_KS`), the best quality-per-bit in the GGUF world. See [QUANTIZATION.md](QUANTIZATION.md). - **Diverging quant naming from mainline** — IQ_K series flags differ; cross-engine GGUF compatibility caveats. - **Spec-decode coverage narrow** — MTP merged but no EAGLE3 / DFlash; lags mainline on those research paths. - **Smaller maintainer surface** — primarily Iwan Kawrakow + a handful of contributors. Not the right pick for production where you need >1 person to debug a kernel issue. diff --git a/docs/QUANTIZATION.md b/docs/QUANTIZATION.md new file mode 100644 index 00000000..dcb2a3f7 --- /dev/null +++ b/docs/QUANTIZATION.md @@ -0,0 +1,124 @@ +# Quantization — a field guide for the club-3090 community + +Quantization is how a 27B model that would need ~54 GB at FP16 fits in 24 GB of VRAM. This page explains the **quant families** you'll see in the wild, what actually differs between them, and which ones this stack ships — including the **IQK imatrix quants** that exist only in [ik_llama.cpp](engines/IK_LLAMA.md). + +> **The one idea to take away:** at the same *bits-per-weight*, not all quants are equal. The two levers that separate good from bad are (1) **non-linear levels** that match the weight distribution and (2) **calibration** (an "importance matrix") that spends bits where they matter. The best quants use both. + +See also: [GLOSSARY.md](GLOSSARY.md) · [DTYPE_MATRIX.md](DTYPE_MATRIX.md) (KV/compute dtypes) · [engines/IK_LLAMA.md](engines/IK_LLAMA.md). + +--- + +## 1. The vocabulary + +- **bpw (bits per weight):** the headline number. FP16 = 16 bpw. A "4-bit" quant is ~4-4.5 bpw once you count the per-block scale/zero-point overhead. Lower bpw = smaller file = more context room, but more quality risk. +- **Block / group:** quants don't store one scale for the whole tensor — they chunk weights into blocks (e.g. 32 weights) and store a scale per block. Smaller blocks = finer = more accurate, but more overhead. +- **imatrix (importance matrix):** a calibration pass over real text that records *which weights matter most* for the model's outputs, so the quantizer protects those and compresses the rest harder. "i-quant" / "IQ" prefixes signal imatrix use. +- **Weight quant vs KV-cache quant:** two independent knobs. One shrinks the *model*; the other shrinks the *context* (see §5). You pick both. + +--- + +## 2. The GGUF ladder (llama.cpp + ik_llama.cpp) + +GGUF is the llama.cpp-family weight format. Roughly in order of quality-per-bit (worst → best at a given bpw): + +| Family | Examples | Calibrated? | Where | Notes | +|---|---|---|---|---| +| **Legacy** | `Q4_0`, `Q4_1`, `Q5_0`, `Q8_0` | ❌ | mainline | Simple round-to-nearest, one scale/block. `Q8_0` is still a great near-lossless choice; the low-bit legacy ones are superseded. | +| **K-quants** | `Q3_K_M`, **`Q4_K_M`**, `Q5_K_M`, `Q6_K` | ❌ (data-free) | mainline | Mixed precision per tensor-type + 2-level block scales. The mainstream default. **`Q4_K_M` is what our shipped `llamacpp/mtp` runs.** Good, but data-free — no calibration. | +| **i-quants** | `IQ2_XXS` … `IQ3_M`, `IQ4_XS` | ✅ imatrix | mainline | Non-linear lattice codebooks + importance matrix. Clearly better quality-per-bit than k-quants, *especially below 4 bpw*. Slightly slower dequant than k-quants. | +| **IQK quants** ⭐ | **`IQ4_KS`**, `IQ5_KS`, `IQ4_K`, `IQ2_K` … | ✅ imatrix | **[ik_llama.cpp](engines/IK_LLAMA.md) only** | Refined grids + imatrix + **kernels co-designed for those grids**. Best quality-per-bit in the GGUF world *and* fast (the dequant path is hand-tuned). Fork-exclusive. | + +**The progression that matters:** `Q4_K_M` (data-free) → `IQ4_XS` (imatrix, mainline) → `IQ4_KS` (imatrix + co-designed kernels, ik fork). Each step is better quality at similar bpw. Our shipped `llamacpp/mtp` is at the *first* rung (`Q4_K_M`); the [ik_llama track](engines/IK_LLAMA.md) is at the *last* (`IQ4_KS`). + +--- + +## 3. What "imatrix" actually buys you + +A data-free quant treats every weight as equally important and rounds uniformly. But in a trained model, a small fraction of weights carry most of the signal. An **importance matrix** is computed by running calibration text through the model and measuring how much each weight influences activations. The quantizer then: +- protects high-importance weights (more bits / closer grid points), and +- compresses low-importance weights harder. + +Result: at 4 bpw, an imatrix quant loses noticeably less quality than a data-free one — and the gap *widens* as you go lower (at 2-3 bpw, imatrix is the difference between usable and broken). The cost is a one-time calibration step when *building* the quant; inference is the same speed. + +> **Calibration corpus matters.** An imatrix calibrated on chat+code+tool-calling preserves those skills; one calibrated on Wikipedia may quietly drop tool-call formatting. This is exactly the kind of thing that shows up in our 8-pack quality tests (see [QUALITY_TEST.md](QUALITY_TEST.md)). + +--- + +## 4. The vLLM / safetensors side (not GGUF) + +vLLM and SGLang don't use GGUF — they load **safetensors** with these quant schemes: + +| Quant | Bits | Calibrated? | Notes | +|---|---|---|---| +| **AutoRound** ⭐ | INT4 | ✅ (sign-gradient) | Intel's method; **what our shipped `vllm/dual` runs** (`qwen3.6-27b-autoround-int4`). Strong 4-bit quality. | +| **AWQ** | INT4 | ✅ (activation-aware) | Protects salient channels by inspecting activations. Widely available. | +| **GPTQ** | INT3/4/8 | ✅ (second-order) | Older, well-supported; AWQ/AutoRound usually edge it at 4-bit. | +| **FP8 (e4m3/e5m2)** | 8 | ❌ | Native on Hopper+; on Ampere it's emulated. We use **fp8 mostly for the KV cache**, not weights. | +| **bitsandbytes** | 4/8 | ❌ | Easy/on-the-fly; lower quality-per-bit than AWQ/AutoRound. | + +These are conceptually the same idea as imatrix i-quants (calibrate, protect what matters) in a different ecosystem. There is **no GGUF↔safetensors interchange** — a quant is tied to its engine family. + +--- + +## 5. KV-cache quantization (a separate knob) + +Independent of the weight quant, you can quantize the **KV cache** — this is what sets your max context, not your model quality: + +| KV type | Bits | Engine | Notes | +|---|---|---|---| +| `f16` | 16 | all | Lossless, biggest. Rarely needed. | +| `q8_0` | 8 | llama.cpp / ik | Near-lossless; good default when context is moderate. | +| `q4_0` | 4 | llama.cpp / ik | Halves KV vs q8_0 → enables **262K on one 3090** (ik IQ4_KS). Tiny quality cost. | +| `fp8_e5m2` | 8 | vLLM | Our `vllm/dual` default. | +| **TQ3 (TurboQuant)** | 3 | vLLM (Genesis) | 3-bit KV — beats fp8 on long-context memory; powers our `dual-turbo`. See [TQ3_MTP_GENESIS.md](TQ3_MTP_GENESIS.md) + [CLIFFS.md](CLIFFS.md). | +| `-khad` (modifier) | — | **ik only** | Hadamard transform on the K-cache → recovers accuracy lost to KV quantization, so you keep quality at q4_0/q8_0. | + +--- + +## 6. Engine × quant support + +| Quant family | vLLM | mainline llama.cpp | ik_llama.cpp | SGLang | +|---|---|---|---|---| +| K-quants (`Q4_K_M`…) | ❌ | ✅ | ✅ | ❌ | +| i-quants (`IQ4_XS`…) | ❌ | ✅ | ✅ | ❌ | +| **IQK (`IQ4_KS`…)** | ❌ | ❌ | ✅ **only** | ❌ | +| AutoRound / AWQ / GPTQ | ✅ | ❌ | ❌ | ✅ | +| FP8 weights | ✅ | ❌ | ❌ | ✅ | + +--- + +## 7. Why doesn't every GGUF repo ship IQK? + +If IQK is the best quality-per-bit, why are most community GGUFs still `Q4_K_M`? + +1. **It's fork-locked.** IQK quants run *only* on ik_llama.cpp. A `Q4_K_M` runs on mainline llama.cpp, Ollama, LM Studio, LocalAI, Jan — everything. Quant authors optimize for reach. +2. **Kernel co-design.** IQK's quality comes partly from kernels written *for* its grids. Porting that to mainline isn't a small patch, and upstreaming has been slow. +3. **Inertia + tooling.** `Q4_K_M` is the well-trodden default; build pipelines, docs, and "recommended download" buttons all point at it. + +So IQK is a deliberate "I'll run the fork to get the better quant" choice — which is exactly the niche the [ik_llama track](engines/IK_LLAMA.md) fills on this stack. + +--- + +## 8. What this stack ships (and why) + +| Path | Quant (weights) | KV | Rationale | +|---|---|---|---| +| `vllm/dual` | AutoRound INT4 | fp8_e5m2 | Production dual-card; deepest Qwen3-Next feature support | +| `vllm/dual-turbo` | AutoRound INT4 | **TQ3** | Max throughput + long context (3-bit KV) | +| `llamacpp/mtp` | **Q4_K_M** | q4_0 | Conservative, mainline image, cliff-immune single-card | +| `ik-llama/iq4ks-mtp` ⭐ | **IQ4_KS** (imatrix) | q4_0 + `-khad` | Advanced-quant track: best quality-per-bit + 262K single-card | + +**Rule of thumb for your own rig:** +- Tightest VRAM / lowest bpw → reach for an **imatrix quant** (`IQ4_XS` mainline, or `IQ4_KS` on ik_llama), not a data-free `Q4_K_M`. +- Want maximum quality-per-bit and willing to run the fork → **ik_llama + IQK**. +- Multi-tenant / vision / tools at scale → **vLLM + AutoRound**. +- "Just works everywhere, no fork" → mainline **llama.cpp + Q4_K_M**. + +--- + +## See also +- [engines/IK_LLAMA.md](engines/IK_LLAMA.md) — the engine that unlocks IQK +- [INFERENCE_ENGINES.md](INFERENCE_ENGINES.md) — engine comparison +- [DTYPE_MATRIX.md](DTYPE_MATRIX.md) — compute/KV dtype matrix +- [CLIFFS.md](CLIFFS.md) + [TQ3_MTP_GENESIS.md](TQ3_MTP_GENESIS.md) — KV-cache quant deep-dives +- [BENCHMARKS.md](../BENCHMARKS.md) — measured quality + TPS per quant/engine diff --git a/docs/engines/IK_LLAMA.md b/docs/engines/IK_LLAMA.md new file mode 100644 index 00000000..4ba67dfe --- /dev/null +++ b/docs/engines/IK_LLAMA.md @@ -0,0 +1,139 @@ +# ik_llama.cpp — the advanced-quant engine + +**Role on this stack:** the engine you reach for when you want **newer, higher-quality-per-bit quants** than mainline llama.cpp ships — specifically the **IQK imatrix family** (`IQ4_KS`, `IQ5_KS`, …) that exists *only* in this fork. It's a llama.cpp fork (ikawrakow), so it inherits llama.cpp's cliff-immune memory model and broad hardware support, then adds a co-designed quant + kernel stack on top. + +> **In one line:** llama.cpp's robustness + fork-exclusive IQK quants + fused CUDA kernels → on our rig, **~62 narr / ~69 code TPS** single-3090 (decode ~64 / ~72) with **MTP**, clean to **262K context on one card** — ~+18-20% over the shipped `llamacpp/mtp` Q4_K_M path at equal-or-better quality. + +For *what the quants actually are* and how IQK compares to k-quants / i-quants / AWQ, see **[../QUANTIZATION.md](../QUANTIZATION.md)**. For the cross-engine overview see **[../INFERENCE_ENGINES.md](../INFERENCE_ENGINES.md)**. + +--- + +## TL;DR + +- **Image:** `ghcr.io/ikawrakow/ik-llama-cpp:cu13-server` (cu13 = CUDA 13.x; matches our 13.2 host driver). Official, digest-pinnable. A `cu12` tag exists for older drivers. +- **Compose:** `models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp.yml` (text) + `iq4ks-mtp-vision.yml` (vision). +- **Model:** ubergarm `Qwen3.6-27B-MTP-IQ4_KS.gguf` (IQK imatrix quant, built-in MTP head). +- **Interface:** same `--jinja` + `--reasoning on|off` server contract as mainline llama.cpp — so the stack-wide thinking-off policy works unchanged. + +```bash +MODEL_DIR=/your/models docker compose \ + -f models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp.yml up -d +curl http://localhost:8020/v1/models +``` + +--- + +## Why pick ik_llama.cpp over mainline llama.cpp? + +Both are cliff-immune (same ggml flat allocator — see [LLAMA_CPP.md](LLAMA_CPP.md) "Why llama.cpp doesn't hit the prefill cliffs"). ik_llama adds, on top: + +1. **IQK imatrix quants (fork-exclusive).** `IQ4_KS` / `IQ5_KS` use refined non-linear grids + an importance matrix + **kernels co-designed for those grids**. Net: better quality-per-bit than mainline `Q4_K_M`, and *faster* than mainline i-quants because the dequant path is hand-tuned. `IQ4_KS` is ~15.1 GB vs `Q4_K_M`'s ~17 GB on Qwen3.6-27B — smaller weights leave room for **262K context** on a single 24 GB card. +2. **Fused CUDA kernels** — `--merge-qkv` (fused QKV projection), `--merge-up-gate-experts` (MoE, no-op on dense), and a fast IQK dequant path. +3. **`-khad` (Hadamard K-cache)** — a Hadamard transform on the K-cache that improves quantized-KV accuracy, so you keep more quality at `q4_0`/`q8_0` KV. +4. **MoE-on-consumer tooling** — `-ser` (smart expert reduction) + on-the-fly MLA tensors make big-MoE-over-VRAM (DeepSeek/Kimi-class) practical, an alternative to ktransformers. +5. **Qwen3.x MTP on `main`** — MTP merged on the fork's main branch (mainline still has it as open [PR #22673](https://github.com/ggml-org/llama.cpp/pull/22673)), so no PR-branch building. + +The cost: it's a **fork** with no tagged releases (rolling `main`, smaller community) — a second engine image to track vs the clean `ggml-org/llama.cpp:server-cuda`. That's the trade for the IQK quants. Use mainline `llamacpp/mtp` for the conservative production path; use ik_llama when you want the cutting-edge quant. + +--- + +## Pros + +- **IQK imatrix quants** — fork-exclusive, best quality-per-bit available in the GGUF world. +- **Cliff-immune** (inherits llama.cpp's ggml allocator — no Cliff 1/2 GDN OOM). +- **262K context on a single 3090** with MTP (q4_0 KV ~5 GB; verified, verify-stress 7/7 incl. 91K needle). +- **Broad hardware** — CUDA (incl. CC 7.0 Volta), ROCm, Apple Metal, Intel, CPU (same as mainline). +- **MoE-over-VRAM** path (`-ser` + MLA) for models bigger than your cards. +- Same `--jinja` / `--reasoning` server contract as mainline → froggeric chat template + thinking-off both work (validated on this fork; see "Gotchas"). + +## Cons + +- **A fork to track** — rolling `main`, no tags, smaller community; pin by image digest. +- **GGUF-only** — no safetensors / vLLM-class continuous batching (single-stream `-np 1` is the sweet spot here). +- **Tooling lag** — some mainline llama.cpp server flags differ or arrive later (e.g. `--alias`, `--no-mmproj-offload` are in source builds but not every published image tag). +- Not the production default — that's vLLM (dual, max TPS) and mainline llama.cpp (single, conservative). + +--- + +## Quick recipe + +### 1. Pull the engine image +```bash +docker pull ghcr.io/ikawrakow/ik-llama-cpp:cu13-server # cu12-server for CUDA 12 drivers +``` + +### 2. Get an IQK GGUF +```bash +hf download ubergarm/Qwen3.6-27B-GGUF Qwen3.6-27B-MTP-IQ4_KS.gguf \ + --local-dir $MODEL_DIR/qwen3.6-27b-gguf/ubergarm-mtp-iq4ks +# Always SHA256-verify multi-GB GGUFs after download. +``` + +### 3. Launch (via our compose) +```bash +MODEL_DIR=$MODEL_DIR docker compose \ + -f models/qwen3.6-27b/ik-llama/compose/single/iq4ks-mtp.yml up -d +``` +Defaults: q4_0 KV, 131K ctx, MTP n=2, froggeric v19 template, thinking-off. Overrides: +- **Max context (262K):** `UBATCH_SIZE=512 CTX_SIZE=262144` (q4_0 KV is the default; ~21.5 GB / 24). +- **Higher KV fidelity:** `KV_TYPE=q8_0` (caps ~131-200K — q8_0 KV @262K OOMs). +- **Reasoning on:** `REASONING=on` (pair with `MTP_DRAFT_N_MAX=5 DRAFT_P_MIN=0.5` — reasoning text drafts deeper). + +--- + +## Tuning levers (ik-specific) + +| Flag | What it does | Default here | +|---|---|---| +| `-ctk` / `-ctv` | KV cache quant (`q4_0` / `q8_0` / `f16`) — biggest VRAM lever | `q4_0` | +| `-khad` / `--k-cache-hadamard` | Hadamard transform on K-cache → better quantized-KV accuracy | on | +| `--merge-qkv` (`-mqkv`) | Fused QKV projection | on | +| `--merge-up-gate-experts` (`-muge`) | Fused MoE up/gate (no-op on dense models) | — | +| `-ser N,f` | Smart expert reduction (big-MoE) | — | +| `--multi-token-prediction` + `--draft-max` / `--draft-p-min` | Built-in MTP spec-decode | n=2 / p-min 0.0 | +| `-b` / `-ub` | Batch / micro-batch — `-ub 512` unlocks higher ctx at ~2-4% TPS | 4096 / 1024 | + +--- + +## ik_llama-specific gotchas + +- **froggeric chat template works here** (unlike mainline). The mainline `llama.cpp` note that froggeric "silently suppresses `--reasoning off`" is a *mainline* issue — on ik_llama, froggeric v19 + `--reasoning off` suppresses thinking cleanly **and** renders tool-calls correctly (validated 2026-05-21). So the ik composes ship froggeric v19; mainline composes stay native. +- **The published image lags source on a few flags.** `--alias` and `--no-mmproj-offload` exist in from-source builds but not the `cu13-server` tag we pull — don't copy a from-source config verbatim. Check `llama-server --help` in the container. +- **Single-stream is the regime.** `-np 1` — this isn't a continuous-batching server. For multi-tenant, use vLLM. +- **MoE flags are no-ops on dense models** (`--merge-up-gate-experts`, `-ser` do nothing on dense Qwen3.6-27B; they matter for the MoE catalog). + +--- + +## Measured on this stack (Qwen3.6-27B, IQ4_KS + MTP, single 3090) + +| Metric | Value | vs shipped `llamacpp/mtp` (Q4_K_M) | +|---|---|---| +| Narrative TPS (wall / decode) | ~62.8 / 64.4 | +18-20% | +| Code TPS (wall / decode) | ~68.7 / 71.9 | | +| Max context (1× 3090) | **262K** (q4_0 KV) | vs 131K | +| verify-stress | 7/7 (incl. 91K Cliff 2 needle) | parity | +| Quality 8-pack | ≈ parity with the vLLM autoround path | — | +| toolcall-15 | 60% — **no regression** (froggeric recovers the formatting) | — | + +> Bench: canonical prompt, 3 warmup + measured runs, q4_0 KV / 131K / MTP n=2 / thinking-off, 370 W cap. See [../../BENCHMARKS.md](../../BENCHMARKS.md). + +--- + +## When ik_llama.cpp is the right pick + +- You want the **best quality-per-bit GGUF** (IQK imatrix) on a single card. +- You want **262K context on one 3090** with MTP decode speed. +- You're running a **big MoE that doesn't fit VRAM** and want `-ser`/MLA instead of ktransformers. +- You're **experimenting with newer quants** generally — this is the stack's advanced-quant track. + +## When to use something else + +- **Production multi-tenant / max dual-card TPS** → vLLM (`dual` / `dual-turbo`). +- **Conservative single-card, mainline image, no fork** → llama.cpp (`llamacpp/mtp`). +- **Apple Silicon** → either llama.cpp (Metal) or ik_llama (also Metal). + +## See also +- [../QUANTIZATION.md](../QUANTIZATION.md) — what IQK / imatrix / k-quants actually are +- [LLAMA_CPP.md](LLAMA_CPP.md) — the mainline sibling (shared cliff-immunity) +- [../INFERENCE_ENGINES.md](../INFERENCE_ENGINES.md) — full cross-engine comparison +- [../../BENCHMARKS.md](../../BENCHMARKS.md) — measured TPS across engines/configs diff --git a/docs/engines/README.md b/docs/engines/README.md index 9e30acd8..d56df34d 100644 --- a/docs/engines/README.md +++ b/docs/engines/README.md @@ -12,6 +12,7 @@ This repo's main path is **vLLM** because it has the deepest support for Qwen3-N |---|---|---|---|---|---|---|---| | **[vLLM](VLLM.md)** ⭐ | **Validated, production-grade** (this repo) | 50-53 narr / 66-70 code | 48K default · 75K IDE-agent · **198K vision · 214K text-only** | ✅ | ✅ | ✅ MTP n=3 | ✅ Full | | **[llama.cpp](LLAMA_CPP.md)** | Works mainline + [Luce DFlash fork](https://github.com/Luce-Org/lucebox-hub) for spec-decode | 35-60 (varies by quant + KV type) | **262K** (Q4_K_M + q4_0 KV) | ✅ (via mmproj) | ⚠️ Limited (no auto-tool-choice in server) | ✅ DFlash N=5 in fork | ⚠️ Partial | +| **[ik_llama.cpp](IK_LLAMA.md)** ⭐ *advanced quants* | **Shipped — advanced-quant track** | ~62 narr / ~69 code (IQ4_KS+MTP) | **262K** (IQ4_KS + q4_0 KV) | ✅ (mmproj) | ✅ (template + parser) | ✅ MTP n=2 | ⚠️ Partial (llama.cpp-class) | | **[SGLang](../../models/qwen3.6-27b/sglang/README.md)** | **Re-test pending** (May 2026). Historical block partially out-of-date — DFlash + MTP have landed natively on SGLang mainline; Marlin pad-sub-tile-n fix status unknown. See sglang README for re-test plan. | n/a (untested) | n/a | ✅ | ✅ | ✅ DFlash + MTP native upstream (untested here) | ✅ Full | --- @@ -100,7 +101,7 @@ Full plan in [models/qwen3.6-27b/sglang/README.md](../../models/qwen3.6-27b/sgla ## Quant choice (orthogonal to engine choice) -The model itself comes in several quant formats. Engine-quant compatibility: +The model itself comes in several quant formats. Engine-quant compatibility (full primer: **[QUANTIZATION.md](../QUANTIZATION.md)**): | Quant | Disk size | Engine fit | Notes | |---|---|---|---| @@ -110,6 +111,7 @@ The model itself comes in several quant formats. Engine-quant compatibility: | GGUF Q4_K_M | ~16.8 GB | llama.cpp ✅ · vLLM ⚠️ experimental · SGLang ❌ | The default GGUF mid-range quant. Strong quality, broad ecosystem (Ollama, LM Studio, etc). | | GGUF UD-Q3_K_XL ([Unsloth](https://huggingface.co/unsloth/Qwen3.6-27B-GGUF)) | **~14.5 GB** | llama.cpp ✅ | Smaller than 4-bit options. Quality cost is small on Qwen3.6 (quantization-friendly), buys substantial KV cache room. | | GGUF Q3_K_M | ~13.6 GB | llama.cpp ✅ | More aggressive 3-bit; quality cost real but acceptable for many workloads. | +| **GGUF IQ4_KS (imatrix)** ⭐ ([ubergarm](https://huggingface.co/ubergarm/Qwen3.6-27B-GGUF)) | **~15.1 GB** | **[ik_llama.cpp](IK_LLAMA.md) only** · llama.cpp ❌ · vLLM ❌ | Best quality-per-bit GGUF (imatrix + kernels co-designed for IQK grids). Smaller than Q4_K_M → **262K single-card**. Fork-exclusive — see [QUANTIZATION.md](../QUANTIZATION.md). | ### AutoRound vs GPTQ vs AWQ (within vLLM) @@ -136,6 +138,7 @@ If MTP isn't a priority for your workload, GPTQ or AWQ are equally valid. - **[VLLM.md](VLLM.md)** — current setup (what this repo ships). Brief recap + tuning levers. - **[LLAMA_CPP.md](LLAMA_CPP.md)** — quick GGUF recipe, vision via mmproj, Luce DFlash fork pointer for spec-decode, gotchas around server feature parity. - **[SGLANG.md](SGLANG.md)** — current blocked state, what would unblock, when to revisit. TBD recipe placeholder until either Marlin pad lands upstream or DeltaNet rollback lands. +- **[IK_LLAMA.md](IK_LLAMA.md)** ⭐ — the advanced-quant engine: fork-exclusive IQK imatrix quants (`IQ4_KS`), 262K single-card, MTP, `-khad` / `--merge-qkv`. Pairs with [QUANTIZATION.md](../QUANTIZATION.md). ---