alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

llama-cpp: document speed-vs-context trade-off + fix stale ub default

Browse Source 
Follow-up to JensJN's question on #170 about pushing mtp-vision beyond 49K
context. Validated end-to-end 2026-05-20:

  Config                              ctx    VRAM     narr TPS  verify-stress
  shipped (ub=1024)                   49K    22.0 GB  56.5      ✓ (shipped)
  override (ub=512 CTX=131072)        131K   21.0 GB  50.0      ✓ 7/7
  override (ub=512 CTX=196608)        192K   22.5 GB  50.9      ✓ 7/7

~10% TPS hit buys 4× more context for agentic vision workloads. Keep
defaults stable (don't break shipped TPS numbers), but document the trade
+ give users explicit env-override examples so they can rebalance for
their workload.

Also corrects a stale `UBATCH_SIZE` default in the table (showed 2048,
actual is 1024 since v0.8.3 cliff-survival re-tune) + adds CTX_SIZE +
KV_TYPE to the table for completeness.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
noonghunna
2026-05-20 13:54:11 +00:00
parent 78556f8815
commit 1b2a76ca89
1 changed files with 27 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

29
models/qwen3.6-27b/llama-cpp/README.md
View File

@@ -45,17 +45,42 @@ Multimodal speed profile — the first stack config combining MTP + vision (the
### Tuning knobs
Both Docker composes expose llama.cpp's batch-size controls without editing YAML:
All three Docker composes expose llama.cpp's batch-size + KV controls without editing YAML:
| Env var | llama.cpp flag | Default | Sensible range on 24 GB | Notes |
|---|---|---:|---:|---|
| `CTX_SIZE` | `-c` | varies by variant | up to ~256K (q4_0 KV) | KV pool size. See per-variant defaults below. |
| `BATCH_SIZE` | `-b` | `4096` | `2048`-`8192` | Logical prompt-processing batch. Higher can improve prefill throughput if VRAM headroom allows. |
| `UBATCH_SIZE` | `-ub` | `2048` | `1024`-`4096` | Physical microbatch. Lower this first if long prompts OOM during prefill. |
| `UBATCH_SIZE` | `-ub` | `1024` | `512`-`4096` | Physical microbatch. **Lower this first if long prompts OOM during prefill** — but it also has a major impact on max-context (see next section). |
| `KV_TYPE` | `--cache-type-k/-v` | `q4_0` | `q4_0`, `q5_0`, `q8_0` | Lower KV bits-per-value = more ctx fits at same VRAM (quality trade-off is small at q4_0 for this model). |
These are throughput-tuning knobs inside llama.cpp. They are orthogonal to
`ESTATE_GPUS` and `ESTATE_PORT`, which only isolate GPU assignment and host port
when `scripts/launch.sh --estate` boots multiple instances.
### Speed vs context — pick your trade-off
`UBATCH_SIZE` (the `-ub` chunked-prefill chunk) is doing two jobs at once: it caps the **per-pass activation buffer** (cliff-survival for tool prefill) AND it eats into the **VRAM budget that could otherwise go to KV cache**. We ship `1024` as the default sweet spot, but you can rebalance:
**For `llamacpp/mtp-vision` specifically** — the vision encoder (mmproj F16, ~0.8 GB) competes for the same VRAM budget. The shipped 49K ctx + ub=1024 is the **speed-optimal** point on a single 3090. If you need more ctx for agentic vision workloads (UI navigation, multi-step tool use, long screenshots-in-context), drop `-ub` to 512 and you can push context up to 192K with full cliff coverage:
```bash
# Tested 2026-05-20 on single 3090, verify-stress 7/7 (incl. 60K + 91K needle):
UBATCH_SIZE=512 CTX_SIZE=196608 bash scripts/switch.sh llamacpp/mtp-vision
```
| Config | ctx | VRAM | narr TPS | verify-stress | When to pick |
|---|---|---:|---:|:---:|---|
| shipped: `ub=1024` | 49K | 22.0 GB | **56.5** | 7/7 ✓ | speed-first, short context |
| override: `ub=512 CTX=131072` | 131K | 21.0 GB | 50.0 | 7/7 ✓ | balanced (extra headroom) |
| override: `ub=512 CTX=196608` | **192K** | 22.5 GB | 50.9 | 7/7 ✓ | **max ctx with cliff coverage** |
So ~10% TPS hit (56.5 → 50.9 narr) buys ~4× more context (49K → 192K). For pure-chat / short-prompt workloads, keep the default. For agentic vision, override.
**For `llamacpp/mtp` (no vision)** — the same `-ub` 512 trade applies but with smaller margins (no mmproj competing for VRAM). Probe with `UBATCH_SIZE=512 CTX_SIZE=196608 bash scripts/switch.sh llamacpp/mtp` if you need more than the shipped 131K — we haven't shipped this as a default but the lever is there.
**For `llamacpp/default`** — already at the model's training-max 262K ctx; `-ub` is not a useful lever (no ctx upside, only TPS cost). Keep the default `1024`.
---
## Recipes (host-binary alternative)