long-text/long-vision/bounded-thinking: middle-ground recovery 130K → 175K / 120K → 140K
After d803278 (130K + 0.95 / 120K + 0.94) shipped, audit surfaced that the
backoff was driven by a synthetic 200K-char (50K-token) single-shot stress
that's heavier than typical agent workloads (ampersandru's repro was ~30K
real tokens; VolandBerlioz's was similar). Realistic agent workloads stay
in the 130K-char (33K-token) class which both 130K + 0.95 and 175K + 0.97
pass.
Recovery: middle-ground configs that keep ~360-720 MiB activation headroom
over the original 0.985 / 0.98 mem-util but recover meaningful context.
long-text: 130K + 0.95 → 175K + 0.97 verify-full 8/8 (AL 2.87),
130K-char stress PASS
bounded-thinking: 130K + 0.95 → 175K + 0.97 parity with long-text
(verified earlier in #134)
long-vision: 120K + 0.94 → 140K + 0.95 verify-full 8/8 (AL 2.49),
130K-char stress PASS
(intermediate 160K + 0.96
booted but failed 130K
stress on vision tower
overhead; 150K + 0.95
wouldn't boot — engine
ceiling at 0.95 vision
is 140352)
200K-char (50K-token) single-shot synthetic stress still cliffs on all
three — that's the FA varlen workspace allocation we can't reach. The bar
that matters for real users (verify-full + 130K-char stress) is met.
Docs updated: SINGLE_CARD.md picker table + activation-budget rationale +
per-variant blurbs; engines/VLLM.md TL;DR + KV cache table; engines/
LLAMA_CPP.md "when to use vLLM instead"; STRUCTURED_COT.md "When to pick
this over long-text"; models/qwen3.6-27b/README.md per-variant lines.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
noonghunna
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
d803278ebc
commit
383b5cc381
+8
-8
@@ -10,9 +10,9 @@ Four recommended options:
|
||||
|
||||
| What you're doing | Compose | Max ctx | Narr / Code TPS | VRAM (24 GB / card) |
|
||||
|---|---|---|---|---|
|
||||
| **Long ctx + vision** (chat, agents, image input) | [`long-vision.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.long-vision.yml) | **120K** | ~50 / ~67 | ~22.1 GB (mem-util 0.94) |
|
||||
| **Long ctx, text-only** (RAG, codebase, books) | [`long-text.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.long-text.yml) | **130K** | ~50 / ~66 | ~22.4 GB (mem-util 0.95) |
|
||||
| **Bounded thinking** (coding agents, structured-CoT, cost-bounded thinking) — see [STRUCTURED_COT.md](STRUCTURED_COT.md) | [`bounded-thinking.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml) | **130K** | ~52 / ~56 | ~22.4 GB (mem-util 0.95) |
|
||||
| **Long ctx + vision** (chat, agents, image input) | [`long-vision.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.long-vision.yml) | **140K** | ~50 / ~67 | ~22.4 GB (mem-util 0.95) |
|
||||
| **Long ctx, text-only** (RAG, codebase, books) | [`long-text.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.long-text.yml) | **175K** | ~50 / ~66 | ~22.8 GB (mem-util 0.97) |
|
||||
| **Bounded thinking** (coding agents, structured-CoT, cost-bounded thinking) — see [STRUCTURED_COT.md](STRUCTURED_COT.md) | [`bounded-thinking.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml) | **175K** | ~52 / ~56 | ~22.8 GB (mem-util 0.97) |
|
||||
| **Bulletproof, no cliffs** (production service, unpredictable inputs) | [`llamacpp/default`](../models/qwen3.6-27b/llama-cpp/compose/docker-compose.yml) | **262K** | 21 / 21 | ~20 GB |
|
||||
|
||||
Run via `bash scripts/launch.sh` (interactive) or `bash scripts/switch.sh <variant>`.
|
||||
@@ -21,7 +21,7 @@ Run via `bash scripts/launch.sh` (interactive) or `bash scripts/switch.sh <varia
|
||||
>
|
||||
> **vLLM single-card variants will crash if you send a single prompt above ~50K tokens.**
|
||||
>
|
||||
> This is Cliff 2 — DeltaNet GDN forward OOMs at 50–60K single-shot regardless of how much VRAM you have left. Both `long-vision.yml` (120K) and `long-text.yml` (130K) are designed for **steady-state accumulation across many turns** — context that builds up across tool calls, replies, retrieved chunks. They are NOT designed for "paste an 80K-token document and ask one question."
|
||||
> This is Cliff 2 — DeltaNet GDN forward OOMs at 50–60K single-shot regardless of how much VRAM you have left. Both `long-vision.yml` (140K) and `long-text.yml` (175K) are designed for **steady-state accumulation across many turns** — context that builds up across tool calls, replies, retrieved chunks. They are NOT designed for "paste an 80K-token document and ask one question."
|
||||
>
|
||||
> **If your workload ever sends single big prompts:** use `llamacpp/default` (262K, no cliffs anywhere — different engine entirely) or move to dual-card (`dual.yml` TP=2, verified at 237K).
|
||||
>
|
||||
@@ -46,7 +46,7 @@ What this says about single-card constraints:
|
||||
- **Model weights** consume ~14 GB (AutoRound INT4 / GGUF Q3_K_XL). Half the card.
|
||||
- **KV cache** is the next biggest line; its size depends on `--kv-cache-dtype` × ctx. fp8 ≈ 1 byte/token/(layer×head); TQ3 ≈ 0.4 bytes/token/(layer×head); fp16 ≈ 2 bytes/token/(layer×head).
|
||||
- **Vision tower** (mmproj) costs ~0.5–1.0 GB extra when on.
|
||||
- **Activations + cudagraph pools** is what's left. At `--gpu-memory-utilization 0.92` (default 48K) you have 2-3 GB of activation headroom — comfortable. **2026-05-01 PM** — long-vision / long-text were previously shipped at 0.98 / 0.985 mem-util but P37/P38 testing surfaced a downstream FA varlen workspace cliff at 50K-token tool prefills that none of our patches reach. Backed off to 130K + 0.95 (long-text) / 120K + 0.94 (long-vision) — adds ~1.5 GiB activation headroom which the FA workspace can grow into. Re-push criteria in [`docs/CLIFFS.md`](CLIFFS.md).
|
||||
- **Activations + cudagraph pools** is what's left. At `--gpu-memory-utilization 0.92` (default 48K) you have 2-3 GB of activation headroom — comfortable. **2026-05-01 PM** — long-vision / long-text were previously shipped at 0.98 / 0.985 mem-util but P37/P38 testing surfaced a downstream FA varlen workspace cliff at 50K-token tool prefills that none of our patches reach. Settled on **175K + 0.97 (long-text)** and **140K + 0.95 (long-vision)** — adds ~360-720 MiB activation headroom which the FA workspace can grow into for realistic-workload tool prefills (130K-char class). The synthetic 200K-char (50K-token single-shot) stress still cliffs — that's beyond what most agent workloads emit. Re-push criteria in [`docs/CLIFFS.md`](CLIFFS.md).
|
||||
|
||||
For the cross-card TP=2 picture, see [`DUAL_CARD.md`](DUAL_CARD.md).
|
||||
|
||||
@@ -58,13 +58,13 @@ For the cross-card TP=2 picture, see [`DUAL_CARD.md`](DUAL_CARD.md).
|
||||
|
||||
**Workload:** chat with images, vision-aware coding agents, multimodal RAG. Anything where the user might paste a screenshot.
|
||||
|
||||
120K + vision tower + TQ3 KV + Genesis MTP n=3 + PN17 + P104 + P37/P38 + compile-safe sidecar at mem-util 0.94. `verify-full.sh` 8/8 (MTP AL 3.09); `verify-stress.sh` 130K-char tool-prefill OK. 200K-char tool-prefill (50K tokens) still cliffs — vision tower's persistent overhead tightens the margin.
|
||||
140K + vision tower + TQ3 KV + Genesis MTP n=3 + PN17 + P104 + P37/P38 + compile-safe sidecar at mem-util 0.95. `verify-full.sh` 8/8 (MTP AL 2.49); `verify-stress.sh` 130K-char tool-prefill OK. 200K-char tool-prefill (50K tokens) still cliffs — vision tower's persistent overhead tightens the margin.
|
||||
|
||||
### Long ctx, text-only — `long-text.yml` ⭐
|
||||
|
||||
**Workload:** RAG ingest, codebase analysis, book/document Q&A, long conversations without image input.
|
||||
|
||||
130K + no vision + TQ3 KV + same patch stack at mem-util 0.95. `verify-full.sh` 8/8 (MTP AL 3.22); `verify-stress.sh` passes 200K-char tool-prefill (50K tokens) — the regime where 218K + 0.985 cliffed before the backoff. Vision drop adds ~1 GB headroom over long-vision so this variant survives the heavier prefills.
|
||||
175K + no vision + TQ3 KV + same patch stack at mem-util 0.97. `verify-full.sh` 8/8 (MTP AL 2.87); `verify-stress.sh` 130K-char tool-prefill OK. Vision drop adds ~1 GB headroom over long-vision so this variant runs at higher ctx + mem-util safely.
|
||||
|
||||
### Bulletproof / no cliffs — `llamacpp/default` ⭐
|
||||
|
||||
@@ -88,7 +88,7 @@ Re-tested 2026-04-30 PM against [`Luce-Org/lucebox-hub`](https://github.com/Luce
|
||||
|
||||
Measured TPS on this rig (RTX 3090, greedy, single-stream, n_gen=1000):
|
||||
|
||||
| Workload | Luce DFlash 3.6+3.6 (TQ3 KV, max_ctx=65K) | vLLM long-text 130K |
|
||||
| Workload | Luce DFlash 3.6+3.6 (TQ3 KV, max_ctx=65K) | vLLM long-text 175K |
|
||||
|---|---|---|
|
||||
| Narrative essay | 37–47 TPS (mean ~40) | 50 TPS |
|
||||
| Code (heap/LRU/AST) | 63–76 TPS (mean ~72) | 66 TPS |
|
||||
|
||||
@@ -49,7 +49,7 @@ andthattoo's headline numbers were on Qwen3.6-35B-A3B MoE Q4_K_M / H100 / llama.
|
||||
|
||||
Our FSM pass@1 lands within 2pp of theirs on both benchmarks — the technique reproduces cleanly across the model+quant+engine substitution. Our accuracy *delta* is bigger because our FREE baseline is weaker (more on this in Caveats).
|
||||
|
||||
## When to pick this over the standard `long-text` 130K
|
||||
## When to pick this over the standard `long-text` 175K
|
||||
|
||||
Pick `bounded-thinking` when **all three** of:
|
||||
|
||||
@@ -57,7 +57,7 @@ Pick `bounded-thinking` when **all three** of:
|
||||
2. You want bounded thinking cost as a structural guarantee (not just "the prompt asks nicely").
|
||||
3. Your workload tolerates a ~10% per-token TPS hit in exchange for ~30× cheaper think output (per-problem wall-clock is faster, not slower).
|
||||
|
||||
Pick `long-text` (the regular variant) when none of those apply. The two composes are otherwise identical (same 130K context, same MTP n=3, same TQ3 KV, same patches). The only difference is one vLLM flag — `--structured-outputs-config.enable_in_reasoning true`, which is what makes grammar enforcement actually fire inside the `<think>` block on this stack.
|
||||
Pick `long-text` (the regular variant) when none of those apply. The two composes are otherwise identical (same 175K context, same MTP n=3, same TQ3 KV, same patches). The only difference is one vLLM flag — `--structured-outputs-config.enable_in_reasoning true`, which is what makes grammar enforcement actually fire inside the `<think>` block on this stack.
|
||||
|
||||
## How to use it
|
||||
|
||||
|
||||
@@ -244,7 +244,7 @@ The q8 → q4_0 jump is **counter-intuitive** because q8 is "higher precision"
|
||||
## When to use vLLM instead
|
||||
|
||||
- You need full OpenAI API parity (tools, streaming, structured output)
|
||||
- You want max context (>130K) on a single 3090 — vLLM single-card currently ships 130K text-only / 120K with vision (backed off from 218K/198K for activation-headroom safety; see [docs/CLIFFS.md](../CLIFFS.md)); llama.cpp goes to 262K
|
||||
- You want max context (>175K) on a single 3090 — vLLM single-card currently ships 175K text-only / 140K with vision (backed off from 218K/198K for activation-headroom safety; see [docs/CLIFFS.md](../CLIFFS.md)); llama.cpp goes to 262K
|
||||
- You need concurrent serving (multi-tenant)
|
||||
- You want MTP spec-decode (the integrated head, not DFlash)
|
||||
- You're hitting llama.cpp's Qwen3-Next limitations and want the actively-developed path
|
||||
|
||||
@@ -10,7 +10,7 @@ This is what the repo's [Quick start](../../README.md#quick-start) ships. Everyt
|
||||
- ✅ Full feature set: vision, tools, streaming, thinking, MTP n=3, TurboQuant 3-bit KV
|
||||
- ✅ Full OpenAI API parity
|
||||
- 51-55 narr / 67-70 code TPS on a single 3090
|
||||
- 48K default ctx · 120K with vision · 130K text-only (backed off from earlier 198K/218K — see [docs/CLIFFS.md](../CLIFFS.md) "vLLM pin compatibility status" for why; Cliff 2 still applies single-prompt >50–60K)
|
||||
- 48K default ctx · 140K with vision · 175K text-only (backed off from earlier 198K/218K — see [docs/CLIFFS.md](../CLIFFS.md) "vLLM pin compatibility status" for why; Cliff 2 still applies single-prompt >50–60K)
|
||||
|
||||
---
|
||||
|
||||
@@ -49,7 +49,7 @@ bash scripts/bench.sh # 3 warmups + 5 measured (narr + code)
|
||||
| Pro | Detail |
|
||||
|---|---|
|
||||
| **Deepest Qwen3-Next feature support** | Vision tower, MTP head, all attention variants supported upstream. |
|
||||
| **TurboQuant 3-bit KV** | Lets us reach 120K + vision or 130K text-only on 24 GB at the current safety-first config. No equivalent in llama.cpp; SGLang has it but blocked by other bugs. |
|
||||
| **TurboQuant 3-bit KV** | Lets us reach 140K + vision or 175K text-only on 24 GB at the current safety-first config. No equivalent in llama.cpp; SGLang has it but blocked by other bugs. |
|
||||
| **MTP speculative decoding** | Works out of the box on the Lorbus quant; mainline llama.cpp doesn't expose MTP. |
|
||||
| **Active development** | Bugs we hit get triaged within days. We've contributed back. |
|
||||
| **Full OpenAI API parity** | Tools, streaming, vision-in-message, reasoning-mode, structured output — everything works. |
|
||||
@@ -80,7 +80,7 @@ Control context vs activation headroom. See the [Activation-memory caveat](../..
|
||||
| `turboquant_4bit_nc` | ~23 KB | ~84K | Untested by us — should work |
|
||||
| `turboquant_3bit_nc` ⭐ | ~17 KB | ~125K | Default v7.14 variant |
|
||||
|
||||
Lower bytes/token = more context, but more dequant scratch + activation pressure. The 3-bit variant is what makes the 120K + vision and 130K text-only tiers reachable on 24 GB.
|
||||
Lower bytes/token = more context, but more dequant scratch + activation pressure. The 3-bit variant is what makes the 140K + vision and 175K text-only tiers reachable on 24 GB.
|
||||
|
||||
### Spec-decode (`--speculative-config`)
|
||||
|
||||
|
||||
@@ -50,9 +50,9 @@ How each config splits the 24 GB / card budget — weights, KV cache, vision tow
|
||||
|
||||
|
||||
As of 2026-04-30 PM, single-card recommended options (see [`docs/SINGLE_CARD.md`](../../docs/SINGLE_CARD.md)):
|
||||
- **`long-text.yml` — 130K text-only** at 0.95 mem-util. Backed off from 218K + 0.985 on 2026-05-01 PM after P37/P38 testing showed the FA varlen workspace cliff at 50K-token tool prefills. P37/P38 + PN17 + P104 + compile-safe sidecar all on; Mamba block alignment forbids smaller chunk size, so the lever is context + mem-util. Re-push criteria in [`docs/CLIFFS.md`](../../docs/CLIFFS.md).
|
||||
- **`long-vision.yml` — 120K + vision** at 0.94 mem-util. Same backoff rationale; vision tower's persistent ~1 GB caps margin slightly tighter than text-only.
|
||||
- **`bounded-thinking.yml` — 130K text-only + structured-CoT grammar in reasoning** at 0.95 mem-util. Same patches as long-text plus `--structured-outputs-config.enable_in_reasoning true`. ~30× cheaper think output on coding workloads with **+4.3pp HE+ / +24pp LCB v6** vs FREE thinking. See [`docs/STRUCTURED_COT.md`](../../docs/STRUCTURED_COT.md).
|
||||
- **`long-text.yml` — 175K text-only** at 0.97 mem-util. Middle-ground after P37/P38 testing surfaced the FA varlen workspace cliff at 50K-token single-shot prefills (synthetic). P37/P38 + PN17 + P104 + compile-safe sidecar all on; Mamba block alignment forbids smaller chunk size, so the lever was context + mem-util. Passes verify-full + 130K-char (33K-token) tool-prefill stress, which matches realistic agent workload thresholds. Re-push criteria in [`docs/CLIFFS.md`](../../docs/CLIFFS.md).
|
||||
- **`long-vision.yml` — 140K + vision** at 0.95 mem-util. Same backoff rationale; vision tower's persistent ~1 GB caps margin tighter than text-only (engine ceiling at 0.95 + vision is 140352).
|
||||
- **`bounded-thinking.yml` — 175K text-only + structured-CoT grammar in reasoning** at 0.97 mem-util. Same patches as long-text plus `--structured-outputs-config.enable_in_reasoning true`. ~30× cheaper think output on coding workloads with **+4.3pp HE+ / +24pp LCB v6** vs FREE thinking. See [`docs/STRUCTURED_COT.md`](../../docs/STRUCTURED_COT.md).
|
||||
- **`llamacpp/default` — 262K + vision** at ~21 TPS. Different engine, no cliffs anywhere — production-safe for unpredictable inputs.
|
||||
|
||||
The **single shipped limitation** on the vLLM variants: Cliff 2 still fires on single prompts >50–60K (DeltaNet GDN forward OOM). Use llama.cpp single or dual-card for one-shot big prompts. See [`docs/CLIFFS.md`](../../docs/CLIFFS.md).
|
||||
|
||||
@@ -158,14 +158,12 @@ services:
|
||||
- float16
|
||||
- --tensor-parallel-size
|
||||
- "1"
|
||||
# 130K + 0.95 — patch parity with long-text.yml (was 218K + 0.985).
|
||||
# Same FA varlen workspace cliff at 50K-token tool prefills; same
|
||||
# rationale; same backoff. See long-text.yml for the full trade-off
|
||||
# discussion and re-push criteria.
|
||||
# 175K + 0.97 — middle-ground parity with long-text.yml. See
|
||||
# long-text.yml for the full trade-off rationale and re-push criteria.
|
||||
- --max-model-len
|
||||
- "130000"
|
||||
- "175000"
|
||||
- --gpu-memory-utilization
|
||||
- "0.95"
|
||||
- "0.97"
|
||||
- --max-num-seqs
|
||||
- "1"
|
||||
- --max-num-batched-tokens
|
||||
|
||||
@@ -196,25 +196,25 @@ services:
|
||||
- float16
|
||||
- --tensor-parallel-size
|
||||
- "1"
|
||||
# 130K + 0.95 — backed off the long-context edge 2026-05-01 PM. Was
|
||||
# 205K + 0.985, before that 218K + 0.985. Trade rationale (per
|
||||
# bench-driven tuning):
|
||||
# * P37/P38 + PN17 + P104 + compile-safe sidecar close the line 903
|
||||
# torch.cat OOM that ampersandru hit, but the cliff moves downstream
|
||||
# to flash_attn_varlen_func internal allocation (turboquant_attn.py:909
|
||||
# → :394 → flash_attn_interface.py:300). 50-MiB workspace allocation
|
||||
# fails with 50.5 MiB free — 1 MiB margin at 0.985 + 200K-char prefill.
|
||||
# * Mamba cache align mode requires block_size <= max_num_batched_tokens
|
||||
# so we can't lower the chunk size below 4128 to shrink the workspace.
|
||||
# * Backing off to 130K + 0.95 frees ~1.5 GiB activation headroom AND
|
||||
# lowers KV pool requirement; 200K-char (~50K-token) tool-prefill
|
||||
# stress now passes cleanly + verify-full 8/8 (MTP AL 3.22).
|
||||
# Re-push toward 200K when an upstream FA fix or Sandermage's next pin
|
||||
# adds workspace clamp coverage at the FA varlen entry point.
|
||||
# 175K + 0.97 — middle-ground config. Was 218K + 0.985 originally;
|
||||
# initial backoff to 130K + 0.95 was conservative (driven by a synthetic
|
||||
# 200K-char single-shot stress that's heavier than real workloads). Real
|
||||
# OpenCode/agent reports (ampersandru, VolandBerlioz) topped out at
|
||||
# ~30K real tokens with multi-turn accumulation, well within the 130K-char
|
||||
# stress class which both 175K + 0.97 and 130K + 0.95 pass.
|
||||
# Trade vs original 205K + 0.985:
|
||||
# * +360 MiB activation headroom (0.985 → 0.97 frees the lever)
|
||||
# * 30K less context capacity
|
||||
# * 200K-char synthetic stress still fails (FA varlen workspace cliff;
|
||||
# no patch reaches that allocation site, Mamba alignment forbids
|
||||
# smaller chunk size). But verify-full 8/8 + 130K-char stress pass.
|
||||
# Re-push toward 200K when upstream FA adds varlen workspace clamping
|
||||
# OR Sandermage's next pin extends PN17 coverage to the kernel entry
|
||||
# point.
|
||||
- --max-model-len
|
||||
- "130000"
|
||||
- "175000"
|
||||
- --gpu-memory-utilization
|
||||
- "0.95"
|
||||
- "0.97"
|
||||
- --max-num-seqs
|
||||
- "1"
|
||||
- --max-num-batched-tokens
|
||||
|
||||
@@ -166,17 +166,21 @@ services:
|
||||
- float16
|
||||
- --tensor-parallel-size
|
||||
- "1"
|
||||
# 130K + 0.94 — backed off the edge 2026-05-01 PM. Was 198K + 0.98.
|
||||
# Same rationale as long-text.yml: P37/P38 + PN17 + P104 + compile-safe
|
||||
# close the line 903 torch.cat OOM but the cliff moves downstream to
|
||||
# FA varlen workspace allocation; Mamba block alignment forbids smaller
|
||||
# chunks (must be ≥4128). Vision adds ~1 GiB persistent so 0.94 here
|
||||
# vs 0.95 on long-text — slightly more conservative to leave room for
|
||||
# vision tower transients alongside FA varlen workspace.
|
||||
# 140K + 0.95 — middle-ground config. Was 198K + 0.98 originally; first
|
||||
# backoff went to 120K + 0.94, intermediate tries 160K + 0.96 (boots,
|
||||
# fails 130K stress — vision tower's ~1 GiB persistent overhead leaves
|
||||
# tighter margin than text-only) and 150K + 0.95 (won't boot — engine
|
||||
# ceiling at 0.95 + vision is 140352 per vLLM). 140K + 0.95 is the
|
||||
# verified middle-ground that boots AND passes 130K-char stress.
|
||||
# Trade vs original 198K + 0.98:
|
||||
# * +720 MiB activation headroom (0.98 → 0.95)
|
||||
# * 58K less context capacity
|
||||
# See long-text.yml for the full FA varlen workspace cliff rationale
|
||||
# and re-push criteria.
|
||||
- --max-model-len
|
||||
- "120000"
|
||||
- "140000"
|
||||
- --gpu-memory-utilization
|
||||
- "0.94"
|
||||
- "0.95"
|
||||
- --max-num-seqs
|
||||
- "1"
|
||||
- --max-num-batched-tokens
|
||||
|
||||
Reference in New Issue
Block a user