The performance-max VibeThinker path — prithivMLmods Q8_0 GGUF on mainline
llama.cpp, single 3090, q8_0 KV, full 131K, -b 4096 -ub 2048. A sibling to
the vLLM bf16 compose (#418), and the better one on every axis.
Why it's better than the vLLM sibling:
- ~166 TPS decode (vs ~110 vLLM bf16), ~6.2 GB at full 131K (vs ~9.8 GB),
prefill ~6,630 tok/s (-b 4096/-ub 2048 = +20%), boot ~9s.
- Q8_0 is near-lossless → quality INTACT, where vLLM's fp8 weight-quant broke
this quant-sensitive 3B (non-terminating empty output). KV quant is fine
(storage-only); fp8 *weights* are the problem.
Validated 2026-06-16 (tool-free packs, temp 0.6, thinking-on):
gsm-symbolic-30 100% · instructfollow-15 100% · reasonmath-15 80% ·
structoutput-15 80% · dataextract-15 40%.
dataextract is a genuine extraction ceiling — full temp sweep {0:27%,
0.6:40%, 1.0:27%} can't lift it (value mismatches persist). A reasoning
specialist, not an extractor. No tool-calling. verify-full 5/9 by design.
Wiring: registry entry + dense-family added to llama-cpp-local engine +
prithivmlmods-q8 weights variant + disk/registry counts (51/52). No MTP head
in the GGUF (plain qwen2 conversion) → no spec-dec. Guard suite 47/47.
Status: 🐣 incubating (hidden from switch.sh --list; --force to launch).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
