7e9f5cb930
The shared served-model-name `qwen3.6-27b-autoround` (from #490) mislabels the non-autoround 27b scenes (fp8 dual-max, lmcache): /v1/models advertises "autoround" while `root` points at qwen3.6-27b-fp8. Same class on gemma-4-31b — after the v0.24.0 consolidation the default is cyankiwi qat-AWQ-INT4 (bf16 KV), yet the LiteLLM route still targeted `gemma-4-31b-autoround` (a latent #482 drift). Fix WITHOUT breaking anything, via vLLM multi-served-name: - Every 27b scene now serves `qwen3.6-27b <its-quant-name>`; every gemma-31b scene serves `gemma-4-31b <its-quant-name>`. The neutral name is PRIMARY (honest /v1/models id); the quant-specific name is retained as a live ALIAS. - LiteLLM: add `qwen3.6-27b` / `gemma-4-31b` canonical public routes; keep the `-autoround` routes as back-compat aliases (same upstream). Repairs the gemma drift. - Migrate our own MODEL= defaults + docs (bench/verify/quality/launch/setup, c3, tui-core, EXAMPLES, ...) to the neutral name. Weights slugs (`-autoround-int4`) untouched; CHANGELOG + results/ history left as-is. Retiring the `-autoround` alias entirely is a deliberate later step once nothing still asks for it. Live-verified on-rig (single/minimal, stock v0.24.0): /v1/models lists BOTH names (root=...-autoround-int4); chat to `qwen3.6-27b` AND `qwen3.6-27b-autoround` both return 200; `qwen3.6-27b-fp8` correctly 404s. Full shell gate 59/59 (1 = known worktree-fixture); c3 pytest 41 passed; served-name arg-order + YAML validated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
Grammar evaluation harness
Tools to A/B-test alternative bounded-thinking grammars against the originally-shipped andthattoo/structured-cot GOAL/APPROACH/EDGE grammar (in docs/STRUCTURED_COT.md). Specifically targeting Holiday_Purpose_3166's tagline grammar from r/LocalLLaMA.
The hypothesis: the K (keywords, 1-5 free tokens) and R (result-keywords, 1-5 free tokens) fields in Holiday's grammar give the model a pressure-relief valve that GOAL/APPROACH/EDGE's rigid 3-line shape lacks. We have 6 documented HE+ regression cases (HE/97, 101, 108, 129, 137, 151) where FSM under-thinks and FREE wins; if Holiday's grammar rescues even some of those without losing too much compression, it's a Pareto improvement worth shipping.
Files
| File | Purpose |
|---|---|
deepseek-scratchpad.gbnf |
Recommended default grammar for bounded-thinking; vLLM/xgrammar (uses underscored rule names — does NOT parse on llama.cpp) |
deepseek-scratchpad.llamacpp.gbnf |
llama.cpp variant of the above — identical language, rule names with no underscores (llama.cpp GBNF is [a-zA-Z0-9-] only). Use this for the llama.cpp bounded-thinking compose. Validated on-rig 2026-05-24. |
holiday-tagline.gbnf |
Translated grammar, xgrammar-compatible |
TRANSLATION.md |
Translation decisions + Phase-1 smoke results |
smoke-test.py |
Phase-1 — verify the grammar parses + applies on vLLM |
subset-bench.py |
Phase-2 — 30-prompt HE+ A/B vs current grammar + FREE + PROMPT_TERSE |
Serving notes
The recommended production/default grammar is the DeepSeek scratchpad. The two engines need different files (same language, different rule-name dialect): vLLM uses deepseek-scratchpad.gbnf as extra_body={"structured_outputs": {"grammar": ...}}; llama.cpp uses deepseek-scratchpad.llamacpp.gbnf (no underscores in rule names) as the request body grammar field — the underscored vLLM version silently fails to parse on llama.cpp and falls back to unconstrained generation. See docs/STRUCTURED_COT.md for engine-specific request examples + validation status.
How to run
Phase 1 — smoke (~3 min):
python3 tools/grammar-eval/smoke-test.py --base-url http://localhost:8020/v1
# or with auto-boot:
python3 tools/grammar-eval/smoke-test.py --boot
5 prompts × tagline-grammar. Pass criterion: all 5 return successfully + match the BODY_RE regex. Validated 2026-05-03 PM — see TRANSLATION.md.
Phase 2 — 30-prompt subset bench (~30-60 min):
/tmp/structured-cot-venv/bin/python tools/grammar-eval/subset-bench.py \
--base-url http://localhost:8020/v1
Subset = 6 known FSM-regress problems + 4 known FREE-regress + 20 random HE+. Compares: FREE vs current GOAL/APPROACH/EDGE vs Holiday tagline vs PROMPT_TERSE. Outputs results/grammar-ab-<timestamp>/{results.jsonl, summary.md}.
Headline question phase 2 answers: of the 6 known HE+ FSM regressions, how many does Holiday's grammar rescue? 0 = drop the experiment. 3+ = phase 3 (full HE+ 164 + LCB v6 50) is worth running.
Phase 3 — full bench (only if phase 2 shows signal):
Re-run the same harness against the full HumanEval+ 164 + LiveCodeBench v6 50. ~6-8 hours of compute. Updates the table in docs/STRUCTURED_COT.md.
Background
The brief that produced these tools is at docs/diagnostics/grammar-eval-codex-brief.md (gitignored). The companion LLM-agnostic design exercise prompt (for refining beyond Holiday's design) is at docs/diagnostics/grammar-design-llm-prompt.md (gitignored).