0b4e694ad3
* Add llama.cpp bounded-thinking compose * fix(structured-cot/llamacpp): ship no-underscore grammar variant + record on-rig validation On-rig validation found the xgrammar deepseek-scratchpad.gbnf does NOT parse under llama.cpp's GBNF — underscored rule names (line_char/line_chars) trip "parse: error parsing grammar: expecting newline or end at _char", and llama-server silently falls back to UNCONSTRAINED generation (FREE==FSM token counts — silent no-fire). Fix: deepseek-scratchpad.llamacpp.gbnf — identical language, rule names with no underscores (llama.cpp GBNF is [a-zA-Z0-9-] only). Validated 2026-05-24 on 1x 3090: parses, fires (reasoning_content = PLAN/NOTE/VERDICT), coexists with --spec-type draft-mtp (draft accept 0.66-0.88). FREE->FSM 2113->1009 tok (2.1x) and 934->331 (2.8x) @ temp 0.6. Repointed the llama.cpp compose + READMEs + STRUCTURED_COT.md llama.cpp examples to the variant; left the vLLM grammar + refs untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * docs(structured-cot): redact pre-existing /home/wasif absolute paths Pre-commit leak scan caught host-local /home/wasif/structured-cot/runs/... paths in the 'What's saved on disk' section — pre-existing (2026-04-30 doc), not introduced by this branch. Redacted to host-local relative form per no-internal-paths-in-public-docs. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> --------- Co-authored-by: noonghunna <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Grammar evaluation harness
Tools to A/B-test alternative bounded-thinking grammars against the originally-shipped andthattoo/structured-cot GOAL/APPROACH/EDGE grammar (in docs/STRUCTURED_COT.md). Specifically targeting Holiday_Purpose_3166's tagline grammar from r/LocalLLaMA.
The hypothesis: the K (keywords, 1-5 free tokens) and R (result-keywords, 1-5 free tokens) fields in Holiday's grammar give the model a pressure-relief valve that GOAL/APPROACH/EDGE's rigid 3-line shape lacks. We have 6 documented HE+ regression cases (HE/97, 101, 108, 129, 137, 151) where FSM under-thinks and FREE wins; if Holiday's grammar rescues even some of those without losing too much compression, it's a Pareto improvement worth shipping.
Files
| File | Purpose |
|---|---|
deepseek-scratchpad.gbnf |
Recommended default grammar for bounded-thinking; vLLM/xgrammar (uses underscored rule names — does NOT parse on llama.cpp) |
deepseek-scratchpad.llamacpp.gbnf |
llama.cpp variant of the above — identical language, rule names with no underscores (llama.cpp GBNF is [a-zA-Z0-9-] only). Use this for the llama.cpp bounded-thinking compose. Validated on-rig 2026-05-24. |
holiday-tagline.gbnf |
Translated grammar, xgrammar-compatible |
TRANSLATION.md |
Translation decisions + Phase-1 smoke results |
smoke-test.py |
Phase-1 — verify the grammar parses + applies on vLLM |
subset-bench.py |
Phase-2 — 30-prompt HE+ A/B vs current grammar + FREE + PROMPT_TERSE |
Serving notes
The recommended production/default grammar is the DeepSeek scratchpad. The two engines need different files (same language, different rule-name dialect): vLLM uses deepseek-scratchpad.gbnf as extra_body={"structured_outputs": {"grammar": ...}}; llama.cpp uses deepseek-scratchpad.llamacpp.gbnf (no underscores in rule names) as the request body grammar field — the underscored vLLM version silently fails to parse on llama.cpp and falls back to unconstrained generation. See docs/STRUCTURED_COT.md for engine-specific request examples + validation status.
How to run
Phase 1 — smoke (~3 min):
python3 tools/grammar-eval/smoke-test.py --base-url http://localhost:8020/v1
# or with auto-boot:
python3 tools/grammar-eval/smoke-test.py --boot
5 prompts × tagline-grammar. Pass criterion: all 5 return successfully + match the BODY_RE regex. Validated 2026-05-03 PM — see TRANSLATION.md.
Phase 2 — 30-prompt subset bench (~30-60 min):
/tmp/structured-cot-venv/bin/python tools/grammar-eval/subset-bench.py \
--base-url http://localhost:8020/v1
Subset = 6 known FSM-regress problems + 4 known FREE-regress + 20 random HE+. Compares: FREE vs current GOAL/APPROACH/EDGE vs Holiday tagline vs PROMPT_TERSE. Outputs results/grammar-ab-<timestamp>/{results.jsonl, summary.md}.
Headline question phase 2 answers: of the 6 known HE+ FSM regressions, how many does Holiday's grammar rescue? 0 = drop the experiment. 3+ = phase 3 (full HE+ 164 + LCB v6 50) is worth running.
Phase 3 — full bench (only if phase 2 shows signal):
Re-run the same harness against the full HumanEval+ 164 + LiveCodeBench v6 50. ~6-8 hours of compute. Updates the table in docs/STRUCTURED_COT.md.
Background
The brief that produced these tools is at docs/diagnostics/grammar-eval-codex-brief.md (gitignored). The companion LLM-agnostic design exercise prompt (for refining beyond Holiday's design) is at docs/diagnostics/grammar-design-llm-prompt.md (gitignored).