d82e89807a
The results/ directory accumulated 21 MB of intermediate investigation output during today's cliff/grammar work. Most of that (residency probes, soak iteration matrices) lived its useful life in issue threads and the gitignored docs/diagnostics/ memos, and isn't worth eternal evidence. What's evidence (committed): 1.5 MB of raw bench output backing published numbers in STRUCTURED_COT.md / BENCHMARKS.md / CHANGELOG: - results/grammar-ab-20260503-224235/ — Phase 2 30-problem subset bench (was the basis for the n=30 PROMPT_TERSE-wins finding that Phase 3 later disproved at scale) - results/grammar-full-20260504-003118-gpu0/ — Phase 3 HE+ shard 1 (82 rows) - results/grammar-full-20260504-003118-gpu1/ — Phase 3 HE+ shard 2 (82 rows) - results/grammar-full-lcb-20260504-021003/ — Phase 3 LCB v6 (50 rows + shard metadata; this is the post-bug-fix re-run after the harness LCB issue surfaced and was patched) - tools/grammar-eval/codex-check-scratchpad.gbnf — fourth grammar candidate exploring DeepSeek + STATE/CHECK pre-VERDICT scaffolding (untested in Phase 3; available for future bench) What's investigation artifact (now gitignored): residency-* (Codex Cliff 2b investigation probes, ~5 MB) + soak-* (~16 MB of cliff probe iterations, final validation matrix already in #41 + cliff2_accumulated_ctx_finding.md memory). .gitignore pattern: `results/*` with explicit negations for `grammar-ab-*`, `grammar-full-*`, and the existing `v0.20-migration` directory. Future grammar bench runs auto-track when committed; future residency/soak runs auto-ignore. The pattern works because git's directory-ignore precedence applies to immediate children, not the parent itself, so negations re-include named subdirs. Total committed: 522 lines of bench output (jsonl + json + summary md). The data is small (per-row generations cap at ~4096 tokens × 5 conditions × 214 problems = ~4 MB raw, much of which compresses well in jsonl). Held back for user decision: tools/residency-instrument/ (Codex's Cliff 2b instrumentation harness — useful tool, no docs yet, may want a README before publication). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
81 lines
1.4 KiB
JSON
81 lines
1.4 KiB
JSON
{
|
|
"phase": "phase3_full",
|
|
"args": {
|
|
"base_url": "http://localhost:8020/v1",
|
|
"model": "qwen3.6-27b-autoround",
|
|
"tokenizer": "/opt/ai/github/qwen36-dual-3090/models/qwen3.6-27b-autoround-int4",
|
|
"structured_cot_dir": "/home/wasif/structured-cot",
|
|
"prior_results": "/home/wasif/structured-cot/runs/full-humaneval-2026-04-30/results.jsonl",
|
|
"current_grammar": "/home/wasif/structured-cot/grammars/fsm_grammar_no_open.gbnf",
|
|
"holiday_grammar": "/opt/ai/github/club-3090/tools/grammar-eval/holiday-tagline.gbnf",
|
|
"deepseek_grammar": "/opt/ai/github/club-3090/tools/grammar-eval/deepseek-scratchpad.gbnf",
|
|
"out_dir": "results/grammar-full-lcb-20260504-021003",
|
|
"seed": 42,
|
|
"max_tokens": 4096,
|
|
"request_timeout": 600.0,
|
|
"timeout": 30,
|
|
"save_raw": false,
|
|
"list_only": false,
|
|
"full": true,
|
|
"include_lcb": true,
|
|
"he_start": 0,
|
|
"he_end": 0,
|
|
"lcb_start": 0,
|
|
"lcb_end": 50,
|
|
"label": "lcb"
|
|
},
|
|
"he_indices": [],
|
|
"lcb_indices": [
|
|
0,
|
|
1,
|
|
2,
|
|
3,
|
|
4,
|
|
5,
|
|
6,
|
|
7,
|
|
8,
|
|
9,
|
|
10,
|
|
11,
|
|
12,
|
|
13,
|
|
14,
|
|
15,
|
|
16,
|
|
17,
|
|
18,
|
|
19,
|
|
20,
|
|
21,
|
|
22,
|
|
23,
|
|
24,
|
|
25,
|
|
26,
|
|
27,
|
|
28,
|
|
29,
|
|
30,
|
|
31,
|
|
32,
|
|
33,
|
|
34,
|
|
35,
|
|
36,
|
|
37,
|
|
38,
|
|
39,
|
|
40,
|
|
41,
|
|
42,
|
|
43,
|
|
44,
|
|
45,
|
|
46,
|
|
47,
|
|
48,
|
|
49
|
|
],
|
|
"plan_size": 50
|
|
} |