alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(bench-agentic): decouple context ramp from tool-call success (#255) (#396)

Browse Source 
bench-agentic.sh drove the ramp with tool_choice='required' and RAISED if a
turn returned no parseable tool call → the main loop caught it and broke,
aborting at the first miss. Reachable depth was bounded by tool-call
reliability at depth, not the TURNS/context budget — so on flaky parsers the
ramp stopped below the ~35K DeltaNet degrade zone the producer exists to
characterize (field-observed abort at turn 11 / ~22K).

Fix: on a no-parseable-tool-call turn, synthesize a tool call so the prompt
keeps growing by the same fixed ~chars (the fixture tool_result is injected
regardless and dominates the growth), flag tool_call_missed, and continue.
Genuine transport errors (HTTP/timeout) still propagate and stop the ramp.
Per-turn rows mark misses; a summary line reports "tool-call misses: N/M".

The optional non-tool RAMP_MODE the issue floats is largely subsumed — the
synthesize-on-miss path already lets the ramp reach configured depth on any
engine regardless of tool-call reliability.

Test: scripts/tests/test-bench-agentic-ramp.sh — mock SSE endpoint; asserts the
ramp reaches turn 3 under 100% tool-call misses (counted) AND the success path
is unchanged (no false misses). Offline, no GPU.

Caveat: guarantees reaching configured TURNS; whether the 15-turn fixture
exceeds 35K is separate (extend the fixture if not). Sibling bench-agentic fix
#498 stays separate.

Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
noonghunna
2026-06-13 18:53:54 +05:00
committed by GitHub
parent c1907dd1ce
commit 7df200163e
2 changed files with 118 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

51
scripts/bench-agentic.sh
View File

@@ -34,14 +34,16 @@
# and requests timed out around ~74K. Treat those as informational
# per-arch_class observations, not universal thresholds.
#
# Ramp-depth caveat:
# The context ramp is driven by tool_choice='required' turns. If the model
# fails to emit a parseable tool call at depth (intermittent on some
# parsers/configs), the ramp stops there — so the reachable depth, and thus
# whether the ~35K degrade zone is observed, is bounded by tool-call
# reliability at depth on the target config, not by this script. A follow-up
# enhancement to decouple the ramp from tool-call success is tracked
# separately.
# Ramp robustness (#255):
# The context ramp is driven by tool_choice='required' turns but does NOT
# depend on tool-call success. If the model fails to emit a parseable tool
# call at depth (intermittent on some parsers/configs), the turn synthesizes a
# tool call so the prompt keeps growing by the same fixed ~chars (the fixture
# tool_result is injected regardless) — the miss is logged + counted, but the
# ramp reaches the configured TURNS so the ~35K degrade zone is observed
# regardless of tool-call reliability. Only genuine transport errors
# (HTTP / timeout) stop the ramp. See the per-turn `tool_call_missed` flag and
# the "tool-call misses" summary line.
#
# Output:
# Per-turn table (turn, prompt_tokens, ttft_ms, decode_tps)
@@ -230,11 +232,22 @@ def run_turn(messages, fixture_turn, session_id, turn_idx):
"function": {"name": s["name"], "arguments": s["args"] or "{}"}}
for i, s in sorted(tool_calls_acc.items()) if s["name"]
]
if not tool_calls_response:
raise RuntimeError(
f"server returned no tool calls despite tool_choice=required "
f"(turn {turn_idx+1}). Check that the endpoint supports tool_choice=required."
)
# #255: decouple the context ramp from tool-call success. A turn that fails
# to emit a parseable tool call (intermittent parser flakiness at depth) used
# to abort the whole ramp via RuntimeError — capping reachable depth below
# the ~35K zone this producer exists to characterize. Instead, synthesize a
# tool call so the prompt keeps growing by the same fixed ~chars (the fixture
# tool_result below is injected regardless and is what dominates the growth),
# log + count the miss, and keep going. Genuine transport errors (HTTP /
# timeout) still propagate from urlopen and stop the ramp — you can't grow
# context off a dead request.
tool_call_missed = not tool_calls_response
if tool_call_missed:
tool_calls_response = [{
"id": f"call_t{turn_idx}_s{session_id}_synthetic",
"type": "function",
"function": {"name": TOOLS[0]["function"]["name"], "arguments": "{}"},
}]
# Sanitize: strip lone surrogates that json.dumps would emit as
# invalid \uD800-\uDFFF sequences, causing server-side 400s.
def _clean(s):
@@ -270,6 +283,7 @@ def run_turn(messages, fixture_turn, session_id, turn_idx):
"prompt_tokens": prompt_tokens,
"tool_calls": len(tool_calls_response),
"result_chars": len(tool_result_content),
"tool_call_missed": tool_call_missed,
}
@@ -277,6 +291,7 @@ def run_turn(messages, fixture_turn, session_id, turn_idx):
# Run sessions and collect per-turn metrics
# ---------------------------------------------------------------------------
per_turn_metrics = [[] for _ in range(TURNS)]
tool_call_misses = 0 # #255: turns where the model emitted no parseable tool call
for session in range(1, SESSIONS + 1):
print(f"\n{'='*72}")
@@ -292,9 +307,12 @@ for session in range(1, SESSIONS + 1):
try:
m = run_turn(messages, fixture_turn, session, turn_idx)
per_turn_metrics[turn_idx].append(m)
if m.get("tool_call_missed"):
tool_call_misses += 1
if not QUIET:
miss = " ⚠ tool-call miss (synthetic result injected)" if m.get("tool_call_missed") else ""
print(f" {turn_idx+1:<5} {m['prompt_tokens']:>10,} {m['ttft_ms']:>9.0f} "
f"{m['decode_tps']:>11.1f} {m['result_chars']:>13,}", flush=True)
f"{m['decode_tps']:>11.1f} {m['result_chars']:>13,}{miss}", flush=True)
except Exception as e:
print(f" turn {turn_idx+1}: FAIL — {e}", flush=True)
break
@@ -306,6 +324,11 @@ for session in range(1, SESSIONS + 1):
print(f"\n\n{'='*72}")
print(f"SUMMARY — multi-turn prefill stress ({SESSIONS} session(s) × {TURNS} turns)")
print(f"{'='*72}")
if tool_call_misses:
turns_run = sum(len(x) for x in per_turn_metrics)
print(f" tool-call misses: {tool_call_misses}/{turns_run} turns — ramp continued via "
f"synthetic results (#255); depth/curve unaffected, but tool-call reliability is "
f"degraded at depth on this config.")
print(f" {'Turn':<5} {'Prompt tok':>10} {'TTFT ms':>9} {'σ ms':>6} {'Decode TPS':>11} Notes")
print(f" {'-'*5} {'-'*10} {'-'*9} {'-'*6} {'-'*11} {'─'*35}")

81
scripts/tests/test-bench-agentic-ramp.sh Executable file
View File

@@ -0,0 +1,81 @@
#!/usr/bin/env bash
# #255: bench-agentic.sh must NOT abort the context ramp when a turn fails to
# emit a parseable tool call — it should synthesize one, inject the fixture
# tool_result, count the miss, and reach the configured TURNS. This test stands
# up a mock SSE endpoint and asserts both the miss path (ramp continues) and the
# success path (normal tool-call flow still works), fully offline (no GPU).
set -euo pipefail
ROOT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd)"
cd "$ROOT_DIR"
tmp="$(mktemp -d)"
port_file="$tmp/port"
cleanup() { [[ -n "${server_pid:-}" ]] && kill "$server_pid" 2>/dev/null || true; rm -rf "$tmp"; }
trap cleanup EXIT
# --- mock OpenAI-compatible SSE endpoint -----------------------------------
# MOCK_EMIT_TOOLCALL=0 → stream content only (no tool_calls) = the parse-miss.
# MOCK_EMIT_TOOLCALL=1 → stream a proper tool_call = the success path.
cat > "$tmp/mock.py" <<'PY'
import json, os, sys
from http.server import BaseHTTPRequestHandler, HTTPServer
EMIT_TC = os.environ.get("MOCK_EMIT_TOOLCALL", "0") == "1"
class H(BaseHTTPRequestHandler):
def log_message(self, *a): pass
def do_GET(self):
if self.path.rstrip("/").endswith("/v1/models"):
body = json.dumps({"data": [{"id": "mock"}]}).encode()
self.send_response(200); self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body))); self.end_headers()
self.wfile.write(body)
else:
self.send_response(404); self.end_headers()
def do_POST(self):
length = int(self.headers.get("Content-Length", 0))
self.rfile.read(length)
self.send_response(200); self.send_header("Content-Type", "text/event-stream"); self.end_headers()
def ev(d): self.wfile.write(f"data: {json.dumps(d)}\n".encode()); self.wfile.flush()
ev({"choices": [{"delta": {"content": "Looking into it."}}]})
if EMIT_TC:
ev({"choices": [{"delta": {"tool_calls": [
{"index": 0, "id": "call_mock_0", "function": {"name": "Read", "arguments": "{}"}}]}}]})
ev({"choices": [{"delta": {}}], "usage": {"prompt_tokens": 100, "completion_tokens": 5}})
self.wfile.write(b"data: [DONE]\n"); self.wfile.flush()
srv = HTTPServer(("127.0.0.1", 0), H)
with open(sys.argv[1], "w") as f: f.write(str(srv.server_address[1]))
srv.serve_forever()
PY
run_bench() { # $1 = MOCK_EMIT_TOOLCALL
rm -f "$port_file"
MOCK_EMIT_TOOLCALL="$1" python3 "$tmp/mock.py" "$port_file" & server_pid=$!
for _ in $(seq 1 50); do [[ -s "$port_file" ]] && break; sleep 0.1; done
local port; port="$(cat "$port_file")"
PREFLIGHT_NO_AUTODETECT=1 URL="http://127.0.0.1:${port}" MODEL=mock \
SESSIONS=1 TURNS=3 QUIET=1 bash scripts/bench-agentic.sh 2>&1
kill "$server_pid" 2>/dev/null || true; wait "$server_pid" 2>/dev/null || true
}
assert_contains() { [[ "$1" == *"$2"* ]] || { echo "ASSERT FAIL: missing '$2'"; echo "$1"; exit 1; }; }
assert_absent() { [[ "$1" != *"$2"* ]] || { echo "ASSERT FAIL: unexpected '$2'"; echo "$1"; exit 1; }; }
echo "── miss path: no parseable tool call → ramp must continue to turn 3 ──"
out="$(run_bench 0)"
assert_absent "$out" "FAIL" # ramp did NOT abort
assert_contains "$out" "tool-call misses: 3/3" # all 3 turns missed, counted
# turn 3 reached (the summary table prints the turn-3 row)
echo "$out" | grep -qE "^\s*3\s" || { echo "ASSERT FAIL: turn 3 not reached"; echo "$out"; exit 1; }
echo " ✓ ramp reached configured depth despite 100% tool-call misses"
echo "── success path: proper tool call → normal flow, zero misses ──"
out="$(run_bench 1)"
assert_absent "$out" "FAIL"
assert_absent "$out" "tool-call misses" # no misses line when all succeed
echo "$out" | grep -qE "^\s*3\s" || { echo "ASSERT FAIL: turn 3 not reached (success path)"; echo "$out"; exit 1; }
echo " ✓ success path intact, no false misses"
echo "test-bench-agentic-ramp: ok"