concurrency-probe: success = tokens generated, not non-empty content
Found testing vibethinker-3b (a reasoning model): with a small max_tokens it spends the budget mid-<think> and returns HTTP 200 with 256 tokens but EMPTY content (never emits the final answer). The old content-based check wrongly flagged those streams as silent-empty failures. For a KV-pool stress test the stream DID run (generated tokens, held KV) -> ok = completion_tokens > 0; a true silent-empty is HTTP 200 with ZERO tokens (what soak-test means). After the fix vibethinker sustains N=8 clean on a 3090 (8x the compose default, 0 post-warm growth). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
This commit is contained in:
@@ -83,9 +83,13 @@ def one(stream, rnd):
|
||||
try:
|
||||
r = json.load(urllib.request.urlopen(req, timeout=600))
|
||||
toks = (r.get("usage") or {}).get("completion_tokens", 0)
|
||||
content = (r.get("choices") or [{}])[0].get("message",{}).get("content","") or ""
|
||||
ok = toks > 0 and len(content.strip()) > 0
|
||||
return {"ok": ok, "toks": toks, "silent": toks == 0 or not content.strip(),
|
||||
# KV-pool stress semantics: a stream "ran" if it GENERATED tokens
|
||||
# (held KV, decoded) — content-emptiness is irrelevant here (reasoning
|
||||
# models legitimately return empty content when max_tokens truncates
|
||||
# mid-<think>). silent-empty = HTTP 200 but ZERO tokens (the real
|
||||
# failure soak-test flags).
|
||||
ok = toks > 0
|
||||
return {"ok": ok, "toks": toks, "silent": toks == 0,
|
||||
"err": None, "dt": time.time()-t0}
|
||||
except Exception as e:
|
||||
return {"ok": False, "toks": 0, "silent": False, "err": str(e)[:80], "dt": time.time()-t0}
|
||||
|
||||
Reference in New Issue
Block a user