Disable Genesis P68/P69 in shipped composes (silent-stop bugfix)

P68 (auto force tool_choice=required) and P69 (inject "must use a tool" reminder) silently fired on prompts > 8000 chars — every IDE agent (Cline, Cursor, OpenCode, Copilot Gateway) blew past that threshold instantly and got silent finish_reason=stop with no content + no tool_calls on greetings or clarifying questions. Bisection on club-3090#2 (HoodOG1 + tenitram): state A (P64+P68+P69+PN8): broken state B (P64+P69+PN8, P68 off): still broken — model loops on "I cannot respond with plain text" then stops mid-reasoning state D (P64+PN8, P68+P69 off): clean — greeting → plain-text reply; tool request → clean read_file call P64 (qwen3coder MTP streaming early-return fix) and PN8 (FP8+MTP draft online-quant memory savings) stay enabled — real bugfixes, no user-intent override. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-29 19:06:06 +00:00
parent 3742244e4d
commit aab8ff4a0e
4 changed files with 24 additions and 5 deletions
@@ -2,6 +2,14 @@

 Changes that span the entire stack — engine version pins, script behavior, repo structure. Per-model dated history lives in `models/<name>/CHANGELOG.md`.

+## 2026-04-29 — Disable Genesis P68/P69 in shipped composes (silent-stop bugfix)
+
+- **`tools-text.yml`** + **`fast-chat.yml`** — `GENESIS_ENABLE_P68_AUTO_FORCE_TOOL` and `GENESIS_ENABLE_P69_LONG_CTX_TOOL_REMINDER` are now commented out (default-off). Caused silent finish_reason=stop with empty content + no tool_calls on greetings and clarifying questions when the prompt exceeded 8000 chars (the patches' threshold). Affected every realistic IDE-agent setup (Cline, Cursor, OpenCode, Copilot Gateway).
+- **Bisected via cross-rig data on club-3090 issue #2** (HoodOG1 + tenitram). State A (all 4 env vars on) reproduced the silent stop. State B (P68 off, P69 on) still broken — model loops on "I cannot respond with plain text" then stops mid-reasoning. State D (both P68 and P69 off) — clean: greeting → "Hello! How can I help you today?", tool request → clean `read_file` call.
+- **P64 and PN8 stay enabled** — both are real targeted bugfixes (P64 = qwen3coder MTP streaming early-return fix from kotori-yan vllm#39598 backport; PN8 = FP8+MTP draft online-quant memory savings, vllm#40849 backport). Neither overrides user intent.
+- Mechanism: `vllm/_genesis/middleware/long_ctx_tool_adherence.py:227` — P68 silently sets `request.tool_choice = "required"` when prompt > 8000 chars; P69 appends "must use a tool" text to the user message. Either alone makes "hi there" + tools fail. Together they're worse.
+- Threshold of 8000 chars is too low for IDE agents (typical context: 20-50K). We may file an upstream issue with Sandermage suggesting either raising the default or making opt-in+threshold mandatory-explicit.
+
 ## 2026-04-29 — UX polish: pre-flight checks + cards-first wizard + PNG embeds + per-page chart split

 - **`scripts/preflight.sh`** (new) — sourceable library of `preflight_docker`, `preflight_gpu [min]`, `preflight_disk <path> <gb>`, `preflight_gpu_idle`, `preflight_running`. Each prints actionable `Fix:` hints on failure rather than a cryptic mid-run crash.
@@ -120,7 +120,9 @@ bash scripts/switch.sh vllm/tools-text

 There's a second wrinkle: Copilot's LLM Gateway sometimes sends very low `max_tokens` (e.g. 64) on probe-style requests. With `tool_choice: required` (which Copilot enforces via `minItems: 1` on its structured-outputs schema), the model must emit a tool-call JSON that wraps a real argument like a file path — and 64 tokens isn't enough to fit `{"name": "read_file", "parameters": {"filePath": "/long/abs/path"}}`. The truncated JSON arrives at the gateway as "empty response." If you see this pattern, it's a client-side limit, not the server. Other OpenAI-compat clients (Cline / Continue.dev / Cursor) tend to send realistic max_tokens by default and don't hit this.

-Background + debug-log analysis: [club-3090 #2](https://github.com/noonghunna/club-3090/issues/2).
+**Server-side fix landed 2026-04-29:** the Genesis P68/P69 long-context tool-adherence patches were silently overriding `tool_choice: auto → required` and injecting "must use a tool" reminders whenever prompt > 8000 chars. That made greetings + clarifying questions stall on every IDE-agent setup (Cline, Cursor, OpenCode, and Copilot Gateway combined). We disabled both in `tools-text.yml` and `fast-chat.yml`. Behavior now: greeting → plain-text reply ("Hello! How can I help you today?"); tool request → clean `read_file({"path": "..."})` call. P64 and PN8 stay enabled (real targeted bugfixes, no user-intent override).
+
+Background + bisection: [club-3090 #2](https://github.com/noonghunna/club-3090/issues/2#issuecomment-4346345554).

 ---

@@ -63,8 +63,11 @@ services:
      # We don't enable P4 (TurboQuant-hybrid) — fp8 KV doesn't need it.
      # We don't enable P65/P67 — fp8 KV doesn't trigger #40880 cudagraph bug.
      - GENESIS_ENABLE_P64_QWEN3CODER_MTP_STREAMING=1
-      - GENESIS_ENABLE_P68_AUTO_FORCE_TOOL=1
-      - GENESIS_ENABLE_P69_LONG_CTX_TOOL_REMINDER=1
+      # P68/P69 deliberately DISABLED (2026-04-29) — see tools-text.yml header
+      # for full reasoning. Short version: 8000-char threshold is too low for
+      # IDE agents and breaks greetings / clarifying questions silently.
+      # - GENESIS_ENABLE_P68_AUTO_FORCE_TOOL=1
+      # - GENESIS_ENABLE_P69_LONG_CTX_TOOL_REMINDER=1
      - GENESIS_ENABLE_PN8_MTP_DRAFT_ONLINE_QUANT=1          # v7.62.x — opt-in test
    shm_size: "16gb"
    ipc: host
@@ -52,8 +52,14 @@ services:
      - VLLM_MARLIN_USE_ATOMIC_ADD=1
      # Genesis v7.14+ opt-in patches (fp8 KV path — no P4/P65/P67 needed)
      - GENESIS_ENABLE_P64_QWEN3CODER_MTP_STREAMING=1
-      - GENESIS_ENABLE_P68_AUTO_FORCE_TOOL=1
-      - GENESIS_ENABLE_P69_LONG_CTX_TOOL_REMINDER=1
+      # P68/P69 deliberately DISABLED (2026-04-29). They override user
+      # tool_choice / inject "must use a tool" reminders when the prompt
+      # exceeds 8000 chars — IDE-agent system prompts blow past that on
+      # the first message and get silent stops on greetings / clarifying
+      # questions. Bisected on club-3090 issue #2 (HoodOG1 + tenitram).
+      # Set to 1 only if you specifically want forced-tool-call behavior.
+      # - GENESIS_ENABLE_P68_AUTO_FORCE_TOOL=1
+      # - GENESIS_ENABLE_P69_LONG_CTX_TOOL_REMINDER=1
      # PN8 (v7.62.x): MTP draft online-quant propagation. Targets fp8+MTP
      # path specifically — frees ~600 MiB by inheriting target quant config
      # into the draft head. Sandermage's claim: closes Cliff 1 on this path.