f6613c869a
Six rounds of cross-rig bisect with Codex/ChatGPT diagnosis closes Cliff 2 at 60K on TP=1 + 24GB. Two shippable variants: Balanced MTP (long-text.yml updated): - Genesis v7.69 dev tip (commit 2db18df) - Codex r1 P103 gate fix applied to nested Genesis tree (cu_seqlens=[0,T] treated as dense single-seq, not multi-seq varlen). Sent to Sander as v7.70 proposal — diff in /tmp/cliff2_v770_cu_seqlens_response.md. - vllm#35975 backport sidecar (patch_inputs_embeds_optional.py) - mem-util 0.93 (down from 0.95) - max_model_len 180000 (admission ceiling at this mem-util) - MTP K=3 retained - 60K probe: HTTP 200 in 623s, recall correct, MTP AL=4.00 Max-context safety (long-text-no-mtp.yml — NEW): - Same patches, but MTP off + mem-util 0.95 - max_model_len 200000 (admission unlocked by removing MTP residency) - 60K probe: HTTP 200 in 537s, recall correct - 90K probe: indeterminate within 25-min curl budget - For long single-shot RAG / codebase analysis Diagnostic chain: 1. Codex r1 identified P103 gate as too broad (cu_seqlens != None bypasses chunking even for single-seq [0,T] case). Applied fix. 2. T=4128 distribution showed chunked path never engages on real serving (vLLM's outer chunked-prefill caps T well below MAX_T). 3. Codex r2: real Cliff 2 source is residency, not gate logic. 4. PR #35975 backport (skip inputs_embeds for text-only) frees 444 MiB at boot — necessary but not sufficient at 0.95 mem-util. 5. mem-util sweep at 0.92/0.93 with MTP+#35975 closed Cliff 2 at 60K. 6. MTP-off + 0.95 + 200K admission validated max-context variant. Codex's P103 gate fix is semantically correct and worth shipping in Genesis v7.70 even though it's not what closes 60K Cliff 2 on this config (the FLA call sees T=4128 already, well below MAX_T). Full diagnostic trail: results/v0.20-migration/v769-codex-r1-test.summary Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
349 lines
18 KiB
YAML
349 lines
18 KiB
YAML
# ===========================================================================
|
||
# Long-text variant — 214K context, text-only (no vision).
|
||
#
|
||
# ⚠️ NOT SAFE FOR IDE-AGENT WORKLOADS (Cline / OpenCode / Roo / Claude Code /
|
||
# Cursor / any flow with sys-prompt + tool schemas). Use `tools-text.yml`
|
||
# instead until Genesis PN25 lands default-on. See club-3090#16.
|
||
#
|
||
# Reproducer (2026-05-01 PM): a 5,900-char system prompt + 10 typical tool
|
||
# schemas + 346-char user request crashes the engine with a 98 MiB OOM at
|
||
# `empty_strided_cuda((s, 17408), ...)` in `inductor_cache/...py:1208`.
|
||
# vLLM's torch.compile inductor inlines `SiluAndMul.forward_native` and
|
||
# bypasses Genesis PN12's `FFNIntermediateCache` pool. PN25 (the proper
|
||
# compile-path opaque-op fix) is on Genesis dev but blocked by a worker-
|
||
# fork registration bug — see Sandermage/genesis-vllm-patches#16.
|
||
#
|
||
# Workloads that ARE safe on this compose:
|
||
# - Pure long-form text generation (essay, doc Q&A) — no tool schemas in prompt
|
||
# - RAG without tool calling (just context + question)
|
||
# - Chat without tools
|
||
# Workloads that should use `tools-text.yml` (75K + fp8 KV + PN8) instead:
|
||
# - Any IDE coding agent
|
||
# - Any flow that puts tool/function schemas in the system prompt
|
||
#
|
||
# Stack: vLLM v0.20.1rc1.dev16 + Genesis v7.69 dev (commit 2db18df) +
|
||
# TurboQuant 3-bit KV + MTP n=3 + vision tower disabled.
|
||
#
|
||
# v7.69 key patches active:
|
||
# - P103 (Cliff 2 chunked fwd_h+fwd_o orchestrator) — chunk.py self-install
|
||
# hook survives `exec vllm serve` worker spawn (v7.69 F2 fix). Also
|
||
# applied: Codex r1 cu_seqlens=[0,T] dense-single-seq gate fix that
|
||
# allows the chunked path to engage on real serving.
|
||
# - PN30 v7.68 part1+2+3 — DS conv state layout + spec-decode AL>1 fix
|
||
# (drift-marker bug fixed in v7.69)
|
||
# - PN32 v2 — GDN _forward_core chunked-prefill (Cliff 2 outer split)
|
||
# - PN34 — workspace_lock relaxation (replaces our former
|
||
# patch_workspace_lock_disable.py sidecar)
|
||
# - PN26b: Genesis-original sparse-V Triton kernel for SM86 (Ampere
|
||
# consumer) — first SM86 sparse-V kernel in any public tree. 27B
|
||
# tuning: BLOCK_KV=8 num_warps=4 threshold=0.01.
|
||
# - P38B / P15B: Cliff 1 mech B compile-safe hook + FA varlen clamp
|
||
#
|
||
# Cliff 2 closure recipe (cross-rig validated 2026-05-02 PM):
|
||
# - vllm#35975 backport sidecar (`patch_inputs_embeds_optional.py`) frees
|
||
# ~444 MiB at boot on text-only models (skips inputs_embeds buffers
|
||
# for both gpu_model_runner.py + llm_base_proposer.py).
|
||
# - --gpu-memory-utilization 0.93 — middle ground between 0.92 (cliff
|
||
# just barely closed) and 0.95 (cliff fires). Recovers ~30K of KV
|
||
# pool concurrency vs 0.92 while keeping ~494 MiB margin at peak.
|
||
# - --max-model-len 180000 — engine pre-check admission ceiling on
|
||
# this config (200K rejected: KV pool needs 3.31 GiB, only 2.19 GiB
|
||
# available at 0.93). Cliff 2 single-prompt safe envelope: ~60K
|
||
# verified pass; ~70-80K is the OOM ceiling.
|
||
#
|
||
# Validation (2026-05-02 PM, branch v7.69-cliff2-test):
|
||
# - 60K MTP-on probe: HTTP 200 in 623s, recall correct, MTP AL=4.00
|
||
# - 90K MTP-on probe: HTTP 500 OOM at 112s (50 MiB / 24.5 MiB free)
|
||
# - Probes 2-6 (tool prefill, IDE-agent, multi-turn, LCB-coding,
|
||
# reasoning-heavy): all PASS
|
||
#
|
||
# For workloads that need >60K single-prompt or 200K declared context,
|
||
# use docker-compose.long-text-no-mtp.yml (max-context safety variant
|
||
# trades MTP decode speedup for prefill stability). For coding agents
|
||
# on dual-card, use docker-compose.dual-turbo.yml.
|
||
#
|
||
# Recommended client max_tokens: 8192 (FREE thinking on this compose; budget
|
||
# for hard reasoning runs). 16384 for competition-grade problems. See
|
||
# docs/EXAMPLES.md "max_tokens defaults" section for the full table.
|
||
#
|
||
# When to pick this over long-vision:
|
||
# - You don't need image input
|
||
# - You want the maximum context on a single 3090
|
||
# - You're doing long-context text workloads (codebase analysis, RAG
|
||
# retrievals, multi-turn chat with steady-state accumulation)
|
||
#
|
||
# When NOT to pick this:
|
||
# - Anything that needs vision — use long-vision (198K + vision) instead
|
||
# - Single-shot prompts ≥50–60K tokens — Cliff 2 (DeltaNet GDN) still
|
||
# applies on single-card. Route those to dual.yml (TP=2 splits the
|
||
# state across cards) or llamacpp/default (262K, different engine).
|
||
#
|
||
# Local sidecar (single one remaining on this compose):
|
||
# - patch_workspace_lock_disable.py — relaxes vllm#39226 strict
|
||
# WorkspaceManager.lock() assertion to a one-shot WARNING. P98 in v7.65
|
||
# covers this functionally but its drift marker auto-skips on v0.20
|
||
# (`UNIFORM_SINGLE_TOKEN_DECODE` false-positive). Drop when Sandermage
|
||
# ships the marker fix.
|
||
#
|
||
# Variants in this dir:
|
||
# File Ctx KV Vision Notes
|
||
# docker-compose.yml (default) 48K TQ3 ✅ Production-safe; below both cliffs
|
||
# docker-compose.long-vision.yml 198K TQ3 ✅ Long ctx + vision (cliffs closed by v0.20 + v7.65)
|
||
# docker-compose.long-text.yml (this) 214K TQ3 ❌ Engine-ceiling text-only on v0.20 + 0.985 mem-util
|
||
# docker-compose.bounded-thinking.yml 214K TQ3 ❌ Long-text + structured-CoT grammar in <think>
|
||
# docker-compose.tools-text.yml 75K fp8 ❌ IDE agents — fp8 KV alternative
|
||
#
|
||
# Run with:
|
||
# cd <repo>/models/qwen3.6-27b/vllm/compose
|
||
# docker compose -f docker-compose.long-text.yml up -d
|
||
# ===========================================================================
|
||
services:
|
||
vllm-qwen36-27b-long-text:
|
||
image: vllm/vllm-openai:nightly-7a1eb8ac2ec4ea69338c51dc7afd4b15010abfa8
|
||
container_name: vllm-qwen36-27b-long-text
|
||
restart: "no"
|
||
ports:
|
||
- "${PORT:-8020}:8000"
|
||
volumes:
|
||
- ${MODEL_DIR:-../../../../models-cache}:/root/.cache/huggingface
|
||
# torch.compile + Triton kernel caches — first boot warms (~60-90 sec
|
||
# for Dynamo trace + Inductor compile + Triton JIT); subsequent boots
|
||
# reuse cached graphs and skip recompile. Pattern from Sandermage's PROD
|
||
# launch (start_27b_int4_TQ_k8v4.sh:31-32). Closes club-3090 #22.
|
||
- ../cache/torch_compile:/root/.cache/vllm/torch_compile_cache
|
||
- ../cache/triton:/root/.triton/cache
|
||
- ../patches/genesis/vllm/_genesis:/usr/local/lib/python3.12/dist-packages/vllm/_genesis:ro
|
||
- ../patches/patch_tolist_cudagraph.py:/patches/patch_tolist_cudagraph.py:ro
|
||
# workspace_lock_disable DROPPED on v7.69 — Genesis PN34 covers the
|
||
# boot-time AND runtime _decode_attention paths. Opt-in via
|
||
# GENESIS_ENABLE_PN34_WORKSPACE_LOCK_RELAX=1 in environment block.
|
||
# patch_inputs_embeds_optional.py — backport of vllm#35975 (~64 MiB GPU +
|
||
# 64 MiB pinned CPU savings on text-only models). Drops when PR merges.
|
||
- ../patches/patch_inputs_embeds_optional.py:/patches/patch_inputs_embeds_optional.py:ro
|
||
# NOTE: patch_pn12_ffn_pool_anchor.py + patch_pn12_compile_safe_custom_op.py
|
||
# + patch_fa_max_seqlen_clamp.py all DROPPED — Genesis v7.65 covers their
|
||
# surface natively (PN12 native + PN25 + PN17 + P15B). See migration
|
||
# commit history if you need to resurrect.
|
||
environment:
|
||
# Uncomment the next line to pin to a specific GPU (e.g. GPU 0):
|
||
# - CUDA_VISIBLE_DEVICES=0
|
||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
|
||
- VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||
- NCCL_CUMEM_ENABLE=0
|
||
- NCCL_P2P_DISABLE=1
|
||
# v0.20 made cudagraph memory profiling default-ON (vllm#38284), eats
|
||
# ~0.0055 mem-util ≈ 120 MiB KV pool. Disabling restores ~4K of ctx.
|
||
- VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0
|
||
- VLLM_NO_USAGE_STATS=1
|
||
- PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512
|
||
- VLLM_FLOAT32_MATMUL_PRECISION=high
|
||
- VLLM_USE_FLASHINFER_SAMPLER=1
|
||
- OMP_NUM_THREADS=1
|
||
- CUDA_DEVICE_MAX_CONNECTIONS=8
|
||
- VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
|
||
- VLLM_MARLIN_USE_ATOMIC_ADD=1
|
||
- TRITON_CACHE_DIR=/root/.triton/cache
|
||
# FULL Genesis v7.65 PROD env-var set per Sandermage's
|
||
# bare_metal_27b_int4_TQ_k8v4.sh. Validated 2026-05-01 PM dual-3090:
|
||
# +50% TPS over the prior subset (code 77 → 116 wall_TPS).
|
||
- GENESIS_ENABLE_P4=1
|
||
- GENESIS_ENABLE_P58_ASYNC_PLACEHOLDER_FIX=1
|
||
- GENESIS_ENABLE_P60_GDN_NGRAM_FIX=1
|
||
- GENESIS_ENABLE_P60B_TRITON_KERNEL=1
|
||
- GENESIS_ENABLE_P61_QWEN3_MULTI_TOOL=1
|
||
- GENESIS_ENABLE_P61B_STREAMING_OVERLAP=1
|
||
- GENESIS_ENABLE_P62_STRUCT_OUT_SPEC_TIMING=1
|
||
- GENESIS_ENABLE_P64_QWEN3CODER_MTP_STREAMING=1
|
||
- GENESIS_ENABLE_P65_TURBOQUANT_SPEC_CG_DOWNGRADE=1
|
||
- GENESIS_ENABLE_P66_CUDAGRAPH_SIZE_FILTER=1
|
||
- GENESIS_ENABLE_P67_TQ_MULTI_QUERY_KERNEL=1
|
||
# P68/P69 — v7.65 raised default threshold 8000 → 50000 chars (~12.5K
|
||
# tokens). Now safe with typical IDE-agent system+tool prompts; below
|
||
# the threshold both gates are no-op. Override via the env var if needed.
|
||
- GENESIS_ENABLE_P68_AUTO_FORCE_TOOL=1
|
||
- GENESIS_ENABLE_P69_LONG_CTX_TOOL_REMINDER=1
|
||
- GENESIS_P68_P69_LONG_CTX_THRESHOLD_CHARS=50000
|
||
- GENESIS_ENABLE_P72_PROFILE_RUN_CAP=1
|
||
- GENESIS_PROFILE_RUN_CAP_M=4128
|
||
- GENESIS_ENABLE_P74_CHUNK_CLAMP=1
|
||
- GENESIS_ENABLE_P83=1
|
||
- GENESIS_ENABLE_P85=1
|
||
- GENESIS_ENABLE_P87=1
|
||
- GENESIS_ENABLE_P91=1
|
||
- GENESIS_ENABLE_P94=1
|
||
- GENESIS_ENABLE_P98=1
|
||
- GENESIS_ENABLE_P99=1
|
||
- GENESIS_ENABLE_P100=1
|
||
- GENESIS_ENABLE_P101=1
|
||
# ─── Cliff 2 closure recipe (v7.69, single-24GB-GPU TP=1) ──────────
|
||
# P103 closes the FLA inner h tensor via chunk.py self-install hook
|
||
# (v7.69 F2 fix — survives `exec vllm serve` worker spawn).
|
||
# PN32 v2 closes the outer FLA call via _forward_core direct patch
|
||
# + initial_state threading (v7.69 F3 fix — composes with P103).
|
||
# Both required for Cliff 2 ceiling > 50K on TP=1.
|
||
- GENESIS_ENABLE_P103=1
|
||
- GENESIS_ENABLE_PN32_GDN_CHUNKED_PREFILL=1
|
||
- GENESIS_PN32_GDN_CHUNK_SIZE=8192
|
||
- GENESIS_PN32_GDN_CHUNK_THRESHOLD=16384
|
||
- GENESIS_FLA_FWD_H_MAX_T=16384
|
||
# PN34 (workspace_lock relaxation, opt-in default OFF). Replaces our
|
||
# patch_workspace_lock_disable.py sidecar.
|
||
- GENESIS_ENABLE_PN34_WORKSPACE_LOCK_RELAX=1
|
||
- GENESIS_ENABLE_PN8_MTP_DRAFT_ONLINE_QUANT=1
|
||
- GENESIS_ENABLE_PN9_INDEPENDENT_DRAFTER_ATTN=1
|
||
- GENESIS_ENABLE_PN11_GDN_AB_CONTIGUOUS=1
|
||
- GENESIS_ENABLE_PN12_FFN_INTERMEDIATE_POOL=1
|
||
- GENESIS_ENABLE_PN13_CUDA_GRAPH_LAMBDA_ARITY=1
|
||
- GENESIS_ENABLE_PN14_TQ_DECODE_OOB_CLAMP=1
|
||
- GENESIS_ENABLE_PN17_FA2_LSE_CLAMP=1
|
||
- GENESIS_ENABLE_PN25_SILU_INDUCTOR_SAFE=1
|
||
# PN30 — RE-ENABLED with our local dst-shaped temp fix
|
||
# (`patch_pn30_dst_shaped_temp_fix.py`, applied during setup.sh). The
|
||
# original Sander a9977d8 .contiguous() approach corrupted DS row strides
|
||
# by raw-memcpying a compact 10240×5 buffer into a 10240×6 destination
|
||
# block (row 1+ landed at the wrong destination offset, causing later
|
||
# TQ store CUDA assert). Our fix builds a destination-shaped temp inside
|
||
# `collect_mamba_copy_meta` and reuses PN30's temp-list lifetime handling.
|
||
# Diagnosis credit: ChatGPT/Codex CLI cross-check 2026-05-01 PM.
|
||
- GENESIS_ENABLE_PN30_DS_LAYOUT_SPEC_DECODE=1
|
||
# PN31 — DISABLED on 1×3090. Per-shape persistent VRAM grows as new
|
||
# shapes are seen during prefill; on TP=1 + 24GB, the residence cost
|
||
# outpaces the malloc-pressure relief. Sander explicitly warned in
|
||
# `753344b` he couldn't validate on 24GB; cross-rig validated 2026-05-01
|
||
# PM here that 0.95→0.93 mem-util can't carry both PN31 + DeltaNet
|
||
# `chunk_fwd_o` activation peak at 29K depth. Reported back on
|
||
# Sandermage/genesis-vllm-patches#15. Use tools-text.yml fp8 path for
|
||
# 25K+ tool-RETURN workloads in the meantime.
|
||
# - GENESIS_ENABLE_PN31_FA_VARLEN_PERSISTENT_OUT=1
|
||
- GENESIS_ENABLE_PN19_SCOPED_MAX_SPLIT=1
|
||
- GENESIS_ENABLE_PN22_LOCAL_ARGMAX_TP=1
|
||
# PN26b sparse-V Triton kernel — first SM86 (Ampere consumer) sparse-V
|
||
# kernel in any public tree. 27B-specific tuning per Sandermage's PROD.
|
||
- GENESIS_ENABLE_PN26_SPARSE_V=1
|
||
- GENESIS_PN26_SPARSE_V_BLOCK_KV=8
|
||
- GENESIS_PN26_SPARSE_V_NUM_WARPS=4
|
||
- GENESIS_PN26_SPARSE_V_THRESHOLD=0.01
|
||
# v7.65 #14 + #15 fixes (compile-safe in-source hook + FA varlen clamp).
|
||
# Replace our local patch_pn12_compile_safe_custom_op.py + P104 sidecars.
|
||
- GENESIS_ENABLE_P38B_COMPILE_SAFE=1
|
||
- GENESIS_ENABLE_P15B_FA_VARLEN_CLAMP=1
|
||
- GENESIS_PREALLOC_TOKEN_BUDGET=4128
|
||
- GENESIS_BUFFER_MODE=shared
|
||
# Explicit OFFs to match Sandermage's PROD env-var set verbatim:
|
||
# P78 (P78_TOLIST_CAPTURE_GUARD) — superseded by our patch_tolist_cudagraph.py
|
||
# P81 (FP8 block-scaled M<=8) — FP8-specific, no-op on our TQ3 path
|
||
# P82 — biased on small-batch single-stream Lorbus INT4 + MTP K=3 (Sander PROD)
|
||
- GENESIS_ENABLE_P78_TOLIST_CAPTURE_GUARD=0
|
||
- GENESIS_ENABLE_P81_FP8_BLOCK_SCALED_M_LE_8=0
|
||
- GENESIS_ENABLE_P82=0
|
||
- GENESIS_P82_THRESHOLD_SINGLE=0.3
|
||
# P82 stays OFF — biased on small-batch single-stream Lorbus INT4 + MTP K=3
|
||
# per Sandermage's PROD memory feedback_p82_*. P78 stays OFF (deprecated).
|
||
# vLLM env knobs from Sandermage's launch:
|
||
# VLLM_SSM_CONV_STATE_LAYOUT=DS — RE-ENABLED with our local PN30 fix
|
||
# (`patch_pn30_dst_shaped_temp_fix.py`, applied during setup.sh).
|
||
# Without our fix, Sander's PN30 a9977d8 corrupts DS row strides on
|
||
# spec-decode AL>1 paths. With our fix, PN30 builds a destination-shaped
|
||
# temp instead of a compact one, preserving DS layout. +6% TPS retained.
|
||
- VLLM_SSM_CONV_STATE_LAYOUT=DS
|
||
- VLLM_USE_FUSED_MOE_GROUPED_TOPK=1
|
||
shm_size: "16gb"
|
||
ipc: host
|
||
deploy:
|
||
resources:
|
||
reservations:
|
||
devices:
|
||
- driver: nvidia
|
||
count: all
|
||
capabilities: [gpu]
|
||
entrypoint:
|
||
- /bin/bash
|
||
- -c
|
||
- |
|
||
set -e
|
||
pip install xxhash pandas scipy -q
|
||
# Pre-flight: Genesis patches must be populated. Empty volume mount
|
||
# = silent no-op apply_all = boot fails later with cryptic upstream
|
||
# error (e.g. "TurboQuant KV not supported for hybrid models", #13).
|
||
if [ ! -f /usr/local/lib/python3.12/dist-packages/vllm/_genesis/patches/apply_all.py ]; then
|
||
echo "ERROR: Genesis patches missing — host volume models/qwen3.6-27b/vllm/patches/genesis/ is empty." >&2
|
||
echo " Run from repo root before 'docker compose up':" >&2
|
||
echo " bash scripts/setup.sh qwen3.6-27b" >&2
|
||
exit 1
|
||
fi
|
||
python3 -m vllm._genesis.patches.apply_all
|
||
# patch_tolist_cudagraph.py — still needed pending upstream
|
||
# vllm#40807 .tolist() capture-stream guard fix.
|
||
python3 /patches/patch_tolist_cudagraph.py
|
||
# patch_inputs_embeds_optional.py — backport of vllm#35975
|
||
# (free ~64 MiB GPU + ~64 MiB pinned CPU on text-only models).
|
||
# Drops when PR merges upstream.
|
||
python3 /patches/patch_inputs_embeds_optional.py
|
||
exec vllm serve "$@"
|
||
- --
|
||
command:
|
||
- --model
|
||
- /root/.cache/huggingface/qwen3.6-27b-autoround-int4
|
||
- --served-model-name
|
||
- qwen3.6-27b-autoround
|
||
- --quantization
|
||
- auto_round
|
||
- --dtype
|
||
- float16
|
||
- --tensor-parallel-size
|
||
- "1"
|
||
# 180K + 0.97 — backed off from 214K + 0.985 on 2026-05-01 PM after
|
||
# verify-stress probe 1 (10K-token long-context needle) crashed with
|
||
# GDN forward OOM (`fla.ops.chunk.chunk_gated_delta_rule_fwd_h`
|
||
# `k.new_empty(B, NT, H, V, K)` requested 98 MiB, 46 MiB free). At
|
||
# 0.985 mem-util, activation budget was ~46 MiB at 10K prefill —
|
||
# tight enough that ANY single prompt over a few thousand tokens was
|
||
# a coin flip. The "Cliff 2 fires only at 50-60K" framing was
|
||
# config-specific to the dev205 + 0.92 default; at tighter mem-util
|
||
# the GDN forward buffer outpaces available activation much earlier.
|
||
#
|
||
# New target (validated 2026-05-01 PM via verify-stress.sh):
|
||
# 180K + 0.97 → ~6.4 GB after weights, ~5.9 GB KV pool, ~500 MiB
|
||
# activation safety. Probe 1 (10K needle) and probe 6 (math reasoning
|
||
# at max_tokens=8192) PASS. Safe single-prompt envelope ~30K; full
|
||
# 180K context for steady-state turn accumulation only.
|
||
#
|
||
# IDE-agent workloads (sys + tool schemas) STILL crash here per
|
||
# club-3090#16 — different bug (Cliff 1 mech B inductor leak), not
|
||
# fixed by mem-util tuning. Use tools-text.yml for those.
|
||
- --max-model-len
|
||
- "180000"
|
||
# 0.95 — backed off from 0.97 on 2026-05-01 PM after PN25 v3 closed
|
||
# Cliff 1 mech B at the FFN intermediate buffer (which now sits in
|
||
# PN12+PN25's resident FFNIntermediateCache pool). Pool residence
|
||
# adds ~140 MiB persistent VRAM, which tightened activation budget
|
||
# at DeltaNet `chunk_fwd_o` (`o = torch.empty_like(v)`, 50 MiB) —
|
||
# 0.97 left only 26 MiB free, OOM at 9.8K probe. 0.95 frees ~480 MiB
|
||
# activation budget, validated up to 30K probes.
|
||
- --gpu-memory-utilization
|
||
- "0.93"
|
||
- --max-num-seqs
|
||
- "1"
|
||
- --max-num-batched-tokens
|
||
- "4128"
|
||
- --kv-cache-dtype
|
||
- turboquant_3bit_nc
|
||
# Drops the vision tower → frees ~1 GB VRAM → lifts engine ceiling from
|
||
# 198K (long-vision at 0.98) to 218K (this variant at 0.985). This is the only difference vs long-vision.
|
||
- --language-model-only
|
||
- --trust-remote-code
|
||
- --reasoning-parser
|
||
- qwen3
|
||
- --enable-auto-tool-choice
|
||
- --tool-call-parser
|
||
- qwen3_coder
|
||
- --enable-prefix-caching
|
||
- --enable-chunked-prefill
|
||
- --no-scheduler-reserve-full-isl
|
||
- --speculative-config
|
||
- '{"method":"mtp","num_speculative_tokens":3}'
|
||
- --host
|
||
- 0.0.0.0
|
||
- --port
|
||
- "8000"
|