alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7248ccb605caf53b40bc25cc474b94de7be88668
club-3090/scripts/lib/profiles/profile_runtime.yml
T
noonghunnaandClaude Opus 4.8 e2efb757a9 Repoint vllm/gemma-26ba4b-single to INT8-PTH long-ctx (#465) 
Repoint the single-card Gemma-4-26B-A4B slug in place from the bf16/16K
AWQ+MTP path to an INT8-per-token-head long-context path. Same weights
(cyankiwi AWQ-4bit, Marlin WNA16 MoE), same external MTP drafter (n=4),
same gemma4 tool-call; only the KV format changes (bf16 -> int8_per_token_head)
via the vendored vLLM PR #40391 hybrid-SWA KV page-size overlay. The dual
(vllm/gemma-26ba4b-dual) is untouched.

INT8-PTH (1 byte/token) lifts the single-card ceiling from bf16 16K to a
176K default. Live-validated on 1x RTX 3090 (2026-06-06): #40391 applies
cleanly, int8_per_token_head KV inits, Marlin WNA16 MoE backend, MTP
SpeculativeConfig active; KV pool 183,357 tok >= 176K; coherent generation
(post cudagraph-warmup) + clean gemma4 tool-call. mem_util 0.94, not 0.96:
0.96 passed the upfront KV check but OOM'd in the drafter's later cudagraph
capture (240 MiB free, needed 256) on a single 24 GB card.

- engine vllm-gemma-stable: + gemma4-swa-moe family, + awq/compressed-tensors
  weight formats
- patches.yml: + gemma-a4b-vllm-pr40391-rebased (model-scoped copy of the
  patch dir; reaches the compose, passes test-patch-attribution)
- compose single/awq/mtp.yml -> single/awq/int8.yml (full rewrite to the
  #40391 overlay + entrypoint + int8 KV; Status: Experimental)
- registry + profile_runtime: engine/kv/max_ctx/mem_util/compose_path/entrypoint
- test-generate-from-profile: repoint the clean-derived-seed fixture off this
  slug (now overlay-carrying) to vllm/gemma-26ba4b-dual + a tp= override

Status stays Experimental; rebench-full + soak are the #464 follow-up. Full
gate suite green (test-submit-bench pre-existing/environmental, fails on
baseline too).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-06 11:30:27 +00:00

819 lines
60 KiB
YAML
Raw Blame History

# Phase A-prime runtime profile data (v0.8.0, club-3090 #359 / PR #147).
#
# WHY THIS FILE EXISTS (storage decision — documented in docs/PATCH_ATTRIBUTION.md):
# The #141 generator (STEP 2/3) needs, per in-scope profile, the whole
# shipped service definition classified into param / governed / constant
# tokens plus the two patch insertion points, and a genesis discriminator.
# It also needs an arch -> model-slug cross-reference and per-model
# trust-remote-code precision. The Phase-A `arch_patches.yml` schema is a
# STRICT closed key-set enforced by scripts/tests/test-patch-attribution.sh
# (`arch_allowed_keys`); adding `model_slugs` directly there would trip the
# "unknown keys" guard, and the test is the contract (RED-LINE: the test is
# not editable in this commit). Therefore the Phase-A-prime fold-ins that
# the brief scopes "onto arch rows" are stored HERE, keyed by the same
# `arch:` string, so arch_patches.yml stays schema-clean and the existing
# test stays green. profile_runtime.yml was chosen over COMPOSE_REGISTRY
# fields because the captured templates are large structured blocks and the
# registry is a thin param-value bridge that the test imports as a Python
# module (keeping it data-thin avoids import-surface churn).
#
# GROUNDING DISCIPLINE: every value below was derived by reading the shipped
# compose at `compose_path` (verbatim) and each model's own config.json on
# this stack. Nothing is guessed. Values that could not be established from an
# artifact are written `unverified` / `none` and listed in the commit report.
schema_version: 1
# --------------------------------------------------------------------------
# Per-profile captured runtime template (in-scope vLLM profiles only).
# genesis_equipped (locked v6 discriminator): true iff the shipped compose
# contains _genesis / GENESIS_PIN / GENESIS_ENABLE OR kv_format starts with
# 'turboquant'. Computed by reading the compose, not by guessing.
# compose_service_template: the WHOLE service definition is the captured
# unit (file at `source`); this block records the token CLASSIFICATION the
# generator applies (param-slot | governed-slot | constant) + the two named
# insertion-point anchors. The `image:` expression is a captured CONSTANT
# (correction #2 — never substituted). `--trust-remote-code` is a
# GOVERNED-slot (correction #1 — never blind-passthrough).
# --------------------------------------------------------------------------
profiles:
vllm/default:
compose_path: models/qwen3.6-27b/vllm/compose/single/autoround-int4/tq3-mtp.yml
genesis_equipped: true
genesis_equipped_evidence: compose contains GENESIS_* / _genesis token
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/single/autoround-int4/tq3-mtp.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8020 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 1 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 48000 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.92 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: turboquant_3bit_nc, value: 'turboquant_3bit_nc' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 1 }
speculative_config: { token: '--speculative-config', source: 'drafter[qwen-mtp-builtin].speculative_config_template' }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/long-text:
compose_path: models/qwen3.6-27b/vllm/compose/single/autoround-int4/long-text.yml
genesis_equipped: true
genesis_equipped_evidence: compose contains GENESIS_* / _genesis token
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/single/autoround-int4/long-text.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8020 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 1 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 180000 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.93 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: turboquant_3bit_nc, value: 'turboquant_3bit_nc' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 1 }
speculative_config: { token: '--speculative-config', source: 'drafter[qwen-mtp-builtin].speculative_config_template' }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/long-text-no-mtp:
compose_path: models/qwen3.6-27b/vllm/compose/single/autoround-int4/long-text-no-mtp.yml
genesis_equipped: true
genesis_equipped_evidence: compose contains GENESIS_* / _genesis token
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/single/autoround-int4/long-text-no-mtp.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8021 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 1 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 200000 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.95 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: turboquant_3bit_nc, value: 'turboquant_3bit_nc' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 1 }
speculative_config: omitted # E.drafter is None -> --speculative-config not emitted
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/long-vision:
compose_path: models/qwen3.6-27b/vllm/compose/single/autoround-int4/long-vision.yml
genesis_equipped: true
genesis_equipped_evidence: compose contains GENESIS_* / _genesis token
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/single/autoround-int4/long-vision.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8020 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 1 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 145000 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.95 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: turboquant_3bit_nc, value: 'turboquant_3bit_nc' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 1 }
speculative_config: { token: '--speculative-config', source: 'drafter[qwen-mtp-builtin].speculative_config_template' }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/bounded-thinking:
compose_path: models/qwen3.6-27b/vllm/compose/single/autoround-int4/bounded-thinking.yml
genesis_equipped: true
genesis_equipped_evidence: compose contains GENESIS_* / _genesis token
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/single/autoround-int4/bounded-thinking.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8020 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 1 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 180000 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.95 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: turboquant_3bit_nc, value: 'turboquant_3bit_nc' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 1 }
speculative_config: { token: '--speculative-config', source: 'drafter[qwen-mtp-builtin].speculative_config_template' }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/tools-text:
compose_path: models/qwen3.6-27b/vllm/compose/single/autoround-int4/tools-text.yml
genesis_equipped: true
genesis_equipped_evidence: compose contains GENESIS_* / _genesis token
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/single/autoround-int4/tools-text.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8020 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 1 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 75000 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.97 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: fp8_e5m2, value: 'fp8_e5m2' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 1 }
speculative_config: { token: '--speculative-config', source: 'drafter[qwen-mtp-builtin].speculative_config_template' }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/minimal:
compose_path: models/qwen3.6-27b/vllm/compose/single/autoround-int4/minimal.yml
genesis_equipped: false
genesis_equipped_evidence: no GENESIS_* token in compose and kv_format is not turboquant*
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/single/autoround-int4/minimal.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8020 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 1 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 32768 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.92 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: fp8_e5m2, value: 'fp8_e5m2' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 1 }
speculative_config: omitted # E.drafter is None -> --speculative-config not emitted
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/dual:
compose_path: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/fp8-mtp.yml
genesis_equipped: false
genesis_equipped_evidence: no GENESIS_* token in compose and kv_format is not turboquant*
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/fp8-mtp.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8010 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 2 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 262144 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.92 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: fp8_e5m2, value: 'fp8_e5m2' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 2 }
speculative_config: { token: '--speculative-config', source: 'drafter[qwen-mtp-builtin].speculative_config_template' }
nvlink_env: { source: E.requires_nvlink, value: false }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/dual-turbo:
compose_path: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/turbo.yml
genesis_equipped: true
genesis_equipped_evidence: compose contains GENESIS_* / _genesis token
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/turbo.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8011 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 2 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 262144 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.85 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: turboquant_3bit_nc, value: 'turboquant_3bit_nc' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 4 }
speculative_config: { token: '--speculative-config', source: 'drafter[qwen-mtp-builtin].speculative_config_template' }
nvlink_env: { source: E.requires_nvlink, value: false }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/dual-dflash:
compose_path: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/dflash.yml
genesis_equipped: false
genesis_equipped_evidence: no GENESIS_* token in compose and kv_format is not turboquant*
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/dflash.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8012 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 2 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 185000 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.95 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: fp16, value: null }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 1 }
speculative_config: { token: '--speculative-config', source: 'drafter[zlab-qwen-dflash].speculative_config_template' }
nvlink_env: { source: E.requires_nvlink, value: false }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/dual-dflash-noviz:
compose_path: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/dflash-noviz.yml
genesis_equipped: false
genesis_equipped_evidence: no GENESIS_* token in compose and kv_format is not turboquant*
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/dflash-noviz.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8013 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 2 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 200000 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.95 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: fp16, value: null }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 1 }
speculative_config: { token: '--speculative-config', source: 'drafter[zlab-qwen-dflash].speculative_config_template' }
nvlink_env: { source: E.requires_nvlink, value: false }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/dual-bf16:
compose_path: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/bf16.yml
genesis_equipped: false
genesis_equipped_evidence: no GENESIS_* token in compose and kv_format is not turboquant*
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/bf16.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8012 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 2 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 200000 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.92 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: bf16, value: null }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 1 }
speculative_config: { token: '--speculative-config', source: 'drafter[qwen-mtp-builtin].speculative_config_template' }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/dual-int8:
compose_path: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/int8.yml
genesis_equipped: false
genesis_equipped_evidence: no GENESIS_* token in compose and kv_format is not turboquant*
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/int8.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8011 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 2 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 262144 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.92 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: int8_per_token_head, value: 'auto+PTH' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 2 }
speculative_config: { token: '--speculative-config', source: 'drafter[qwen-mtp-builtin].speculative_config_template' }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/dual-tq3-mtp:
compose_path: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/tq3-mtp.yml
genesis_equipped: true
genesis_equipped_evidence: kv_format 'turboquant_3bit_nc' starts with 'turboquant'
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/tq3-mtp.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8013 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 2 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 262144 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.92 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: turboquant_3bit_nc, value: 'turboquant_3bit_nc' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 2 }
speculative_config: { token: '--speculative-config', source: 'drafter[qwen-mtp-builtin].speculative_config_template' }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/dual-tq3-mtp-genesis:
compose_path: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/tq3-mtp-genesis.yml
genesis_equipped: true
genesis_equipped_evidence: compose contains GENESIS_* / _genesis token
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/tq3-mtp-genesis.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8015 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 2 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 262144 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.85 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: turboquant_3bit_nc, value: 'turboquant_3bit_nc' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 2 }
speculative_config: { token: '--speculative-config', source: 'drafter[qwen-mtp-builtin].speculative_config_template' }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/dual-tq3-nomtp:
compose_path: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/tq3-nomtp.yml
genesis_equipped: true
genesis_equipped_evidence: kv_format 'turboquant_3bit_nc' starts with 'turboquant'
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/tq3-nomtp.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8014 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 2 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 262144 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.92 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: turboquant_3bit_nc, value: 'turboquant_3bit_nc' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 2 }
speculative_config: omitted # E.drafter is None -> --speculative-config not emitted
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/dual-carnice-bf16mtp:
compose_path: models/qwen3.6-27b/vllm/compose/dual/carnice-bf16mtp/bf16-mtp.yml
genesis_equipped: false
genesis_equipped_evidence: no GENESIS_* token in compose and kv_format is not turboquant*
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/dual/carnice-bf16mtp/bf16-mtp.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8070 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 2 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 262144 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.92 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: fp8_e5m2, value: 'fp8_e5m2' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 2 }
speculative_config: { token: '--speculative-config', source: 'drafter[qwen-mtp-builtin].speculative_config_template' }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/dual-qwopus-bf16mtp:
compose_path: models/qwen3.6-27b/vllm/compose/dual/qwopus-bf16mtp/bf16-mtp.yml
genesis_equipped: false
genesis_equipped_evidence: no GENESIS_* token in compose and kv_format is not turboquant*
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/dual/qwopus-bf16mtp/bf16-mtp.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8071 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 2 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 262144 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.92 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: fp8_e5m2, value: 'fp8_e5m2' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 2 }
speculative_config: { token: '--speculative-config', source: 'drafter[qwen-mtp-builtin].speculative_config_template' }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/dual4:
compose_path: models/qwen3.6-27b/vllm/compose/multi4/autoround-int4/fp8-mtp.yml
genesis_equipped: false
genesis_equipped_evidence: no GENESIS_* token in compose and kv_format is not turboquant*
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/multi4/autoround-int4/fp8-mtp.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8015 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 4 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 262144 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.92 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: fp8_e5m2, value: 'fp8_e5m2' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 4 }
speculative_config: { token: '--speculative-config', source: 'drafter[qwen-mtp-builtin].speculative_config_template' }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/dual4-dflash:
compose_path: models/qwen3.6-27b/vllm/compose/multi4/autoround-int4/dflash.yml
genesis_equipped: false
genesis_equipped_evidence: no GENESIS_* token in compose and kv_format is not turboquant*
compose_service_template:
source: models/qwen3.6-27b/vllm/compose/multi4/autoround-int4/dflash.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8016 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 4 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 262144 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.95 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: fp16, value: null }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 2 }
speculative_config: { token: '--speculative-config', source: 'drafter[zlab-qwen-dflash].speculative_config_template' }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/gemma-mtp-tp1:
compose_path: models/gemma-4-31b/vllm/compose/single/autoround-int4/fp8-mtp.yml
genesis_equipped: false
genesis_equipped_evidence: no GENESIS_* token in compose and kv_format is not turboquant*
compose_service_template:
source: models/gemma-4-31b/vllm/compose/single/autoround-int4/fp8-mtp.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8031 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 1 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 8192 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.95 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: fp8_e4m3, value: 'fp8' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 256 }
speculative_config: { token: '--speculative-config', source: 'drafter[gemma-it-assistant].speculative_config_template' }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:v0.21.0}' # captured constant; Gemma 31B vLLM survivors pin immutable v0.21.0
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: absent # sidecar invoke anchor
vllm/gemma-bf16-mtp:
compose_path: models/gemma-4-31b/vllm/compose/dual/autoround-int4/bf16-mtp.yml
genesis_equipped: false
genesis_equipped_evidence: no GENESIS_* token in compose and kv_format is not turboquant*
compose_service_template:
source: models/gemma-4-31b/vllm/compose/dual/autoround-int4/bf16-mtp.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8030 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 2 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 32768 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.92 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: bf16, value: null }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 4 }
speculative_config: { token: '--speculative-config', source: 'drafter[gemma-it-assistant].speculative_config_template' }
nvlink_env: { source: E.requires_nvlink, value: false }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:v0.21.0}' # captured constant; Gemma 31B vLLM survivors pin immutable v0.21.0
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/gemma-int8-mtp:
compose_path: models/gemma-4-31b/vllm/compose/dual/autoround-int4/int8.yml
genesis_equipped: false
genesis_equipped_evidence: no GENESIS_* token in compose and kv_format is not turboquant*
compose_service_template:
source: models/gemma-4-31b/vllm/compose/dual/autoround-int4/int8.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8032 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 2 }
max_model_len: { token: '${MAX_MODEL_LEN:-${CTX}}', source: E.max_ctx, value: 98304 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.95 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: int8_per_token_head, value: 'auto+PTH' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 4 }
speculative_config: { token: '--speculative-config', source: 'drafter[gemma-it-assistant].speculative_config_template' }
nvlink_env: { source: E.requires_nvlink, value: false }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:v0.21.0}' # captured constant; Gemma 31B vLLM survivors pin immutable v0.21.0
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: present # sidecar invoke anchor
vllm/gemma-26ba4b-single:
compose_path: models/gemma-4-26b-a4b/vllm/compose/single/awq/int8.yml
genesis_equipped: false
genesis_equipped_evidence: no GENESIS_* token in compose and kv_format is not turboquant*
compose_service_template:
source: models/gemma-4-26b-a4b/vllm/compose/single/awq/int8.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8040 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 1 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 176000 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.94 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: int8_per_token_head, value: 'auto+PTH' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 256 }
speculative_config: { token: '--speculative-config', source: 'drafter[gemma-26b-it-assistant].speculative_config_template' }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:v0.22.0}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor (PR #40391 mount)
entrypoint: present # sidecar invoke anchor (PR #40391 install.sh entrypoint)
vllm/gemma-26ba4b-dual:
compose_path: models/gemma-4-26b-a4b/vllm/compose/dual/awq/mtp.yml
genesis_equipped: false
genesis_equipped_evidence: no GENESIS_* token in compose and kv_format is not turboquant*
compose_service_template:
source: models/gemma-4-26b-a4b/vllm/compose/dual/awq/mtp.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8041 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 2 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 262144 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.92 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: bf16, value: null }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 256 }
speculative_config: { token: '--speculative-config', source: 'drafter[gemma-26b-it-assistant].speculative_config_template' }
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:v0.22.0}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: absent # sidecar invoke anchor (#40886 install entrypoint dropped — AWQ keys native in v0.22.0)
vllm/qwen-a3b-preview-single:
compose_path: models/qwen3.6-35b-a3b/vllm/compose/single/autoround-int4/preview.yml
genesis_equipped: false
genesis_equipped_evidence: no GENESIS_* token in compose and kv_format is not turboquant*
compose_service_template:
source: models/qwen3.6-35b-a3b/vllm/compose/single/autoround-int4/preview.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8050 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 1 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 8192 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.92 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: fp8_e5m2, value: 'fp8_e5m2' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 1 }
speculative_config: omitted # E.drafter is None -> --speculative-config not emitted
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: absent # sidecar invoke anchor
vllm/qwen-35b-a3b-dual:
compose_path: models/qwen3.6-35b-a3b/vllm/compose/dual/autoround-int4/fp8.yml
genesis_equipped: false
genesis_equipped_evidence: no GENESIS_* token in compose and kv_format is not turboquant*
compose_service_template:
source: models/qwen3.6-35b-a3b/vllm/compose/dual/autoround-int4/fp8.yml # whole service definition captured verbatim from this file
param_slots:
ports: { source: E.default_port, value: 8051 }
tensor_parallel_size: { token: '${TP}', source: E.tp, value: 2 }
max_model_len: { token: '${MAX_MODEL_LEN}', source: E.max_ctx, value: 262144 }
gpu_memory_utilization: { token: '${GPU_MEMORY_UTILIZATION}', source: E.mem_util, value: 0.92 }
kv_cache_dtype: { token: '${KV_CACHE_DTYPE}', source: 'kv_arg(E.kv_format)', kv_format: fp8_e5m2, value: 'fp8_e5m2' }
max_num_seqs: { token: '--max-num-seqs', source: E.max_num_seqs, value: 1 }
speculative_config: omitted # E.drafter is None -> --speculative-config not emitted
governed_slots:
trust_remote_code:
token: '--trust-remote-code'
present_in_shipped_compose: true
policy: governed # locked design v6 §88 security model
emit_rule: 'only if arch trc gate resolves evidence-cited false-permitted; never blind-passthrough'
constants:
image: '${VLLM_IMAGE:-vllm/vllm-openai:v0.22.0}' # captured constant; engine pin flows via env, NOT rewritten
verbatim_from_compose: [shm_size, ipc, deploy.resources, model_cache_volume, base_environment, base_entrypoint, '--model', '--served-model-name', '--quantization', '--dtype', '--tool-call-parser', '--chat-template', '--reasoning-parser', '--default-chat-template-kwargs', '--enable-prefix-caching', '--enable-chunked-prefill', '--enforce-eager']
insertion_points:
volumes: present # overlay/sidecar mount anchor
entrypoint: absent # sidecar invoke anchor
# --------------------------------------------------------------------------
# Phase A-prime fold-in: arch -> model_slugs + trust-remote-code precision.
# Keyed by the same `arch:` string used in arch_patches.yml. `model_slugs`
# are the compose_registry `model` values served by that arch on this
# stack. `trust_remote_code` is derived from each listed model's OWN
# config.json on /mnt/models/huggingface (auto_map absent + no *modeling*.py
# modeling file shipped with the weights => false, evidence-cited). Generic
# common-transformers lower-bound rows have no on-stack weights to inspect
# => model_slugs [] and trust_remote_code stays unverified (honest: nothing
# to cite). arch_patches.yml is the C0 authority for requires_trust_remote_code;
# this block is the evidence ledger the generator's step-5 trc gate reads to
# decide the governed-slot emission. The two MUST agree for in-scope arches.
# --------------------------------------------------------------------------
arch_model_xref:
Qwen3NextForCausalLM:
family: qwen3-next-hybrid
model_slugs: [qwen3.6-27b]
trust_remote_code: false
trust_remote_code_evidence: "qwen3.6-27b weights config.json (qwen3.6-27b-autoround-int4) has architectures=[Qwen3_5ForConditionalGeneration], auto_map absent, no *modeling*.py modeling file shipped with weights -> loads built-in transformers/vLLM class, no remote code execution"
golden_triple_profiles: [vllm/minimal, vllm/dual]
Qwen3_5MoeForConditionalGeneration:
family: qwen3-next-moe
model_slugs: [qwen3.6-35b-a3b]
trust_remote_code: false
trust_remote_code_evidence: "qwen3.6-35b-a3b weights config.json (qwen3.6-35b-a3b-autoround-int4) has architectures=[Qwen3_5MoeForConditionalGeneration], auto_map absent, no *modeling*.py shipped -> built-in class"
Gemma4ForConditionalGeneration:
family: gemma4-swa-dense
model_slugs: [gemma-4-31b]
trust_remote_code: false
trust_remote_code_evidence: "gemma-4-31b weights config.json (gemma-4-31b-autoround-int4) has architectures=[Gemma4ForConditionalGeneration], auto_map absent, no *modeling*.py shipped -> built-in class"
golden_triple_profiles: [vllm/gemma-bf16-mtp, vllm/gemma-int8-mtp, vllm/gemma-mtp-tp1]
Gemma4ForCausalLM:
family: gemma4-swa-dense
model_slugs: []
trust_remote_code: unverified
trust_remote_code_evidence: none # no on-stack model uses this exact text-only arch string; brief requires deriving from the model's OWN config -> nothing to cite
Gemma4ForConditionalGenerationMoE:
family: gemma4-swa-moe
model_slugs: [gemma-4-26b-a4b]
trust_remote_code: false
trust_remote_code_evidence: "gemma-4-26b-a4b weights config.json (gemma-4-26b-a4b-autoround-int4-mixed AND gemma-4-26b-a4b-awq-4bit) both have architectures=[Gemma4ForConditionalGeneration], auto_map absent, no *modeling*.py shipped -> built-in class"
LlamaForCausalLM:
family: dense
model_slugs: []
trust_remote_code: unverified
trust_remote_code_evidence: none # generic common-transformers lower-bound row, no on-stack weights to inspect
MistralForCausalLM:
family: dense
model_slugs: []
trust_remote_code: unverified
trust_remote_code_evidence: none
Qwen2ForCausalLM:
family: dense
model_slugs: []
trust_remote_code: unverified
trust_remote_code_evidence: none
Qwen3ForCausalLM:
family: dense
model_slugs: []
trust_remote_code: unverified
trust_remote_code_evidence: none
Phi3ForCausalLM:
family: dense
model_slugs: []
trust_remote_code: unverified
trust_remote_code_evidence: none
Gemma2ForCausalLM:
family: dense
model_slugs: []
trust_remote_code: unverified
trust_remote_code_evidence: none
Reference in New Issue View Git Blame Copy Permalink