nvfp4 composes: document the unsloth (compressed-tensors) provider variant

--quantization modelopt matches NVIDIA's export only; unsloth's NVFP4
declares compressed-tensors and errors against the flag (first Discord
report 2026-07-12). Inline note: delete the flag pair for unsloth,
auto-detect takes over; provider A/B tied so either checkpoint is fine.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
This commit is contained in:
noonghunna
2026-07-12 01:03:31 +00:00
parent be2ce99223
commit f19139fbef
2 changed files with 10 additions and 0 deletions

View File

@@ -163,6 +163,11 @@ services:
- qwen3.6-27b-nvfp4
# NVIDIA model-card recipe: modelopt quantization (MIXED_PRECISION
# NVFP4 + FP8 — vLLM reads hf_quant_config.json / quantization_config).
# ⚠ This flag matches NVIDIA's export (nvidia/Qwen3.6-27B-NVFP4) ONLY.
# Serving unsloth/Qwen3.6-27B-NVFP4 instead? Its config declares
# compressed-tensors → DELETE the two lines below (vLLM auto-detects;
# keeping them errors: "Quantization method ... (compressed-tensors)
# does not match ... (modelopt)"). Provider A/B 2026-07-11: quality/TPS tie.
- --quantization
- modelopt
- --dtype

View File

@@ -130,6 +130,11 @@ services:
- qwen3.6-27b-autoround
- qwen3.6-27b-nvfp4
# NVIDIA model-card recipe: modelopt quantization.
# ⚠ This flag matches NVIDIA's export (nvidia/Qwen3.6-27B-NVFP4) ONLY.
# Serving unsloth/Qwen3.6-27B-NVFP4 instead? Its config declares
# compressed-tensors → DELETE the two lines below (vLLM auto-detects;
# keeping them errors: "Quantization method ... (compressed-tensors)
# does not match ... (modelopt)"). Provider A/B 2026-07-11: quality/TPS tie.
- --quantization
- modelopt
- --dtype