nvfp4 composes: document the unsloth (compressed-tensors) provider variant
--quantization modelopt matches NVIDIA's export only; unsloth's NVFP4 declares compressed-tensors and errors against the flag (first Discord report 2026-07-12). Inline note: delete the flag pair for unsloth, auto-detect takes over; provider A/B tied so either checkpoint is fine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
This commit is contained in:
@@ -163,6 +163,11 @@ services:
|
||||
- qwen3.6-27b-nvfp4
|
||||
# NVIDIA model-card recipe: modelopt quantization (MIXED_PRECISION
|
||||
# NVFP4 + FP8 — vLLM reads hf_quant_config.json / quantization_config).
|
||||
# ⚠ This flag matches NVIDIA's export (nvidia/Qwen3.6-27B-NVFP4) ONLY.
|
||||
# Serving unsloth/Qwen3.6-27B-NVFP4 instead? Its config declares
|
||||
# compressed-tensors → DELETE the two lines below (vLLM auto-detects;
|
||||
# keeping them errors: "Quantization method ... (compressed-tensors)
|
||||
# does not match ... (modelopt)"). Provider A/B 2026-07-11: quality/TPS tie.
|
||||
- --quantization
|
||||
- modelopt
|
||||
- --dtype
|
||||
|
||||
@@ -130,6 +130,11 @@ services:
|
||||
- qwen3.6-27b-autoround
|
||||
- qwen3.6-27b-nvfp4
|
||||
# NVIDIA model-card recipe: modelopt quantization.
|
||||
# ⚠ This flag matches NVIDIA's export (nvidia/Qwen3.6-27B-NVFP4) ONLY.
|
||||
# Serving unsloth/Qwen3.6-27B-NVFP4 instead? Its config declares
|
||||
# compressed-tensors → DELETE the two lines below (vLLM auto-detects;
|
||||
# keeping them errors: "Quantization method ... (compressed-tensors)
|
||||
# does not match ... (modelopt)"). Provider A/B 2026-07-11: quality/TPS tie.
|
||||
- --quantization
|
||||
- modelopt
|
||||
- --dtype
|
||||
|
||||
Reference in New Issue
Block a user