Skip to main content

GGUF

GGUF

Warning

Please note that GGUF support in vLLM is highly experimental and under-optimized at the moment, it might be incompatible with other features. Currently, you can use GGUF as a way to reduce memory footprint. If you encounter any issues, please report them to the vLLM team.

Note

GGUF support has migrated to OOT vllm-gguf-plugin. Make sure you have GGUF plugin installed before serving a GGUF model.

Before serving a GGUF model, make sure to install the vllm-gguf-plugin:

uv pip install vllm-gguf-plugin

To run a GGUF model with vLLM, you can use the repo_id:quant_type format to load directly from HuggingFace. For example, to load a Q4_K_M quantized model from unsloth/Qwen3-0.6B-GGUF:

# We recommend using the tokenizer from base model to avoid long-time and buggy tokenizer conversion.
vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M --tokenizer Qwen/Qwen3-0.6B

You can also add --tensor-parallel-size 2 to enable tensor parallelism inference with 2 GPUs:

vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M \
   --tokenizer Qwen/Qwen3-0.6B \
   --tensor-parallel-size 2

Alternatively, you can download and use a local GGUF file:

wget https://huggingface.co/unsloth/Qwen3-0.6B-GGUF/resolve/main/Qwen3-0.6B-Q4_K_M.gguf
vllm serve ./Qwen3-0.6B-Q4_K_M.gguf --tokenizer Qwen/Qwen3-0.6B

Warning

We recommend using the tokenizer from base model instead of GGUF model. Because the tokenizer conversion from GGUF is time-consuming and unstable, especially for some models with large vocab size.

GGUF assumes that HuggingFace can convert the metadata to a config file. In case HuggingFace doesn't support your model you can manually create a config and pass it as hf-config-path

# If your model is not supported by HuggingFace you can manually provide a HuggingFace compatible config path
vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M \
   --tokenizer Qwen/Qwen3-0.6B \
   --hf-config-path Qwen/Qwen3-0.6B

You can also use the GGUF model directly through the LLM entrypoint:

Code

Edit on GitHub