Skip to main content

Draft Models

Draft Models

The following code configures vLLM in an offline mode to use speculative decoding with a draft model, speculating 5 tokens at a time.

from vllm import LLM, SamplingParams

prompts = ["The future of AI is"]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(
    model="Qwen/Qwen3-8B",
    tensor_parallel_size=1,
    speculative_config={
        "model": "Qwen/Qwen3-0.6B",
        "num_speculative_tokens": 5,
        "method": "draft_model",
    },
)
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

To perform the equivalent launch in online mode, use the following server-side code:

vllm serve Qwen/Qwen3-4B-Thinking-2507 \
    --host 0.0.0.0 \
    --port 8000 \
    --seed 42 \
    -tp 1 \
    --max-model-len 2048 \
    --gpu-memory-utilization 0.8 \
    --speculative-config '{"model": "Qwen/Qwen3-0.6B", "num_speculative_tokens": 5, "method": "draft_model"}'

The code used to request completions as a client remains unchanged:

Draft Model Method with heterogeneous vocabs

By default, vLLM requires the draft and target models to share the same vocabulary. Setting use_heterogeneous_vocab: true enables the Token-Level Intersection (TLI) algorithm, which allows draft models from a different model family with a different tokenizer.

Currently, use_heterogeneous_vocab requires draft_sample_method='greedy' (the default). Probabilistic draft sampling is not yet supported and will be added in a future release.

from vllm import LLM, SamplingParams

llm = LLM(
    model="Qwen/Qwen3-8B",
    speculative_config={                               
        "method": "draft_model",
        "model": "HuggingFaceTB/SmolLM2-135M-Instruct",
        "num_speculative_tokens": 3,
        "use_heterogeneous_vocab": True,
    },
    gpu_memory_utilization=0.5,
)
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Warning

Note: Please use --speculative-config to set all configurations related to speculative decoding. The previous method of specifying the model through --speculative-model and adding related parameters such as --num-speculative-tokens separately has been deprecated. For supported keys and examples, see the --speculative-config schema.

Edit on GitHub