FLy Verification

FLy is an approximate verification policy that can defer a native rejection at a high-entropy position when the following draft tokens remain aligned with native verification.

Usage

from vllm import LLM

llm = LLM(
    model="Qwen/Qwen3-8B",
    speculative_config={
        "method": "draft_model",
        "model": "Qwen/Qwen3-0.6B",
        "num_speculative_tokens": 8,
        "rejection_sample_method": "fly",
        "fly_window_size": 6,
        "fly_entropy_threshold": 0.3,
        "fly_entropy_top_k": 3,
    },
)

FLy requires at least two speculative tokens. fly_window_size is the number of subsequent native acceptance decisions checked; it must be smaller than num_speculative_tokens and defaults to min(6, num_speculative_tokens - 1). The entropy gate uses the three largest processed target probabilities by default. Set fly_entropy_top_k in speculative_config to change that number; it must be positive and is capped at the vocabulary size.

FLy supports greedy draft sampling with target-only acceptance and probabilistic draft sampling with standard p/q acceptance. Token-Level Intersection can be used for cross-vocabulary draft models with greedy draft sampling, but is incompatible with use_local_argmax_reduction.

FLy is supported by both model runners. With ModelRunnerV2, use a supported proposer such as MTP, EAGLE, or DFlash; standalone draft_model proposing still requires ModelRunnerV1.

Validated Configurations

The following evaluation covers the V1 draft_model proposer; MRV2 validation is summarized below. Configurations not listed here are not necessarily unsupported, but have not yet been experimentally validated.

FLy was faster than Standard SD in all 40 primary configurations. Across all four target/draft pairs, the 29 configurations with ratio-based quality measurements had a minimum quality retention of 97.9%.

The 10-benchmark suite contains GSM8K, MGSM, MATH-500, AIME 2024, AIME 2025, GPQA-Diamond, HumanEval, MBPP, Spec-Bench, and HLE. Subsets of GSM8K and MATH-500 were also validated with temperature 1 and probabilistic draft sampling.

ModelRunner V2

MRV2 runs used greedy draft sampling, fly_window_size=2, fly_entropy_threshold=0.3, and fly_entropy_top_k=3. Decode throughput was measured with 1,024 output tokens per request at batch sizes 1, 4, 8, 16, 32, and 64, excluding prefill.

Single-pass quality evaluations compared Target-only, Standard SD, and FLy on GSM8K (100 examples), MATH-500 (100), AIME25 (30), and GPQA-Diamond (198). Relative to Target-only, the minimum measured accuracy retention was 96.43% on AMD and 96.30% on NVIDIA; both corresponded to one fewer correct answer on AIME25.