Skip to main content

CPU

还没翻译译自英文（docs.vllm.ai）本页 107 段都还没译

CPU

vLLM is a Python library that supports the following CPU variants. Select your CPU type to see vendor specific instructions:

Intel/AMD x86ARM AArch64Apple siliconIBM Z (S390X)

vLLM supports basic model inferencing and serving on x86 CPU platform, with data types FP32, FP16 and BF16.

Technical Discussions

The main discussions happen in the #sig-cpu channel of vLLM Slack.

When open a Github issue about the CPU backend, please add [CPU Backend] in the title and it will be labeled with cpu for better awareness.

Requirements

  • Python: 3.11 -- 3.14

Intel/AMD x86ARM AArch64Apple siliconIBM Z (S390X)

  • OS: Linux
  • CPU flags: avx512f (Recommended), avx2 (Limited features)

Tip

Use lscpu to check the CPU flags.

Set up using Python

Create a new Python environment

It's recommended to use uv, a very fast Python environment manager, to create and manage Python environments. Please follow the documentation to install uv. After installing uv, you can create a new Python environment using the following commands:

uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate

Pre-built wheels

When specifying the index URL, please make sure to use the cpu variant subdirectory. For example, the nightly build index is: https://wheels.vllm.ai/nightly/cpu/.

Intel/AMD x86ARM AArch64Apple siliconIBM Z (S390X)

Pre-built vLLM wheels for x86 with AVX512/AVX2 are available since version 0.17.0. To install release wheels:

export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//')

# use uv
uv pip install "https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cpu-cp38-abi3-manylinux_2_34_x86_64.whl" --torch-backend cpu

pip

set LD_PRELOAD

Before use vLLM CPU installed via wheels, make sure Intel OpenMP is added to LD_PRELOAD:

# manually find the path
sudo find / -iname *libiomp5.so
IOMP_PATH=...

# add it to LD_PRELOAD
export LD_PRELOAD="$IOMP_PATH:$LD_PRELOAD"

Install the latest code

To install the wheel built from the latest main branch:

uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu --index-strategy first-index --torch-backend cpu

Install specific revisions

If you want to access the wheels for previous commits (e.g. to bisect the behavior change, performance regression), you can specify the commit hash in the URL:

export VLLM_COMMIT=730bd35378bf2a5b56b6d3a45be28b3092d26519 # use full commit hash from the main branch
uv pip install vllm --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT}/cpu --index-strategy first-index --torch-backend cpu

Build wheel from source

Set up using Python-only build (without compilation)

This method requires pre-built wheels for your platform.

Please refer to the instructions for Python-only build on GPU, and replace the build commands with:

VLLM_USE_PRECOMPILED=1 VLLM_PRECOMPILED_WHEEL_VARIANT=cpu VLLM_TARGET_DEVICE=cpu uv pip install --editable .

Full build (with compilation)

Intel/AMD x86ARM AArch64Apple siliconIBM Z (s390x)

Install recommended compiler. We recommend to use gcc/g++ >= 12.3.0 as the default compiler to avoid potential problems. For example, on Ubuntu 22.4, you can run:

sudo apt-get update -y
sudo apt-get install -y gcc-12 g++-12 libnuma-dev
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10 --slave /usr/bin/g++ g++ /usr/bin/g++-12

It's recommended to use uv, a very fast Python environment manager, to create and manage Python environments. Please follow the documentation to install uv. After installing uv, you can create a new Python environment using the following commands:

uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate

Clone the vLLM project:

git clone https://github.com/vllm-project/vllm.git vllm_source
cd vllm_source

Install the required dependencies:

uv pip install -r requirements/build/cpu.txt --torch-backend cpu --index-strategy unsafe-best-match
uv pip install -r requirements/cpu.txt --torch-backend cpu --index-strategy unsafe-best-match

pip

Build and install vLLM:

VLLM_TARGET_DEVICE=cpu uv pip install . --no-build-isolation

If you want to develop vLLM, install it in editable mode instead.

VLLM_TARGET_DEVICE=cpu python3 setup.py develop

Optionally, build a portable wheel which you can then install elsewhere:

VLLM_TARGET_DEVICE=cpu uv build --wheel --no-build-isolation
uv pip install dist/*.whl

pip

set LD_PRELOAD

Before using vLLM CPU installed via wheels, make sure TCMalloc and Intel OpenMP are installed and added to LD_PRELOAD:

# install TCMalloc, Intel OpenMP is installed with vLLM CPU
sudo apt-get install -y --no-install-recommends libtcmalloc-minimal4

# manually find the path
sudo find / -iname *libtcmalloc_minimal.so.4
sudo find / -iname *libiomp5.so
TC_PATH=...
IOMP_PATH=...

# add them to LD_PRELOAD
export LD_PRELOAD="$TC_PATH:$IOMP_PATH:$LD_PRELOAD"

Troubleshooting

  • NumPy ≥2.0 error: Downgrade using pip install "numpy<2.0".
  • CMake picks up CUDA: Add CMAKE_DISABLE_FIND_PACKAGE_CUDA=ON to prevent CUDA detection during CPU builds, even if CUDA is installed.
  • AMD requires at least 4th gen processors (Zen 4/Genoa) or higher to support AVX512 to run vLLM on CPU.
  • If you receive an error such as: Could not find a version that satisfies the requirement torch==X.Y.Z+cpu+cpu, consider updating pyproject.toml to help pip resolve the dependency.
pyproject.toml
[build-system]
requires = [
  "cmake>=3.26.1",
  ...
  "torch==X.Y.Z+cpu"   # <-------
]

Set up using Docker

Pre-built images

Intel/AMD x86ARM AArch64Apple siliconIBM Z (S390X)

You can pull the latest available CPU image from Docker Hub:

docker pull vllm/vllm-openai-cpu:latest-x86_64

To pull an image for a specific vLLM version:

export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//')
docker pull vllm/vllm-openai-cpu:v${VLLM_VERSION}-x86_64

All available image tags are here: https://hub.docker.com/r/vllm/vllm-openai-cpu/tags

You can run these images via:

docker run \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    --env "HF_TOKEN=<secret>" \
    vllm/vllm-openai-cpu:latest-x86_64 <args...>

Serve with vLLM Recipes

The CPU image includes the vLLM Recipes deployment tool. Recipe-assisted serving queries the site at runtime, so the image can use the latest published recipe for the selected model and hardware.

To find the value for --hardware, open vLLM Recipes, select the model, and choose the target in the Hardware picker. The selected page URL also contains the hardware key; for example, ?hardware=xeon6 maps to --hardware xeon6.

Keep the normal image entrypoint for regular serving. For recipe-assisted serving, override it with serve_with_recipe.sh:

docker run --rm \
    --shm-size=4g \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    --env "HF_TOKEN=<secret>" \
    --entrypoint tools/recipes/serve_with_recipe.sh \
    vllm/vllm-openai-cpu:latest-x86_64 \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --hardware xeon6

For Xeon 6, the script enables hardware detection automatically, generates config.yml and env.sh, and starts vllm serve. The image keeps /vllm-workspace as its working directory so relative recipe assets under examples/ continue to resolve.

See the Recipes tool documentation for interactive discovery and additional options.

Build image from source

Intel/AMD x86ARM AArch64Apple siliconIBM Z (S390X)

Building for your target CPU

docker build -f docker/Dockerfile.cpu \
        --build-arg VLLM_CPU_X86=<false (default)|true> \ # For cross-compilation
        --tag vllm-cpu-env \
        --target vllm-openai .

Building with AMD Zen optimizations

For AMD Zen 4 / Zen 5 hosts (linux/amd64 only), use the vllm-openai-zen target. It extends the default vllm-openai image and adds zentorch via the vllm[zen] extra so ZenCpuPlatform auto-activates at runtime:

docker build -f docker/Dockerfile.cpu \
        --tag vllm-cpu-zen-env \
        --target vllm-openai-zen .

The resulting image accepts the same arguments and environment variables as vllm-openai (see Launching the OpenAI server below); no extra flag is needed to engage Zen optimizations. See AMD Zen optimizations for runtime behavior and the supported-dtype caveats.

Launching the OpenAI server

docker run --rm \
            --security-opt seccomp=unconfined \
            --cap-add SYS_NICE \
            --shm-size=4g \
            -p 8000:8000 \
            -e VLLM_CPU_KVCACHE_SPACE=<KV cache space> \
            vllm-cpu-env \
            meta-llama/Llama-3.2-1B-Instruct \
            --dtype=bfloat16 \
            other vLLM OpenAI server arguments

AMD Zen optimizations

On AMD Zen CPUs, vLLM auto-selects ZenCpuPlatform (a subclass of CpuPlatform) which dispatches linear layers through zentorch's ZenDNN-optimized kernels. See the FAQ entry How do I enable AMD Zen optimizations? for the install command.

Detection rules

ZenCpuPlatform is selected when all of the following hold:

  • vLLM is built for CPU
  • /proc/cpuinfo reports AuthenticAMD and avx512
  • import zentorch succeeds

Otherwise, vLLM falls back to the default CpuPlatform (oneDNN / sgl-kernel paths).

Supported dtypes

ZenCpuPlatform.supported_dtypes always advertises bfloat16 and float32, and adds float16 when torch.ops.zentorch.zentorch_is_fp16_supported() reports AVX512-FP16, which AMD ships from Venice (6th Gen EPYC) onwards. On Turin (5th Gen EPYC) and earlier, models declared with torch_dtype=float16 are auto-downcast to bfloat16 at load time with the standard "Your device 'cpu' doesn't support torch.float16. Falling back to torch.bfloat16 for compatibility." warning emitted from vllm/config/model.py.

bfloat16 is listed first either way, so --dtype auto keeps resolving to it.

Environment variables

  • VLLM_ZENTORCH_WEIGHT_PREPACK (default 1): eagerly prepacks linear weights into ZenDNN's blocked layout at model load time, eliminating per-inference layout conversion overhead. Set to 0 to disable.

Docker

The vllm-openai-zen Docker target (in docker/Dockerfile.cpu) extends the default vllm-openai image with vllm[zen]. Build it with docker build -f docker/Dockerfile.cpu --target vllm-openai-zen . — see Building with AMD Zen optimizations for the full command and run instructions.

Reference

For the design rationale, see RFC #35089: In-Tree AMD Zen CPU Backend via zentorch.

Related runtime environment variables

  • VLLM_CPU_KVCACHE_SPACE: specify the KV Cache size (e.g, VLLM_CPU_KVCACHE_SPACE=40 means 40 GiB space for KV cache), larger setting will allow vLLM to run more requests in parallel. This parameter should be set based on the hardware configuration and memory management pattern of users. Default value is 0.
  • VLLM_CPU_OMP_THREADS_BIND: specify the CPU cores dedicated to the OpenMP threads, can be set as CPU id lists, auto (by default), or nobind (to disable binding to individual CPU cores and to inherit user-defined OpenMP variables). For example, VLLM_CPU_OMP_THREADS_BIND=0-31 means there will be 32 OpenMP threads bound on 0-31 CPU cores. VLLM_CPU_OMP_THREADS_BIND=0-31|32-63 means there will be 2 tensor parallel processes, 32 OpenMP threads of rank0 are bound on 0-31 CPU cores, and the OpenMP threads of rank1 are bound on 32-63 CPU cores. By setting to auto, the OpenMP threads of each rank are bound to the CPU cores in each NUMA node respectively. If set to nobind, the number of OpenMP threads is determined by the standard OMP_NUM_THREADS environment variable.
  • VLLM_CPU_NUM_OF_RESERVED_CPU: specify the total number of CPU cores which are not dedicated to OpenMP threads. The variable only takes effect when VLLM_CPU_OMP_THREADS_BIND is set to auto. By default, x86, ARM, and RISC-V reserve local_world_size CPUs in total, while PowerPC and S390X retain a one-CPU default. When KV transfer is enabled, one additional CPU is reserved per local rank. Reserved CPUs are distributed across the local ranks.
  • CPU_VISIBLE_MEMORY_NODES: specify visible NUMA memory nodes for vLLM CPU workers, similar to CUDA_VISIBLE_DEVICES. The variable only takes effect when VLLM_CPU_OMP_THREADS_BIND is set to auto. The variable provides more control for the auto thread-binding feature, such as masking nodes and changing nodes binding sequence.
  • VLLM_ZENTORCH_WEIGHT_PREPACK (AMD Zen only): when ZenCpuPlatform is active, eagerly prepack linear weights into ZenDNN's blocked layout at model load time, eliminating per-inference layout conversion overhead. Default is 1 (enabled). See AMD Zen optimizations.

FAQ

Which dtype should be used?

  • Currently, vLLM CPU uses model default settings as dtype. However, due to unstable float16 support in torch CPU, it is recommended to explicitly set dtype=bfloat16 if there are any performance or accuracy problem.
  • On AMD Zen CPUs (ZenCpuPlatform), float16 requires AVX512-FP16, available from Venice (6th Gen EPYC) onwards. On Turin (5th Gen EPYC) and earlier only bfloat16 and float32 are accepted, and models declared with float16 are auto-downcast to bfloat16 at model load time. See AMD Zen optimizations.

How to launch a vLLM service on CPU?

  • When using the online serving, it is recommended to reserve 1-2 CPU cores for the serving framework to avoid CPU oversubscription. For example, on a platform with 32 physical CPU cores, reserving CPU 31 for the framework and using CPU 0-30 for inference threads:
export VLLM_CPU_KVCACHE_SPACE=40
export VLLM_CPU_OMP_THREADS_BIND=0-30
vllm serve facebook/opt-125m --dtype=bfloat16

or using default auto thread binding:

export VLLM_CPU_KVCACHE_SPACE=40
export VLLM_CPU_NUM_OF_RESERVED_CPU=1
vllm serve facebook/opt-125m --dtype=bfloat16

Note, it is recommended to manually reserve 1 CPU for vLLM front-end process when world_size == 1.

What are supported models on CPU?

For the full and up-to-date list of models validated on CPU platforms, please see the official documentation: Supported Models on CPU

How to find benchmark configuration examples for supported CPU models?

For any model listed under Supported Models on CPU, optimized runtime configurations are provided in the vLLM Benchmark Suite’s CPU test cases, defined in cpu test cases as serving-tests-cpu.json. Full test cases for Text-only models, Multi-Modal models and Embedded models are in cpu Text-Only test cases as serving-tests-cpu-text.json, cpu Multi-Modal test cases as serving-tests-cpu-multimodal.json and cpu Embedded test cases as serving-tests-cpu-embed.json.
For details on how these optimized configurations are determined, see: performance-benchmark-details. To benchmark the supported models using these optimized settings, follow the steps in running vLLM Benchmark Suite manually and run the Benchmark Suite on a CPU environment.

Below is an example command to benchmark all CPU-supported models using optimized configurations.

ON_CPU=1 bash .buildkite/performance-benchmarks/scripts/run-performance-benchmarks.sh

The benchmark results will be saved in ./benchmark/results/. In the directory, the generated .commands files contain all example commands for the benchmark.

We recommend configuring tensor-parallel-size to match the number of NUMA nodes on your system. Note that the current release does not support tensor-parallel-size=6. To determine the number of NUMA nodes available, use the following command:

lscpu | grep "NUMA node(s):" | awk '{print $3}'

For performance reference, users may also consult the vLLM Performance Dashboard , which publishes default-model CPU results produced using the same Benchmark Suite.

Dry-Run

For users only need to get the optimized runtime configurations without running benchmark, a Dry-Run mode is provided. By passing an environment variable DRY_RUN=1 with run-performance-benchmarks.sh, all commands will be generated under ./benchmark/results/.

ON_CPU=1 DRY_RUN=1 bash .buildkite/performance-benchmarks/scripts/run-performance-benchmarks.sh

By providing different JSON file, users can get runtime configurations for different models such as Embedded Models.

ON_CPU=1 SERVING_JSON=serving-tests-cpu-embed.json DRY_RUN=1 bash .buildkite/performance-benchmarks/scripts/run-performance-benchmarks.sh

By providing MODEL_FILTER and DTYPE_FILTER, only commands for related model ID and Data Type will be generated.

ON_CPU=1 SERVING_JSON=serving-tests-cpu-text.json DRY_RUN=1 MODEL_FILTER=meta-llama/Llama-3.1-8B-Instruct DTYPE_FILTER=bfloat16  bash .buildkite/performance-benchmarks/scripts/run-performance-benchmarks.sh

How do I enable AMD Zen optimizations?

On an AMD Zen 4 / Zen 5 CPU, install the CPU wheel with the zen extra so vLLM pulls the tested zentorch version for that release:

export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//')
uv pip install "vllm[zen]" --extra-index-url https://wheels.vllm.ai/${VLLM_VERSION}/cpu --index-strategy first-index --torch-backend cpu

Warning

zentorch is not supported with vLLM 0.27.0 and 0.27.1.

vLLM auto-detects the platform and routes linear layers through ZenDNN-optimized kernels - no flag needed. To verify it is engaged, look for the platform-selection line in the server's startup logs:

vllm serve Qwen/Qwen3-0.6B 2>&1 | grep "AMD Zen CPU detected with zentorch installed"

For per-backend dispatch details (which kernel each linear layer was bound to), re-run with VLLM_LOGGING_LEVEL=DEBUG and grep for CPU unquantized GEMM dispatch.

See AMD Zen optimizations for detection rules, supported dtypes, and the VLLM_ZENTORCH_WEIGHT_PREPACK knob.

How to decide VLLM_CPU_OMP_THREADS_BIND?

  • Default auto thread-binding is recommended for most cases. Ideally, each OpenMP thread will be bound to a dedicated physical core respectively, threads of each rank will be bound to the same NUMA node respectively, and the default reserved CPUs will be kept for other vLLM components. On x86, ARM, and RISC-V, local_world_size CPUs are reserved in total, plus one additional CPU per local rank when KV transfer is enabled. If you have any performance problems or unexpected binding behaviours, please try to bind threads as following.

  • On a hyper-threading enabled platform with 16 logical CPU cores / 8 physical CPU cores:

Commands

  • When deploying vLLM CPU backend on a multi-socket machine with NUMA and enable tensor parallel or pipeline parallel, each NUMA node is treated as a TP/PP rank. So be aware to set CPU cores of a single rank on the same NUMA node to avoid cross NUMA node memory access.

How to decide VLLM_CPU_KVCACHE_SPACE?

This value is 0 by default, which means vLLM doesn't override the KV cache size through this variable. When set, the value is in GiB (e.g, VLLM_CPU_KVCACHE_SPACE=40 means 40 GiB). Larger space can support more concurrent requests, longer context length. However, users should take care of memory capacity of each NUMA node. The memory usage of each TP rank is the sum of weight shard size and VLLM_CPU_KVCACHE_SPACE, if it exceeds the capacity of a single NUMA node, the TP worker will be killed with exitcode 9 due to out-of-memory. This is a legacy setting; use --kv-cache-memory-bytes instead.

How to do performance tuning for vLLM CPU?

First of all, please make sure the thread-binding and KV cache space are properly set and take effect. You can check the thread-binding by running a vLLM benchmark and observing CPU cores usage via htop.

Use multiples of 32 as --block-size, which is 128 by default.

Inference batch size is an important parameter for the performance. A larger batch usually provides higher throughput, a smaller batch provides lower latency. Tuning the max batch size starting from the default value to balance throughput and latency is an effective way to improve vLLM CPU performance on specific platforms. There are two important related parameters in vLLM:

  • --max-num-batched-tokens, defines the limit of token numbers in a single batch, has more impacts on the first token performance. The default value is set as:
    • Offline Inference: 4096 * world_size
    • Online Serving: 2048 * world_size
  • --max-num-seqs, defines the limit of sequence numbers in a single batch, has more impacts on the output token performance.
    • Offline Inference: 256 * world_size
    • Online Serving: 128 * world_size

vLLM CPU supports data parallel (DP), tensor parallel (TP) and pipeline parallel (PP) to leverage multiple CPU sockets and memory nodes. For more details of tuning DP, TP and PP, please refer to Optimization and Tuning. For vLLM CPU, it is recommended to use DP, TP and PP together if there are enough CPU sockets and memory nodes.

Which quantization configs does vLLM CPU support?

  • vLLM CPU supports quantizations:
    • AWQ (x86, s390x, Power)
    • GPTQ (x86, s390x, Power)
    • compressed-tensor INT8 W8A8 (x86, s390x only)

Why do I see get_mempolicy: Operation not permitted when running in Docker?

In some container environments (like Docker), NUMA-related syscalls used by vLLM (e.g., get_mempolicy, migrate_pages) are blocked/denied in the runtime's default seccomp/capabilities settings. This may lead to warnings like get_mempolicy: Operation not permitted. Functionality is not affected, but NUMA memory binding/migration optimizations may not take effect and performance can be suboptimal.

To enable these optimizations inside Docker with the least privilege, you can follow below tips:

docker run ... --cap-add SYS_NICE --security-opt seccomp=unconfined  ...

# 1) `--cap-add SYS_NICE` is to address `get_mempolicy` EPERM issue.

# 2) `--security-opt seccomp=unconfined` is to enable `migrate_pages` for `numa_migrate_pages()`.
# Actually, `seccomp=unconfined` bypasses the seccomp for container,
# if it's unacceptable, you can customize your own seccomp profile,
# based on docker/runtime default.json and add `migrate_pages` to `SCMP_ACT_ALLOW` list.

# reference : https://docs.docker.com/engine/security/seccomp/

Alternatively, running with --privileged=true also works but is broader and not generally recommended.

In K8S, the following configuration can be added to workload yaml to achieve the same effect as above:

securityContext:
  seccompProfile:
    type: Unconfined
  capabilities:
    add:
    - SYS_NICE
Edit on GitHub
AI 翻译查看英文原文