Using Kubernetes

Deploying vLLM on Kubernetes is a scalable and efficient way to serve machine learning models. This guide walks you through deploying vLLM using native Kubernetes.

• Deployment with CPUs
• Deployment with GPUs
• Serving with gRPC
• Troubleshooting
  • Startup Probe or Readiness Probe Failure, container log contains "KeyboardInterrupt: terminated"
• Conclusion

Alternatively, you can deploy vLLM to Kubernetes using any of the following:

• Helm
• NVIDIA Dynamo
• InftyAI/llmaz
• llm-d
• KAITO
• KServe
• Kthena
• KubeRay
• kubernetes-sigs/lws
• meta-llama/llama-stack
• substratusai/kubeai
• vllm-project/AIBrix
• vllm-project/production-stack

Deployment with CPUs

First, create a Kubernetes PVC and Secret for downloading and storing Hugging Face model:

cat <<EOF |kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vllm-models
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 50Gi
---
apiVersion: v1
kind: Secret
metadata:
  name: hf-token-secret
type: Opaque
stringData:
  token: "REPLACE_WITH_TOKEN"
EOF

Here, the token field stores your Hugging Face access token. For details on how to generate a token, see the Hugging Face documentation.

Next, start the vLLM server as a Kubernetes Deployment and Service.

Note that you will want to configure your vLLM image based on your processor arch:

VLLM_IMAGE=public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:latest       # use this for x86_64
VLLM_IMAGE=public.ecr.aws/q9t5s3a7/vllm-arm64-cpu-release-repo:latest # use this for arm64
cat <<EOF |kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: vllm
  template:
    metadata:
      labels:
        app.kubernetes.io/name: vllm
    spec:
      containers:
      - name: vllm
        image: $VLLM_IMAGE
        command: ["/bin/sh", "-c"]
        args: [
          "vllm serve meta-llama/Llama-3.2-1B-Instruct"
        ]
        env:
        - name: HF_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token-secret
              key: token
        ports:
          - containerPort: 8000
        volumeMounts:
          - name: llama-storage
            mountPath: /root/.cache/huggingface
      volumes:
      - name: llama-storage
        persistentVolumeClaim:
          claimName: vllm-models
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-server
spec:
  selector:
    app.kubernetes.io/name: vllm
  ports:
  - protocol: TCP
    port: 8000
    targetPort: 8000
  type: ClusterIP
EOF

We can verify that the vLLM server has started successfully via the logs (this might take a couple of minutes to download the model):

kubectl logs -l app.kubernetes.io/name=vllm
...
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Deployment with GPUs

Pre-requisite: Ensure that you have a running Kubernetes cluster with GPUs.

1. Create a PVC, Secret and Deployment for vLLM

   PVC is used to store the model cache and it is optional, you can use hostPath or other storage options

   Yaml

   Secret is optional and only required for accessing gated models, you can skip this step if you are not using gated models

   apiVersion: v1
   kind: Secret
   metadata:
     name: hf-token-secret
     namespace: default
   type: Opaque
   stringData:
     token: "REPLACE_WITH_TOKEN"

   Next to create the deployment file for vLLM to run the model server. The following example deploys the Mistral-7B-Instruct-v0.3 model.

   Here are two examples for using NVIDIA GPU and AMD GPU.

   NVIDIA GPU:

   Yaml

   AMD GPU:

   You can refer to the deployment.yaml below if using AMD ROCm GPU like MI300X.

   Yaml

   You can get the full example with steps and sample yaml files from https://github.com/ROCm/k8s-device-plugin/tree/master/example/vllm-serve.

2. Create a Kubernetes Service for vLLM

   Next, create a Kubernetes Service file to expose the mistral-7b deployment:

   Yaml

3. Deploy and Test

   Apply the deployment and service configurations using kubectl apply -f <filename>:

   kubectl apply -f deployment.yaml
   kubectl apply -f service.yaml

   To test the deployment, run the following curl command:

   curl http://mistral-7b.default.svc.cluster.local/v1/completions \
     -H "Content-Type: application/json" \
     -d '{
           "model": "mistralai/Mistral-7B-Instruct-v0.3",
           "prompt": "San Francisco is a",
           "max_tokens": 7,
           "temperature": 0
         }'

   If the service is correctly deployed, you should receive a response from the vLLM model.

Serving with gRPC

vLLM can serve models over gRPC instead of HTTP by passing the --grpc flag. This requires the optional gRPC dependencies:

pip install vllm[grpc]

When using --grpc, the server exposes the standard gRPC Health Checking Protocol (grpc.health.v1.Health), which integrates with Kubernetes native gRPC probes (available since Kubernetes 1.24).

To deploy with gRPC, change the vllm serve command to include --grpc and replace httpGet probes with grpc probes:

containers:
- name: mistral-7b
  image: vllm/vllm-openai:latest
  command: ["/bin/sh", "-c"]
  args: [
    "pip install vllm[grpc] && vllm serve mistralai/Mistral-7B-Instruct-v0.3 --grpc --port 50051 --trust-remote-code"
  ]
  ports:
  - containerPort: 50051
  livenessProbe:
    grpc:
      port: 50051
    initialDelaySeconds: 120
    periodSeconds: 10
  readinessProbe:
    grpc:
      port: 50051
    initialDelaySeconds: 120
    periodSeconds: 5

You can also verify the health service manually with grpcurl:

grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check

Troubleshooting

Startup Probe or Readiness Probe Failure, container log contains "KeyboardInterrupt: terminated"

If the startup or readiness probe failureThreshold is too low for the time needed to start up the server, Kubernetes scheduler will kill the container. A couple of indications that this has happened:

1. container log contains "KeyboardInterrupt: terminated"
2. kubectl get events shows message Container $NAME failed startup probe, will be restarted

To mitigate, increase the failureThreshold to allow more time for the model server to start serving. You can identify an ideal failureThreshold by removing the probes from the manifest and measuring how much time it takes for the model server to show it's ready to serve.

Conclusion

Deploying vLLM with Kubernetes allows for efficient scaling and management of ML models leveraging GPU resources. By following the steps outlined above, you should be able to set up and test a vLLM deployment within your Kubernetes cluster. If you encounter any issues or have suggestions, please feel free to contribute to the documentation.