vLLM can be run on a cloud based GPU machine with dstack, an open-source framework for running LLMs on any cloud. This tutorial assumes that you have already configured credentials, gateway, and GPU quotas on your cloud environment.
To install dstack client, run:
pip install dstack[all]dstack server
Next, to configure your dstack project, run:
mkdir -p vllm-dstackcd vllm-dstackdstack init
Next, to provision a VM instance with LLM of your choice (NousResearch/Llama-2-7b-chat-hf for this example), create the following serve.dstack.yml file for the dstack Service:
After the provisioning, you can interact with the model by using the OpenAI SDK:
from openai import OpenAIclient = OpenAI( base_url="https://gateway.<gateway domain>", api_key="<YOUR-DSTACK-SERVER-ACCESS-TOKEN>",)completion = client.chat.completions.create( model="NousResearch/Llama-2-7b-chat-hf", messages=[ { "role": "user", "content": "Compose a poem that explains the concept of recursion in programming.", } ],)print(completion.choices[0].message.content)
Note
dstack automatically handles authentication on the gateway using dstack's tokens. Meanwhile, if you don't want to configure a gateway, you can provision dstack Task instead of Service. The Task is for development purpose only. If you want to know more about hands-on materials how to serve vLLM using dstack, check out this repository
