vLLM integration

vLLM can be deployed as a server that mimics the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API. This server can be queried in the same format as OpenAI API.

Overview

This will help you get started with vLLM chat models, which leverages the langchain-openai package. For detailed documentation of all ChatOpenAI features and configurations head to the API reference.

Integration details

Class	Package	Serializable	JS support	Downloads	Version
ChatOpenAI	langchain_openai	beta	❌

Model features

Specific model features, such as tool calling, support for multi-modal inputs, support for token-level streaming, etc., will depend on the hosted model.

Setup

See the vLLM documentation.

To access vLLM models through LangChain, you'll need to install the langchain-openai integration package.

Credentials

Authentication will depend on specifics of the inference server.

To enable automated tracing of your model calls, set your LangSmith API key:

os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_API_KEY"] = getpass.getpass("Enter your LangSmith API key: ")

Installation

The LangChain vLLM integration can be accessed via the langchain-openai package:

pip install -qU langchain-openai

Instantiation

Now we can instantiate our model object and generate chat completions:

from langchain.messages import HumanMessage, SystemMessage
from langchain_openai import ChatOpenAI

inference_server_url = "http://localhost:8000/v1"

llm = ChatOpenAI(
    model="mosaicml/mpt-7b",
    api_key="your api key goes here",
    base_url=inference_server_url,
    max_tokens=5,
    temperature=0,
)

Invocation

messages = [
    SystemMessage(
        content="You are a helpful assistant that translates English to Italian."
    ),
    HumanMessage(
        content="Translate the following sentence from English to Italian: I love programming."
    ),
]
llm.invoke(messages)

AIMessage(content=' Io amo programmare', additional_kwargs={}, example=False)

API reference

For detailed documentation of all features and configurations exposed via langchain-openai, head to the API reference: reference.langchain.com/python/langchain-openai/chat_models/base/ChatOpenAI

Refer to the vLLM documentation as well.