LlamaIndex

还没翻译译自英文（docs.vllm.ai）本页 5 段都还没译

LlamaIndex

vLLM is also available via LlamaIndex .

To install LlamaIndex, run

pip install llama-index-llms-vllm -q

To run inference on a single or multiple GPUs, use Vllm class from llamaindex.

from llama_index.llms.vllm import Vllm

llm = Vllm(
    model="microsoft/Orca-2-7b",
    tensor_parallel_size=4,
    max_new_tokens=100,
    vllm_kwargs={"gpu_memory_utilization": 0.5},
)

Please refer to this Tutorial for more details.

Edit on GitHub

AI 翻译查看英文原文