Menu

Home
GitHub(새 탭에서 열림)

Home

Using vLLM

Quickstart

Installation
GPU
CPU

Examples

vLLM V1
Frequently Asked Questions
Production Metrics
Reproducibility
Security
Troubleshooting
Usage Stats Collection

Offline Inference

Online Serving
Derenderer APIs
Generative Scoring
OpenAI-Compatible Server
Renderer APIs
Speech to Text APIs
Tokens In <> Tokens Out API
Trace Replay

Context Parallel Deployment
Data Parallel Deployment
Troubleshooting distributed deployments
Expert Parallel Deployment
Parallelism and Scaling

Claude Code
Codex
LangChain
LlamaIndex

Using Docker
Using Kubernetes
Using Nginx

Anyscale
AnythingLLM
AutoGen
BentoML
Cerebrium
Chatbox
Crusoe
Dify
dstack
Haystack
Helm
Hugging Face Inference Endpoints
LiteLLM
Lobe Chat
LWS
Modal
Nebius Serverless AI
Open WebUI
Retrieval-Augmented Generation
RunPod
SkyPilot
Streamlit
NVIDIA Triton

AIBrix
NVIDIA Dynamo
KAITO
KServe
Kthena
KubeAI
KubeRay
Llama Stack
llm-d
llmaz
Production stack

Async Reinforcement Learning
What is Layerwise (Re)loading?
Prompt Token ID Logprobs (Teacher Scoring)
Reinforcement Learning from Human Feedback
Routed Experts Return (MoE Routing Replay / R3)
Sampling Mask (Distribution Replay)
Transformers Reinforcement Learning
Weight Checker

Weight Transfer
Base Classes and Custom Engines
IPC Engine
NCCL M2N Engine
ModelExpress Engine
NCCL Engine
Sharded RDT Engine

Configuration Options
Conserving Memory
Engine Arguments
Environment Variables
Model Resolution
Optimization and Tuning
Server Arguments

Supported Models
Generative Models

Pooling Models
Classification Usages
Embedding Usages
Reward Usages
Scoring Usages
Specific Model Examples
Token Classification Usages
Token Embedding Usages

Structured Decisions Models

Fastsafetensor
Loading Model Weights with InstantTensor
Loading models with Run:ai Model Streamer
Loading models with CoreWeave's Tensorizer

CPU - Intel® Xeon®
XPU - Intel® GPUs

Features
Automatic Prefix Caching
Batch Invariance
Context Extension
Cross-Encoder Output Reuse
Custom Arguments
Custom Logits Processors
Disaggregated Encoder
Disaggregated Prefilling (experimental)
CPU EC Connector Usage Guide
Engram: conditional memory via n-gram lookups
IndexCache
Initialized engine snapshots
Interleaved Thinking
KV Offloading Usage Guide
LoRA Adapters
MooncakeConnector Usage Guide
ECMooncakeConnector Usage Guide
MooncakeStoreConnector Usage Guide
MoRIIOConnector Usage Guide
Multimodal Inputs
NixlConnector Compatibility Matrix
NixlConnector Usage Guide
Per-Request Metrics
Preload
Prompt Embedding Inputs
Reasoning Outputs
Sleep Mode
Structured Outputs
Tool Calling
Text watermarking

Quantization
AutoAWQ
b12x Linear and MoE Backends
BitsAndBytes
FP8 ViT Encoder Attention
GGUF
GPTQModel
Intel Quantization Support
NVIDIA Model Optimizer
Online Quantization
Quantized KV Cache
AMD Quark
TorchAO

LLM Compressor
FP8 W8A8
INT4 W4A16
INT8 W4A8
INT8 W8A8

Speculative Decoding
Per-Request Acceptance Metrics
Adaptive Verification
Draft Models
Dynamic Speculative Decoding
EAGLE Draft Models
Hidden State Extraction
FLy Verification
LiLiCorr
MTP (Multi-Token Prediction)
N-Gram Speculation
Parallel Draft Models
vLLM-Project/Speculators
Suffix Decoding

Contributing to vLLM

Deprecation Policy
Dockerfile
Editing Agent Instructions
Incremental Compilation Workflow
JIT Kernel Warmup
Labels
Profiling vLLM
Releasing vLLM
Vulnerability Management

Summary
Basic Model
Registering a Model
Unit Testing
Multi-Modal Support
Speech-to-Text (Transcription/Translation) Support

CI Failures
Nightly Builds of vLLM Wheels
Update PyTorch version on vLLM OSS CI/CD

Endpoint Plugins
IO Processor Plugins
LoRA Resolver Plugins
Plugin System

Architecture Overview
Attention Backend Feature Support
CUDA Graphs
Vision Encoder (ViT) CUDA Graphs
CustomOp
Dual Batch Overlap
How to debug the vLLM-torch.compile integration
Expert substitution (MoNE)
Fused MoE Modular Kernel
Fusion torch.compile passes
HiSparse local KV offload architecture
Integration with Hugging Face
Hybrid KV Cache Manager
Logits Processors
Metrics
Multi-Modal Data Processing
Model Runner V2 Design Document
Fused MoE Kernel Features
Python Multiprocessing
NIXL KV Cache Lease Renewal
NIXL push-mode KV transfer
Optimization Levels
Paged Attention
Automatic Prefix Caching
Octave KV: a 3-bit KV cache for ROCm
Zero-Copy Shared-Memory Tensor Arena for MessageQueue
torch.compile integration
torch.compile with Multimodal Encoders
vLLM IR: Functional Intermediate Representation

Benchmark Suites
Benchmark CLI
Parameter Sweeps
Performance Dashboard

Summary

vLLM CLI Guide

Contact Us
Meetups
Reviewers
Sponsors

Collaboration Policy
Committers
Governance Process

本页还没有译文

本页 2 段还没译成한국어，下面显示的是英文原文。

NVIDIA Triton

还没翻译译自英文（docs.vllm.ai）本页 2 段都还没译

原文

NVIDIA Triton

The Triton Inference Server hosts a tutorial demonstrating how to quickly deploy a simple facebook/opt-125m model using vLLM. Please see Deploying a vLLM model in Triton for more details.

Edit on GitHub

AI 翻译·查看英文原文

Adapted from docs.vllm.ai/en/latest/deployment/frameworks/triton by the vLLM project (LICENSE names no copyright holder), licensed under Apache-2.0. Changes were made: this page is re-rendered from the original's source.

본문 넓게 보기