Skip to main content

TorchAO

还没翻译译自英文（docs.vllm.ai）本页 7 段都还没译

TorchAO

TorchAO is an architecture optimization library for PyTorch, it provides high performance dtypes, optimization techniques and kernels for inference and training, featuring composability with native PyTorch features like torch.compile, FSDP etc.. Some benchmark numbers can be found here.

We recommend installing the latest torchao nightly with

# Install the latest TorchAO nightly build
# Choose the CUDA version that matches your system (cu126, cu128, etc.)
pip install \
    --pre torchao>=10.0.0 \
    --index-url https://download.pytorch.org/whl/nightly/cu126

Quantizing HuggingFace Models

You can quantize your own huggingface model with torchao, e.g. transformers and diffusers, and save the checkpoint to huggingface hub like this with the following example code:

Code

Alternatively, you can use the TorchAO Quantization space for quantizing models with a simple UI.

Edit on GitHub
AI 翻译查看英文原文