⚠️ Warning:
The AutoAWQ library is deprecated. This functionality has been adopted by the vLLM project in llm-compressor.
For the recommended quantization workflow, please see the AWQ examples in llm-compressor. For more details on the deprecation, refer to the original AutoAWQ repository.
To create a new 4-bit quantized model, you can leverage AutoAWQ.
Quantization reduces the model's precision from BF16/FP16 to INT4 which effectively reduces the total model memory footprint.
The main benefits are lower latency and memory usage.
After installing AutoAWQ, you are ready to quantize a model. Please refer to the AutoAWQ documentation for further details. Here is an example of how to quantize mistralai/Mistral-7B-Instruct-v0.2:
from awq import AutoAWQForCausalLMfrom transformers import AutoTokenizermodel_path = "mistralai/Mistral-7B-Instruct-v0.2"quant_path = "mistral-instruct-v0.2-awq"quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM"}# Load modelmodel = AutoAWQForCausalLM.from_pretrained( model_path, low_cpu_mem_usage=True, use_cache=False,)tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)# Quantizemodel.quantize(tokenizer, quant_config=quant_config)# Save quantized modelmodel.save_quantized(quant_path)tokenizer.save_pretrained(quant_path)print(f'Model is quantized and saved at "{quant_path}"')
AWQ models are also supported directly through the LLM entrypoint:
from vllm import LLM, SamplingParams# Sample prompts.prompts = [ "Hello, my name is", "The president of the United States is", "The capital of France is", "The future of AI is",]# Create a sampling params object.sampling_params = SamplingParams(temperature=0.8, top_p=0.95)# Create an LLM.llm = LLM(model="TheBloke/Llama-2-7b-Chat-AWQ", quantization="auto_awq")# Generate texts from the prompts. The output is a list of RequestOutput objects# that contain the prompt, generated text, and other information.outputs = llm.generate(prompts, sampling_params)# Print the outputs.for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")