AutoRound
What it is
AutoRound is an advanced quantization toolkit for Large Language Models (LLMs) and Vision-Language Models (VLMs) that enables high-accuracy low-bit inference (2–4 bits). It utilizes sign-gradient descent and supports multiple data types across CPU, XPU, and CUDA hardware.
The problem it solves
High memory and compute requirements for deploying large language models, and the accuracy loss typically associated with aggressive quantization.
What you could use it for
- Quantizing LLMs to 2-4 bits for low-memory inference
- Deploying models on vLLM or SGLang with high accuracy
- Generating mixed-precision quantization schemes
- Quantizing models for GGUF format
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.