Twice as Fast, Until It Isn't: Quantized Serving of Mid-Size Open LLMs
What it is
This paper analyzes the performance and accuracy trade-offs of quantizing 9B-20B open-weight LLMs for serving on vLLM and SGLang. It aggregates public measurements to show that while 4-bit quantization doubles speed at low batch sizes, the benefit vanishes at high batch sizes where FP8 becomes more effective, and notes that many reported accuracy differences are within statistical noise.
The problem it solves
The lack of comprehensive, verified data on the speedup and accuracy trade-offs for quantized mid-size LLMs, leading to confusion about which quantization recipe is optimal for specific serving conditions.
What you could use it for
- Selecting a quantization recipe (INT4, FP8, etc.) for serving mid-size LLMs
- Estimating throughput gains for batch-1 vs. high-batch inference workloads
- Evaluating the statistical significance of accuracy benchmarks for quantized models
Why it matters
Provides a clear, evidence-based framework for choosing quantization methods, preventing practitioners from relying on misleading speedup or accuracy claims that do not hold across different batch sizes and hardware configurations.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.