RAQ-MLA: Role-Aware Quantization for Multi-Head Latent Attention Caches
What it is
This paper proposes RAQ-MLA, a quantization method for Multi-Head Latent Attention (MLA) caches that treats the latent and positional key tensors differently based on their roles in attention. It fits invertible transforms to minimize attention-weighted value error for the latent and score perturbation for the positional key, achieving ~2.5 bits per element without training.
The problem it solves
Existing KV-cache quantization methods are designed for standard attention and fail to account for the specific structure of MLA caches (compressed latent + positional key), leading to significant accuracy loss at low bit widths.
What you could use it for
- Reducing memory footprint of MLA-based LLMs during inference
- Increasing theoretical KV-cache capacity for longer context windows
- Quantizing KV caches for models like DeepSeek-V2/V3 or Kimi
Why it matters
Significantly lowers the memory cost of serving MLA models, enabling longer contexts and higher batch sizes on existing hardware while maintaining near-lossless performance on reasoning and coding tasks.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.