VFold: Symmetry-Aware Cross-Layer Value Cache Compression
What it is
This paper proposes a symmetry-aware strategy for merging value caches across layers in Large Language Models to reduce memory usage during decoding. It demonstrates that this method can be combined with existing compression techniques like quantization and pruning to achieve higher compression ratios without significant performance loss.
The problem it solves
KV cache memory dominates usage at long context lengths, limiting the practical context window size for LLMs on memory-constrained hardware.
What you could use it for
- Reducing memory footprint for long-context LLM inference
- Scaling context windows on hardware with limited VRAM
- Optimizing KV cache storage in LLM serving systems
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.