HeteroCache
What it is
HeteroCache is a training-free dynamic KV cache compression framework designed for long-context LLM inference. It works by categorizing attention heads based on temporal stability and spatial similarity, using a hierarchical storage mechanism to selectively retrieve context and reduce memory overhead.
The problem it solves
Linear memory growth of KV caches in long-context tasks and the failure of static compression methods to preserve globally important information.
What you could use it for
- Accelerating inference for long-context tasks (e.g., 224K context length)
- Reducing memory footprint of KV caches in LLMs
- Evaluating multiple KV cache compression methods (e.g., FullKV, H2O, SnapKV) in a unified framework
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.