DHSA: Dynamic Hierarchical Sparse Attention
What it is
DHSA is a research implementation of a data-driven plugin for long-context LLM inference that predicts attention sparsity on the fly without retraining the base model. It extends Block-Sparse Attention with variable-sized query and key blocks to improve accuracy under highly sparse regimes, supporting Qwen2/Qwen2.5 and Llama models.
The problem it solves
The quadratic cost of dense attention in LLMs creates severe memory and latency bottlenecks for long-context processing, and existing static sparse methods fail to adapt to content, leading to accuracy degradation.
What you could use it for
- Reducing memory and latency costs for long-context inference on LLMs like Llama-3.1-8B and Qwen2.5-3B
- Accelerating prompt prefill while maintaining benchmark accuracy (NIAH, LongBench, RULER) compared to full FlashAttention-2
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.