Subquadratic Attention with a Hierarchical Tree-Structured Indexer
What it is
This paper introduces Zyphra Hierarchical Attention (ZHA), a method that replaces quadratic attention with a subquadratic alternative using a hierarchical tree-structured indexer. It works by constructing a tree of recursively compressed key summaries over the KV cache, allowing queries to traverse the tree top-down to retrieve a sparse subset of keys, thereby reducing computational scaling.
The problem it solves
The quadratic scaling of standard attention mechanisms is a fundamental bottleneck that limits the long-context capabilities and computational efficiency of large language models.
What you could use it for
- Extending the effective context window of Large Language Models to hundreds of millions or billions of tokens
- Improving long-context retrieval and reasoning performance in LLMs
- Optimizing agentic coding and mathematical task performance with large context inputs
Why it matters
If successful, ZHA could dramatically outperform competing sparse attention methods beyond 1 million context tokens, potentially enabling LLMs to process and reason over extremely long documents, codebases, or conversation histories without prohibitive computational cost.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.