AttentionIndex: Accelerating Learned Sparse Attention with a Causal Inverted-File Index
What it is
AttentionIndex is a proposed method for speeding up the key-selection stage of learned sparse attention, where a lightweight indexer must score every key in the prefix before a query attends to only k of them. It replaces exhaustive or block-based two-stage selection with an inverted-file index that groups indexer keys by similarity, then refines the retrieved candidates using the native indexer score. The index is built incrementally during prefill, reuses prior clustering to keep maintenance cheap, and restricts each query to clusters derived from its own causal prefix.
The problem it solves
In learned sparse attention the indexer still scores every key in the prefix, so at one million tokens selection consumes more than half of prefill time; existing block-based two-stage selection (HiSA) reduces that cost but loses recall as context grows at a fixed candidate budget.
What you could use it for
- Speeding up prefill for long-context LLM inference that uses learned sparse attention (e.g. DeepSeek/GLM-style indexer architectures)
- Reducing the cost of top-k key selection at context lengths around one million tokens
- Improving recall of the selected top-k keys at a fixed candidate budget compared with block-based two-stage selection
- Long-context benchmark accuracy improvement at equal candidate budget
Why it matters
If the reported results hold, it could cut a dominant share of prefill time in million-token sparse-attention inference while recovering more of the exhaustive top-k than block-based selection, potentially improving both throughput and long-context accuracy for production sparse-attention models.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.