LongContextAttention (yunchang)
What it is
A library providing Unified Sequence Parallel (USP) attention for training and inference of long-context Large Language Models. It combines the high throughput of DeepSpeed-Ulysses-Attention with the scalability of Ring-Attention to overcome limitations in both methods.
The problem it solves
Resolves the trade-offs between DeepSpeed-Ulysses (limited by head count and complex TP compatibility) and Ring-Attention (lower compute/communication efficiency and risk of deadlocks).
What you could use it for
- Training LLMs with long context windows on distributed systems
- Efficient inference of models using Grouped Query Attention (GQA) or Multi-Query Attention (MQA)
- Scaling model training across multiple GPUs with improved memory and communication efficiency
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.