HyScale: Scaling Long-Context Training for Learned Sparse Attention
What it is
HyScale is a training system designed to optimize long-context training for learned sparse attention models. It introduces HyTopK to find exact top-k indices without materializing full score tables and uses selection replay to reuse forward-pass selection during recomputation, significantly speeding up training for models like GLM-5.1 and DeepSeek-V4-Flash.
The problem it solves
Current systems for learned sparse attention suffer from training bottlenecks because they materialize full score tables and repeat selection during recomputation, which is inefficient for million-token contexts.
What you could use it for
- Training large language models with million-token context windows
- Optimizing sparse attention layers in production models like DeepSeek and Qwen
- Reducing training bottlenecks in learned sparse attention architectures
Why it matters
Significantly accelerates the training of long-context LLMs, potentially reducing the cost and time required to develop models capable of handling million-token sequences.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.