ALiBi (Attention with Linear Biases)
What it is
ALiBi is a method for transformer language models that enables input length extrapolation by adding a linear bias to attention scores instead of using traditional position embeddings. It allows models trained on short sequences to perform inference on much longer sequences without performance degradation.
The problem it solves
Standard position embeddings (learned or sinusoidal) often cause models to fail or become unstable when processing sequences longer than those seen during training.
What you could use it for
- Training language models that need to handle long contexts (e.g., long documents) with limited training data/compute.
- Improving performance in low-resource language modeling settings.
- Implementing transformer models that require efficient memory and speed during long-sequence inference.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.