RangeSWA: Training-Free Length Extrapolation for Continuous Diffusion Language Models
What it is
This paper proposes RangeSWA, a training-free method to enable continuous diffusion language models to generate text longer than their training length. It works by combining local sliding-window attention, remote block summaries, and dynamic content retrieval, while using positional remapping and adaptive window scheduling to mitigate quality degradation.
The problem it solves
Continuous diffusion language models suffer from text repetition, semantic drift, and quality degradation when the generation length exceeds the length they were trained on.
What you could use it for
- Generating long-form text with continuous diffusion language models without retraining
- Reducing text repetition and semantic drift in long-context generation tasks
Why it matters
Enables existing continuous diffusion language models to handle arbitrarily long sequences efficiently, potentially making them viable for long-document generation and summarization tasks without the cost of fine-tuning.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.