Towards Looped Models Done Right, Part II: Rethinking at Fixed Points
What it is
This paper proposes improvements to the training of looped language models by focusing on fixed points to enable efficiency gains. It introduces a learned depth prior and orthogonal input injection to reduce perplexity and allow for truncated backpropagation, terminal KV sharing, and faster RL updates.
The problem it solves
High computational cost in training, decoding, and RL for looped language models due to the need to process full recurrent trajectories and manage large KV caches.
What you could use it for
- Training looped language models with reduced memory overhead via KV sharing
- Accelerating prefill and decoding in recurrent architectures
- Optimizing reinforcement learning updates for looped models
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.