TaH2: Adaptive Looped Transformers for Test-Time Scaling
What it is
This paper introduces TaH2, a method that improves the efficiency of looped transformers during test-time scaling. It uses a joint post-training approach with an iteration decider to apply extra computational loops only to tokens that benefit from them, rather than all tokens.
The problem it solves
Existing looped transformers often underperform non-looped baselines at matched compute because they waste iterations on tokens that do not benefit from additional processing, leading to inefficient test-time scaling.
What you could use it for
- Improving the accuracy-compute trade-off in large language models during inference
- Optimizing test-time scaling for mathematical reasoning benchmarks like AIME
- Reducing the computational cost of looped transformer architectures
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.