Multi-Token Prediction Done Right
What it is
This paper identifies a flaw in standard Multi-Token Prediction (MTP) training where a noisy residual gradient interferes with learning. It proposes Soft-MTP, a technique that uses a KL gradient-recovery trick to isolate the useful self-distillation signal, improving performance and inference speed without adding parameters or overhead.
The problem it solves
Standard MTP training suffers from a low signal-to-noise ratio in the gradient, where a residual component buries the useful KL self-distillation signal, hindering learning as model scale increases.
What you could use it for
- Improving the efficiency of speculative decoding in large language models
- Enhancing zero-shot accuracy and reasoning capabilities in LLM pre-training
- Optimizing MTP training for DeepSeek- and Meta-style architectures
Why it matters
Provides a 'free lunch' for MTP optimization, enabling faster inference and better downstream performance at no additional computational or parameter cost.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.