Rounding-Location Alignment Stabilizes LLM Reinforcement Learning
What it is
This paper investigates how numerical precision differences between inference and training stacks destabilize reinforcement learning for large language models. It proposes aligning rounding locations in specific regions (RoPE, SwiGLU, and residual RMSNorm) by retaining intermediates in FP32 until BF16 outputs, which prevents training collapse without changing the inference engine or learning objective.
The problem it solves
Implementation differences between inference and training systems cause intended on-policy updates to become off-policy, leading to training instability and collapse in BF16 reinforcement learning for large language models.
What you could use it for
- Stabilizing BF16 reinforcement learning training for Qwen-family models
- Preventing off-policy updates caused by implementation differences between inference and training systems
- Improving the robustness of LLM fine-tuning pipelines that use separate inference and training stacks
Why it matters
Enables stable and efficient BF16 reinforcement learning for large language models without requiring full stack alignment or global precision changes, potentially reducing the computational overhead and complexity of maintaining consistent numerical behavior across distributed training environments.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.