CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning
What it is
CARM is a sequence-level masking method for LLM reinforcement learning that prevents opposing probability changes from canceling out in off-policy correction. It takes the absolute value of token log-ratios before averaging to better detect policy drift.
The problem it solves
Standard sequence-level masking using geometric mean of token probability ratios can hide substantial bidirectional policy drift because positive and negative log-ratios cancel each other out, leading to ineffective off-policy correction in LLM RL training.
What you could use it for
- Improving mathematical reasoning capabilities in LLMs during RL post-training
- Enhancing code generation performance in LLMs by correcting off-policy issues
- Stabilizing RL training when there is a mismatch between rollout and training engines
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.