There Is No Free Lunch in RLVR: Every pass@1 Gain Has a pass@k Price
What it is
This paper presents a theoretical analysis of Reinforcement Learning with Verifiable Rewards (RLVR), proving that improvements in first-try success rates (pass@1) inevitably come at the cost of reduced success rates when given multiple attempts (pass@k). It establishes that this trade-off is a fundamental mathematical constraint rather than a transient training artifact, unifying entropy collapse and performance laws under a single mechanism.
The problem it solves
The unexplained phenomenon where RL-tuned models perform better on single attempts but worse than their base counterparts when given multiple sampling opportunities, which has been debated as either a failure of reasoning or a transient training issue.
What you could use it for
- Designing RL training objectives that account for the pass@1/pass@k trade-off
- Diagnosing whether a model's performance degradation at high sampling budgets is due to fundamental RL constraints or implementation errors
- Setting realistic expectations for the generalization capabilities of RL-tuned language models
Why it matters
Provides a rigorous theoretical bound on the capabilities of RL-tuned models, preventing wasted effort on trying to 'fix' a fundamental mathematical limitation and guiding the development of constrained updates that trace the optimal performance frontier.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.