Model Selection for Off-Policy Evaluation via Pairwise Bellman Validation
What it is
This paper introduces Pairwise Bellman Validation (PBV), a tuning-free minimax procedure for selecting the best fitted Q-function or occupancy-ratio estimator in off-policy evaluation. It works by constructing critics from pairwise differences between candidates to estimate Bellman moments, avoiding the need for auxiliary validation models or separate optimization.
The problem it solves
Errors in fitted Q-functions and occupancy-ratio estimates cannot be directly evaluated from held-out transitions, and existing validation methods often require costly auxiliary regressions or separate critic optimization.
What you could use it for
- Selecting the best hyperparameters for a Q-function in offline reinforcement learning
- Choosing the most accurate occupancy-ratio estimator for off-policy evaluation
- Validating fitted models in settings where direct evaluation of estimation error is impossible
Why it matters
Reduces the computational and tuning overhead of off-policy evaluation by providing a theoretically grounded, tuning-free method for model selection that offers finite-sample oracle guarantees.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.