CriticOPD: Critique-Guided On-Policy Distillation
What it is
This paper proposes CriticOPD, a method to improve on-policy distillation by addressing the 'recoverability' failure where teachers follow student errors. It uses the teacher to critique incorrect student rollouts, allowing the student to rewrite the solution with that feedback before scoring.
The problem it solves
Standard on-policy distillation fails when student models generate incorrect prefixes because the teacher model cannot recover the correct answer from those erroneous states, leading to poor supervision signals.
What you could use it for
- Training smaller language models to match the reasoning capabilities of larger teacher models
- Improving mathematical reasoning performance in LLMs
- Refining on-policy distillation pipelines for complex problem-solving tasks
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.