Order-Only On-Policy Distillation
What it is
This paper proposes Order-Only On-Policy Distillation (O3PD), a method for training smaller language models by matching the teacher's ranking of top-K tokens rather than their exact probability values. By focusing on ordinal supervision, the approach eliminates sensitivity to teacher temperature and avoids transferring the teacher's confidence calibration, which smaller students cannot effectively reproduce.
The problem it solves
Standard on-policy distillation is sensitive to teacher temperature and forces smaller students to reproduce the teacher's probability values, which entangle token preference with the teacher's specific capacity and calibration that the student cannot match.
What you could use it for
- Training smaller student models to match the reasoning capabilities of larger teacher models on mathematical benchmarks
- Improving on-policy distillation pipelines by removing the need for temperature tuning
- Transferring ranking-based knowledge from large models to smaller, more efficient models
Why it matters
If effective, this method could significantly lower the cost and complexity of distilling large language models by simplifying the training objective to ranking, potentially yielding stronger smaller models without the calibration mismatches inherent in probability-matching distillation.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.