From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation
What it is
This paper investigates how multi-teacher on-policy distillation (MOPD) affects model parameters by analyzing gradients, optimizer updates, and learning curves in Qwen3-1.7B. It identifies that loss averaging strategies, Adam optimizer dynamics, and BF16 precision rounding significantly influence the final student model's capabilities.
The problem it solves
The lack of understanding regarding how specific teacher signals and optimization mechanics (like averaging and precision) translate into parameter changes and final task performance in multi-teacher distillation.
What you could use it for
- Designing effective multi-teacher distillation pipelines for LLMs
- Optimizing loss averaging strategies to balance domain contributions
- Understanding the impact of mixed-precision training on distillation fidelity
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.