Distribution Matching Distillation for Continuous Diffusion Language Models
What it is
This paper proposes two distillation methods, Simplex-DMD and Reinforce-DMD, to reduce the number of network evaluations required for high-quality generation in continuous diffusion language models. It connects student output parameterizations to gradient estimators, using either continuous relaxations with pathwise gradients or categorical sampling with REINFORCE.
The problem it solves
High-quality generation with continuous diffusion language models typically requires hundreds of network evaluations, making inference expensive and slow.
What you could use it for
- Reducing inference latency for diffusion-based text generation models
- Training efficient student models that match the distribution of larger teacher diffusion models
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.