Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data
What it is
This paper proposes a method for accelerating Mixture-of-Experts (MoE) decoding by selecting which expert channels to load without requiring offline calibration data. It demonstrates that using a top-k selection strategy outperforms calibrated thresholds in accuracy and stability, and introduces a weight-based reader that uses ternary sketches and low-rank maps to recover skipped channels.
The problem it solves
MoE models are bottlenecked by loading routed-expert weights from memory, and existing channel selection methods require calibration data and suffer from inconsistent read budgets and accuracy collapse.
What you could use it for
- Accelerating interactive decoding for MoE models at small batch sizes
- Reducing memory bandwidth bottlenecks in LLM inference
- Improving the stability of channel selection to avoid repetition loops
Why it matters
Significantly faster inference speeds (1.6x-2.69x) for MoE models while retaining high accuracy, removing the need for complex calibration pipelines.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.