Structure-Preserving Expert Pruning for MoE LLMs
What it is
This paper demonstrates that expert outputs in Mixture-of-Experts (MoE) models exhibit structured redundancy, characterized by low stable rank and entropy effective rank. It proposes two pruning methods, PEEP and DCEP, that treat expert selection as a structure-preserving subset optimization problem rather than independent scoring, using router weights as priors to maintain global structural relationships.
The problem it solves
Practical deployment of MoE models requires storing all experts, leading to substantial memory overhead despite the sparse activation mechanism.
What you could use it for
- Reducing memory footprint of MoE models like DeepSeek-V2-Lite and Qwen3-30B-A3B for deployment
- Compressing LLMs while preserving performance on general and mathematical reasoning tasks
- Optimizing inference efficiency by retaining only a subset of experts (e.g., 50-75% retention)
Why it matters
Significantly reduces the storage and memory overhead of deploying large MoE models without sacrificing performance, making high-capacity models more accessible for practical applications.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.