Foil: A Method for Looping Mixture-of-Experts Models
What it is
This paper introduces Foil, a design strategy for looped Mixture-of-Experts (MoE) models that flattens expert layers to increase the routing pool and unties attention parameters across passes. It demonstrates that this approach improves pretraining loss and routing balance compared to unflattened baselines while keeping expert parameters and compute fixed.
The problem it solves
Standard looped MoE models suffer from suboptimal expert usage and routing imbalance, limiting the benefits of reusing layers for additional computation.
What you could use it for
- Designing efficient sparse MoE architectures for large language models
- Optimizing expert utilization in fixed-parameter transformer models
- Researching the interaction between layer looping and sparse routing
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.