Pruning and Distilling Mixture-of-Experts into Dense Language Models
27 May 2026
Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models, yet it requires all expert parameters to be loaded in memory, making it less preferable for memory-constrained deployment.