Mixture of Experts
A neural architecture that routes each input to a subset of specialized feed-forward experts, increasing model capacity without activating all parameters for every token.
Related Concepts
- Transformer
- Router
- Sparse Model
- Load Balancing