AllenAI's EMO MoE model induces modular experts from data in end-to-end pretraining

AllenAI announced EMO on May 8, 2026 — a mixture — of-experts model pretrained end-to-end so modular structure emerges directly from data rather than from human — defined priors. The model is a large sparse MoE with 14 billion total parameters and 1 billion active parameters; it activates eight experts per token from a pool of 128 and was pretrained on roughly 1 trillion tokens. EMO is reported to allow using only 12.5% of experts for a given task while retaining near full-model performance, whereas a standard MoE trained on the same data showed severe degradation when experts were selected selectively. That selective — use capability could let operators cut serving costs and tailor behavior to specific domains.

The release positions EMO against two persistent deployment challenges: monolithic LLMs become costly to host and adapt as they scale toward trillions of parameters, and conventional MoEs often still require the full model because tokens tend to activate diverse experts. Prior attempts to enforce modularity by routing with predefined semantic domain labels — for example in BTX and the FlexOlmo project — depend on domain labels, can introduce human bias, and lock in a fixed modular structure.

For builders and operators, EMO’s selective expert use presents concrete trade — offs. Teams can load and serve small expert subsets tailored to a task or domain to reduce memory and compute with only a modest accuracy hit, or use all experts when general — purpose behavior is needed. That composable serving model aims to make sparse architectures more practical for domain — specific services, edge-adjacent deployments, and cost-sensitive hosting.

EMO treats modularity as an explicit training objective and modifies how routing and expert specialization develop during pretraining. Instead of forcing token — to-domain mappings with labels, the router is trained so tokens from similar contexts tend to activate similar subsets of experts, encouraging coherent expert groups that can be selected or composed at inference time. This approach addresses a common failure mode in standard MoEs, where experts specialize in low-level lexical patterns rather than higher — level capabilities.

AllenAI published supporting artifacts to help builders evaluate and reproduce results: a technical report, the codebase, a model collection, and an interactive visualization that demonstrates model behavior. The team emphasizes that EMO can function both as a strong general — purpose model when all experts are used and as a composable model when small expert subsets are selected, offering a new path for sparse — model deployment and experimentation.