Revisiting Efficiency–Accuracy Scaling in Mixture-of-Experts Architectures

Mixture-of-Experts (MoEs) have become a central component of many state-of-the-art open-source and proprietary large language models. Despite their widespread adoption, it remains unclear how close existing MoE architectures are to optimal for inference cost, as measured by accuracy per floating-point operation and per parameter. In this work, we revisit MoE design from a hardware-software co-design perspective, grounded in empirical and theoretical considerations. We characterize key performance bottlenecks across diverse deployment regimes, spanning offline high-throughput execution and online, latency-critical inference. Guided by these insights, we introduce LatentMoE, a new model architecture resulting from systematic design exploration and optimized for maximal accuracy per unit of compute. Empirical design space exploration at scales of up to 95B parameters and over a 1T-token training horizon, together with supporting theoretical analysis, shows that LatentMoE consistently outperforms standard MoE architectures in terms of accuracy per FLOP and per parameter. Given its strong performance, the LatentMoE architecture has been adopted by the flagship Nemotron-3 Super and Ultra models and scaled to substantially larger regimes, including longer token horizons and larger model sizes, as reported in (NVIDIA et al., 2025, arXiv:2512.20856).

The large language models behind today's AI assistants are powerful but expensive to run, and a growing share of that cost comes from serving them — answering each user query. A popular way to build bigger, smarter models without paying proportionally more for each answer is the "Mixture-of-Experts" design, where a model is split into many specialized sub-networks ("experts") and only a few are used for any given input. But these designs were mostly tuned for bulk, behind-the-scenes processing, and pay too little attention to the full range of real deployment — both serving many users at once and responding quickly to each individual one. We took a fresh look at this design by reasoning carefully about how the underlying hardware actually spends its time. This led to LatentMoE, which shrinks the experts into a more compact form. This cuts the two costs that dominate real serving — shuffling data between processors, which bottlenecks high-volume serving, and loading the experts' parameters from memory, which bottlenecks fast, interactive serving. The savings let the model use many more experts at no extra cost, which makes it more accurate in both settings. In experiments on models up to 95 billion parameters, LatentMoE delivered higher accuracy at the same serving cost, and it has since been adopted in NVIDIA's publicly released Nemotron-3 models. Our analysis further projects that at the trillion-parameter scale, a LatentMoE model could match the quality of a conventional model roughly a third larger — pointing to large potential savings in serving cost, energy, and carbon as models continue to grow.