RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing

Sparse Mixture of Experts (MoE) models scale more efficiently than dense models by routing tokens to modular expert networks that are only active when relevant to the task. A leading hypothesis for the performance of MoE models is that each expert specialises in a single, coherent domain. However, interpretability efforts that assume this hypothesis have generally been unsuccessful. We propose and present evidence for an alternative account that we call the Superposed Specialisation Hypothesis (SSH): experts specialise in a disjoint union of fine-grained features rather than one broad domain. Leveraging the SSH, we introduce RouterInterp, a method for interpreting expert routing that identifies Sparse Autoencoder features most predictive of routing decisions and produces unified natural language explanations. On gpt-oss-20b, RouterInterp explains expert routing with 57% higher detection accuracy than prior token statistics based methods. This work provides a scalable method for generating concise and more accurate explanations of expert routing and increases our understanding of a previously uninterpretable component of foundation models.

Today's large language models stay efficient by sending each word they process to just a handful of internal sub-networks, called experts. A small built-in component called the router decides which experts to use on the fly. Many researchers assumed each expert specialises in a single field (e.g. medicine or law) — but nobody has been able to confirm this, leaving the routing process a black box. By analysing which inputs the router sends to each expert across thousands of examples, we found that experts don't specialise in one field at all. Instead, each expert handles small parts of many unrelated fields at once. The router itself has no built-in labels for its experts: it operates purely on numbers, so there is no off-the-shelf way to describe what any given expert is for. To fix this, we built RouterInterp, a method that automatically generates descriptions of what each expert is in plain English. RouterInterp explains routing decisions more accurately than any previous method, giving developers and the public a clearer window into the inner workings of modern AI systems.