MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unification

Mixture-of-Experts (MoE) models scale capacity by combining specialized experts, but most existing approaches assume centralized access to training data. In practice, data are distributed across clients and cannot be shared due to privacy constraints, making unified MoE training challenging. We propose MetaMoE, a privacy-preserving framework that unifies independently trained, domain-specialized experts into a single MoE using public proxy data as surrogates for inaccessible private data. Central to MetaMoE is diversity-aware proxy selection, which selects client-domain–relevant and diverse samples from public data to effectively approximate private data distributions and supervise router learning. These proxies are further used to align expert training, improving expert coordination at unification time, while a context-aware router enhances expert selection across heterogeneous inputs. Experiments on computer vision and natural language processing benchmarks demonstrate that MetaMoE consistently outperforms recent privacy-preserving MoE unification methods. Code is available at https://github.com/ws-jiang/MetaMoE.

Organizations often train their own specialized AI models on private data they cannot share due to legal or confidentiality constraints. Ideally, these separately trained models could be combined into a single, more capable system, but doing so typically requires access to all the original training data, which is off-limits. We propose MetaMoE, a framework that merges multiple privately trained expert models into one unified system without ever accessing private data. Instead, we select carefully chosen samples from publicly available data to stand in for each organization's private data. Our selection method balances two goals: picking public samples that resemble the private data, and ensuring those samples are diverse enough to cover the full range of each organization's domain. These public stand-ins then guide a routing mechanism that learns when to consult which expert for a given input. We also align each expert's training with these public samples so that the experts and the router work well together from the start. Our approach consistently outperforms existing methods on both image recognition and language understanding tasks, demonstrating that effective multi-expert collaboration is possible even when the underlying data must remain strictly private.