Learning to Route Languages for Multilingual Policy Optimization

Large language models~(LLMs) are trained on heterogeneous multilingual corpora, yet existing policy optimization methods often implicitly restrict each training question to a single response language or rely on a fixed dominant language for supervision. We propose language-routed policy optimization (LRPO), an online reinforcement learning framework that treats language as a selectable variable. LRPO elicits multilingual rollouts for each training question and integrates their relative quality into preference-based policy updates, increasing the diversity and informativeness of training signals under the fixed rollout budget. To adaptively determine which languages to explore during reinforcement learning, we introduce a trainable language router formulated as a multi-armed bandit, balancing exploration of underutilized languages with exploitation of more informative ones. Extensive experiments show that LRPO consistently improves multilingual performance, demonstrating that adaptive language routing enables effective cross-lingual knowledge exploitation for training. We release all the resources at \url{https://github.com/Guochry/LRPO}.

AI assistants are used around the world, but they do not understand and use every language equally well. When improving these models, current training methods often ask the model to answer each question in the same input language or assume that English is always the best language to learn from. This can miss useful knowledge that the model may already have in other languages. We introduce LRPO, a training method that lets the model try answering the same question in multiple languages and learn from the most helpful responses. To identify helpful responses, LRPO adjusts the scoring process so that answers written in different languages can be compared more fairly. LRPO also introduces a language router, which learns which languages are most useful for different kinds of questions, such as math, general knowledge, or region-specific facts. Across several multilingual benchmarks and model families, LRPO improves overall performance. This suggests that improving multilingual AI requires explicitly modeling how knowledge differs across languages during training, rather than treating language choice as a fixed detail.