Multi-Task GRPO: Reliable LLM Reasoning Across Tasks

RL-based post-training with GRPO is widely used to improve large language models on individual reasoning tasks. However, real-world deployment requires reliable performance across diverse tasks. A straightforward multi-task adaptation of GRPO often leads to imbalanced outcomes, with some tasks dominating optimization while others stagnate. Moreover, tasks can vary widely in how frequently prompts yield zero advantages (and thus zero gradients), which further distorts their effective contribution to the optimization signal. To address these issues, we propose a novel Multi-Task GRPO (MT-GRPO) algorithm that (i) dynamically adapts task weights to explicitly optimize worst-task performance and promote balanced progress across tasks, and (ii) introduces a ratio-preserving sampler to ensure task-wise policy gradients reflect the adapted weights. Experiments on both 3-task and 9-task settings show that MT-GRPO consistently outperforms baselines in worst-task accuracy. In particular, MT-GRPO achieves 16–28\% and 6\% absolute improvement on worst-task performance over standard GRPO and DAPO, respectively, while maintaining competitive average accuracy. Moreover, MT-GRPO requires 50\% fewer training steps to reach 50\% worst-task accuracy in the 3-task setting, demonstrating substantially improved efficiency in achieving reliable performance across tasks.

Modern AI language models are general‑purpose, but the extra training used to sharpen their reasoning is usually done one skill at a time, for example on math problems or science questions. If you instead try to improve several of these skills at once in a single model, a common problem shows up: it keeps getting better at whichever tasks come easiest and barely improves on the harder ones. The result is a model that looks fine on average but is unreliable, because there is always some skill it never really picked up. This work is about training one model to be reasonably good at all of its tasks, not just good on average. Our method, MT‑GRPO, does this by steering more of the training toward the tasks the model is currently worst at, and by making sure that extra effort actually reaches the model during training. (In standard methods, some hard tasks can end up contributing almost nothing to learning, so just telling the model to focus on them is not enough.) We tried this across different model sizes and on mixtures of math, science, and reasoning tasks. In each case the model's weakest skill improved while its average performance stayed about the same, and it reached that balanced state faster than the methods we compared against.