Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for Frozen LLMs

Large Language Models (LLMs) have demonstrated remarkable general capabilities, but enhancing skills such as reasoning often demands substantial computational resources and may compromise generalization. While Parameter-Efficient Fine-Tuning (PEFT) methods offer a more resource-conscious alternative, they typically require retraining for each LLM backbone due to architectural dependencies. To address these challenges, we propose Universal Reasoner (UniR)—a lightweight, composable, and plug-and-play reasoning module that can be used with larger frozen LLMs to provide specialized reasoning capabilities. Specifically, UniR decomposes the reward into a standalone reasoning module trained in a decoupled manner using verifiable rewards, effectively translating trajectory-level signals into token-level guidance. Once trained, UniR is combined with frozen LLMs at inference time by simply adding its output logits to those of the backbone. This additive structure enables modular composition: multiple UniR modules trained for different tasks can be jointly applied by summing their logits, enabling complex reasoning via composition. Furthermore, UniR demonstrates weak-to-strong generalization, where reasoning modules trained on smaller models effectively guide much larger LLMs in the same model family, and generalize across domains such as in vision language models and medical reasoning. Experiments on mathematical reasoning and machine translation show that UniR surpasses existing fine-tuning methods. Code is open-sourced at https://github.com/hangeol/UniR.

Large language models can solve increasingly difficult reasoning tasks, but improving their reasoning ability often requires expensive fine-tuning of the entire model. Existing lightweight fine-tuning methods are more efficient, but they are usually tied to one specific model architecture and are difficult to reuse across larger models or combine across tasks. We introduce Universal Reasoner (UniR), a small plug-and-play reasoning module that guides a frozen language model without changing its parameters. UniR is trained using automatically checkable rewards, such as whether a math answer is correct or whether a translation is accurate. At inference time, UniR steers the backbone model by adding its token-level guidance to the model’s own predictions. This design makes reasoning more modular. A UniR module trained with a smaller model can improve larger models in the same family, and multiple UniR modules can be combined for tasks requiring different skills, such as translating and solving math problems. Experiments show that UniR improves mathematical reasoning, translation, visual math reasoning, and medical prediction tasks while preserving the general abilities of the frozen backbone.