Strat-Reasoner: Reinforcing Strategic Reasoning of LLMs in Multi-Agent Games

While Large Language Models (LLMs) excel in certain reasoning tasks, they struggle in multi-agent games where the final outcome depends on the joint strategies of all agents. In multi-agent games, the non-stationarity of other agents brings significant challenges on the evaluation of the reasoning process and the credit assignment over multiple reasoning steps. Existing single-agent reinforcement learning (RL) approaches and their multi-agent extensions fail to address these challenges as they do not incorporate other agents in the reasoning process. In this work, we propose Strat-Reasoner, a novel RL-based framework that improves LLMs' strategic reasoning ability in multi-agent games. We introduce a novel recursive reasoning paradigm where an agent's reasoning also integrates other agents' reasoning processes. To provide effective reward signals for the intermediate reasoning sequences, we employ a centralized Chain-of-Thought (CoT) comparison module to evaluate the reasoning quality. Finally, we compute an accurate hybrid advantage and develop a group-relative RL approach to optimize the LLM policy. Experimental results show that Strat-Reasoner substantially improves strategic abilities of underlying LLMs, achieving 22.1\% average performance improvements across various multi-agent games. Code is publicly available at https://github.com/ydhe1012/Strat-Reasoner.

Large language models (LLMs) can solve many reasoning tasks, but they often struggle in games or real-world situations where success depends on other agents’ strategies. In these settings, an agent must think not only about what it should do, but also about what others may believe, intend, and do next. We propose Strat-Reasoner, a reinforcement learning framework that improves this kind of strategic reasoning in LLM-based agents. Strat-Reasoner teaches an agent to reason recursively about other agents’ reasoning processes. It also gives the agent intermediate feedback by comparing its predictions about another agent’s reasoning with that agent’s actual reasoning, rather than relying only on the final game outcome. This feedback is combined with standard game rewards to train better policies. Across several multi-agent games, Strat-Reasoner improves average strategic performance by 22.1\%, showing that large language models can become stronger strategic decision makers when they are trained to reason about others.