Post-Training LLMs as Better Decision-Making Agents: A Regret-Minimization Approach

Large language models (LLMs) are increasingly deployed as agents for decision-making (DM) in interactive and dynamic environments. However, since they are not originally designed for DM, recent studies show that LLMs struggle in basic online DM settings. We introduce ITERATIVE REGRET-MINIMIZATION FINE-TUNING (ITERATIVE RMFT), a post-training procedure that repeatedly distills low-regret decision trajectories into the base model. Unlike prior methods that rely on distilling known algorithms or enforcing manually designed reasoning formats, our approach leverages regret as a training signal to elicit improved decision-making behavior while incorporating model-generated reasoning in natural language. Empirically, ITERATIVE RMFT improves DM performance across models, including numerical Transformers, lightweight open-weight LLMs, and the closed-weight model GPT-4o mini, while exhibiting generalization across varying horizons, action spaces, reward processes, and natural-language-described DM scenarios. Overall, we position our approach as an initial exploration, calling for more principled and novel post-training paradigms for LLMs when it comes to addressing DM tasks.

AI language models are increasingly being asked to work to solve a specific task: for example, choosing what to do next after seeing feedback from a changing situation. But these models are mostly trained to predict text, so they can make poor choices when a task requires learning from trial and error over many steps. We study how to teach these models to make better decisions after they are built. Our method gives the model extra training on examples of decision paths that led to low regret, meaning fewer missed opportunities compared with better choices that could have been made. The training is repeated: the model generates decisions and explanations, we identify better low-regret behavior, and then we train the model to imitate that behavior. This lets the model improve without being forced to copy a hand-written decision rule or a fixed reasoning template. We find that this approach improves several kinds of models, from small open models to GPT-4o mini. The gains also carry over to tasks with different lengths, choices, reward patterns, and natural-language descriptions. Our results suggest that regret-based training is a promising first step toward making language models more reliable decision-making agents.