TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination

Multi-agent LLM systems can improve reasoning and tool use, yet recent evidence shows their gains are often unstable and sensitive to interaction design. A promising direction is to train collaboration, but team post-training introduces a moving-target effect: when agents interact through a shared context, updating one agent shifts the context distribution faced by the others, which can regress coordination under naive sequential updates. We propose TeamTR, a trust-region framework for fine-tuning heterogeneous LLM teams that explicitly controls this occupancy shift. TeamTR evaluates each agent update on rollouts from the intermediate team induced by partially applied updates, and enforces per-agent trust regions via a token-decomposed reverse KL that is directly monitorable from those rollouts. This yields population-level per-update and per-stage improvement lower bounds whose functional form applies to any realized update order, and motivates a practical certificate proxy computed from logged surrogates and KL terms. We instantiate TeamTR for router-based text handoff with sequence-level returns and bounded group-normalized advantages, and show empirically that it mitigates coordination regressions, improves training stability across heterogeneous teams, and supports modular component replacement via a trust-region alignment step.

Agentic systems are increasingly built as teams: several language models with different specialties working together to solve hard problems like math, logic puzzles, and step-by-step planning. Surprisingly, some research reported that these multi agent system often perform worse than a single strong model. We set out to understand why, and uncovered a hidden flaw in how AI teams are trained. When trainers improve the team's members one at a time, each change shifts what conversations the team actually has. But trainers typically judge the next member's progress against stale conversation records collected before those earlier changes happened. It is a bit like coaching a basketball player using last season's game footage after half the lineup has been replaced. The longer the chain of updates, the bigger the mismatch grows. We show mathematically that this gap balloons quickly as the team gets larger, which explains why naïvely trained AI teams can fall apart even when each individual change looks like an improvement. Our method TeamTR, is straightforward: refresh the practice data after every teammate is updated, and prevent any single update from changing a member too drastically at once. This keeps every adjustment grounded in what the team is actually doing now. The result: AI teams that consistently outperform both single models and existing team-training methods by about 7% on average, train more stably, and even support swapping in upgraded members, like substituting a stronger player into the starting lineup without disrupting the rest of the team.