Agentic Monte Carlo: Simulating Reinforcement Learning for Black-Box Agents

LLM agents operate in two distinct regimes: open-weight agents amenable to reinforcement learning (RL) and black-box agents whose behaviour must be controlled purely at test time. Although black-box agents are often backed by state-of-the-art proprietary LLMs, API-only access precludes parameter-level optimization, rendering most RL methods inapplicable. To address this limitation, we turn to a known equivalence between RL and Bayesian inference. We propose Agentic Monte Carlo (AMC) to directly sample from the optimal policy of a black-box agent rather than training it through RL. The optimal policy is a posterior over trajectories whose prior we define as the fixed black-box LLM agent. We employ Sequential Monte Carlo to sample from this posterior by learning a value function to steer the agent while leaving the underlying black-box model unchanged. We validate AMC on three diverse environments from the AgentGym benchmark, demonstrating significant improvements over prompting baselines and even outperforming Group Relative Policy Optimization (GRPO) as we scale the test-time compute of our method. AMC demonstrates the feasibility of performing principled RL-style optimization of black-box LLM agents.

Closed AI models like ChatGPT are difficult to improve because outside researchers cannot access or change their internal functions. Typically, AI models’ internal functions are modified using reinforcement learning (RL), a highly effective process where AI learns through trial-and-error feedback. To overcome this, we introduce Agentic Monte Carlo (AMC). Instead of trying to modify the AI model, AMC partners it with a smaller guiding model that can help steer the large model. AMC works by simultaneously testing slightly different approaches the large model takes for the same task. The guiding model evaluates these attempts step-by-step, allowing AMC to discard less promising solutions and continue exploring more successful ones according to the guide. This process achieves similar benefits to if we had done RL training on the original large model. When tested on complex tasks like e-commerce and scientific reasoning, AMC showed comparable performance with methods that require full access to the internal functions. Furthermore, AMC allows cheaper AI systems to perform like advanced ones, reducing costs by over 50%. This proves that closed AI can be improved, making high-performance technology accessible to researchers on a budget.