Gecko: A Simulation Environment with Stateful Feedback for Refining Agent Tool Calls

The ability to use tools is fundamental for large language model (LLM) agents. Given a task, existing systems use LLMs to plan and generate tool calls, which are executed by real-world tools to complete the task. However, tool calls are prone to errors because they are generated primarily from the intrinsic capabilities of LLMs. Moreover, while it is useful to let LLMs iteratively refine the tool-call sequence using execution results from real tools, this process can be expensive and may cause unsafe side effects. To improve LLM tool calls and address issues caused by using real tools for refinement, we introduce Gecko, a stateful simulation environment that provides informative feedback for refining LLM tool calls before real execution. Specifically, Gecko combines rules and LLMs to check the validity of tool names and arguments, synthesize schema-conforming and state-consistent responses, and judge task completion against the user objective. These three types of feedback allow LLMs to refine their tool calls in simulation, forming a simple yet effective test-time scaling method named GATS. On BFCLv3 and $\tau^2$-bench, GATS consistently improves the performance of various LLMs, including GPT-4o, GPT-5, and Gemini-3.0-pro.

Large language model agents often need to use external tools, but they can make mistakes when choosing tools or providing inputs. Directly using real tools to debug and refine these mistakes can be costly and may cause unwanted side effects. We introduce Gecko, a simulation environment where agents can safely test and improve their planned tool use before final real execution. Gecko checks whether tool calls are valid, gives realistic feedback, tracks task progress, and judges whether the task is completed. This allows agents to refine their tool-use plans in simulation and then use the improved plans with real tools. Experiments show that this simulated refinement improves the tool-use performance of several strong AI models.