Tvcache: A Tool-Value Cache for Post-Training LLM Agents

In RL post-training of LLM agents, calls to external tools take several seconds or even minutes, leaving allocated GPUs idle and inflating post-training time and cost. While many tool invocations repeat across parallel rollouts and could in principle be cached, naively caching their outputs for reuse is incorrect since tool outputs depend on the environment state induced by prior agent interactions. We present TVCACHE, a stateful tool-value cache for LLM agent post-training. TVCACHE maintains a tree of observed tool-call sequences and performs longest-prefix matching for cache lookups: a hit occurs only when the agent’s full tool history matches a previously executed sequence, guaranteeing identical environment state. On three diverse workloads—terminal-based tasks, SQL generation, and video understanding—TVCACHE achieves cache hit rates of up to 70% and reduces median tool call execution time by up to 6.9×, with no degradation in post-training reward accumulation.

Modern AI agents are becoming more powerful because they can use tools to interact with the world beyond text, such as running code, querying databases, and analyzing videos. This capability is taught during post-training, where the agent repeatedly tries to solve the same tasks using a given set of tools. Across these repeated attempts, agents often rerun the same expensive tool operations; while those tools execute on external systems, the accelerators assigned to generate the agent's next tokens must wait for the tool outputs. Caching could reuse tool results from earlier attempts and reduce this waiting time on later invocations, but doing so safely is challenging. A simple cache that remembers only the tool name and input can return incorrect results, because the same tool call may appear in different attempts after different sequences of earlier tool calls. We built TVCache, a system that safely reuses repeated tool results during agent post-training. Instead of caching each tool call in isolation, TVCache tracks the sequence of tool calls that led to the current state of the environment where tools execute, so it only reuses a result when the previous steps match correctly. It also intelligently saves selected snapshots of this tool-execution environment, so it can restart from useful intermediate points instead of rerunning everything. Across coding, database, and video-understanding tasks, TVCache reduces median tool-call execution time by up to $6.9\times$ with no loss in training performance. This can make training AI agents that use tools, faster, cheaper, and more efficient.