$\tau$-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge

Conversational agents are increasingly deployed in knowledge-intensive settings, where correct behavior depends on retrieving and applying domain-specific knowledge from large, proprietary, and unstructured corpora during live interactions with users. Yet most existing benchmarks evaluate retrieval or tool use independently of each other, creating a gap in realistic, fully agentic evaluation over unstructured data in long-horizon interactions. We introduce $\tau$-Knowledge, an extension of $\tau$-Bench for evaluating agents in environments where success depends on coordinating external, natural-language knowledge with tool outputs to produce verifiable, policy-compliant state changes. Our new domain, $\tau$-Banking, models realistic fintech customer support workflows in which agents must navigate roughly 700 interconnected knowledge documents while executing tool-mediated account updates. Across embedding-based retrieval and terminal-based search, even frontier models with high reasoning budgets achieve only $\sim$25.5\% pass^1, with reliability degrading sharply over repeated trials. Agents struggle to retrieve the correct documents from densely interlinked knowledge bases and to reason accurately over complex internal policies. Overall, $\tau$-Knowledge provides a realistic testbed for developing agents that integrate unstructured knowledge in human-facing deployments.

AI assistants are increasingly being used in settings like banking customer support, where they must do more than answer simple questions. To behave correctly, they often need to search through large collections of company-specific documents, understand the relevant policies, use software tools, and take the right action during a live conversation. Existing benchmarks usually test these skills separately, but real customer interactions require agents to combine them over many turns. We introduce τ-Knowledge, a benchmark that tests whether AI agents can use written knowledge together with tool results to complete realistic tasks. Our new environment, τ-Banking, simulates fintech customer support workflows where agents must navigate about 700 connected policy and support documents while making account updates through tools. These tasks are designed so that success can be checked automatically: the agent must produce the correct, policy-compliant change in the system. We find that this is still very difficult. Even strong frontier models succeed only about 25.5% of the time on the first try, and their reliability drops across repeated attempts, showing that today’s agents still struggle to find the right information and apply complex policies consistently.