BFCL Audio: An Audio Function Calling Evaluation for Large Language Models

Audio agents are increasingly deployed to execute tools from spoken requests, yet audio tool use poses challenges beyond text-only function calling: perception errors (e.g., homophones, noise, disfluencies) can corrupt entities and arguments, and natural interactions often require clarification that changes the tool-calling protocol. We introduce BFCL Audio, a large-scale benchmark for audio function calling with 6.2K expert-verified tasks across two suites that mirror common deployments: BFCL Text Audio (pipelined $\text{ASR}\rightarrow\text{LLM}\rightarrow\text{tools}$ via transcripts) and BFCL True Audio (end-to-end $\text{audio-in}\rightarrow\text{tool calls}$). BFCL Audio includes controlled speech and acoustic perturbations (accent and speaking-rate variation, content disfluencies, and background noise) generated through a controllable audio synthesis/augmentation pipeline. We provide automatic grading for both function names and argument values using AST-based matching for single-turn calls and response/state-based metrics for multi-turn interactions, enabling scalable evaluation without LLM judges. Across a broad set of models, we propose a failure-mode taxonomy and analyze which speech and noise factors most strongly impact tool-calling accuracy. We release the benchmark, evaluation harness, and audio pipeline to support research on reliable speech-based agents.

When you ask a voice assistant to do something ("book a flight to Austin"), it has to turn your spoken words into an exact command. But speech is messy — accents, stumbles, background noise — so the assistant can mishear a detail and confidently do the wrong thing (book "Boston" instead). Until now there was no good way to test how often this happens or why. The authors built BFCL Audio: a kit of ~6,200 tasks that deliberately adds realistic noise, accents, and stumbles to measure how reliably voice assistants carry out spoken requests, and grades them automatically. Key finding: noise makes every system noticeably worse (even the best dropped ~10%), and the worst mistakes are quiet ones — mishearing names or numbers, or babbling instead of acting. The test kit is being released publicly so others can build more trustworthy voice assistants.