RealtimeTool: Parallel Decoding for Real-Time LLM Function Calling

LLM-based function calling enables intelligent agents to interact with external tools and environments, yet autoregressive decoding imposes a fundamental latency bottleneck that limits real-time applications such as embodied intelligence, game AI, and interactive avatars (e.g., 10 Hz control frequency). We observe that function calling differs fundamentally from free-form text generation: structured outputs exhibit substantial token redundancy (delimiters, parameter names) and weak causal dependencies among arguments---two properties that must be exploited jointly to achieve real-time performance. We present RealtimeTool, which introduces special tokens that serve a dual role: compressing low-entropy tokens (4--6× reduction) while acting as mode selectors that enable independent parallel generation of function name and arguments. This synergistic design achieves 3--6× end-to-end speedup (up to 9.6×) with only +8.2% parallelization overhead, while maintaining competitive or improved accuracy across five benchmarks on Qwen-series models (0.5B--14B). With quantization on a consumer-grade GPU, RealtimeTool reaches 61.2 ms P50 latency at 4B scale---enabling 16 Hz real-time control and bridging the gap between LLM function calling and latency-critical real-world deployment.

LLM can call external tools---looking up the weather, controlling a smart home device, or moving a robot arm---but they do this by writing out their answer step by step, with each step depending on the previous one. For a chatbot the delay is tolerable, but for a robot or a game character that needs to react several times per second, even half a second of latency makes the system unusable. We noticed two things about these single-user, real-time scenarios. First, the GPU running the model spends most of its time waiting on memory rather than computing, leaving a large amount of computing power idle. Second, when a language model calls a tool, the different pieces of its answer (the tool's name, the values to pass in) don't really depend on each other in the way that words in a sentence do---so they don't actually need to be produced one after another. We exploited both observations together: we taught the model to generate the different parts of a tool call in parallel, using the otherwise-idle computing power, rather than one piece at a time. On an ordinary gaming GPU (an RTX 4090), a 4-billion-parameter model can now issue up to 16 tool calls per second---fast enough to control robots, game characters, and other systems that need to react in real time.