AgentVocab: Structure-Aware Vocabulary Adaptation for Efficient LLM Agents

Recent large language models (LLMs) have demonstrated strong capabilities across challenging tasks, enabling their widespread adoption in agentic systems that interact with external tools. In such deployments, however, LLMs are typically trained with general-purpose tokenizers designed for broad language coverage, while their usage is dominated by narrow, structured tool-calling interactions. This training–deployment mismatch leads to inefficient tokenization, where repetitive structural patterns and frequent semantic units in function calls are fragmented into long sequences of low-level tokens, increasing decoding overhead. To address this gap, we introduce $\textbf{AgentVocab}$, a structure-aware vocabulary adaptation framework for efficient LLM agents. AgentVocab derives specialized vocabulary entries from real tool-calling traces and adapts the model vocabulary to better reflect structural and semantic regularities, without task-specific schema engineering. Experiments on $\tau$ and $\tau^2$-bench show that AgentVocab significantly improves decoding efficiency, reducing latency by approximately 15-25\% relative to the vanilla baseline, while preserving tool-calling performance. Our approach is orthogonal to existing fine-tuning and agent-training methods and integrates seamlessly into standard agent pipelines. Source code and models will be available at https://anonymous.4open.science/r/AgentVocab-28CC.