PinTok: Tokenizers Deserve Dedicated Pinned CPU-Compute and Memory

Tokenization is the first point of contact between large language models (LLMs) and text data, yet it has not been viewed by many as a component of LLMs worth accelerating. During inference, tokenizers typically rely on simple dictionary lookups and are executed on CPUs as standard processes. This approach, however, introduces significant overhead from scheduling delays, core selection, data copying, and other system-level costs. These inefficiencies become problematic in latency-sensitive applications such as embedding, small language models, and agentic AI. In this paper, we present the Pinned Tokenizer (PinTok), a novel tokenizer architecture that reduces redundant hardware, operating system, and networking overhead through three key innovations: core and memory pinning, scheduling and context switch avoidance, and duplicate network packet copy and processing avoidance. Our implementation of PinTok can serve as a drop-in replacement for existing tokenizer deployments, delivering latency reductions of up to 95% (average), 97% (P50), 94% (P90), and 87% (P99) along with throughput improvements of up to 2,084%.

While tokenization is one of the main step in LLM inference, it has not been viewed as a critical piece of logic worth accelerating. We wanted to raise multiple systematic issues with exiting tokenization processes and provide generalizable methods to accelerate tokenization. Our paper presents a novel tokenization architecture to enable faster tokenization by pinning CPU and RAM to for tokenization and running tokenization as part of the network stack. This enables not only faster tokenization, but also predictable tokenization latencies, which are critical for latency sensitive LLM applications. Our contributions have implications on how we improve LLM performances, especially when there are more and more agentic applications that chain series of LLM calls in sequence.