TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs). Despite the importance of tokenization, its role in LM performance and behavior is poorly understood due to the challenge of measuring the impact of tokenization in isolation. To address this need, we present TokSuite, a collection of models and a benchmark that supports research into tokenization's influence on LMs. Specifically, we release fourteen pre-trained models that use different off-the-shelf tokenizers but are otherwise identical, using the same architecture, dataset, training budget, and initialization. We also release a multilingual robustness benchmark that measures model performance under real-world perturbations in English, Chinese, Farsi, Italian, and Turkish, curated by native annotators. Together, TokSuite allows robust decoupling of the influence of a model's tokenizer, supporting a series of novel findings that elucidate the respective benefits and shortcomings of a wide range of popular tokenizers.

Before a language model reads any text, it first splits it into small pieces called "tokens" from a fixed, limited vocabulary. Unlike humans, who can encounter a new word and learn it as its own, language models can only work with units they already know. For example, the word "doctor" might be split into "doc" and "tor". This process, called tokenization, is one of the earliest decisions made when building a language model, yet it is often treated as an afterthought, with many models simply borrowing whatever tokenizer a previous model used. We study how much this choice actually matters. The challenge is that existing models differ in too many ways (their size, training data, architecture), making it hard to pin performance differences on the tokenizer alone. So we trained 14 models that are identical in every way except for their tokenizer, and paired them with a new benchmark of about 5,000 test cases covering real-world language variations across five languages, as well as mathematical notation and scientific content, all curated by native speakers. We found that tokenizer design consistently shapes how robust a model is to everyday imperfections like typos, foreign scripts, or formatted equations, more so than model size or training duration. Even a tiny whitespace difference inside a math formula could cause a model to fail completely. We hope TokSuite helps the community make more informed tokenizer choices in the future.