Sampling from Your Language Model One Byte at a Time

Tokenization is used almost universally by modern language models, enabling efficient text representation using multi-byte or multi-character tokens. However, prior work has shown that tokenization can introduce distortion into the model's generations, an issue known as the Prompt Boundary Problem (PBP). For example, users are often advised not to end their prompts with a space because it prevents the model from including the space as part of the next token. While this heuristic is effective in English, the underlying PBP continues to affect code generation and languages such as Chinese, where tokens often do not line up with word and syntactic boundaries. In this work, we present an inference-time method to convert any autoregressive LM with a BPE tokenizer into a character-level or byte-level LM. Our method efficiently solves the PBP and is also able to unify the vocabularies of language models with different tokenizers, allowing one to ensemble LMs with different tokenizers at inference time or transfer the post-training from one model to another using proxy-tuning.

Language models break text into pieces called tokens before reading or writing it. This is efficient, but it can cause mistakes when a user’s prompt ends in the middle of a token-like unit, such as part of a word, a code identifier, or a Chinese phrase. As a result, the model may assign too little probability to natural continuations, even when it otherwise “knows” the right answer. We introduce ByteSampler, a method that lets a standard language model reason one byte or character at a time without retraining the model. This fixes token-boundary errors at inference time and preserves the model’s original behavior over complete text. It also makes it easier to combine or transfer behavior between language models that use different tokenizers. This matters because tokenization is usually hidden from users, but it can affect reliability in multilingual text, code generation, and systems that combine multiple models. ByteSampler provides a practical way to reduce these artifacts while reusing existing models.