Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models

Byte-level tokenization enables language models to handle any Unicode input, but models can generate invalid UTF-8 sequences when encountering rare or unseen characters. We investigate the relationship between training scale and UTF-8 generation reliability with a 355M parameter model trained on 80B tokens from a balanced multilingual corpus of English, Japanese, Korean, and Chinese. We introduce multiple evaluation protocols that isolate UTF-8 structural validity from language modeling. UTF-8 validity convergence lags perplexity by a roughly a factor of two: perplexity stabilizes after 2.1B tokens, but UTF-8 validity requires 4.2B tokens. In context-free generation, rare characters achieve higher structural validity than common characters, suggesting over-specialization of frequent character representations. Through experiments, we observed that reliable UTF-8 generation is a distinct capability requiring evaluation beyond perplexity.

When language models read or write text in languages like Chinese, Japanese, or Korean, they often don't have a built-in piece for every character. Instead, they fall back to spelling each character out as a short sequence of raw bytes, the same way computers store text underneath the surface. The model has to produce those bytes in exactly the right order, or the output comes out as garbled junk. We trained a small multilingual model and tracked, step by step, how long it takes to learn this skill. Two findings stood out. First, learning to "sound fluent" on average happens roughly twice as fast as learning to spell rare characters out correctly, so a model can look well-trained on standard quality scores and still produce broken characters. Second, even when the byte sequence is technically valid, the model frequently picks the wrong character. Tests on ten public models from five different labs, ranging up to 9 billion parameters, show the same pattern. Reliable handling of rare characters is a distinct capability that needs to be measured on its own, not assumed.