What Language is This? Ask Your Tokenizer.

Language Identification (LID) is an important component of many multilingual natural language processing pipelines, where it facilitates corpus curation, training data analysis, and cross-lingual evaluation of large language models. Despite near- perfect performance on high-resource languages, existing systems remain brittle in low-resource and closely related language settings. We introduce UniLID, a simple and efficient LID method based on the UnigramLM tokenization algorithm, leveraging its probabilistic framing, parameter estimation technique and inference strategy. In short, to predict a string’s language label, we simply ask: under which language’s unigram distribution is this string most likely? Our formulation is data- and compute-efficient, supports incremental addition of new languages without retraining existing models, and can naturally be integrated into existing language model tokenization pipelines. Empirical evaluations against widely used baselines, including fastText, GlotLID-M, and CLD3, show that UniLID achieves competitive performance on standard benchmarks, substantially improves sample efficiency in low-resource settings—surpassing 70% accuracy with as few as five labeled samples per language—and delivers large gains on fine-grained dialect identification.

When AI systems are trained on text from the web, the first step is figuring out which language each piece is in. Current tools do this well for common languages like English, but stumble on languages which are less widely spoken, closely related (like Croatian vs. Serbian), or only observed in a short (potentially messy) text passage. As a result, many of the world's languages get left out of modern AI systems. We wondered if a different approach to language identification could help. AI systems read text by breaking it into small pieces, and different languages use different pieces. So we ask: which language's typical pieces fit a given text best? Surprisingly, this works very well. With as few as five example sentences per language (far less than existing tools require) our method, UniLID, identifies languages correctly most of the time, and is much better at telling apart closely related dialects. We are releasing UniLID as open source so others can use it to build more inclusive AI systems.