DNACHUNKER: Learnable Tokenization for DNA Language Models

DNA language models are increasingly used to represent genomic sequence, yet their effectiveness depends critically on how raw nucleotides are converted into model inputs. Unlike natural language, DNA offers no canonical boundaries, making fixed tokenizations a brittle design choice under shifts, indels, and local repeats. We introduce DNAChunker, a masked DNA language model that incorporates a learnable adaptive segmentation module to produce context-dependent, variable-length units. Building on a dynamic segmentation procedure, DNAChunker learns to allocate finer granularity to functionally enriched regions while compressing repetitive or redundant sequence. We pretrain DNAChunker on the human reference genome and evaluate it across five benchmarks, where it consistently improves over strong fixed-tokenization baselines. Further analyses and ablations indicate that unlike fixed tokenizations, segmentation is learned in a biologically-informed, mutation-resilient manner.

DNA is often compared to a language, but unlike English, it does not come with clear spaces between “words.” This makes it difficult for AI models to decide how to break long DNA sequences into useful pieces before learning from them. In this work, we introduce DNAChunker, a DNA language model that learns how to split DNA into meaningful chunks instead of relying on fixed, hand-designed rules. The model keeps more detail in biologically important regions, such as regulatory signals, while compressing repetitive or less informative parts of the genome. This helps the model study long DNA sequences more efficiently without losing important local information. Across five genomics benchmarks, DNAChunker performs better than strong existing models on many tasks, including predicting regulatory activity, splice sites, epigenetic marks, and long-range genomic interactions. We also find that its learned chunks often preserve known biological motifs as single units and remain more stable when DNA is mutated. These results suggest that allowing AI models to learn their own “DNA words” can lead to more accurate, efficient, and biologically meaningful genomic prediction tools.