annDNA: Learning Annotation-Aware Genomic Representations via Knowledge Distillation

Genomic language models (GLMs) learn contextual representations of DNA sequences, but current approaches rely solely on sequence patterns without incorporating known genomic and functional annotations. To address this, we present annDNA, which involves two stages: (1) annotation-aware pre-training that creates tokens explicitly encoding functional information from GENCODE and ENCODE, and (2) cross-modal knowledge distillation that transfers these annotation-aware representations to a sequence-only model. Annotation-aware models achieve 15.5% higher AUROC than sequence-only baselines in variant effect prediction. The distilled model, with one-third of the parameters, achieves an 11.2% improvement over the sequence-only baseline while requiring only sequence input at inference. Our results demonstrate the effectiveness of using annotations during training, offering a general framework for transferring biological knowledge to sequence-only models.