Understanding Transfer Learning of RNA Foundation Models on Downstream Tasks

Foundation models (FMs) pretrained on large-scale sequence data have emerged as a promising paradigm for RNA biology, yet the mechanisms underlying their transferability remain unclear. In this work, we conduct a systematic investigation of transfer learning in RNA FMs across diverse structural and functional tasks. Our results demonstrate that frozen representations from pretrained RNA FMs are not universally transferable, and that the hierarchical feature reuse paradigm prevalent in computer vision does not generally extend to RNA FMs. Instead, pretraining primarily benefits downstream tasks by providing a favorable optimization initialization when pretraining and downstream objectives are well aligned, which accelerates convergence toward flatter minima associated with improved generalization. Overall, our findings characterize pretraining as an optimization prior whose effectiveness is governed by task alignment and model capacity, offering principled guidance for future RNA FMs.

RNA molecules play many essential roles in living cells, and predicting their properties can support biological research and future medical applications. Recent artificial intelligence models first learn from large collections of RNA sequences and are then adapted to specific prediction problems, but it remains unclear why this initial learning helps. In this work, we compare models that have already learned general RNA patterns with identical models trained from the beginning on individual tasks. Across several RNA prediction problems, models with prior learning usually perform better and reach solutions that are less sensitive to small changes in the model. These findings help explain how learning from large RNA collections can improve the reliability and effectiveness of AI tools for studying RNA.