GENEB: Why Genomic Models Are Hard to Compare

Progress in genomic foundation models is difficult to assess due to fragmented benchmarks, incompatible evaluation protocols, and task-specific reporting. As a result, claims of superiority or generality across models are often not directly comparable. We introduce GENEB, a large-scale diagnostic benchmark that evaluates frozen representations from 40 genomic foundation models across 100 tasks spanning 13 functional categories under a unified probing-based protocol, including few-shot regimes. GENEB enables controlled comparison across model scale, architecture, tokenization, and pretraining data while explicitly exposing task-level trade-offs. Our analysis shows that aggregate leaderboards are unstable: model rankings vary sharply across task categories, scale provides only modest and inconsistent gains, and architectural and pretraining alignment frequently outweigh parameter count. These results highlight limitations of current evaluation practices and position GENEB as a reference framework for principled comparison and category-aware model selection in genomic machine learning.

Researchers have built dozens of "foundation models" - large neural networks trained on DNA sequences - to help with biological prediction tasks, such as identifying where genes start or which species a sequence came from. The trouble is that every paper introducing a new model tests it on a different set of tasks, with different evaluation rules, so it is impossible to tell which model is actually better and at what. We built GENEB, a benchmark that evaluates 40 such models on the same 100 DNA prediction tasks, grouped into 13 biological categories, under one shared protocol. This lets us compare models fairly across size, architecture, the type of "vocabulary" they use to read DNA, and the kinds of organisms they were trained on. Our central finding is that bigger is not always better: the model that wins on one category often loses on another, and a model's design and training data frequently matter more than its size. This means single overall leaderboards can be misleading, and practitioners should pick a model based on the specific biological problem they care about. We release GENEB as a public reference so that future progress in genomic machine learning can be measured consistently.