Scaling Laws and Architectural Frontiers in Metagenomic Foundation Models

Foundation models for genomics have the potential to revolutionize therapeutic design, yet the optimal architectural choices for modeling the vast and diverse distribution of metagenomic data remain under-explored. In this work, we present the machine learning methodology behind EDEN, a family of metagenomic foundation models scaled up to 28 billion parameters and trained on 9.7 trillion nucleotide tokens. We provide a systematic empirical study of architectural trade-offs between autoregressive Transformers (Llama-style), State-Space Models (Mamba), and Long-convolutional architectures (Hyena) for nucleotide-level modeling. Contrary to recent trends favoring linear-time sequence models for long-range biological data, we demonstrate that the Llama architecture exhibits superior scaling efficiency and semantic retrieval capabilities as the model capacity grows. We derive a set of quality-aware scaling laws for metagenomics, showing how model performance follows predictable power-law behavior across three orders of magnitude in parameters and data. Through extensive benchmarking, spanning unsupervised zeroshot fitness prediction, semantic completion, and gene recovery, we establish a blueprint for scaling biological foundation models and provide empirical evidence demonstrating why Transformer-based architectures define the current frontier.

The microbial world is staggeringly vast - a single gram of soil can hold thousands of unknown species - yet most of this genetic diversity, sometimes called "microbial dark matter," remains a mystery. Hidden within it could be the next generation of antibiotics, therapies, and biotechnologies, but we lack the tools to read and understand DNA at this scale. AI models similar to those behind modern chatbots can in principle learn the "language" of DNA, but no one had systematically tested which type of model design works best for the enormous diversity of microbial genetic data. We built EDEN, a family of AI models trained on 9.7 trillion DNA "letters" from across the microbial world, scaling up to 28 billion parameters. We carefully compared three leading model designs at multiple sizes, and - contrary to recent trends favoring newer, faster alternatives - found that the same Transformer design that powers today's chatbots learned biology most faithfully as it grew larger. We also showed that the quality and diversity of training data matters just as much as model size. EDEN provides a blueprint for building AI systems that can help decode microbial dark matter, accelerate the discovery of new medicines, and design therapies grounded in the hidden biology around us.