A Probabilistic Framework for LLM-Based Model Discovery

Automated methods for discovering mechanistic simulator models from observational data offer a promising path toward accelerating scientific progress. Such methods often take the form of agentic-style iterative workflows that repeatedly propose and revise candidate models by imitating human discovery processes. However, existing LLM-based approaches typically implement such workflows via hand-crafted heuristic procedures, without an explicit probabilistic formulation. We recast model discovery as probabilistic inference, i.e., as sampling from an unknown distribution over mechanistic models capable of explaining the data. This perspective provides a unified way to reason about model proposal, refinement, and selection within a single inference framework. As a concrete instantiation of this view, we introduce ModelSMC, an algorithm based on Sequential Monte Carlo sampling. ModelSMC represents candidate models as particles which are iteratively proposed and refined by an LLM, and weighted using likelihood-based criteria. Experiments on real-world scientific systems illustrate that this formulation discovers models with interpretable mechanisms and improves posterior predictive checks. More broadly, this perspective provides a probabilistic lens for understanding and developing LLM-based approaches to model discovery.

Scientists rely on computer simulations to understand complex systems, from how viruses spread to how neurons fire in the brain. However, building these simulations is slow and requires deep expertise. Therefore, researchers have started asking whether machine learning methods can automatically discover good simulation models from data. Recent approaches use large language models (LLM) to write and revise simulation code, but most are built as ad-hoc workflows without a clear mathematical foundation, making them hard to analyze or improve systematically. We reframe automatic discovery of simulation models as a probabilistic problem. Instead of searching for a single best model, we ask which collection of models is consistent with the data. Our method, ModelSMC, maintains a population of candidate models that are iteratively proposed by an LLM and scored by how well they explain observations, similar to how a scientist might weigh competing hypotheses. Applied to real neuroscience and pharmacology problems, ModelSMC identifies not just one plausible model but a ranked set of alternatives, revealing which mechanistic hypotheses are strongly supported, which remain uncertain, and which are ruled out by the data.