Position: Reinforcement Learning Foundation Models Should Already Be A Thing

Foundation models for language and vision are powered by internet-scale data: the raw material is abundant and cheap to collect. Structured domains (tabular prediction, time-series forecasting, graph learning, reinforcement learning, etc.) do not enjoy this abundance. The natural substitute is synthetic data, which shifts the burden from collection to prior design. Such priors now exist for many structured tasks; TabPFN and its successors are the most prominent example, solving tabular classification via a transformer pretrained on a synthetic Bayesian prior. This paper makes two points. First, reinforcement learning is the conspicuous gap in this landscape: sampling a synthetic MDP is no harder than sampling a synthetic tabular dataset, yet none of the existing in-context RL literature treats prior design as a primary objective. We argue it should be. Second, MDPs admit a natural sufficient statistic that is fixed-size, independent of the number of episodes observed, and tabular in shape. That makes them directly amenable to the same set-transformer architectures used for tabular foundation models, with an actor-critic head replacing the supervised prediction target. Together, these two points define the agenda for an RL foundation model.