When Data Is Scarce: The Strength of the Prior in Tabular Foundation Models

We present a systematic evaluation of Prior-Data Fitted Networks (PFNs) in small-sample ($n < 500$) prediction tasks. First, through synthetic experiments, we quantify how informative parameter priors must be for correctly specified parametric models to match PFN predictive performance. Our results indicate that PFNs are competitive even against models with strong, well-calibrated priors. Second, using subsamples of the TabArena benchmark, we show that PFNs outperform traditional regression and tree-based methods across both classification and regression tasks in small-sample settings.