Pocket Foundation Models: Distilling TFMs into CPU-Ready GBDTs

A fraud scorer needs to answer in under 2 ms. The strongest tabular foundation models (TFMs) take 151 to 1,275 ms on GPU. We close this gap by distilling the TFM offline into an XGBoost or CatBoost student that runs natively on CPU. The central obstacle is specific to in-context learning: when an ICL teacher scores an example that already lives in its own context, the output collapses to a near one-hot vector and carries no inter-class structure to distill. Stratified out-of-fold (OOF) teacher labeling removes this leakage and is the only component of the pipeline that is genuinely required. We benchmark this recipe across 153 classification datasets drawn from TALENT, OpenML-CC18, TabZilla, TabArena, with four TFM teachers and four student families. Distilling TabICLv2 into XGBoost yields a macro-mean ROC-AUC of 0.882 (97.1% of TabICLv2's solo AUC; 96.5% of the per-dataset best-teacher AUC) at 1.9 ms on a single CPU core, 79x faster than the teacher, and beats a tuned CatBoost baseline on 51% of datasets (Wilcoxon p=0.0008; wins average +0.021 AUC, losses -0.010). Three secondary findings sharpen the picture. Teacher rank transfers exactly to student rank in every student family, so teacher selection collapses to a single held-out comparison. Gains concentrate on low-dimensional data (<=21 features: +0.011 over CatBoost, n=79; >21 features: +0.001, n=74); on high-dimensional tasks where the teacher itself trails CatBoost, distillation makes things worse. Multi-teacher averaging adds +0.006 AUC for MLP students (p=0.003) but is practically negligible for tree students (+0.0006).