Realistic Evaluation of TabPFN v2.5 in Open Environments

Tabular data, owing to its ubiquitous presence in real-world domains, has garnered significant attention in machine learning research. While tree-based models have long dominated tabular machine learning tasks, the recently proposed deep learning model TabPFN v2.5 has emerged, demonstrating unparalleled performance and scalability potential. Although extensive research has been conducted on TabPFN v2.5 to further improve performance, the majority of this research remains confined to closed environments, neglecting the challenges that frequently arise in open environments. This naturally leads to an important question of whether TabPFN v2.5 can maintain strong performance under open-environment challenges. To this end, we conduct the first comprehensive evaluation of TabPFN v2.5's adaptability in open environments. We construct a unified evaluation framework covering various real-world challenges and assess the robustness of TabPFN v2.5 under this framework. Empirical results demonstrate that TabPFN v2.5 shows significant limitations in data-fitting scenarios but is suitable for covariate-shifted, class-imbalanced, and prior-driven tasks. To advance future research in open environments, we advocate for open-environment tabular benchmarks featuring multi-metric evaluation, a strengthened emphasis on robustness, dedicated modules for TabPFN v2.5, and the use of synthetic data that reflects open-environment characteristics during training.