Lost in Aggregation: How Benchmarks Overlook Irreplaceable Model Strengths

Tabular machine learning benchmarks typically summarize performance by averaging scores, ranks, or pairwise wins across datasets. Such aggregates are useful for selecting robust default models, but they can obscure a different question: which models are necessary to attain peak performance on particular datasets? We argue that benchmark evaluation should also consider the data-centric peak performance frontier, defined by the best statistically supported performance achieved on each dataset. From this perspective, a model may be irreplaceable, sufficient, redundant, or fallible depending on whether it uniquely reaches the frontier, ties with frontier models, is dominated by alternatives, or performs significantly worse than typical competitors. Applying this framework to TabArena, we find that common leaderboard metrics largely measure consistency and failure avoidance, and are much less aligned with dataset-level irreplaceability. As a result, models with unique dataset-specific strengths can appear mediocre under standard aggregation. Our results suggest that benchmark progress should be measured not only by improvements in average score, but also by whether new models expand the set of attainable peak performances across datasets.