Beyond Accuracy on RelBench: Item Response Theory Analysis of Relational Deep Learning Benchmarks

Machine learning benchmarks typically rank models by aggregate performance metrics, implicitly assuming all test instances are equally informative. We apply Item Response Theory (IRT) to two benchmark suites for relational deep learning, RelBench and DBInfer, to reveal properties of benchmark structure: which items drive rankings, where models diverge, and whether certain models are systematically over or under-ranked. Across 17 datasets (10 RelBench + 7 DBInfer, spanning 55 tasks), we evaluate up to 39 models from 13 families, including gradient-boosted decision trees, graph neural networks (HeteroGraph-SAGE, GatedGNN), graph foundation models (RelGT, Griffin, GNN+TabPFN-2.5), single-table foundation models applied via flat-table features (TabPFN, TabICL, Context-Tab, TabSTAR), relational DFS-based learners (RDBLearn), and AutoML (AutoGluon). We construct binary response matrices and fit Rasch, 2PL, and LogisticFM models with bootstrap resampling (N =100). Our key findings are: (1) IRT ability rankings correlate strongly with accuracy on 11 of 17 datasets (ρ > 0.8), but diverge substantially when models have nearly identical accuracy (Spearman ρ as low as 0.011 on dbinfer-diginetica); (2) graph-based models show strong domain-dependence - HeteroGraphSAGE ranks #1 by IRT on 4 datasets but drops to the bottom tercile on others; (3) RDBLearn’s Deep Feature Synthesis improves IRT ability on some datasets (e.g., ∆ˆθ = +0.286 on rel-ratebeer) but hurts on others; (4) IRT flags datasets where near-zero accuracy spread makes aggregate rankings unreliable for model selection (e.g., dbinfer-diginetica and dbinfer-retailrocket); (5) binarization ablation studies confirm robustness of IRT findings. In summary, IRT serves as a diagnostic tool: it can confirm when rankings are stable, and flag cases where rankings are compressed, item-driven, or uninformative for model selection.