Aggregate Models, Not Explanations: Improving Feature Importance Estimation

Feature-importance methods show promise for transforming machine learning (ML) models from predictive engines into tools for scientific discovery. However, expressive models can be unstable due to data sampling and algorithmic stochasticity, leading to inaccurate variable importance estimates, undermining their utility in critical biomedical applications. While ensembling offers a remedy, the choice between explaining a single ensemble model or aggregating individual model explanations is non-trivial due to the non-linearity of importance measures, and remains largely understudied. Our theoretical analysis, developed under assumptions accommodating complex state-of-the-art ML models, reveals that this choice is governed by a trade-off involving the model's excess risk. In contrast to prior literature, we show that ensembling at the model level provides more accurate variable-importance estimates, particularly for expressive models, by reducing this leading error term. We validate these findings on classical benchmarks and a large-scale proteomic study from the UK Biobank.

Explainable AI promises to reveal what complex models, often considered black boxes, learn during training. This holds the potential to increase trust in predictions and deepen our understanding of complex problems. For instance, explaining a model trained to predict disease risk from large clinical cohorts can help identify new risk factors and ways to prevent disease. However, complex AI models are prone to instability, making them hard to introspect; small perturbations in training can change the inner workings of these models. This is known as the Rashomon effect, after a movie depicting conflicting witness accounts of the same incident during a trial. We study how best to mitigate this effect. Should we aggregate the explanations of multiple models, or explain a single larger model that itself aggregates predictions? To pursue the analogy: should multiple juries each hear one witness and then aggregate their verdicts, or should a single jury hear all witnesses and deliver one verdict? Our analysis reveals that the second approach is more effective, providing practitioners with a simple guideline for obtaining more reliable explanations from complex models.