Hide&Seek: Learning to Explain in an End-to-End Differentiable Network

Instance-wise feature selection is a valuable tool for interpreting labeled data and the predictions of black-box models. In contrast to global feature selection techniques, instance-wise methods dynamically identify important features for each instance. A growing number of methods learn a selector, which identifies important features, and a predictor, which uses these to make predictions. However, these pioneering methods face challenges including information leakage and lack of differentiability, which can slow training. In this paper, we present Hide&Seek, an end-to-end differentiable model for instance-wise feature selection. We jointly learn feature selection and prediction under a single objective without information leakage. Hide&Seek outperforms existing state-of-the-art models across a range of experiments and is fast to train. We achieve this by reformulating feature removal as a differentiable operation where instead of discretely removing features, we replace a proportion of each feature. Training is further stabilized via a parsimony-weight annealing framework.

When a model makes a prediction, it draws on many features. Usually, only a handful matter, and the ones that matter can change from case to case. For example, diet and exercise might drive one patient's diagnosis, while genetics could be the important factor for another. To identify these features, some algorithms play a game of hide and seek: if masking a feature (hide) leads to a worse prediction (seek), that feature was probably important. The catch is how to hide. Most methods blank a feature out by setting it to zero. But the seeker can secretly read "this slot is zeroed" as a hidden signal, appearing to ignore the feature while still learning from it. Most methods also switch features fully on or off: an all-or-nothing choice whose mathematical workarounds can make models slow to train. Hide&Seek fixes both problems. Instead of blanking a feature, we swap in a plausible random value, leaving no tell-tale gap to exploit. Instead of an on/off switch, we use a dimmer dial, swapping out a proportion of each feature. The result: faster, more faithful explanations that can reveal relationships in data and help peer inside black-box models.