Fairness-Preserving Regularizer: Balancing Core and Spurious Features

Real world visual data contains multiple attributes, e.g. color, foreground, background, etc. To solve a specific learning task, machine learning models should use a specific set of attributes. In principle, selecting which set of attributes as the core feature is defined by the task regardless of how heavily other attributes are (spuriously) correlated with the label. Without prior knowledge for identifying the core feature or spurious one, we can hardly tell a learned correlation is spurious or not in real world scenarios. In this work, we dive into this realistic setting and since there is no prior knowledge to determine which feature is core or spurious, we aim to learn a regularized predictor to fairly balance both core and spurious features.To achieve this, we start by formalizing fairness of learned features in a linear predictor under multiview data distribution assumption (Allen-Zhu &Li, 2023). We prove that achieving this fairness can be bounded by a simple regularization term and finally design fairness-preserving regularizer. Experiments on Waterbirds, CelebA and Wilds-FMOW datasets validate the effectiveness of our method.