SCISSOR: Mitigating Semantic Bias through Cluster-Aware Siamese Networks for Robust Classification

Shortcut learning undermines model generalization to out-of-distribution data. While the literature attributes shortcuts to biases in superficial features, we show that imbalances in the semantic distribution of sample embeddings induce spurious semantic correlations, compromising model robustness. To address this issue, we propose SCISSOR (Semantic Cluster Intervention for Suppressing ShORtcut), a Siamese network-based debiasing approach that remaps the semantic space by discouraging latent clusters exploited as shortcuts. Unlike prior data-debiasing approaches, SCISSOR eliminates the need for data augmentation and rewriting. We evaluate SCISSOR on 6 models across 4 benchmarks: Chest-XRay and Not-MNIST in computer vision, and GYAFC and Yelp in NLP tasks. Compared to several baselines, SCISSOR reports +5.3 absolute points in F1 score on GYAFC, +7.3 on Yelp, +7.7 on Chest-XRay, and +1 on Not-MNIST. SCISSOR is also highly advantageous for lightweight models with ∼9.5% improvement on F1 for ViT on computer vision datasets and ∼11.9% for BERT on NLP. Our study redefines the landscape of model generalization by addressing overlooked semantic biases, establishing SCISSOR as a foundational framework for mitigating shortcut learning and fostering more robust, bias-resistant AI systems.

When we train classifiers, they often learn to make judgement based on misleading patterns, what is widely known as shortcut learning. For example, a model trained on bussiness reviews might assume that any food-related term indicates the review is positive, even when it isn't. These shortcuts make models less reliable in real-world applications.In our research, we found that the shortcut learning does not just come from obvious features like specific words, but also from how similar examples are grouped in the model’s internal understanding of meaning. If these groups are unbalanced, the model may unfairly rely on shortcuts.To solve this, we created a new method called SCISSOR that helps the model avoid learning from these biases. SCISSOR changes how the model understands the data, encouraging it to focus on the true reasons. Unlike other solutions, our approach doesn’t need more balanced data, it works as a lightweight add-on to existing models. This study improved accuracy and fairness of machine learning models in real-world cases, fostering more robust, bias-resistant AI systems.