Improving the Sensitivity of Backdoor Detectors via Class Subspace Orthogonalization

Most post-training backdoor detection methods rely on attacked models exhibiting extreme outlier detection statistics for the target class of an attack, compared to non-target classes. However, these approaches may fail: (1) when some (non-target) classes are easily discriminable from all others, in which case they may naturally achieve extreme detection statistics (e.g., decision confidence); and (2) when the backdoor is subtle, i.e., with its features weak relative to intrinsic class-discriminative features. A key observation is that the backdoor target class has contributions to its detection statistic from both the backdoor trigger and from its intrinsic features, whereas non-target classes only have contributions from their intrinsic features. To achieve more sensitive detectors, we thus propose to suppress intrinsic features while optimizing the detection statistic for a given class. For non-target classes, such suppression will drastically reduce the achievable statistic, whereas for the target class the (significant) contribution from the backdoor trigger remains. In practice, we formulate a constrained optimization problem, leveraging a small set of clean examples from a given class, and optimizing the detection statistic while orthogonalizing with respect to the class's intrinsic features. We dub this approach ''class subspace orthogonalization'' (CSO). CSO can be ''plug-and-play'' applied to a wide variety of existing detectors. We demonstrate its effectiveness in improving several well-known detectors, comparing with a variety of baseline detectors, against a variety of attacks, on the CIFAR-10, GTSRB, and TinyImageNet domains. Moreover, to make the detection problem even more challenging, we also evaluate against a novel mixed clean/dirty-label poisoning attack that is more surgical and harder to detect than traditional dirty-label attacks. Finally, we evaluate CSO against an adaptive attack designed to defeat it, with promising detection results.

How can backdoor attacks still be detected when their signatures are subtle? Backdoor attacks are a serious security concern for modern AI systems. In these attacks, a model is secretly manipulated during training so that it behaves normally on most inputs, but produces incorrect predictions whenever a hidden trigger appears. Most existing backdoor detection methods work by searching for unusually strong or suspicious patterns associated with potential target classes in the model. However, this can lead to mistakes. Some classes are naturally easier for a model to distinguish, making them appear suspicious even when no attack exists. At the same time, subtle backdoors can hide within the model’s normal decision-making process, making malicious behavior difficult to separate from legitimate class features. Our work introduces a new technique called Class Subspace Orthogonalization (CSO) to make subtle backdoor signals easier to detect. The idea is that a poisoned target class contains both class discriminative features and malicious trigger-related features, while non-target classes contain only class discriminative features. CSO suppresses the class discriminative information so that any hidden backdoor behavior becomes more visible. Importantly, the method is designed to strengthen existing detection approaches rather than replace them entirely, allowing it to be easily integrated into current defenses. Across multiple base detectors and attack settings, CSO consistently improves the ability to identify backdoored models. CSO is also computationally efficient and requires only a small amount of clean data, making it practical for real-world deployment. By improving the reliability of post-training backdoor detection, this work contributes toward safer and more trustworthy deployment of machine learning systems in security-sensitive applications.