Rethinking Evaluation Paradigms in IBP-based Certified Training

Deep neural networks achieve strong performance on many supervised learning tasks but remain vulnerable to adversarial perturbations. Neural network verification provides mathematically rigorous robustness guarantees, yet at substantial computational cost. To mitigate this, certified training techniques optimise for verifiable robustness during training, typically inducing a trade-off between natural and certified accuracy controlled by method-specific hyperparameters. Because these metrics are inherently conflicting, the common practice of reporting a single configuration is problematic: it can mislead conclusions about overall performance and prevents unbiased assessments of the state of the art. We address this by evaluating certified training methods via Pareto front comparisons over the natural--certified accuracy trade-off. To enable fair, method-agnostic comparisons, we perform efficient automated multi-objective hyperparameter optimisation to identify a set of Pareto-optimal configurations for each method. This approach often uncovers substantial undertuning in previously reported configurations, yielding superior performance and establishing a new state of the art. Leveraging these fronts, we present the first comprehensive multi-objective comparison of certified training approaches, showing that prior advancements are less pronounced than assumed and revealing previously unreported performance complementarities.

Modern deep learning systems often perform very well on ordinary data, but they can be fooled by small, carefully chosen changes to their inputs that may be imperceptible to humans. This raises concerns about the reliability of AI systems, especially in safety-critical applications, where inputs may change naturally or be deliberately manipulated by an attacker. One way to address this problem is to train neural networks using certified training techniques. These methods aim to produce models whose robustness against certain input changes can be proven mathematically. However, certified robustness usually comes at a cost: models that are more robust often make more mistakes on clean, unmodified data. This creates a fundamental trade-off between certified robustness and standard accuracy. Certified training methods expose hyperparameters that control this trade-off. Previous work, however, has typically evaluated each method using only one fixed setting. Such evaluations can be misleading, because they do not show the full range of trade-offs a method can achieve. In this work, we argue that certified training methods should instead be compared across the entire trade-off spectrum. To make this possible, we introduce an efficient method for identifying the best attainable trade-offs, known as a Pareto front: a set of configurations where improving robustness would require sacrificing clean accuracy, or vice versa. Our analysis shows that previous evaluations often underestimated the performance of certified training methods, especially older ones, due to suboptimal hyperparameter choices. We also find that different methods are complementary: some perform better when high clean accuracy is desired, while others are preferable when stronger certified robustness is required. Overall, our work provides a more complete and fair way to evaluate certified training methods.