How Out-of-Distribution Detection Learning Theory Enhances Transformer: Learnability and Reliability

Transformers excel in natural language processing and computer vision tasks. However, they still face challenges in generalizing to Out-of-Distribution (OOD) datasets, i.e. data whose distribution differs from that seen during training. OOD detection aims to distinguish outliers while preserving in-distribution (ID) data performance. This paper introduces the OOD detection Probably Approximately Correct (PAC) Theory for transformers, which establishes the conditions for data distribution and model configurations for the OOD detection learnability of transformers. It shows that outliers can be accurately represented and distinguished with sufficient data under conditions. The theoretical implications highlight the trade-off between theoretical principles and practical training paradigms. By examining this trade-off, we naturally derived the rationale for leveraging auxiliary outliers to enhance OOD detection. Our theory suggests that by penalizing the misclassification of outliers within the loss function and strategically generating soft synthetic outliers, one can robustly bolster the reliability of transformer networks. This approach yields a novel algorithm that ensures learnability and refines the decision boundaries between inliers and outliers. In practice, the algorithm consistently achieves state-of-the-art (SOTA) performance across various data formats.

Modern Transformer models perform extremely well on images and language, but they often fail when they encounter unfamiliar or “out-of-distribution” (OOD) data that differs from what they saw during training, creating reliability and safety risks in real-world applications. Modern Transformer models perform extremely well on images and language, but they often fail when they encounter unfamiliar or “out-of-distribution” (OOD) data that differs from what they saw during training, creating reliability and safety risks in real-world applications. The proposed theory and algorithm together make Transformer systems more trustworthy and robust, achieving state-of-the-art performance across both vision and language tasks while providing principled guidance for building safer AI systems deployed in open-world environments.