Approximation Theory for Lipschitz Continuous Transformers

Stability and robustness are critical for deploying Transformers in safety-sensitive settings. A principled way to enforce such behavior is to constrain the model's Lipschitz constant. However, approximation-theoretic guarantees for architectures that explicitly preserve Lipschitz continuity have yet to be established. In this work, we bridge this gap by introducing a class of gradient-descent-type in-context Transformers that are Lipschitz-continuous by construction. We realize both MLP and attention blocks as explicit Euler steps of negative gradient flows, ensuring inherent stability without sacrificing expressivity. We prove a universal approximation theorem for this class within a Lipschitz-constrained function space. Crucially, our analysis adopts a measure-theoretic formalism, interpreting Transformers as operators on probability measures, to yield approximation guarantees independent of token count. These results provide a rigorous theoretical foundation for the design of robust, Lipschitz continuous Transformer architectures.

Transformers are widely used in scientific and safety-sensitive applications, where stable and predictable behavior is important. However, it is difficult to mathematically guarantee that small changes in the input will not cause large changes in the output. In this work, we develop a class of Transformer architectures that are stable by design through a Lipschitz continuity constraint. Our models are inspired by gradient-descent dynamics and remain expressive despite the stability constraint. We also prove a theoretical result showing that these stable Transformers can still approximate a broad class of functions. This work provides a mathematical foundation for building more reliable and robust Transformer models.