Approximation Bounds for Transformer Networks with Application to Regression

We develop approximation and statistical theory for standard Transformer networks in sequence modeling. Given a sequence-to-sequence target on $[0,1]^{d_x \times n}$ whose entries are $\gamma$-Hölder for $\gamma \in (0,1]$ or belong to a first-order Sobolev class, we establish explicit $L^p$-approximation bounds for all $p \in [1,\infty]$, including the previously elusive endpoint $p=\infty$ under softmax attention. In particular, achieving error $\varepsilon$ in $L^p$-norm requires $\mathcal{O}(\varepsilon^{-d_x n/\gamma})$ parameters for $\gamma$-Hölder targets and $\mathcal{O}(\varepsilon^{-d_x n})$ parameters for Sobolev targets, matching the best known scalings in ambient dimension $d_x n$. We further study nonparametric regression with sequential and dependent observations using Transformer networks. Assuming stationary $\beta$-mixing covariates whose temporal dependence weakens over time, we analyze a sliding-window empirical risk minimization procedure and establish excess-risk guarantees for the Transformer-based estimators. Our analysis clarifies the role of attention and enables extensions beyond softmax.

Transformers are the core technology behind many modern AI systems that process text, speech, and other sequential data. Yet despite their practical success, we still lack a precise mathematical understanding of what sequence-to-sequence rules they can represent and learn reliably. This paper develops theoretical guarantees for standard Transformer networks. We show how many model parameters are sufficient for a Transformer to accurately approximate broad classes of smooth sequence-to-sequence functions, including guarantees for worst-case prediction error. We also analyze learning from dependent sequential observations, where nearby data points may influence each other, as commonly occurs in language, time series, and sensor data. Our results help explain why the attention mechanism in Transformers is powerful from a theoretical perspective. More broadly, they provide a step toward understanding when Transformer-based methods can be expected to learn reliably from complex sequential data.