Floating-Point Networks with Automatic Differentiation Can Represent Almost All Floating-Point Functions and Their Gradients

Theoretical studies show that for any differentiable function on a compact domain, there exists a neural network that approximates both the function values and gradients. However, such a result cannot be used in practice since it assumes real parameters and exact internal operations. In contrast, real implementations only use a finite subset of reals and machine operations with round-off errors. In this work, we investigate whether a similar result holds for neural networks under floating-point arithmetic, when the gradient with respect to the input is computed by the automatic differentiation algorithm $D^\mathtt{AD}$. We first show that given a floating-point function $\phi$ (e.g., a loss function), arbitrary function values and gradients can be represented by a floating-point network $f$ and $D^\mathtt{AD}(\phi\circ f)$, respectively. We further extend this result: given $\phi_1,\dots,\phi_n$, $D^\mathtt{AD}(\phi_i\circ f)$ can simultaneously represent arbitrary gradients while $f$ represents the target values, under mild conditions. Our results hold for practical activation functions, e.g., ReLU, ELU, GELU, Swish, Sigmoid, and tanh.

Many modern AI applications rely not only on the outputs of neural networks but also on their gradients. Existing mathematical results show that neural networks can represent both function values and gradients, but these results assume idealized computations with perfectly accurate real numbers. Real computers, however, use floating-point arithmetic and introduce rounding errors during computation. In this work, we study whether similar guarantees hold for neural networks running on real computer hardware. We show that floating-point neural networks can represent almost any desired function values and gradients computed by automatic differentiation, the algorithm used to compute gradients in modern deep learning systems. We further show that these gradients can behave in ways that are impossible under idealized real-number arithmetic. Our results reveal a fundamental gap between neural networks studied in mathematical theory and neural networks implemented on real computers. By accounting for the effects of floating-point arithmetic, our work can help build a more realistic understanding of modern AI systems.