On Minimum Depth and Width of Floating-Point Neural Networks for Representing Floating-Point Functions

Research on the expressive power of neural networks has identified the minimum depth and width of neural networks that enable universal approximation and memorization. However, existing results are derived under exact arithmetic and cannot be directly applied to real implementations on computers, which can only use a finite set of numbers and inexact machine operations with round-off errors. In this work, we study floating-point ReLU networks that have floating-point parameters and use floating-point operations. Specifically, we investigate their minimum depth and width to represent all functions from the set of floating-point vectors $\mathbb F^d$ to the set of floating-point numbers $\mathbb F$. We first show that the minimum depth for representing all functions from $\mathbb F^d$ to $\mathbb F$ is exactly three, where two layers can be sufficient if we consider a smaller domain and/or codomain. We further show that the minimum width for representing all functions from $\mathbb F^d$ to $\mathbb F$ lies between $2d$ and $2d+4$. In addition, if we restrict the domain to non-negative floats, it lies between $d$ and $d+4$, where it can be smaller for a smaller domain, even beyond $d$. Our results show that the existing results analyzed under exact arithmetic do not extend to the floating-point setup.

AI systems are usually studied using ideal mathematics, where calculations are perfectly accurate. Real computers do not work this way: they constantly round numbers during calculations. We wanted to understand how this rounding changes what neural networks can actually do on real hardware. We found that rounding errors can fundamentally change how neural networks behave. In some cases, networks that look simple in theory become much harder to build on real computers, requiring larger or more complicated designs. We also found that the difficulty depends on the kinds of numbers given to the network as input. Our work shows that there can be a large gap between AI systems studied in theory and AI systems running on actual computers. By studying neural networks under realistic computer arithmetic, we hope to build mathematical foundations that better reflect how modern AI systems truly operate.