Floating-Point Neural Networks Can Represent Almost All Floating-Point Functions

Existing works on the expressive power of neural networks typically assume real-valued parameters and exact mathematical operations during the evaluation of networks. However, neural networks run on actual computers can take parameters only from a small subset of the reals and perform inexact mathematical operations with round-off errors and overflows. In this work, we study the expressive power of floating-point neural networks, i.e., networks with floating-point parameters and operations. We first observe that for floating-point neural networks to represent all functions from floating-point vectors to floating-point vectors, it is necessary to distinguish different inputs: the first layer of a network should be able to generate different outputs for different inputs. We also prove that such distinguishability is sufficient, along with mild conditions on activation functions. Our result shows that with practical activation functions, floating-point neural networks can represent floating-point functions from a wide domain to all finite or infinite floats. For example, the domain is all finite floats for Sigmoid and tanh, and it is all finite floats of magnitude less than 1/8 times the largest float for ReLU, ELU, SeLU, GELU, Swish, Mish, and sin.

Neural networks are a key part of modern AI, that has been used for face recognition, language translation, and medical image analysis. Most research about how neural networks work assumes they can do perfect math. But in the real world, computers cannot; they use rounded numbers that can introduce tiny errors.We investigate the following question: Can neural networks still work well using these imperfect numbers? We prove that the answer is yes. Despite the rounding and limitations of computer-based math, these networks can still perform nearly any task they need to.This is important because it shows that the AI tools we use in the real world are just as powerful as the ones described in theory assuming perfect math. The results help confirm that computers can run neural networks effectively, even with the small errors that come from working with real hardware. In short, this research builds confidence that practical AI systems are as capable as researchers expect them to be.