Rational Neural Networks have Expressivity Advantages

We study neural networks with trainable low-degree rational activation functions and show that they are more expressive and parameter-efficient than modern piecewise-linear and smooth activations such as ELU, LeakyReLU, LogSigmoid, PReLU, ReLU, SELU, CELU, Sigmoid, SiLU, Mish, Softplus, Tanh, Softmin, Softmax, and LogSoftmax. For an error target of $\varepsilon>0$, we establish approximation-theoretic separations: Any network built from standard fixed activations can be uniformly approximated on compact domains by a rational-activation network with only $\mathrm{poly}(\log\log(1/\varepsilon))$ overhead in size, while the converse provably requires $\Omega(\log(1/\varepsilon))$ parameters in the worst case. This exponential gap persists at the level of full networks and extends to gated activations and transformer-style nonlinearities. In practice, rational activations integrate seamlessly into standard architectures and training pipelines, allowing rationals to match or outperform fixed activations under identical architectures and optimizers.

A small rational neural networks can sometimes do the work of a much larger standard network. For sharp transitions, high curvature, and nearby singularities, rational functions can reach a target accuracy with far fewer degrees of freedom than many smooth or piecewise-linear alternatives. This paper turns that approximation fact into a network-size result, comparing rational activations with ReLU, GELU, SiLU, and related standard choices. The gap favors rational networks: they can imitate many standard-activation networks compactly, while standard activations may need much larger networks to imitate some compact rational ones. The separation also covers nonlinear blocks used in modern architectures, including gated activations and transformer MLPs. We demonstrated this in practice for image classification, vision transformers, and offline control, low-degree rational activations often improve accuracy, convergence, or parameter efficiency.