Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models

Modern vision backbones treat pointwise activations (e.g., ReLU, GELU) and exponential softmax as essential sources of nonlinearity, but we demonstrate they are not required within MetaFormer-style vision backbones. We design activation-free polynomial alternatives for three core primitives (MLPs, convolutions, and attention), where Hadamard products replace standard nonlinearities to yield polynomial functions of the input. These modules integrate seamlessly into existing architectures: instantiated within MetaFormer, a modular framework for vision backbones, our PolyNeXt models match or exceed activation-based counterparts across model scales on ImageNet classification, ADE20K semantic segmentation, and out-of-distribution robustness. We also substantially outperform prior polynomial networks at reduced computational cost, showing that polynomial variants of standard modules beat complex custom architectures. Our code is available at https://github.com/jjwang8/PolyNeXt.

Modern AI systems that recognize images are built from mathematical building blocks called "activation functions". Activation functions are small operations that let a network capture complex patterns. Researchers have long treated these functions as essential. Yet they create a practical obstacle: they are very hard to run on encrypted data, which blocks AI that could analyze private information without exposing it. Our paper studies whether these functions are truly necessary, or just one option among many. We found they can be removed entirely and replaced with plain multiplication: the network simply multiplies two processed versions of the data together. This turns the network into a "polynomial," an expression built only from additions and multiplications, like those in high-school algebra. We rebuilt the core components of a standard image-recognition network this way, and developed techniques to train these networks stably. Our proposed models, called PolyNeXt, recognize images as accurately as conventional networks and clearly outperform earlier multiplication-only designs. This shows that activation functions are not a fundamental requirement for AI vision, but one design choice among several. It is also a step toward AI that could one day analyze sensitive images, such as medical scans, while the data stays encrypted.