Faster Activation Functions at the Edge for Post-Training Speedups

On-device AI has gained significant attention for enabling efficient, low-latency inference on edge devices. However, tight resource constraints on these platforms make the deployment of accurate and lightweight deep learning models challenging. In particular, advanced activation functions (AFs) like Swish and GELU often incur high inference overhead due to the lack of hardware fast-paths for exponentiation and division, restricting edge-ML applications to simple AFs like ReLU, limiting model accuracy. To address this, we propose FFCC, a compiler that automatically generates efficient approximations of AFs through floating-point reinterpretation. These functions do not require hardware fast-paths, meaning they remain fast on edge devices, but are accurate enough to be used as post-training drop-ins. FFCC takes a specification of AFs using basic floating-point operators and applies derivation rules to lower these expressions into efficient instruction sequences. Our experiments show that FFCC provides fast approximations of AFs, achieving order-of-magnitude speed-ups over accurate baselines on Arm M7, Aarch64 and Intel platforms. Using ConvNeXt as an example, we demonstrate how these activation-level gains translate to end-to-end speed-ups, and do not result in significant loss of model accuracy.

Running artificial intelligence (AI) directly on everyday devices -- like smartphones, smart home gadgets, and medical sensors -- is essential for fast, private, and offline processing. However, these small devices have strict hardware limits, making it hard to run complex AI models efficiently. A major bottleneck lies in **activation functions** -- the mathematical checkpoints in an AI model that help it learn complex patterns. While more advanced functions make models highly accurate, they rely on very complicated math. Because small device hardware lacks built-in "fast-paths" to perform these calculations quickly, developers are forced to use oversimplified math, which severely hurts model accuracy. To solve this dilemma, we introduce **FFCC (Fast Floating-point Compiler)**. FFCC automatically replaces complex activation functions with some funny shortcuts that just so happen to approximate the complicated math very accurately. It does this by cleverly recognising that the 1s and 0s of computer memory form useful patterns for short-cutting exactly the kind of math machine learning models rely on. Our results show that these approximations are accurate enough and can be dropped directly into pre-existing models without ruining their final performance. Across embedded, mobile, and desktop computers, FFCC achieves speedups of $10\times$ or more. When tested on a real-world image recognition network (ConvNeXt), these mathematical shortcuts translated directly into measurable end-to-end device speedups with almost no loss in model accuracy.