Who Said Neural Networks Aren't Linear?

Neural networks are famously nonlinear. However, linearity is defined relative to a pair of vector spaces, $f:\mathcal{X}\to\mathcal{Y}$. Leveraging the algebraic concept of transport of structure, we propose a method to explicitly identify non-standard vector spaces where a neural network acts as a linear operator. When sandwiching a linear operator $A$ between two invertible neural networks, $f(x)=g_y^{-1}(A g_x(x))$, the corresponding vector spaces $\mathcal{X}$ and $\mathcal{Y}$ are induced by newly defined addition and scaling actions derived from $g_x$ and $g_y$. We term this kind of architecture a Linearizer. This framework makes the entire arsenal of linear algebra, including SVD, pseudo-inverse, orthogonal projection and more, applicable to nonlinear mappings. Furthermore, we show that the composition of two Linearizers that share a neural network is also a Linearizer. We leverage this property and demonstrate that training diffusion models using our architecture makes the hundreds of sampling steps collapse into a single step. We further utilize our framework to enforce idempotency (i.e.\ $f(f(x))=f(x)$) on networks leading to a globally projective generative model and to demonstrate modular style transfer.

Modern neural networks are powerful because they reshape data in extraordinarily complicated ways — but that same complexity is also why they are slow, hard to invert, and hard to reason about. The simplest mathematical transformation, by contrast, is a straight scaling-and-mixing — what mathematicians call linear. Linear operations are fast, reversible, and well understood; they are usually thought to lack the expressive power needed for modern machine learning. This paper bridges that gap. We design a class of neural networks, which we call Linearizers, that look complicated from the outside but secretly behave like a simple linear operation once their input and output are viewed in the right coordinate system. The network learns both the linear "core" and the two reversible translators that map ordinary data into and out of the special coordinate system where this core lives. Because the core is genuinely linear, every standard tool from linear algebra transfers over. We use this to turn a modern image generator — which normally needs hundreds of slow sampling steps — into a one-step generator with no quality loss; to interpolate smoothly between artistic styles; and to build a generative model that cleanly snaps any input onto the kind of data it was trained on.