Do Neural Operators Forget Geometry? The Forgetting Hypothesis in Deep Operator Learning

Neural operators perform well on structured domains, yet their behaviour on irregular geometries remains poorly understood. We show that this limitation is not merely an encoding issue, but a depth-wise failure mode inherent to deep operator architectures. We formalise the Geometric Forgetting Hypothesis: due to the Markovian structure of operator layers and their reliance on global mixing mechanisms, neural operators progressively lose access to domain geometry as depth increases. Using layer-wise geometric probing, we demonstrate that both spectral and attention-based operators systematically lose geometric fidelity. We show that this geometric forgetting degrades accuracy, stability, and generalisation. To counteract it, we introduce a lightweight geometry memory injection mechanism that restores geometric constraints at intermediate depths with minimal architectural overhead. This simple intervention consistently mitigates forgetting and exposes a geometric shortcut instability in transformer-based operators, revealing that geometric retention is a structural requirement rather than a design choice.

People are using AI models to quickly predict how physical systems behave, such as air flow, heat movement, or fluid motion. These models work well when the space has a simple, regular shape, but often struggle when the shape is irregular, as in many real-world engineering problems. We showed that the issue is not just how the shape is first given to the model. Instead, as information passes through the model, it gradually loses track of the shape and boundaries of the physical space. We call this “geometric forgetting.” We showed that this forgetting makes predictions less accurate, less stable, and worse at handling new shapes. To address this, we added a simple “geometry memory injection” mechanism that reminds the model of the shape at different stages. This small change improves performance across several tests and shows that keeping track of shape throughout the model is essential.