Functional Attention: From Pairwise Affinities to Functional Correspondences

Learning mappings between infinite-dimensional function spaces, or operator learning, is essential for many machine learning applications. Although transformer-based operators are popular, they often rely on token-wise attention. These methods treat continuous fields as discrete tokens and usually ignore the global functional structure. We introduce Functional Attention, which reinterprets attention as a functional correspondence between adaptive bases. Inspired by geometric functional maps, our method replaces softmax affinities with structured linear operators. This yields a compact, generalizable, resolution-invariant representation that explicitly captures global dependencies. Experiments demonstrate that Functional Attention can match state-of-the-art performance in many operator learning tasks, including solving PDEs, 3D segmentation, and regression, while remaining robust to varying discretizations. Project page is available at https://github.com/xjffff/FUNCATTN.

Many scientific problems require AI to process continuous physical quantities sampled at thousands of points. Examples include predicting airflow around an airplane wing, simulating how materials bend under stress, and analyzing 3D molecular shapes. Modern AI relies on a mechanism called "attention," originally designed for text, that treats each measurement point as an isolated piece of data. This perspective ignores the smooth, continuous nature of physical signals, becomes computationally expensive at large scales, and struggles when the same problem is given at a different resolution. We propose Functional Attention, a new way to think about how attention works. Instead of comparing every pair of points, our method first learns a small set of adaptive "building blocks" that capture the essential patterns in the data, and then describes the input and output relationship using only these blocks. The idea is inspired by a mathematical tool originally developed in geometry for matching 3D shapes. Our method matches or outperforms leading approaches on diverse tasks, including fluid simulation, material deformation prediction, and 3D biological structure analysis. It also generalizes better to unseen physical conditions and resolutions. By making AI-based scientific simulation more accurate and flexible, this work could help accelerate engineering design and scientific discovery.