Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation

We introduce Robust Filter Attention (RFA), a formulation of self-attention as a robust state estimator. Each token is treated as a noisy observation of a latent trajectory governed by a linear stochastic differential equation (SDE), and attention weights are determined by consistency under this model rather than static feature similarity. Under isotropic noise and decay assumptions, RFA matches the computational complexity of standard attention. On language modeling benchmarks, RFA achieves lower perplexity than RoPE within the training window while remaining stable under zero-shot extrapolation to longer contexts. The framework also provides a dynamical interpretation of standard positional mechanisms, connecting rotational embeddings and recency biases to transport and uncertainty propagation induced by stochastic dynamics.

Modern transformer models rely on attention mechanisms to decide which pieces of information are most relevant when processing sequences such as text or time series data. However, most existing attention methods do not explicitly model uncertainty or account for noisy or unreliable information. We introduce Robust Filter Attention (RFA), a new attention framework inspired by probabilistic estimation and filtering theory. RFA models how uncertainty evolves over time and uses this information to adaptively weight interactions between tokens. This allows the model to reduce the influence of inconsistent or noisy inputs while preserving important structure in the data. Our framework also provides a new interpretation of positional encoding methods in transformers by connecting them to dynamical systems and uncertainty propagation. Experiments show that RFA improves robustness and modeling performance while building a closer connection between modern attention mechanisms and classical statistical estimation methods.