Controlled Dynamics Attractor Transformer

Transformer architectures have dramatically advanced representation learning and inference in deep models through self-attention mechanisms. In parallel, associative memory frameworks map representations onto energy landscapes, offering interpretable retrieval mechanisms. However, their continuous-time inference dynamics lack the biological plausibility of classical Continuous Attractor Neural Networks. To bridge this gap, we propose Controlled Dynamics Attractor Transformer (CDAT), which couples a mixture von Mises–Fisher attention energy with a Hopfield refinement energy, while augmenting energy descent with a CANN-inspired excitation–inhibition modulation. CDAT instantiates a topology-constrained dynamical system whose couplings encode relational structure among tokens, thereby linking attractor-style dynamics to modern energy-based attention. We further provide a constructive dissipation analysis to formally establish their controlled inference dynamics. Benefiting from these robust and structured dynamics, CDAT achieves state-of-the-art performance across multiple benchmarks in graph anomaly detection and graph classification.

Transformers are powerful but hard to interpret, while brain-inspired networks that settle into stable patterns are more transparent yet less accurate. We built CDAT, a model that combines transformer-style attention with a memory mechanism that pulls answers toward stable patterns, guided by a brain-inspired balance of "push and pull" signals between units. We also proved mathematically that it always settles into reliable answers rather than drifting unpredictably. On network-shaped data such as social graphs and transactions, CDAT outperforms existing methods at spotting unusual patterns and sorting graphs into categories—a step toward AI that is both stable and interpretable, with uses in fraud detection, cybersecurity, and scientific discovery.