Skip to yearly menu bar Skip to main content


Visual Attention Emerges from Recurrent Sparse Reconstruction

Baifeng Shi · Yale Song · Neel Joshi · Trevor Darrell · Xin Wang

Hall E #418

Keywords: [ DL: Robustness ] [ SA: Trustworthy Machine Learning ] [ APP: Neuroscience, Cognitive Science ] [ APP: Computer Vision ] [ DL: Attention Mechanisms ]


Visual attention helps achieve robust perception under noise, corruption, and distribution shifts in human vision, which are areas where modern neural networks still fall short. We present VARS, Visual Attention from Recurrent Sparse reconstruction, a new attention formulation built on two prominent features of the human visual attention mechanism: recurrency and sparsity. Related features are grouped together via recurrent connections between neurons, with salient objects emerging via sparse regularization. VARS adopts an attractor network with recurrent connections that converges toward a stable pattern over time.Network layers are represented as ordinary differential equations (ODEs), formulating attention as a recurrent attractor network that equivalently optimizes the sparse reconstruction of input using a dictionary of ``templates'' encodingunderlying patterns of data. We show that self-attention is a special case of VARS with a single-step optimization and no sparsity constraint. VARS can be readily used as a replacement for self-attention in popular vision transformers, consistently improving their robustness across various benchmarks.

Chat is not available.