Timezone: »

 
Poster
Visual Attention Emerges from Recurrent Sparse Reconstruction
Baifeng Shi · Yale Song · Neel Joshi · Trevor Darrell · Xin Wang

Thu Jul 21 03:00 PM -- 05:00 PM (PDT) @ Hall E #418
in Poster Session 3 »

Visual attention helps achieve robust perception under noise, corruption, and distribution shifts in human vision, which are areas where modern neural networks still fall short. We present VARS, Visual Attention from Recurrent Sparse reconstruction, a new attention formulation built on two prominent features of the human visual attention mechanism: recurrency and sparsity. Related features are grouped together via recurrent connections between neurons, with salient objects emerging via sparse regularization. VARS adopts an attractor network with recurrent connections that converges toward a stable pattern over time.Network layers are represented as ordinary differential equations (ODEs), formulating attention as a recurrent attractor network that equivalently optimizes the sparse reconstruction of input using a dictionary of ``templates'' encodingunderlying patterns of data. We show that self-attention is a special case of VARS with a single-step optimization and no sparsity constraint. VARS can be readily used as a replacement for self-attention in popular vision transformers, consistently improving their robustness across various benchmarks.

Keywords: APP: Computer Vision · APP: Neuroscience, Cognitive Science · DL: Attention Mechanisms · DL: Robustness · SA: Trustworthy Machine Learning

Author Information

Baifeng Shi (UC Berkeley)
Yale Song (Microsoft Research)
Neel Joshi (MICROSOFT RESEARCH)
Trevor Darrell (University of California at Berkeley)
Xin Wang (Microsoft Research, Redmond)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors