Event2Vec: Processing neuromorphic events directly by representations in vector space

Neuromorphic event cameras possess superior temporal resolution, power efficiency, and dynamic range compared to traditional cameras. However, their asynchronous and sparse data format poses a significant challenge for conventional deep learning methods. Most existing methods either densify events into frames, sacrificing their sparse asynchronous nature, or use irregular models that are less compatible with GPU acceleration. Inspired by word-to-vector models, we propose event2vec, a novel representation that allows Transformers to process events directly. We demonstrate the effectiveness of event2vec on the DVS Gesture, ASL-DVS, and DVS-Lip benchmarks, showing that event2vec is remarkably parameter-efficient, features high throughput and low latency, and achieves high accuracy even with an extremely low number of events or low spatial resolutions. These results show that sparse asynchronous event data can be directly integrated into high-throughput Transformer architectures, offering an efficient paradigm for real-time neuromorphic vision. The code is provided at https://github.com/Intelligent-Computing-Lab-Panda/event2vec.

Traditional cameras capture frame-by-frame static images just like a flipbook, while neuromorphic cameras inspired by the biological retina only record changes in pixel brightness. This mechanism greatly improves temporal resolution and energy efficiency. However, its irregular sparse data format makes it difficult for conventional artificial intelligence to process directly. To solve this problem, inspired by natural language processing techniques, we propose the event2vec method. It converts isolated visual events into a mathematical representation format that can be directly understood by modern advanced AI models such as Transformer. Experimental results show that this method features an extremely streamlined model and ultra-low running latency. It maintains high accuracy even under extreme conditions with very limited data or extremely low resolution. This breakthrough provides a highly promising new paradigm for the real-time and efficient application of brain-inspired visual sensors in real-world scenarios.