BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language

Modeling the bidirectional correspondence between external sensory stimuli and internal neural activity has emerged as a critical frontier in neuroscience. However, existing approaches predominantly treat brain encoding and decoding as isolated tasks, relying heavily on unimodal alignment and external priors while overlooking the brain's intrinsic nature as a multimodal integration system. To address these limitations, we propose BrainJanus, the first unified brain model that integrates brain, vision, and language within a single framework. Specifically, we introduce a Unified Brain Tokenizer to quantize continuous neural dynamics into discrete tokens aligned with visual and linguistic representations in a shared Omni space. Building on this, we utilize an All-in-One autoregressive architecture that leverages next-token prediction to enable seamless any-to-any generation, which encompasses image-to-brain and text-to-brain encoding, and brain-to-image and brain-to-text decoding. Extensive experiments demonstrate that BrainJanus achieves superior performance across diverse benchmarks. Furthermore, our framework exhibits zero-shot generalization and preserves interpretable biological topography, highlighting its potential as a general-purpose brain modeling paradigm. The code is available at \href{https://github.com/HaitaoWuTJU/BrainJanus}{GitHub}.

(1) Understanding the two-way street between what we see or hear and how our brain reacts is a major goal in neuroscience. However, current AI tools usually treat interpreting brain signals and predicting brain responses as completely separate tasks, ignoring how our brains naturally blend different types of information together. (2) To bridge this gap, we created BrainJanus, a unified AI system that links brain signals, images, and text into a single, shared language. First, we developed a tool that translates complex brain waves into a "digital alphabet" that matches how the AI understands pictures and words. Then, similar to how modern chatbots predict the next word in a sentence, our system can seamlessly translate in any direction. It can predict how a brain will react to a specific image or sentence, or do the exact opposite and reconstruct pictures and words directly from someone's brain signals. (3) BrainJanus significantly outperforms previous methods and successfully adapts to new scenarios it wasn't explicitly trained for. By preserving the brain's natural biological patterns, this framework provides a powerful new foundation for advanced brain-computer interfaces, which could eventually help individuals with severe communication or sensory disabilities.