Image-to-Brain Signal Generation for Visual Prosthesis with CLIP Guided Multimodal Diffusion Models

Visual prostheses hold great promise for restoring vision in blind individuals. While researchers have successfully utilized M/EEG signals to evoke visual perceptions during the brain decoding stage of visual prostheses, the complementary process of converting images into M/EEG signals in the brain encoding stage remains largely unexplored, hindering the formation of a complete functional pipeline. In this work, we present a novel image-to-brain signal framework that generates M/EEG from images by leveraging the diffusion transformer architecture enhanced with cross-attention mechanisms. Specifically, we employ a diffusion transformer (DiT) architecture based on denoising diffusion implicit models (DDIM) to achieve brain signal generation. To realize the goal of image-to-brain signal conversion, we use cross-attention mechanisms to align brain signal embeddings with CLIP image embeddings. Moreover, we leverage large language models (LLMs) to generate image captions, and concatenate the resulting CLIP text embeddings with CLIP image embeddings to form unified embeddings for cross-attention alignment, enabling our model to capture core semantic information. Furthermore, we introduce a learnable spatio-temporal position encoding that combines brain region embeddings with temporal embeddings to capture both spatial and temporal characteristics of brain signals. We evaluate the framework on two multimodal benchmark datasets (THINGS-EEG2 and THINGS-MEG) and demonstrate that it generates biologically plausible brain signals.

Many people with severe vision loss could benefit from visual prostheses, devices that aim to create visual sensations by stimulating the nervous system. A key challenge is deciding what brain-like signal should be produced from a camera image so that the resulting stimulation is biologically meaningful. While much recent work has studied how to reconstruct images from brain activity, the reverse problem, turning images into brain signals, has received less attention. We propose a machine learning framework that generates EEG or MEG brain signals from images. Our model combines information about what appears in an image with a plain-language description of the image, then uses a diffusion-based generative model to produce brain signals over both brain regions and time. We test the method on two public datasets that contain paired images and human brain recordings, and show that the generated signals are closer to real brain responses than those produced by previous approaches. This work is a step toward better brain encoding models for visual prostheses and may also help researchers study how the brain represents visual information.