Mitigating the Modality Gap in Vision–Language Models with Fractal Spectral Geometry

Vision–language models such as CLIP embed images and text into a shared space, but still suffer from a modality gap, where image and text features cluster separately and nearest neighbors are dominated by same-modality rather than true cross-modal matches. Existing works alleviate the modality gap by strengthening cross-modal losses, post-processing embeddings or similarities, or imposing geometric regularization, but they primarily enforce global alignment and can distort local geometry, limiting gains in local ranking and zero-shot accuracy. We propose Fractal Spectral Alignment (FSAlign), which reduces the modality gap by shaping and matching the multi-scale geometry of image and text embeddings. By enforcing Ahlfors-regularity and sub-Gaussian heat kernel bounds, FSAlign constructs a shared fractal multi-scale structure for multiple modalities. This structure captures geometry across scales, from local neighborhoods to global structure, and ensures shared fractal spectral geometry across modalities. Based on this structure, we introduce a fractal spectral zeta score derived from multi-scale heat kernels and minimize the discrepancy between pairwise image–text samples to align their multi-scale neighborhoods. We theoretically demonstrate that FSAlign can guarantee the alignment of local spectral measures and global fractional Dirichlet energies.

Vision-language models learn to connect images and text in the same representation space. However, image features and text features can still form separate groups, even when they describe the same concept. This problem is known as the modality gap. It can hurt image-text retrieval because a model may place visually similar images or textually similar captions ahead of the correct matched pair. This paper proposes FSAlign, a method that reduces this gap by improving how image and text neighborhoods are organized. Instead of only pulling each image and its caption closer together, FSAlign makes the local structure around matched images and texts more similar across different scales. In simple terms, the method checks whether an image and its caption have similar nearby examples, from very close neighbors to broader groups. It then learns lightweight geometry mappings on top of frozen vision-language models, so the original model does not need to be retrained from scratch. Experiments on image-text retrieval and zero-shot classification show that FSAlign improves performance across several datasets and model backbones. These results suggest that better local alignment between image and text representations can make vision-language models more reliable for matching, searching, and transferring knowledge across modalities.