OpenMAG: A Comprehensive Benchmark for Multimodal-Attributed Graph

Multimodal-Attributed Graph (MAG) learning has achieved remarkable success in modeling complex real-world systems by integrating graph topology with rich attributes from multiple modalities. With the rapid proliferation of novel MAG models capable of handling intricate cross-modal semantics and structural dependencies, establishing a rigorous and unified evaluation standard has become imperative. Although existing benchmarks have facilitated initial progress, they exhibit critical limitations in domain coverage, encoder flexibility, model diversity, and task scope, presenting significant challenges to fair evaluation. To bridge this gap, we present OpenMAG, a comprehensive benchmark that integrates 19 datasets across 6 domains and incorporates 16 encoders to support both static and trainable feature encoding. OpenMAG further implements a standardized library of 24 state-of-the-art models and supports 8 downstream tasks, enabling fair comparisons within a unified framework. Through systematic assessment of necessity, data quality, effectiveness, robustness, and efficiency, we derive 14 fundamental insights into MAG learning to guide future advancements. Our code is available at https://github.com/YUKI-N810/OpenMAG.

Many real-world systems must reason about items that are connected to one another and described in more than one form, such as products linked by co-purchases with both images and titles, or social posts linked by interactions with text and pictures. Today, researchers lack a broad and fair way to test models for this kind of data, because existing benchmarks cover only a few domains, tasks, or feature-processing choices. To bridge this gap, we present OpenMAG, a comprehensive benchmark that serves as a standardized testing ground. It integrates 19 datasets across 6 domains, 16 encoders and standardizes a library of 24 state-of-the-art models. We test these models across 8 downstream tasks, challenging them to perform both structural reasoning and content generation. OpenMAG provides the research community with a unified framework to measure progress accurately. Through systematic assessment, we derive 14 fundamental insights that offer a clear roadmap for future machine learning advancements.