Amodal Instance Segmentation with IRAIS Dataset for Sim-to-Real Transfer

Amodal instance segmentation is hindered by the scarcity of scalable and transferable annotations. We introduce MaviGen, an automated 3D retail scene modeling and rendering framework that generates photorealistic multi-view images with complete amodal masks. Building on MaviGen, we present the IRAIS dataset, a sim-to-real benchmark comprising a large-scale synthetic multi-view set (3D-IRAIS) and a human-annotated real image set (Real-IRAIS), both sharing unified label definitions and evaluation protocols to facilitate rigorous transfer studies. We propose EUREKA, an encoder-only, query-efficient network for amodal instance segmentation that performs full-image multi-task inference via unified amodal/visible queries and dual mask heads. The dual heads enable mutual supervision between complete and visible masks, while the conditional masked self-attention mechanism further strengthens occlusion reasoning. Experiments establish strong baselines on IRAIS and achieve state-of-the-art performance on D2SA and COCOA-cls, demonstrating substantial improvements in sim-to-real transfer.

Real-world objects are often partially hidden, but most training data only labels what is visible and does not describe the object’s full shape. This makes it hard for models to learn how to predict complete objects under occlusion, especially when large-scale, reusable annotations are needed. To address this, we introduce an automated pipeline that builds 3D retail scenes and renders photorealistic images from multiple viewpoints while automatically producing complete “behind-the-occlusion” object masks alongside visible ones. Using this pipeline, we create the IRAIS benchmark for studying transfer from simulation to reality: it includes a large synthetic multi-view set and a smaller human-annotated real-image set, designed with consistent label definitions and evaluation rules so transfer can be measured rigorously. We also propose a new segmentation model, EUREKA, that predicts both the visible part and the full extent of each object in a single pass, and uses these two predictions to correct each other. Our experiments establish strong baselines on IRAIS and show improved performance on existing benchmarks, indicating more effective transfer from synthetic training data to real images.