EPS3D: End-to-End Feed-Forward 3D Panoptic Segmentation

This paper introduces EPS3D, a new end-to-end feed-forward framework for open-vocabulary 3D panoptic segmentation. Unlike existing methods relying on additional preprocessing, we design an end-to-end architecture, with a distillation-based training strategy on diverse 3D scenes to predict 3D-aware semantic and instance features from multi-view images, improving 3D consistency and avoiding error accumulation. We further propose a mutual enhancement module to enforce inherent semantic-instance consistency. By aligning semantics within instances (Ins2Sem) and refining instance features with semantic guidance (Sem2Ins), we achieve more coherent 3D scene understanding. Ultimately, EPS3D outperforms SOTA baselines on two benchmarks (e.g., +13% mIoU for semantics on Replica) with high efficiency (e.g., 1s per scene), supporting tasks like robotic manipulation and 3D scene editing.

How can computers understand a 3D scene from ordinary photos taken from different viewpoints? A useful system must recognize what objects are present, such as chairs or tables, and also distinguish each individual object, such as one chair from another. Previous methods often needed extra preprocessing steps before they could begin, which made them slower and introduced errors. We propose EPS3D, a single-step system that directly learns to understand 3D scenes from multiple 2D images. A key part of EPS3D lets two tasks help each other: identifying what an object is and deciding which individual object it belongs to. This makes the final 3D scene understanding more consistent. EPS3D performs much better than existing methods on widely used test datasets while remaining fast, processing an entire scene in about one second. This combination of speed and accuracy makes it useful for real-world applications, such as helping robots manipulate objects or allowing users to edit individual elements in 3D scenes.