ReferSplat: Referring Segmentation in 3D Gaussian Splatting

We introduce Referring 3D Gaussian Splatting Segmentation (R3DGS), a new task that aims to segment target objects in a 3D Gaussian scene based on natural language descriptions, which often contain spatial relationships or object attributes. This task requires the model to identify newly described objects that may be occluded or not directly visible in a novel view,posing a significant challenge for 3D multi-modal understanding. Developing this capability is crucial for advancing embodied AI.To support research in this area, we construct the first R3DGS dataset, Ref-LERF.Our analysis reveals that 3D multi-modal understanding and spatial relationship modeling are key challenges for R3DGS.To address these challenges, we propose ReferSplat, a framework that explicitly models 3D Gaussian points with natural language expressions in a spatially aware paradigm. ReferSplat achieves state-of-the-art performance on both the newly proposed R3DGS task and 3D open-vocabulary segmentation benchmarks.Dataset and code are available at https://github.com/heshuting555/ReferSplat.

Imagine describing a scene with words like “the red chair next to the table” and having a computer automatically find that object in 3D Gaussian scenes. In our work, we explore how to teach AI systems to understand such descriptions and locate the right objects — even if they're partially hidden or viewed from new angles.We introduce a new task called Referring 3D Gaussian Splatting Segmentation (R3DGS), which asks AI to identify target objects in a 3D Gaussian scene using natural language. To support this, we built the first dedicated dataset, called Ref-LERF.Understanding spatial relationships in 3D is hard, so we created a method called ReferSplat that links 3D scene points to language in a way that’s aware of spatial layouts. Our approach not only handles complex language cues but also achieves state-of-the-art performance in multiple 3D understanding tasks.We’ve released both our dataset and code to help others explore this new challenge: https://github.com/heshuting555/ReferSplat.