AmbiRefer3D: 3D Visual Grounding with Referential Ambiguity

Traditional 3D visual grounding typically assumes that natural language expressions unambiguously refer to target objects in a 3D scene. However, in practical applications, human instructions are often ambiguous or insufficient, which may lead existing models to associate the query with multiple possible objects, resulting in incorrect results. In this paper, we propose a new task, 3D visual grounding with referential ambiguity, which allows for referential ambiguity in language descriptions, making it more broadly applicable to real-world scenarios. To tackle this task, we propose an interactive grounding framework that performs multi-round question-answer interactions, in which the model actively generates clarifying questions and receives human-provided answers to acquire additional object attributes, spatial relationships, and other contextual information, thereby resolving referential ambiguity and achieving accurate grounding. To support the learning of interactive grounding, we construct a large-scale dataset named AmbiRefer3D, which contains 47,085 samples with 141,255 annotations of question-answer dialogues that capture interactive disambiguation processes, covering 7,316 indoor 3D scenes. Furthermore, we establish multi-round evaluation metrics to measure both disambiguation efficiency and grounding accuracy. The code is available at https://github.com/yearnallover/AmbiRefer3D.

People often describe objects in ways that are not perfectly clear. For example, in a room with several similar chairs, saying “the chair near the table” may not be enough to identify exactly which chair is meant. Most existing 3D vision systems assume that a description clearly refers to one object, but this assumption often fails in real situations. This paper studies how a 3D vision system can handle such unclear instructions. We introduce a new task where the system must first recognize that several objects may match the same description, then ask or use clarifying information to narrow down the correct target. We also build AmbiRefer3D, a large dataset containing ambiguous object descriptions, candidate objects, and clarification dialogues across thousands of indoor 3D scenes. Our goal is to make 3D vision systems better at understanding natural human instructions, especially in realistic settings where people may give incomplete or ambiguous descriptions.