TVDRNet: Text-driven Viewpoint Optimization via Differentiable Rendering for 3D Reasoning Segmentation

Three-dimensional (3D) reasoning segmentation aims at segmenting target objects based on text instructions and 3D spatial cues. Recent efforts in 3D reasoning leverage Multimodal Large Language Models (MLLMs) to bridge the gap between text and 3D data. However, since MLLMs are primarily trained on text-image pairs, directly adapting them to unstructured 3D point clouds often fails to capture implicit semantic intent and reliably localize objects. This paper introduces TVDRNet to address these challenges. Inspired by Active Vision theory, where humans selectively choose optimal viewpoints to better observe targets, TVDRNet employs a differentiable renderer to simulate this active process in 3D perception. By using text instructions as supervision to optimize intrinsic and extrinsic rendering parameters, the TVDRNet identifies the optimal viewpoints for observing the 3D scene, and therefore learning 'where to look' based on what the text instruction 'asked to find'. This process generates informative, task-relevant 2D images that are compatible with MLLMs. TVDRNet comprises: (1) the AVPL module, establishing a learnable mapping from semantics to optimal rendering parameters; and (2) the MGL module, fusing multi-modalities via semantic grouping to guide mask generation. Experiments show TVDRNet achieves the state-of-the-art performance in 3D reasoning segmentation (Reason3D, Instruct3D) and 3D visual grounding (ScanRefer) benchmarks.

When humans look for an object in a 3D space, like searching for keys on a messy desk, we instinctively move our heads to get the best viewing angle rather than taking in the whole room at once. Our research teaches Artificial Intelligence to mimic this natural human behavior. We developed a system called TVDRNet that uses written instructions—like "find the car keys"—to automatically figure out the best virtual camera angles to look at a 3D scene. Instead of blindly processing the entire 3D environment, our AI learns how to snap the most helpful virtual "photos" of the area based specifically on what it has been asked to find. Our tests show that this "active looking" approach makes the AI significantly more accurate at locating and identifying objects within complex 3D environments.