Learning Gaussian Mixture-distributed Prototypes for 3D Scene Graph Generation from RGB-D Sequences

3D Scene Graph Generation (3DSGG) aims to create a structured representation of 3D environment by identifying objects as nodes and their relations as edges. Existing 3DSGG methods based on RGB-D sequences typically put much focus on the adaption of neural networks to robust node and edge feature extraction in complex 3D scenes, yet ignoring the inherent intra-class diversity within each class and inter-class similarity between different categories associated with nodes and edges. In this work, we develop GMPSSG, a novel Gaussian Mixture-distributed Prototype mining framework for 3DSGG. Specifically, we model different categories with independent Gaussian Mixture-distributed Prototype to effectively mitigate inter-class similarity, while employing multiple Gaussian components within each prototype to capture intra-class diversity. Moreover, Prototype-anchored Representation Learning is introduced to construct a well-structured and mutually independent category space; Topology-aware Prototype Interaction is devised to capture implicit co-occurrence priors within the scene, and leverage them to calibrate prototype distributions, thereby ensuring the plausibility of node-edge matching. Experiments on 3DSSG dataset demonstrate GMPSSG outperforms various top-leading methods. Our code is available at GMPSSG.

Robots and intelligent systems need to understand 3D indoor spaces by not only recognizing objects, but also identifying the relationships between them, such as a chair on the floor or a picture hanging on a wall. However, this is challenging because objects from the same category may look very different, while different objects or relationships may appear visually similar. We propose GMPSSG, which represents each object and relationship category with multiple typical patterns instead of a single fixed template. This allows the model to capture diversity within the same category and better distinguish visually similar categories. We also use common object-relation combinations in scenes to make the predictions more plausible. Our method improves 3D scene graph generation from RGB-D sequences, helping robots build a more reliable understanding of indoor environments for tasks such as navigation, localization, and scene reasoning.