FoundObj: Self-supervised Foundation Models as Rewards for Label-free 3D Object Segmentation

We address the challenging task of 3D object segmentation in complex scene point clouds without relying on any scene-level human annotations during training. Existing methods are typically constrained to identifying simple objects, primarily due to insufficient object priors in the learning process. In this paper, we present FoundObj, a novel framework featuring a superpoint-based object discovery agent that incrementally merges suitable neighboring superpoints, guided by our innovative semantic and geometric reward modules. These modules synergistically leverage semantic and geometric priors from self-supervised 2D/3D foundation models, providing complementary feedback to the object discovery agent and enabling robust identification of multi-class objects through reinforcement learning. Extensive experiments on diverse benchmarks demonstrate that our approach consistently outperforms existing baselines. Notably, our method exhibits strong generalization in zero-shot and long-tail scenarios, underscoring its potential for scalable, label-free 3D object segmentation. Code is available at https://github.com/vLAR-group/FoundObj

Accurately identifying and separating different objects from complex 3D scenes remains a major challenge. Existing methods rely heavily on labor-intensive human-annotated scene data and often struggle to recognize anything beyond simple objects, limiting their real-world applicability. We introduce FoundObj, a new framework that requires no human annotations at all during training. It first breaks down 3D scenes into small superpoint units, then uses an intelligent agent to gradually merge adjacent superpoints belonging to the same object. This merging process is guided by two innovative reward modules that extract semantic and geometric knowledge from pre-trained general-purpose AI models. Through reinforcement learning, the agent continuously improves its segmentation performance using this complementary feedback. Extensive experiments show our method consistently outperforms existing techniques across multiple standard benchmarks. Notably, it excels at recognizing never-before-seen objects and rare objects, enabling scalable, low-cost 3D object segmentation.