3D-DLP: Self-supervised 3D Object-centric Scene Representation Learning

We introduce 3D-DLP, a self-supervised object-centric representation learning model that decomposes scene-level RGB-D or voxel observations into a set of 3D latent particles. Building on the Deep Latent Particles (DLP) framework, each particle encodes disentangled attributes, including 3D keypoint position, bounding box dimensions, and appearance features, and represents a distinct entity in the scene. The model learns interpretable per-particle segmentation maps through an end-to-end self-supervised reconstruction objective. We demonstrate on both simulated and real-world datasets that the learned latent space is interpretable and controllable: by manipulating particle positions and decoding, we can generate novel scene configurations. Furthermore, we show that leveraging these compact 3D latent particles for downstream robotic manipulation improves performance over baselines that either lack explicit 3D information or rely on memory-intensive dense 3D inputs without object-centric structure. Code and videos are available at https://eubooks3003.github.io/3d-dlp

For robots to act reliably in the real world, they need to understand the scenes around them: where each object is, how big it is, and what it looks like. Today's AI systems struggle with this. Some learn from flat camera images, which lose 3D geometry, making it hard for a robot to grasp a cup at the right angle or reach around an obstacle. Others use full 3D data, but treat the scene as one undifferentiated whole, with no notion of separate objects. The few methods that do separate objects usually require a person to first label what counts as an object — which is expensive and limits where they can be used. This paper introduces 3D-DLP, a method that decomposes a 3D scene into individual objects entirely on its own, without any human labels. Given a 3D scene — say, a tabletop with several items — the system outputs a compact description of each object's location, size, and appearance. A user can directly move or resize objects in the description and watch the scene update. We then test whether this helps robots. Across 22 simulated manipulation tasks — picking up tools, assembling parts, arranging objects — robots using our representation succeed more often than those given either flat images or unstructured 3D inputs.