PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation

State-of-the-art single-image 3D reconstruction methods often rely on complex hybrid architectures and loss functions (e.g., MoGe), or necessitate compressing geometry into latent spaces (e.g., GeometryCrafter) to leverage pre-trained latent diffusion models. In this work, we demonstrate that such architectural overhead and intricate loss formulations are unnecessary. We introduce a minimalist pixel-space Diffusion Transformer built on a plain ViT, which operates directly on raw 3D point map patches and is conditioned on image tokens from a pre-trained DINOv3. Unlike existing latent diffusion-based approaches, we train our diffusion backbone entirely from scratch, eliminating the need for point map tokenizers. We show that this streamlined approach yields results superior to complex latent-based diffusion models while remaining significantly simpler than hybrid alternatives. Notably, our model produces sharper geometric structures and achieves significantly better results on highly ambiguous regions, such as transparent objects.

We introduce PointDiT, a minimalist framework that generates 3D structures (point maps) directly from a single image. By pairing a clean, standard ViT architecture with a robust pre-trained visual encoder DINOv3, our model learns to accurately predict sharp 3D shapes from a single image in a streamlined manner. PointDiT produces cleaner, more precise geometry than heavier existing models, especially around fine boundaries and notoriously tricky surfaces like transparent glass, paving the way for simpler, scalable, highly efficient 3D and 4D spatial computing.