Fast-SAM3D: 3Dfy Anything in Images but Faster

SAM3D enables scalable, open-world 3D reconstruction from complex scenes, yet its deployment is hindered by prohibitive inference latency. In this work, we conduct the **first systematic investigation** into its inference dynamics, revealing that generic acceleration strategies are brittle in this context. We demonstrate that these failures stem from neglecting the pipeline's inherent multi-level **heterogeneity**: the kinematic distinctiveness between shape and layout, the intrinsic sparsity of texture refinement, and the spectral variance across geometries. To address this, we present **Fast-SAM3D**, a training-free framework that dynamically aligns computation with instantaneous generation complexity. Our approach integrates three heterogeneity-aware mechanisms: (1) *Modality-Aware Step Caching* to decouple structural evolution from sensitive layout updates; (2) *Joint Spatiotemporal Token Carving* to concentrate refinement on high-entropy regions; and (3) *Spectral-Aware Token Aggregation* to adapt decoding resolution. Extensive experiments demonstrate that Fast-SAM3D delivers up to **2.67$\times$** end-to-end speedup with negligible fidelity loss, establishing a new Pareto frontier for efficient single-view 3D generation.

Reconstructing 3D objects from a single image is useful for virtual reality, digital content creation, robotics, and embodied AI, but strong models such as SAM3D are often too slow for practical use. This paper introduces Fast-SAM3D, which accelerates SAM3D without retraining by giving more computation to complex shapes and detailed regions while reducing computation on simpler or redundant parts. This allows the model to keep high-quality geometry and appearance while finishing much faster. Experiments show that Fast-SAM3D achieves up to 2.67× speedup with little loss in reconstruction quality.