PRISM: Learning Realistic Depth via Physics-Grounded Noise Disentanglement with Semantic-Geometric Collaboration

Real-world physical sensing exhibits complex, heterogeneous noise patterns that deviate significantly from idealized simulation, posing a fundamental bottleneck for sim-to-real transfer. Existing sensor modelings typically treat depth noise as a monolithic black-box process, overlooking the distinct physical mechanisms that govern different error modalities. In this work, we introduce a physics-grounded paradigm that disentangles monolithic noise into two complementary modalities: sensing invalidation and measurement inaccuracy, enabling a tailored treatment of noise sources based on their physical origins. Building on this insight, we propose PRISM, a tripartite framework that distills 3D Visual Foundation Model features as rich spatial-semantic priors for physics-based reasoning. To address the inherent sparsity and class imbalance of invalidation regions, we develop Hierarchical Positive-Prioritized Supervision, integrating multi-scale positive-weighted objectives with a positive-preserving dynamic hard mining strategy to enforce precise artifact delineation. Extensive benchmarks demonstrate that PRISM achieves state-of-the-art fidelity in noisy depth synthesis. Furthermore, downstream robotic experiments show that PRISM facilitates a 92.1% average success rate in the real world, marking a significant improvement over monolithic baselines.

Robots trained in computer simulation often fail when deployed in the real world, because simulated depth sensors produce perfectly clean 3D measurements while real sensors are riddled with errors. Existing approaches lump very different failure types together and try to model them as a single, undifferentiated source of noise. We propose PRISM, a system that treats two failure types separately. First, it asks: will this pixel return any signal at all? Second, for pixels that do return a signal, it asks: how far off will that measurement be? To answer the first question intelligently, PRISM borrows a pre-trained AI model that already "understands" the 3D geometry of scenes, using its knowledge to predict which surfaces are likely to confuse the sensor. The answers from both questions are then combined to synthesize realistic, corrupted depth images that closely match what a real sensorwould produce. Training robot policies on this more realistic synthetic data leads to dramatically better real-world performance: robots equipped with PRISM-augmented training achieved a 92.1% average success rate across six manipulation tasks without any additional demostration or adaptation at real-world deployment.