SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-Body Manipulation

Simulating deformable objects under rich interactions remains a fundamental challenge for real-to-sim robot manipulation, with dynamics jointly driven by environmental effects and robot actions. Existing simulators rely on predefined physics or data-driven dynamics without robot-conditioned control, limiting accuracy, stability, and generalization. This paper presents \textbf{SoMA}, a 3D Gaussian Splat simulator for soft-body manipulation. SoMA couples deformable dynamics, environmental forces, and robot joint actions in a unified latent neural space for end-to-end real-to-sim simulation. Modeling interactions over learned Gaussian splats enables controllable, stable long-horizon manipulation and generalization beyond observed trajectories without predefined physical models. SoMA improves resimulation accuracy and generalization on real-world robot manipulation by 20\%, enabling stable simulation of complex tasks such as long-horizon cloth folding. Project Page: city-super.github.io/SoMA

Robots need large amounts of practice data to learn how to handle soft objects such as cloth, but collecting this data in the real world is slow and expensive. This paper introduces SoMA, a neural simulator that learns from real robot videos and can reproduce how soft objects move when a robot interacts with them. By combining visual observations with robot actions, SoMA can simulate both previously seen motions and new manipulation actions more accurately than prior methods. This can help researchers test and improve robot manipulation methods in simulation before running them on real robots.