Learning 3D-Gaussian Simulators from RGB Videos

Realistic simulation is critical for applications ranging from robotics to animation. Video generation models have emerged as a way to capture real-world physics from data, but they often face challenges in maintaining spatial consistency and object permanence, relying on memory mechanisms to compensate. As a complementary direction, we present 3DGSim, a learned 3D simulator that directly learns physical interactions from multi-view RGB videos. 3DGSim adopts MVSplat to learn a latent particle-based representation of 3D scenes, a Point Transformer for the particle dynamics, a Temporal Merging module for consistent temporal aggregation, and Gaussian Splatting to produce novel view renderings. By jointly training inverse rendering and dynamics forecasting, 3DGSim embeds physical properties into point-wise latent features. This enables the model to capture diverse behaviors, from rigid and elastic to cloth-like dynamics and boundary conditions (e.g., fixed cloth corners), while producing realistic lighting effects. We show that 3DGSim can generate physically plausible results even in out of distribution cases, e.g. ground removal or multi-object interactions, despite being trained only on single-body collisions.

Realistic simulation matters for fields like robotics and animation. A popular recent approach uses AI "video generation" models, which learn physics by watching real footage and predicting upcoming frames. But because these models reason only about flat images, they lose track of the 3D world: objects drift, deform, or even vanish when they leave the frame, and the models lean on memory tricks to paper over the gaps. We built 3DGSim, a simulator that learns physics directly in 3D rather than in 2D video. It represents a scene as a cloud of particles, learns how those particles move and collide over time, and then renders the result into images you can view from any angle. By training it to reconstruct scenes and forecast their motion at the same time, physical properties become embedded in the particles themselves. Because it reasons about genuine 3D structure instead of pixels, 3DGSim handles situations it never saw during training. Although it learned only from simple single-object collisions, it still behaves sensibly when the ground is removed or several objects interact at once, and it captures rigid, elastic, and cloth-like materials with realistic lighting. This shows that building explicit 3D structure into a model, rather than learning physics implicitly from video, is what lets it generalize beyond its training data.