From Kepler to Newton: Inductive Biases Guide Learned World Models in Transformers

Vafa et al. recently showed that a transformer fails to acquire an internal Newtonian world model when trained on synthetic planetary-motion data. How can we fix this problem? We find that inductive biases are key to learning the veridical world model: (1) Spatial smoothness is required for any world model to be learned. However, naive tokenization may disrupt smoothness since two close points in physical space may be far apart in token embedding space without sufficient training or data. We fix this by formulating the prediction problem as regression instead of classification. (2) Spatial stability makes the prediction robust to noise, which is not guaranteed by default, but can be taught via correcting in-context noise perturbations. (3) With both spatial smoothness and stability built in, further imposing temporal locality induces a Newtonian world model, while the lack of this knowledge induces a Keplerian world model -- fitting elliptical parameters instead of computing gravitational forces. Our results suggest that even simple general inductive biases are powerful enough to induce correct and specific world models. The inductive biases do not need to know that much about the underlying law to be learned, but without them, it is impossible to learn.

Modern AI systems can make remarkably accurate predictions, but an important question remains: do they truly understand how the world works, or are they simply recognizing patterns? A recent study suggested that even when a transformer is trained on data generated by Newton's laws of gravity, it does not learn the underlying physical law. Instead, it appears to rely on shortcuts that predict future motion without understanding the forces that cause it. In this work, we investigate why this happens and how it can be fixed. We find that the key is not providing more data or larger models, but giving the learning system a few simple assumptions that match the structure of the physical world. For example, nearby locations in space should be represented similarly, predictions should remain stable when observations are slightly noisy, and recent events should matter more than distant ones. When these assumptions are incorporated into training, the model learns a much more faithful internal representation of physics. Rather than merely recognizing the shape of planetary orbits, it begins to infer the underlying gravitational forces that generate them. Without these assumptions, the model often learns a different strategy that makes accurate predictions but does not reflect the true physical mechanism. Our results suggest a broader lesson for artificial intelligence. Learning accurate models of the world may require the right inductive biases—general principles that guide learning—even when those principles contain little direct knowledge about the specific laws being discovered. In other words, some prior structure may be essential for AI systems to develop genuine scientific understanding rather than sophisticated pattern matching.