Can Vision Language Models Learn Intuitive Physics from Interaction?

Pre-trained vision language models do not have good intuitions about the physical world. Recent work has shown that supervised fine-tuning can improve model performance on simple physical tasks. However, fine-tuned models do not appear to learn robust physical rules that can generalize to new contexts. Based on research in cognitive science, we hypothesize that models need to interact with an environment to properly learn its physical dynamics. We train models that learn through interaction with a simulated environment using reinforcement learning. While learning from interaction allows models to improve their within-task performance, it fails to produce models with generalizable physical intuitions. We find that models trained on one task do not reliably generalize to related tasks, even if the tasks share visual statistics and physical principles, and regardless of whether the models are trained through interaction.

Modern large models that can take both images and text as inputs often do not have a good understanding of simple physical processes, such as whether a tower of blocks is stable. We let models interact with an environment, with the hope that this will allow them to gain an intuitive physical understanding. The models can learn new tasks, such as building bigger block towers, but they do not gain a broad physical understanding that allows them to solve other, related tasks.