A Deep Learning Model of Mental Rotation Informed by Interactive VR Experiments

Mental rotation—the ability to compare objects seen from different viewpoints—is a fundamental example of mental simulation and spatial world modeling in humans. Here we propose a mechanistic model of human mental rotation, leveraging recent advances in deep, equivariant, and neuro-symbolic learning. Our model consists of three stacked components: (1) an equivariant neural encoder, producing 3D spatial representations of objects from images, (2) a neuro-symbolic object encoder, deriving symbolic objects descriptions from these spatial representations, and (3) a neural decision agent, comparing these symbolic descriptions to prescribe rotation simulations in 3D latent space via a recurrent pathway. Our model design is guided by the existing experimental literature on mental rotation, which we complemented with experiments in VR where participants could at times manipulate the objects to compare. Our model captures well the performance, response times and behavior of participants in our and others' experiments, and through ablation studies we demonstrate the necessity of each component. Our work adds to a recent collection of deep neural models of human spatial reasoning, further demonstrating the potency of integrating deep, equivariant, and symbolic representations to model the human mind.

A striking example of spatial reasoning is humans' ability to recognize objects from different viewpoints—a skill known as mental rotation. In a classical mental rotation task, participants are shown two objects oriented differently and must determine whether the objects are identical or mirror images of each other. Humans are able to solve this task quite easily, and yet, despite the progress made in AI for visual tasks, it remains difficult to replicate this ability in artificial systems. Drawing on existing research and our own experimental observations of how people perform mental rotation, we built an AI model that combines two types of representation: one that captures the smooth, continuous transformations needed to rotate an object, and one that abstracts the spatial pose of objects to judge their similarity. We found that this model not only solves the task but also recapitulates human behavioral patterns, suggesting that this combination of continuous and discrete representations may be key to modeling human spatial reasoning.