MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Models

Entity state tracking is a necessary component of world modeling that requires maintaining coherent representations of entities over time. Previous work has benchmarked entity tracking performance in purely text-based tasks. We introduce MET-Bench, a multimodal entity tracking benchmark designed to evaluate the ability of vision-language models to track entity states across modalities. Using three domains, we assess how effectively current models integrate textual and image-based state updates. Our findings reveal a significant performance gap between text-based and image-based entity tracking. We empirically show this discrepancy primarily stems from deficits in visual reasoning rather than perception. We further show that explicit text-based reasoning strategies improve performance, yet limitations remain, especially in long-horizon multimodal tasks. We apply reinforcement learning to improve entity tracking in open-source VLMs. This yields substantial in-modality gains, but does not transfer robustly across input modalities. Our results highlight the need for improved multimodal representations and reasoning techniques to bridge the gap between textual and visual entity tracking.

As AI systems take on more real-world tasks, they need to understand how the world changes as actions happen. For example, a cooking robot may need to track a refrigerator’s contents by observing what goes in and what comes out. This requires connecting information from images and text over time. We introduce MET-Bench, a benchmark that tests whether AI models can track changing world states in three game settings: Chess, Shell Game, and Minecraft. We find that models are much worse at tracking changes from images than from text. Surprisingly, the main difficulty is not simply recognizing what each image shows: in our tasks, models often identify the images correctly but struggle to use them to update their understanding of the state of the world. We then show that learning from trial and error can improve models' ability to track state. These findings reveal a critical gap in today's AI systems and point toward the need for new ways of creating models that reason about the world from visual inputs.