GameVerse: Can Vision-Language Models Learn from Video-based Reflection?

Human gameplay is a visually grounded interaction loop in which players act, reflect on failures, and watch tutorials to refine strategies. Can Vision-Language Models (VLMs) also learn from video-based reflection? We present GameVerse, a comprehensive video game benchmark that enables a reflective visual interaction loop. Moving beyond traditional fire-and-forget evaluations, it uses a novel reflect-and-retry paradigm to assess how VLMs internalize visual experience and improve policies. To facilitate systematic and scalable evaluation, we also introduce a cognitive hierarchical taxonomy spanning 15 globally popular games, dual action space for both semantic and GUI control, and milestone evaluation using advanced VLMs to quantify progress. Our experiments show that VLMs benefit from video-based reflection in varied settings, and perform best by combining failure trajectories and expert tutorials—a training-free analogue to reinforcement learning (RL) plus supervised fine-tuning (SFT).

When stuck in a video game, humans watch tutorials and review their mistakes to pass the level. Traditional AI testing, however, is "fire-and-forget"—giving models only one shot with no chance to learn from failure. To fix this, researchers introduced GameVerse, a benchmark spanning 15 popular games that tests AI using a "reflect-and-retry" loop. Instead of failing instantly, Vision-Language Models (VLMs) can watch videos of their own failed attempts alongside expert tutorials to self-correct. The study found that AI can significantly improve its gameplay without any retraining. The best results came from combining its own mistakes with expert guides—proving that AI, like humans, can successfully learn and evolve through visual reflection alone.