Rethinking Visual Intelligence: Insights from Video Pretraining

Large language models (LLMs) have demonstrated that large-scale pretraining enables systems to adapt rapidly to new problems with little supervision in the language domain. This success, however, has not translated as effectively to the visual domain, where models, including LLMs, continue to struggle with compositional understanding, sample efficiency, and general-purpose problem-solving. We investigate Video Diffusion Models (VDMs) as a promising direction for bridging this gap. Pretraining on spatiotemporal data endows these models with strong inductive biases for structure and dynamics, which we hypothesize can support broad task adaptability. To test this, we design a controlled evaluation in which both a pretrained LLM and a pretrained VDM are equipped with lightweight adapters and presented with tasks in their natural modalities. Across benchmarks including ARC-AGI, ConceptARC, visual games, route planning, and cellular automata, VDMs demonstrate higher data efficiency than their language counterparts. Taken together, our results indicate that video pretraining offers inductive biases that support progress toward visual foundation models.

Modern AI systems based on language models can learn many new tasks from only a few examples, but this ability is still much weaker for visual problems. In this paper, we study whether video generation models can help close this gap. These models are trained on videos, so they may learn useful knowledge about objects, shapes, movement, and how visual scenes change over time. We compare a video model and a language model on several visual reasoning tasks, including abstract puzzles, games, route planning, and cellular automata. Both models are adapted in a similar lightweight way, but each receives the task in the format most natural for it: images for the video model and text for the language model. Our results show that the video model often learns visual tasks from fewer examples than the language model. This suggests that training on video can give AI systems useful visual knowledge and may be a step toward more general-purpose visual AI.