SLAP: The Semantic Least Action Principle for Variational Video-Language Modeling

In the era of Large Video-Language Models (LVLMs), the computational necessity of sparse frame sampling creates a fundamental ``temporal gap'', rendering models blind to critical causal transitions. Existing solutions relying on generative hallucination (e.g., latent diffusion) or autoregressive extrapolation often fail to maintain semantic consistency over long horizons, suffering from object vanishing and energetic instability. We propose a paradigm shift from probabilistic generation to variational mechanics with the Semantic Least Action Principle (SLAP). Drawing a rigorous isomorphism between classical mechanics and semantic dynamics, we model the latent video trajectory as a path on a Riemannian manifold governed by a Semantic Lagrangian. By formulating the interpolation task as a Boundary Value Problem (BVP) solved via the discrete Euler-Lagrange equations, SLAP naturally enforces object persistence without pixel-level rendering. Extensive experiments on multiple challenging datasets show the effectiveness of our proposed SLAP.

AI systems that answer questions about videos often cannot look at every frame, especially when the video is long. To save computation, they usually sample only a small number of frames. This can make them miss important events that happen between the sampled frames, such as an object entering a tunnel, staying hidden for a while, and then coming out later. As a result, the system may give answers that are inconsistent with what likely happened. This paper proposes a new way to help video-language models reason about these missing moments. Instead of trying to generate the missing video frames, our method estimates the most consistent ``story'' between the frames the model actually sees. It is inspired by the idea that natural motion usually follows a simple and stable path. Guided by the text question, the method fills in the missing visual meaning in a compact representation, helping the model preserve objects and reason more reliably over time. Our experiments show that this approach improves video question answering in challenging cases involving sparse observations and occlusion, while using much less computation than methods that generate missing video content.