FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs

Although Multimodal Large Language Models (MLLMs) demonstrate strong omni-modal perception, their ability to forecast future events from audio-visual cues remains largely unexplored, as existing benchmarks focus mainly on retrospective understanding. To bridge this gap, we introduce FutureOmni, the first benchmark designed to evaluate omni-modal future forecasting from audio-visual environments. The evaluated models are required to perform cross-modal causal and temporal reasoning, as well as effectively leverage internal knowledge to predict future events. FutureOmni is constructed via a scalable LLM-assisted, human-in-the-loop pipeline and contains 919 videos and 1,034 multiple-choice QA pairs across 8 primary domains. Evaluations on 13 omni-modal and 7 video-only models show that current systems struggle with audio-visual future prediction, particularly in speech-heavy scenarios, with the best accuracy of 64.8% achieved by Gemini 3 Flash. To mitigate this limitation, we curate a 7K-sample instruction-tuning dataset and propose an Omni-Modal Future Forecasting (OFF) training strategy. Evaluations on FutureOmni along with standard audio-visual and video-only benchmarks show that OFF improves future forecasting performance and generalization. Code and data are available at https://github.com/OpenMOSS/FutureOmni.

Many AI systems can describe what has already happened in a video, but real-world decisions often depend on predicting what will happen next. For example, a driving assistant may need to combine a car horn with the position of a pedestrian to anticipate danger. We introduce FutureOmni, a new test for AI systems that must use both video and sound to forecast future events. The test contains 919 videos and 1,034 multiple-choice questions across everyday, educational, movie, game, documentary, emergency, surveillance, and cartoon scenarios. We evaluated 20 leading AI models and found that they still struggle with this task, especially when spoken information is important; the best model answered only 64.8% of questions correctly. To help improve these systems, we created an additional training set and a method called Omni-Modal Future Forecasting, which teaches models to focus on earlier audio and visual clues rather than future frames. This training improves forecasting and also helps with other audio-video understanding tasks. Our work provides a practical way to measure and improve AI systems that need to anticipate future events from the world around them.