MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models

Evaluation benchmarks are essential for assessing vision--language models (VLMs), but most multimodal benchmarks are static, making them vulnerable to temporal staleness, data contamination, and costly maintenance. We present MMBench-Live, a continuously evolving multimodal benchmark built by a multi-agent-driven automated pipeline. Our framework treats benchmark evolution as task-guided dataset construction, integrating structured benchmark specification, feedback-controlled real-time data acquisition, and verifiable QA generation with executable reasoning. To maintain cross-version comparability, we introduce a distribution-consistent update strategy that extracts task-related visual patterns from the original benchmark to guide data collection and filtering. Instantiated from MMBench, MMBench-Live contains 5.9K newly generated evaluation instances with a high answer correctness rate, while each update costs about \$30 and takes 1--2 hours. Extensive evaluations show that MMBench-Live preserves stable model rankings, maintains semantic alignment with the original benchmark, and exhibits weaker contamination-related memorization signals, suggesting a practical and scalable paradigm for sustainable multimodal benchmark evolution. The project is available at \url{https://github.com/PRIS-CV/MMBench-Live}.

Vision-language models, such as AI systems that answer questions about images, are often evaluated using fixed benchmark datasets. But fixed tests can become outdated, and some test examples may appear in the data used to train new models, making results look better than they really are. We introduce MMBench-Live, a benchmark that can be refreshed automatically with recent real-world images while keeping the original evaluation goals. Our system uses several AI agents to describe the tasks in an existing benchmark, collect new images that match those tasks, generate questions and answers, and check the answers through step-by-step tool-based verification. To make results comparable over time, the system learns the visual patterns of the original benchmark and uses feedback during data collection to avoid drifting away from those patterns. Built from MMBench, MMBench-Live contains 5.9K new evaluation examples. Each update takes about 1–2 hours and costs about $30, making frequent benchmark updates practical. Experiments show that MMBench-Live keeps model rankings stable while showing weaker signs of benchmark memorization than the original MMBench. This work offers a scalable way to keep multimodal AI evaluation fresh, fairer, and easier to maintain as models continue to evolve.