Rethinking Multimodal Time-Series Forecasting Evaluation

We introduce a new context-enriched, multimodal time series forecasting benchmark TimesX. TimesX contains a wide selection of high-quality real-world time series with diverse domains and textual contexts obtained from an automated data generation pipeline, which helps address three main issues of existing multimodal forecasting benchmarks: (1) poor generalization due to the small scale and synthetic nature of benchmark data, (2) very limited types of textual contexts in the benchmarks, and (3) an inability to mitigate data leakage in evaluation. We conduct a thorough empirical study of zero-shot multimodal forecasting approaches on TimesX. Our results suggest that many approaches that perform well on existing benchmarks may fail on TimesX. In contrast, simple ensemble methods that leverage rich textual context accompanying time-series can outperform strong baselines on the TimesX benchmark.

Time-series forecasting is important for decision-making in areas such as markets, public interest, and exchange rates. We empirically show that existing multimodal (time+text) forecasting benchmarks suffer from information leakage, limited context quality, and fidelity issues. To address these problems, we introduce TimesX, a real-world multimodal time-series forecasting benchmark covering 19 domains and pairing time-series data with diverse context, including metadata, holidays, related variables, and events. More importantly, TimesX is designed to be automatically refreshable through carefully designed dataset agents, enabling it to continually reduce leakage from LLM knowledge and expand in scale. Our experiments show that simple ensembles of LLMs and time-series foundation models achieve strong zero-shot forecasting performance, while also revealing the instability issue.