TiMi: Empower Time Series Transformers with Multimodal Mixture of Experts

Multimodal time series forecasting has garnered significant attention for its potential to provide more robust and accurate predictions than traditional single-modality models by leveraging rich information inherent in other modalities. However, due to fundamental challenges in modality alignment, existing methods often struggle to effectively incorporate multimodal data into predictions, particularly textual information that has a causal influence on time series fluctuations, such as emergency reports and policy announcements. In this paper, we reflect on the role of textual information in numerical forecasting and propose Time series transformers with Multimodal Mixture-of-Experts, TiMi, to unleash the causal reasoning capabilities of LLMs. Concretely, TiMi utilizes language models to generate inferences on future developments, which then serve as guidance for time series forecasting. To seamlessly integrate both exogenous factors and time series into predictions, we introduce a Multimodal Mixture-of-Experts (MMoE) module as a lightweight plug-in to empower Transformer-based time series models for multimodal forecasting, eliminating the need for explicit representation-level alignment. Experimentally, our proposed TiMi demonstrates consistent state-of-the-art performance on sixteen real-world multimodal forecasting benchmarks, outperforming advanced baselines while offering strong adaptability and interpretability.

Good forecasts often depend on more than past numbers. Flu cases, fuel prices, traffic, and unemployment can all be affected by reports, news, or policy announcements written in text. However, such text is hard for forecasting models to use because it does not directly match the numerical data. We propose TiMi, a forecasting method that uses a language model to read relevant text and extract clues about likely future changes, then uses these clues to guide a time series model. TiMi also learns from the historical numerical patterns themselves. Across sixteen real-world forecasting tasks, TiMi improves prediction accuracy over strong existing methods and makes the role of text and past trends easier to understand.