Evaluating System 1 vs. 2 Reasoning Approaches for Zero-Shot Time Series Forecasting: A Benchmark and Insights

Reasoning ability is crucial for solving challenging tasks. With the advancement of foundation models, such as the emergence of large language models (LLMs), a wide range of reasoning strategies has been proposed, including test-time enhancements, such as Chain-of-Thought, and post-training optimizations, as used in DeepSeek-R1. While these reasoning strategies have demonstrated effectiveness across various challenging language or vision tasks, their applicability and impact on time-series forecasting (TSF), particularly the challenging zero-shot TSF, remain largely unexplored. In particular, it is unclear whether zero-shot TSF benefits from reasoning and, if so, what types of reasoning strategies are most effective. To bridge this gap, we propose REC4TS, the first benchmark that systematically evaluates the effectiveness of popular reasoning strategies when applied to zero-shot TSF tasks. REC4TS conducts comprehensive evaluations across datasets spanning eight domains, covering unimodal and multimodal settings under short- and long-term forecasting horizons. More importantly, REC4TS provides key insights: (1) Self-consistency emerges as the most effective test-time reasoning strategy; (2) Hybrid reasoning with low-to-moderate reasoning effort often gives the best win-rate trade-off; (3) DeepSeek-R1 performs best among evaluated System 2 models, suggesting that outcome-oriented GRPO-style training may be promising while remaining confounded by model-family and training-data differences; (4) Multimodal TSF benefits more from reasoning strategies compared to unimodal TSF. Beyond these insights, REC4TS establishes two building blocks to support future zero-shot TSF reasoning research: (1) A novel dataset, TIME-THINKING, containing forecasting samples annotated with reasoning trajectories from multiple advanced LLMs, and (2) A simple test-time scaling trend for foundation TSF models inspired by self-consistency. All data and code are accessible at: https://anonymous.4open.science/r/REC4TS-2700