TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems

We introduce TFRBench, the first benchmark designed to evaluate the reasoning capabilities of forecasting systems. Traditionally, time-series forecasting has been evaluated solely on numerical accuracy, treating foundation models as "black boxes." Unlike existing benchmarks, TFRBench provides a protocol for evaluating the reasoning generated by forecasting systems--specifically their analysis of cross-channel dependencies, trends, and external events. To enable this, we propose a systematic multi-agent framework that utilizes an iterative verification loop to synthesize numerically grounded reasoning traces. Spanning ten datasets across five domains, our evaluation confirms that this reasoning is causally effective; useful for evaluation; and prompting LLMs with our generated traces significantly improves forecasting accuracy compared to direct numerical prediction (e.g., avg. $\sim40.2$% $\rightarrow$ $\sim56.6$%), validating the quality of our reasoning. Conversely, benchmarking experiments reveal that off-the-shelf LLMs consistently struggle with both reasoning (lower LLM-as-a-Judge scores) and numerical forecasting, frequently failing to capture domain-specific dynamics. TFRBench thus establishes a new standard for interpretable, reasoning-based evaluation in time-series forecasting. Our benchmark is available at: https://tfrbench.github.io

Forecasting systems are widely used to predict future events such as electricity demand, traffic volume, product sales, cloud workload, and stock prices. However, most forecasting benchmarks only check whether the final numbers are accurate, without asking whether the system can explain why those numbers should happen. This makes it difficult to know whether a model truly understands important causes, such as holidays, weather changes, seasonal patterns, or business events, or whether it is only guessing from past trends. In this paper, we introduce TFRBench, a benchmark designed to evaluate both forecasting accuracy and the reasoning behind the forecast. TFRBench uses a multi-agent framework to collect relevant external information, generate reasoning traces, verify them, and connect them to numerical predictions across ten datasets from five domains. The benchmark then evaluates whether forecasting models can produce explanations that are logically connected to the predicted values. Our results show that better reasoning is strongly associated with better forecasting performance. TFRBench provides a new way to test forecasting systems, not only by asking "Was the prediction correct?" but also "Was the explanation reliable?" This can help build more transparent and trustworthy forecasting tools for real-world decision-making.