Beyond Model Ranking: Predictability-Aligned Evaluation for Time Series Forecasting

In the era of increasingly complex AI models for time series forecasting, progress is often measured by marginal improvements on benchmark leaderboards. However, standard evaluations rely on aggregate metrics (e.g., MSE) that conflate model capability with the intrinsic difficulty of the evaluated instances. To address this, we propose a diagnostic framework anchored in **Spectral Coherence Predictability (SCP)**, which provides an efficient $\mathcal{O}(N\log N)$ per-instance difficulty reference and yields a corresponding linear MSE lower bound. Complementing this, we introduce the **Linear Utilization Ratio (LUR)** to quantify how effectively models exploit linearly predictable structures across frequencies. Experiments on synthetic and real-world benchmarks show that SCP aligns strongly with realized forecasting errors across diverse state-of-the-art forecasters. Using this lens, we uncover ``predictability drift,'' revealing that task difficulty is not static but fluctuates significantly over time and variables. Furthermore, stratified evaluation exposes complementary architectural strengths across distinct frequency bands and difficulty regimes. Overall, we advocate moving beyond leaderboard-style ranking toward a more insightful, predictability-aware evaluation that fosters fairer model comparisons and a deeper understanding of model behavior. Code and data are available at https://github.com/WanjinVon/TS_Predictability.

Time series forecasting models predict future values such as traffic, weather, or electricity demand. Current evaluations often compare models using one average error score, but this can be misleading because some examples are naturally much easier to predict than others. This paper proposes a diagnostic framework that estimates how predictable each forecasting example is before judging model performance. It also measures how well a model uses the predictable structure in the data. Across synthetic and real-world datasets, we find that this predictability estimate closely matches actual model errors and reveals that forecasting difficulty changes over time and across variables. Our results suggest that forecasting models should be evaluated with awareness of example difficulty, rather than only by average benchmark scores.