Revealing Scaling Paradox in Large-scale Time Series Models: Implications for More Efficient and Accurate Forecasting

Large-scale models are at the forefront of time series (TS) forecasting, dominated by two paradigms: fine-tuning text-based Large Language Models for TS (LLM4TS) and training Time Series Foundation Models (TSFMs) from scratch. Both approaches share a foundational assumption that scaling up model capacity and data volume leads to improved performance. However, we observe a scaling paradox in TS models, revealing a puzzling phenomenon that larger models do NOT always achieve better performance. Through extensive experiments on two model families across four scales (100M to 1.7B parameters) and diverse data (up to 6B observations), we rigorously confirm that the scaling paradox is a pervasive issue. We then diagnose its root cause by analyzing internal representations, identifying a phenomenon we call few-layer dominance: only a small subset of layers are functionally important, while the majority are redundant, under-utilized, and can even distract training. Based on this discovery, we propose a practical method to automatically identify and retain only these dominant layers. In our models, retaining only 21% of the parameters achieves up to a 12% accuracy improvement and a 2.7x inference speedup. We validate the universality of our method on 8 prominent SOTA models (LLM4TS and TSFMs, 90M to 6B), showing that retaining less than 30% layers achieves superior accuracy in over 95% tasks.

Time series data appear in many real-world applications, such as weather, traffic, finance, energy systems, and healthcare. A common belief is that making forecasting models larger and training them on more data will naturally make them better. In this work, we show that this belief does not always hold. Across many experiments, we find that larger time series models often fail to outperform smaller ones, even when they use much more data and many more parameters. To understand why this happens, we look inside these models and find that only a small number of layers play a major role in making predictions. Many other layers contribute little, and in some cases may even make training less effective. Based on this observation, we develop a simple method that automatically keeps the useful layers and removes the less useful ones. Our results show that much smaller versions of large time series models can often be more accurate and faster than the original models. In some cases, keeping only about one fifth of the model parameters improves prediction accuracy while also making inference much faster. We further test this idea on a wide range of popular time series models and find that the same pattern appears consistently. These findings suggest that building better time series models is not only about making them larger, but also about understanding which parts of the model are truly useful.