Universal Redundancies in Time Series Foundation Models

Time Series Foundation Models (TSFMs) leverage extensive pretraining to accurately predict unseen time series during inference, without the need for task-specific fine-tuning. Through large-scale evaluations of standard benchmarks, we find that leading transformer-based TSFMs exhibit redundant components in their intermediate layers. We introduce a set of tools for mechanistic interpretability of TSFMs, including ablations of specific components and direct logit attribution on the residual stream. Our findings are consistent across several leading TSFMs with diverse architectures, and across a diverse set of real-world and synthetic time-series datasets. We discover that all models in our study are robust to ablations of entire layers. Furthermore, we develop a theoretical framework framing transformers as kernel regressors, motivating a purely intrinsic strategy for ablating heads based on the stable rank of the per-head projection matrices. Using this approach, we uncover the specific heads responsible for degenerate phenomena widely observed in TSFMs, such as parroting of motifs from the context and seasonality bias. Our study sheds light on the universal properties of this emerging class of architectures for continuous-time sequence modeling.

We discover redundant components within large transformer-based time series foundation models (TSFMs). TSFM are pretrained on diverse datasets and have demonstrated a strong ability to forecast datasets unseen during training, beating classical statistical approaches on many data-limited tasks. We show that many components (and even entire layers) of TSFM can be removed without significantly affecting performance. We develop an interpretability toolbox that allows us to trace where phenomena like seasonality are represented within large models, and we present a theoretical framework explaining our findings. We show how models' internal representations of time series are updated from layer to layer, and we visualize the formation or redundant "lines of thought" in middle layers. Our study provides strong evidence that TSFM can be made much smaller while maintaining their performance, motivating future work in interpretability and model compression.