Olivia: Harmonizing Time Series Foundation Models with Power Spectral Density

Time series foundation models rely on large-scale pretraining over diverse datasets across domains, yet their heterogeneity in temporal patterns could hinder the effectiveness of training and learning transferable time series representations. Inspired a fundamental concept, normalized power spectral density (PSD) in signal processing, we assume harmonizing datasets via PSDs in the spectral domain could reduce mismatches and enhance pretraining. We then go beyond the direct intractable minimization optimization and innovatively reformulate it as a principled harmonization approach. Specifically, we propose Harmonizer, a module that reshapes spectral structures and implicitly harmonizing PSDs across datasets, which theoretically corresponds to a shared reparameterization of second-order temporal correlations. Our theoretical analysis further reveals token interactions with Harmonizer can be efficiently mediated by a compact set of resonators, motivating a HarmonicAttention design that performs self-attention in a low-dimensional interaction space. Then, we propose Olivia, a novel time series foundation model built upon these harmonization mechanisms. Extensive experiments on several large-scale benchmarks (TSLib, GIFT-Eval, and GluonTS), demonstrate Olivia consistently achieves state-of-the-art performance under zero-shot, few-shot, and full-shot forecasting scenarios. Our code is at \url{https://github.com/TSTS13/Olivia}.

Recently, large time series foundation models have been trained on massive collections of datasets from many different domains in order to learn general forecasting abilities. However, these datasets often exhibit very different temporal patterns, making it difficult for a single model to learn transferable representations across them. In this work, we explore whether these differences can be reduced by harmonizing datasets in the spectral domain, where temporal patterns can be analyzed through their frequency characteristics. Inspired by normalized power spectral density (PSD) in signal processing, we propose Olivia, a new time series foundation model that aligns temporal structures across datasets before learning from them. Olivia introduces a Harmonizer module to reshape spectral structures and a HarmonicAttention mechanism to efficiently model token interactions. Experiments on large-scale forecasting benchmarks show that Olivia consistently improves forecasting performance across zero-shot, few-shot, and full-shot settings, demonstrating stronger generalization across diverse real-world datasets.