Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization

How to best develop foundational models for time series forecasting remains an important open question. Tokenization is a crucial consideration in this effort: what is an effective discrete vocabulary for a real-valued sequential input? To address this question, we develop WaveToken, a wavelet-based tokenizer that allows models to learn complex representations directly in the space of time-localized frequencies. Our method first scales and decomposes the input time series, then thresholds and quantizes the wavelet coefficients, and finally pre-trains an autoregressive model to forecast coefficients for the forecast horizon. By decomposing coarse and fine structures in the inputs, wavelets provide an eloquent and compact language for time series forecasting that simplifies learning. Empirical results on a comprehensive benchmark, including 42 datasets for both in-domain and zero-shot settings, show that WaveToken: i) performs on par or better than recently proposed foundation models for forecasting while using a much smaller vocabulary (1024 tokens), and is competitive with modern deep learning models trained specifically on each dataset; ii) exhibits superior generalization capabilities, achieving the best average rank across all datasets for three complementary metrics; and iii) easily captures complex temporal patterns of practical relevance that are challenging for other recent pre-trained models, including trends, sparse spikes, and non-stationary time series with varying frequencies evolving over time.

Predicting future events from past data (time series forecasting) is crucial in many fields, from finance to climate science. While powerful AI models (referred to as "foundation models") excel at language, adapting them to understand continuous time series data is challenging. A key hurdle is "tokenization": finding an effective way to convert real-valued sequences into a discrete "language" these models can process, especially one that captures both broad trends and sudden changes efficiently. This paper introduces WaveToken, a novel method that uses wavelets—mathematical tools that break down signals into different frequencies at specific times. WaveToken first decomposes the time series into these wavelet components, then simplifies and converts them into a compact set of "tokens". A foundation model is then trained to predict these future wavelet tokens. WaveToken performs as well as or better than existing advanced forecasting models, even those specifically trained for individual datasets, but with a significantly smaller vocabulary (fewer "words"). This compactness helps it generalize better to unseen data. Crucially, WaveToken can accurately capture complex patterns like trends, sudden spikes, and signals with evolving frequencies—areas where other models often falter, making it a promising approach for more robust and efficient time series forecasting.