Elad Hazan: Spectral State Space Models

We'll describe a new approach for sequence modeling for prediction with long range dependencies motivated by large language models. We start with a formulation for state space models based on learning linear dynamical systems with the spectral filtering algorithm (https://arxiv.org/abs/1711.00946). This gives rise to a novel sequence prediction architecture we call spectral state space models that have two primary advantages. First, they have provable robustness properties as their memory performance depends on neither the spectrum of the underlying dynamics nor the dimensionality of the problem. Second, these models are constructed with fixed convolutional filters that do not require learning. The resulting models are evaluated on synthetic dynamical systems and long-range prediction tasks supporting the theoretical benefits of spectral filtering for tasks requiring very long-range memory. The talk is based on this new paper (https://arxiv.org/abs/2312.06837) and previous work on spectral filtering.