HiPPO Zoo: Explicit Memory Mechanisms for Interpretable State Space Models

Representing the past in a compressed, efficient, and informative manner is a central problem for systems trained on sequential data. The \textit{HiPPO} framework, originally proposed by Gu & Dao et al., provides a principled approach to sequential compression by projecting signals onto orthogonal polynomial (OP) bases via structured linear ordinary differential equations. Subsequent works have embedded these dynamics in state space models (SSMs), where HiPPO structure serves as an initialization. Nonlinear successors of these SSM methods such as Mamba are state-of-the-art for many tasks with long-range dependencies, but the mechanisms by which they represent and prioritize history remain largely implicit. In this work, we revisit the HiPPO framework with the goal of making these mechanisms explicit. We show how polynomial representations of history can be extended to support capabilities of modern SSMs such as adaptive memory allocation and associative memory, while retaining direct interpretability in the OP basis. We introduce a unified framework comprising five such extensions, which we collectively refer to as a ``HiPPO zoo.'' Each extension exposes a specific modeling capability through an explicit, interpretable modification of the HiPPO framework. The resulting models adapt their memory online and train in streaming settings with efficient updates. We illustrate the behaviors and modeling advantages of these extensions through a range of synthetic sequence modeling tasks, demonstrating that capabilities typically associated with modern SSMs can be realized through explicit, interpretable polynomial memory structures.

When we process sequences of information such as speech, text, or sensor readings, we need to decide what to remember and what to forget. Modern machine learning systems called state space models (SSMs) have become remarkably good at this, but we cannot easily inspect what they are remembering or why. This paper revisits a framework called HiPPO which represents memory explicitly as polynomials that store history as a structured, readable summary rather than an opaque numerical state. The original HiPPO framework was interpretable but lacked capabilities that make modern SSMs powerful, such as the ability to selectively prioritize important inputs, store and retrieve specific associations, or represent information across multiple timescales simultaneously. We introduce five extensions to HiPPO (a "HiPPO Zoo") that add these capabilities in an explicit, interpretable way. Each extension exposes a specific capability as a transparent modification of how memory is structured. For example, one extension learns to allocate more memory to informative inputs and less to irrelevant ones, while another stores explicit key-value associations that can be directly inspected. The result is a toolkit of memory mechanisms that are less powerful than state-of-the-art SSMs on standard benchmarks, but whose internal workings remain directly readable. We view these models as interpretable alternatives suited for scientific and streaming applications where understanding what a model remembers matters as much as raw predictive performance.