Next Generation of Sequence Modeling Architectures

This workshop aims to bring together various researchers to chart the course for the next generation of sequence models. The focus will be on better understanding the limitations of existing models like transformer architectures, recurrent neural networks, and state space models (e.g., S4, Mamba), as well as describing existing open problems. We will touch on topics such as memory, long-range context and in-context learning, optimization stability of these architectures, and their ability to represent different classes of problems. We will also cover interpretability and pragmatic aspects of getting these models to be efficient and perform well: how they should be scaled up, and the trade-offs and limitations imposed by current hardware. We will place additional emphasis on the discussion regarding how we should evaluate and benchmark sequential models at scale, for example, in the context of language or other domains like vision, audio, or biological signals.