HyperMLP: An Integrated Perspective for Sequence Modeling

Self-attention is often viewed as probabilistic query-key lookup, motivating designs that preserve normalized attention scores and fixed positional semantics. We advocate a simpler and more unified perspective: an autoregressive attention head can be viewed as a dynamic two-layer MLP whose weights are instantiated from the context history. From this view, attention scores form an ever-growing hidden representation, and standard MLP activations such as ReLU or GLU naturally implement input-conditioned selection over a context-dependent memory pool rather than a probability distribution. Based on this formulation, we introduce \textbf{HyperMLP} and \textbf{HyperGLU}, which learn dynamic mixing in both feature space and sequence space, using a reverse-offset (lag) layout to align temporal mixing with autoregressive semantics. We provide theoretical characterizations of the expressivity and implications of this structure, and empirically show that HyperMLP/HyperGLU consistently outperform strong softmax-attention baselines under matched parameter budgets.

Modern AI models for language, vision, and many other domains rely on a component called attention, which lets the model decide which parts of its input to focus on at each step. Today this is almost always implemented in one specific way, by turning scores into probabilities and taking a weighted average, and most efforts to improve attention still try to preserve that exact shape. We step back and show that attention is more naturally described as a small neural network whose internal connections are built on the fly from the surrounding context, which means the "probability" interpretation is not essential. Building on this view, we introduce two new components, HyperMLP and HyperGLU, that learn richer ways of mixing information across both features and time while respecting the causal structure that language models depend on. Under matched compute and parameter budgets, they consistently outperform standard attention across language modeling, image classification, and time-series forecasting.