Hourglass MLP: Rethinking the Shape of Residual Architectures

Multi-layer perceptrons (MLPs) conventionally adopt a narrow--wide--narrow design, where residual connections operate at input/output dimensions and computation occurs in expanded hidden spaces. We revisit this convention with wide--narrow--wide (Hourglass) MLP blocks, where residual connections act at the widest stage while computation flows through narrow bottlenecks. This inversion leverages expressive high-dimensional residual spaces for stable feature refinement. A key enabler is an initial expansion projection that can remain fixed at random initialization, reducing training and inference costs. We evaluate Hourglass MLPs on generative tasks and extend the design to Vision Transformers (ViTs) across multiple scales. Experiments demonstrate consistently superior performance--parameter Pareto frontiers compared to conventional designs. The reduced bottleneck cost enables flexible parameter reallocation toward wider residual representations or increased depth under fixed budgets. Our findings establish skip connection placement as a critical design principle for compute-optimal scaling in residual architectures.