Catformer: Designing Stable Transformers via Sensitivity Analysis

Jared Quincy Davis · Albert Gu · Krzysztof Choromanski · Tri Dao · Christopher Re · Chelsea Finn · Percy Liang

Keywords: [ Architectures ] [ Privacy, Anonymity, and Security ]

[ Abstract ]
[ Paper ]
[ Visit Poster at Spot A6 in Virtual World ]
Tue 20 Jul 9 p.m. PDT — 11 p.m. PDT
Spotlight presentation: AutoML and Neural Network Architectures 2
Tue 20 Jul 7 p.m. PDT — 8 p.m. PDT


Transformer architectures are widely used, but training them is non-trivial, requiring custom learning rate schedules, scaling terms, residual connections, careful placement of submodules such as normalization, and so on. In this paper, we improve upon recent analysis of Transformers and formalize a notion of sensitivity to capture the difficulty of training. Sensitivity characterizes how the variance of activation and gradient norms change in expectation when parameters are randomly perturbed. We analyze the sensitivity of previous Transformer architectures and design a new architecture, the Catformer, which replaces residual connections or RNN-based gating mechanisms with concatenation. We prove that Catformers are less sensitive than other Transformer variants and demonstrate that this leads to more stable training. On DMLab30, a suite of high-dimension reinforcement tasks, Catformer outperforms other transformers, including Gated Transformer-XL---the state-of-the-art architecture designed to address stability---by 13%.

Chat is not available.