Timezone: »

 
Poster
Catformer: Designing Stable Transformers via Sensitivity Analysis
Jared Quincy Davis · Albert Gu · Krzysztof Choromanski · Tri Dao · Christopher Re · Chelsea Finn · Percy Liang

Tue Jul 20 09:00 PM -- 11:00 PM (PDT) @
in Poster Session 2 »

Transformer architectures are widely used, but training them is non-trivial, requiring custom learning rate schedules, scaling terms, residual connections, careful placement of submodules such as normalization, and so on. In this paper, we improve upon recent analysis of Transformers and formalize a notion of sensitivity to capture the difficulty of training. Sensitivity characterizes how the variance of activation and gradient norms change in expectation when parameters are randomly perturbed. We analyze the sensitivity of previous Transformer architectures and design a new architecture, the Catformer, which replaces residual connections or RNN-based gating mechanisms with concatenation. We prove that Catformers are less sensitive than other Transformer variants and demonstrate that this leads to more stable training. On DMLab30, a suite of high-dimension reinforcement tasks, Catformer outperforms other transformers, including Gated Transformer-XL---the state-of-the-art architecture designed to address stability---by 13%.

Keywords: Architectures · Privacy, Anonymity, and Security

Author Information

Jared Quincy Davis (DeepMind & Stanford University)
Albert Gu (Stanford University)
Krzysztof Choromanski (Google Brain Robotics)
Tri Dao (Stanford)
Christopher Re (Stanford)
Chelsea Finn (Google)
Percy Liang (Stanford University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors