Catformer: Designing Stable Transformers via Sensitivity Analysis

Jared Quincy Davis * 1 2 Albert Gu * 1 Krzysztof Choromanski 3 4 Tri Dao 1 Christopher Re 1 Chelsea Finn 1 3
                         Percy Liang 1

             Abstract                 to ameliorate these challenges, they require a combination
   Transformer architectures are widely used, but        of techniques such as complex optimizers and learning rate
   training them is non-trivial, requiring custom        schedules (Da ...