Synthesizer: Rethinking Self-Attention for Transformer Models

        Yi Tay 1 Dara Bahri 1 Donald Metzler 1 Da-Cheng Juan 1 Zhe Zhao 1 Che Zheng 1

             Abstract                widely attributed to this self-attention mechanism since fully
                                 connected token graphs, which are able to model long-range
   The dot product self-attention is known to be cen-
                                 dependencies, provide a robust inductive bias.
   tral and indispensabl ...