Untangling tradeoffs between recurrence and
       self-attention in neural networks
            
  Giancarlo Kerg1,2,    Bhargav Kanuparthi 1,2,    Anirudh Goyal 1,2    Kyle Goyette 1,2,3
          Yoshua Bengio1,2,4            Guillaume Lajoie1,2,5

                        Abstract
     Attention and self-attention mechanisms, are now central to state-of-the-art deep
     learning on sequential tasks. However, most recent progress hinges on heuristic
     approaches with limited understandi ...