Non-normal Recurrent Neural Network (nnRNN):
  learning long time dependencies while improving
     expressivity with transient dynamics

     Giancarlo Kerg1,2,  
                   Kyle Goyette1,2,3,     Maximilian Puelma Touzel1,4
  Gauthier Gidel1,2   Eugene Vorontsov1,5     Yoshua Bengio1,2,6    Guillaume Lajoie1,7

                        Abstract
      A recent strategy to circumvent the exploding and vanishing gradient problem in
      RNNs, and to allow the stable propagation of signa ...