Understanding the Role of Momentum in
         Stochastic Gradient Methods

       Igor Gitman  Hunter Lang    Pengchuan Zhang Lin Xiao
                 Microsoft Research AI
                Redmond, WA 98052, USA
      {igor.gitman, hunter.lang, penzhan, lin.xiao}@microsoft.com

                          Abstract
     The use of momentum in stochastic gradient methods has become a widespread
     practice in machine learning. Different variants of momentum, including heavy-
     ball momentum,  ...