Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent

                   Kangqiao Liu * 1 Liu Ziyin * 1 Masahito Ueda 1 2 3

             Abstract                 and Teh, 2011). When the noise is due to minibatch sam-
                                  pling, the noise is called the SGD noise or minibatch noise.
   In the vanishing learning rate regime, stochas-
                                  So far, nearly all the theoretical attempts at understand-
   tic gradient desce ...