Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts
                  Generalization

Stanisaw Jastrzebski 1 2 Devansh Arpit 3 Oliver strand 2 Giancarlo Kerg 4 Huan Wang 3 Caiming Xiong 3
              Richard Socher 3 Kyunghyun Cho * 2 5 Krzysztof J. Geras * 1 2

             Abstract                 commonly observed that using a moderately large learning
                                 rate in the early phase of training results in better generaliza-
   The early phase of tr ...