The Surprising Simplicity of the Early-Time Learning
      Dynamics of Neural Networks

      Wei Hu      Lechao Xiao      Ben Adlam      Jeffrey Pennington§

                        Abstract
      Modern neural networks are often regarded as complex black-box functions whose
      behavior is difficult to understand owing to their nonlinear dependence on the data
      and the nonconvexity in their loss landscapes. In this work, we show that these
      common perceptions can be completely f ...