Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes

    James Lucas 1 2 Juhan Bae 1 2 Michael R. Zhang 1 2 Stanislav Fort 3 Richard Zemel 1 2 Roger Grosse 1 2

             Abstract
    Linear interpolation between initial neural net-
    work parameters and converged parameters after
    training with stochastic gradient descent (SGD)
    typically leads to a monotonic decrease in the
    training objective. This Monotonic Linear Inter-
    polation (MLI) property, first ...