Conformal Symplectic and Relativistic Optimization
Guilherme Franca,Jeremias Sulam,Daniel Robinson,Rene Vidal
Arguably, the two most popular accelerated or momentum-based optimizationmethods are Nesterovs accelerated gradient and Polyakss heavy ball,both corresponding to different discretizations of a particular second orderdifferential equation with a friction term. Such connections withcontinuous-time dynamical systems have been instrumental in demystifyingacceleration phenomena in optimization.Here we study structure-preserving discretizations for a certain class ofdissipative (conformal) Hamiltonian systems, allowing us to analyze thesymplectic structure of both Nesterov and heavy ball, besides providingseveral new insights into these methods.Moreover, we propose a new algorithm based on a dissipative relativisticsystem that normalizes the momentum and may result in more stable/fasteroptimization. Importantly, such a method generalizes both Nesterov and heavyball, each being recovered as distinct limiting cases, and has potentialadvantages at no additional cost.
