On Second Order Behaviour in Augmented Neural ODEs
Alexander Norcliffe,Cristian Bodnar,Ben Day,Nikola Simidjievski,Pietro Liu00f3
Neural Ordinary Differential Equations (NODEs) are a new class of models thattransform data continuously through infinite-depth architectures. The continuousnature of NODEs has made them particularly suitable for learning the dynamicsof complex physical systems. While previous work has mostly been focused onfirst order ODEs, the dynamics of many systems, especially in classical physics,are governed by second order laws. In this work, we consider Second OrderNeural ODEs (SONODEs). We show how the adjoint sensitivity method can beextended to SONODEs and prove that the optimisation of a first order coupledODE is equivalent and computationally more efficient. Furthermore, we extend thetheoretical understanding of the broader class of Augmented NODEs (ANODEs)by showing they can also learn higher order dynamics with a minimal numberof augmented dimensions, but at the cost of interpretability. This indicates thatthe advantages of ANODEs go beyond the extra space offered by the augmenteddimensions, as originally thought. Finally, we compare SONODEs and ANODEson synthetic and real dynamical systems and demonstrate that the inductive biasesof the former generally result in faster training and better performance.
