Closed-form continuous-time neural networks

Continuous-time neural networks are a class of machine learning systems for representation learning on spatiotemporal decision-making tasks, typically represented by continuous differential equations. Their expressive power on computers is often bottlenecked by numerical differential equation solvers, limiting scaling and understanding of natural physical phenomena such as nervous system dynamics. Ideally, one would circumvent this bottleneck by solving the dynamical system in closed form, which is generally intractable. Here, we show it is possible to closely approximate, efficiently and in closed form, the interaction between neurons and synapses constructed by liquid time-constant (LTC) networks. We compute a tightly bounded approximation of an integral in LTC dynamics that previously had no known closed-form solution. This closed-form solution impacts the design of continuous-time and continuous-depth neural models by making time explicit and relaxing the need for numerical solvers. Consequently, we obtain models between one and five orders of magnitude faster in training and inference than differential equation-based counterparts. Closed-form networks also scale remarkably well versus other deep learning instances and perform strongly on time-series modelling compared with advanced recurrent neural networks. Continuous-depth architectures built by ODEs transform depth/time into a continuous vector field and have shown promise in density estimation and modelling sequential and irregularly sampled data.