We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.

Neural Ordinary Differential Equations

Received November 1, 1962. Revised April 22, 1963. * If the function f(y) satisfies a Lipschitz condition and h is sufficiently small, then the equations defining g(1>, g(2), • • • , gw have a unique solution which may be found by iteration (see Appendix). t It will be assumed throughout that f (y) and all its derivatives exist and are continuous so that the Taylor expansions for y and y may be terminated at any term with an error of the same order as the first term omitted. 50

/pdf/implicit-runge-kutta-processes-1wpbjw46ie.pdf

Implicit Runge-Kutta processes

The optimization field suffers from the metaphor-based “pseudo-novel” or “fancy” optimizers. Most of these cliche methods mimic animals' searching trends and possess a small contribution to the optimization process itself. Most of these cliche methods suffer from the locally efficient performance, biased verification methods on easy problems, and high similarity between their components' interactions. This study attempts to go beyond the traps of metaphors and introduce a novel metaphor-free population-based optimization method based on the mathematical foundations and ideas of the Runge Kutta (RK) method widely well-known in mathematics. The proposed RUNge Kutta optimizer (RUN) was developed to deal with various types of optimization problems in the future. The RUN utilizes the logic of slope variations computed by the RK method as a promising and logical searching mechanism for global optimization. This search mechanism benefits from two active exploration and exploitation phases for exploring the promising regions in the feature space and constructive movement toward the global best solution. Furthermore, an enhanced solution quality (ESQ) mechanism is employed to avoid the local optimal solutions and increase convergence speed. The RUN algorithm's efficiency was evaluated by comparing with other metaheuristic algorithms in 50 mathematical test functions and four real-world engineering problems. The RUN provided very promising and competitive results, showing superior exploration and exploitation tendencies, fast convergence rate, and local optima avoidance. In optimizing the constrained engineering problems, the metaphor-free RUN demonstrated its suitable performance as well. The authors invite the community for extensive evaluations of this deep-rooted optimizer as a promising tool for real-world optimization. The source codes, supplementary materials, and guidance for the developed method will be publicly available at different hubs at http://imanahmadianfar.com and http://aliasgharheidari.com/RUN.html .

RUN beyond the metaphor: An efficient optimization algorithm based on Runge Kutta method

A survey of direct time-integration methods in computational structural dynamics—I. Explicit methods

We consider a set of η first order simultaneous differential equations in the dependent variables y1, y2, …, yn and the independent variable x ⋮ No loss of gernerality results from taking the functions f1, f2, …, fn to be independent of x, for if this were not so an additional dependent variable yn+1, anc be introduced which always equals x and thus satisfies the differential equation

/pdf/coefficients-for-the-study-of-runge-kutta-integration-d5qfs972yu.pdf

W. Kutta

Papers

Beitrag zur Naherungsweisen Integration Totaler Differentialgleichungen