Controlling chaos

to be done in this area. Face recognition is a problem that scarcely clamoured for attention before the computer age but, having surfaced, has involved a wide range of techniques and has attracted the attention of some fine minds (David Mumford was a Fields Medallist in 1974). This singular application of mathematical modelling to a messy applied problem of obvious utility and importance but with no unique solution is a pretty one to share with students: perhaps, returning to the source of our opening quotation, we may invert Duncan's earlier observation, 'There is an art to find the mind's construction in the face!'.

Linear and nonlinear waves, by G. B. Whitham. Pp.636. £50. 1999. ISBN 0 471 35942 4 (Wiley).

Many problems in the sciences and engineering can be rephrased as optimization problems on matrix search spaces endowed with a so-called manifold structure. This book shows how to exploit the special structure of such problems to develop efficient numerical algorithms. It places careful emphasis on both the numerical formulation of the algorithm and its differential geometric abstraction--illustrating how good algorithms draw equally from the insights of differential geometry, optimization, and numerical analysis. Two more theoretical chapters provide readers with the background in differential geometry necessary to algorithmic development. In the other chapters, several well-known optimization methods such as steepest descent and conjugate gradients are generalized to abstract manifolds. The book provides a generic development of each of these methods, building upon the material of the geometric chapters. It then guides readers through the calculations that turn these geometrically formulated methods into concrete numerical algorithms. The state-of-the-art algorithms given as examples are competitive with the best existing algorithms for a selection of eigenspace problems in numerical linear algebra. Optimization Algorithms on Matrix Manifolds offers techniques with broad applications in linear algebra, signal processing, data mining, computer vision, and statistical analysis. It can serve as a graduate-level textbook and will be of interest to applied mathematicians, engineers, and computer scientists.

http://www.eeci-institute.eu/GSC2011/Photos-EECI/EECI-GSC-2011-M5/book_AMS.pdf

Optimization Algorithms on Matrix Manifolds

Hamiltonian Monte Carlo (HMC) is a Markov chain Monte Carlo (MCMC) algorithm that avoids the random walk behavior and sensitivity to correlated parameters that plague many MCMC methods by taking a series of steps informed by first-order gradient information. These features allow it to converge to high-dimensional target distributions much more quickly than simpler methods such as random walk Metropolis or Gibbs sampling. However, HMC's performance is highly sensitive to two user-specified parameters: a step size e and a desired number of steps L. In particular, if L is too small then the algorithm exhibits undesirable random walk behavior, while if L is too large the algorithm wastes computation. We introduce the No-U-Turn Sampler (NUTS), an extension to HMC that eliminates the need to set a number of steps L. NUTS uses a recursive algorithm to build a set of likely candidate points that spans a wide swath of the target distribution, stopping automatically when it starts to double back and retrace its steps. Empirically, NUTS performs at least as efficiently as (and sometimes more effciently than) a well tuned standard HMC method, without requiring user intervention or costly tuning runs. We also derive a method for adapting the step size parameter e on the fly based on primal-dual averaging. NUTS can thus be used with no hand-tuning at all, making it suitable for applications such as BUGS-style automatic inference engines that require efficient "turnkey" samplers.

/pdf/the-no-u-turn-sampler-adaptively-setting-path-lengths-in-2gyr5zn9cx.pdf

The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo

Initial-value ordinary differential equation solution via variable order Adams method (SIVA/DIVA) package is collection of subroutines for solution of nonstiff ordinary differential equations. There are versions for single-precision and double-precision arithmetic. Requires fewer evaluations of derivatives than other variable-order Adams predictor/ corrector methods. Option for direct integration of second-order equations makes integration of trajectory problems significantly more efficient. Written in FORTRAN 77.

Solving Ordinary Differential Equations

Hamiltonian Systems. Examples of Hamiltonian Systems. Symplecticness. The solution operator. Preservation of area. Checking preservation of area: Jacobians. Checking preservation of area: differential forms. Symplectic transformations. Conservation of volume. Numerical Methods. Numerical integrators. Stiff problems. Runge-Kutta methods. Partitioned Runge-Kutta methods. Runge-Kutta-Nystrom methods. Composition of methods - adjoints. Order conditions. Order conditions for Runge-Kutta methods. The local error in Runge-Kutta methods. Order conditions for PRK methods. The local error in Partitioned Runge-Kutta methods. Order conditions for Runge-Kutta-Nystrom methods. The local error in Runge-Kutta-Nystrom mehthods. Implementation. Variable step sizes. Embedded pairs. Numerical experience with variable step sizes. Implementing implicit methods. Fourth-order Gauss method. Symplectic integration. Symplectic methods. Symplectic Runge-Kutta methods. Symplectic partitioned Runge-Kutta methods. Symplectic Runge-Kutta-Nystrom methods. Necessity of the symplecticness conditions. Symplectic order conditions. Prelimiaries. Order conditions for symplectic RK methods. Order conditions for symplectic PRK methods. Order conditions for symplectic RKN methods. Homogenous form of the order conditions. Available symplectic methods. Symplecticness of the Gauss methods. Diagonally implicity Runge-Kutta methods. Other symplectic Runge-Kutta methods. Explicit partitioned Runge-Kutta methods. Available symplectic Runge-Kutta-Nystrom methods. Numerical experiments. A comparison of symplectic integrators. Variable step sizes for symplectic methods. Conclusions and recommendations. Properties of symplectic integrators. Backward error interpretation. An alternative approach. Conservation of energy. KAM theory. Generating functions. The concept of generating function. Hamilton-Jacobi equations. Integrators based on generating functions. Generating functions for RK methods. Canonical order theory. Lie formalism. The Poisson bracket. Lie operators and Lie series. The Baker-Campbell-Hausdorff formula. Application to fractional-step methods. Extension to the non-Hamilton case. High-order methods. High-order Lie methods. High-order Runge-Kutta-Nystrom methods. A comparison of order 8 symplectic integrators. Extensions. Partitioned Runge-Kutta methods for nonseparable Hamiltonian systems. Canonical B-series. Conjugate symplectic methods. Trapezoidal rule. Constrained systems. General Poisson structures. Multistep methods. Partial differential equations. Reversable systems. Volume preserving flows.

Numerical Hamiltonian Problems

In the sciences, situations where dissipation is not significant may invariably be modelled by Hamiltonian systems of ordinary, or partial, differential equations. Symplectic integrators are numerical methods specifically aimed at advancing in time the solution of Hamiltonian systems. Roughly speaking, ‘symplecticness’ is a characteristic property possessed by the solutions of Hamiltonian problems. A numerical method is called symplectic if, when applied to Hamiltonian problems, it generates numerical solutions which inherit the property of symplecticness.

https://www.cambridge.org/core/services/aop-cambridge-core/content/view/S0962492900002282

Symplectic integrators for Hamiltonian problems: an overview

We study the application of Runge-Kutta schemes to Hamiltonian systems of ordinary differential equations We investigate which schemes possess the canonical property of the Hamiltonian flow We also consider the issue of exact conservation in the time-discretization of the continuous invariants of motion

Runge-Kutta schemes for Hamiltonian systems

We investigate the properties of the Hybrid Monte Carlo algorithm (HMC) in high dimensions.
HMC develops a Markov chain reversible w.r.t. a given target distribution . by using separable Hamiltonian dynamics with potential -log .. The additional momentum variables are chosen at random from the Boltzmann distribution and the continuous-time Hamiltonian dynamics are then discretised using the leapfrog scheme. The induced bias is removed via a Metropolis-
Hastings accept/reject rule. In the simplified scenario of independent, identically distributed components, we prove that, to obtain an O(1) acceptance probability as the dimension d of the state space tends to ., the leapfrog step-size h should be scaled as h=l ×d−1/
4 . Therefore, in high dimensions, HMC requires O(d1/
4 ) steps to traverse the state space. We also identify
analytically the asymptotically optimal acceptance probability, which turns out to be 0.651
(to three decimal places). This is the choice which optimally balances the cost of generating a
proposal, which decreases as l increases (because fewer steps are required to reach the desired final integration time), against the cost related to the average number of proposals required to obtain acceptance, which increases as l increases

/pdf/optimal-tuning-of-the-hybrid-monte-carlo-algorithm-eygg2k3kiv.pdf

Optimal tuning of the hybrid Monte Carlo algorithm

Considered are numerical integration schemes for nondissipative dynamical systems in which multiple time scales are present. It is assumed that one can do an explicit separation of the RHS "forces" into fast forces and slow forces such that (i) the fast forces contain the high frequency part of the solution, (ii) the fast forces are conservative, and (iii) the reduced problem consisting only of the fast forces can be integrated much more cheaply than the full problem. The fast forces are allowed to have low frequency components. Particular applications for which the schemes are intended include N-body problems (for which most of the forces are slow) and nonlinear wave phenomena (for which the fast forces can be propagated by spectral methods). The assumption of cheap integration of fast forces implies that the overall cost of integration is primarily determined by the step size used to sample the slow forces. A long-time-step method is one in which this step size exceeds half the period of the fastest normal mode present in the full system. An existing method that comes close to qualifying is the "impulse" method, also known as Verlet-I and r-RESPA. It is shown that it might fail, though, for a couple of reasons. First, it suffers a serious loss of accuracy if the step size is near a multiple of the period of a normal mode, and, second, it is unstable if the step size is near a multiple of half the period of a normal mode. Proposed in this paper is a "mollified" impulse method having an error bound that is independent of the frequency of the fast forces. It is also shown to possess superior stability properties. Theoretical results are supplemented by numerical experiments. The method is efficient and reasonably easy to implement.

Jesús María Sanz-Serna

Papers

Numerical Hamiltonian Problems

Symplectic integrators for Hamiltonian problems: an overview

Runge-Kutta schemes for Hamiltonian systems

Optimal tuning of the hybrid Monte Carlo algorithm

Long-Time-Step Methods for Oscillatory Differential Equations