Low-rank matrix approximations, such as the truncated singular value decomposition and the rank-revealing QR decomposition, play a central role in data analysis and scientific computing. This work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low-rank matrix approximation. These techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets. This paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions. These methods use random sampling to identify a subspace that captures most of the action of a matrix. The input matrix is then compressed—either explicitly or implicitly—to this subspace, and the reduced matrix is manipulated deterministically to obtain the desired low-rank factorization. In many cases, this approach beats its classical competitors in terms of accuracy, robustness, and/or speed. These claims are supported by extensive numerical experiments and a detailed error analysis. The specific benefits of randomized techniques depend on the computational environment. Consider the model problem of finding the $k$ dominant components of the singular value decomposition of an $m \times n$ matrix. (i) For a dense input matrix, randomized algorithms require $\bigO(mn \log(k))$ floating-point operations (flops) in contrast to $ \bigO(mnk)$ for classical algorithms. (ii) For a sparse input matrix, the flop count matches classical Krylov subspace methods, but the randomized approach is more robust and can easily be reorganized to exploit multiprocessor architectures. (iii) For a matrix that is too large to fit in fast memory, the randomized techniques require only a constant number of passes over the data, as opposed to $\bigO(k)$ passes for classical algorithms. In fact, it is sometimes possible to perform matrix approximation with a single pass over the data.

/pdf/finding-structure-with-randomness-probabilistic-algorithms-11r587jy79.pdf

Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions

This purpose of this introductory paper is threefold. First, it introduces the Monte Carlo method with emphasis on probabilistic machine learning. Second, it reviews the main building blocks of modern Markov chain Monte Carlo simulation, thereby providing and introduction to the remaining papers of this special issue. Lastly, it discusses new interesting research horizons.

/pdf/an-introduction-to-mcmc-for-machine-learning-22slt5b6x5.pdf

An introduction to MCMC for machine learning

Low-rank matrix approximations, such as the truncated singular value decomposition and the rank-revealing QR decomposition, play a central role in data analysis and scientific computing. This work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low-rank matrix approximation. These techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets. 
This paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions. These methods use random sampling to identify a subspace that captures most of the action of a matrix. The input matrix is then compressed---either explicitly or implicitly---to this subspace, and the reduced matrix is manipulated deterministically to obtain the desired low-rank factorization. In many cases, this approach beats its classical competitors in terms of accuracy, speed, and robustness. These claims are supported by extensive numerical experiments and a detailed error analysis.

Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions

We estimated the world’s technological capacity to store, communicate, and compute information, tracking 60 analog and digital technologies during the period from 1986 to 2007. In 2007, humankind was able to store 2.9 × 10 20 optimally compressed bytes, communicate almost 2 × 10 21 bytes, and carry out 6.4 × 10 18 instructions per second on general-purpose computers. General-purpose computing capacity grew at an annual rate of 58%. The world’s capacity for bidirectional telecommunication grew at 28% per year, closely followed by the increase in globally stored information (23%). Humankind’s capacity for unidirectional information diffusion through broadcasting channels has experienced comparatively modest annual growth (6%). Telecommunication has been dominated by digital technologies since 1990 (99.9% in digital format in 2007), and the majority of our technological memory has been in digital format since the early 2000s (94% digital in 2007).

/pdf/the-world-s-technological-capacity-to-store-communicate-and-39dybc0mey.pdf

The World’s Technological Capacity to Store, Communicate, and Compute Information

The paper proposes Metropolis adjusted Langevin and Hamiltonian Monte Carlo sampling methods defined on the Riemann manifold to resolve the shortcomings of existing Monte Carlo algorithms when sampling from target densities that may be high dimensional and exhibit strong correlations. The methods provide fully automated adaptation mechanisms that circumvent the costly pilot runs that are required to tune proposal densities for Metropolis–Hastings or indeed Hamiltonian Monte Carlo and Metropolis adjusted Langevin algorithms. This allows for highly efficient sampling even in very high dimensions where different scalings may be required for the transient and stationary phases of the Markov chain. The methodology proposed exploits the Riemann geometry of the parameter space of statistical models and thus automatically adapts to the local structure when simulating paths across this manifold, providing highly efficient convergence and exploration of the target density. The performance of these Riemann manifold Monte Carlo methods is rigorously assessed by performing inference on logistic regression models, log-Gaussian Cox point processes, stochastic volatility models and Bayesian estimation of dynamic systems described by non-linear differential equations. Substantial improvements in the time-normalized effective sample size are reported when compared with alternative sampling approaches. MATLAB code that is available from http://www.ucl.ac.uk/statistics/research/rmhmc allows replication of all the results reported.

/pdf/riemann-manifold-langevin-and-hamiltonian-monte-carlo-254effw0xm.pdf

Riemann manifold Langevin and Hamiltonian Monte Carlo methods

The Metropolis Algorithm has been the most successful and influential of all the members of the computational species that used to be called the "Monte Carlo method". Today, topics related to this algorithm constitute an entire field of computational science supported by a deep theory and having applications ranging from physical simulations to the foundations of computational complexity. Since the rejection method invention (J. von Neumann), it has been developed extensively and applied in a wide variety of settings. The Metropolis Algorithm can be formulated as an instance of the rejection method used for generating steps in a Markov chain.

/pdf/the-metropolis-algorithm-2ru92j5l89.pdf

The Metropolis Algorithm

For Monte Carlo simulations of systems of size [ital M], either kinetic simulations or equilibrium simulations that use the method of Bortz, Kalos, and Liebowitz [J. Comput. Phys. [bold 17], 10 (1975)], the best computer time per event has been [ital O]([ital M][sup 1/2]). We present two methods whose computer time per event is [ital O]([ital M][sup 1/[ital K]]) or [ital O](log[ital M]). In practice, for typical simulation sizes, [ital K] = 4 or [ital K] = 5 is fastest, requiring even less computer time than the [ital O](log[ital M]) method. For typical simulation sizes, we are able to achieve speedup factors of 5 to 7 over the [ital O]([ital M][sup 1/2]) technique.

Faster Monte Carlo simulations.

Although there are certainly revolutionary advances in technology, their effect accumulates slowly and the ultimate impact often contradicts the initial prognostications.

http://ieeexplore.ieee.org/iel5/5992/5725228/05725233.pdf

Watchful Waiting

Approximating the Permanent via Importance Sampling with Application to the Dimer Covering Problem

Importance sampling is an underappreciated Monte Carlo technique. The authors have used it with great success. They share their insights. Importance sampling is designed to reduce the variance of the estimators for a given-size sample.

Isabel Beichl

Papers

The Metropolis Algorithm

Faster Monte Carlo simulations.

Watchful Waiting

Approximating the Permanent via Importance Sampling with Application to the Dimer Covering Problem

The importance of importance sampling