Showing papers in "Siam Review in 2009"
TL;DR: This survey provides an overview of higher-order tensor decompositions, their applications, and available software.
Abstract: This survey provides an overview of higher-order tensor decompositions, their applications, and available software. A tensor is a multidimensional or $N$-way array. Decompositions of higher-order tensors (i.e., $N$-way arrays with $N \geq 3$) have applications in psycho-metrics, chemometrics, signal processing, numerical linear algebra, computer vision, numerical analysis, data mining, neuroscience, graph analysis, and elsewhere. Two particular tensor decompositions can be considered to be higher-order extensions of the matrix singular value decomposition: CANDECOMP/PARAFAC (CP) decomposes a tensor as a sum of rank-one tensors, and the Tucker decomposition is a higher-order form of principal component analysis. There are many other tensor decompositions, including INDSCAL, PARAFAC2, CANDELINC, DEDICOM, and PARATUCK2 as well as nonnegative variants of all of the above. The N-way Toolbox, Tensor Toolbox, and Multilinear Engine are examples of software packages for working with tensors.
9,227 citations
TL;DR: This work proposes a principled statistical framework for discerning and quantifying power-law behavior in empirical data by combining maximum-likelihood fitting methods with goodness-of-fit tests based on the Kolmogorov-Smirnov (KS) statistic and likelihood ratios.
Abstract: Power-law distributions occur in many situations of scientific interest and have significant consequences for our understanding of natural and man-made phenomena. Unfortunately, the detection and characterization of power laws is complicated by the large fluctuations that occur in the tail of the distribution—the part of the distribution representing large but rare events—and by the difficulty of identifying the range over which power-law behavior holds. Commonly used methods for analyzing power-law data, such as least-squares fitting, can produce substantially inaccurate estimates of parameters for power-law distributions, and even in cases where such methods return accurate answers they are still unsatisfactory because they give no indication of whether the data obey a power law at all. Here we present a principled statistical framework for discerning and quantifying power-law behavior in empirical data. Our approach combines maximum-likelihood fitting methods with goodness-of-fit tests based on the Kolmogorov-Smirnov (KS) statistic and likelihood ratios. We evaluate the effectiveness of the approach with tests on synthetic data and give critical comparisons to previous approaches. We also apply the proposed methods to twenty-four real-world data sets from a range of different disciplines, each of which has been conjectured to follow a power-law distribution. In some cases we find these conjectures to be consistent with the data, while in others the power law is ruled out.
8,753 citations
TL;DR: The aim of this paper is to introduce a few key notions and applications connected to sparsity, targeting newcomers interested in either the mathematical aspects of this area or its applications.
Abstract: A full-rank matrix ${\bf A}\in \mathbb{R}^{n\times m}$ with $n
2,372 citations
TL;DR: The history and basic formulation of WENO schemes are reviewed, the main ideas in using WenO schemes to solve various hyperbolic PDEs and other convection dominated problems are outlined, and a collection of applications in areas including computational fluid dynamics, computational astronomy and astrophysics, semiconductor device simulation, traffic flow models, computational biology, and some non-PDE applications are presented.
Abstract: High order accurate weighted essentially nonoscillatory (WENO) schemes are relatively new but have gained rapid popularity in numerical solutions of hyperbolic partial differential equations (PDEs) and other convection dominated problems. The main advantage of such schemes is their capability to achieve arbitrarily high order formal accuracy in smooth regions while maintaining stable, nonoscillatory, and sharp discontinuity transitions. The schemes are thus especially suitable for problems containing both strong discontinuities and complex smooth solution features. WENO schemes are robust and do not require the user to tune parameters. At the heart of the WENO schemes is actually an approximation procedure not directly related to PDEs, hence the WENO procedure can also be used in many non-PDE applications. In this paper we review the history and basic formulation of WENO schemes, outline the main ideas in using WENO schemes to solve various hyperbolic PDEs and other convection dominated problems, and present a collection of applications in areas including computational fluid dynamics, computational astronomy and astrophysics, semiconductor device simulation, traffic flow models, computational biology, and some non-PDE applications. Finally, we mention a few topics concerning WENO schemes that are currently under investigation.
831 citations
TL;DR: This paper proposes a variation on Hodrick-Prescott (H-P) filtering, a widely used method for trend estimation that substitutes a sum of absolute values for the sum of squares used in H-P filtering to penalize variations in the estimated trend.
Abstract: The problem of estimating underlying trends in time series data arises in a variety of disciplines. In this paper we propose a variation on Hodrick-Prescott (H-P) filtering, a widely used method for trend estimation. The proposed $\ell_1$ trend filtering method substitutes a sum of absolute values (i.e., $\ell_1$ norm) for the sum of squares used in H-P filtering to penalize variations in the estimated trend. The $\ell_1$ trend filtering method produces trend estimates that are piecewise linear, and therefore it is well suited to analyzing time series with an underlying piecewise linear trend. The kinks, knots, or changes in slope of the estimated trend can be interpreted as abrupt changes or events in the underlying dynamics of the time series. Using specialized interior-point methods, $\ell_1$ trend filtering can be carried out with not much more effort than H-P filtering; in particular, the number of arithmetic operations required grows linearly with the number of data points. We describe the method and some of its basic properties and give some illustrative examples. We show how the method is related to $\ell_1$ regularization-based methods in sparse signal recovery and feature selection, and we list some extensions of the basic method.
577 citations
TL;DR: A new backward error analysis of the scaling and squaring method is given that employs sharp bounds for the truncation errors and leads to an implementation of essentially optimal efficiency.
Abstract: The scaling and squaring method is the most widely used method for computing the matrix exponential, not least because it is the method implemented in the MATLAB function expm. The method scales the matrix by a power of 2 to reduce the norm to order 1, computes a Pade approximant to the matrix exponential, and then repeatedly squares to undo the effect of the scaling. We give a new backward error analysis of the method (in exact arithmetic) that employs sharp bounds for the truncation errors and leads to an implementation of essentially optimal efficiency. We also give a new rounding error analysis that shows the computed Pade approximant of the scaled matrix to be highly accurate. For IEEE double precision arithmetic the best choice of degree of Pade approximant turns out to be 13, rather than the 6 or 8 used by previous authors. Our implementation of the scaling and squaring method always requires at least two fewer matrix multiplications than the expm function in MATLAB 7.0 when the matrix norm exceeds 1, which can amount to a 37% saving in the number of multiplications, and it is typically more accurate, owing to the fewer required squarings. We also investigate a different scaling and squaring algorithm proposed by Najfeld and Havel that employs a Pade approximation to the function $x \coth(x)$. This method is found to be essentially a variation of the standard one with weaker supporting error analysis.
309 citations
TL;DR: Recent theoretical and experimental progress on making objects invisible to detection by electromagnetic waves is described and ideas for devices that would once have seemed fanciful may now be at least approximately implemented physically using a new class of artificially structured materials called metamaterials.
Abstract: We describe recent theoretical and experimental progress on making objects invisible to detection by electromagnetic waves. Ideas for devices that would once have seemed fanciful may now be at least approximately implemented physically using a new class of artificially structured materials called metamaterials. Maxwell's equations have transformation laws that allow for the design of electromagnetic material parameters that steer light around a hidden region, returning it to its original path on the far side. Not only would observers be unaware of the contents of the hidden region, they would not even be aware that something was being hidden. An object contained in the hidden region, which would have no shadow, is said to be cloaked. Proposals for, and even experimental implementations of, such cloaking devices have received the most attention, but other designs having striking effects on wave propagation are possible. All of these designs are initially based on the transformation laws of the equations that govern wave propagation but, due to the singular parameters that give rise to the desired effects, care needs to be taken in formulating and analyzing physically meaningful solutions. We recount the recent history of the subject and discuss some of the mathematical and physical issues involved.
276 citations
TL;DR: Results demonstrate that recent trends in memory system organization have reduced the eficacy of traditional cache- blocking optimizations, and represent one of the most extensive analyses of stencil optimizations and performance modeling to date.
Abstract: Stencil-based kernels constitute the core of many important scientific applications on block-structured grids. Unfortunately, these codes achieve a low fraction of peak performance, due primarily to the disparity between processor and main memory speeds. In this paper, we explore the impact of trends in memory subsystems on a variety of stencil optimization techniques and develop performance models to analytically guide our optimizations. Our work targets cache reuse methodologies across single and multiple stencil sweeps, examining cache-aware algorithms as well as cache-oblivious techniques on the Intel Itanium2, AMD Opteron, and IBM Power5. Additionally, we consider stencil computations on the heterogeneous multicore design of the Cell processor, a machine with an explicitly managed memory hierarchy. Overall our work represents one of the most extensive analyses of stencil optimizations and performance modeling to date. Results demonstrate that recent trends in memory system organization have reduced the efficacy of traditional cache-blocking optimizations. We also show that a cache-aware implementation is significantly faster than a cache-oblivious approach, while the explicitly managed memory on Cell enables the highest overall efficiency: Cell attains 88% of algorithmic peak while the best competing cache-based processor achieves only 54% of algorithmic peak performance.
243 citations
TL;DR: Research on multigrid methods for optimization problems is reviewed and problems considered include shape design, parameter optimization, and optimal control problems governed by partial differential equations of elliptic, parabolic, and hyperbolic type.
Abstract: Research on multigrid methods for optimization problems is reviewed. Optimization problems considered include shape design, parameter optimization, and optimal control problems governed by partial differential equations of elliptic, parabolic, and hyperbolic type.
181 citations
TL;DR: This expository paper demonstrates estimation and feedback control of quantum mechanical systems in what is essentially a noncommutative version of the binomial model that is popular in mathematical finance and allows a full development of the theory while remaining completely within the setting of finite-dimensional Hilbert spaces.
Abstract: The engineering and control of devices at the quantum mechanical level—such as those consisting of small numbers of atoms and photons—is a delicate business The fundamental uncertainty that is inherently present at this scale manifests itself in the unavoidable presence of noise, making this a novel field of application for stochastic estimation and control theory In this expository paper we demonstrate estimation and feedback control of quantum mechanical systems in what is essentially a noncommutative version of the binomial model that is popular in mathematical finance The model is extremely rich and allows a full development of the theory while remaining completely within the setting of finite-dimensional Hilbert spaces (thus avoiding the technical complications of the continuous theory) We introduce discretized models of an atom in interaction with the electromagnetic field, obtain filtering equations for photon counting and homodyne detection, and solve a stochastic control problem using dynamic programming and Lyapunov function methods
133 citations
TL;DR: A methodology is introduced that generates a sequence converging to the volume of $\mathbf{K}$ that is obtained from optimal values of a hierarchy of either semidefinite or linear programs, and every finite vector of moments of the probability measure can be approximated as closely as desired.
Abstract: Given a basic compact semialgebraic set $\mathbf{K}\subset\mathbb{R}^n$, we introduce a methodology that generates a sequence converging to the volume of $\mathbf{K}$. This sequence is obtained from optimal values of a hierarchy of either semidefinite or linear programs. Not only the volume but also every finite vector of moments of the probability measure that is uniformly distributed on $\mathbf{K}$ can be approximated as closely as desired, which permits the approximation of the integral on $\mathbf{K}$ of any given polynomial; the extension to integration against some weight functions is also provided. Finally, some numerical issues associated with the algorithms involved are briefly discussed.
TL;DR: In this paper, a survey of acute and nonobtuse simplices and associated spatial partitions is presented, including path-simplices, the generalization of right triangles to higher dimensions.
Abstract: This paper surveys some results on acute and nonobtuse simplices and associated spatial partitions. These partitions are relevant in numerical mathematics, including piecewise polynomial approximation theory and the finite element method. Special attention is paid to a basic type of nonobtuse simplices called path-simplices, the generalization of right triangles to higher dimensions. In addition to applications in numerical mathematics, we give examples of the appearance of acute and nonobtuse simplices in other areas of mathematics.
TL;DR: It is shown how the computational power and programmability of modern graphics processing units (GPUs) can be used to efficiently solve large-scale pixel-based material distribution problems using a gradient-based optimality criterion method.
Abstract: We show how the computational power and programmability of modern graphics processing units (GPUs) can be used to efficiently solve large-scale pixel-based material distribution problems using a gradient-based optimality criterion method. To illustrate the principle, a so-called topology optimization problem that results in a constrained nonlinear programming problem with over 4 million decision variables is solved on a commodity GPU.
TL;DR: An overview of vortex lattice theory from 1957 to the present is given, highlighting recent experiments in Bose-Einstein condensate systems and formulating questions that can be addressed by understanding the singular value decomposition of the configuration matrix.
Abstract: Recent experiments on the formation of vortex lattices in Bose-Einstein condensates has produced the need for a mathematical theory that is capable of predicting a broader class of lattice patterns, ones that are free of discrete symmetries and can form in a random environment. We give an overview of an $N$-particle based Hamiltonian theory which, if formulated in terms of the $O(N^2)$ interparticle distances, leads to the analysis of a nonnormal “configuration” matrix whose nullspace structure determines the existence or nonexistence of a lattice. The singular value decomposition of this matrix leads to a method in which all lattice patterns and the associated particle strengths, in principle, can be classified and calculated by a random-walk scheme which systematically uses the $m$ smallest singular values as a ratchet mechanism to home in on lattices with $m$-dimensional nullspaces, where $0 < m \le N$. The resulting distribution of singular values encodes detailed geometric properties of the lattice and allows us to identify and calculate important quantitative measures associated with each lattice, including its size (as measured by the Frobenius or 2-norms), distance between the lattices (hence lattice density), robustness, and Shannon entropy as a quantitative measure of its level of disorder. This article gives an overview of vortex lattice theory from 1957 to the present, highlighting recent experiments in Bose-Einstein condensate systems and formulating questions that can be addressed by understanding the singular value decomposition of the configuration matrix. We discuss some of the computational challenges associated with producing large $N$ lattices, the subtleties associated with understanding and exploiting complicated Hamiltonian energy surfaces in high dimensions, and we highlight ten important directions for future research in this area.
TL;DR: It is shown that the gap in the spectrum of the alignment matrix is proportional to the square of the size of the overlap of the local parameterizations, thus deriving a quantitative measure of how stably the null space can be computed numerically.
Abstract: Local methods for manifold learning generate a collection of local parameterizations which is then aligned to produce a global parameterization of the underlying manifold. The alignment procedure is carried out through the computation of a partial eigendecomposition of a so-called alignment matrix. In this paper, we present an analysis of the eigenstructure of the alignment matrix giving both necessary and sufficient conditions under which the null space of the alignment matrix recovers the global parameterization. We show that the gap in the spectrum of the alignment matrix is proportional to the square of the size (the precise definition of the size is given in section sec:sp) of the overlap of the local parameterizations, thus deriving a quantitative measure of how stably the null space can be computed numerically. We also give a perturbation analysis of the null space of the alignment matrix when the computation of the local parameterizations is subject to error. Our analysis provides insights into the behaviors and performance of local manifold learning algorithms.
TL;DR: A model to describe the optimal distributions of residents and services in a prescribed urban area using the Monge-Kantorovich mass transportation theory and the theory of nonconvex functionals defined on measures is proposed.
Abstract: We propose a model to describe the optimal distributions of residents and services in a prescribed urban area. The cost functional takes into account the transportation costs (according to a Monge-Kantorovich-type criterion) and two additional terms which penalize concentration of residents and dispersion of services. The tools we use are the Monge-Kantorovich mass transportation theory and the theory of nonconvex functionals defined on measures.
TL;DR: It is shown how the spectral perspective allows the extension of the conjugate point approach to variants of the classic problems in the literature, such as problems with Neumann-Neumann boundary conditions.
Abstract: The theory of conjugate points in the calculus of variations is reconsidered with a perspective emphasizing the connection to finite-dimensional optimization. The object of central importance is the spectrum of the second-variation operator, analogous to the eigenvalues of the Hessian matrix in finite dimensions. With a few basic properties of this spectrum, one can gain a new perspective on the classic result that “stability requires the lack of conjugate points.” Furthermore, we show how the spectral perspective allows the extension of the conjugate point approach to variants of the classic problems in the literature, such as problems with Neumann-Neumann boundary conditions.
TL;DR: The so-called Kac ring model is described, suggested by Mark Kac in 1956, which illustrates coarse-graining in a setting so simple that all aspects can be exposed both through elementary, explicit computation and through easy numerical simulation.
Abstract: The process of coarse-graining—here, in particular, of passing from a deterministic, simple, and time-reversible dynamics at the microscale to a typically irreversible description in terms of averaged quantities at the macroscale—is of fundamental importance in science and engineering. At the same time, it is often difficult to grasp and, if not interpreted correctly, implies seemingly paradoxical results. The kinetic theory of gases, historically the first and arguably most significant example, occupied physicists for the better part of the 19th century and continues to pose mathematical challenges to this day.
In this paper, we describe the so-called Kac ring model, suggested by Mark Kac in 1956, which illustrates coarse-graining in a setting so simple that all aspects can be exposed both through elementary, explicit computation and through easy numerical simulation. In this setting, we explain a Boltzmannian “Stoszahlansatz,” ensemble averages, the difference between ensemble averaged and “typical” system behavior, and the notion of entropy.
[...]
TL;DR: This paper considers several variants of the hat guessing problem, united by the common theme that the guessing strategies are required to be deterministic and the objective is to maximize the number of correct answers in the worst case.
Abstract: Hat problems have become a popular topic in recreational mathematics. In a typical hat problem, each of $n$ players tries to guess the color of the hat he or she is wearing by looking at the colors of the hats worn by some of the other players. In this paper we consider several variants of the problem, united by the common theme that the guessing strategies are required to be deterministic and the objective is to maximize the number of correct answers in the worst case. We also summarize what is currently known about the worst-case analysis of deterministic hat guessing problems with a finite number of players.
TL;DR: This paper characterized the close relationship between these distributed dynamical systems and the disk-covering and sphere-packing cost functions from geometric optimization and established various asymptotic convergence properties of the laws.
Abstract: Emerging applications for networked and cooperative robots motivate the study of motion coordination for groups of agents. For example, it is envisioned that groups of agents will perform a variety of useful tasks including surveillance, exploration, and environmental monitoring. This paper deals with basic interactions among mobile agents such as “move away from the closest other agent” or “move toward the furthest vertex of your own Voronoi polygon.” These simple interactions amount to distributed dynamical systems because their implementation requires only minimal information about neighboring agents. We characterize the close relationship between these distributed dynamical systems and the disk-covering and sphere-packing cost functions from geometric optimization. Our main results are as follows: (i) we characterize the smoothness properties of these geometric cost functions, (ii) we show that the interaction laws are variations of the nonsmooth gradient of the cost functions, and (iii) we establish various asymptotic convergence properties of the laws. The technical approach relies on concepts from computational geometry, nonsmooth analysis, and nonsmooth stability theory.
TL;DR: This work proposes a least-squares-based halftoning model with a substantial contribution from the HVS model, and designs a robust computational algorithm based on Markov random walks, and rigorously proves the gradient-descent property of the Markov-stochastic algorithm.
Abstract: Halftoning is the core algorithm governing most digital printing or imaging devices, by which images of continuous tones are converted to ensembles of discrete or quantum dots. It is through the human vision system (HVS) that such fields of quantum dots can be perceived to be almost identical to the original continuous images. In the current work, we propose a least-squares-based halftoning model with a substantial contribution from the HVS model, and we design a robust computational algorithm based on Markov random walks. Furthermore, we discuss and quantify the important role of spatial smoothing by the HVS and rigorously prove the gradient-descent property of the Markov-stochastic algorithm. Computational results on typical test images further confirm the performance of the new approach. The proposed algorithm and its mathematical analysis are generically applicable to similar discrete or nonlinear programming problems.
TL;DR: Energy transfers from internal energy to surface energy are found to be comparable to viscously dissipated energy in thermo-capillary systems, which is essential to assessing the validity of numerical simulations of such systems.
Abstract: We examine the energy budget associated with a multiphase fluid system in the presence of a fluid-fluid interface with variable surface tension. Such an interface is found in thermo-capillary systems and also describes solutions of miscible liquids separated from a third fluid by an interface. We incorporate local density variations and derive an energy budget from the Navier-Stokes and advection-diffusion equations. In addition to the usual potential, kinetic, surface, and dissipated energy, three additional terms are found which correspond to transfers from thermodynamic internal energy to surface, potential, or kinetic energy, respectively. Energy transfers from internal energy to surface energy are found to be comparable to viscously dissipated energy in thermo-capillary systems. Taking into account such transfers is essential to assessing the validity of numerical simulations of such systems.
TL;DR: In this paper, the mathematics of an implementation of an iris recognition system is examined and it is shown that the determination of a crucial spread parameter is made on implicit assumptions that are not fulfilled.
Abstract: Most probability and statistics textbooks are loaded with dice, coins, and balls in urns. These are perfect metaphors for actual phenomena where uncertainty plays a role. However, students will greatly appreciate a real-life example. In this paper we examine the mathematics of an implementation of an iris recognition system. We show that the determination of a crucial spread parameter is made on implicit assumptions that are not fulfilled. Some elementary probability theory calculations show that this leads in general to an optimistic assessment of the reliability of the identification system. Then we use a famous inequality to quantify this optimism.
TL;DR: A solution to the problem of pricing a perpetual warrant of the American type using linear programming duality under a simple random walk assumption for the stock price process is offered, enabling a classroom exposition of the problem in graduate courses on linear programming without assuming a background in stochastic processes.
Abstract: A warrant is an option that entitles the holder to purchase shares of a common stock at some prespecified price during a specified interval. The problem of pricing a perpetual warrant (with no specified interval) of the American type (that can be exercised any time) is one of the earliest contingent claim pricing problems in mathematical economics. The problem was first solved by Samuelson and McKean in 1965 under the assumption of a geometric Brownian motion of the stock price process. It is a well-documented exercise in stochastic processes and continuous-time finance curricula. The present paper offers a solution to this time-honored problem from an optimization point of view using linear programming duality under a simple random walk assumption for the stock price process, thus enabling a classroom exposition of the problem in graduate courses on linear programming without assuming a background in stochastic processes.
Journal Article•
TL;DR: The two papers in this issue deal with differential equations, one with the numerical solution of partial differential equation solutions, and the other one with analytic solutions for ordinary differential equations.
Abstract: The two papers in this issue deal with differential equations, one with the numerical solution of partial differential equations, and the other one with analytic solutions for ordinary differential equations.
In his paper "From Functional Analysis to Iterative Methods", Robert Kirby is concerned with linear systems arising from discretizations of partial differential equations (PDEs). Specifically, the PDEs are elliptic and describe boundary value problems; the discretizations are done via finite elements, and at issue is the convergence rate of iterative methods for solving the linear systems. The author's approach is to go back to the underlying variational problem in a Hilbert space, and to make ample use of the Riesz representation theorem. This point of view results in short and elegant proofs, as well as the construction of efficient preconditioners. The general theory is illustrated with two concrete model problems of PDEs for convection diffusion and planar elasticity. This paper will appeal to anybody who has an interest in the numerical solution of PDEs.
In 1963 the mathematician/meteorologist Edward Lorenz formulated a system of three coupled nonlinear ordinary differential equations, whose long-term behavior is described by an attractor with fractal structure. You can see a beautiful rendition of the thus named Lorenz attractor on the cover of this issue. Although it is "easy" to plot solutions of the Lorenz system, it is much harder to determine them mathematically. This is what motivated the paper "Complex Singularities and the Lorenz Attractor" by Divakar Viswanath and Sonmez Sahutoglu. Their idea is to allow the time variable to be complex, rather than real; to focus on singular solutions; and to express these singular solutions in terms of so-called psi series. After all is said and done, the authors end up with a two-parameter family of complex solutions to the Lorenz system. This a highly readable and very enjoyable paper, with concrete steps for future research, and connections to thunderstorms and analytic function theory.