Showing papers in "Siam Review in 2018"
TL;DR: The authors provides a review and commentary on the past, present, and future of numerical optimization algorithms in the context of machine learning applications and discusses how optimization problems arise in machine learning and what makes them challenging.
Abstract: This paper provides a review and commentary on the past, present, and future of numerical optimization algorithms in the context of machine learning applications. Through case studies on text classification and the training of deep neural networks, we discuss how optimization problems arise in machine learning and what makes them challenging. A major theme of our study is that large-scale machine learning represents a distinctive setting in which the stochastic gradient (SG) method has traditionally played a central role while conventional gradient-based nonlinear optimization techniques typically falter. Based on this viewpoint, we present a comprehensive theory of a straightforward, yet versatile SG algorithm, discuss its practical behavior, and highlight opportunities for designing algorithms with improved performance. This leads to a discussion about the next generation of optimization methods for large-scale machine learning, including an investigation of two main streams of research on techniques th...
2,238 citations
TL;DR: In many situations across computational science and engineering, multiple computational models are available that describe a system of interest as discussed by the authors, and these different models have varying evaluation costs, i.e.
Abstract: In many situations across computational science and engineering, multiple computational models are available that describe a system of interest. These different models have varying evaluation costs...
678 citations
TL;DR: Random graph null models have found widespread application in diverse research communities analyzing network datasets, including social, information, and economic networks, as well as food webs, pr....
Abstract: Random graph null models have found widespread application in diverse research communities analyzing network datasets, including social, information, and economic networks, as well as food webs, pr...
172 citations
TL;DR: The computation of the singular value decomposition, or SVD, has a long history with many improvements over the years, both in its implementations and algorithmically.
Abstract: The computation of the singular value decomposition, or SVD, has a long history with many improvements over the years, both in its implementations and algorithmically. Here, we survey the evolution...
76 citations
TL;DR: A general framework for Shanks transformations of sequences of elements in a vector space is presented and it is shown that Minimal Polynomial Extrapolation (MPE), Modified Minimal polynomial E...
Abstract: This paper presents a general framework for Shanks transformations of sequences of elements in a vector space. It is shown that Minimal Polynomial Extrapolation (MPE), Modified Minimal Polynomial E...
68 citations
TL;DR: An essay on the mathematical development of inverse scattering theory for time-harmonic waves over the past fifty years together with some personal memories of their participation in these studies are presented.
Abstract: We present an essay on the mathematical development of inverse scattering theory for time-harmonic waves over the past fifty years together with some personal memories of our participation in these...
54 citations
TL;DR: As institutions consider new and evolving educational programs, it is essential to consider the broader research challenges and opportunities that provide the context for CSE education and workforce development.
Abstract: This report presents challenges, opportunities and directions for computational science and engineering (CSE) research and education for the next decade. Over the past two decades the field of CSE has penetrated both basic and applied research in academia, industry, and laboratories to advance discovery, optimize systems, support decision-makers, and educate the scientific and engineering workforce. Informed by centuries of theory and experiment, CSE performs computational experiments to answer questions that neither theory nor experiment alone is equipped to answer. CSE provides scientists and engineers with algorithmic inventions and software systems that transcend disciplines and scales. CSE brings the power of parallelism to bear on troves of data. Mathematics-based advanced computing has become a prevalent means of discovery and innovation in essentially all areas of science, engineering, technology, and society; and the CSE community is at the core of this transformation. However, a combination of disruptive developments—including the architectural complexity of extreme-scale computing, the data revolution and increased attention to data-driven discovery, and the specialization required to follow the applications to new frontiers—is redefining the scope and reach of the CSE endeavor. With these many current and expanding opportunities for the CSE field, there is a growing demand for CSE graduates and a need to expand CSE educational offerings. This need includes CSE programs at both the undergraduate and graduate levels, as well as continuing education and professional development programs, exploiting the synergy between computational science and data science. Yet, as institutions consider new and evolving educational programs, it is essential to consider the broader research challenges and opportunities that provide the context for CSE education and workforce development.
53 citations
TL;DR: Noise plays a fundamental role in a wide variety of physical and biological dynamical systems as discussed by the authors, and it can arise from an external forcing or due to random dynamics internal to the system.
Abstract: Noise plays a fundamental role in a wide variety of physical and biological dynamical systems. It can arise from an external forcing or due to random dynamics internal to the system. It is well est...
53 citations
TL;DR: Efficient schemes are developed to dynamically evolve the rank of the reduced solution and to ensure the orthogonality of the basis matrix while preserving its smooth evolution over time.
Abstract: Quantifying the uncertainty of Lagrangian motion can be performed by solving a large number of ordinary differential equations with random velocities or, equivalently, a stochastic transport partia...
51 citations
TL;DR: In this article, a high-dimensional phase space and a large number of instabilities are ubiquitous among many complex systems in science and engineering, including dynamical systems characterized by both a high dimensional phase space (HDS) and a number of dynamical instabilities.
Abstract: Turbulent dynamical systems characterized by both a high-dimensional phase space and a large number of instabilities are ubiquitous among many complex systems in science and engineering, including ...
51 citations
TL;DR: In this article, an innovative Gillespie algorithm for renewal processes on the basis of the Laplace transform has been proposed, which makes use of the fact that a class of point processes is represented as a mixture of Poisson processes with different ev...
Abstract: The Gillespie algorithm provides statistically exact methods for simulating stochastic dynamics modeled as interacting sequences of discrete events including systems of biochemical reactions or earthquake occurrences, networks of queuing processes or spiking neurons, and epidemic and opinion formation processes on social networks. Empirically, the inter-event times of various phenomena obey long-tailed distributions. The Gillespie algorithm and its variants either assume Poisson processes (i.e., exponentially distributed inter-event times), use particular functions for time courses of the event rate, or work for non-Poissonian renewal processes, including the case of long-tailed distributions of inter-event times, but at a high computational cost. In the present study, we propose an innovative Gillespie algorithm for renewal processes on the basis of the Laplace transform. The algorithm makes use of the fact that a class of point processes is represented as a mixture of Poisson processes with different ev...
TL;DR: Evidence is presented of universal behavior in modulationally unstable media, including an ensemble of nonlinear evolution equations that arise in a variety of applications in the physical and mathematical sciences, including water waves, optics, acoustics, Bose-Einstein condensation, and more.
Abstract: Evidence is presented of universal behavior in modulationally unstable media. An ensemble of nonlinear evolution equations, including three partial differential equations, an integro-differential e...
TL;DR: It is shown that a so-called Catch-Up Rule for determining the order of kicking would not only make the penalty shootout fairer but also is essentially strategyproof.
Abstract: The rules of many sports are not fair---they do not ensure that equally skilled competitors have the same probability of winning. As an example, the penalty shootout in soccer, wherein a coin toss determines which team kicks first on all five penalty kicks, gives a substantial advantage to the first-kicking team, both in theory and in practice. We show that a so-called Catch-Up Rule for determining the order of kicking would not only make the shootout fairer but is also essentially strategyproof. By contrast, the so-called Standard Rule now used for the tiebreaker in tennis is fair. We briefly consider several other sports, all of which involve scoring a sufficient number of points to win, and show how they could benefit from certain rule changes which would be straightforward to implement.
TL;DR: This paper studies how the right preconditioner changes the Krylov subspaces where the CGLS iterates live, and draws a tighter connection between Bayesian inference and KrylovSubspace methods.
Abstract: The solution of linear inverse problems when the unknown parameters outnumber data requires addressing the problem of a nontrivial null space. After restating the problem within the Bayesian framework, a priori information about the unknown can be utilized for determining the null space contribution to the solution. More specifically, if the solution of the associated linear system is computed by the conjugate gradient for least squares (CGLS) method, the additional information can be encoded in the form of a right preconditioner. In this paper we study how the right preconditioner changes the Krylov subspaces where the CGLS iterates live, and we draw a tighter connection between Bayesian inference and Krylov subspace methods. The advantages of a Bayes-meets-Krylov approach to the solution of underdetermined linear inverse problems is illustrated with computed examples.
TL;DR: A high frequency (HF) trading strategy where the HF trader uses her superior speed to process information and to post limit sell and buy orders.
Abstract: We develop a high frequency (HF) trading strategy where the HF trader uses her superior speed to process information and to post limit sell and buy orders. By introducing a multifactor mutually exc...
TL;DR: New analytical and numerical methods for the plasmonic spheres system are developed by clarifying the connection between transformation optics and the method of image charges, and a hybrid numerical scheme for computing the field distribution produced by an arbitrary number of spheres is developed.
Abstract: When metallic (or plasmonic) nanospheres are nearly touching, strong concentration of light can occur in the narrow gap regions. This phenomenon has potential applications in nanophotonics, biosensing, and spectroscopy. Understanding the strong interaction between the plasmonic spheres turns out to be quite challenging; indeed, an extremely high computational cost is required to compute the electromagnetic field. Also, the classical method of image charges, which is effective for the dielectric spheres system, is not valid for plasmonic spheres because of their negative permittivities. Here we develop new analytical and numerical methods for the plasmonic spheres system by clarifying the connection between transformation optics and the method of image charges. We derive fully analytic solutions valid for two plasmonic spheres. We then develop a hybrid numerical scheme for computing the field distribution produced by an arbitrary number of spheres. Our method is highly efficient and accurate even in the ne...
TL;DR: This work establishes rigorously that each iterate produced by one of these three algorithms can be viewed as a Newton's method iterate followed by a normalization.
Abstract: The $l_2$ normalized inverse, shifted inverse, and Rayleigh quotient iterations are classic algorithms for approximating an eigenvector of a symmetric matrix. This work establishes rigorously that each iterate produced by one of these three algorithms can be viewed as a Newton's method iterate followed by a normalization. The equivalences given here are not meant to suggest changes to the implementations of the classic eigenvalue algorithms. However, they add further understanding to the formal structure of these iterations, and they provide an explanation for their good behavior despite the possible need to solve systems with nearly singular coefficient matrices. A historical development of these eigenvalue algorithms is presented. Using our equivalences and traditional Newton's method theory helps to gain understanding as to why normalized Newton's method, inverse iteration, and shifted inverse iteration are only linearly convergent and not quadratically convergent, as would be expected, and why a new l...
TL;DR: The asymptotic density of spots and their heights for any spatially dependent feed rate $A(x)$ is characterized and a novel phenomenon which only happens when the feed rate is sufficiently inhomogeneous in space is demonstrated.
Abstract: We develop novel mathematical techniques to study spot patterns in reaction-diffusion systems with space-dependent feed rate. The techniques are illustrated on the Schnakenberg model, which is a pr...
TL;DR: In this article, the authors discuss the way transitions can be reduced to discontinuities without trivializing them, by preserving so-called hidden terms, and present a prototype for piecewise-smooth models from the asymptotics of systems with rapid transitions.
Abstract: Transitions between steady dynamical regimes in diverse applications are often modeled using discontinuities, but doing so introduces problems of uniqueness. No matter how quickly a transition occurs, its inner workings can affect the dynamics of the system significantly. Here we discuss the way transitions can be reduced to discontinuities without trivializing them, by preserving so-called hidden terms. We review the fundamental methodology, its motivations, and where their study seems to be heading. We derive a prototype for piecewise-smooth models from the asymptotics of systems with rapid transitions, sharpening Filippov's convex combinations by encoding the tails of asymptotic series into nonlinear dependence on a switching parameter. We present a few examples that illustrate the impact of these on our standard picture of smooth or only piecewise-smooth dynamics.
TL;DR: In this article, a general mathematical framework for trajectory stratification for simulating rare events is presented, which decomposes trajectories of the underlying process into fraggences, which are then used to simulate rare events.
Abstract: We present a general mathematical framework for trajectory stratification for simulating rare events. Trajectory stratification involves decomposing trajectories of the underlying process into frag...
TL;DR: Variational problems that involve Wasserstein distances and more generally optimal transport theory are playing an increasingly important role in data sciences as discussed by the authors, and such problems can be used to solve data problems.
Abstract: Variational problems that involve Wasserstein distances and more generally optimal transport (OT) theory are playing an increasingly important role in data sciences. Such problems can be used to fo...
TL;DR: This contribution summarizes the state of such models in the simulation of electric circuits via cosimulation technique and addresses the existence of solutions for these complex systems as well as structural properties as the DAE index.
Abstract: Modeling with partial differential-algebraic equations is a natural and universal approach valid for various applications with coupled subsystems. This contribution summarizes the state of such models in the simulation of electric circuits; that is, we place known facts and techniques into an overall context. In fact, we mainly discuss the modeling and analysis aspects of several important settings. In the modeling context, we embed the network equations into the context of Maxwell's equations and address the three main types of coupling: modeling with subsystems of the same type, refined models, and multiphysics. In the analysis context, we address the existence of solutions for these complex systems as well as structural properties as the DAE index (after spatial semidiscretization). For the numerical simulations, we give results for the cosimulation technique (also referred to as dynamic iteration), which is a standard method for coupled systems.
TL;DR: Fantasy sports have experienced a surge in popularity in the past decade and one of the consequences of this recent rapid growth is increased scrutiny surrounding the legal aspects of the games.
Abstract: Fantasy sports have experienced a surge in popularity in the past decade. One of the consequences of this recent rapid growth is increased scrutiny surrounding the legal aspects of the games, which...
TL;DR: The main purpose of this paper is to present some of the recent results about the second-order necessary conditions for stochastic optimal controls with the control variable entering into both the drift and the diffusion terms.
Abstract: The main purpose of this paper is to present some of our recent results about the second-order necessary conditions for stochastic optimal controls with the control variable entering into both the drift and the diffusion terms. In particular, when the control region is convex, a pointwise second-order necessary condition for stochastic singular optimal controls in the classical sense is established, whereas when the control region is allowed to be nonconvex, we obtain a pointwise second-order necessary condition for stochastic singular optimal controls in the sense of the Pontryagin-type maximum principle. Unlike deterministic optimal control problems or stochastic optimal control problems with control-independent diffusions, there exist some essential difficulties in deriving the pointwise second-order necessary optimality conditions from the integral conditions when the controls act in the diffusion terms of the stochastic control systems. Some techniques from Malliavin calculus are employed to overcome...
TL;DR: A Markov jump process approximation for SPDEs, which is referred to as the spectral random walk method (SPECTRWM) is introduced and the accuracy and ergodicity are verified in the context of a heat and overdamped Langevin SPDE, respectively.
Abstract: The numerical solution of stochastic partial differential equations (SPDEs) presents challenges not encountered in the simulation of PDEs or SDEs. Indeed, the roughness of the noise in conjunction with nonlinearities in the drift typically make it difficult to construct, operate, and validate numerical methods for SPDEs. This is especially true if one is interested in path-dependent expected values, long-time simulations, or in the simulation of SPDEs whose solutions have constraints on their domains. To address these numerical issues, this paper introduces a Markov jump process approximation for SPDEs, which we refer to as the spectral random walk method (SPECTRWM). The accuracy and ergodicity of SPECTRWM are verified in the context of a heat and an overdamped Langevin SPDE, respectively. We also apply the method to Burgers and KPZ SPDEs. The article includes a MATLAB implementation of SPECTRWM.
TL;DR: This work compares two different linear programming (LP) relaxations of the metric TSP, namely, the path version of the Held--Karp LP relaxation for the TSP and a weaker LP relaxation, and shows that both LPs have the same (fractional) optimal value.
Abstract: We study the metric $s$--$t$ path traveling salesman problem (TSP) An, Kleinberg, and Shmoys [Proceedings of the 44th ACM Symposium on Theory of Computing, 2012, pp 875--886] improved on the long-standing $\frac{5}{3}$-approximation factor and presented an algorithm that achieves an approximation factor of $\frac{1+\sqrt{5}}{2}\approx161803$ Later, Sebo [Proceedings of the 16th Conference on Integer Programming and Combinatorial Optimization, 2013, pp 362--374] further improved the approximation factor to $\frac{8}{5}$ We present a simple, self-contained analysis that unifies both results; our main contribution is a unified correction vector Additionally, we compare two different linear programming (LP) relaxations of the $s$--$t$ path TSP, namely, the path version of the Held--Karp LP relaxation for the TSP and a weaker LP relaxation, and we show that both LPs have the same (fractional) optimal value Also, we show that the minimum cost of integral solutions of the two LPs are within a factor of $