Steven T. Smith

Bio: Steven T. Smith is an academic researcher from Massachusetts Institute of Technology. The author has contributed to research in topics: Adaptive filter & Power graph analysis. The author has an hindex of 18, co-authored 45 publications receiving 3942 citations. Previous affiliations of Steven T. Smith include Harvard University.

The Geometry of Algorithms with Orthogonality Constraints

Abstract: In this paper we develop new Newton and conjugate gradient algorithms on the Grassmann and Stiefel manifolds. These manifolds represent the constraints that arise in such areas as the symmetric eigenvalue problem, nonlinear eigenvalue problems, electronic structures computations, and signal processing. In addition to the new algorithms, we show how the geometrical framework gives penetrating new insights allowing us to create, understand, and compare algorithms. The theory proposed here provides a taxonomy for numerical linear algebra algorithms that provide a top level mathematical view of previously unrelated algorithms. It is our hope that developers of new algorithms and perturbation theories will benefit from the theory, methods, and examples in this paper.

Optimization Techniques on Riemannian Manifolds.

Abstract: The techniques and analysis presented in this paper provide new methods to solve optimization problems posed on Riemannian manifolds. A new point of view is offered for the solution of constrained optimization problems. Some classical optimization techniques on Euclidean space are generalized to Riemannian manifolds. Several algorithms are presented and their convergence properties are analyzed employing the Riemannian structure of the manifold. Specifically, two apparently new algorithms, which can be thought of as Newton's method and the conjugate gradient method on Riemannian manifolds, are presented and shown to possess, respectively, quadratic and superlinear convergence. Examples of each method on certain Riemannian manifolds are given with the results of numerical experiments. Rayleigh's quotient defined on the sphere is one example. It is shown that Newton's method applied to this function converges cubically, and that the Rayleigh quotient iteration is an efficient approximation of Newton's method. The Riemannian version of the conjugate gradient method applied to this function gives a new algorithm for finding the eigenvectors corresponding to the extreme eigenvalues of a symmetric matrix. Another example arises from extremizing the function $\mathop{\rm tr} {\Theta}^{\scriptscriptstyle\rm T}Q{\Theta}N$ on the special orthogonal group. In a similar example, it is shown that Newton's method applied to the sum of the squares of the off-diagonal entries of a symmetric matrix converges cubically.

Covariance, subspace, and intrinsic Crame/spl acute/r-Rao bounds

Abstract: Crame/spl acute/r-Rao bounds on estimation accuracy are established for estimation problems on arbitrary manifolds in which no set of intrinsic coordinates exists. The frequently encountered examples of estimating either an unknown subspace or a covariance matrix are examined in detail. The set of subspaces, called the Grassmann manifold, and the set of covariance (positive-definite Hermitian) matrices have no fixed coordinate system associated with them and do not possess a vector space structure, both of which are required for deriving classical Crame/spl acute/r-Rao bounds. Intrinsic versions of the Crame/spl acute/r-Rao bound on manifolds utilizing an arbitrary affine connection with arbitrary geodesics are derived for both biased and unbiased estimators. In the example of covariance matrix estimation, closed-form expressions for both the intrinsic and flat bounds are derived and compared with the root-mean-square error (RMSE) of the sample covariance matrix (SCM) estimator for varying sample support K. The accuracy bound on unbiased covariance matrix estimators is shown to be about (10/log 10)n/K/sup 1/2/ dB, where n is the matrix order. Remarkably, it is shown that from an intrinsic perspective, the SCM is a biased and inefficient estimator and that the bias term reveals the dependency of estimation accuracy on sample support observed in theory and practice. The RMSE of the standard method of estimating subspaces using the singular value decomposition (SVD) is compared with the intrinsic subspace Crame/spl acute/r-Rao bound derived in closed form by varying both the signal-to-noise ratio (SNR) of the unknown p-dimensional subspace and the sample support. In the simplest case, the Crame/spl acute/r-Rao bound on subspace estimation accuracy is shown to be about (p(n-p))/sup 1/2-1/2/SNR/sup -1/2/ rad for p-dimensional subspaces. It is seen that the SVD-based method yields accuracies very close to the Crame/spl acute/r-Rao bound, establishing that the principal invariant subspace of a random sample provides an excellent estimator of an unknown subspace. The analysis approach developed is directly applicable to many other estimation problems on manifolds encountered in signal processing and elsewhere, such as estimating rotation matrices in computer vision and estimating subspace basis vectors in blind source separation.

Statistical resolution limits and the complexified Crame/spl acute/r-Rao bound

Abstract: Array resolution limits and accuracy bounds on the multitude of signal parameters (e.g., azimuth, elevation, Doppler, range, cross-range, depth, frequency, chirp, polarization, amplitude, phase, etc.) estimated by array processing algorithms are essential tools in the evaluation of system performance. The case in which the complex amplitudes of the signals are unknown is of particular practical interest. A computationally efficient formulation of these bounds (from the perspective of derivations and analysis) is presented for the case of deterministic and unknown signal amplitudes. A new derivation is given using the unknown complex signal parameters and their complex conjugates. The new formula is readily applicable to obtaining either symbolic or numerical solutions to estimation bounds for a very wide class of problems encountered in adaptive sensor array processing. This formula is shown to yield several of the standard Crame/spl acute/r-Rao results for array processing, along with new results of fundamental interest. Specifically, a new closed-form expression for the statistical resolution limit of an aperture for any asymptotically unbiased superresolution algorithm (e.g., MUSIC, ESPRIT) is provided. The statistical resolution limit is defined as the source separation that equals its own Crame/spl acute/r-Rao bound, providing an algorithm-independent bound on the resolution of any high-resolution method. It is shown that the statistical resolution limit of an array or coherent integration window is about 1.2/spl middot/SNR/sup -1/4/ relative to the Fourier resolution limit of 2/spl pi//N radians (large number N of array elements). That is, the highest achievable resolution is proportional to the reciprocal of the fourth root of the signal-to-noise ratio (SNR), in contrast to the square-root (SNR/sup -1/2/) dependence of standard accuracy bounds. These theoretical results are consistent with previously published bounds for specific superresolution algorithms derived by other methods. It is also shown that the potential resolution improvement obtained by separating two collinear arrays (synthetic ultra-wideband), each with a fixed aperture B wavelengths by M wavelengths (assumed large), is approximately (M/B)/sup 1/2/, in contrast to the resolution improvement of M/B for a full aperture. Exact closed-form results for these problems with their asymptotic approximations are presented.

The Geometry of Algorithms with Orthogonality Constraints

Abstract: In this paper we develop new Newton and conjugate gradient algorithms on the Grassmann and Stiefel manifolds. These manifolds represent the constraints that arise in such areas as the symmetric eigenvalue problem, nonlinear eigenvalue problems, electronic structures computations, and signal processing. In addition to the new algorithms, we show how the geometrical framework gives penetrating new insights allowing us to create, understand, and compare algorithms. The theory proposed here provides a taxonomy for numerical linear algebra algorithms that provide a top level mathematical view of previously unrelated algorithms. It is our hope that developers of new algorithms and perturbation theories will benefit from the theory, methods, and examples in this paper.

Block-iterative frequency-domain methods for Maxwell’s equations in a planewave basis

Abstract: We describe a fully-vectorial, three-dimensional algorithm to compute the definite-frequency eigenstates of Maxwell's equations in arbitrary periodic dielectric structures, including systems with anisotropy (birefringence) or magnetic materials, using preconditioned block-iterative eigensolvers in a planewave basis. Favorable scaling with the system size and the number of computed bands is exhibited. We propose a new effective dielectric tensor for anisotropic structures, and demonstrate that O Delta x;2 convergence can be achieved even in systems with sharp material discontinuities. We show how it is possible to solve for interior eigenvalues, such as localized defect modes, without computing the many underlying eigenstates. Preconditioned conjugate-gradient Rayleigh-quotient minimization is compared with the Davidson method for eigensolution, and a number of iteration variants and preconditioners are characterized. Our implementation is freely available on the Web.

The Geometry of Algorithms with Orthogonality Constraints

Abstract: In this paper we develop new Newton and conjugate gradient algorithms on the Grassmann and Stiefel manifolds. These manifolds represent the constraints that arise in such areas as the symmetric eigenvalue problem, nonlinear eigenvalue problems, electronic structures computations, and signal processing. In addition to the new algorithms, we show how the geometrical framework gives penetrating new insights allowing us to create, understand, and compare algorithms. The theory proposed here provides a taxonomy for numerical linear algebra algorithms that provide a top level mathematical view of previously unrelated algorithms. It is our hope that developers of new algorithms and perturbation theories will benefit from the theory, methods, and examples in this paper.

Optimization Algorithms on Matrix Manifolds

Abstract: Many problems in the sciences and engineering can be rephrased as optimization problems on matrix search spaces endowed with a so-called manifold structure. This book shows how to exploit the special structure of such problems to develop efficient numerical algorithms. It places careful emphasis on both the numerical formulation of the algorithm and its differential geometric abstraction--illustrating how good algorithms draw equally from the insights of differential geometry, optimization, and numerical analysis. Two more theoretical chapters provide readers with the background in differential geometry necessary to algorithmic development. In the other chapters, several well-known optimization methods such as steepest descent and conjugate gradients are generalized to abstract manifolds. The book provides a generic development of each of these methods, building upon the material of the geometric chapters. It then guides readers through the calculations that turn these geometrically formulated methods into concrete numerical algorithms. The state-of-the-art algorithms given as examples are competitive with the best existing algorithms for a selection of eigenspace problems in numerical linear algebra. Optimization Algorithms on Matrix Manifolds offers techniques with broad applications in linear algebra, signal processing, data mining, computer vision, and statistical analysis. It can serve as a graduate-level textbook and will be of interest to applied mathematicians, engineers, and computer scientists.