When discrete ill-posed problems are analyzed and solved by various numerical regularization techniques, a very convenient way to display information about the regularized solution is to plot the norm or seminorm of the solution versus the norm of the residual vector. In particular, the graph associated with Tikhonov regularization plays a central role. The main purpose of this paper is to advocate the use of this graph in the numerical treatment of discrete ill-posed problems. The graph is characterized quantitatively, and several important relations between regularized solutions and the graph are derived. It is also demonstrated that several methods for choosing the regularization parameter are related to locating a characteristic L-shaped “corner” of the graph.

https://www.jstor.org/stable/info/2132628

Analysis of discrete ill-posed problems by means of the L-curve

In this book, which focuses on the use of iterative methods for solving large sparse systems of linear equations, templates are introduced to meet the needs of both the traditional user and the high-performance specialist Templates, a description of a general algorithm rather than the executable object or source code more commonly found in a conventional software library, offer whatever degree of customization the user may desire

/pdf/templates-for-the-solution-of-linear-systems-building-blocks-28xlh68wjc.pdf

Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods

The Conjugate Gradient Method is the most prominent iterative method for solving sparse systems of linear equations. Unfortunately, many textbook treatments of the topic are written so that even their own authors would be mystified, if they bothered to read their own writing. For this reason, an understanding of the method has been reserved for the elite brilliant few who have painstakingly decoded the mumblings of their forebears. Nevertheless, the Conjugate Gradient Method is a composite of simple, elegant ideas that almost anyone can understand. Of course, a reader as intelligent as yourself will learn them almost effortlessly. The idea of quadratic forms is introduced and used to derive the methods of Steepest Descent, Conjugate Directions, and Conjugate Gradients. Eigenvectors are explained and used to examine the convergence of the Jacobi Method, Steepest Descent, and Conjugate Gradients. Other topics include preconditioning and the nonlinear Conjugate Gradient Method. I have taken pains to make this article easy to read. Sixty-two illustrations are provided. Dense prose is avoided. Concepts are explained in several different ways. Most equations are coupled with an intuitive interpretation.

/pdf/an-introduction-to-the-conjugate-gradient-method-without-the-4x5ver5l5d.pdf

An Introduction to the Conjugate Gradient Method Without the Agonizing Pain

"Parameter Estimation and Inverse Problems, 2/e" provides geoscience students and professionals with answers to common questions like how one can derive a physical model from a finite set of observations containing errors, and how one may determine the quality of such a model. This book takes on these fundamental and challenging problems, introducing students and professionals to the broad range of approaches that lie in the realm of inverse theory. The authors present both the underlying theory and practical algorithms for solving inverse problems. The authors' treatment is appropriate for geoscience graduate students and advanced undergraduates with a basic working knowledge of calculus, linear algebra, and statistics. "Parameter Estimation and Inverse Problems, 2/e" introduces readers to both Classical and Bayesian approaches to linear and nonlinear problems with particular attention paid to computational, mathematical, and statistical issues related to their application to geophysical problems. The textbook includes Appendices covering essential linear algebra, statistics, and notation in the context of the subject. This book includes a companion website that features computational examples (including all examples contained in the textbook) and useful subroutines using MATLAB. It: includes appendices for review of needed concepts in linear, statistics, and vector calculus; features a companion website that contains comprehensive MATLAB code for all examples, which readers can reproduce, experiment with, and modify; offers an online instructor's guide that helps professors teach, customize exercises, and select homework problems; and, is accessible to students and professionals without a highly specialized mathematical background.

/pdf/parameter-estimation-and-inverse-problems-370q4164ci.pdf

Parameter estimation and inverse problems

The package REGULARIZATION TOOLS consists of 54 Matlab routines for analysis and solution of discrete ill-posed problems, i.e., systems of linear equations whose coefficient matrix has the properties that its condition number is very large, and its singular values decay gradually to zero. Such problems typically arise in connection with discretization of Fredholm integral equations of the first kind, and similar ill-posed problems. Some form of regularization is always required in order to compute a stabilized solution to discrete ill-posed problems. The purpose of REGULARIZATION TOOLS is to provide the user with easy-to-use routines, based on numerical robust and efficient algorithms, for doing experiments with regularization of discrete ill-posed problems. By means of this package, the user can experiment with different regularization strategies, compare them, and draw conclusions from these experiments that would otherwise require a major programming effert. For discrete ill-posed problems, which are indeed difficult to treat numerically, such an approach is certainly superior to a single black-box routine. This paper describes the underlying theory gives an overview of the package; a complete manual is also available.

/pdf/regularization-tools-a-matlab-package-for-analysis-and-3c2tydm07j.pdf

REGULARIZATION TOOLS: A Matlab package for analysis and solution of discrete ill-posed problems

It has been observed that the rate of convergence of Conjugate Gradients increases when one or more of the extreme Ritz values have sufficiently converged to the corresponding eigenvalues (the “superlinear convergence” of CG). In this paper this will be proved and made quantitative. It will be shown that a very modest degree of convergence of an extreme Ritz value already suffices for an increased rate of convergence to occur.

The rate of convergence of conjugate gradients

In this chapter we will digress on two classes of methods for solving the large sparse matrix systems arising from tomographic problems, viz. ART-like methods and projection methods.1 The former class has been in use for several decades, the latter is more recent.

A. van der Sluis

Papers

The rate of convergence of conjugate gradients

Numerical solution of large, sparse linear algebraic systems arising from tomographic problems

SIRT- and CG-type methods for the iterative solution of sparse linear least-squares problems

The convergence behavior of ritz values in the presence of close eigenvalues

Further results on the convergence behavior of conjugate-gradients and Ritz values