scispace - formally typeset
Search or ask a question

Showing papers on "Rate of convergence published in 1985"


Journal ArticleDOI
TL;DR: In this paper, it is shown that consistency between the tangent operator and the integration algorithm employed in the solution of the incremental problem plays crucial role in preserving the quadratic rate of asymptotic convergence of iterative solution schemes based upon Newton's method.

1,702 citations


Journal ArticleDOI
TL;DR: It is found that the adaptive coefficient μ, which controls the rate of convergence of the algorithm, must be restricted to an interval significantly smaller than the domain commonly stated in the literature.
Abstract: Statistical analysis of the least mean-squares (LMS) adaptive algorithm with uncorrelated Gaussian data is presented. Exact analytical expressions for the steady-state mean-square error (mse) and the performance degradation due to weight vector misadjustment are derived. Necessary and sufficient conditions for the convergence of the algorithm to the optimal (Wiener) solution within a finite variance are derived. It is found that the adaptive coefficient μ, which controls the rate of convergence of the algorithm, must be restricted to an interval significantly smaller than the domain commonly stated in the literature. The outcome of this paper, therefore, places fundamental limitations on the mse performance and rate of convergence of the LMS adaptive scheme.

392 citations


Journal ArticleDOI
TL;DR: Two classes of matrix splittings are presented and applications to the parallel iterative solution of systems of linear equations are given, resulting in algorithms which can be implemented efficiently on parallel computing systems.
Abstract: We present two classes of matrix splittings and give applications to the parallel iterative solution of systems of linear equations. These splittings generalize regular splittings and P-regular splittings, resulting in algorithms which can be implemented efficiently on parallel computing systems. Convergence is established, rate of convergence is discussed, and numerical examples are given.

353 citations


Journal ArticleDOI
TL;DR: In this article, a general theory of rates of convergence for the Rayleigh-Ritz variational method is given for the ground states of atoms and molecules, and the theory shows what functions should be added to the basis set to improve the rate of convergence.
Abstract: A general theory of rates of convergence for the Rayleigh–Ritz variational method is given for the ground states of atoms and molecules. The theory shows what functions should be added to the basis set to improve the rate of convergence, and gives explicit formulas for estimating corrections to variational energies and wave functions. An application of this general theory to a CI calculation on the ground state of the helium atom yields an explicit large L asymptotic formula for the ‘‘L’’‐limit energies E L . The increments are found to obey the formula E L −E L−1 =−3C 1(L+ 1/2 )− 4−4C 2(L+ 1/2 )− 5+O(L − 6), where the constants C 1 and C 2 are given by explicit integrals over the exact wave function evaluated at r 1 2=0. Numerical evaluation of these integrals yields 3C 1≅0.0741 and 4C 2≅0.0309, in excellent agreement with the empirical results 3C 1≅0.0740 and 4C 2≅0.031 found by Carroll, Silverstone, and Metzger.

323 citations


Proceedings ArticleDOI
01 Dec 1985
TL;DR: In this paper, a theoretical analysis of simulated annealing based on a time-inhomogeneous Markov chain is presented and a bound on the departure of the probability distribution of the state at finite time from the optimum is given.
Abstract: Simulated Annealing is a randomized algorithm which has been proposed for finding globally optimum least-cost configurations in large NP-complete problems with cost functions which may have many local minima. A theoretical analysis of Simulated Annealing based on its precise model, a time-inhomogeneous Markov chain, is presented. An annealing schedule is given for which the Markov chain is strongly ergodic and the algorithm converges to a global optimum. The finite-time behavior of Simulated Annealing is also analyzed and a bound obtained on the departure of the probability distribution of the state at finite time from the optimum. This bound gives an estimate of the rate of convergence and insights into the conditions on the annealing schedule which gives optimum performance.

307 citations


Journal ArticleDOI
Basilis Gidas1
TL;DR: In this article, the authors studied the asymptotic behavior of nonstationary Markov chains and proved the convergence of the annealing algorithm in Monte Carlo simulations, and provided a rigorous procedure for choosing the optimal schedule.
Abstract: We study the asymptotic behavior as timet → + ∞ of certain nonstationary Markov chains, and prove the convergence of the annealing algorithm in Monte Carlo simulations. We find that in the limitt → + ∞, a nonstationary Markov chain may exhibit “phase transitions.” Nonstationary Markov chains in general, and the annealing algorithm in particular, lead to biased estimators for the expectation values of the process. We compute the leading terms in the bias and the variance of the sample-means estimator. We find that the annealing algorithm converges if the temperatureT(t) goes to zero no faster thanC/log(t/t0) ast→+∞, with a computable constantC andt0 the initial time. The bias and the variance of the sample-means estimator in the annealing algorithm go to zero likeO(t−1+e) for some 0⩽e<1, with e=0 only in very special circumstances. Our results concerning the convergence of the annealing algorithm, and the rate of convergence to zero of the bias and the variance of the sample-means estimator, provide a rigorous procedure for choosing the optimal “annealing schedule.” This optimal choice reflects the competition between two physical effects: (a) The “adiabatic” effect, whereby if the temperature is loweredtoo abruptly the system may end up not in a ground state but in a nearby metastable state, and (b) the “super-cooling” effect, whereby if the temperature is loweredtoo slowly the system will indeed approach the ground state(s) but may do so extremely slowly.

259 citations


Journal ArticleDOI
TL;DR: In this article, an algorithm is proposed that finds the constrained minimum of the maximum of finitely many ratios, which involves a sequence of linear subproblems if the ratios are linear (convex-concave).
Abstract: An algorithm is suggested that finds the constrained minimum of the maximum of finitely many ratios. The method involves a sequence of linear (convex) subproblems if the ratios are linear (convex-concave). Convergence results as well as rate of convergence results are derived. Special consideration is given to the case of (a) compact feasible regions and (b) linear ratios.

202 citations


Journal ArticleDOI
TL;DR: In this paper, the authors considered a nonparametric regression model where the zero mean errors are uncorrelated with common variance a2 and the response function f is assumed only to have a bounded square integrable qth derivative.
Abstract: Linear estimation is considered in nonparametric regression models of the form Yi = f (xi) + ei, xi E= (a, b), where the zero mean errors are uncorrelated with common variance a2 and the response function f is assumed only to have a bounded square integrable qth derivative. The linear estimator which minimizes the maximum mean squared error summed over the observation points is derived, and the exact minimax rate of convergence is obtained. For practical problems where bounds on 11 f2q) 11 and a2 may be unknown, generalized cross-validation is shown to give an adaptive estimator which achieves the minimax optimal rate under the additional assumption of normality. 1. The model. Consider the nonparametric regression model

185 citations


Journal ArticleDOI
TL;DR: In this article, the Alternating Group Explicit (AGE) method is applied to derive the solution of a 2-point boundary value problem and the analysis clearly shows the method to be analogous to the A.I.D. method.
Abstract: In this paper, the Alternating Group Explicit (AGE) method is developed and applied to derive the solution of a 2 point boundary value problem. The analysis clearly shows the method to be analogous to the A.D.I. method. The extension of the method to ultidimensional problems and techniques for improving the convergence rate and attaining higher order accuracy are also given.

171 citations


Journal ArticleDOI
TL;DR: A primal–dual decomposition method is presented to solve the separable convex programming problem, equivalent to the proximal point algorithm applied to a certain maximal monotone multifunction.
Abstract: A primal–dual decomposition method is presented to solve the separable convex programming problem Convergence to a solution and Lagrange multiplier vector occurs from an arbitrary starting point The method is equivalent to the proximal point algorithm applied to a certain maximal monotone multifunction In the nonseparable case, it specializes to a known method, the proximal method of multipliers Conditions are provided which guarantee linear convergence

163 citations


Journal ArticleDOI
TL;DR: This paper proves the convergence of the multilevel iterative method for solving linear equations that arise from elliptic partial differential equations in terms of the generalized condition number $\kappa $ of the matrix A and the smoothing matrix B.
Abstract: In this paper, we prove the convergence of the multilevel iterative method for solving linear equations that arise from elliptic partial differential equations. Our theory is presented entirely in terms of the generalized condition number $\kappa $ of the matrix A and the smoothing matrix B. This leads to a completely algebraic analysis of the method as an iterative technique for solving linear equations; the properties of the elliptic equation and discretization procedure enter only when we seek to estimate $\kappa $, just as in the case of most standard iterative methods. Here we consider the fundamental two-level iteration, and the V and W cycles of the j-level iteration ($j > 2$). We prove that the V and W cycles converge even when only one smoothing iteration is used. We present several examples of the computation of $\kappa $ using both Fourier analysis and standard finite element techniques. We compare the predictions of our theorems with the actual rate of convergence. Our analysis also shows that...

Proceedings ArticleDOI
01 Jan 1985
TL;DR: The generalized minimal residual algorithm is modified to handle nonlinear equations characteristic of computational fluid dynamics, and a formulation is developed that allows GMRES to be preconditioned by the solution procedures already built into existing computer codes.
Abstract: The generalized minimal residual algorithm (GMRES) is a conjugate-gradient like method that applies directly to nonsymmetric linear systems of equations. In this paper, GMRES is modified to handle nonlinear equations characteristic of computational fluid dynamics. Attention is devoted to the concept of preconditioning and the role it plays in assuring rapid convergence. A formulation is developed that allows GMRES to be preconditioned by the solution procedures already built into existing computer codes. Examples are provided that demonstrate the ability of GMRES to greatly improve the robustness and rate of convergence of current state-of-the-art fluid dynamics codes. Theoretical aspects of GMRES are presented that explain why it works. Finally, the advantage GMRES enjoys over related methods such as conjugate gradients are discussed.

Journal ArticleDOI
TL;DR: The PARTAN technique and heuristic variations of the Frank-Wolfe algorithm are described which serve to significantly improve the convergence rate with no significant increase in memory requirements.
Abstract: We discuss methods for speeding up convergence of the Frank-Wolfe algorithm for solving nonlinear convex programs. Models involving hydraulic networks, road networks and factory-warehouse networks are described. The PARTAN technique and heuristic variations of the Frank-Wolfe algorithm are described which serve to significantly improve the convergence rate with no significant increase in memory requirements. Computational results for large-scale models are reported.

Journal ArticleDOI
TL;DR: It is shown that the method of Takahashi corresponds to a modified block Gauss-Seidel step and aggregation, whereas that of Vantilborgh corresponds to an modified block Jacobistep and aggregation.
Abstract: Iterative aggregation/disaggregation methods provide an efficient approach for computing the stationary probability vector of nearly uncoupled (also known as nearly completely decomposable) Markov chains. Three such methods that have appeared in the literature recently are considered and their similarities and differences are outlined. Specifically, it is shown that the method of Takahashi corresponds to a modified block Gauss-Seidel step and aggregation, whereas that of Vantilborgh corresponds to a modified block Jacobi step and aggregation. The third method, that of Koury et al., is equivalent to a standard block Gauss-Seidel step and iteration. For each of these methods, a lemma is established, which shows that the unique fixed point of the iterative scheme is the left eigenvector corresponding to the dominant unit eigenvalue of the stochastic transition probability matrix. In addition, conditions are established for the convergence of the first two of these methods; convergence conditions for the third having already been established by Stewart et al. All three methods are shown to have the same asymptotic rate of convergence.

Journal ArticleDOI
TL;DR: In this paper, Richardson extrapolation, overrelaxation, bordering by a singularity condition and appending the usual linear model by a quadratic term are described in detail.
Abstract: The paper begins with a review of the convergence theory for Newton’s method near simple and other regular singularities, followed by a brief discussion of the inherent difficulty of singular problems and the effect of rounding errors on the achievable solution accuracy. Then several ways to accelerate the slow convergence of Newton’s method are described in some detail. These are: Richardson extrapolation, overrelaxation, bordering by a singularity condition and appending the usual linear model by a quadratic term. The last two approaches require limited second derivative information which can be obtained analytically, by differencing or through secant updating. Numerical comparisons reported in the final sections lead to the conclusion that the collection of the additional information pays off in terms of speed and accuracy.

Journal ArticleDOI
TL;DR: In this paper, a model-following control scheme for a class of nonlinear plants is proposed, which guarantees that tracking error remains bounded and tends to a neighborhood of the origin with a rate not inferior to an exponential one; furthermore, the designer can arbitrarily prescribe the rate of convergence and the size of the set of ultimate boundedness.
Abstract: We propose a new model-following control scheme for a class of nonlinear plants. The feedback control signal is a continuous function of all its arguments. It is shown that this scheme guarantees that tracking error remains bounded and tends to a neighborhood of the origin with a rate not inferior to an exponential one; furthermore, it allows the designer to arbitrarily prescribe the rate of convergence and the size of the set of ultimate boundedness.

Journal ArticleDOI
TL;DR: A globally convergent algorithm is presented for the solution of a wide class of semi-infinite programming problems, and usually has a second order convergence rate.
Abstract: A globally convergent algorithm is presented for the solution of a wide class of semi-infinite programming problems. The method is based on the solution of a sequence of equality constrained quadratic programming problems, and usually has a second order convergence rate. Numerical results illustrating the method are given.

Journal ArticleDOI
TL;DR: In this paper, the authors present an existence-comparison theorem and an iterative method for a nonlinear finite difference system which corresponds to a class of semilinear parabolic and elliptic boundary value problems.
Abstract: This paper presents an existence-comparison theorem and an iterative method for a nonlinear finite difference system which corresponds to a class of semilinear parabolic and elliptic boundary-value problems. The basic idea of the iterative method for the computation of numerical solutions is the monotone approach which involves the notion of upper and lower solutions and the construction of monotone sequences from a suitable linear discrete system. Using upper and lower solutions as two distinct initial iterations, two monotone sequences from a suitable linear system are constructed. It is shown that these two sequences converge monotonically from above and below, respectively, to a unique solution of the nonlinear discrete equations. This formulation leads to a well-posed problem for the nonlinear discrete system. Applications are given to several models arising from physical, chemical and biological systems. Numerical results are given to some of these models including a discussion on the rate of convergence of the monotone sequences.

Journal ArticleDOI
TL;DR: In this paper, the authors analyzed and further developed the adiabatic invariance method for computing semiclassical eigenvalues, which was recently introduced by Solov'ev.
Abstract: In this paper we analyze and further develop the adiabatic invariance method for computing semiclassical eigenvalues. This method, which was recently introduced by Solov’ev, is basically an application of the Ehrenfest adiabatic hypothesis. The eigenvalues are determined from a classical calculation of the energy as the time dependent Hamiltonian H(t)=H0+s(t)H1 is switched adiabatically from the separable reference Hamiltonian H0 to the system Hamiltonian H0+H1. A systematic study is carried out to determine the best form for the switching function, s(t), to maximize the rate of convergence of the energy to its adiabatic limit. Five switching functions, including the linear function used by Solov’ev, are defined and tested on three different systems. The linear function is found to have a very slow convergence rate compared to the others. The classical energy is shown to be a periodic function of the angle coordinates of H0. The coefficients of the Fourier series representation of this function are then s...

Journal ArticleDOI
TL;DR: In this paper, a simple one-dimensional example is examined in some detail, where the validity and rate of convergence of this scheme depends on the spectrum of an associated non-selfadjoint Hamiltonian which is found numerically.
Abstract: As part of a program to evaluate expectations in complex distributions by longterm averages of solutions to Langevin equations with complex dirft, a simple one-dimensional example is examined in some detail. The validity and rate of convergence of this scheme depends on the spectrum of an associated non-selfadjoint Hamiltonian which is found numerically. In the regime where the stochastic evaluation should be accurate numerical solution of the Langevin equation shows this to be the case.

Journal ArticleDOI
TL;DR: In this paper, the mathematical requirements that the expansion functions must satisfy in the method of moments (MM) solution of an operator equation are discussed and a simple differential equation is solved to demonstrate these requirements.
Abstract: One of the objectives of this paper is to discuss the mathematical requirements that the expansion functions must satisfy in the method of moments (MM) solution of an operator equation. A simple differential equation is solved to demonstrate these requirements. The second objective is to study the numerical stability of point matching method, Galerkin's method, and the method of least squares. Pocklington's integral equation is considered and numerical results are presented to illustrate the effect of various choices of weighting functions on the rate of convergence. Finally, it is shown that certain choices of expansion and weighting functions yield numerically acceptable results even though they are not admissible from a strictly mathematical point of view. The reason for this paradox is outlined.

Proceedings ArticleDOI
01 Sep 1985
TL;DR: This work analyzes a nonstationary finite state Markov chain whose state space Ω is the domain of the cost function to be minimized, and considers an arbitrary partition optimization {I, J} of Ω of this chain focusing on those issues most important for optimization.
Abstract: Simulated annealing is a popular Monte Carlo algorithm for combinatorial optimization. The annealing algorithm simulates a nonstationary finite state Markov chain whose state space ? is the domain of the cost function to be minimized. We analyze this chain focusing on those issues most important for optimization. In all of our results we consider an arbitrary partition optimization {I, J} of ? important special cases are when I is the set of minimum cost states or a set of all states with sufficiently small cost. We give a lower bound on the probability that the chain visits I at some time ? k, for k = 1,2, .... This bound may be useful even when the algorithm does not converge. We give conditions under which the chain converges to I in probability and obtain an estimate of the rate of convergence as well. We also give conditions under which the chain visits I infinitely often, visits I almost always, or does not converge to I, with probability 1.

Journal ArticleDOI
TL;DR: In this article, the implementation of a 9-node Lagrange element with uniform reduced quadrature (2 × 2) and spurious mode control for plates and shells is described.

Book ChapterDOI
01 Jan 1985
TL;DR: In this article, the authors discuss the convergence properties of various methods for the approximate solution of ill-posed equations and show that these methods in general converge arbitrarily slow, i.e. there may exist some y for which the approximations have a good convergence rate, but in general for each order of convergence there are right hand sides y with a worse convergence rate.
Abstract: The aim of this note is to discuss the convergence properties of various methods for the approximate solution of ill-posed equations. If an operator T between Banach spaces has a non-closed range, then there exists no linear uniformly convergent approximation, method, but at most pointwise approximation methods. for the approximate solution of an equation (1) Tx = y. These methods in general converge arbitrarily slow, i.e. there may exist some y for which the approximations have a good convergence rate, but in general for each order of convergence there are right hand sides y with a worse convergence rate. This phenomenon is not restricted to ill-posed equations. Recently I have shown [121 that many common approximation schemes for integral equations of the second kind show arbitrarily slow convergence, too.

Journal ArticleDOI
TL;DR: For circulant systems, the Landweber iteration with a special step size is shown to be equivalent to a variation of the projection method, and it is conjectured that the projection technique has a better convergence rate.
Abstract: Iterative signal reconstruction and restoration have made considerable use of variations of the Landweber iteration and, more recently, of projection onto convex sets. For circulant systems, the Landweber iteration with a special step size is shown to be equivalent to a variation of the projection method. Based on experimental evidence, it is conjectured that the projection technique has a better convergence rate. Additionally, the projection method makes it easier to combine other a priori information with the basic iteration.

Journal ArticleDOI
TL;DR: In this article, the approximation in the L?-norm of variational inequalities with non-linear operators and somewhat irregular obstacles is studied and it is shown that the order of convergence will be the same as that of the equation associated with the nonlinear operator if the discrete maximum principle is verified.
Abstract: We are interested in the approximation in theL ?-norm of variational inequalities with non-linear operators and somewhat irregular obstacles We show that the order of convergence will be the same as that of the equation associated with the non-linear operator if the discrete maximum principle is verified

Journal ArticleDOI
TL;DR: In this paper, the convergence of the Laplacian with Dirichlet boundary conditions on the surfaces of some identical (small) neighborhoods of randomly distributed points was shown to be δ − C(x), whereC(x) is the limit density of electrostatic capacity of the "obstacles".
Abstract: We consider the Laplacianδ m in ℝ3 (or in a bounded region of ℝ3) with Dirichlet boundary conditions on the surfaces of some identical (small) neighborhoods ofm randomly distributed points, in the limit whenm goes to infinity and their linear size decreases as 1/m. We give here a stronger form of the result showing the convergence of the above operator toδ − C(x), whereC(x) is the limit density of electrostatic capacity of the “obstacles.” In particular results on the rate of convergence and on the fluctuations ofδ m around the limit operator are given.

Journal ArticleDOI
TL;DR: Local convergence at a second order rate is established for a generalized Newton method when the minimizer satisfies nondegeneracy, strict complementarity and second order sufficiency conditions and necessary and sufficient conditions for a superlinear rate of convergence for curvature approximating methods are established.
Abstract: This paper considers local convergence and rate of convergence results for algorithms for minimizing the composite functionF(x)=f(x)+h(c(x)) wheref andc are smooth buth(c) may be nonsmooth. Local convergence at a second order rate is established for the generalized Gauss—Newton method whenh is convex and globally Lipschitz and the minimizer is strongly unique. Local convergence at a second order rate is established for a generalized Newton method when the minimizer satisfies nondegeneracy, strict complementarity and second order sufficiency conditions. Assuming the minimizer satisfies these conditions, necessary and sufficient conditions for a superlinear rate of convergence for curvature approximating methods are established. Necessary and sufficient conditions for a two-step superlinear rate of convergence are also established when only reduced curvature information is available. All these local convergence and rate of convergence results are directly applicable to nonlinearing programming problems.

Journal ArticleDOI
TL;DR: An exact performance analysis of the BLMS algorithm is presented and an optimal choice of this gain is presented resulting in the fastest convergence rate of the algorithm.
Abstract: An exact performance analysis of the BLMS algorithm is presented in this paper. Based on equations describing second statistics behavior, bounds are derived for the adaptation gain guaranteeing convergence of the algorithm. Within these bounds an optimal choice of this gain is presented resulting in the fastest convergence rate of the algorithm.

Journal ArticleDOI
TL;DR: A quadratically convergent algorithm for minimizing a nonlinear function subject to nonlinear equality constraints is derived and an extension of Kantorovich's theorem shows that the algorithm maintains quadratic convergence even if the basis of the tangent space changes abruptly from iteration to iteration.
Abstract: We derive a quadratically convergent algorithm for minimizing a nonlinear function subject to nonlinear equality constraints. We show, following Kaufman [4], how to compute efficiently the derivative of a basis of the subspace tangent to the feasible surface. The derivation minimizes the use of Lagrange multipliers, producing multiplier estimates as a by-product of other calculations. An extension of Kantorovich's theorem shows that the algorithm maintains quadratic convergence even if the basis of the tangent space changes abruptly from iteration to iteration. The algorithm and its quadratic convergence are known but the drivation is new, simple, and suggests several new modifications of the algorithm.