Showing papers on "Rate of convergence published in 1971"
TL;DR: In this article, the authors give two derivative-free computational algorithms for nonlinear least squares approximation, which are finite difference analogues of the Levenberg-Marquardt and Gauss methods.
Abstract: In this paper we give two derivative-free computational algorithms for nonlinear least squares approximation. The algorithms are finite difference analogues of the Levenberg-Marquardt and Gauss methods. Local convergence theorems for the algorithms are proven. In the special case when the residuals are zero at the minimum, we show that certain computationally simple choices of the parameters lead to quadratic convergence. Numerical examples are included.
381 citations
TL;DR: It is proved that successful convergence is obtained provided that the objective function has a strictly positive definite second derivative matrix for all values of its variables.
Abstract: The variable metric algorithm is a frequently used method for calculating the least value of a function of several variables. However it has been proved only that the method is successful if the objective function is quadratic, although in practice it treats many types of objective functions successfully. This paper extends the theory, for it proves that successful convergence is obtained provided that the objective function has a strictly positive definite second derivative matrix for all values of its variables. Moreover it is shown that the rate of convergence is super-linear.
183 citations
01 Apr 1971
TL;DR: In this article, a novel approach to the weight optimization of indeterminate structures under multiple loading conditions with strength and displacement constraints has been developed and is presented by using this method significant improvements in computational time have been achieved.
Abstract: : The report considers the state of the art in methods of structural optimization. Mathematical programming based methods, while extremely successful with problems of moderate size tend to become prohibitively costly when applied to large scale structures. A novel approach to the weight optimization of indeterminate structures under multiple loading conditions with strength and displacement constraints has been developed and is presented herein. Using this method significant improvements in computational time have been achieved over direct numerical search methods. In some cases the numbers of iterations required to determine the least weight have been reduced by factors of over 20. The rate of convergence is independent of problem size permitting application to large scale structures. Examples of application of the new approach to a number of problems are included. (Author)
158 citations
TL;DR: In this paper, the authors present three numerical schemes for solving the Falkner-Skan equation with positive or negative wall shear, using Newton's method with the aid of variational equations and yields quadratic convergence.
144 citations
TL;DR: In this paper, a technique for solving the finite difference biharmonic equation as a coupled pair of harmonic difference equations is proposed, which is a general block SOR method with convergence rate O(h^(h 1 / 2 )$ on a square, where h is mesh size.
Abstract: A technique is proposed for solving the finite difference biharmonic equation as a coupled pair of harmonic difference equations. Essentially, the method is a general block SOR method with convergence rate $O(h^{{1 / 2}} )$ on a square, where h is mesh size.
88 citations
TL;DR: A family of iterative methods of order of convergence is given for computing the closest unitary matrix, measured in Euclidean norm, to a given rectangular matrix A, to be the unitary factor in the polar decomposition of A.
Abstract: The closest unitary matrix, measured in Euclidean norm, to a given rectangular matrix A is known to be the unitary factor in the polar decomposition of A The paper gives a family of iterative methods of order of convergence $p + 1,\, p = 1,2,3, \cdots $, for computing this matrix The methods are especially efficient when the columns of A are not too far from being orthonormal The choice of order of convergence to minimize the amount of computation is discussed Global convergence properties for the methods of order $ \leqq 4$ are studied and sufficient conditions for convergence in terms of $\| {I - A^H A} \|$ are given
87 citations
TL;DR: In this paper, the rate of convergence of the mode shapes and frequencies by the finite element method using consistent and lumped mass formulations has been established, and it has been found that for a system of differential equations of second order such as the equations of equilibrium in terms of displacement in the theory of elasticity, membrane etc, a proper lump-mass formulation will not suffer any loss of rate of convergence utilizing simple elements However, in the case of higher order differential equations or when the use of more complicated elements is required or desired, a consistent mass formulation often will provide a
73 citations
TL;DR: In this paper, a modified Muller's method for the convergence of column temperature profile is proposed, which is simplified by using matrix notation, which also has the advantage that any interstage flow pattern is allowed.
Abstract: Iterative method for the determination of stage temperatures, stage reaction rates and interstage flow rates in the problem of multicomponent distillation accompanied by a simultaneous chemical reaction is discussed, and the use of a modified Muller''s method for the convergence of the column temperature profile is proposed. Derivation of the equation is simplified by using matrix notation, which also has the advantage that any interstage flow pattern is allowed. For the solution of the linearized material balance equation, the tridiagonal matrix algorithm is employed. Some problems are discussed to demonstrate the feasibility of the calculations and the fact that quadratic convergence is obtained.
66 citations
TL;DR: It is shown how various timing recovery schemes are reasonable approximations of the maximum likelihood strategy for estimating an unknown timing parameter in additive white gaussian noise.
Abstract: It is shown how various timing recovery schemes are reasonable approximations of the maximum likelihood strategy for estimating an unknown timing parameter in additive white gaussian noise. These schemes derive an appropriate error signal from the received data which is then used in a closed-loop system to change the timing phase of a voltage-controlled oscillator. The technique of stochastic approximation is utilized to cast the synchronization problem as a regression problem and to develop an estimation algorithm which rapidly converges to the desired sampling time. This estimate does not depend upon knowledge of the system impulse response, is independent of the noise distribution, is computed in real time, and can be synthesized as a feedback structure. As is characteristic of stochastic approximation algorithms, the current estimate is the sum of the previous estimate and a time-varying weighted approximation of the estimation error. The error is approximated by sampling the derivative of the received signal, and the mean-square error of the resulting estimate is minimized by optimizing the choice of the gain sequence. If the receiver is provided with an ideal reference (or if the data error rate is small) it is shown that both the bias and the jitter (mean-square error) of the estimator approach zero as the number of iterations becomes large. The rate of convergence of the algorithm is derived and examples are provided which indicate that reliable synchronization information can be quickly acquired.
55 citations
TL;DR: In this paper, a new minimization procedure is presented for solving the general eigenvalue problem that arises from the dynamic analysis of a structure by the finite element method: it consists in seeking the stationary points of the Rayleigh quotient and thus does not need the physical assembling of the structural matrices K and M.
48 citations
TL;DR: In this article, a sequence of decision problems is considered where for each problem the observation has discrete probability function of the form p(x) = h(x), beta (lambda) lambda to the power x, x = 0,1,2,..., and where lambda is selected independently for each decision according to an unknown prior distribution G(lambda).
Abstract: : A sequence of decision problems is considered where for each problem the observation has discrete probability function of the form p(x) = h(x) beta (lambda) lambda to the power x, x = 0,1,2,..., and where lambda is selected independently for each problem according to an unknown prior distribution G(lambda). It is supposed that for each problem one of two possible actions (e.g., 'accept' or 'reject') must be selected. Under various assumptions about h(x) and G(lambda) the rate at which the risk of the nth problem approaches the smallest possible risk is determined for standard empirical Bayes procedures. It is shown that for most practical situations, the rate of convergence to 'optimality' will be at least as fast as L(n)/n where L(n) is a slowly varying function (e.g., log n). The rate cannot be faster than 1/n and this exact rate is achieved in some cases. Arbitrarily slow rates will occur in certain pathological situations. (Author)
TL;DR: It is shown that it is possible to obtain the highest rate of convergence by this refinement and the character of the proper refinement of the elements around the boundary is studied.
Abstract: The character of the proper refinement of the elements (mesh) around the boundary is studied. It is shown that it is possible to obtain the highest (optimal) rate of convergence by this refinement.
TL;DR: In this article, a fully implicit scheme was developed along with a functional iteration method for solving the system of nonlinear difference equations, which is mathematically the most preferable of all functional iteration methods because of its quadratic convergence.
Abstract: In the solution of nonlinear parabolic partial differential equations, such as the Richards equation, classical implicit schemes often oscillate and fail to converge. A fully implicit scheme has been developed along with a functional iteration method for solving the system of nonlinear difference equations. Newton's iteration technique is mathematically the most preferable of all functional iteration methods because of its quadratic convergence. The Richards equation, Newton-linearized with respect to relative permeability and saturation as functions of capillary pressure, is particularly aided by this new approach for problems in which saturations vary rapidly with time (infiltration fronts, cone of depression near a well bore, and so forth). Although the computing time is almost twice as long for a time step with Newton's iteration scheme, the smaller time truncation than that of classical implicit schemes and the stability in cases in which classical schemes are unstable permit the use of much larger time steps. To demonstrate the method, heterogeneous (layered) soil systems are used to simulate sharp infiltration fronts caused by ponding at the soil surface.
TL;DR: The diakoptics technique has been examined as an alternative to the widely used Hardy Cross method for the solution of pipe network problems and was found to be faster.
TL;DR: Bounds on the variance, valid for large signal-to-noise ratios, indicate that the new algorithm not only converges faster, but also has a smaller variance asymptotically than the present algorithm for moderate intersymbol interference and the same variance asyspymbol interference.
Abstract: Currently used adaptive equalizers for the minimization of mean-square error in digital communications commonly employ a fixed-step-size gradient-search procedure. The algorithm to be described here employs variable step sizes designed to minimize the error after a specified number of iterations. The resultant convergence rate provides considerable improvement over the fixed-step-size approach. Bounds on the variance, valid for large signal-to-noise ratios, indicate that the new algorithm not only converges faster, but also has a smaller variance asymptotically than the present algorithm for moderate intersymbol interference and the same variance asymptotically for large intersymbol interference. Computer simulation studies have verified these results.
TL;DR: A great deal of insight is gained by working out the actual asymptotic convergence rates of these modified constructs, as well as the rates of convergence of the optimal relative-cost function.
Abstract: The modified method of successive approximations for solving Markovian decision problems as formulated by White, Schweitzer, MacQueen, and Odoni, concentrates attention on cost differences either between successive stages in the same state, or relative to a base state in the same stage, rather than on the total cost function. The former bound the (discounted) gain of the optimal policy, while the latter relative-cost function determines the policy to be chosen at each stage. While these authors have demonstrated that these modified constructs converge to the gain and the optimal relative-cost function under rather general circumstances (undiscounted, single-chain, aperiodic processes), little is known about the rates of convergence. [Note that convergence of the relative-cost function guarantees optimality of a currently repeating policy, as noted by Howard.) A great deal of insight into this mathematically difficult question may be gained by working out the actual asymptotic convergence rates of these co...
TL;DR: Both classes of methods are analyzed as to parameter selection requirements, convergence to first and second-order Kuhn-Tucker Points, rate of convergence, matrix conditioning problems and computations required.
Abstract: The relative merits of using sequential unconstrained methods for solving: minimizef(x) subject togi(x) ź 0, i = 1, ź, m, hj(x) = 0, j = 1, ź, p versus methods which handle the constraints directly are explored. Nonlinearly constrained problems are emphasized. Both classes of methods are analyzed as to parameter selection requirements, convergence to first and second-order Kuhn-Tucker Points, rate of convergence, matrix conditioning problems and computations required.
TL;DR: The penalty-function approach is an attractive method for solving constrained nonlinear programming problems, since it brings into play all of the well-developed unconstrained optimization techniques as discussed by the authors.
Abstract: The penalty-function approach is an attractive method for solving constrained nonlinear programming problems, since it brings into play all of the well-developed unconstrained optimization techniques, If, however, the classical steepest-descent method is applied to the standard penalty-function objective, the rate of convergence approaches zero as the penalty coefficient is increased to yield a close approximation to the true solution.
TL;DR: In this paper, it is shown how to match the accuracy of the discovery of the minimum in the problem with a penalty function with the value of the coefficient of the penalty function.
Abstract: THE closeness of the solution of the problem with a penalty to the solution of the original problem is estimated at a conditional minimum. It is shown how to match the accuracy of the discovery of the minimum in the problem with a penalty function with the value of the coefficient of the penalty function. THE closeness of the solution of the problem with a penalty to the solution of the original problem is estimated at a conditional minimum. It is shown how to match the accuracy of the discovery of the minimum in the problem with a penalty function with the value of the coefficient of the penalty function. The idea of the penalty function method-the reduction of problems at a conditional extremum to problems without constraints by the introduction of a penalty on the infringement of constraints - has been known for a long time and has been discussed by many authors. At the present time there are numerous papers devoted to various modifications of the method, their applications to particular problems etc, (references specially devoted to this method can be found in the book [l], or in [2], section 12). However, the theoretical study of the method is far from complete, since the investigation is usually confined to the proof of its convergence, without estimating the rate of convergence. One of the few exceptions is [3] in which the closeness of the solution of the problem with a penalty to the solution of the original problem is estimated. However, in [3l only the convex case is considered (that is, these results are not applicable to non-linear constraints of the equality type), and also the estimate of closeness was there obtained with respect to a functional and to the constraints, but not with respect to the actual variables.Another important question left open is the following.In the practical realization of the penalty function method the discovery of the exact solution of each problem at an unconditional minimum is impossible. How must the accuracy of the solution of this auxiliary problem be chosen, in order that, on the one hand, the computing time is not increased too greatly, and on the other, the convergence of the method is not destroyed? Finally, it is known that in the penalty function method we obtain approximations for the dual variables (Lagrange multipliers) simultaneously. But the accuracy of these approximations has not been investigated. In the present paper an answer to these questions is given for the case of constraints given by equations and the simplest form of penalty.
TL;DR: The rate of convergence and the computational effort in the pth order method are studied, and the optimum p is given in terms of the numbers of columns of A and B.
Abstract: Apth order iterative method for computing $A^\dag B$ is studied, where $p \geqq 2$, A and B are arbitrary complex matrices with equal number of rows and $A^\dag $ is the Moore–Penrose generalized inverse of A. The rate of convergence and the computational effort in the pth order method are studied, and the optimum p is given in terms of the numbers of columns of A and B.
TL;DR: The results indicate that, on the average, the new algorithms lead to faster tracking of changes in the channel characteristics and thereby result in a smaller error rate.
Abstract: This paper is concerned with the design of second-order algorithms for an equalizer in a training or a tracking mode. The algorithms govern the iterative adjustment of the equalizer parameters for the minimization of the mean-squared error. On the basis of estimated bounds for the eigenvalues of the signal plus noise correlation matrix, an optimal second-order algorithm is derived. The resultant convergence is considerably faster than the commonly used first-order fixed-size gradient-search procedure. The variance of the optimal algorithm is shown to have a slightly larger bound than the present first-order fixed-step algorithm. However, a computer simulation for an input signal-to-noise ratio of 30 dB shows that for large intersymbol interference the improvement in the convergence of the mean more than compensates for the small increase in variance. For moderate intersymbol interference the simulation shows no variance increase. Suboptimum second-order algorithms with smaller improvement in the convergence rate and smaller increase in the variance bound are also considered. The results indicate that, on the average, the new algorithms lead to faster tracking of changes in the channel characteristics and thereby result in a smaller error rate.
01 Oct 1971
TL;DR: It has been proved by two different methods that the algorithm converges to the sought value in the mean-square sense and with probability one.
Abstract: A new algorithm for stochastic approximation has been proposed, along with the assumptions and conditions necessary for convergence. It has been proved by two different methods that the algorithm converges to the sought value in the mean-square sense and with probability one. The rate of convergence of the new algorithm is shown to be better than two existing algorithms under certain conditions. Results of simulation have been given, making a realistic comparison between the three algorithms.
TL;DR: In this paper, the alternating direction method of Peaceman and Rachford is considered for elliptic difference schemes of second order and with two independent variables, and a connection between that result and theorems on optimal scaling of band matrices is established.
Abstract: The alternating direction method of Peaceman and Rachford is considered for elliptic difference schemes of second order and with two independent variables. An earlier result of the author's on the rapid convergence of multi-parameter noncommutative prob- lems is described and a connection is established between that result and theorems on optimal scaling of band matrices. Simple algorithms to decrease the condition number and increase the rate of convergence are discussed. 1. Introduction. In this paper we shall consider the alternatinlg direction implicit (ADI) method of Peaceman and Rachford (12) when applied to difference approxi- mations to elliptic problems with two independent variables. It is known that this method is often quite powerful, especially when different acceleration parameters are used in the different iteration steps. Usually, these parameters are chosen in a cyclic way. We shall assume that this is the case and denote the cycle length by m. It has been proved that the method always converges when m = 1, but for the potentially much more powerful multi-parameter case the theory is still not satis- factory. Indeed, there seems to be little hope that there will ever be a very general convergence theory because of the fact that divergence has been observed in numer- ical experiments. Under certain additional restrictions on the problem, we can theoretically ex- plain the full power of the method. Thus, there exists a very satisfactory theory in the case when the two matrices, corresponding to the different independent variables, commute. Cf. Varga (15) or Wachspress (16). The commutativity condition is how- ever very limiting because, as was shown by Birkhoff and Varga (1), it imposes severe restrictions on the coefficients as well as on the region. The region thus has to be rectangular. In fact all problems giving rise to commutative problems can be handled by separation-of-variables techniques. It is of interest to note that for sepa- rable problems there now exist faster methods than the ADI or SOR methods. Cf. Hockney (10) and Buzbee, Golub and Nielson (2) for methods which are in fact very efficient computer implementations of the separation-of-variables idea. We shall now make a short survey of results for the noncommutative case. (Cf. Wachspress (16) for more details.) One of the more interesting results follows from an observation by Guilinger (8). It is thus possible to prove the convergence of
TL;DR: The concept of most favorable bias function is introduced and the calculus of variations is used to derive the lower bound on the average mean-square error, which serves as a standard to judge the merits of the stochastic-estimation algorithm.
Abstract: This paper is concerned with stochastic-approximation algorithms for estimating signal parameters. Emphasis will be on the performance of the algorithm for a finite number of observations as opposed to the asymptotic convergence rate. We use as an upper bound a result due to Dvoretzky. A lower bound on the average mean-square error is derived. This new bound is based on the Cramer-Rao inequality. The conventional Cramer-Rao bound is not directly applicable, because it requires the knowledge of the bias function, which is difficult to find in a recursive estimation scheme. To avoid this difficulty, we introduce the concept of most favorable bias function and use the calculus of variations to derive the lower bound. The new bound also serves as a standard to judge the merits of the stochastic-estimation algorithm, since under some general conditions no estimate can yield smaller error. It is shown that under some conditions the two bounds are nearly equal, and hence the algorithm is near optimal. The asymptotic efficiency of the algorithm is compared with Sakrison's result. A stochastic-estimation algorithm is derived for estimating Doppler frequency, and performance curves in terms of the error bounds are presented.
TL;DR: This study deals with the design and analysis of an adaptive array processor in which the individual filters consist of tapped delay lines and adjustable gains and shows that the adaptive system eventually acts to eliminate the effect of the directional component.
Abstract: This study deals with the design and analysis of an adaptive array processor in which the individual filters consist of tapped delay lines and adjustable gains. The gains are adjusted automatically by an iterative procedure which is a modified form of the stochastic approximation method of Robbins and Monro. The criterion for optimality is that the mean‐square error between filter output and an ideal signal be a minimum. However, it is shown that the ideal signal need not be available to the processor; it suffices that the signal autocorrelation function and spatial direction be available. General expressions are obtained for convergence of the iterative algorithm, and an explicit expression for an upper bound on the rate of convergence is derived. System performance is analyzed by considering a noise field consisting of a spatially isotropic component and a single directional component. It is shown that the adaptive system eventually acts to eliminate the effect of the directional component. The theoretical behavior of the system has been verified experimentally using actual sonar data and a six‐element array.
TL;DR: SOR iterations for linear systems arising from finite element approximations to $2m$th order elliptic problems are considered and it is shown that, in general, the asymptotic rate of convergence is O(h 2m), but if the relaxation parameters are suitably chosen, this can be increased to O( h^m).
Abstract: SOR iterations for linear systems arising from finite element approximations to $2m$th order elliptic problems are considered It is shown that, in general, the asymptotic rate of convergence is $O(h^{2m} )$, where h denotes the mesh length However, if the relaxation parameters are suitably chosen, this can be increased to $O(h^m )$
TL;DR: In this article, an extension of Ostrowski's point of attraction theorem to multistep iterative procedures for finding a zero of a non-near function defined on $R^n $ was given.
Abstract: In this paper we shall give an extension of Ostrowski's point of attraction theorem to multistep iterative procedures for finding a zero of a nonljnear function defined on $R^n $. We shall also obtain a rate of convergence statement in terms of the spectral radius of certain matrices. These results will then be applied to a class of procedures obtained by composing one-dimensional iterative methods with the Jacobi and successive-overrelaxation processes. The rate of convergence of these procedures will be shown to be independent of the rate of convergence of the one-dimensional methods.
TL;DR: In this article, the authors consider the problem of estimating the convergence rate of a variational solution to an inhomogeneous equation and obtain a number of a priori estimates of the asymptotic convergence rate which are easy to compute, and which are likely to be realistic in practice.
Abstract: We consider the problem of estimating the convergence rate of a variational solution to an inhomogeneous equation. This problem is not soluble in general without imposing conditions on both the class of expansion functions and the class of problems considered; we introduce the concept of "asymptotically diagonal systems," which is particularly appropriate for classical variational expansions as applied to elliptic partial differential equations. For such systems, we obtain a number of a priori estimates of the asymptotic convergence rate which are easy to compute, and which are likely to be realistic in practice. In the simplest cases these estimates reduce the problem of variational convergence to the simpler problem of Fourier series convergence, which is considered in a companion paper. We also produce estimates for the convergence rate of the individual expansion coefficients a. . thus categorising the convergence completely.
TL;DR: In this paper, the classical convergence theory for numerical solutions to initial and boundary value problems with continuous data (the right-hand side) was extended to problems with Riemann integrable data.
Abstract: This paper extends the classical convergence theory for numerical solutions to initial and boundary value problems with continuous data (the right-hand side) to problems with Riemann integrable data. Order of convergence results are also obtained.
01 Jan 1971
TL;DR: In this article, the authors present a survey of results on the rate of convergence of finite difference schemes applied to initial-value problems for hyperbolic equations and discuss the dependence of this rate upon the smoothness of the initial values.
Abstract: Publisher Summary This chapter highlights the rate of convergence of difference schemes for hyperbolic equations. It presents a survey of results on the rate of convergence of finite difference schemes applied to initial-value problems for hyperbolic equations. It discusses the dependence of this rate of convergence upon the smoothness of the initial values. The chapter discusses the spaces of functions in which the results are expressed and also states some interpolation properties of these spaces. It presents a general convergence result that applies to the case when the hyperbolic system is correctly posed and the difference operator is stable in the Lp space under consideration. As these assumptions are in general satisfied only in L2, their applicability to other Lp spaces is limited to special cases. The chapter presents a maximum-norm convergence result based on the L2-theory and on an embedding lemma of the Sobolev type. The chapter also explores the scalar one-dimensional case.