Showing papers on "Rate of convergence published in 1970"
TL;DR: In order to avoid the time delays associated with linearly convergent division based on subtraction, other iterative schemes can be used based on series expansion of the reciprocal, multiplicative sequence, or additive sequence convergent to the quotient.
Abstract: In order to avoid the time delays associated with linearly convergent division based on subtraction, other iterative schemes can be used. These are based on 1) series expansion of the reciprocal, 2) multiplicative sequence, or 3) additive sequence convergent to the quotient. These latter techniques are based on finding the root of an arbitrary function at either the quotient or reciprocal value. A Newton-Raphson iteration or root finding iteration can be used.
153 citations
TL;DR: In this paper, the authors derived necessary and sufficient conditions on the range of one such parameter to guarantee stability of the method, and showed that the parameter effects only the length, not the direction, of the search vector at each step, and used this result to derive several computational algorithms.
Abstract: Quasi-Newton methods accelerate gradient methods for minimizing a function by approximating the inverse Hessian matrix of the function. Several papers in recent literature have dealt with the generation of classes of approximating matrices as a function of a scalar parameter. This paper derives necessary and sufficient conditions on the range of one such parameter to guarantee stability of the method. It further shows that the parameter effects only the length, not the direction, of the search vector at each step, and uses this result to derive several computational algorithms. The algorithms are evaluated on a series of test problems.
147 citations
TL;DR: The paper shows that proper refinement of the elements around the corners leads to the rate of convergence which is the same as it would be on domain with smooth boundary.
Abstract: The rate of convergence of the finite element method is greatly influenced by the existence of corners on the boundary. The paper shows that proper refinement of the elements around the corners leads to the rate of convergence which is the same as it would be on domain with smooth boundary.
131 citations
TL;DR: In this paper, it was shown that higher-order terms in the Wiener-Hermite expansion are capable of representing shocks, which dissipate the energy, even for a nearly Gaussian field of evolving three-dimensional turbulence.
Abstract: Meecham and his co-workers have developed a theory of turbulence involving a truncated Wiener–Hermite expansion of the velocity field. The randomness is taken up by a white-noise function associated, in the original version of the theory, with the initial state of the flow. The mechanical problem then reduces to a set of coupled integro-differential equations for deterministic kernels. We have solved numerically an analogous set for Burgers's model equation and have computed, for the sake of comparison, actual random solutions of the Burgers equation. We find that the theory based on the first two terms of the Wiener–Hermite expansion predicts an insufficient rate of energy decay for Reynolds numbers larger than two, because the equations for the kernels contain no convolution integrals in wave-number space and therefore permit no cascade of energy. An energy cascade in wave-number space corresponds to a cascade up through successive terms of the Wiener-Hermite expansion. Pictures of the Gaussian and non-Gaussian components of an actual solution of the Burgers equation show directly that only higher-order terms in the Wiener–Hermite expansion are capable of representing shocks, which dissipate the energy. Higher-order terms would be needed even for a nearly Gaussian field of evolving three-dimensional turbulence. ‘Gaussianity’, in the experimentalist's sense, has no bearing on the rate of convergence of a Wiener–Hermite expansion whose white-noise function is associated with the initial state. Such an expansion would converge only if the velocity field and its initial state were joint-normally distributed. The question whether a time-varying white-noise function can speed the convergence is treated in the paper following this one.
71 citations
TL;DR: In this paper, some methods of minimization under constraints of the equation type are discussed, in which the iterative process is based both on the initial variables and on dual variables (Lagrange multipliers).
Abstract: Some methods of minimization under constraints of the equation type will be discussed, in which the iterative process is based both on the initial variables and on dual variables (Lagrange multipliers). In the classification given in [1], these methods are of the duality type. Alternatively, they may be interpreted as iterative methods for finding the stationary (in particular, saddle) points of the Lagrange function. We assume that both the functional to be minimized, and the constraints, are reasonably smooth. In Section 1 we describe the methods, prove their local convergence, and estimate the rate of convergence. In Section 2 we consider the merits and drawbacks of the methods from the computational point of view, and outline a means for selecting the initial approximation.
40 citations
TL;DR: The quadratically convergent approach to solving the correct SCF equations for general open-shell systems (with orthogonal orbitals) is derived and used to discuss other less complicated approaches.
40 citations
TL;DR: In this article, the convergence of finite difference approximations for the general eigenvalue and boundary value problem of ODEs is proved under the condition of consistency and stability.
Abstract: In this paper the convergence of finite difference approximations for the general eigenvalue and boundary value problem of ordinary differential equations is proved under the condition of consistency and stability. The eigenvalues are shown to converge preserving multiplicity. Estimates are given for the rate of convergence of difference quotients and eigenvalues.
38 citations
TL;DR: In this article, the authors present a practical formulation for the solution of the elastostatic problem of a semi-infinite solid cylinder with the long sides free from stress and self-equilibrated tractions applied on the end.
Abstract: This paper presents a practical formulation for the solution of the elastostatic problem of a semi-infinite solid cylinder with the long sides free from stress and self-equilibrated tractions applied on the end. The formulation is entirely in terms of stresses and displacement related auxiliary variables of the same differential order as the stresses. The equilibrium equations, the Beltrami-Michell equations of compatibility and the definitions of the auxiliary variables are used to write a first order matrix differential equation. The solution of this non-self adjoint equation yields a series of non-orthogonal vector eigenfunctions. The coefficients of this series are chosen by use of a generalized biorthogonality condition. Numerical solutions of trial problems are presented as an indication of the rate of convergence of this series.
36 citations
TL;DR: In this article, the authors apply the theory of interpolation spaces to different parts of approximation theory and study the rate of convergence of summation processes of Fourier series and Fourier integrals.
Abstract: In this paper we apply the theory of interpolation spaces to different parts of Approximation theory. We study the rate of convergence of summation processes of Fourier series and Fourier integrals. The main body of the paper is devoted to a study of the rate of convergence of solutions of difference schemes for parabolic initialvalue problems with constant coefficients and to related problems.
36 citations
TL;DR: This correspondence extends two algorithms for unconstrained minimization in Rn, Davidon's method and a projected gradient algorithm, to optimal control problems, and shows that recent computational results indicate that this may improve the rate of convergence.
Abstract: This correspondence extends two algorithms for unconstrained minimization in Rn, Davidon's method and a projected gradient algorithm, to optimal control problems. Both require only the value and gradient of the functional being minimized; both find the current search direction by operating on the negative gradient with a dyadic operator; and both generate conjugate directions when applied to a quadratic functional. To compute the direction of search at iteration i , the Davidon algorithm requires that 2i + 2 functions, generated in past and current cycles, be stored. The projected gradient method requires only i + 2 . Both decrease the value of the functional being minimized at each step. The storage demands will require that both methods be restarted periodically. However, recent computational results indicate that this may improve the rate of convergence.
33 citations
TL;DR: In this article, a theory of the convergence of minimization processes convex functionals that reduce the value of the functional at each step is presented, and convergence conditions are derived and the rate of convergence and stability of the process are studied in this terminology.
Abstract: This article sets out a theory of the convergence of minimization processes convex functionals that reduce the value of the functional at each step. A geometrical language, independent of the algorithmic structure, is used to describe the processes: the language of relaxation angles and factors. Convergence conditions are derived and the rate of convergence and stability of the process are studied in this terminology. Translation from the language of concrete algorithms to the geometrical terminology is not difficult, and thanks to this the theory has a wide area of applications: gradient and operator-gradient processes, processes of Newtonian type, coordinate relaxation, Jacobi processes and relaxation for the Rayleigh functional.
IBM1
TL;DR: In this paper, two methods for the iterative synthesis of an array processor are discussed: steepest descent and conjugate gradients with projection, and a bound for the rate of convergence is obtained for these iterative procedures.
Abstract: Two methods for the iterative synthesis of an array processor are discussed: the method of steepest descent and the method of conjugate gradients with projection. These methods require no intermediate statistics such as the covariance matrix function or the cross-power spectral matrix, and therefore, require less storage space than the conventional synthesis methods. A bound for the rate of convergence is obtained for these iterative procedures and it is shown that the convergence is geometric. The algorithms are then applied to seismic data of the Montana large aperture seismic array. Simulation results indicate that the convergence is so fast that a few iterations are enough from the practical viewpoint. Therefore, these methods can also save significant computation time as well.
TL;DR: A first-order method for solving the problem: minimize f(x) subject to Ax - b \geqq 0 is presented and contains ideas based on variable reduction with anti-zig-zagging and acceleration devices based on the Variable Metric Method.
Abstract: A first-order method for solving the problem: minimize f(x) subject to Ax − b ≧ 0 is presented. The method contains ideas based on variable reduction with anti-zig-zagging and acceleration devices based on the Variable Metric Method. Proof of convergence to a Kuhn-Tucker Point, and statement of the rate of convergence when the strict second order sufficiency conditions hold are given.
TL;DR: In this article, a framework was developed to give a simplified proof of conditions given by Eaves and Zangwill under which inactive constraints may be dropped after each subproblem in cutting-plane algorithms.
Abstract: This note shows that a framework I developed earlier can be used to give a simplified proof of conditions given by Eaves and Zangwill (which weaken the uniform concavity requirement on my earlier objective function) under which inactive constraints may be dropped after each subproblem in cutting-plane algorithms. Here the convergence rate I established previously as an extension of the results of Levitin and Polyak is improved and its application extended.
01 Jan 1970
TL;DR: If the strict second-order sufficiency conditions hold, the rate of convergence of the algorithm is shown to be superlinear or even quadratic with aLipschitz condition on the second derivatives of f(x) subject to Ax − b ≥ 0.
Abstract: An algorithm using second derivatives for solving the problem: minimize f(x) subject to Ax − b ≥ 0 is presented. Convergence to a Second-Order Kuhn Tucker Point is proved. If the strict second-order sufficiency conditions hold, the rate of convergence of the algorithm is shown to be superlinear or even quadratic with aLipschitz condition on the second derivatives of f(x).
TL;DR: In this article, a method of analysis of nonhomogeneous elastic solids involving general three-dimensional states of stress was presented, and the displacement equations of equilibrium were based on the finite-element variational procedure.
TL;DR: In this article, the rate of convergence in mean square of the Bayes solution for a discretized parameter space has been investigated and the asymptotic solution and rates of convergence of a class of minimum-integral-square-difference algorithms are determined.
Abstract: There are several approaches to unsupervised estimation that have application to problems of communications, control, and pattern recognition. This paper presents properties of several different digitally implemented algorithms suitable for unsupervised estimation. One result is the rate of convergence in mean square of the Bayes solution for a discretized parameter space. A regression function that is the expected value of the natural logarithm of the mixture probability density function naturally arises from the Bayes approach. This regression function can be used to devise unsupervised estimation algorithms of the stochastic approximation form. Also, the asymptotic solution and rates of convergence in mean square of a class of minimum-integral-square-difference algorithms are determined. Two other estimators that use a "net" on the parameter space are also presented.
TL;DR: In this paper, the authors studied the rate of convergence of penalty function algorithms and proposed a sequence of unconstrained minimisation problems to transform the penalty function problem into a series of nonconstrained problems.
TL;DR: In this article, the authors present the stochastic approximation for minimizing functions and discuss its application to recovering functions from noisy measurements of their values, including the potential function method. But they do not discuss the relationship of stochastically approximating functions to other techniques such as the potential functions method.
Abstract: Publisher Summary This chapter presents the stochastic approximation for minimizing functions and discusses its application to recovering functions from noisy measurements of their values. The chapter describes the relationship of stochastic approximation to other techniques such as the potential function method. There are two approaches for accelerating the convergence of stochastic approximation procedures. The first approach is to accelerate convergence by selecting a proper weighting sequence. The choice of the weighting sequence based on information concerning the behavior of the regression function should improve the rate of convergence. The second approach for accelerating convergence is by taking more observations at each stage of iteration. The algorithms for finding zeroes of functions can be modified to estimate parameters that are time-varying and are varying in either a deterministic manner or in a random fashion with their variances going to zero as time tends to infinity.
TL;DR: A related imbedding method has been presented, where a sequence of locally convergent iteration processes produces a global convergence even without any knowledge of a first approximation.
Abstract: In the solution methods introduced byDavidenko andGavurin the nonlinear equation is simulated by an initial value problem involving a differential equation. In this paper a related imbedding method has been presented, where a sequence of locally convergent iteration processes produces a global convergence even without any knowledge of a first approximation.
TL;DR: Theorem 1 of the present paper shows that, under certain conditions, minimization algorithms can be based on approximation of the initial functional by a segment of a Taylor series with an arbitrary number of terms.
Abstract: The results obtained in [1] will be developed below. At the present time, minimization methods based on linearization of the initial functional (gradient methods) are well known [2–5] for problems on an unconstrained extremum and for minimization problems in a closed convex region. There are also well-known second-order methods, convergent from any initial approximation (they are in essence a generalization of Newton's method); such niethods were discussed in [6] for the unconstrained extremum problem, and in [1] for the constrained problem. Second-order methods are based on quadratic approximation of the functional. Theorem 1 of the present paper indicates a general approach to the construction of methods of this type; it shows that, under certain conditions, minimization algorithms can be based on approximation of the initial functional by a segment of a Taylor series with an arbitrary number of terms. All the methods that can be obtained by means of Theorem 1 are convergent from any initial approximation; this is achieved by regulating the length of step in the direction of motion. The algorithm for selecting the length of step proposed in the theorem is unified for all methods, and differs from previous algorithms. Its advantages and usefulness are discussed in the paper. We observe, for instance, that our algorithm for step length selection in secondorder methods enables a higher order estimate to be obtained for the rate of convergence than that obtained in [6], while at the same time reducing the labour of each iteration. This result follows from Theorem 2. We show in Theorem 3 that the convergence rate estimate obtained in [6] for a second-order method for solving unconstrained extremum problems, still holds for solving problems of minimization in a closed convex set (with the same method of step length selection). The approach used in Theorem 1 for constructing algorithms for solving problems of minimization in a closed region, may also be used for proving the convergence of algorithms employed for solving problems with supplementary restrictions in the form of equations (Theorem 4). Algorithms for solving such problems were considered in [1], but with a different method of step length selection from that used in Theorem 4. By using the method of Theorem 4, the number of computations in such problems may be considerably reduced.
TL;DR: A modification to the classical approach of the quasi-Newton method that takes into account the structure of the network and shows that this approach represents a clear gain in terms of computational time without increasing the requirement of memory space.
Abstract: The backpropagation algorithm is the most popular procedure to train self-learning feedforward neural networks. However, the rate of convergence of this algorithm is slow because the backpropagation algorithm is mainly a steepest descent method. Several researchers have proposed other approaches to improve the rate of convergence: conjugate gradient methods, dynamic modification of learning parameters, full quasi-Newton or Newton methods, stochastic methods, etc. Quasi-Newton methods were criticized because they require significant computation time and memory space to perform the update of the hessian matrix. This paper proposes a modification to the classical approach of the quasi-Newton method that takes into account the structure of the network. With this modification, the size of the problem is not proportional to the total number of weights but depends on the number of neurons of each level. The modified quasi-Newton method is tested on two examples and is compared to classical approaches. The numerical results show that this approach represents a clear gain in terms of computational time without increasing the requirement of memory space.
TL;DR: In this paper, the authors developed approximation methods for nonlinear programming problems using function-values only, where the idea is to approximate the objective function by a quadratic interpolating polynomial.
Abstract: In the paper the authors develop approximation methods for nonlinear programming problems using function-values only. The idea is to approximate the objective function by a quadratic interpolating polynomial. It is shown how to construct methods of an arbitrarily high convergence rate. By aid of the efficiency index methods are selected from the class in view which require a minimal number of function-values in order to guarantee a given exactness.
TL;DR: In this paper, the necessary and sufficient conditions for the convergence of the generalized overrelaxation method (go-method) and the extrapolated Jacobi method (ECM) were discussed.
19 Aug 1970
TL;DR: In this paper, an approach to the development of a second and higher-order perturbation theory for two-body trajectories is presented, and the radii of convergence for these series solutions are determined.
Abstract: An approach to the development of a second- and higher-order perturbation theory for two- body trajectories is presented. It is shown that the higher order analysis can be developed in a systematic manner in terms of series solutions to the two-body problem as functions of the time variable. Simple algebraic recursive formulas for the determination of the series coeffi- cients are derived and the radii of convergence for these series solutions are determined. Their accuracy and the rate of convergence are investigated in a number of numerical cases. ingly more precise mission objectives and with long flight times. It is the purpose of this paper to present an approach to the higher order perturbation analysis of two-body trajec- tories. A second-order theory, involving second partial derivatives of the six-dimensional state vector of a spacecraft at a given time with respect to the state vector at an initial epoch, is con- sidered in detail. It is shown that a second as well as higher order theory can be developed in a simple and systematic manner using power series solutions of the two-body problem as functions of the time variable, a technique suggested by the theory of Lie series.1 In this paper, the series representa- tions associated with the second-order theory are developed; they are uniformly valid for all two-body conies. It is shown that the coefficients of these series expansions can be generated by simple algebraic recursive formulas, and their radii of con- vergence can be determined analytically. The rate of convergence and truncation error associated with these series representations are investigated in a number of numerical cases. Results indicate that the convergence characteristics and accuracies can in general be estimated in terms of the eccentricity and the position vector at the initial epoch. Generally speaking, the results exhibit good con- vergence characteristics for elliptical orbits covering less than one orbital revolution. Results for hyperbolic orbits are less favorable.
TL;DR: The truncated functional expansion of a nonlinear system is representative of the system response when the rate of convergence is fast and this fact has been exploited to find an optimisation approach, subject to accuracy constraint, when the input is a step with probabilistic amplitude.
Abstract: The truncated functional expansion of a nonlinear system is representative of the system response when the rate of convergence is fast. This fact has been exploited to find an optimisation approach, subject to accuracy constraint, when the input is a step with probabilistic amplitude.
01 Jan 1970
TL;DR: In this article, the rate of convergence and local convergence of iterative processes were discussed from a general viewpoint and then for several important special processes, such as the Newton attraction theorem, the Ostrowski theorem, and Newton's method.
Abstract: This chapter discusses the rate of convergence and local convergence of iterative processes, first from a general viewpoint and then for several important special processes. The chapter also discusses the one-step stationary processes of the form x k+1 = Gx k , where k = 0,1,… The Ostrowski theorem, the linear convergence theorem, and Newton's method and some of its modifications are further discussed in the chapter. The chapter also discusses the Newton attraction theorem. High-order iterative processes may be generated by the composition of two lower-order processes. The chapter further discusses generalized linear iterations and the Newton–Peaceman–Rachford theorem and explains how some of the point-of-attraction results are applied to continuation methods.