Showing papers on "Nonlinear conjugate gradient method published in 1992"

Global Convergence Properties of Conjugate Gradient Methods for Optimization

Abstract: This paper explores the convergence of nonlinear conjugate gradient methods without restarts, and with practical line searches. The analysis covers two classes of methods that are globally convergent on smooth, nonconvex functions. Some properties of the Fletcher–Reeves method play an important role in the first family, whereas the second family shares an important property with the Polak–Ribiere method. Numerical experiments are presented.

Optimization for training neural nets

Abstract: Various techniques of optimizing criterion functions to train neural-net classifiers are investigated. These techniques include three standard deterministic techniques (variable metric, conjugate gradient, and steepest descent), and a new stochastic technique. It is found that the stochastic technique is preferable on problems with large training sets and that the convergence rates of the variable metric and conjugate gradient techniques are similar. >

Conjugate gradient techniques for adaptive filtering

Abstract: The application of the conjugate gradient technique for the solution of the adaptive filtering problem is discussed. An algorithm that does not require a line search or a knowledge of the Hessian is developed based on the conjugate gradient method. The choice of the gradient average window in the algorithm is shown to provide a trade-off between computational complexity and convergence performance. The method is capable of providing convergence comparable to recursive least squares (RLS) schemes at a computational complexity that is intermediate between the least mean square (LMS) and the RLS methods and does not suffer from any known instability problems. >

A Biconjugate Gradient FFT Solution for Scattering by Planar Plates

Abstract: An efficient numerical solution of the scattering by planar perfectly conducting or resistive plates is presented. The electric field integral equation is discretized using roof–top subdo–main functions as testing and expansion basis and the resulting system is solved via the biconjugate gradient (BiCG) method in conjunction with the fast Fourier transform (FFT). Unlike other formulations employed in conjunction with the conjugate gradient FFT (CG–FFT) method, in this formulation the derivatives associated with the dyadic Green's function are transferred to the testing and expansion basis, thus reducing the singularity of the kernel. This leads to substantial improvements in the convergence of the solution as demonstrated by the included results.

Following gradient extremal paths

Abstract: For any point on a gradient extremal path, the gradient is an eigenvector of the hessian. Two new methods for following the gradient extremal path are presented. The first greatly reduces the number of second derivative calculations needed by using a modified updating scheme for the hessian. The second method follows the gradient extremal using only the gradient, avoiding the hessian evaluation entirely. The latter algorithm makes it possible to use gradient extremals to explore energy surfaces at higher levels of theory for which analytical hessians are not available.

Implementations of Affine Scaling Methods: Approximate Solutions of Systems of Linear Equations Using Preconditioned Conjugate Gradient Methods

Abstract: The conjugate gradient method has been proposed for solving system of linear equations arising at each iteration of interior point methods. This paper studies several problems associated with developing such implementations. This includes development of a termination criteria, computation of an effective preconditioner, and the lack of positive definiteness of the matrix. INFORMS Journal on Computing, ISSN 1091-9856, was published as ORSA Journal on Computing from 1989 to 1995 under ISSN 0899-1499.

Generalized Polak-Ribiegre algorithm

Abstract: A new generalized Polak-Ribiere conjugate gradient algorithm is proposed for unconstrained optimization, and its numerical and theoretical properties are discussed. The new method is, in fact, a particular type of two-dimensional Newton method and is based on a finite-difference approximation to the product of a Hessian and a vector.

Comparison and evaluation of variants of the conjugate gradient method for efficient learning in feed-forward neural networks with backward error propagation

Abstract: Backward error propagation is a widely used procedure for computing the gradient of the error for a feed-forward network and thus allows the error to be minimized (learning). Simple gradient descent is ineffective unless the step size used is very small and it is then unacceptably slow. Conjugate gradient methods are now increasingly used as they allow second-derivative information to be used, thus improving learning. Two different implementations are described; one using an exact line search to find the minimum of the error along the current search direction, the other avoids the line search by controlling the positive indefiniteness of the Hessian matrix. The two implementations are compared and evaluated in the context of an image recognition problem using input bit-maps with a resolution of 128 by 128 pixels.

The numerical calculation of rough surface scattering by the conjugate gradient method

Abstract: The conjugate gradient method has been used to compute the solutions of the magnetic field integral equation for two-dimensional, perfectly conducting, Gaussian, rough surfaces. For surfaces whose roughness is of similar order to the incident wavelength, the rate of convergence of the conjugate gradient method is controlled by the RMS surface slope. In some cases, solutions are obtained with a significant reduction in the computation that direct methods require. Using the solution, the scattered field in the vicinity of the surface was calculated, and it is shown that the scattered field satisfies the extinction theorem to a good approximation. >

Time complexity of a parallel conjugate gradient solver for light scattering simulations: theory and SPMD implementation

Abstract: We describe parallelization for distributed memory computers of a preconditioned Conjugate Gradient method, applied to solve systems of equations emerging from Elastic Light Scattering simulations. The execution time of the Conjugate Gradient method is analyzed theoretically. First expressions for the execution time for three different data decompositions are derived. Next two processor network topologies are taken into account and the theoretical execution times are further specified as a function of these topologies. The Conjugate Gradient method was implemented with a rowblock data decomposition on a ring of transputers. The measured and theoretically calculated execution times agree within 5 %. Finally convergence properties of the algorithm are investigated and the suitability of a polynomial preconditioner is examined.

Conjugate progector preconditioning for the solution of contact problems

Abstract: We present a new algorithm for numerical solution of contactproblems without friction. The method is based on enhancing conjugate projector preconditioning into the standard conjugate gradient method for the solution of quadratic programming problems with inequality constraints. We show that the algorithm is correct and discuss the efficiency of the algorithm. The relation of the algorithm to the domain decomposition methods is also explained. The paper is supplied with numerical experiment.

Semi-srochastic approximation by the response surface methodology (RMS)

Abstract: Stochastic approximation procedures, eg stochastic gradient methods for the minimization of a mean value unction on a cover feasible domain can be accelerated considerably by using deterministic descent directions or more exact gradient estimates at certain iteration points The gradient estimator estimator considered here is defined by the gradient of a so-called first or second order polynomial reponse surface model f of the unknow objective function f,where the estimate f of F is constructed by regression analysisThe accuracy of this gradient estimator is examined and the convergence behavior of the resulting hybride procedure—in comparison with standard stochastic approximation—is considered

Computational experience with improved conjugate gradient methods for unconstrained minimization

Abstract: We are concerned with the finding of a local minimum x* € IR\" of the function F : X —> IR on an open set X C R\" , i .e. a point x* € IR\" tha t satisfies the inequality F{x*) 0 , where B(x*,e) = {x €lR : || x-x* | |

Implementation of a parallel conjugate gradient method for simulation of elastic light scattering

Abstract: We simulate elastic light scattering with the coupled dipole method. The kernel of this method is a large set of linear equations. The n ×n system matrix is complex, symmetric, full, and diagonally dominant. This application is a typical example of problems arising in computational electromagnetics. The matrix equations are usually solved with (preconditioned) conjugate gradient methods. For realistic problems the size of the matrix is very large (n ~ 10 4 to 106). In that case sustained calculation speeds in the Gflop/s range are required to keep execution times acceptable.We introduce a methodology to parallelize the conjugate gradient method for this type of problems, with emphasis on coarse grain distributed memory implementations. We present results for an implementation on a transputer network.

Successive linearization methods for large-scale nonlinear programming problems

Abstract: We propose a sparsity preserving algorithm for solving large-scale, nonlinear programming problems. The algorithm solves at each iteration a subproblem, which contains a linearized objective function augmented by a simple quadratic term and linearized constraints. The quadratic term added to the linearized objective function plays the role of step restriction which is essential in ensuring global convergence of the algorithm. If the conjugate gradient method or successive over-relaxation method is used to solve the subproblems, the sparsity of the original problem is preserved, because those methods only require simple operations on the rows of the constraint matrix. Thus, large-scale problems can be dealt with when the constraint matrices are sparse enough to be stored in a compact form. Practical implementation of the algorithm is described and computational results are reported.

Boundary-orthogonal biharmonic grids via preconditioned gradient methods

Abstract: A finite difference approximation of the biharmonic equation is solved using three preconditioned gradient methods for the generation of curvilinear boundary-orthogonal grids in two dimensions. The developed algorithms based on the conjugated gradient, the steepest descent, and the minimal residual method are applied in a number of domains of engineering interest for the purpose of comparison in terms of CPU time and number of iterations to convergence.

A supermemory gradient projection algorithm for optimization problem with nonlinear constraints

Abstract: In this paper, by extending concept of the supermemory gradient method for unconstrained optimization problems, we present a supermemory gradient projection algorithm for nonlinear programming with nonlinear constraints. Under some suitable conditions we prove its global convergence.

A class of descent algorithms for nonlinear function minimization with linear equality constraints

Abstract: We consider the application of the ABS algorithms, introduced by [I], mainly for the solution of linear algebraic systems, to constrained optimization problems with equality constraints. A general formulation of feasible direction algorithms is obtained and the ABS formulation is given of the reduced gradient method of Wolfe (1967) and of the projected gradient method of Rosen (1960)

Hybrid optimization-an experimental study

Abstract: The authors compare the performance of a hybrid optimization method to that of pure gradient based methods. The hybrid optimization method comprises an initial adaptive ordinal search phase followed by a gradient ascent (descent) phase. The adaptive ordinal search phase consists of fixing the size of the design population and ranking the members of the population using an estimated value of the performance. Members of the design population for the next stage are picked using the top designs of the previous population. This process is achieved via a variation on the standard genetic algorithm (see D. E. Goldberg, 1989). Ho et al. (1992) showed that ranks of populations are relatively insensitive to simulation noise, and as the experimental data show, this fact is useful in using short simulation runs to improve the search efficiency before the onset of the final gradient ascent (descent) phase. >

Signal selection for non-symmetric sources in two dimensions

Abstract: An attempt is made to determine which two-dimensional signal constellation minimizes the probability of error under an average power constraint for nonsymmetric sources. Two different approaches to the problem are presented. First, a numerical method with the help of Lagrange multipliers is described. Then, a normalized conjugate gradient method and a normalized gradient method based on the approach given by G.J. Foschini et al. (1974) are provided. The gradient-based methods are powerful tools that can be applied to signal sets of any size and of any statistics. The normalized conjugate gradient method is a faster than the normalized gradient method. >

Interior dual proximal point algorithm using preconditioned conjugate gradient

Abstract: Serial and parallel implementations of the interior dual proximal point algorithm for the solution of large linear programs are described. A preconditioned conjugate gradient method is used to solve the linear system of equations that arises at each interior point interation. Numerical results for a set of multicommodity network flow problems are given. For larger problem preconditioned conjugate gradient method outperforms direct methods of solution. In fact it is impossible to handle very large problems by direct methods

The local convergence property of the nonlinear Voevodin-ABS algorithm

Abstract: We consider a subclass of the nonlinear ABS algorithm for solving nonlinear systems of equations corresponding to a generalization of the Voevodin method for linear systems which includes all conjugate gradient type methods. A local Q-order convergence theorem is obtained under only the standard conditions on the mapping, without additional conditions on the parameters defining the class.

Solving combinatorial optimization problems by gradient flows

Abstract: The relationship between certain NP-hard combinatorial optimization problems and gradient flows on the special orthogonal group is examined. It is shown that there is a correspondence between the local minima of a heuristic local search algorithm and the local minima of the gradient flow on the group. >

1992acc/ta1 0 conjugate gradient approach to parallel processing indynamic simulation of power systems

Abstract: Inthispaperwepresent a newapproach tosoving the powersystem dynamic simulation problem using theGeneralised Minimal Residual (GMRES)algorithm. Thisisoneof theseveral methods inthefamily ofConjugate Gradient methodsforsolving systems ofequatiom oftheformAz= b.Although someofthese methods require A besymmetric positive definite, GMRES hasnosuchrestriction. Thetechnique isillustrated ona39bus, 10machine powersystem using theCray Y-MP.