Showing papers on "Line search published in 1988"

Learning Algorithms for Connectionist Networks: Applied Gradient Methods of Nonlinear Optimization

Abstract: The problem of learning using connectionist networks, in which network connection strengths are modified systematically so that the response of the network increasingly approximates the desired response can be structured as an optimization problem. The widely used back propagation method of connectionist learning [19, 21, 18] is set in the context of nonlinear optimization. In this framework, the issues of stability, convergence and parallelism are considered. As a form of gradient descent with fixed step size, back propagation is known to be unstable, which is illustrated using Rosenbrock's function. This is contrasted with stable methods which involve a line search in the gradient direction. The convergence criterion for connectionist problems involving binary functions is discussed relative to the behavior of gradient descent in the vicinity of local minima. A minimax criterion is compared with the least squares criterion. The contribution of the momentum term [19, 18] to more rapid convergence is interpreted relative to the geometry of the weight space. It is shown that in plateau regions of relatively constant gradient, the momentum term acts to increase the step size by a factor of 1/1-μ, where μ is the momentum term. In valley regions with steep sides, the momentum constant acts to focus the search direction toward the local minimum by averaging oscillations in the gradient. Comments University of Pennsylvania Department of Computer and Information Science Technical Report No. MSCIS-88-62. This technical report is available at ScholarlyCommons: http://repository.upenn.edu/cis_reports/597 LEARNING ALGORITHMS FOR CONNECTIONIST NETWORKS: APPLIED GRADIENT METHODS OF NONLINEAR OPTIMIZATION

Experiments with successive quadratic programming algorithms

Abstract: There are many variants of successive quadratic programming (SQP) algorithms Important issues include: the choice of either line search or trust region strategies; the QP formulation to be used; and how the QP is to be solved Here, we consider the QP's proposed by Fletcher and Powell and discuss a specialized reduced-gradient procedure for solving them A computer implementation is described, and the various options are compared on some well-known test problems Factors influencing robustness and speed are identified

A discrete search technique for global optimization

Abstract: We have developed a new algorithm which uses the trajectories of a discrete dynamical system to sample the domain of an objective function in search of global minima. We demonstrate the effectiveness of this algorithm by applying it to a model geometry optimization problem. Significant improvements in optimization efficiency are demonstrated, in that our algorithm returns lower minima than conventional line minimization in 79% of the optimization runs we made. The method is orders of magnitude less computationally intensive than simulated annealing, while returning good minima for functions possessing many local minima. The method is simple to program, and requires only two-point gradient calculations for its implementation.

Numerical experiments with variations of the Gauss-Newton algorithm for nonlinear least squares

Abstract: In this paper, the classical Gauss-Newton method for the unconstrained least squares problem is modified by introducing a quasi-Newton approximation to the second-order term of the Hessian. Various quasi-Newton formulas are considered, and numerical experiments show that most of them are more efficient on large residual problems than the Gauss-Newton method and a general purpose minimization algorithm based upon the BFGS formula. A particular quasi-Newton formula is shown numerically to be superior. Further improvements are obtained by using a line search that exploits the special form of the function.

Newton-type algorithms with nonmonotone line search for large-scale unconstrained optimization

Abstract: In this paper we define globally convergent algorithms for the solution of large dimensional unconstrained minimization problems. The algorithms proposed employ a nonmonotone steplength selection rule along the search direction which is determined by means of a Truncated-Newton algorithm. Numerical results obtained for a set of test problems are reported.

The analysis of consolidation by a quasi‐Newton technique

Abstract: A quasi-Newton algorithm is implemented for the solution of multi-dimensional, linear consolidation problems. The study is motivated by the need to implement an efficient equation-solving technique for the solution of large systems of equations typical in problems of consolidation of saturated porous media. The proposed procedure obviates the need to reassemble and re-factorize the global coefficient matrix every load increment, albeit the time step may be held variable in the analysis. The method employs the combined techniques of ‘line search’ and BFGS updates applied to the coupled equations. A numerical example is presented to show that the proposed method is computationally more efficient than the conventional direct equation-solving scheme, particularly when solving large systems of finite element equations.

Optimization with the Mixed Coordination Method

Abstract: This paper studies static optimization with equality constraints by using the mixed coordination method. The idea is to relax equality constraints via Lagrange multipliers, and creat a hierarchy where the Lagrange multipliers and part of the decision variables are selected as high level variables. The method was proposed about ten years ago with a simple high level updating scheme. In this paper we show that this simple updating scheme has a linear convergence rate under appropriate conditions. To obtain faster convergence, the Modified Newton's Method is adopted at the high level. There are two difficulties associated with the Modified Newton's Method. One is how to obtain the Hessian matrix in determining the Newton direction, as second order derivatives of the objective function with respect to all high level variables are needed. The second is when to stop in performing a line search along the Newton direction, as the high level problem is a maxmini problem looking for a saddle point. In this paper, the Hessian matrix is obtained by using a kind of sensitivity analysis. The line search stopping criterion, on the other hand, is based on the norm of the gradient vector. Extensive numerical testing results are provided in the paper. Since the low level is a set of independent subproblems, the method is well suited for parallel processing. Furthermore, since convexification terms can be added while maintaining separability of the original problem, the method is promising for nonconvex problems.

Parallel deterministic and stochastic optimization algorithms for computer vision

Abstract: Two classes of computational vision models-mechanical models and their statistical equivalent-Markov random fields models are examined for various vision problems. Different vision problems are formulated as optimization problems, and new algorithms are developed to obtain the optimal solution for both the deterministic and stochastic formulations. The equivalence between direct optimization of functionals in the deterministic set up and the stochastic maximum aposteriori (MAP) solutions is demonstrated using Gibbs energy functions. The weak membrane model is shown to belong to a class of compound Gauss Markov random field models. Graduated non-convexity algorithms originating in the deterministic formulation are extended to MAP estimation in the presence of additive noise. The Gibbs energy function is used to formulate a unified framework for many different vision problems. This unified methodology can serve to combine multiple sources of information in order to achieve a more robust vision system. The robust line search conjugate gradient algorithm is suggested for optimizing the nonlinear Gibbs energy functions, arising from the unified formulation. The special structure of the Gibbs energy function is used to derive a pyramid implementation for the algorithm. For many early vision problems, calculus of variations transforms the optimization problems to a set of linear or nonlinear Poisson equations. We derive new efficient algorithms based on direct Poisson solvers and embedding techniques. This formulation is also used to derive a new algorithm for SFS from stereo images. Texture segmentation is an important component in an image understanding system. Two different optimality criteria-MAP and minimum probability of error per pixel (MPM), were compared for this problem. We experimented with the corresponding stochastic algorithm-stochastic relaxation for MAP and the MPM algorithm. The deterministic iterative conditional modes algorithm suggested by Besag to approximate the MAP solution was also implemented. We conclude that labelling problems are highly non-convex and should be treated with an algorithm that consists of a random element. We also show that one can obtain estimates that interpolate between the MAP and MPM estimates by running the MPM algorithm with temperature strictly smaller than 1. (Copies available exclusively from Micrographics Department, Doheny Library, USC, Los Angeles, CA 90089-0182.)