Showing papers on "Convergence (routing) published in 1994"

Convergence of the Cahn-Hilliard equation to the Hele-Shaw model

Abstract: We prove that level surfaces of solutions to the Cahn-Hilliard equation tend to solutions of the Hele-Shaw problem under the assumption that classical solutions of the latter exist. The method is based on a new matched asymptotic expansion for solutions, a spectral analysis for linearizd operators, and an estimate for the difference between the true solutions and certain approximate ones.

Simulated annealing: a proof of convergence

Abstract: We prove the convergence of the simulated annealing procedure when the decision to change the current configuration is blind of the cost of the new configuration. In case of filtering binary images, the proof easily generalizes to other procedures, including that of Metropolis. We show that a function Q associated with the algorithm must be chosen as large as possible to provide a fast rate of convergence. The worst case (Q constant) is associated with the "blind" algorithm. On the other hand, an appropriate Q taking sufficiently high values yields a better rate of convergence than that of Metropolis procedure. >

On the Convergence of a Class of Infeasible Interior-Point Methods for the Horizontal Linear Complementarity Problem

Abstract: Interior-point methods require strictly feasible points as starting points. In theory, this requirement does not seem to be particularly restrictive, but it can be costly in computation. To overcom...

Analysis of the back-propagation algorithm with momentum

Abstract: In this letter, the back-propagation algorithm with the momentum term is analyzed. It is shown that all local minima of the sum of least squares error are stable. Other equilibrium points are unstable. >

Discrete time high-order schemes for viscosity solutions of Hamilton-Jacobi-Bellman equations

Abstract: A general method for constructing high-order approximation schemes for Hamilton-Jacobi-Bellman equations is given. The method is based on a discrete version of the Dynamic Programming Principle. We prove a general convergence result for this class of approximation schemes also obtaining, under more restrictive assumptions, an estimate in $L^\infty$ of the order of convergence and of the local truncation error. The schemes can be applied, in particular, to the stationary linear first order equation in ${\Bbb R}^n$ . We present several examples of schemes belonging to this class and with fast convergence to the solution.

On relaxation algorithms in computation of noncooperative equilibria

Abstract: This paper considers a special class of numerical algorithms, the so-called relaxation algorithm, for Nash equilibrium points in noncooperative games. The relaxation algorithms have been studied by various authors for the deterministic case. Convergence conditions of this algorithm are based on fixed point theorems. For example, Basar (1987) and Li and Basar (1987) have proved its convergence for a two-player game via the contraction mapping theorem. For the quadratic case these conditions can be easily checked. For other nonlinear payoff functions it is sometimes difficult to check these convergence conditions. In this paper, the authors propose an alternative approach using the residual terms of the Nikaido-Isoda function. The convergence theorem is proved for nonsmooth weakly convex-concave Nikaido-Isoda functions. The family of weakly convex-concave functions is broad enough for applications, since if includes the family of smooth functions. When the payoff functions are twice continuously differentiable, the condition for the residual terms is reduced to strict positiveness of a matrix representing the difference of the Hessians of the Nikaido-Isoda function with respect to the first and second groups of variables. An analogous condition was used by Uryas'ev (1988) to prove convergence of the gradient-type algorithm for the Nash equilibrium problem. In this paper the authors discuss only the deterministic case; nevertheless this approach can be generalized for the stochastic Nash equilibrium problems with uncertainties in parameters. >

Direct adaptive control of parabolic systems: algorithm synthesis and convergence and stability analysis

Abstract: This paper presents a model reference adaptive control of a class of distributed parameter systems described by linear, n-dimensional, parabolic partial differential equations. Unknown parameters appearing in the system equation are either constant or spatially-varying. Distributed sensing and actuation are assumed to be available. Adaptation laws are obtained by the Lyapunov redesign method. It Is shown that the concept of persistency of excitation, which guarantees the parameter error convergence to zero in finite-dimensional adaptive systems, in infinite-dimensional adaptive systems should be investigated in relation to time variable, spatial variable, and also boundary conditions. Unlike the finite-dimensional case, in infinite-dimensional adaptive systems even a constant input is shown to be persistently exciting in the sense that it guarantees the convergence of parameter errors to zero. Averaging theorems for two-time scale systems which involve a finite dimensional slow system and an infinite dimensional fast system are developed. The exponential stability of the adaptive system, which is critical in finite dimensional adaptive control in terms of tolerating disturbances and unmodeled dynamics, is shown by applying averaging. >

Global convergence of damped Newton's method for nonsmooth equations via the path search

Abstract: A natural damping of Newton's method for nonsmooth equations is presented. This damping, via the path search instead of the traditional line search, enlarges the domain of convergence of Newton's method and therefore is said to be globally convergent. Convergence behavior is like that of line search damped Newton's method for smooth equations, including Q-quadratic convergence rates under appropriate conditions. Applications of the path search include damping Robinson-Newton's method for nonsmooth normal equations corresponding to nonlinear complementarity problems and variational inequalities, hence damping both Wilson's method (sequential quadratic programming) for nonlinear programming and Josephy-Newton's method for generalized equations. Computational examples from nonlinear programming are given.

Statistical Physics Algorithms That Converge

Abstract: In recent years there has been significant interest in adapting techniques from statistical physics, in particular mean field theory, to provide deterministic heuristic algorithms for obtaining approximate solutions to optimization problems. Although these algorithms have been shown experimentally to be successful there has been little theoretical analysis of them. In this paper we demonstrate connections between mean field theory methods and other approaches, in particular, barrier function and interior point methods. As an explicit example, we summarize our work on the linear assignment problem. In this previous work we defined a number of algorithms, including deterministic annealing, for solving the assignment problem. We proved convergence, gave bounds on the convergence times, and showed relations to other optimization algorithms.

Shape preserving properties and convergence of univariate multiquadric quasi-interpolation

Abstract: With a suitable modiication at the endpoints of the range, quasi{interpolation with univariate multiquadrics (x) = p c 2 + x 2 is shown to preserve convexity and monotonic-ity. If h is the maximum distance of centres, convergence of the quasi{interpolant is of order O(h 2 j log hj) if c = O(h). The log term can not be removed by introducing diierent boundary conditions or special placements of the centres.

Polynomiality of infeasible-interior-point algorithms for linear programming

Abstract: Kojima, Megiddo, and Mizuno investigate an infeasible-interior-point algorithm for solving a primal--dual pair of linear programming problems and they demonstrate its global convergence. Their algorithm finds approximate optimal solutions of the pair if both problems have interior points, and they detect infeasibility when the sequence of iterates diverges. Zhang proves polynomial-time convergence of an infeasible-interior-point algorithm under the assumption that both primal and dual problems have feasible points. In this paper, we show that a modification of the Kojima--Megiddo--Mizuno algorithm "solves" the pair of problems in polynomial time without assuming the existence of the LP solution. Furthermore, we develop anO(nL)-iteration complexity result for a variant of the algorithm.

Approximation in nonhierarchic system optimization

Abstract: This paper reports on the effectiveness of a nonhierarchic system optimization algorithm in application to complex coupled systems problems. A second-order nonhierarchic system optimization algorithm developed in earlier studies is modified in this study to provide for individual constraint/state modeling. A cumulative constraint formulation was used in previous implementation studies. The test problems in this study are each complex coupled systems. Complex coupled systems require an iterative solution strategy to evaluate system states. Nonhierarchic algorithm development is driven by these types of problems, and their study is imperative. The algorithm successfully optimizes each of the complex coupled systems. A significant reduction in the number of system analyses required for optimization is observed as compared with conventional optimization using the generalized reduced-gradient method. in the design database. The design database stores design site information generated during the subspace optimizations. A quadratic polynomial approximation to the design is formed using the strategy of Vanderplaats. 6 A weighted least-squares solution strategy is employed to solve for the second-order terms in Vanderplaats' strategy. Exact data in the design data- base are more heavily weighted in the least-squares solution procedure. The resulting quadratic polynomial forms the basis function of accumulated approximation replacing the linear basis used in the original formulation. In Renaud and Gabriele5 improved convergence is observed for the welded beam test problem. The improved convergence is attributed to the im- proved accuracy of cumulative constraint approximations when using second-order-based approximating functions. Additional studies using the second-order-based coordina- tion procedure of system approximation indicated that replac- ing the cumulative constraints with their component con- straints may improve algorithm performance. Implementation of the second-order-based coordination procedure of system approximation was less effective in reducing cycling when applied to the Golinski speed reducer problem. The speed reducer cumulative constraints were composed of a large num- ber of individual constraints as compared with the cumulative constraints in the welded beam test problem. With a larger number of individual constraints assigned to a cumulative constraint, it will more likely undergo a change in its active set during the coordination procedure. It is difficult to approxi- mate these changes in the cumulative constraints. Inaccurate cumulative constraint approximations reduce algorithm per- formance and delay convergence. Approximating individual constraints/states in the coor-

Stochastic Optimization by Simulation: Convergence Proofs for the GI/G/1 Queue in Steady-State

Abstract: Approaches like finite differences with common random numbers, infinitesimal perturbation analysis, and the likelihood ratio method have drawn a great deal of attention recently as ways of estimating the gradient of a performance measure with respect to continuous parameters in a dynamic stochastic system. In this paper, we study the use of such estimators in stochastic approximation algorithms, to perform so-called "single-run optimizations" of steady-state systems. Under mild conditions, for an objective function that involves the mean system time in a GI/G/1 queue, we prove that many variant of these algorithms converge to the minimizer. In most cases, however, the simulation length must be increased from iteration to iteration, otherwise the algorithm may converge to the wrong value. One exception is a particular implementation of infinitesimal perturbation analysis, for which the single-run optimization converges to the optimum even with a fixed and small number of ends of service per iteration. As a by-product of our convergence proofs, we obtain some properties of the derivative estimators that could be of independent interest. Our analysis exploits the regenerative structure of the system, but our derivative estimation and optimization algorithms do not always take advantage of that regenerative structure. In a companion paper, we report numerical experiments with an M/M/1 queue, which illustrate the basis convergence properties and possible pitfalls of the various techniques.

An optimal neural network process model for plasma etching

Abstract: Neural network models of semiconductor processes have recently been shown to offer advantages in both accuracy and predictive ability over traditional statistical methods. However, model development is complicated by the fact that back-propagation neural networks contain several adjustable parameters whose optimal values are initially unknown. These include learning rate, initial weight range, momentum, and training tolerance, as well as the network architecture. The effect of these factors on network performance is investigated here by means of a D-optimal experiment. The goal is to determine how the factors impact network performance and to derive a set of parameters which optimize performance based on several criteria. The network responses optimized are learning capability, predictive capability, and training time. Learning and prediction accuracy are quantified by the experimental error of the model. The process modeled is polysilicon etching in a CCl/sub 4/-based plasma. Statistical analysis of the experimental results reveals that learning capability and convergence speed depend mostly on the learning parameters, whereas prediction is controlled primarily by the number of hidden layer neurons. An optimal network structure and parameter set has been determined which minimizes learning error, prediction error, and training time individually as well as collectively. >

Parallel Variable Distribution

Abstract: An approach is presented here for solving optimization problems in which the variables are distributed among p processors. Each processor has primary responsibility for updating its own block of variables in parallel while allowing the remaining variables to change in a restricted fashion, e.g., along a steepest descent, quasi-Newton, or any arbitrary direction. This “forget-me-not” approach is a distinctive feature of the algorithm that has not been analyzed before. The parallelization step is followed by a fast synchronization step wherein the affine hull of the points computed by the parallel processors and the current point is searched for an optimal point. Convergence to a stationary point under continuous differentiability is established for the unconstrained case, as well as a linear convergence rate under the additional assumption of a Lipschitzian gradient and strong convexity. For problems constrained to lie in the Cartesian product of closed convex sets, convergence is established to a point sa...

Serial and parallel backpropagation convergence via nonmonotone perturbed minimization

Abstract: A general convergence theorem is proposed for a family of serial and parallel nonmonotone unconstrained minimization methods with perturbations. A principal application of the theorem is to establish convergence of backpropagation (BP), the classical algorithm for training artificial neural networks. Under certain natural assumptions, such as divergence of the sum of the learning rates and convergence of the sum of their squares, it is shown that every accumulation point of the BP iterates is a stationary point of the error function associated with the given set of training examples. The results presented cover serial and parallel BP, as well as modified BP with a momentum term.

Back propagation through adjoints for the identification of nonlinear dynamic systems using recurrent neural models

Abstract: In this paper, back propagation is reinvestigated for an efficient evaluation of the gradient in arbitrary interconnections of recurrent subsystems. It is shown that the error has to be back-propagated through the adjoint model of the system and that the gradient can only be obtained after a delay. A faster version, accelerated back propagation, that eliminates this delay, is also developed. Various schemes including the sensitivity method are studied to update the weights of the network using these gradients. Motivated by the Lyapunov approach and the adjoint model, the predictive back propagation and its variant, targeted back propagation, are proposed. A further refinement, predictive back propagation with filtering is then developed, where the states of the model are also updated. The convergence of this scheme is assured. It is shown that it is sufficient to back propagate as many time steps as the order of the system for convergence. As a preamble, convergence of online batch and sample-wise updates in feedforward models is analyzed using the Lyapunov approach. >

Asynchronous two-stage iterative methods

Abstract: Parallel block two-stage iterative methods for the solution of linear systems of algebraic equations are studied. Convergence is shown for monotone matrices and for H-matrices. Two different asynchronous versions of these methods are considered and their convergence investigated.

Evolution Strategies on Noisy Functions: How to Improve Convergence Properties

Abstract: Evolution Strategies are reported to be robust in the presence of noise which in general hinders the optimization process. In this paper we discuss the influence of some of the stratey parameters and strategy variants on the convergence process and discuss measures for improvement of the convergence properties. After having a broad look to the theory for the dynamics of a (1,λ)-ES on a simple quadratic function we numerically investigate the influence of the parent population size and the introduction of recombination. Finally we compare the effects of multiple sampling of the objective function versus the enlargment of the population size for the convergence precision as well as the convergence reliability by the example of the multimodal Rastrigins function.

Adaptive, global, extended Kalman filters for training feedforward neural networks

Abstract: Finding methods for the optimization of weights in feedforward neural networks has become an ongoing developmental process in connectionist research. The current focus on finding new methods for the optimization of weights is mostly the result of the slow and unreliable convergence properties of the gradient descent optimization used in the original back-propagation algorithm. More accurate and computationally expensive second-order gradient methods have displaced earlier first-order gradient optimization of the network connection weights. The global, extended Kalman filter is among the most accurate and computationally expensive of these second-order weight optimization methods. The iterative, second-order nature of the filter results in a large number of calculations for each sweep of the training set. This can increase the training time dramatically when training is conducted with data sets that contain large numbers of training patterns. In this paper an adaptive variant of the global, extended Kalman filter that exhibits substantially improved convergence properties is presented and discussed. The adaptive mechanism permits more rapid convergence of network training by identifying data that contain redundant information and avoiding calculations based on this redundant information.

Convergence properties of backpropagation for neural nets via theory of stochastic gradient methods. Part 1

Abstract: We study here convergence properties of serial and parallel backpropagation algorithm for training of neural nets, as well as its modification with momentum term. It is shown that these algorithms can be put into the general framework of the stochastic gradient methods. This permits to consider from the same positions both stochastic and deterministic rules for the selection of components (training examples) of the error function to minimize at each iteration. We obtained weaker conditions on the stepsize for deterministic case and provide quite general synchronization rule for parallel version.

An Optimal Control Formulation and Related Numerical Methods for a Problem in Shape Reconstruction

Abstract: The main problem considered in this paper is the construction of numerical methods and proofs of their convergence for the problem of "shape from shading." In the first part of the paper, it is assumed that the height function that describes the surface to be reconstructed is known at all local minima (or maxima). These points are a subset of the singular points, which are the brightest points in the image. A pair of optimal control problems are defined that provide representations for the height function. Numerical schemes based on these representations are then constructed. While both schemes lead to the same approximation, one yields a more efficient algorithm, while the other is more convenient in the convergence analysis. The proof of convergence is based on a representation of the approximation to the height as a functional of a controlled Markov chain. In a later part of the paper the assumption that the height must be known at all local minima (or maxima) is dropped. An extension of the algorithm is described that is capable of reconstruction without this information. Numerical experiments for both algorithms on synthetic and real data are included.

Approximation of scalar waves in the space-frequency domain

Abstract: A numerical method for approximating a pseudodifferential system describing attenuated, scalar waves is introduced and analyzed. Analytic properties of the solutions of the pseudodifferential systems are determined and used to show convergence of the numerical method. Experiments using the method are reported.

Large deviations of semimartingales via convergence of the predictable characteristics

Abstract: We establish conditions for the large deviation principle in the Skorohod topology to hold for a sequence of semimartingales in terms of the convergence of their predictable characteristics. We also develop “a theory of weak convergence for rate functions”

Convergence of subdivision and degree elevation

Abstract: This paper presents a short, simple, and general proof showing that the control polygons generated by subdivision and degree elevation converge to the underlying splines, box-splines, or multivariate Bezier polynomials, respectively. The proof is based only on a Taylor expansion. Then the results are carried over to rational curves and surfaces. Finally, an even shorter but as simple proof is presented for the fact that subdivided Bezier polygons converge to the corresponding curve.