scispace - formally typeset
Search or ask a question

Showing papers on "Rate of convergence published in 1995"


Journal ArticleDOI
TL;DR: In this paper, the wavelet-vaguelette decomposition (WVD) is used to solve ill-posed linear inverse problems by nonlinearly shrinking the WVD coefficients of the noisy, indirect data.

684 citations


Journal ArticleDOI
TL;DR: In this paper, the authors proved that the Landweber iteration is a stable method for solving nonlinear ill-posed problems and proposed a stopping rule that yields the convergence rate of O(O( √ Δ ) under appropriate conditions.
Abstract: In this paper we prove that the Landweber iteration is a stable method for solving nonlinear ill-posed problems. For perturbed data with noise level \(\delta \) we propose a stopping rule that yields the convergence rate\(O (\delta ^{1/2}\) ) under appropriate conditions. We illustrate these conditions for a few examples.

589 citations


Journal ArticleDOI
TL;DR: The approach is to use a modular network architecture, reducing a K-class problem to a set of K two-class problems, with a separately trained network for each of the simpler problems.
Abstract: The rate of convergence of net output error is very low when training feedforward neural networks for multiclass problems using the backpropagation algorithm. While backpropagation will reduce the Euclidean distance between the actual and desired output vectors, the differences between some of the components of these vectors increase in the first iteration. Furthermore, the magnitudes of subsequent weight changes in each iteration are very small, so that many iterations are required to compensate for the increased error in some components in the initial iterations. Our approach is to use a modular network architecture, reducing a K-class problem to a set of K two-class problems, with a separately trained network for each of the simpler problems. Speedups of one order of magnitude have been obtained experimentally, and in some cases convergence was possible using the modular approach but not using a nonmodular network. >

407 citations


Journal ArticleDOI
TL;DR: This EM gradient algorithm approximately solves the M-step of the EM algorithm by one iteration of Newton's method, and the proof of global convergence applies and improves existing theory for the EM algorithms.
Abstract: In many problems of maximum likelihood estimation, it is impossible to carry out either the E-step or the M-step of the EM algorithm. The present paper introduces a gradient algorithm that is closely related to the EM algorithm. This EM gradient algorithm approximately solves the M-step of the EM algorithm by one iteration of Newton's method. Since Newton's method converges quickly, the local properties of the EM gradient algorithm are almost identical with those of the EM algorithm. Any strict local maximum point of the observed likelihood locally attracts the EM and EM gradient algorithm at the same rate of convergence, and near the maximum point the EM gradient algorithm always produces an increase in the likelihood. With proper modification the EM gradient algorithm also exhibits global convergence properties that are similar to those of the EM algorithm. Our proof of global convergence applies and improves existing theory for the EM algorithm. These theoretical points are reinforced by a discussion of three realistic examples illustrating how the EM gradient algorithm can succeed where the EM algorithm is intractable

371 citations


Journal ArticleDOI
TL;DR: Preliminary numerical testing of the algorithms on simulated data suggest that the convex algorithm and the ad hoc gradient algorithm are computationally superior to the EM algorithm.
Abstract: This paper reviews and compares three maximum likelihood algorithms for transmission tomography. One of these algorithms is the EM algorithm, one is based on a convexity argument devised by De Pierro (see IEEE Trans. Med. Imaging, vol.12, p.328-333, 1993) in the context of emission tomography, and one is an ad hoc gradient algorithm. The algorithms enjoy desirable local and global convergence properties and combine gracefully with Bayesian smoothing priors. Preliminary numerical testing of the algorithms on simulated data suggest that the convex algorithm and the ad hoc gradient algorithm are computationally superior to the EM algorithm. This superiority stems from the larger number of exponentiations required by the EM algorithm. The convex and gradient algorithms are well adapted to parallel computing. >

368 citations


Journal ArticleDOI
TL;DR: This paper presents space-alternating generalized EM (SAGE) algorithms for image reconstruction, which update the parameters sequentially using a sequence of small "hidden" data spaces, rather than simultaneously using one large complete-data space.
Abstract: Most expectation-maximization (EM) type algorithms for penalized maximum-likelihood image reconstruction converge slowly, particularly when one incorporates additive background effects such as scatter, random coincidences, dark current, or cosmic radiation. In addition, regularizing smoothness penalties (or priors) introduce parameter coupling, rendering intractable the M-steps of most EM-type algorithms. This paper presents space-alternating generalized EM (SAGE) algorithms for image reconstruction, which update the parameters sequentially using a sequence of small "hidden" data spaces, rather than simultaneously using one large complete-data space. The sequential update decouples the M-step, so the maximization can typically be performed analytically. We introduce new hidden-data spaces that are less informative than the conventional complete-data space for Poisson data and that yield significant improvements in convergence rate. This acceleration is due to statistical considerations, not numerical overrelaxation methods, so monotonic increases in the objective function are guaranteed. We provide a general global convergence proof for SAGE methods with nonnegativity constraints. >

308 citations


Journal ArticleDOI
TL;DR: This work provides a solid foundation for performance analysis either by analytical methods or by simulation of open multiclass queueing networks, which are common models of communication networks, and complex manufacturing systems such as wafer fabrication facilities.
Abstract: The subject of this paper is open multiclass queueing networks, which are common models of communication networks, and complex manufacturing systems such as wafer fabrication facilities. We provide sufficient conditions for the existence of bounds on long-run average moments of the queue lengths at the various stations, and we bound the rate of convergence of the mean queue length to its steady-state value. Our work provides a solid foundation for performance analysis either by analytical methods or by simulation. These results are applied to several examples including re-entrant lines, generalized Jackson networks, and a general polling model as found in computer networks applications. >

290 citations


Journal ArticleDOI
TL;DR: The adaptive observers presented in this note guarantee arbitrarily fast exponential convergence both of parameter and state estimates to actual parameters and states, while previous adaptive observers guarantee only exponential (not arbitrarily fast) convergence.
Abstract: Concerns the same class of linearly parameterized single-output nonlinear systems that the authors previously identified in (1992) in terms of differential geometric conditions. When persistency of excitation conditions are satisfied, the adaptive observers presented in this note guarantee arbitrarily fast exponential convergence both of parameter and state estimates to actual parameters and states, while previous adaptive observers guarantee only exponential (not arbitrarily fast) convergence. This extends earlier results for linear systems. >

284 citations


Journal ArticleDOI
TL;DR: It is shown that the EM algorithm can be regarded as a variable metric algorithm with its searching direction having a positive projection on the gradient of the log likelihood and an acceleration technique that yields a significant speedup in simulation experiments.

278 citations


Journal ArticleDOI
TL;DR: In this paper, it is shown that if the regressor vector is constructed out of radial basis function approximants, it will be persistently exciting, provided a kind of "ergodic" condition is satisfied.
Abstract: In this paper, identification algorithms whose convergence and rate of convergence hinge on the regressor vector being persistently exciting are discussed. It is then shown that if the regressor vector is constructed out of radial basis function approximants, it will be persistently exciting, provided a kind of "ergodic" condition is satisfied. In addition, bounds on parameters associated with the persistently exciting regressor vector are provided; these parameters are connected with both the convergence and rates of convergence of the algorithms involved.

274 citations


Journal ArticleDOI
TL;DR: This paper presents an analysis of the filtered-X LMS algorithm using stochastic methods and some derived bounds and predicted dynamic behavior are found to correspond very well to simulation results.
Abstract: The presence of a transfer function in the auxiliary-path following the adaptive filter and/or in the error-path, as in the case of active noise control, has been shown to generally degrade the performance of the LMS algorithm. Thus, the convergence rate is lowered, the residual power is increased, and the algorithm can even become unstable. To ensure convergence of the algorithm, the input to the error correlator has to be filtered by a copy of the auxiliary-error-path transfer function. This paper presents an analysis of the filtered-X LMS algorithm using stochastic methods. The influence of off-line and on-line estimation of the error-path filter on the algorithm is also investigated. Some derived bounds and predicted dynamic behavior are found to correspond very well to simulation results.

Journal ArticleDOI
TL;DR: A modification of the abstract convergence theory of the additive and multiplicative Schwarz methods that makes the relation to traditional iteration methods more explicit, making convergence proofs of multilevel and domain decomposition methods clearer, or, at least, more classical.
Abstract: In recent years, it has been shown that many modern iterative algorithms (multigrid schemes, multilevel preconditioners, domain decomposition methods etc.) for solving problems resulting from the discretization of PDEs can be interpreted as additive (Jacobi-like) or multiplicative (Gauss-Seidel-like) subspace correction methods. The key to their analysis is the study of certain metric properties of the underlying splitting of the discretization space \(V\) into a sum of subspaces \(V_j\) and the splitting of the variational problem on \(V\) into auxiliary problems on these subspaces. In this paper, we propose a modification of the abstract convergence theory of the additive and multiplicative Schwarz methods, that makes the relation to traditional iteration methods more explicit. The analysis of the additive and multiplicative Schwarz iterations can be carried out in almost the same spirit as in the traditional block-matrix situation, making convergence proofs of multilevel and domain decomposition methods clearer, or, at least, more classical. In addition, we present a new bound for the convergence rate of the appropriately scaled multiplicative Schwarz method directly in terms of the condition number of the corresponding additive Schwarz operator. These results may be viewed as an appendix to the recent surveys [X], [Ys].

Journal ArticleDOI
TL;DR: New linear convergence results for iterative methods for solving the variational inequality problem are presented, including the extragradient method, the proximal point method, a matrix splitting method and a certain feasible descent method.

Journal ArticleDOI
TL;DR: In this article, the authors established the best possible rate of convergence for estimating the mixing distribution in finite mixture models and showed that the key for estimating mixing distribution is the knowledge of the number of components in the mixture.
Abstract: In finite mixture models, we establish the best possible rate of convergence for estimating the mixing distribution. We find that the key for estimating the mixing distribution is the knowledge of the number of components in the mixture. While a $\sqrt n$-consistent rate is achievable when the exact number of components is known, the best possible rate is only $n^{-1/4}$ when it is unknown. Under a strong identifiability condition, it is shown that this rate is reached by some minimum distance estimators. Most commonly used models are found to satisfy the strong identifiability condition.

Journal ArticleDOI
TL;DR: A new learning procedure is presented which is based on a linearization of the nonlinear processing elements and the optimization of the multilayer perceptron layer by layer, which yields results in both accuracy and convergence rates which are orders of magnitude superior compared to conventional backpropagation learning.
Abstract: Multilayer perceptrons are successfully used in an increasing number of nonlinear signal processing applications. The backpropagation learning algorithm, or variations hereof, is the standard method applied to the nonlinear optimization problem of adjusting the weights in the network in order to minimize a given cost function. However, backpropagation as a steepest descent approach is too slow for many applications. In this paper a new learning procedure is presented which is based on a linearization of the nonlinear processing elements and the optimization of the multilayer perceptron layer by layer. In order to limit the introduced linearization error a penalty term is added to the cost function. The new learning algorithm is applied to the problem of nonlinear prediction of chaotic time series. The proposed algorithm yields results in both accuracy and convergence rates which are orders of magnitude superior compared to conventional backpropagation learning. >

Journal ArticleDOI
TL;DR: In this article, the Gibbs sampling scheme converges geometrically in terms of Pearson χ 2 -distance for both systematic and random scans under conditions that guarantee the compactness of the Markov forward operator and irreducibility of the corresponding chain.
Abstract: This paper presents results on covariance structure and convergence for the Gibbs sampler with both systematic and random scans. It is shown that, under conditions that guarantee the compactness of the Markov forward operator and irreducibility of the corresponding chain, the Gibbs sampling scheme converges geometrically in terms of Pearson χ 2 -distance. In particular, for the random scan, the autocovariance can be expressed as variances of iterative conditional expectations. As a consequence, the autocorrelations are all positive and decrease monotonically

Proceedings ArticleDOI
13 Dec 1995
TL;DR: In this article, an indicator process is formulated and its properties are examined, and it is shown that ordinal comparison converges monotonically in the case of averaging normal random variables.
Abstract: Recent research has demonstrated that ordinal comparison converges fast despite possible presence of large estimation noise in the design of discrete event dynamic systems. In this paper, we address a fundamental problem of characterizing the convergence of ordinal comparison. To achieve the goal, an indicator process is formulated and its properties are examined. For several performance measures frequently used in simulation, rate of convergence for the indicator process is proven to be exponential for regenerative simulations. Therefore, the fast convergence of ordinal comparison is supported and explained in a rigorous framework. Many performance measures of averaging type have asymptotic normal distributions. The results of this paper show that ordinal comparison converges monotonically in the case of averaging normal random variables. Such monotonicity is useful in simulation planning.

Journal ArticleDOI
TL;DR: In this article, the authors studied the simulation of one-dimensional reflected (or regulated) Brownian motion and showed that the discretization error associated with the Euler scheme for simulation of such a process has both a strong and weak order of convergence of precisely 1/2.
Abstract: This paper is concerned with various aspects of the simulation of one-dimensional reflected (or regulated) Brownian motion. The main result shows that the discretization error associated with the Euler scheme for simulation of such a process has both a strong and weak order of convergence of precisely 1/2. This contrasts with the faster order 1 achievable for simulations of SDE's without reflecting boundaries. The asymptotic distribution of the discretization error is described using Williams' decomposition of a Brownian path at the time of a minimum. Improved methods for simulation of reflected Brownian motion are discussed.

Journal ArticleDOI
TL;DR: This paper is essentially expository but it presents however new results concerning some problems of rate of convergence in Finance theory.
Abstract: We present several results and methods concerning the convergence of numerical schemes for problems arising in Finance theory. This paper is essentially expository but we present however new results concerning some problems of rate of convergence.

Journal ArticleDOI
Samer S. Saab1
TL;DR: It is proved that the same condition is sufficient for the global robustness of the proposed learning algorithm to state disturbances, measurement noise at the output, and reinitialization error are present at each iteration.
Abstract: A discretized version of the D-type learning control algorithm is presented for a MIMO linear discrete-time system. A necessary and sufficient condition for uniform convergence of the proposed learning algorithm is presented. Then, we prove that the same condition is sufficient for the global robustness of the proposed learning algorithm to state disturbances, measurement noise at the output, and reinitialization error are present at each iteration. A numerical example is given to illustrate the results. >

Journal ArticleDOI
TL;DR: A global convergence result is established, with a quadratic rate under the regularity assumption, for the minimization of the convex composite optimization functionh ο F.
Abstract: An extension of the Gauss—Newton method for nonlinear equations to convex composite optimization is described and analyzed. Local quadratic convergence is established for the minimization ofh ο F under two conditions, namelyh has a set of weak sharp minima,C, and there is a regular point of the inclusionF(x) ∈ C. This result extends a similar convergence result due to Womersley (this journal, 1985) which employs the assumption of a strongly unique solution of the composite functionh ο F. A backtracking line-search is proposed as a globalization strategy. For this algorithm, a global convergence result is established, with a quadratic rate under the regularity assumption.

Journal ArticleDOI
TL;DR: In this paper, a globally convergent computational scheme is established to approximate a topological multivortex solution in the recently discovered self-dual Chern-Simons theory in R 2.
Abstract: In this paper a globally convergent computational scheme is established to approximate a topological multivortex solution in the recently discovered self-dual Chern-Simons theory in R 2 . Our method which is constructive and numerically efficient finds the most superconducting solution in the sense that its Higgs field has the largest possible magnitude. The method consists of two steps: first one obtains by a convergent monotone iterative algorithm a suitable solution of the bounded domain equations and then one takes the large domain limit and approximates the full piane solutions. It is shown that with a special choice of the initial guess function, the approximation sequence approaches exponentially fast a solution in R 2 . The convergence rate implies that the truncation errors away from local regions are insignificant.

Posted Content
TL;DR: In this article, the authors define new binomial models, where the calculated option prices converge smoothly to the Black-Scholes solution and remarkably, they even achieve order of convergence two with much smaller initial error.
Abstract: Binomial models, which rebuild the continuous setup in the limit, serve for approximative valuation of options, especially where formulas cannot be derived mathematically. Even with the valuation of European call options distorting irregularities occur. For this case, sources of convergence patterns are explained. Furthermore, it is proved order of convergence one for the Cox--Ross--Rubinstein[79] model as well as for the tree parameter selections of Jarrow and Rudd[83], and Tian[93]. Then, we define new binomial models, where the calculated option prices converge smoothly to the Black--Scholes solution and remarkably, we even achieve order of convergence two with much smaller initial error. Notably, solely the formulas to determine the constant up- and down- factors change. Finally, all tree approaches are compared with respect to speed and accuracy calculating relative root--mean--squared error of approximative option values for a sample of randomly selected parameters across a set of refinements. Approximation of American type options with the new models exhibits order of convergence one but smaller initial error than previously existing binomial models.

Journal ArticleDOI
TL;DR: Rates of convergence for nearest neighbor estimation are established in a general framework in terms of metric covering numbers of the underlying space and a consistency result is established for k/sub n/-nearest neighbor estimation under arbitrary sampling and a convergence rate matching established rates for i.i.d. sampling is established.
Abstract: Rates of convergence for nearest neighbor estimation are established in a general framework in terms of metric covering numbers of the underlying space. The first result is to find explicit finite sample upper bounds for the classical independent and identically distributed (i.i.d.) random sampling problem in a separable metric space setting. The convergence rate is a function of the covering numbers of the support of the distribution. For example, for bounded subsets of R/sup r/, the convergence rate is O(1/n/sup 2/r/). The main result is to extend the problem to allow samples drawn from a completely arbitrary random process in a separable metric space and to examine the performance in terms of the individual sample sequences. The authors show that for every sequence of samples the asymptotic time-average of nearest neighbor risks equals twice the time-average of the conditional Bayes risks of the sequence. Finite sample upper bounds under arbitrary sampling are again obtained in terms of the covering numbers of the underlying space. In particular, for bounded subsets of R/sup r/ the convergence rate of the time-averaged risk is O(1/n/sup 2/r/). The authors then establish a consistency result for k/sub n/-nearest neighbor estimation under arbitrary sampling and prove a convergence rate matching established rates for i.i.d. sampling. Finally, they show how their arbitrary sampling results lead to some classical i.i.d. sampling results and in fact extend them to stationary sampling. The framework and results are quite general while the proof techniques are surprisingly elementary. >

Book
31 Jul 1995
TL;DR: In this paper, the convergence rate of Fourier series and best approximations in the spaces Lp and Lp are presented. But they do not consider the problem of approximating functions and their derivatives by Fourier sums.
Abstract: Preface. Introduction. 1. Classes of periodic functions. 2. Integral representations of deviations of linear means of Fourier series. 3. Approximations by Fourier sums in the spaces c and L1. 4. Simultaneous approximation of functions and their derivatives by Fourier sums. 5. Convergence rate of Fourier series and best approximations in the spaces Lp. 6. Best approximations in the spaces C and L. Bibliographical notes. References. Index.

Journal ArticleDOI
TL;DR: The superlinear or quadratic convergence rate of a Newton-like algorithm is proved and it is shown that it is superlinearly convergent and a characterization of superlinear convergence extending the result of Boggs, Tolle and Wang is given.
Abstract: In this paper, some Newton and quasi-Newton algorithms for the solution of inequality constrained minimization problems are considered. All the algorithms described produce sequences {x k } convergingq-superlinearly to the solution. Furthermore, under mild assumptions, aq-quadratic convergence rate inx is also attained. Other features of these algorithms are that only the solution of linear systems of equations is required at each iteration and that the strict complementarity assumption is never invoked. First, the superlinear or quadratic convergence rate of a Newton-like algorithm is proved. Then, a simpler version of this algorithm is studied, and it is shown that it is superlinearly convergent. Finally, quasi-Newton versions of the previous algorithms are considered and, provided the sequence defined by the algorithms converges, a characterization of superlinear convergence extending the result of Boggs, Tolle, and Wang is given.

Journal ArticleDOI
TL;DR: The authors present a comprehensive analysis of the performance of this new frequency-domain LMS adaptive scheme, the generalized multidelay filter (GMDF), and provide insight into the influence of impulse response segmentation on the behavior of the adaptive algorithm.
Abstract: Frequency-domain adaptive filters have long been recognized as an attractive alternative to time-domain algorithms when dealing with systems with large impulse response and/or correlated input. New frequency-domain LMS adaptive schemes have been proposed. These algorithms essentially retain the attractive features of frequency-domain implementations, while requiring a processing delay considerably smaller than the length of the impulse response. The authors show that these algorithms can be seen as particular implementations of a more general scheme, the generalized multidelay filter (GMDF). Within this general class of algorithms, we focus on implementations based on the weighted overlap and add reconstruction algorithms; these variants, overlooked in previous contributions, provide an independent control of the overall processing delay and of the rate of update of the filter coefficients, allowing a trade-off between the computational complexity and the rate of convergence. We present a comprehensive analysis of the performance of this new scheme and to provide insight into the influence of impulse response segmentation on the behavior of the adaptive algorithm. Exact analytical expressions for the steady-state mean-square error are first derived. Necessary and sufficient conditions for the convergence of the algorithm to the optimal solution within finite variance are then obtained, and are translated into bounds for the stepsize parameter. Simulations are presented to support our analysis and to demonstrate the practical usefulness of the GMDF algorithm in applications where large impulse response has to be processed. >

Journal ArticleDOI
TL;DR: Several long-standing problems concerning the basic properties of LS-based STR, such as stability, optimality, consistency and the best convergence rate, are solved within a unified framework.

Journal ArticleDOI
TL;DR: In this paper, a multi-grid method for a periodic heterogeneous medium in 1-D is presented based on homogenization theory, special integrid transfer operators have been developed to simulate a low frequency response of the differential equations with oscillatory coefficients.

Journal ArticleDOI
TL;DR: It is proved that the rate of convergence of the second method is optimal uniformly in the number of variables and the approximate Hessian strategy significantly improves the total arithmetical complexity of the method.
Abstract: In this paper we establish the efficiency estimates for two cutting plane methods based on the analytic barrier We prove that the rate of convergence of the second method is optimal uniformly in the number of variables We present a modification of the second method In this modified version each test point satisfies an approximate centering condition We also use the standard strategy for updating approximate Hessians of the logarithmic barrier function We prove that the rate of convergence of the modified scheme remains optimal and demonstrate that the number of Newton steps in the auxiliary minimization processes is bounded by an absolute constant We also show that the approximate Hessian strategy significantly improves the total arithmetical complexity of the method