Showing papers on "Rate of convergence published in 1994"
TL;DR: In this article, an element-free Galerkin method which is applicable to arbitrary shapes but requires only nodal data is applied to elasticity and heat conduction problems, where moving least-squares interpolants are used to construct the trial and test functions for the variational principle.
Abstract: An element-free Galerkin method which is applicable to arbitrary shapes but requires only nodal data is applied to elasticity and heat conduction problems. In this method, moving least-squares interpolants are used to construct the trial and test functions for the variational principle (weak form); the dependent variable and its gradient are continuous in the entire domain. In contrast to an earlier formulation by Nayroles and coworkers, certain key differences are introduced in the implementation to increase its accuracy. The numerical examples in this paper show that with these modifications, the method does not exhibit any volumetric locking, the rate of convergence can exceed that of finite elements significantly and a high resolution of localized steep gradients can be achieved. The moving least-squares interpolants and the choices of the weight function are also discussed in this paper.
5,324 citations
TL;DR: ECME as discussed by the authors is a generalization of the ECM algorithm, which is itself an extension of the EM algorithm (Dempster, Laird & Rubin, 1977), which can be obtained by replacing some CM-steps of ECM, which maximise the constrained expected complete-data loglikelihood function, with steps that maximize the correspondingly constrained actual likelihood function.
Abstract: A generalisation of the ECM algorithm (Meng & Rubin, 1993), which is itselfan extension of the EM algorithm (Dempster, Laird & Rubin, 1977), can be obtained by replacing some CM-steps of ECM, which maximise the constrained expected complete-data loglikelihood function, with steps that maximise the correspondingly constrained actual likelihood function. This algorithm, which we call ECME algorithm, for Expectation /Conditional Maximisation Either, shares with both EM and ECM their stable monotone convergence and basic simplicity of implementation relative to competing faster converging methods. Moreover, ECME can have a substantially faster convergence rate than either EM or ECM, measured using either the number of iterations or actual computer time
604 citations
TL;DR: It is shown that if this computation is replaced by an approximate solution produced by an arbitrary iterative method, then with relatively modest requirements on the accuracy of the approximate solution, the resulting inexact Uzawa algorithm is convergent, with a convergence rate close to that of the exact algorithm.
Abstract: Variants of the Uzawa algorithm for solving symmetric indefinite linear systems are developed and analyzed. Each step of this algorithm requires the solution of a symmetric positive- definite system of linear equations. It is shown that if this computation is replaced by an approximate solution produced by an arbitrary iterative method, then with relatively modest requirements on the accuracy of the approximate solution, the resulting inexact Uzawa algorithm is convergent, with a convergence rate close to that of the exact algorithm. In addition, it is shown that preconditioning can be used to improve performance. The analysis is illustrated and supplemented using several examples derived from mixed finite element discretization of the Stokes equations.
487 citations
TL;DR: The problem of the synthesis of a feedback control, assuring that the system state is ultimately bounded within a given compact set containing the origin with an assigned rate of convergence, is investigated and it is shown that such a function may be derived by numerically efficient algorithms involving polyhedral sets.
Abstract: In this note, linear discrete-time systems affected by both parameter and input uncertainties are considered. The problem of the synthesis of a feedback control, assuring that the system state is ultimately bounded within a given compact set containing the origin with an assigned rate of convergence, is investigated. It is shown that the problem has a solution if and only if there exists a certain Lyapunov function which does not belong to a preassigned class of functions (e.g., the quadratic ones), but it is determined by the target set in which ultimate boundedness is desired. One of the advantages of this approach is that we may handle systems with control constraints. No matching assumptions are made. For systems with linearly constrained uncertainties, it is shown that such a function may be derived by numerically efficient algorithms involving polyhedral sets. The resulting compensator may be implemented as a linear variable-structure control. >
431 citations
TL;DR: This paper presents a decomposition method for solving convex minimization problems that preserves the good features of the proximal method of multipliers, with the additional advantage that it leads to a decoupling of the constraints, and is thus suitable for parallel implementation.
Abstract: This paper presents a decomposition method for solving convex minimization problems. At each iteration, the algorithm computes two proximal steps in the dual variables and one proximal step in the primal variables. We derive this algorithm from Rockafellar's proximal method of multipliers, which involves an augmented Lagrangian with an additional quadratic proximal term. The algorithm preserves the good features of the proximal method of multipliers, with the additional advantage that it leads to a decoupling of the constraints, and is thus suitable for parallel implementation. We allow for computing approximately the proximal minimization steps and we prove that under mild assumptions on the problem's data, the method is globally convergent and at a linear rate. The method is compared with alternating direction type methods and applied to the particular case of minimizing a convex function over a finite intersection of closed convex sets.
420 citations
TL;DR: In this paper, the convergence rate of the iterative solution of the Stokes problem is derived for a general class of block preconditioners, where the partitioning into blocks corresponds to the natural partitioning of the velocity and pressure variables.
Abstract: Mixed finite element approximation of the classical Stokes problem describing slow viscous incompressible flow gives rise to symmetric indefinite systems for the discrete velocity and pressure variables. Iterative solution of such indefinite systems is feasible and is an attractive approach for large problems. Part I of this work described a conjugate gradient-like method (the method of preconditioned conjugate residuals) which is applicable to symmetric indefinite problems [A. J. Wathen and D. J. Silvester, SIAM J. Numer. Anal., 30 (1993), pp. 630–649]. Using simple arguments, estimates for the eigenvalue distribution of the discrete Stokes operator on which the convergence rate of the iteration depends are easily derived. Part I discussed the important case of diagonal preconditioning (scaling). This paper considers the more general class of block preconditioners, where the partitioning into blocks corresponds to the natural partitioning into the velocity and pressure variables. It is shown that, provid...
399 citations
TL;DR: It is proved that the convergence of the simulated annealing procedure when the decision to change the current configuration is blind of the cost of the new configuration, including that of Metropolis, is proved.
Abstract: We prove the convergence of the simulated annealing procedure when the decision to change the current configuration is blind of the cost of the new configuration. In case of filtering binary images, the proof easily generalizes to other procedures, including that of Metropolis. We show that a function Q associated with the algorithm must be chosen as large as possible to provide a fast rate of convergence. The worst case (Q constant) is associated with the "blind" algorithm. On the other hand, an appropriate Q taking sufficiently high values yields a better rate of convergence than that of Metropolis procedure. >
383 citations
TL;DR: In this article, the authors extended the classical empirical process theory for Vapnik-Cervonenkis classes to stationary sequences of dependent variables and established a uniform convergence rate of O(n −s/(1+s) ) for V-C classes.
Abstract: Classical empirical process theory for Vapnik-Cervonenkis classes deals mainly with sequences of independent variables. This paper extends the theory to stationary sequences of dependent variables. It establishes rates of convergence for $\beta$-mixing and $\phi$-mixing empirical processes indexed by classes of functions. The method of proof depends on a coupling of the dependent sequence with sequences of independent blocks, to which the classical theory can be applied. A uniform $O(n^{-s/(1+s)})$ rate of convergence over V-C classes is established for sequences whose mixing coefficients decay slightly faster than $O(n^{-s})$.
363 citations
TL;DR: In this paper, the convergence rate of a sieve estimate is governed by (a) the local expected values, variances and $L_2$ entropy of the criterion differences and (b) the approximation error of the sieve.
Abstract: In this paper, we develop a general theory for the convergence rate of sieve estimates, maximum likelihood estimates (MLE's) and related estimates obtained by optimizing certain empirical criteria in general parameter spaces. In many cases, especially when the parameter space is infinite dimensional, maximization over the whole parameter space is undesirable. In such cases, one has to perform maximization over an approximating space (sieve) of the original parameter space and allow the size of the approximating space to grow as the sample size increases. This method is called the method of sieves. In the case of the maximum likelihood estimation, an MLE based on a sieve is called a sieve MLE. We found that the convergence rate of a sieve estimate is governed by (a) the local expected values, variances and $L_2$ entropy of the criterion differences and (b) the approximation error of the sieve. A robust nonparametric regression problem, a mixture problem and a nonparametric regression problem are discussed as illustrations of the theory. We also found that when the underlying space is too large, the estimate based on optimizing over the whole parameter space may not achieve the best possible rates of convergence, whereas the sieve estimate typically does not suffer from this difficulty.
361 citations
TL;DR: In this paper, the authors compared the performance of the Picard and Newton iterative methods in one-, two-, and three-dimensional finite element simulations involving both steady state and transient flow, and concluded that the Picard or relaxed Picard schemes are often adequate for solving Richards' equation, but in cases where these fail to converge or converge slowly, the Newton method should be used.
Abstract: Picard iteration is a widely used procedure for solving the nonlinear equation governing flow in variably saturated porous media. The method is simple to code and computationally cheap, but has been known to fail or converge slowly. The Newton method is more complex and expensive (on a per-iteration basis) than Picard, and as such has not received very much attention. Its robustness and higher rate of convergence, however, make it an attractive alternative to the Picard method, particularly for strongly nonlinear problems. In this paper the Picard and Newton schemes are implemented and compared in one-, two-, and three-dimensional finite element simulations involving both steady state and transient flow. The eight test cases presented highlight different aspects of the performance of the two iterative methods and the different factors that can affect their convergence and efficiency, including problem size, spatial and temporal discretization, initial solution estimates, convergence error norm, mass lumping, time weighting, conductivity and moisture content characteristics, boundary conditions, seepage faces, and the extent of fully saturated zones in the soil. Previous strategies for enhancing the performance of the Picard and Newton schemes are revisited, and new ones are suggested. The strategies include chord slope approximations for the derivatives of the characteristic equations, relaxing convergence requirements along seepage faces, dynamic time step control, nonlinear relaxation, and a mixed Picard-Newton approach. The tests show that the Picard or relaxed Picard schemes are often adequate for solving Richards' equation, but that in cases where these fail to converge or converge slowly, the Newton method should be used. The mixed Picard-Newton approach can effectively overcome the Newton scheme's sensitivity to initial solution estimates, while comparatively poor performance is reported for the various chord slope approximations. Finally, given the reliability and efficiency of current conjugate gradient-like methods for solving linear nonsymmetric systems, the only real drawback of using Newton rather than Picard iteration is the algebraic complexity and computational cost of assembling the derivative terms of the Jacobian matrix, and it is suggested that both methods can be effectively implemented and used in numerical models of Richards' equation.
318 citations
TL;DR: A robust BP learning algorithm is derived that is resistant to the noise effects and is capable of rejecting gross errors during the approximation process, and its rate of convergence is improved since the influence of incorrect samples is gracefully suppressed.
Abstract: The backpropagation (BP) algorithm allows multilayer feedforward neural networks to learn input-output mappings from training samples. Due to the nonlinear modeling power of such networks, the learned mapping may interpolate all the training points. When erroneous training data are employed, the learned mapping can oscillate badly between data points. In this paper we derive a robust BP learning algorithm that is resistant to the noise effects and is capable of rejecting gross errors during the approximation process. The spirit of this algorithm comes from the pioneering work in robust statistics by Huber and Hampel. Our work is different from that of M-estimators in two aspects: 1) the shape of the objective function changes with the iteration time; and 2) the parametric form of the functional approximator is a nonlinear cascade of affine transformations. In contrast to the conventional BP algorithm, three advantages of the robust BP algorithm are: 1) it approximates an underlying mapping rather than interpolating training samples; 2) it is robust against gross errors; and 3) its rate of convergence is improved since the influence of incorrect samples is gracefully suppressed. >
01 Jan 1994
TL;DR: Domain decomposition methods provide powerful preconditioners for the iterative solution of the large systems of algebraic equations that arise in finite element or finite difference approximations.
Abstract: Domain decomposition methods provide powerful preconditioners for the iterative solution of the large systems of algebraic equations that arise in finite element or finite difference approximations of partial differential equations. The preconditioners are constructed from exact or approximate solvers for the same partial differential equation restricted to a set of subregions into which the given region has been divided. In addition, the preconditioner is often augmented by a coarse, second-level approximation that provides additional, global exchange of information that can enhance the rate of convergence considerably. The iterative substructuring methods, based on decompositions of the region into nonoverlapping subregions, form one of the main families of such algorithms. Many domain decomposition algorithms can conveniently be described and analyzed as Schwarz methods. These algorithms are fully defined in terms of a set of subspaces and auxiliary bilinear forms. A general theoretical framework has previously been developed. In this paper, these techniques are used in an analysis of iterative substructuring methods for elliptic problems in three dimensions. A special emphasis is placed on the difficult problem of designing good coarse models and obtaining robust methods for which the rate of convergence is insensitive to large variations in the coefficients of the differential equation. Domain decomposition algorithms can conveniently be built from modules that represent local and global components of the preconditioner. In this paper, a number of such possibilities are explored, and it is demonstrated how a great variety of fast algorithms can be designed and analyzed.
TL;DR: In this paper, a condition number estimate for the iterative operator, which is independent of the coefficients, and grows at most as the square of the number of levels, is established.
Abstract: Multilevel Schwarz methods are developed for a conforming finite element approximation of second order elliptic problems. We focus on problems in three dimensions with possibly large jumps in the coefficients across the interface separating the subregions. We establish a condition number estimate for the iterative operator, which is independent of the coefficients, and grows at most as the square of the number of levels. We also characterize a class of distributions of the coefficients, called quasi-monotone, for which the weighted\(L^{2}\) -projection is stable and for which we can use the standard piecewise linear functions as a coarse space. In this case, we obtain optimal methods, i.e. bounds which are independent of the number of levels and subregions. We also design and analyze multilevel methods with new coarse spaces given by simple explicit formulas. We consider nonuniform meshes and conclude by an analysis of multilevel iterative substructuring methods.
TL;DR: Domain decomposition methods provide powerful preconditioners for the iterative solution of the large systems of algebraic equations that arise in finite element or finite difference approximations as discussed by the authors.
Abstract: Domain decomposition methods provide powerful preconditioners for the iterative solution of the large systems of algebraic equations that arise in finite element or finite difference approximations...
TL;DR: This method simultaneously triangularizes by orthogonal equivalences a sequence of matrices associated with a cyclic pencil formulation related to the Euler-Lagrange difference equations to extract a basis for the stable deflating subspace of the extended pencil, from which the Riccati solution is obtained.
Abstract: In this paper we present a method for the computation of the periodic nonnegative definite stabilizing solution of the periodic Riccati equation. This method simultaneously triangularizes by orthogonal equivalences a sequence of matrices associated with a cyclic pencil formulation related to the Euler-Lagrange difference equations. In doing so, it is possible to extract a basis for the stable deflating subspace of the extended pencil, from which the Riccati solution is obtained. This algorithm is an extension of the standard QZ algorithm and retains its attractive features, such as quadratic convergence and small relative backward error. A method to compute the optimal feedback controller gains for linear discrete time periodic systems is dealt with. >
TL;DR: In this article, a new four-node shell element for nonlinear analysis which is useful for explicit time integration with single point quadrature is presented, and an assumed strain method is used to stabilize the zero-energy modes of the element.
TL;DR: In this article, the authors show that the convergence test and the mean-reversion test are not equivalent and provide an empirical example in which the null hypothesis of no mean reversion is rejected but the null hypotheses of no convergence is not rejected.
Abstract: The authors show that, contrary to the beliefs of some previous analysts of international economic growth, the hypotheses of convergence and of mean-reversion are not equivalent. Under some assumptions, the rate of convergence is independent of the degree of mean-reversion; under other assumptions, mean-reversion is a necessary, but not a sufficient, condition for convergence. The authors show the relationship between the convergence test and the mean-reversion test and provide an empirical example in which the null hypothesis of no mean-reversion is rejected but the null hypothesis of no convergence is not rejected. Copyright 1994 by MIT Press.
TL;DR: In this article, a general method for constructing high-order approximation schemes for Hamilton-Jacobi-Bellman equations is given, based on a discrete version of the Dynamic Programming Principle.
Abstract: A general method for constructing high-order approximation schemes for
Hamilton-Jacobi-Bellman equations is given. The method is based on a
discrete version of the Dynamic Programming Principle. We prove a
general convergence result for this class of approximation schemes also
obtaining, under more restrictive assumptions, an estimate in
$L^\infty$
of the order of convergence and of the local truncation error. The
schemes can be applied, in particular, to the stationary linear first
order equation in
${\Bbb R}^n$
. We present several
examples of schemes
belonging to this class and with fast convergence to the solution.
TL;DR: In this article, a condition of semistability is shown to ensure the quadratic convergence of Newton's method and the superlinear convergence of some quasi-Newton algorithms, provided the sequence defined by the algorithm exists and converges.
Abstract: This paper presents some new results in the theory of Newton-type methods for variational inequalities, and their application to nonlinear programming. A condition of semistability is shown to ensure the quadratic convergence of Newton's method and the superlinear convergence of some quasi-Newton algorithms, provided the sequence defined by the algorithm exists and converges. A partial extension of these results to nonsmooth functions is given. The second part of the paper considers some particular variational inequalities with unknowns (x, λ), generalizing optimality systems. Here only the question of superlinear convergence of {x
k
} is considered. Some necessary or sufficient conditions are given. Applied to some quasi-Newton algorithms they allow us to obtain the superlinear convergence of {x
k
}. Application of the previous results to nonlinear programming allows us to strengthen the known results, the main point being a characterization of the superlinear convergence of {x
k
} assuming a weak second-order condition without strict complementarity.
TL;DR: In this paper, the convergence rate of RBF nets with respect to the number of hidden units is investigated and the existence of a consistent estimator for RBF networks is proven constructively.
TL;DR: The genetic algorithm-based cost minimization technique is shown to perform very well in terms of robustness to noise, rate of convergence and quality of the final edge image.
TL;DR: It is shown that fast solvers for discrete elliptic variational inequalities of the first kind (obstacle problems) as resulting from the approximation of related continuous problems by piecewise linear finite elements are derived by using basic ideas of successive subspace correction.
Abstract: We derive fast solvers for discrete elliptic variational inequalities of the first kind (obstacle problems) as resulting from the approximation of related continuous problems by piecewise linear finite elements. Using basic ideas of successive subspace correction, we modify well-known relaxation methods by extending the set of search directions. Extended underrelaxations are called monotone multigrid methods, if they are quasioptimal in a certain sense. By construction, all monotone multigrid methods are globally convergent. We take a closer look at two natural variants, the standard monotone multigrid method and a truncated version. For the considered model problems, the asymptotic convergence rates resulting from the standard approach suffer from insufficient coarse-grid transport, while the truncated monotone multigrid method provides the same efficiency as in the unconstrained case.
TL;DR: Theoretical analysis of the convergence and the stability of fictitious dynamical methods for electrons shows that a particular damped second-order dynamics has a much faster rate of convergence to the ground state than first-order steepest-descent algorithms while retaining their numerical cost per time step.
Abstract: We study the convergence and the stability of fictitious dynamical methods for electrons. first, we show that a particular damped second-order dynamics has a much faster rate of convergence to the ground state than first-order steepest-descent algorithms while retaining their numerical cost per time step. Our damped dynamics has efficiency comparable to that of conjugate gradient methods in typical electronic minimization problems. Then, we analyze the factors that limit the size of the integration time step in approaches based on plane-wave expansions. The maximum allowed time step is dictated by the highest frequency components of the fictitious electronic dynamics. These can result either from the larger wave vector components of the kinetic energy or from the small wave vector components of the Coulomb potential giving rise to the so called charge sloshing problem. We show how to eliminate large wave vector instabilities by adopting a preconditioning scheme in the context of Car-Parrinello ab initio molecular-dynamics simulations of the ionic motion. We also show how to solve the charge sloshing problem when this is present. We substantiate our theoretical analysis with numerical tests on a number of different silicon and carbon systems having both insulating and metallic character.
TL;DR: A general method to quickly determine convergence rate and fault-tolerance for any member of a broad family of convergent voting algorithms is presented, developed under a realistic mixed-mode fault model comprised of asymmetric, symmetric, and benign fault modes.
Abstract: In a fault-tolerant distributed system, different non-faulty processes may arrive at different values for a given system parameter. To resolve this disagreement, processes must exchange and vote upon their respective local values. Faulty processes may attempt to inhibit agreement by acting in a malicious or "Byzantine" manner. Approximate agreement defines one form of agreement in which the voted values obtained by the non-faulty processes need not be identical. Instead, they need only agree to within a predefined tolerance. Approximate agreement can be achieved by a sequence of convergent voting rounds, in which the range of values held by non-faulty processes is reduced in each round. Historically, each new convergent voting algorithm has been accompanied by ad-hoc proofs of its convergence rate and fault-tolerance, using an overly conservative fault model in which all faults exhibit worst-case Byzantine behavior. This paper presents a general method to quickly determine convergence rate and fault-tolerance for any member of a broad family of convergent voting algorithms. This method is developed under a realistic mixed-mode fault model comprised of asymmetric, symmetric, and benign fault modes. These results are employed to more accurately analyze the properties of several existing voting algorithms, to derive a sub-family of optimal mixed-mode voting algorithms, and to quickly determine the properties of proposed new voting algorithms. >
TL;DR: Some new techniques, based on error-reducing transformations of the integrand, are described that have been shown to be useful both in estimating high-dimensional integrals and in solving integral equations.
Abstract: Much of the recent work dealing with quasi-random methods has been aimed at establishing the best possible asymptotic rates of convergence to zero of the error resulting when a finite-dimensional integral is replaced by a finite sum of integrand values. In contrast with this perspective to concentrate on asymptotic convergence rates, this paper emphasizes quasi-random methods that are effective for all sample sizes. Throughout the paper, the problem of estimating finite-dimensional integrals is used to illustrate the major ideas, although much of what is done applies equally to the problem of solving certain Fredholm integral equations. Some new techniques, based on error-reducing transformations of the integrand, are described that have been shown to be useful both in estimating high-dimensional integrals and in solving integral equations. These techniques illustrate the utility of carrying over to the quasi-Monte Carlo method certain devices that have proven to be very valuable in statistical (pseudoran...
TL;DR: The authors show that the iterative algorithm converges if the density of the sampling set exceeds a certain minimum value which naturally increases with the bandwidth of the data, and suggest those for which the convergence rate of the recovery algorithm is maximum or minimum.
Abstract: Analyze the performance of an iterative algorithm, similar to the discrete Papoulis-Gerchberg algorithm, and which can be used to recover missing samples in finite-length records of band-limited data. No assumptions are made regarding the distribution of the missing samples, in contrast with the often studied extrapolation problem, in which the known samples are grouped together. Indeed, it is possible to regard the observed signal as a sampled version of the original one, and to interpret the reconstruction result studied as a sampling result. The authors show that the iterative algorithm converges if the density of the sampling set exceeds a certain minimum value which naturally increases with the bandwidth of the data. They give upper and lower bounds for the error as a function of the number of iterations, together with the signals for which the bounds are attained. Also, they analyze the effect of a relaxation constant present in the algorithm on the spectral radius of the iteration matrix. From this analysis they infer the optimum value of the relaxation constant. They also point out, among all sampling sets with the same density, those for which the convergence rate of the recovery algorithm is maximum or minimum. For low-pass signals it turns out that the best convergence rates result when the distances among the missing samples are a multiple of a certain integer. The worst convergence rates generally occur when the missing samples are contiguous. >
TL;DR: In this article, a modification of kernel density estimation is proposed, where the first step is ordinary kernel estimation of the density and its cdf, and the second step the data are transformed, using this estimated cDF, to an approximate uniform (or normal or other target) distribution.
Abstract: A modification of kernel density estimation is proposed. The first step is ordinary kernel estimation of the density and its cdf. In the second step the data are transformed, using this estimated cdf, to an approximate uniform (or normal or other target) distribution. The density and cdf of the transformed data are then estimated by the kernel method and, by change of variable, converted to new estimates of the density and the cdf of the original data. This process is repeated for a total of $k$ steps for some integer $k$ greater than 1. If the target density is uniform, then the order of the bias is reduced, provided that the density of the observed data is sufficiently smooth. By proper choice of bandwidth, rates of squared-error convergence equal to those of higher-order kernels are attainable. More precisely, $k$ repetitions of the process are equivalent, in terms of rate of convergence, to a $2k$-th-order kernel. This transformation-kernel estimate is always a bona fide density and appears to be more effective at small sample sizes than higher-order kernel estimators, at least for densities with interesting features such as multiple modes. The main theoretical achievement of this paper is the rigorous establishment of rates of convergence under multiple iteration. Simulations using a uniform target distribution suggest that the possibility of improvement over ordinary kernel estimation is of practical significance for samples sizes as low as 100 and can become appreciable for sample sizes around 400.
TL;DR: In this article, conditions for the geometric rate of convergence of the Gibbs sampler algorithm for discrete and continuous parameter spaces are derived, and an illustrative exponential family example is given.
Abstract: SUMMARY The rate of convergence of the Gibbs sampler is discussed. The Gibbs sampler is a Monte Carlo simulation method with extensive application to computational issues in the Bayesian paradigm. Conditions for the geometric rate of convergence of the algorithm for discrete and continuous parameter spaces are derived, and an illustrative exponential family example is given. This paper investigates conditions under which the Gibbs sampler (Gelfand and Smith, 1990; Tanner and Wong, 1987; Geman and Geman, 1984) converges at a geometric rate. The main results appear in Sections 2 and 3, where geometric convergence results are established, with respect to total variation and supremum norms under fairly natural conditions on the underlying distribution. For ease of exposition, we shall concentrate on the two most commonly encountered situations, where the state space is finite or continuous. All our results will establish uniform convergence, a strong form of geometric convergence, under appropriate regularity conditions. Uniform convergence is a useful property in its own right but also happens to be a sufficient condition for certain ergodic central limit theorems. Such results are important for estimation in Markov chain simulation but will not be considered in detail here. Our approach is to apply the theory of Markov chains to the specific Gibbs sampler case. In the finite state space case, uniform ergodicity is automatic. However, the situation is more complicated for continuous state spaces where even well-behaved underlying distributions can give rise to Markov chains which converge slowly, or have unbounded kernels. We give two results in this context, corollary 2 and corollary 3 which establish uniform convergence under different sets of conditions on the underlying density. Finally we apply these results to an example of a Bayesian hierarchical model where regularity conditions for geometric convergence are naturally satisfied. Here the hierarchical structure of the model is crucial in permitting application of corollary 3.
TL;DR: In this article, the authors generalized the EM algorithm to the ECM algorithm, a more flexible and applicable iterative algorithm proposed recently by Meng and Rubin, and showed that intuitions accurate for complete-data iterative algorithms may not be trustworthy in the presence of missing data.
Abstract: The fundamental result on the rate of convergence of the EM algorithm has proven to be theoretically valuable and practically useful. Here, this result is generalized to the ECM algorithm, a more flexible and applicable iterative algorithm proposed recently by Meng and Rubin. Results on the rate of convergence of variations of ECM are also presented. An example is given to show that intuitions accurate for complete-data iterative algorithms may not be trustworthy in the presence of missing data.
TL;DR: In this article, the authors extended the penalized likelihood approach to more general stationary processes and established asymptotic rates of convergence with respect to the spectral density of Gaussian processes.
Abstract: The penalized likelihood approach is not well developed in time series analysis, even though it has been applied successfully in a number of nonparametric function estimation problems. Chow and Grenander proposed a penalized likelihood-type approach to the nonparametric estimation of the spectral density of Gaussian processes. In this article this estimator is extended to more general stationary processes, its practical implementation is developed in some detail, and some asymptotic rates of convergence are established. Its performance is also compared to more widely used alternatives in the field. A computational algorithm involving an iterative least squares, initialized by the log-periodogram, is first developed. Then, motivated by an asymptotic linearization, an estimator of the integrated squared error between the estimated and true log-spectral densities is proposed. From this, a data-dependent procedure for selection of the amount of smoothing is constructed. The methodology is illustrated...