Showing papers on "Rate of convergence published in 2003"

Inferential Theory for Factor Models of Large Dimensions

Abstract: This paper develops an inferential theory for factor models of large dimensions. The principal components estimator is considered because it is easy to compute and is asymptotically equivalent to the maximum likelihood estimator (if normality is assumed). We derive the rate of convergence and the limiting distributions of the estimated factors, factor loadings, and common components. The theory is developed within the framework of large cross sections (N) and a large time dimension (T), to which classical factor analysis does not apply. We show that the estimated common components are asymptotically normal with a convergence rate equal to the minimum of the square roots of N and T. The estimated factors and their loadings are generally normal, although not always so. The convergence rate of the estimated factors and factor loadings can be faster than that of the estimated common components. These results are obtained under general conditions that allow for correlations and heteroskedasticities in both dimensions. Stronger results are obtained when the idiosyncratic errors are serially uncorrelated and homoskedastic. A necessary and sufficient condition for consistency is derived for large N but fixed T.

Boosting With the L2 Loss

Abstract: This article investigates a computationally simple variant of boosting, L2Boost, which is constructed from a functional gradient descent algorithm with the L2-loss function. Like other boosting algorithms, L2Boost uses many times in an iterative fashion a prechosen fitting method, called the learner. Based on the explicit expression of refitting of residuals of L2Boost, the case with (symmetric) linear learners is studied in detail in both regression and classification. In particular, with the boosting iteration m working as the smoothing or regularization parameter, a new exponential bias-variance trade-off is found with the variance (complexity) term increasing very slowly as m tends to infinity. When the learner is a smoothing spline, an optimal rate of convergence result holds for both regression and classification and the boosted smoothing spline even adapts to higher-order, unknown smoothness. Moreover, a simple expansion of a (smoothed) 0–1 loss function is derived to reveal the importance of the d...

A Trigonometric Mutation Operation to Differential Evolution

Abstract: Previous studies have shown that differential evolution is an efficient, effective and robust evolutionary optimization method. However, the convergence rate of differential evolution in optimizing a computationally expensive objective function still does not meet all our requirements, and attempting to speed up DE is considered necessary. In this paper, a new local search operation, trigonometric mutation, is proposed and embedded into the differential evolution algorithm. This modification enables the algorithm to get a better trade-off between the convergence rate and the robustness. Thus it can be possible to increase the convergence velocity of the differential evolution algorithm and thereby obtain an acceptable solution with a lower number of objective function evaluations. Such an improvement can be advantageous in many real-world problems where the evaluation of a candidate solution is a computationally expensive operation and consequently finding the global optimum or a good sub-optimal solution with the original differential evolution algorithm is too time-consuming, or even impossible within the time available. In this article, the mechanism of the trigonometric mutation operation is presented and analyzed. The modified differential evolution algorithm is demonstrated in cases of two well-known test functions, and is further examined with two practical training problems of neural networks. The obtained numerical simulation results are providing empirical evidences on the efficiency and effectiveness of the proposed modified differential evolution algorithm.

On the construction of blending elements for local partition of unity enriched finite elements

Abstract: For computational efficiency, partition of unity enrichments are preferably localized to the sub-domains where they are needed. It is shown that an appropriate construction of the elements in the blending area, the regionwhere the enriched elements blend to unenriched elements, is often crucial for good performance of local partition of unity enrichments. An enhanced strain formulation is developed which leads to good performance; the optimal rate of convergence is achieved. For polynomial enrichments, it is shown that a proper choice of the finite element shape functions and partition of unity shape functions also improves the accuracy and convergence. The methods are illustrated by several examples. The examples deal primarily with the signed distance function enrichment for treating discontinuous derivatives inside an element, but other enrichments are also considered. Results show that both methods provide optimal rates of convergence.

Numerical analysis of the Allen-Cahn equation and approximation for mean curvature flows

Abstract: We propose and analyze a semi-discrete (in time) scheme and a fully discrete scheme for the Allen-Cahn equation u t −Δu+ɛ−2 f(u)=0 arising from phase transition in materials science, where ɛ is a small parameter known as an ``interaction length''. The primary goal of this paper is to establish some useful a priori error estimates for the proposed numerical methods, in particular, by focusing on the dependence of the error bounds on ɛ. Optimal order and quasi-optimal order error bounds are shown for the semi-discrete and fully discrete schemes under different constraints on the mesh size h and the time step size k and different regularity assumptions on the initial datum function u 0 . In particular, all our error bounds depend on only in some lower polynomial order for small ɛ. The cruxes of the analysis are to establish stability estimates for the discrete solutions, to use a spectrum estimate result of de Mottoni and Schatzman [18, 19] and Chen [12] and to establish a discrete counterpart of it for a linearized Allen-Cahn operator to handle the nonlinear term. Finally, as a nontrivial byproduct, the error estimates are used to establish convergence and rate of convergence of the zero level set of the fully discrete solution to the motion by mean curvature flow and to the generalized motion by mean curvature flow.

Enhancing the resolving power of least‐squares inversion with active constraint balancing

Abstract: Most geophysical inverse problems are solved using least‐squares inversion schemes with damping or smoothness constraints to improve stability and convergence rate. Since the Lagrangian multiplier controls resolution and stability of the inverse problem, we always want to use the optimum multiplier, which is not easy to get and is usually obtained by experience or a time‐consuming optimization process.We present a new regularization approach, in which the Lagrangian multiplier is set as a spatial variable at each parameterized block and automatically determined via the parameter resolution matrix and spread function analysis. For highly resolvable parameters, a small value of the Lagrangian multiplier is assigned, and vice versa. This approach, named “active constraint balancing” (ACB), tries to balance the constraints of the least‐squares inversion according to sensitivity for a given problem so that it enhances the resolution as well as the stability of the inversion process. We demonstrate the performa...

Reachability analysis of nonlinear systems using conservative approximation

Abstract: In this paper we present an approach to approximate reachability computation for nonlinear continuous systems. Rather than studying a complex nonlinear system x = g(x), we study an approximating system x = f(x) which is easier to handle. The class of approximating systems we consider in this paper is piecewise linear, obtained by interpolating g over a mesh. In order to be conservative, we add a bounded input in the approximating system to account for the interpolation error. We thus develop a reachability method for systems with input, based on the relation between such systems and the corresponding autonomous systems in terms of reachable sets. This method is then extended to the approximate piecewise linear systems arising in our construction. The final result is a reachability algorithm for nonlinear continuous systems which allows to compute conservative approximations with as great degree of accuracy as desired, and more importantly, it has good convergence rate. If g is a C2 function, our method is of order 2. Furthermore, the method can be straightforwardly extended to hybrid systems.

Enhanced accuracy by post-processing for finite element methods for hyperbolic equations

Abstract: We consider the enhancement of accuracy, by means of a simple post-processing technique, for finite element approximations to transient hyperbolic equations. The post-processing is a convolution with a kernel whose support has measure of order one in the case of arbitrary unstructured meshes; if the mesh is locally translation invariant, the support of the kernel is a cube whose edges are of size of the order of Δx only. For example, when polynomials of degree k are used in the discontinuous Galerkin (DG) method, and the exact solution is globally smooth, the DG method is of order k+1/2 in the L2-norm, whereas the post-processed approximation is of order 2k + 1; if the exact solution is in L2 only, in which case no order of convergence is available for the DG method, the post-processed approximation converges with order k + 1/2 in L2(Ω0), where Ω0 is a subdomain over which the exact solution is smooth. Numerical results displaying the sharpness of the estimates are presented.

Convergence properties of three spike-triggered analysis techniques

Abstract: We analyse the convergence properties of three spike-triggered data analysis techniques. Our results are obtained in the setting of a probabilistic linear-nonlinear (LN) cascade neural encoding model; this model has recently become popular in the study of the neural coding of natural signals. We start by giving exact rate-of-convergence results for the common spike-triggered average technique. Next, we analyse a spike-triggered covariance method, variants of which have been recently exploited successfully by Bialek, Simoncelli and colleagues. Unfortunately, the conditions that guarantee that these two estimators will converge to the correct parameters are typically not satisfied by natural signal data. Therefore, we introduce an estimator for the LN model parameters which is designed to converge under general conditions to the correct model. We derive the rate of convergence of this estimator, provide an algorithm for its computation and demonstrate its application to simulated data as well as physiological data from the primary motor cortex of awake behaving monkeys. We also give lower bounds on the convergence rate of any possible LN estimator. Our results should prove useful in the study of the neural coding of high-dimensional natural signals.

Sparse finite elements for elliptic problems with stochastic loading

Abstract: We formulate elliptic boundary value problems with stochastic loading in a bounded domain D?? d We show well-posedness of the problem in stochastic Sobolev spaces and we derive a deterministic elliptic PDE in D×D for the spatial correlation of the random solution We show well-posedness and regularity results for this PDE in a scale of weighted Sobolev spaces with mixed highest order derivatives Discretization with sparse tensor products of any hierarchic finite element (FE) spaces in D yields optimal asymptotic rates of convergence for the spatial correlation even in the presence of singularities or for spatially completely uncorrelated data Multilevel preconditioning in D×D allows iterative solution of the discrete equation for the correlation kernel in essentially the same complexity as the solution of the mean field equation

Convergence of the Monte Carlo expectation maximization for curved exponential families

Abstract: The Monte Carlo expectation maximization (MCEM) algorithm is a versatile tool for inference in incomplete data models, especially when used in combination with Markov chain Monte Carlo simulation methods. In this contribution, the almost-sure convergence of the MCEM algorithm is established. It is shown, using uniform versions of ergodic theorems for Markov chains, that MCEM converges under weak conditions on the simulation kernel. Practical illustrations are presented, using a hybrid random walk Metropolis Hastings sampler and an independence sampler. The rate of convergence is studied, showing the impact of the simulation schedule on the fluctuation of the parameter estimate at the convergence. A novel averaging procedure is then proposed to reduce the simulation variance and increase the rate of convergence.

A Particle-Partition of Unity Method Part V: Boundary Conditions

Abstract: In this sequel to [12, 13, 14, 15] we focus on the implementation of Dirichlet boundary conditions in our partition of unity method. The treatment of essential boundary conditions with meshfree Galerkin methods is not an easy task due to the non-interpolatory character of the shape functions. Here, the use of an almost forgotten method due to Nitsche from the 1970’s allows us to overcome these problems at virtually no extra computational costs. The method is applicable to general point distributions and leads to positive definite linear systems. The results of our numerical experiments, where we consider discretizations with several million degrees of freedom in two and three dimensions, clearly show that we achieve the optimal convergence rates for regular and singular solutions with the (adaptive) h-version and (augmented) p-version.

On the rate of convergence of regularized boosting classifiers

Abstract: A regularized boosting method is introduced, for which regularization is obtained through a penalization function. It is shown through oracle inequalities that this method is model adaptive. The rate of convergence of the probability of misclassification is investigated. It is shown that for quite a large class of distributions, the probability of error converges to the Bayes risk at a rate faster than n-(V+2)/(4(V+1)) where V is the VC dimension of the "base" class whose elements are combined by boosting methods to obtain an aggregated classifier. The dimension-independent nature of the rates may partially explain the good behavior of these methods in practical problems. Under Tsybakov's noise condition the rate of convergence is even faster. We investigate the conditions necessary to obtain such rates for different base classes. The special case of boosting using decision stumps is studied in detail. We characterize the class of classifiers realizable by aggregating decision stumps. It is shown that some versions of boosting work especially well in high-dimensional logistic additive models. It appears that adding a limited labelling noise to the training data may in certain cases improve the convergence, as has been also suggested by other authors.

Preconditioned iterative minimization for linear-scaling electronic structure calculations

Abstract: Linear-scaling electronic structure methods are essential for calculations on large systems. Some of these approaches use a systematic basis set, the completeness of which may be tuned with an adjustable parameter similar to the energy cut-off of plane-wave techniques. The search for the electronic ground state in such methods suffers from an ill-conditioning which is related to the kinetic contribution to the total energy and which results in unacceptably slow convergence. We present a general preconditioning scheme to overcome this ill-conditioning and implement it within our own first-principles linear-scaling density functional theory method. The scheme may be applied in either real space or reciprocal space with equal success. The rate of convergence is improved by an order of magnitude and is found to be almost independent of the size of the basis.

Convergence rate for compressible Euler equations with damping and vacuum

Abstract: We study the asymptotic behavior of L∞ weak-entropy solutions to the compressible Euler equations with damping and vacuum. Previous works on this topic are mainly concerned with the case away from the vacuum and small initial data. In the present paper, we prove that the entropy-weak solution strongly converges to the similarity solution of the porous media equations in Lp(R) (2≤p<∞) with decay rates. The initial data can contain vacuum and can be arbitrary large. A new approach is introduced to control the singularity near vacuum for the desired estimates.

On the approximation of fixed points of weak contractive mappings

Abstract: In this paper, the class of weak contractive type mappings, introduced in (3) and stud- ied in (3) and (4) is compared to some other well known contractive type mappings in Rhoades' classification. As corollaries of our main results, we obtain several convergence theorems for approxi- mating fixed points by means of Picard iteration. These complete or extend the corresponding results in literature by providing error estimates, rate of convergence for used iterative method as well as results concerning the data dependence of the fixed points.

Parameter optimization in iterative learning control

Abstract: In this paper parameter optimization through a quadratic performance index is introduced as a method to establish a new iterative learning control law. With this new algorithm, monotonic convergence of the error to zero is guaranteed if the original system is a discrete-time LTI system and it satisfies a positivity condition. If the original system is not positive, two methods are derived to make the system positive. The effect of the choice of weighting parameters in the performance index on convergence rate is analysed. As a result adaptive weights are introduced as a method to improve the convergence properties of the algorithm. A high-order version of the algorithm is also derived and its convergence analysed. The theoretical findings in this paper are highlighted with simulations.

The SDFEM for a Convection-Diffusion Problem with a Boundary Layer: Optimal Error Analysis and Enhancement of Accuracy

Abstract: The streamline-diffusion finite element method (SDFEM) is applied to a convection-diffusion problem posed on the unit square, using a Shishkin rectangular mesh with piecewise bilinear trial functions. The hypotheses of the problem exclude interior layers but allow exponential boundary layers. An error bound is proved for $\|u^I-u^N\|_{SD}$, where $u^I$ is the interpolant of the solution $u$, $u^N$ is the SDFEM solution, and $\|\cdot\|_{SD}$ is the streamline-diffusion norm. This bound implies that $\|u-u^N\|_{L^2}$ is of optimal order, thereby settling an open question regarding the $L^2$-accuracy of the SDFEM on rectangular meshes. Furthermore, the bound shows that $u^N$ is superclose to $u^I$, which allows the construction of a simple postprocessing that yields a more accurate solution. Enhancement of the rate of convergence by using a discrete streamline-diffusion norm is also discussed. Finally, the verification of these rates of convergence by numerical experiments is examined, and it is shown that this practice is less reliable than was previously believed.

Finite element superconvergence on Shishkin mesh for 2-D convection-diffusion problems

Abstract: In this work, the bilinear finite element method on a Shishkin mesh for convection-diffusion problems is analyzed in the two-dimensional setting. A superconvergence rate O(N-2 ln2 N + eN-1.5lnN) in a discrete e-weighted energy norm is established under certain regularity assumptions. This convergence rate is uniformly valid with respect to the singular perturbation parameter e. Numerical tests indicate that the rate O(N-2ln2 N) is sharp for the boundary layer terms. As a by-product, an e-uniform convergence of the same order is obtained for the L2-norm. Furthermore, under the same regularity assumption, an e-uniform convergence of order N-3/2ln5/2N + eN-1ln1/2N in the L∞ norm is proved for some mesh points in the boundary layer region.

Conservative Front Tracking with Improved Accuracy

Abstract: We propose a fully conservative front tracking algorithm for systems of nonlinear conservation laws. The algorithm improves by one order in its convergence rate over most finite difference schemes. Near tracked discontinuities in the solution, the proposed algorithm has ${\mathcal O}(\Delta x)$ errors, improving over ${\mathcal O}(1)$ errors commonly found near a discontinuity. Numerical experiments which confirm these assertions are presented.

Multichannel affine and fast affine projection algorithms for active noise control and acoustic equalization systems

Abstract: In the field of adaptive signal processing, it is well known that affine projection algorithms or their low-computational implementations fast affine projection algorithms can produce a good tradeoff between convergence speed and computational complexity. Although these algorithms typically do not provide the same convergence speed as recursive-least-squares algorithms, they can provide a much improved convergence speed compared to stochastic gradient descent algorithms, without the high increase of the computational load or the instability often found in recursive-least-squares algorithms. In this paper, multichannel affine and fast affine projection algorithms are introduced for active noise control or acoustic equalization. Multichannel fast affine projection algorithms have been previously published for acoustic echo cancellation, but the problem of active noise control or acoustic equalization is a very different one, leading to different structures, as explained in the paper. The computational complexity of the new algorithms is evaluated, and it is shown through simulations that not only can the new algorithms provide the expected tradeoff between convergence performance and computational complexity, they can also provide the best convergence performance (even over recursive-least-squares algorithms) when nonideal noisy acoustic plant models are used in the adaptive systems.

Iterative Solution Methods for Modeling Multiphase Flow in Porous Media Fully Implicitly

Abstract: We discuss several fully implicit techniques for solving the nonlinear algebraic system arising in an expanded mixed finite element or cell-centered finite difference discretization of two- and three-phase porous media flow. Every outer nonlinear Newton iteration requires solution of a nonsymmetric Jacobian linear system. Two major types of preconditioners, supercoarsening multigrid (SCMG) and two-stage, are developed for the GMRES iteration applied to the solution of the Jacobian system. The SCMG reduces the three-dimensional system to two dimensions using a vertical aggregation followed by a two-dimensional multigrid. The two-stage preconditioners are based on decoupling the system into a pressure and concentration equations. Several pressure preconditioners of different types are described. Extensive numerical results are presented using the integrated parallel reservoir simulator (IPARS) and indicate that these methods have low arithmetical complexity per iteration and good convergence rates.