Showing papers on "Rate of convergence published in 2010"

Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Łojasiewicz Inequality

Abstract: We study the convergence properties of an alternating proximal minimization algorithm for nonconvex structured functions of the type: L(x,y)=f(x)+Q(x,y)+g(y), where f and g are proper lower semicontinuous functions, defined on Euclidean spaces, and Q is a smooth function that couples the variables x and y. The algorithm can be viewed as a proximal regularization of the usual Gauss-Seidel method to minimize L. We work in a nonconvex setting, just assuming that the function L satisfies the Kurdyka-Łojasiewicz inequality. An entire section illustrates the relevancy of such an assumption by giving examples ranging from semialgebraic geometry to “metrically regular” problems. Our main result can be stated as follows: If L has the Kurdyka-Łojasiewicz property, then each bounded sequence generated by the algorithm converges to a critical point of L. This result is completed by the study of the convergence rate of the algorithm, which depends on the geometrical properties of the function L around its critical points. When specialized to $Q(x,y)=\Vert x-y \Vert ^2$ and to f, g indicator functions, the algorithm is an alternating projection mehod (a variant of von Neumann's) that converges for a wide class of sets including semialgebraic and tame sets, transverse smooth manifolds or sets with “regular” intersection. To illustrate our results with concrete problems, we provide a convergent proximal reweighted l1 algorithm for compressive sensing and an application to rank reduction problems.

A General Framework for a Class of First Order Primal-Dual Algorithms for Convex Optimization in Imaging Science

Abstract: We generalize the primal-dual hybrid gradient (PDHG) algorithm proposed by Zhu and Chan in [An Efficient Primal-Dual Hybrid Gradient Algorithm for Total Variation Image Restoration, CAM Report 08-34, UCLA, Los Angeles, CA, 2008] to a broader class of convex optimization problems. In addition, we survey several closely related methods and explain the connections to PDHG. We point out convergence results for a modified version of PDHG that has a similarly good empirical convergence rate for total variation (TV) minimization problems. We also prove a convergence result for PDHG applied to TV denoising with some restrictions on the PDHG step size parameters. We show how to interpret this special case as a projected averaged gradient method applied to the dual functional. We discuss the range of parameters for which these methods can be shown to converge. We also present some numerical comparisons of these algorithms applied to TV denoising, TV deblurring, and constrained $l_1$ minimization problems.

Optimal rates of convergence for covariance matrix estimation

Abstract: Covariance matrix plays a central role in multivariate statistical analysis. Significant advances have been made recently on developing both theory and methodology for estimating large covariance matrices. However, a minimax theory has yet been developed. In this paper we establish the optimal rates of convergence for estimating the covariance matrix under both the operator norm and Frobenius norm. It is shown that optimal procedures under the two norms are different and consequently matrix estimation under the operator norm is fundamentally different from vector estimation. The minimax upper bound is obtained by constructing a special class of tapering estimators and by studying their risk properties. A key step in obtaining the optimal rate of convergence is the derivation of the minimax lower bound. The technical analysis requires new ideas that are quite different from those used in the more conventional function/sequence estimation problems.

Distributed Sparse Linear Regression

Abstract: The Lasso is a popular technique for joint estimation and continuous variable selection, especially well-suited for sparse and possibly under-determined linear regression problems. This paper develops algorithms to estimate the regression coefficients via Lasso when the training data are distributed across different agents, and their communication to a central processing unit is prohibited for e.g., communication cost or privacy reasons. A motivating application is explored in the context of wireless communications, whereby sensing cognitive radios collaborate to estimate the radio-frequency power spectrum density. Attaining different tradeoffs between complexity and convergence speed, three novel algorithms are obtained after reformulating the Lasso into a separable form, which is iteratively minimized using the alternating-direction method of multipliers so as to gain the desired degree of parallelization. Interestingly, the per agent estimate updates are given by simple soft-thresholding operations, and inter-agent communication overhead remains at affordable level. Without exchanging elements from the different training sets, the local estimates consent to the global Lasso solution, i.e., the fit that would be obtained if the entire data set were centrally available. Numerical experiments with both simulated and real data demonstrate the merits of the proposed distributed schemes, corroborating their convergence and global optimality. The ideas in this paper can be easily extended for the purpose of fitting related models in a distributed fashion, including the adaptive Lasso, elastic net, fused Lasso and nonnegative garrote.

Guaranteed Rank Minimization via Singular Value Projection

Abstract: Minimizing the rank of a matrix subject to affine constraints is a fundamental problem with many important applications in machine learning and statistics. In this paper we propose a simple and fast algorithm SVP (Singular Value Projection) for rank minimization under affine constraints (ARMP) and show that SVP recovers the minimum rank solution for affine constraints that satisfy a restricted isometry property (RIP). Our method guarantees geometric convergence rate even in the presence of noise and requires strictly weaker assumptions on the RIP constants than the existing methods. We also introduce a Newton-step for our SVP framework to speed-up the convergence with substantial empirical gains. Next, we address a practically important application of ARMP - the problem of low-rank matrix completion, for which the defining affine constraints do not directly obey RIP, hence the guarantees of SVP do not hold. However, we provide partial progress towards a proof of exact recovery for our algorithm by showing a more restricted isometry property and observe empirically that our algorithm recovers low-rank incoherent matrices from an almost optimal number of uniformly sampled entries. We also demonstrate empirically that our algorithms outperform existing methods, such as those of [5, 18, 14], for ARMP and the matrix completion problem by an order of magnitude and are also more robust to noise and sampling schemes. In particular, results show that our SVP-Newton method is significantly robust to noise and performs impressively on a more realistic power-law sampling scheme for the matrix completion problem.

A Newton-CG Augmented Lagrangian Method for Semidefinite Programming

Abstract: We consider a Newton-CG augmented Lagrangian method for solving semidefinite programming (SDP) problems from the perspective of approximate semismooth Newton methods. In order to analyze the rate of convergence of our proposed method, we characterize the Lipschitz continuity of the corresponding solution mapping at the origin. For the inner problems, we show that the positive definiteness of the generalized Hessian of the objective function in these inner problems, a key property for ensuring the efficiency of using an inexact semismooth Newton-CG method to solve the inner problems, is equivalent to the constraint nondegeneracy of the corresponding dual problems. Numerical experiments on a variety of large-scale SDP problems with the matrix dimension $n$ up to $4,110$ and the number of equality constraints $m$ up to $2,156,544$ show that the proposed method is very efficient. We are also able to solve the SDP problem fap36 (with $n=4,110$ and $m=1,154,467$) in the Seventh DIMACS Implementation Challenge much more accurately than in previous attempts.

Convergence Rates of Best N -term Galerkin Approximations for a Class of Elliptic sPDEs

Abstract: Deterministic Galerkin approximations of a class of second order elliptic PDEs with random coefficients on a bounded domain D⊂ℝd are introduced and their convergence rates are estimated. The approximations are based on expansions of the random diffusion coefficients in L 2(D)-orthogonal bases, and on viewing the coefficients of these expansions as random parameters y=y(ω)=(y i (ω)). This yields an equivalent parametric deterministic PDE whose solution u(x,y) is a function of both the space variable x∈D and the in general countably many parameters y. We establish new regularity theorems describing the smoothness properties of the solution u as a map from y∈U=(−1,1)∞ to $V=H^{1}_{0}(D)$. These results lead to analytic estimates on the V norms of the coefficients (which are functions of x) in a so-called “generalized polynomial chaos” (gpc) expansion of u. Convergence estimates of approximations of u by best N-term truncated V valued polynomials in the variable y∈U are established. These estimates are of the form N −r , where the rate of convergence r depends only on the decay of the random input expansion. It is shown that r exceeds the benchmark rate 1/2 afforded by Monte Carlo simulations with N “samples” (i.e., deterministic solves) under mild smoothness conditions on the random diffusion coefficients. A class of fully discrete approximations is obtained by Galerkin approximation from a hierarchic family $\{V_{l}\}_{l=0}^{\infty}\subset V$of finite element spaces in D of the coefficients in the N-term truncated gpc expansions of u(x,y). In contrast to previous works, the level l of spatial resolution is adapted to the gpc coefficient. New regularity theorems describing the smoothness properties of the solution u as a map from y∈U=(−1,1)∞ to a smoothness space W⊂V are established leading to analytic estimates on the W norms of the gpc coefficients and on their space discretization error. The space W coincides with $H^{2}(D)\cap H^{1}_{0}(D)$in the case where D is a smooth or convex domain. Our analysis shows that in realistic settings a convergence rate $N_{\mathrm{dof}}^{-s}$in terms of the total number of degrees of freedom N dof can be obtained. Here the rate s is determined by both the best N-term approximation rate r and the approximation order of the space discretization in D.

Irregular Identification, Support Conditions, and Inverse Weight Estimation

Abstract: In weighted moment condition models, we show a subtle link between identification and estimability that limits the practical usefulness of estimators based on these models. In particular, if it is necessary for (point) identification that the weights take arbitrarily large values, then the parameter of interest, though point identified, cannot be estimated at the regular (parametric) rate and is said to be irregularly identified. This rate depends on relative tail conditions and can be as slow in some examples as n−1/4. This nonstandard rate of convergence can lead to numerical instability and/or large standard errors. We examine two weighted model examples: (i) the binary response model under mean restriction introduced by Lewbel (1997) and further generalized to cover endogeneity and selection, where the estimator in this class of models is weighted by the density of a special regressor, and (ii) the treatment effect model under exogenous selection (Rosenbaum and Rubin (1983)), where the resulting estimator of the average treatment effect is one that is weighted by a variant of the propensity score. Without strong relative support conditions, these models, similar to well known “identified at infinity” models, lead to estimators that converge at slower than parametric rate, since essentially, to ensure point identification, one requires some variables to take values on sets with arbitrarily small probabilities, or thin sets. For the two models above, we derive some rates of convergence and propose that one conducts inference using rate adaptive procedures that are analogous to Andrews and Schafgans (1998) for the sample selection model.

Approximation accuracy, gradient methods, and error bound for structured convex optimization

Abstract: Convex optimization problems arising in applications, possibly as approximations of intractable problems, are often structured and large scale. When the data are noisy, it is of interest to bound the solution error relative to the (unknown) solution of the original noiseless problem. Related to this is an error bound for the linear convergence analysis of first-order gradient methods for solving these problems. Example applications include compressed sensing, variable selection in regression, TV-regularized image denoising, and sensor network localization.

Mean-Field Dynamics: Singular Potentials and Rate of Convergence

Abstract: We consider the time evolution of a system of N identical bosons whose interaction potential is rescaled by N−1. We choose the initial wave function to describe a condensate in which all particles are in the same one-particle state. It is well known that in the mean-field limit N → ∞ the quantum N-body dynamics is governed by the nonlinear Hartree equation. Using a nonperturbative method, we extend previous results on the mean-field limit in two directions. First, we allow a large class of singular interaction potentials as well as strong, possibly time-dependent external potentials. Second, we derive bounds on the rate of convergence of the quantum N-body dynamics to the Hartree dynamics.

Asymptotic behavior of a Cahn-Hilliard-Navier-Stokes system in 2D

Abstract: We consider a model for the flow of a mixture of two homogeneous and incompressible fluids in a two-dimensional bounded domain. The model consists of a Navier–Stokes equation governing the fluid velocity coupled with a convective Cahn–Hilliard equation for the relative density of atoms of one of the fluids. Endowing the system with suitable boundary and initial conditions, we analyze the asymptotic behavior of its solutions. First, we prove that the initial and boundary value problem generates a strongly continuous semigroup on a suitable phase-space which possesses the global attractor A . Then we establish the existence of an exponential attractors E . Thus A has finite fractal dimension. This dimension is then estimated from above in terms of the physical parameters. Moreover, assuming the potential to be real analytic and in absence of volume forces, we demonstrate that each trajectory converges to a single equilibrium. We also obtain a convergence rate estimate in the phase-space metric.

A new multiscale finite element method for high-contrast elliptic interface problems

Abstract: We introduce a new multiscale finite element method which is able to accurately capture solutions of elliptic interface problems with high contrast coefficients by using only coarse quasiuniform meshes, and without resolving the interfaces. A typical application would be the modelling of flow in a porous medium containing a number of inclusions of low (or high) permeability embedded in a matrix of high (respectively low) permeability. Our method is H^1- conforming, with degrees of freedom at the nodes of a triangular mesh and requiring the solution of subgrid problems for the basis functions on elements which straddle the coefficient interface but which use standard linear approximation otherwise. A key point is the introduction of novel coefficientdependent boundary conditions for the subgrid problems. Under moderate assumptions, we prove that our methods have (optimal) convergence rate of O(h) in the energy norm and O(h^2) in the L_2 norm where h is the (coarse) mesh diameter and the hidden constants in these estimates are independent of the “contrast” (i.e. ratio of largest to smallest value) of the PDE coefficient. For standard elements the best estimate in the energy norm would be O(h^(1/2−e)) with a hidden constant which in general depends on the contrast. The new interior boundary conditions depend not only on the contrast of the coefficients, but also on the angles of intersection of the interface with the element edges.

Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDEs

Abstract: Parametric partial dierential equations are commonly used to model physical systems. They also arise when Wiener chaos expansions are used as an alternative to Monte Carlo when solving stochastic elliptic problems. This paper considers a model class of second order, linear, parametric, elliptic PDEs in a bounded domain D with coecients depending on possibly countably many parameters. It shows that the dependence of the solution on the parameters in the diusion coecient is analytically smooth. This analyticity is then exploited to prove that under very weak assumptions on the diusion coecients, the entire family of solutions to such equations can be simultaneously approximated by multivariate polynomials (in the parameters) with coecients taking values in the Hilbert space V = H 1 0(D) of weak solutions of the elliptic problem with a controlled number of terms N. The convergence rate in terms of N does not depend on the number of parameters in V which may be countable, therefore breaking the curse of dimensionality. The discretization of the coecients from a family of continuous, piecewise linear Finite Element functions in D is shown to yield finite dimensional approximations whose convergence rate in terms of the overall number Ndof of degrees of freedom is the minimum of the convergence rates aorded by the best N-term sequence approximations in the parameter space and the rate of Finite Element approximations in D for a single instance of the parametric problem.

Particle swarm optimization with fractional-order velocity

Abstract: This paper proposes a novel method for controlling the convergence rate of a particle swarm optimization algorithm using fractional calculus (FC) concepts. The optimization is tested for several well-known functions and the relationship between the fractional order velocity and the convergence of the algorithm is observed. The FC demonstrates a potential for interpreting evolution of the algorithm and to control its convergence.

Convergence rate for consensus with delays

Abstract: We study the problem of reaching a consensus in the values of a distributed system of agents with time-varying connectivity in the presence of delays. We consider a widely studied consensus algorithm, in which at each time step, every agent forms a weighted average of its own value with values received from the neighboring agents. We study an asynchronous operation of this algorithm using delayed agent values. Our focus is on establishing convergence rate results for this algorithm. In particular, we first show convergence to consensus under a bounded delay condition and some connectivity and intercommunication conditions imposed on the multi-agent system. We then provide a bound on the time required to reach the consensus. Our bound is given as an explicit function of the system parameters including the delay bound and the bound on agents' intercommunication intervals.

A Reproducing Kernel Hilbert Space Approach to Functional Linear Regression

Abstract: We study in this paper a smoothness regularization method for functional linear regression and provide a unified treatment for both the prediction and estimation problems. By developing a tool on simultaneous diagonalization of two positive definite kernels, we obtain shaper results on the minimax rates of convergence and show that smoothness regularized estimators achieve the optimal rates of convergence for both prediction and estimation under conditions weaker than those for the functional principal components based methods developed in the literature. Despite the generality of the method of regularization, we show that the procedure is easily implementable. Numerical results are obtained to illustrate the merits of the method and to demonstrate the theoretical developments.

Distributed Estimation Over an Adaptive Incremental Network Based on the Affine Projection Algorithm

Abstract: We study the problem of distributed estimation based on the affine projection algorithm (APA), which is developed from Newton's method for minimizing a cost function. The proposed solution is formulated to ameliorate the limited convergence properties of least-mean-square (LMS) type distributed adaptive filters with colored inputs. The analysis of transient and steady-state performances at each individual node within the network is developed by using a weighted spatial-temporal energy conservation relation and confirmed by computer simulations. The simulation results also verify that the proposed algorithm provides not only a faster convergence rate but also an improved steady-state performance as compared to an LMS-based scheme. In addition, the new approach attains an acceptable misadjustment performance with lower computational and memory cost, provided the number of regressor vectors and filter length parameters are appropriately chosen, as compared to a distributed recursive-least-squares (RLS) based method.

Convergence Rates of Distributed Average Consensus With Stochastic Link Failures

Abstract: We consider a distributed average consensus algorithm over a network in which communication links fail with independent probability. In such stochastic networks, convergence is defined in terms of the variance of deviation from average. We first show how the problem can be recast as a linear system with multiplicative random inputs which model link failures. We then use our formulation to derive recursion equations for the second order statistics of the deviation from average in networks with and without additive noise. We give expressions for the convergence behavior in the asymptotic limits of small failure probability and large networks. We also present simulation-free methods for computing the second order statistics in each network model and use these methods to study the behavior of various network examples as a function of link failure probability.

Second-order full-discretization method for milling stability prediction

Abstract: This paper proposes a second-order full-discretization method for milling stability prediction based on the direct integration scheme. The model of the milling dynamics taking the regenerative effect into account in the state-space form is firstly represented in the integral form. After the time period being equally discretized into a finite set of intervals, the full-discretization method is developed to handle the integration term of the system. On each small time interval, the second-degree Lagrange polynomial is employed to interpolate the state item, and the linear interpolation is utilized to approximate the time-periodic and time delay items, respectively. Then, a discrete dynamical map is deduced to establish the state transition matrix on one time period to predict the milling stability via Floquet theory. The rate of convergence of the method is discussed, and the benchmark example is utilized to verify the effectiveness of the presented algorithm. The MATLAB code of the algorithm is attached in the Appendix.

Newton-Like Extremum-Seeking for the Control of Thermoacoustic Instability

Abstract: In practice, the convergence rate and stability of perturbation based extremum-seeking schemes can be very sensitive to the curvature of the plant map. An example of this can be seen in the use of extremum-seeking to reduce the amplitude of thermoacoustic oscillations in premixed, gas-turbine combustors. This sensitivity to the plant map curvature arises from the use of a gradient descent adaptation algorithm. Such extremum-seeking schemes may need to be conservatively tuned in order to maintain stability over a wide range of operating conditions, resulting in slower optimization than could be achieved for a fixed operating condition. This can severely reduce the effectiveness of perturbation based extremum-seeking schemes in some applications. In this paper, a sinusoidally perturbed extremum-seeking scheme using a Newton-like step is developed. Non-local stability results for the scheme are formulated using a Lyapunov analysis. A local analysis of the scheme is given to investigate the influence of plant dynamics and to show that the local rate of convergence is independent of the plant map curvature. The benefit of this plant map curvature independence is then experimentally demonstrated in minimizing the thermoacoustic oscillations in a model premixed combustor.

Quasi-Optimal Convergence Rate of an Adaptive Discontinuous Galerkin Method

Abstract: We analyze an adaptive discontinuous finite element method (ADFEM) for symmetric second order linear elliptic operators. The method is formulated on nonconforming meshes made of simplices or quadrilaterals, with any polynomial degree and in any dimension $\geq2$. We prove that the ADFEM is a contraction for the sum of the energy error and the scaled error estimator between two consecutive adaptive loops. We design a refinement procedure that maintains the level of nonconformity uniformly bounded and prove that the approximation classes using continuous and discontinuous finite elements are equivalent. The geometric decay and the equivalence of classes are instrumental in deriving the optimal cardinality of the ADFEM. We show that the ADFEM (and the AFEM on nonconforming meshes) yields a decay rate of energy error plus oscillation in terms of the number of degrees of freedom as dictated by the best approximation for this combined nonlinear quantity.

On Hermitian and Skew-Hermitian Splitting Ietration Methods for the Continuous Sylvester Equations

Abstract: We present a Hermitian and skew-Hermitian splitting (HSS) iteration method for solving large sparse continuous Sylvester equations with non-Hermitian and positive definite/semidefinite matrices. The unconditional convergence of the HSS iteration method is proved and an upper bound on the convergence rate is derived. Moreover, to reduce the computing cost, we establish an inexact variant of the HSS iteration method and analyze its convergence property in detail. Numerical results show that the HSS iteration method and its inexact variant are efficient and robust solvers for this class of continuous Sylvester equations.