Showing papers on "Rate of convergence published in 2007"

A New TwIST: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration

Abstract: Iterative shrinkage/thresholding (1ST) algorithms have been recently proposed to handle a class of convex unconstrained optimization problems arising in image restoration and other linear inverse problems. This class of problems results from combining a linear observation model with a nonquadratic regularizer (e.g., total variation or wavelet-based regularization). It happens that the convergence rate of these 1ST algorithms depends heavily on the linear observation operator, becoming very slow when this operator is ill-conditioned or ill-posed. In this paper, we introduce two-step 1ST (TwIST) algorithms, exhibiting much faster convergence rate than 1ST for ill-conditioned problems. For a vast class of nonquadratic convex regularizers (lscrP norms, some Besov norms, and total variation), we show that TwIST converges to a minimizer of the objective function, for a given range of values of its parameters. For noninvertible observation operators, we introduce a monotonic version of TwIST (MTwIST); although the convergence proof does not apply to this scenario, we give experimental evidence that MTwIST exhibits similar speed gains over IST. The effectiveness of the new methods are experimentally confirmed on problems of image deconvolution and of restoration with missing samples.

Gradient methods for minimizing composite objective function

Abstract: In this paper we analyze several new methods for solving optimization problems with the objective function formed as a sum of two convex terms: one is smooth and given by a black-box oracle, and another is general but simple and its structure is known. Despite to the bad properties of the sum, such problems, both in convex and nonconvex cases, can be solved with eciency typical for the good part of the objective. For convex problems of the above structure, we consider primal and dual variants of the gradient method (converge as O ‡ 1 k · ), and an accelerated multistep version with convergence rate O ‡ 1 k2 · , where k is the iteration counter. For all methods, we suggest some ecient “line search” procedures and show that the additional computational work necessary for estimating the unknown problem class parameters can only multiply the complexity of each iteration by a small constant factor. We present also the results of preliminary computational experiments, which confirm the superiority of the accelerated scheme.

Methodology and convergence rates for functional linear regression

Abstract: In functional linear regression, the slope "parameter" is a function. Therefore, in a nonparametric context, it is determined by an infinite number of unknowns. Its estimation involves solving an ill-posed problem and has points of contact with a range of methodologies, including statistical smoothing and deconvolution. The standard approach to estimating the slope function is based explicitly on functional principal components analysis and, consequently, on spectral decomposition in terms of eigenvalues and eigen-functions. We discuss this approach in detail and show that in certain circumstances, optimal convergence rates are achieved by the PCA technique. An alternative approach based on quadratic regularisation is suggested and shown to have advantages from some points of view.

A randomized Kaczmarz algorithm with exponential convergence

Abstract: The Kaczmarz method for solving linear systems of equations is an iterative algorithm that has found many applications ranging from computer tomography to digital signal processing. Despite the popularity of this method, useful theoretical estimates for its rate of convergence are still scarce. We introduce a randomized version of the Kaczmarz method for consistent, overdetermined linear systems and we prove that it converges with expected exponential rate. Furthermore, this is the first solver whose rate does not depend on the number of equations in the system. The solver does not even need to know the whole system, but only a small random part of it. It thus outperforms all previously known methods on general extremely overdetermined systems. Even for moderately overdetermined systems, numerical simulations as well as theoretical analysis reveal that our algorithm can converge faster than the celebrated conjugate gradient algorithm. Furthermore, our theory and numerical simulations confirm a prediction of Feichtinger et al. in the context of reconstructing bandlimited functions from nonuniform sampling.

Analysis of the Parareal Time-Parallel Time-Integration Method

Abstract: The parareal algorithm is a method to solve time-dependent problems parallel in time: it approximates parts of the solution later in time simultaneously to parts of the solution earlier in time. In this paper the relation of the parareal algorithm to space-time multigrid and multiple shooting methods is first briefly discussed. The focus of the paper is on new convergence results that show superlinear convergence of the algorithm when used on bounded time intervals, and linear convergence for unbounded intervals.

A convergence rates result for Tikhonov regularization in Banach spaces with non-smooth operators

Abstract: There exists a vast literature on convergence rates results for Tikhonov regularized minimizers. We are concerned with the solution of nonlinear ill-posed operator equations. The first convergence rates results for such problems were developed by Engl, Kunisch and Neubauer in 1989. While these results apply for operator equations formulated in Hilbert spaces, the results of Burger and Osher from 2004, more generally, apply to operators formulated in Banach spaces. Recently, Resmerita and co-workers presented a modification of the convergence rates result of Burger and Osher which turns out to be a complete generalization of the rates result of Engl and co-workers. In all these papers relatively strong regularity assumptions are made. However, it has been observed numerically that violations of the smoothness assumptions of the operator do not necessarily affect the convergence rate negatively. We take this observation and weaken the smoothness assumptions on the operator and prove a novel convergence rate result. The most significant difference in this result from the previous ones is that the source condition is formulated as a variational inequality and not as an equation as previously. As examples, we present a phase retrieval problem and a specific inverse option pricing problem, both previously studied in the literature. For the inverse finance problem, the new approach allows us to bridge the gap to a singular case, where the operator smoothness degenerates just when the degree of ill-posedness is minimal.

Convergence rates of posterior distributions for noniid observations

Abstract: We consider the asymptotic behavior of posterior distributions and Bayes estimators based on observations which are required to be neither independent nor identically distributed. We give general results on the rate of convergence of the posterior measure relative to distances derived from a testing criterion. We then specialize our results to independent, nonidentically distributed observations, Markov processes, stationary Gaussian time series and the white noise model. We apply our general results to several examples of infinite-dimensional statistical models including nonparametric regression with normal errors, binary regression, Poisson regression, an interval censoring model, Whittle estimation of the spectral density of a time series and a nonlinear autoregressive model.

Accelerating the cubic regularization of Newton's method on convex problems

Abstract: In this paper we propose an accelerated version of the cubic regularization of Newton’s method (Nesterov and Polyak, in Math Program 108(1): 177–205, 2006). The original version, used for minimizing a convex function with Lipschitz-continuous Hessian, guarantees a global rate of convergence of order $$O\big({1 \over k^2}\big)$$, where k is the iteration counter. Our modified version converges for the same problem class with order $$O\big({1 \over k^3}\big)$$, keeping the complexity of each iteration unchanged. We study the complexity of both schemes on different classes of convex problems. In particular, we argue that for the second-order schemes, the class of non-degenerate problems is different from the standard class.

Immersed-Interface Finite-Element Methods for Elliptic Interface Problems with Nonhomogeneous Jump Conditions

Abstract: In this work, a class of new finite-element methods, called immersed-interface finite-element methods, is developed to solve elliptic interface problems with nonhomogeneous jump conditions. Simple non-body-fitted meshes are used. A single function that satisfies the same nonhomogeneous jump conditions is constructed using a level-set representation of the interface. With such a function, the discontinuities across the interface in the solution and flux are removed, and an equivalent elliptic interface problem with homogeneous jump conditions is formulated. Special finite-element basis functions are constructed for nodal points near the interface to satisfy the homogeneous jump conditions. Error analysis and numerical tests are presented to demonstrate that such methods have an optimal convergence rate. These methods are designed as an efficient component of the finite-element level-set methodology for fast simulation of interface dynamics that does not require remeshing.

Convergence Rates of General Regularization Methods for Statistical Inverse Problems and Applications

Abstract: Previously, the convergence analysis for linear statistical inverse problems has mainly focused on spectral cut-off and Tikhonov-type estimators. Spectral cut-off estimators achieve minimax rates for a broad range of smoothness classes and operators, but their practical usefulness is limited by the fact that they require a complete spectral decomposition of the operator. Tikhonov estimators are simpler to compute but still involve the inversion of an operator and achieve minimax rates only in restricted smoothness classes. In this paper we introduce a unifying technique to study the mean square error of a large class of regularization methods (spectral methods) including the aforementioned estimators as well as many iterative methods, such as $ u$-methods and the Landweber iteration. The latter estimators converge at the same rate as spectral cut-off but require only matrix-vector products. Our results are applied to various problems; in particular we obtain precise convergence rates for satellite gradiometry, $L^2$-boosting, and errors in variable problems.

Linear convergence of iterative soft-thresholding

Abstract: In this article a unified approach to iterative soft-thresholding algorithms for the solution of linear operator equations in infinite dimensional Hilbert spaces is presented. We formulate the algorithm in the framework of generalized gradient methods and present a new convergence analysis. As main result we show that the algorithm converges with linear rate as soon as the underlying operator satisfies the so-called finite basis injectivity property or the minimizer possesses a so-called strict sparsity pattern. Moreover it is shown that the constants can be calculated explicitly in special cases (i.e. for compact operators). Furthermore, the techniques also can be used to establish linear convergence for related methods such as the iterative thresholding algorithm for joint sparsity and the accelerated gradient projection method.

Convergence rates for sparse chaos approximations of elliptic problems with stochastic coefficients

Abstract: A scalar, elliptic boundary-value problem in divergence form with stochastic diffusion coefficient a (x, co) in a bounded domain D R d is reformulated as a deterministic, infinite-dimensional, parametric problem by separation of deterministic (x ∈ D) and stochastic (ω e Q) variables in a(x,ω) via Karhunen-Loeve or Legendre expansions of the diffusion coefficient. Deterministic, approximate solvers are obtained by projection of this problem into a product probability space of finite dimension M and sparse discretizations of the resulting M-dimensional parametric problem. Both Galerkin and collocation approximations are considered. Under regularity assumptions on the fluctuation of a(x, ω) in the deterministic variable x, the convergence rate of the deterministic solution algorithm is analysed in terms of the number N of deterministic problems to be solved as both the chaos dimension M and the multiresolution level of the sparse discretization resp. the polynomial degree of the chaos expansion increase simultaneously.

Posterior convergence rates of Dirichlet mixtures at smooth densities

Abstract: We study the rates of convergence of the posterior distribution for Bayesian density estimation with Dirichlet mixtures of normal distributions as the prior. The true density is assumed to be twice continuously differentiable. The bandwidth is given a sequence of priors which is obtained by scaling a single prior by an appropriate order. In order to handle this problem, we derive a new general rate theorem by considering a countable covering of the parameter space whose prior probabilities satisfy a summability condition together with certain individual bounds on the Hellinger metric entropy. We apply this new general theorem on posterior convergence rates by computing bounds for Hellinger (bracketing) entropy numbers for the involved class of densities, the error in the approximation of a smooth density by normal mixtures and the concentration rate of the prior. The best obtainable rate of convergence of the posterior turns out to be equivalent to the well-known frequentist rate for integrated mean squared error n -2/5 up to a logarithmic factor.

Robin-Robin Domain Decomposition Methods for the Stokes-Darcy Coupling

Abstract: In this paper we consider a coupled system made of the Stokes and Darcy equations, and we propose some iteration-by-subdomain methods based on Robin conditions on the interface. We prove the convergence of these algorithms, and for suitable finite element approximations we show that the rate of convergence is independent of the mesh size $h$. Special attention is paid to the optimization of the performance of the methods when both the kinematic viscosity $ u$ of the fluid and the hydraulic conductivity tensor $K$ of the porous medium are very small.

Hybridization methods for the analysis of nonlinear systems

Abstract: In this article, we describe some recent results on the hybridization methods for the analysis of nonlinear systems. The main idea of our hybridization approach is to apply the hybrid systems methodology as a systematic approximation method. More concretely, we partition the state space of a complex system into regions that only intersect on their boundaries, and then approximate its dynamics in each region by a simpler one. Then, the resulting hybrid system, which we call a hybridization, is used to yield approximate analysis results for the original system. We also prove important properties of the hybridization, and propose two effective hybridization construction methods, which allow approximating the original nonlinear system with a good convergence rate.

Remarks about the Inviscid Limit of the Navier–Stokes System

Abstract: In this paper we prove two results about the inviscid limit of the Navier-Stokes system. The first one concerns the convergence in H s of a sequence of solutions to the Navier-Stokes system when the viscosity goes to zero and the initial data is in H s . The second result deals with the best rate of convergence for vortex patch initial data in 2 and 3 dimensions. We present here a simple proof which also works in the 3D case. The 3D case is new.

Phase retrieval of arbitrary complex-valued fields through aperture-plane modulation

Abstract: We propose and experimentally demonstrate a simple and robust approach for retrieving arbitrary complex-valued fields from three or more diffraction intensity recordings. We need no a priori knowledge about the object field. The convergence rate is rapid. We obtained good results using experimental data with only 80 iterations (160 fast Fourier transforms). The method does not suffer any stagnation or ambiguity problem, and it also exhibits a high immunity to noise. The technique exhibits great potential in lensless phase-contrast imaging, wave-front sensing, and metrology for a wide spectral range.

Diluted maximum-likelihood algorithm for quantum tomography

Abstract: We propose a refined iterative likelihood-maximization algorithm for reconstructing a quantum state from a set of tomographic measurements. The algorithm is characterized by a very high convergence rate and features a simple adaptive procedure that ensures likelihood increase in every iteration and convergence to the maximum-likelihood state. We apply the algorithm to homodyne tomography of optical states and quantum tomography of entangled spin states of trapped ions and investigate its convergence properties.

Gaussian Mean-Shift Is an EM Algorithm

Abstract: The mean-shift algorithm, based on ideas proposed by Fukunaga and Hosteller, is a hill-climbing algorithm on the density defined by a finite mixture or a kernel density estimate Mean-shift can be used as a nonparametric clustering method and has attracted recent attention in computer vision applications such as image segmentation or tracking We show that, when the kernel is Gaussian, mean-shift is an expectation-maximization (EM) algorithm and, when the kernel is non-Gaussian, mean-shift is a generalized EM algorithm This implies that mean-shift converges from almost any starting point and that, in general, its convergence is of linear order For Gaussian mean-shift, we show: 1) the rate of linear convergence approaches 0 (superlinear convergence) for very narrow or very wide kernels, but is often close to 1 (thus, extremely slow) for intermediate widths and exactly 1 (sublinear convergence) for widths at which modes merge, 2) the iterates approach the mode along the local principal component of the data points from the inside of the convex hull of the data points, and 3) the convergence domains are nonconvex and can be disconnected and show fractal behavior We suggest ways of accelerating mean-shift based on the EM interpretation

Error estimate, optimal shape factor, and high precision computation of multiquadric collocation method

Abstract: Multiquadric (MQ) collocation method is highly efficient for solving partial differential equations due to its exponential error convergence rate. A special feature of the method is that error can be reduced by increasing the value of shape constant c in the MQ basis function, without refining the grid. It is believed that in a numerical solution without roundoff error, infinite accuracy can be achieved by letting c → ∞ . Using the arbitrary precision computation, this paper tests the above conjecture. A sharper error estimate than previously obtained is presented. A formula for a finite, optimal c value that minimizes the solution error for a given grid size is obtained. Using residual errors, constants in error estimate and optimal c formula can be obtained. These results are supported by numerical examples.

Symmetric Indefinite Preconditioners for Saddle Point Problems with Applications to PDE-Constrained Optimization Problems

Abstract: We consider large scale sparse linear systems in saddle point form. A natural property of such indefinite 2-by-2 block systems is the positivity of the (1,1) block on the kernel of the (2,1) block. Many solution methods, however, require that the positivity of the (1,1) block is satisfied everywhere. To enforce the positivity everywhere, an augmented Lagrangian approach is usually chosen. However, the adjustment of the involved parameters is a critical issue. We will present a different approach that is not based on such an explicit augmentation technique. For the considered class of symmetric and indefinite preconditioners, assumptions are presented that lead to symmetric and positive definite problems with respect to a particular scalar product. Therefore, conjugate gradient acceleration can be used. An important class of applications are optimal control problems. It is typical for such problems that the cost functional contains an extra regularization parameter. For control problems with elliptic state equations and distributed control, a special indefinite preconditioner for the discretized problem is constructed, which leads to convergence rates of the preconditioned conjugate gradient method that are not only independent of the mesh size but also independent of the regularization parameter. Numerical experiments are presented for illustrating the theoretical results.

On successive-overrelaxation acceleration of the Hermitian and skew-Hermitian splitting iterations

Abstract: We further generalize the technique for constructing the Hermitian/skew-Hermitian splitting (HSS) iteration method for solving large sparse non-Hermitian positive definite system of linear equations to the normal/skew-Hermitian (NS) splitting obtaining a class of normal/skew-Hermitian splitting (NSS) iteration methods. Theoretical analyses show that the NSS method converges unconditionally to the exact solution of the system of linear equations. Moreover, we derive an upper bound of the contraction factor of the NSS iteration which is dependent solely on the spectrum of the normal splitting matrix, and is independent of the eigenvectors of the matrices involved. We present a successive-overrelaxation (SOR) acceleration scheme for the NSS iteration, which specifically results in an acceleration scheme for the HSS iteration. Convergence conditions for this SOR scheme are derived under the assumption that the eigenvalues of the corresponding block Jacobi iteration matrix lie in certain regions in the complex plane. A numerical example is used to show that the SOR technique can significantly accelerate the convergence rate of the NSS or the HSS iteration method. Copyright © 2006 John Wiley & Sons, Ltd.

Error bounds for monotone approximation schemes for parabolic Hamilton-Jacobi-Bellman equations

Abstract: . We obtain nonsymmetric upper and lower bounds on the rate of convergence of general monotone approximation/numerical schemes for parabolic Hamilton-Jacobi-Bellman equations by introducing a new notion of consistency. Our results are robust and general - they improve and extend earlier results by Krylov, Barles, and Jakobsen. We apply our general results to various schemes including Crank-Nicholson type finite difference schemes, splitting methods, and the classical approximation by piecewise constant controls. In the first two cases our results are new, and in the last two cases the results are obtained by a new method which we develop here.