Showing papers on "Conjugate gradient method published in 2009"

Fast motion deblurring

Abstract: This paper presents a fast deblurring method that produces a deblurring result from a single image of moderate size in a few seconds. We accelerate both latent image estimation and kernel estimation in an iterative deblurring process by introducing a novel prediction step and working with image derivatives rather than pixel values. In the prediction step, we use simple image processing techniques to predict strong edges from an estimated latent image, which will be solely used for kernel estimation. With this approach, a computationally efficient Gaussian prior becomes sufficient for deconvolution to estimate the latent image, as small deconvolution artifacts can be suppressed in the prediction. For kernel estimation, we formulate the optimization function using image derivatives, and accelerate the numerical process by reducing the number of Fourier transforms needed for a conjugate gradient method. We also show that the formulation results in a smaller condition number of the numerical system than the use of pixel values, which gives faster convergence. Experimental results demonstrate that our method runs an order of magnitude faster than previous work, while the deblurring quality is comparable. GPU implementation facilitates further speed-up, making our method fast enough for practical use.

A Randomized Kaczmarz Algorithm with Exponential Convergence

Abstract: The Kaczmarz method for solving linear systems of equations is an iterative algorithm that has found many applications ranging from computer tomography to digital signal processing. Despite the popularity of this method, useful theoretical estimates for its rate of convergence are still scarce. We introduce a randomized version of the Kaczmarz method for consistent, overdetermined linear systems and we prove that it converges with expected exponential rate. Furthermore, this is the first solver whose rate does not depend on the number of equations in the system. The solver does not even need to know the whole system but only a small random part of it. It thus outperforms all previously known methods on general extremely overdetermined systems. Even for moderately overdetermined systems, numerical simulations as well as theoretical analysis reveal that our algorithm can converge faster than the celebrated conjugate gradient algorithm. Furthermore, our theory and numerical simulations confirm a prediction of Feichtinger et al. in the context of reconstructing bandlimited functions from nonuniform sampling.

DL-FIND: An Open-Source Geometry Optimizer for Atomistic Simulations

Abstract: Geometry optimization, including searching for transition states, accounts for most of the CPU time spent in quantum chemistry, computational surface science, and solid-state physics, and also plays an important role in simulations employing classical force fields. We have implemented a geometry optimizer, called DL-FIND, to be included in atomistic simulation codes. It can optimize structures in Cartesian coordinates, redundant internal coordinates, hybrid-delocalized internal coordinates, and also functions of more variables independent of atomic structures. The implementation of the optimization algorithms is independent of the coordinate transformation used. Steepest descent, conjugate gradient, quasi-Newton, and L-BFGS algorithms as well as damped molecular dynamics are available as minimization methods. The partitioned rational function optimization algorithm, a modified version of the dimer method and the nudged elastic band approach provide capabilities for transition-state search. Penalty function, gradient projection, and Lagrange-Newton methods are implemented for conical intersection optimizations. Various stochastic search methods, including a genetic algorithm, are available for global or local minimization and can be run as parallel algorithms. The code is released under the open-source GNU LGPL license. Some selected applications of DL-FIND are surveyed.

Matrices, Moments and Quadrature with Applications

Abstract: This computationally oriented book describes and explains the mathematical relationships among matrices, moments, orthogonal polynomials, quadrature rules, and the Lanczos and conjugate gradient algorithms. The book bridges different mathematical areas to obtain algorithms to estimate bilinear forms involving two vectors and a function of the matrix. The first part of the book provides the necessary mathematical background and explains the theory. The second part describes the applications and gives numerical examples of the algorithms and techniques developed in the first part.Applications addressed in the book include computing elements of functions of matrices; obtaining estimates of the error norm in iterative methods for solving linear systems and computing parameters in least squares and total least squares; and solving ill-posed problems using Tikhonov regularization.This book will interest researchers in numerical linear algebra and matrix computations, as well as scientists and engineers working on problems involving computation of bilinear forms.

Gradient based iterative solutions for general linear matrix equations

Abstract: In this paper, we present a gradient based iterative algorithm for solving general linear matrix equations by extending the Jacobi iteration and by applying the hierarchical identification principle. Convergence analysis indicates that the iterative solutions always converge fast to the exact solutions for any initial values and small condition numbers of the associated matrices. Two numerical examples are provided to show that the proposed algorithm is effective.

Concurrent number cruncher: a GPU implementation of a general sparse linear solver

Abstract: A wide class of numerical methods needs to solve a linear system, where the matrix pattern of non-zero coefficients can be arbitrary. These problems can greatly benefit from highly multithreaded computational power and large memory bandwidth available on graphics processor units (GPUs), especially since dedicated general purpose APIs such as close-to-metal (CTM) (AMD-ATI) and compute unified device architecture (CUDA) (NVIDIA) have appeared. CUDA even provides a BLAS implementation, but only for dense matrices (CuBLAS). Other existing linear solvers for the GPU are also limited by their internal matrix representation. This paper describes how to combine recent GPU programming techniques and new GPU dedicated APIs with high performance computing strategies (namely block compressed row storage (BCRS), register blocking and vectorization), to implement a sparse general-purpose linear solver. Our implementation of the Jacobi-preconditioned conjugate gradient algorithm outperforms by up to a factor of 6.0 × leading-edge CPU counterparts, making it attractive for applications which are content with single precision.

3D magnetic data-space inversion with sparseness constraints

Abstract: I have developed an inversion approach that determines a 3D susceptibility distribution that produces a given magnetic anomaly. The subsurface model consists of a 3D, equally spaced array of dipoles. The inversion incorporates a model norm that enforces sparseness and depth weighting of the solution. Sparseness is imposed by using the Cauchy norm on model parameters. The inverse problem is posed in the data space, leading to a linear system of equations with dimensions based on the number of data, N . This contrasts with the standard least-squares solution, derived through operations within the M -dimensional model space ( M being the number of model parameters). Hence, the data-space method combined with a conjugate gradient algorithm leads to computational efficiency by dealing with an N×N system versus an M×M one, where N≪M . Tests on synthetic data show that sparse inversion produces a much more focused solution compared with a standard model-space, least-squares inversion. The inversion of aeromagnet...

Link Propagation: A Fast Semi-supervised Learning Algorithm for Link Prediction

Abstract: We propose Link Propagation as a new semi-supervised learning method for link prediction problems, where the task is to predict unknown parts of the network structure by using auxiliary information such as node similarities. Since the proposed method can fill in missing parts of tensors, it is applicable to multi-relational domains, allowing us to handle multiple types of links simultaneously. We also give a novel efficient algorithm for Link Propagation based on an accelerated conjugate gradient method.

Modified nonlinear conjugate gradient methods with sufficient descent property for large-scale optimization problems

Abstract: It is well known that the sufficient descent condition is very important to the global convergence of the nonlinear conjugate gradient method. In this paper, some modified conjugate gradient methods which possess this property are presented. The global convergence of these proposed methods with the weak Wolfe–Powell (WWP) line search rule is established for nonconvex function under suitable conditions. Numerical results are reported.

A hybridizable discontinuous Galerkin method for linear elasticity

Abstract: This paper describes the application of the so-called hybridizable discontinuous Galerkin (HDG) method to linear elasticity problems. The method has three significant features. The first is that the only globally coupled degrees of freedom are those of an approximation of the displacement defined solely on the faces of the elements. The corresponding stiffness matrix is symmetric, positive definite, and possesses a block-wise sparse structure that allows for a very efficient implementation of the method. The second feature is that, when polynomials of degree k are used to approximate the displacement and the stress, both variables converge with the optimal order of k+1 for any k⩾0. The third feature is that, by using an element-by-element post-processing, a new approximate displacement can be obtained that converges at the order of k+2, whenever k⩾2. Numerical experiments are provided to compare the performance of the HDG method with that of the continuous Galerkin (CG) method for problems with smooth solutions, and to assess its performance in situations where the CG method is not adequate, that is, when the material is nearly incompressible and when there is a crack. Copyright © 2009 John Wiley & Sons, Ltd.

Speeding up the scaled conjugate gradient algorithm and its application in neuro-fuzzy classifier training

Abstract: The aim of this study is to speed up the scaled conjugate gradient (SCG) algorithm by shortening the training time per iteration. The SCG algorithm, which is a supervised learning algorithm for network-based methods, is generally used to solve large-scale problems. It is well known that SCG computes the second-order information from the two first-order gradients of the parameters by using all the training datasets. In this case, the computation cost of the SCG algorithm per iteration is more expensive for large-scale problems. In this study, one of the first-order gradients is estimated from the previously calculated gradients without using the training dataset. To estimate this gradient, a least square error estimator is applied. The estimation complexity of the gradient is much smaller than the computation complexity of the gradient for large-scale problems, because the gradient estimation is independent of the size of dataset. The proposed algorithm is applied to the neuro-fuzzy classifier and the neural network training. The theoretical basis for the algorithm is provided, and its performance is illustrated by its application to several examples in which it is compared with several training algorithms and well-known datasets. The empirical results indicate that the proposed algorithm is quicker per iteration time than the SCG. The algorithm decreases the training time by 20–50% compared to SCG; moreover, the convergence rate of the proposed algorithm is similar to SCG.

An Overlapping Schwarz Algorithm for Almost Incompressible Elasticity

Abstract: Overlapping Schwarz methods are extended to mixed finite element approximations of linear elasticity which use discontinuous pressure spaces. The coarse component of the preconditioner is based on a low-dimensional space previously developed for scalar elliptic problems and a domain decomposition method of iterative substructuring type, i.e., a method based on nonoverlapping decompositions of the domain, while the local components of the preconditioner are based on solvers on a set of overlapping subdomains. A bound is established for the condition number of the algorithm which grows in proportion to the logarithm of the number of degrees of freedom in individual subdomains and, essentially, to the third power of the relative overlap between the overlapping subdomains, and which is independent of the Poisson ratio as well as jumps in the Lame parameters across the interface between the subdomains. A positive definite reformulation of the discrete problem makes the use of the standard preconditioned conjugate gradient method straightforward. Numerical results, which include a comparison with problems of compressible elasticity, illustrate the findings.

An Improved Super-Resolution Source Reconstruction Method

Abstract: The source reconstruction method (SRM) is a technique developed for antenna diagnostics and for carrying out near-field (NF) to far-field (FF) transformation. The SRM is based on the application of the electromagnetic equivalence principle, in which one establishes an equivalent current distribution that radiates the same fields as the actual currents induced in the antenna under test (AUT). The knowledge of the equivalent currents allows the determination of the antenna radiating elements, as well as the prediction of the AUT-radiated fields outside the equivalent currents domain. The unique feature of the novel methodology presented in this paper is that it can resolve equivalent currents that are smaller than half a wavelength in size, thus providing super-resolution. Furthermore, the measurement field samples can be taken at field spacings greater than half a wavelength, thus going beyond the classical sampling criteria. These two distinctive features are possible due to the choice of a model-based parameter estimation methodology where the unknowns are approximated by a continuous basis and, secondly, through the use of the analytic Green's function. The latter condition also guarantees the invertibility of the electric field operator and provides a stable solution for the currents even when evanescent waves are present in the measurements. In addition, the use of the singular value decomposition in the solution of the matrix equations provides the user with a quantitative tool to assess the quality and the quantity of the measured data. Alternatively, the use of the iterative conjugate gradient (CG) method in solving the ill-conditioned matrix equations can also be implemented. Two examples of an antenna diagnostics method are presented to illustrate the applicability and accuracy of the proposed methodology.

A parallel algebraic multigrid solver on graphics processing units

Abstract: The paper presents a multi-GPU implementation of the preconditioned conjugate gradient algorithm with an algebraic multigrid preconditioner (PCG-AMG) for an elliptic model problem on a 3D unstructured grid. An efficient parallel sparse matrix-vector multiplication scheme underlying the PCG-AMG algorithm is presented for the many-core GPU architecture. A performance comparison of the parallel solver shows that a singe Nvidia Tesla C1060 GPU board delivers the performance of a sixteen node Infiniband cluster and a multi-GPU configuration with eight GPUs is about 100 times faster than a typical server CPU core.

Conjugate gradient method

Abstract: The conjugate gradient (CG) method for optimization and equation solving is described, along with three principal families of algorithms derived from it. In each case, a foundational CG algorithm is formulated mathematically and followed by a brief discussion of refinements and variants within its family. The hierarchical nature of the subject matter is highlighted, because each foundational CG algorithm presented is a descendant of one earlier in the list from which it inherits fundamental algorithmic properties. Copyright © 2009 John Wiley & Sons, Inc. For further resources related to this article, please visit the WIREs website.

Hybrid domain decomposition algorithms for compressible and almost incompressible elasticity

Abstract: Overlapping Schwarz methods are considered for mixed finite element approximations of linear elasticity, with discontinuous pressure spaces, as well as for compressible elasticity approximated by standard conforming finite elements. The coarse components of the preconditioners are based on spaces, with a number of degrees of freedom per subdomain which are uniformly bounded, which are similar to those previously developed for scalar elliptic problems and domain decomposition methods of iterative substructuring type, i.e. methods based on nonoverlapping decompositions of the domain. The local components of the new preconditioners are based on solvers on a set of overlapping subdomains. In the current study, the dimension of the coarse spaces is smaller than in recently developed algorithms; in the compressible case all independent face degrees of freedom have been eliminated while in the almost incompressible case five out of six are not needed. In many cases, this will result in a reduction of the dimension of the coarse space by about one half compared with that of the algorithm previously considered. In addition, in spite of using overlapping subdomains to define the local components of the preconditioner, values of the residual and the approximate solution need only to be retained on the interface between the subdomains in the iteration of the new hybrid Schwarz algorithm. The use of discontinuous pressures makes it possible to work exclusively with symmetric, positive-definite problems and the standard preconditioned conjugate gradient method. Bounds are established for the condition number of the preconditioned operators. The bound for the almost incompressible case grows in proportion to the square of the logarithm of the number of degrees of freedom of individual subdomains and the third power of the relative overlap between the overlapping subdomains, and it is independent of the Poisson ratio as well as jumps in the Lame parameters across the interface between the subdomains. Numerical results illustrate the findings. Copyright © 2009 John Wiley & Sons, Ltd.

A modified PRP conjugate gradient method

Abstract: This paper gives a modified PRP method which possesses the global convergence of nonconvex function and the R-linear convergence rate of uniformly convex function. Furthermore, the presented method has sufficiently descent property and characteristic of automatically being in a trust region without carrying out any line search technique. Numerical results indicate that the new method is interesting for the given test problems.

Modelling of a vertical ground coupled heat pump system by using artificial neural networks

Abstract: This paper describes the applicability of artificial neural networks (ANNs) to estimate of performance of a vertical ground coupled heat pump (VGCHP) system used for cooling and heating purposes experimentally. The system involved three heat exchangers in the different depths at 30 (VB1), 60 (VB2) and 90 (VB3)m. The experimental results were obtained in cooling and heating seasons of 2006-2007. ANNs have been used in varied applications and they have been shown to be particularly useful in system modeling and system identification. In this study, the back-propagation learning algorithm with three different variants, namely Levenberg-Marguardt (LM), Pola-Ribiere conjugate gradient (CGP), and scaled conjugate gradient (SCG), and tangent sigmoid transfer function were used in the network so that the best approach could be found. The most suitable algorithm and neuron number in the hidden layer were found as LM with 8 neurons for both cooling and heating modes.

Hybrid Conjugate Gradient Algorithm for Unconstrained Optimization

Abstract: In this paper a new hybrid conjugate gradient algorithm is proposed and analyzed. The parameter β k is computed as a convex combination of the Polak-Ribiere-Polyak and the Dai-Yuan conjugate gradient algorithms, i.e. β =(1−θ k )β +θ k β . The parameter θ k in the convex combination is computed in such a way that the conjugacy condition is satisfied, independently of the line search. The line search uses the standard Wolfe conditions. The algorithm generates descent directions and when the iterates jam the directions satisfy the sufficient descent condition. Numerical comparisons with conjugate gradient algorithms using a set of 750 unconstrained optimization problems, some of them from the CUTE library, show that this hybrid computational scheme outperforms the known hybrid conjugate gradient algorithms.

Iterative Methods for Finding a Trust-region Step

Abstract: We consider methods for large-scale unconstrained minimization based on finding an approximate minimizer of a quadratic function subject to a two-norm trust-region inequality constraint. The Steihaug-Toint method uses the conjugate-gradient algorithm to minimize the quadratic over a sequence of expanding subspaces until the iterates either converge to an interior point or cross the constraint boundary. The benefit of this approach is that an approximate solution may be obtained with minimal work and storage. However, the method does not allow the accuracy of a constrained solution to be specified. We propose an extension of the Steihaug-Toint method that allows a solution to be calculated to any prescribed accuracy. If the Steihaug-Toint point lies on the boundary, the constrained problem is solved on a sequence of evolving low-dimensional subspaces. Each subspace includes an accelerator direction obtained from a regularized Newton method applied to the constrained problem. A crucial property of this direction is that it can be computed by applying the conjugate-gradient method to a positive-definite system in both the primal and dual variables of the constrained problem. The method includes a parameter that allows the user to take advantage of the tradeoff between the overall number of function evaluations and matrix-vector products associated with the underlying trust-region method. At one extreme, a low-accuracy solution is obtained that is comparable to the Steihaug-Toint point. At the other extreme, a high-accuracy solution can be specified that minimizes the overall number of function evaluations at the expense of more matrix-vector products.

An out-of-core sparse Cholesky solver

Abstract: Direct methods for solving large sparse linear systems of equations are popular because of their generality and robustness. Their main weakness is that the memory they require usually increases rapidly with problem size. We discuss the design and development of the first release of a new symmetric direct solver that aims to circumvent this limitation by allowing the system matrix, intermediate data, and the matrix factors to be stored externally. The code, which is written in Fortran and called HSL_MA77, implements a multifrontal algorithm. The first release is for positive-definite systems and performs a Cholesky factorization. Special attention is paid to the use of efficient dense linear algebra kernel codes that handle the full-matrix operations on the frontal matrix and to the input/output operations. The input/output operations are performed using a separate package that provides a virtual-memory system and allows the data to be spread over many files; for very large problems these may be held on more than one device.Numerical results are presented for a collection of 30 large real-world problems, all of which were solved successfully.