scispace - formally typeset
Search or ask a question

Showing papers on "Non-linear least squares published in 2009"


17 Feb 2009
TL;DR: MPFIT is a port to IDL of the non-linear least squares fitting program MINPACK-1 that inherits the robustness of the original FORTRAN version of MINPack-1, but is optimized for performance and convenience in IDL.
Abstract: MPFIT is a port to IDL of the non-linear least squares fitting program MINPACK-1. MPFIT inherits the robustness of the original FORTRAN version of MINPACK-1, but is optimized for performance and convenience in IDL. In addition to the main fitting engine, MPFIT, several specialized functions are provided to fit 1-D curves and 2-D images/ 1-D and 2-D peaks/ and interactive fitting from the IDL command line. Several constraints can be applied to model parameters, including fixed constraints, simple bounding constraints, and 'tying' the value to another parameter. Several data weighting methods are allowed, and the parameter covariance matrix is computed. Extensive diagnostic capabilities are available during the fit, via a call-back subroutine, and after the fit is complete. Several different forms of documentation are provided, including a tutorial, reference pages, and frequently asked questions. The package has been translated to C and Python as well. The full IDL and C packages can be found at http://purl.com/net/mpfit

658 citations


Journal ArticleDOI
TL;DR: In this article, a more general and practical intensity change model is employed with consideration of the linear intensity change of the deformed image, followed by an iterative least squares algorithm for calculating displacement field with sub-pixel accuracy.

362 citations


Proceedings Article
01 Jan 2009
TL;DR: A new oracle inequality is established for kernelbased, regularized least squares regression methods, which uses the eigenvalues of the associated integral operator as a complexity measure and it turns out that these rates are independent of the exponent of the regularization term.
Abstract: We establish a new oracle inequality for kernelbased, regularized least squares regression methods, which uses the eigenvalues of the associated integral operator as a complexity measure. We then use this oracle inequality to derive learning rates for these methods. Here, it turns out that these rates are independent of the exponent of the regularization term. Finally, we show that our learning rates are asymptotically optimal whenever, e.g., the kernel is continuous and the input space is a compact metric space.

284 citations


Journal ArticleDOI
TL;DR: The results show that both estimators, on average, are able to recover the true location of the single molecule in all scenarios the authors examined, but in the absence of modeling inaccuracies and low noise levels, the maximum likelihood estimator is more accurate than the nonlinear least squares estimator, as measured by the standard deviations of its estimates.
Abstract: Estimating the location of single molecules from microscopy images is a key step in many quantitative single molecule data analysis techniques. Different algorithms have been advocated for the fitting of single molecule data, particularly the nonlinear least squares and maximum likelihood estimators. Comparisons were carried out to assess the performance of these two algorithms in different scenarios. Our results show that both estimators, on average, are able to recover the true location of the single molecule in all scenarios we examined. However, in the absence of modeling inaccuracies and low noise levels, the maximum likelihood estimator is more accurate than the nonlinear least squares estimator, as measured by the standard deviations of its estimates, and attains the best possible accuracy achievable for the sets of imaging and experimental conditions that were tested. Although neither algorithm is consistently superior to the other in the presence of modeling inaccuracies or misspecifications, the maximum likelihood algorithm emerges as a robust estimator producing results with consistent accuracy across various model mismatches and misspecifications. At high noise levels, relative to the signal from the point source, neither algorithm has a clear accuracy advantage over the other. Comparisons were also carried out for two localization accuracy measures derived previously. Software packages with user-friendly graphical interfaces developed for single molecule location estimation (EstimationTool) and limit of the localization accuracy calculations (FandPLimitTool) are also discussed.

170 citations


Book
01 Jan 2009
TL;DR: In this article, a generalization of convex sets, called quasi-convex sets is proposed to solve nonlinear least squares problems, where the convexity of sets is restricted.
Abstract: Nonlinear Least Squares.- Nonlinear Inverse Problems: Examples and Difficulties.- Computing Derivatives.- Choosing a Parameterization.- Output Least Squares Identifiability and Quadratically Wellposed NLS Problems.- Regularization of Nonlinear Least Squares Problems.- A generalization of convex sets.- Quasi-Convex Sets.- Strictly Quasi-Convex Sets.- Deflection Conditions for the Strict Quasi-convexity of Sets.

145 citations


Journal ArticleDOI
TL;DR: In this paper, a weighted least squares solution to general coupled Sylvester matrix equations is proposed to solve the problem, and the optimal step sizes such that the convergence rates of the algorithms are maximized and established.

132 citations


Posted Content
TL;DR: MPFIT as discussed by the authors is a port to IDL of the non-linear least squares fitting program MINPACK-1, which inherits the robustness of the original FORTRAN version of MINPAC, but is optimized for performance and convenience in IDL.
Abstract: MPFIT is a port to IDL of the non-linear least squares fitting program MINPACK-1. MPFIT inherits the robustness of the original FORTRAN version of MINPACK-1, but is optimized for performance and convenience in IDL. In addition to the main fitting engine, MPFIT, several specialized functions are provided to fit 1-D curves and 2-D images; 1-D and 2-D peaks; and interactive fitting from the IDL command line. Several constraints can be applied to model parameters, including fixed constraints, simple bounding constraints, and "tying" the value to another parameter. Several data weighting methods are allowed, and the parameter covariance matrix is computed. Extensive diagnostic capabilities are available during the fit, via a call-back subroutine, and after the fit is complete. Several different forms of documentation are provided, including a tutorial, reference pages, and frequently asked questions. The package has been translated to C and Python as well. The full IDL and C packages can be found at this http URL

131 citations


Journal ArticleDOI
TL;DR: This study describes a general alternative to existing methods for the calibration of NMR diffusion measurements which is parameterised as the exponential of a power series.

110 citations


Journal ArticleDOI
TL;DR: A comparison of the proposed and existing methods shows that the new method is significantly more accurate and robust with regard to noise and the amount of raw data.

101 citations


Journal ArticleDOI
TL;DR: In this article, a nonlinear least squares method for measuring the power system frequency is presented, wherein the voltage at the measurement point is modeled by using the Fourier series. But the robustness of this algorithm with respect to change in various parameters is studied through simulation and the results are validated by hardware implementation using a Virtex IV field-programmable gate array.
Abstract: This paper presents a nonlinear least squares method for measuring the power system frequency, wherein the voltage at the measurement point is modeled by using the Fourier series. The estimation of the fundamental frequency is a nonlinear problem in this formulation and is solved by performing a 1-D search over the range of allowed frequency variation. The voltage signal is used for frequency estimation because it is typically less distorted than the line current, resulting in computational efficiency. The robustness of this algorithm with respect to change in various parameters is studied through simulation and the results are validated by hardware implementation using a Virtex IV field-programmable gate array. An application of this algorithm to a shunt active power filter is also presented.

100 citations


Book
14 Dec 2009
TL;DR: In this article, the authors introduce different mathematical methods of scientific computation to solve minimization problems using examples ranging from locating an aircraft, finding the best time to replace a computer, analyzing developments on the stock market, and constructing phylogenetic trees.
Abstract: Using real-life applications, this graduate-level textbook introduces different mathematical methods of scientific computation to solve minimization problems using examples ranging from locating an aircraft, finding the best time to replace a computer, analyzing developments on the stock market, and constructing phylogenetic trees. The textbook focuses on several methods, including nonlinear least squares with confidence analysis, singular value decomposition, best basis, dynamic programming, linear programming, and various optimization procedures. Each chapter solves several realistic problems, introducing the modeling optimization techniques and simulation as required. This allows readers to see how the methods are put to use, making it easier to grasp the basic ideas. There are also worked examples, practical notes, and background materials to help the reader understand the topics covered. Interactive exercises are available at www.cambridge.org/9780521849890.

Journal ArticleDOI
TL;DR: The method uses a statistical linear regression technique which is based on the orthogonal least squares (OLS) algorithm, substituting a QR algorithm for the traditional Gram-Schmidt algorithm, to find the connected weight of the hidden layer neurons.
Abstract: In this paper we present a method for improving the generalization performance of a radial basis function (RBF) neural network. The method uses a statistical linear regression technique which is based on the orthogonal least squares (OLS) algorithm. We first discuss a modified way to determine the center and width of the hidden layer neurons. Then, substituting a QR algorithm for the traditional Gram-Schmidt algorithm, we find the connected weight of the hidden layer neurons. Cross-validation is utilized to determine the stop training criterion. The generalization performance of the network is further improved using a bootstrap technique. Finally, the solution method is used to solve a simulation and a real problem. The results demonstrate the improved generalization performance of our algorithm over the existing methods.

Journal ArticleDOI
TL;DR: Least squares percentage regression as discussed by the authors is a method of least squares regression based on percentage errors, which is not to be confused with semi-log regression, and is linked to the multiplicative error model in the same way that the standard additive error model was linked to ordinary least square regression.
Abstract: When using a model for prediction, or for representing the data, the percentage error may be more important than the absolute error. We therefore present the method of least squares regression based on percentage errors. Exact expressions are derived for the coefficients, and we show how models can be estimated easily using existing regression software. (The proposed method is not to be confused with semi-log regression.) Least squares percentage regression is linked to the multiplicative error model in the same way that the standard additive error model is linked to ordinary least squares regression. The method should therefore also prove useful when the data does not have constant variance (heteroscedastic data). The coefficients are shown to be unbiased. When the relative error is normally distributed, least squares percentage regression is shown to provide maximum likelihood estimates.

Journal ArticleDOI
15 May 2009
TL;DR: This study shows that this algorithm allows for surface tension measurements corresponding to Bond numbers previously shown to be ill suited for pendant drop measurements.
Abstract: The pendant drop method is one of the most widely used techniques to measure the surface tension between gas-liquid and liquid-liquid interfaces. The method consists of fitting the Young-Laplace equation to the digitized shape of a drop suspended from the end of a capillary tube. The first use of digital computers to solve this problem utilized nonlinear least squares fitting and since then numerous subroutines and algorithms have been reported for improving efficiency and accuracy. However, current algorithms which rely on gradient based methods have difficulty converging for almost spherical drop shapes (i.e. low Bond numbers). We present a non-gradient based algorithm based on the Nelder-Mead simplex method to solve the least squares problem. The main advantage of using a non-gradient based fitting routine is that it is robust against poor initial guesses and works for almost spherical bubble shapes. We have tested the algorithm against theoretical and experimental drop shapes to demonstrate both the efficiency and the accuracy of the fitting routine for a wide range of Bond numbers. Our study shows that this algorithm allows for surface tension measurements corresponding to Bond numbers previously shown to be ill suited for pendant drop measurements.

Journal ArticleDOI
TL;DR: A semidefinite relaxation RD-based positioning algorithm, which makes use of the admissible source position information, is proposed and its estimation performance is contrasted with the two-step weighted least squares method and nonlinear least squares estimator as well as Cramer-Rao lower bound.
Abstract: A common technique for passive source localization is to utilize the range-difference (RD) measurements between the source and several spatially separated sensors. The RD information defines a set of hyperbolic equations from which the source position can be calculated with the knowledge of the sensor positions. Under the standard assumption of Gaussian distributed RD measurement errors, it is well known that the maximum-likelihood (ML) position estimation is achieved by minimizing a multimodal cost function which corresponds to a difficult task. In this correspondence, we propose to approximate the nonconvex ML optimization by relaxing it to a convex optimization problem using semidefinite programming. A semidefinite relaxation RD-based positioning algorithm, which makes use of the admissible source position information, is proposed and its estimation performance is contrasted with the two-step weighted least squares method and nonlinear least squares estimator as well as Cramer-Rao lower bound.

Journal ArticleDOI
TL;DR: In this paper, a covariate-adjusted nonlinear regression model is proposed to estimate the distorting functions by nonparametrically regressing the predictors and response on a distorting covariate; then, nonlinear least squares estimators for the parameters are obtained using the estimated response and predictors.
Abstract: In this paper, we propose a covariate-adjusted nonlinear regression model. In this model, both the response and predictors can only be observed after being distorted by some multiplicative factors. Because of nonlinearity, existing methods for the linear setting cannot be directly employed. To attack this problem, we propose estimating the distorting functions by nonparametrically regressing the predictors and response on the distorting covariate; then, nonlinear least squares estimators for the parameters are obtained using the estimated response and predictors. Root $n$-consistency and asymptotic normality are established. However, the limiting variance has a very complex structure with several unknown components, and confidence regions based on normal approximation are not efficient. Empirical likelihood-based confidence regions are proposed, and their accuracy is also verified due to its self-scale invariance. Furthermore, unlike the common results derived from the profile methods, even when plug-in estimates are used for the infinite-dimensional nuisance parameters (distorting functions), the limit of empirical likelihood ratio is still chi-squared distributed. This property eases the construction of the empirical likelihood-based confidence regions. A simulation study is carried out to assess the finite sample performance of the proposed estimators and confidence regions. We apply our method to study the relationship between glomerular filtration rate and serum creatinine.

Journal ArticleDOI
TL;DR: In this article, seven calibration equations were selected to evaluate the fitting agreement of the resistance-temperature data of four types of thermistors in this study and the parameters of these calibration equations are estimated using least squares method.

Journal ArticleDOI
TL;DR: An efficient iterative approach to solving separable nonlinear least squares problems that arise in large-scale inverse problems using a variable projection Gauss-Newton method and Tikhonov regularization is presented, providing a nonlinear solver that requires very little input from the user.
Abstract: We present an efficient iterative approach to solving separable nonlinear least squares problems that arise in large-scale inverse problems. A variable projection Gauss-Newton method is used to solve the nonlinear least squares problem, and Tikhonov regularization is incorporated using an iterative hybrid scheme. Regularization parameters are chosen automatically using a weighted generalized cross validation method, thus providing a nonlinear solver that requires very little input from the user. Applications from image deblurring and digital tomosynthesis illustrate the effectiveness of the resulting numerical scheme.

Journal ArticleDOI
TL;DR: The proposed method significantly improves the quality of parameter estimation as the amplitude of the errors in variables becomes larger, and the optimal number of fiber families in the multi-fiber family model with respect to the experimental data balancing between variance and bias errors is investigated.

Journal ArticleDOI
TL;DR: The simulation studies indicate that the proposed algorithms can effectively estimate the parameters of the C-ARMA models.
Abstract: This paper presents a two-stage least squares based iterative algorithm, a residual based interactive least squares algorithm and a residual based recursive least squares algorithm for identifying controlled autoregressive moving average (C-ARMA) models. The simulation studies indicate that the proposed algorithms can effectively estimate the parameters of the C-ARMA models.

Journal ArticleDOI
TL;DR: Multi-platform software has been developed for the analysis of powder diffraction data, with particular focus on structure solution, which provides a Rietveld optimization engine with the possibility of refining parameters describing both the sample and the instrument model.
Abstract: Multi-platform software has been developed for the analysis of powder diffraction data, with particular focus on structure solution. The program provides a Rietveld optimization engine, with the possibility of refining parameters describing both the sample and the instrument model. Geometric constraints such as rigid fragments and torsion angles can be defined for the atomic structure, to reduce the number of degrees of freedom of the model. An innovative hierarchical description of the asymmetric unit has been adopted, which allows, in principle, the definition of arbitrarily complex geometric relationships. Additionally, global optimization algorithms may be used in place of the standard nonlinear least squares, when particularly challenging problems are being faced.

Journal ArticleDOI
TL;DR: A class of error estimates previously introduced by the authors are extended to the least squares solution of consistent and inconsistent linear systems, and their application to various direct and iterative regularization methods is discussed.
Abstract: The a posteriori estimate of the errors in the numerical solution of ill-conditioned linear systems with contaminated data is a complicated problem. Several estimates of the norm of the error have been recently introduced and analyzed, under the assumption that the matrix is square and nonsingular. In this paper we study the same problem in the case of a rectangular and, in general, rank-deficient matrix. As a result, a class of error estimates previously introduced by the authors (Brezinski et al., Numer Algorithms, in press, 2008) are extended to the least squares solution of consistent and inconsistent linear systems. Their application to various direct and iterative regularization methods are also discussed, and the numerical effectiveness of these error estimates is pointed out by the results of an extensive experimentation.

Journal ArticleDOI
TL;DR: A generalized iterated conditional modes (ICM) algorithm operating on blocks instead of sites is proposed which is shown to converge considerably faster than the conventional ICM algorithm and show a clear reduction of RMSE and variance as well as, in some cases, reduced estimation bias.
Abstract: Dynamic contrast-enhanced magnetic resonance (DCE-MR) imaging can be used to study microvascular structure in vivo by monitoring the abundance of an injected diffusible contrast agent over time. The resulting spatially resolved intensity-time curves are usually interpreted in terms of kinetic parameters obtained by fitting a pharmacokinetic model to the observed data. Least squares estimates of the highly nonlinear model parameters, however, can exhibit high variance and can be severely biased. As a remedy, we bring to bear spatial prior knowledge by means of a generalized Gaussian Markov random field (GGMRF). By using information from neighboring voxels and computing the maximum a posteriori solution for entire parameter maps at once, both bias and variance of the parameter estimates can be reduced thus leading to smaller root mean square error (RMSE). Since the number of variables gets very big for common image resolutions, sparse solvers have to be employed. To this end, we propose a generalized iterated conditional modes (ICM) algorithm operating on blocks instead of sites which is shown to converge considerably faster than the conventional ICM algorithm. Results on simulated DCE-MR images show a clear reduction of RMSE and variance as well as, in some cases, reduced estimation bias. The mean residual bias (MRB) is reduced on the simulated data as well as for all 37 patients of a prostate DCE-MRI dataset. Using the proposed algorithm, average computation times only increase by a factor of 1.18 (871 ms per voxel) for a Gaussian prior and 1.51 (1.12 s per voxel) for an edge-preserving prior compared to the single voxel approach (740 ms per voxel).

Proceedings ArticleDOI
14 Jun 2009
TL;DR: This paper shows that under a mild condition, a class of generalized eigenvalue problems in machine learning can be formulated as a least squares problem, and reports experimental results that confirm the established equivalence relationship.
Abstract: Many machine learning algorithms can be formulated as a generalized eigenvalue problem. One major limitation of such formulation is that the generalized eigenvalue problem is computationally expensive to solve especially for large-scale problems. In this paper, we show that under a mild condition, a class of generalized eigenvalue problems in machine learning can be formulated as a least squares problem. This class of problems include classical techniques such as Canonical Correlation Analysis (CCA), Partial Least Squares (PLS), and Linear Discriminant Analysis (LDA), as well as Hypergraph Spectral Learning (HSL). As a result, various regularization techniques can be readily incorporated into the formulation to improve model sparsity and generalization ability. In addition, the least squares formulation leads to efficient and scalable implementations based on the iterative conjugate gradient type algorithms. We report experimental results that confirm the established equivalence relationship. Results also demonstrate the efficiency and effectiveness of the equivalent least squares formulations on large-scale problems.

Proceedings ArticleDOI
05 Jan 2009
TL;DR: In this article, a local extremum diminishing property (LEM) based algorithm for meshless Euler equations is presented. The algorithm is suitable for use with many meshless schemes, three of which are detailed here.
Abstract: An algorithm for use with several meshless schemes is presented based on a local extremum diminishing property. The scheme is applied to the Euler equations in two dimensions. The algorithm is suitable for use with many meshless schemes, three of which are detailed here. First, a method based on Taylor series expansion and least squares is highlighted. Next, a similar least squares method is used, but using polynomial basis functions with fixed Gaussian weighting. A third method makes use of the Hardy multiquadric radial basis functions on a local cloud of points. Results indicate that all three methods perform essentially equally well for flows without shocks. For flows with shocks, the least squares methods perform significantly better than the radial basis method, which displays discrepancies in shock location and magnitude. All methods are compared to an established finite volume method for validation purposes.

Journal ArticleDOI
TL;DR: In this article, a mesh-free discrete least squares meshless (DLSM) method is presented for the solution of elliptic partial differential equations, where the computational domain is discretized by some nodes and the set of simultaneous algebraic equations are built by minimizing a least square functional with respect to the nodal parameters.
Abstract: A meshfree method namely, discrete least squares meshless (DLSM) method, is presented in this paper for the solution of elliptic partial differential equations. In this method, computational domain is discretized by some nodes and the set of simultaneous algebraic equations are built by minimizing a least squares functional with respect to the nodal parameters. The least squares functional is defined as the sum of squared residuals of the differential equation and its boundary condition calculated at a set of points called sampling points, generally different from nodal points. A moving least squares (MLS) technique is used to construct the shape functions. The proposed method automatically leads to symmetric and positive-definite system of equations. The proposed method does not need any background mesh and, therefore, it is a truly meshless method. The solutions of several one- and two-dimensional examples of elliptic partial differential equations are presented to illustrate the performance of the proposed method. Sensitivity analysis on the parameters of the method is also carried out and the results are presented.

Journal ArticleDOI
TL;DR: The weighted least-squares analysis is employed to calculate the differences in the storage product concentration between the model predictions and the experimental data as the sum of squared weighted errors, suggesting that this approach could be useful to evaluate the product formation kinetics of mixed cultures like activated sludge.

Journal ArticleDOI
TL;DR: In this article, the authors investigate the biases in the estimation of the b-value of the Gutenberg-Richter law and its uncertainty made through the least squares technique and show that the least square technique should not be used to determine the slope of the GRL.
Abstract: We investigate conceptually, analytically, and numerically the biases in the estimation of the b-value of the Gutenberg-Richter law and of its uncertainty made through the least squares technique. The biases are introduced by the cumulation operation for the cumulative form of the Gutenberg-Richter law, by the logarithmic transformation, and by the measurement errors on the magnitude. We find that the least squares technique, applied to the cumulative and binned form of the Gutenberg-Richter law, produces strong bias in the b-value and its uncertainty, whose amplitudes depend on the size of the sample. Furthermore, the logarithmic transformation produces two different endemic bends in the Log(N) versus M curve. This means that this plot might produce fake significant departures from the Gutenberg-Richter law. The effect of the measurement errors is negligible compared to those of cumulation operation and logarithmic transformation. The results obtained show that the least squares technique should never be used to determine the slope of the Gutenberg-Richter law and its uncertainty.

Journal ArticleDOI
TL;DR: In this article, a distribution-weighted least squares estimator is proposed to recover directions in the central subspace, then use the distribution weighted estimator as a seed vector and project it onto a Krylov space by partial least squares to avoid computing the inverse of the covariance of predictors.
Abstract: Summary. Because highly correlated data arise from many scientific fields, we investigate parameter estimation in a semiparametric regression model with diverging number of predictors that are highly correlated. For this, we first develop a distribution-weighted least squares estimator that can recover directions in the central subspace, then use the distribution-weighted least squares estimator as a seed vector and project it onto a Krylov space by partial least squares to avoid computing the inverse of the covariance of predictors. Thus, distrbution-weighted partial least squares can handle the cases with high dimensional and highly correlated predictors. Furthermore, we also suggest an iterative algorithm for obtaining a better initial value before implementing partial least squares. For theoretical investigation, we obtain strong consistency and asymptotic normality when the dimension p of predictors is of convergence rate O{n1/2/ log (n)} and o(n1/3) respectively where n is the sample size. When there are no other constraints on the covariance of predictors, the rates n1/2 and n1/3 are optimal. We also propose a Bayesian information criterion type of criterion to estimate the dimension of the Krylov space in the partial least squares procedure. Illustrative examples with a real data set and comprehensive simulations demonstrate that the method is robust to non-ellipticity and works well even in ‘small n–large p’ problems.