Showing papers on "Non-linear least squares published in 2010"

Sparse partial least squares regression for simultaneous dimension reduction and variable selection

Abstract: Partial least squares regression has been an alternative to ordinary least squares for handling multicollinearity in several areas of scientific research since the 1960s. It has recently gained much attention in the analysis of high dimensional genomic data. We show that known asymptotic consistency of the partial least squares estimator for a univariate response does not hold with the very large p and small n paradigm. We derive a similar result for a multivariate response regression with partial least squares. We then propose a sparse partial least squares formulation which aims simultaneously to achieve good predictive performance and variable selection by producing sparse linear combinations of the original predictors. We provide an efficient implementation of sparse partial least squares regression and compare it with well-known variable selection and dimension reduction approaches via simulation experiments. We illustrate the practical utility of sparse partial least squares regression in a joint analysis of gene expression and genomewide binding data.

A New Vector Waveform Inversion Algorithm for Simultaneous Updating of Conductivity and Permittivity Parameters From Combination Crosshole/Borehole-to-Surface GPR Data

Abstract: We have developed a new full-waveform groundpenetrating radar (GPR) multicomponent inversion scheme for imaging the shallow subsurface using arbitrary recording configurations. It yields significantly higher resolution images than conventional tomographic techniques based on first-arrival times and pulse amplitudes. The inversion is formulated as a nonlinear least squares problem in which the misfit between observed and modeled data is minimized. The full-waveform modeling is implemented by means of a finite-difference time-domain solution of Maxwell's equations. We derive here an iterative gradient method in which the steepest descent direction, used to update iteratively the permittivity and conductivity distributions in an optimal way, is found by cross-correlating the forward vector wavefield and the backward-propagated vectorial residual wavefield. The formulation of the solution is given in a very general, albeit compact and elegant, fashion. Each iteration step of our inversion scheme requires several calculations of propagating wavefields. Novel features of the scheme compared to previous full-waveform GPR inversions are as follows: 1) The permittivity and conductivity distributions are updated simultaneously (rather than consecutively) at each iterative step using improved gradient and step length formulations; 2) the scheme is able to exploit the full vector wavefield; and 3) various data sets/survey types (e.g., crosshole and borehole-to-surface) can be individually or jointly inverted. Several synthetic examples involving both homogeneous and layered stochastic background models with embedded anomalous inclusions demonstrate the superiority of the new scheme over previous approaches.

A robust methodology for in vivo T1 mapping.

Abstract: In this article, a robust methodology for in vivo T(1) mapping is presented. The approach combines a gold standard scanning procedure with a novel fitting procedure. Fitting complex data to a five-parameter model ensures accuracy and precision of the T(1) estimation. A reduced-dimension nonlinear least squares method is proposed. This method turns the complicated multi-parameter minimization into a straightforward one-dimensional search. As the range of possible T(1) values is known, a global grid search can be used, ensuring that a global optimal solution is found. When only magnitude data are available, the algorithm is adapted to concurrently restore polarity. The performance of the new algorithm is demonstrated in simulations and phantom experiments. The new algorithm is as accurate and precise as the conventionally used Levenberg-Marquardt algorithm but much faster. This gain in speed makes the use of the five-parameter model viable. In addition, the new algorithm does not require initialization of the search parameters. Finally, the methodology is applied in vivo to conventional brain imaging and to skin imaging. T(1) values are estimated for white matter and gray matter at 1.5 T and for dermis, hypodermis, and muscle at 1.5 T, 3 T, and 7 T.

Response-Based Segmentation Using Finite Mixture Partial Least Squares

Abstract: When applying multivariate analysis techniques in information systems and social science disciplines, such as management information systems (MIS) and marketing, the assumption that the empirical data originate from a single homogeneous population is often unrealistic. When applying a causal modeling approach, such as partial least squares (PLS) path modeling, segmentation is a key issue in coping with the problem of heterogeneity in estimated cause-and-effect relationships. This chapter presents a new PLS path modeling approach which classifies units on the basis of the heterogeneity of the estimates in the inner model. If unobserved heterogeneity significantly affects the estimated path model relationships on the aggregate data level, the methodology will allow homogenous groups of observations to be created that exhibit distinctive path model estimates. The approach will, thus, provide differentiated analytical outcomes that permit more precise interpretations of each segment formed. An application on a large data set in an example of the American customer satisfaction index (ACSI) substantiates the methodology’s effectiveness in evaluating PLS path modeling results.

Nonlinear Least Squares for Inverse Problems

Abstract: The first € price and the £ and $ price are net prices, subject to local VAT. Prices indicated with * include VAT for books; the €(D) includes 7% for Germany, the €(A) includes 10% for Austria. Prices indicated with ** include VAT for electronic products; 19% for Germany, 20% for Austria. All prices exclusive of carriage charges. Prices and other details are subject to change without notice. All errors and omissions excepted. G. Chavent Nonlinear Least Squares for Inverse Problems

Generalization of total least-squares on example of unweighted and weighted 2D similarity transformation

Abstract: In this contribution it is shown that the so-called “total least-squares estimate” (TLS) within an errors-in-variables (EIV) model can be identified as a special case of the method of least-squares within the nonlinear Gauss–Helmert model. In contrast to the EIV-model, the nonlinear GH-model does not impose any restrictions on the form of functional relationship between the quantities involved in the model. Even more complex EIV-models, which require specific approaches like “generalized total least-squares” (GTLS) or “structured total least-squares” (STLS), can be treated as nonlinear GH-models without any serious problems. The example of a similarity transformation of planar coordinates shows that the “total least-squares solution” can be obtained easily from a rigorous evaluation of the Gauss–Helmert model. In contrast to weighted TLS, weights can then be introduced without further limitations. Using two numerical examples taken from the literature, these solutions are compared with those obtained from certain specialized TLS approaches.

Robust linear least squares regression

Abstract: We consider the problem of robustly predicting as well as the best linear combination of $d$ given functions in least squares regression, and variants of this problem including constraints on the parameters of the linear combination. For the ridge estimator and the ordinary least squares estimator, and their variants, we provide new risk bounds of order $d/n$ without logarithmic factor unlike some standard results, where $n$ is the size of the training data. We also provide a new estimator with better deviations in the presence of heavy-tailed noise. It is based on truncating differences of losses in a min--max framework and satisfies a $d/n$ risk bound both in expectation and in deviations. The key common surprising factor of these results is the absence of exponential moment condition on the output distribution while achieving exponential deviations. All risk bounds are obtained through a PAC-Bayesian analysis on truncated differences of losses. Experimental results strongly back up our truncated min--max estimator.

Tests for nonlinear cointegration

Abstract: This paper develops tests for the null hypothesis of cointegration in the nonlinear regression model with I(1) variables. The test statistics we use in this paper are Kwiatkowski, Phillips, Schmidt, and Shin’s (1992; KPSS hereafter) tests for the null of stationarity, though using other kinds of tests is also possible. The tests are shown to depend on the limiting distributions of the estimators and parameters of the nonlinear model when they use full-sample residuals from the nonlinear least squares and nonlinear leads-and-lags regressions. This feature makes it difficult to use them in practice. As a remedy, this paper develops tests using subsamples of the regression residuals. For these tests, first, the nonlinear least squares and nonlinear leads-and-lags regressions are run and residuals are calculated. Second, the KPSS tests are applied using subresiduals of size b. As long as b/T → 0 as T → ∞, where T is the sample size, the tests using the subresiduals have limiting distributions that are not affected by the limiting distributions of the full-sample estimators and the parameters of the model. Third, the Bonferroni procedure is used for a selected number of the subresidual-based tests. Monte Carlo simulation shows that the tests work reasonably well in finite samples for polynomial and smooth transition regression models when the block size is chosen by the minimum volatility rule. In particular, the subresidual-based tests using the leads-and-lags regression residuals appear to be promising for empirical work. An empirical example studying the U.S. money demand equation illustrates the use of the tests.

How far is SLAM from a linear least squares problem

Abstract: Most people believe SLAM is a complex nonlinear estimation/optimization problem However, recent research shows that some simple iterative methods based on linearization can sometimes provide surprisingly good solutions to SLAM without being trapped into a local minimum This demonstrates that hidden structure exists in the SLAM problem that is yet to be understood In this paper, we first analyze how far SLAM is from a convex optimization problem Then we show that by properly choosing the state vector, SLAM problem can be formulated as a nonlinear least squares problem with many quadratic terms in the objective function, thus it is clearer how far SLAM is from a linear least squares problem Furthermore, we explain that how the map joining approaches reduce the nonlinearity/nonconvexity of the SLAM problem

Sparse non-linear least squares optimization for geometric vision

Abstract: Several estimation problems in vision involve the minimization of cumulative geometric error using non-linear least-squares fitting. Typically, this error is characterized by the lack of interdependence among certain subgroups of the parameters to be estimated, which leads to minimization problems possessing a sparse structure. Taking advantage of this sparseness during minimization is known to achieve enormous computational savings. Nevertheless, since the underlying sparsity pattern is problem-dependent, its exploitation for a particular estimation problem requires non-trivial implementation effort, which often discourages its pursuance in practice. Based on recent developments in sparse linear solvers, this paper provides an overview of sparseLM, a generalpurpose software package for sparse non-linear least squares that can exhibit arbitrary sparseness and presents results from its application to important sparse estimation problems in geometric vision.

PLS-MGA: A Non-Parametric Approach to Partial Least Squares-based Multi-Group Analysis.

Abstract: This paper adds to an often applied extension of Partial Least Squares (PLS) path modeling, namely the comparison of PLS estimates across subpopulations, also known as multi-group analysis. Existing PLS-based approaches to multi-group analysis have the shortcoming that they rely on distributional assumptions. This paper develops a non-parametric PLS-based approach to multi-group analysis: PLS-MGA. Both the existing approaches and the new approach are applied to a marketing example of customer switching behavior in a liberalized electricity market. This example provides first evidence of favorable operation characteristics of PLS-MGA

On a Global Complexity Bound of the Levenberg-Marquardt Method

Abstract: In this paper, we investigate a global complexity bound of the Levenberg-Marquardt method (LMM) for the nonlinear least squares problem. The global complexity bound for an iterative method solving unconstrained minimization of φ is an upper bound to the number of iterations required to get an approximate solution, such that ‖∇φ(x)‖≤e. We show that the global complexity bound of the LMM is O(e −2).

Evaluation of spectral pretreatments, partial least squares, least squares support vector machines and locally weighted regression for quantitative spectroscopic analysis of soils

Abstract: Soil testing requires the analysis of large numbers of samples in the laboratory that is often time consuming and expensive. Mid-infrared spectroscopy (mid-IR) and near infrared (NIR) spectroscopy are fast, non-destructive and inexpensive analytical methods that have been used for soil analysis, in the laboratory and in the field, to reduce the need for measurements using complex chemical/physical analyses. A comparison of the use of spectral pretreatment as well as the implementation of linear and non-linear regression methods was performed. This study presents an overview of the use of infrared spectroscopy for the prediction of five physical (sand, silt and clay) and chemical (total carbon and total nitrogen) soil parameters with near and mid-infrared units in bench top and field set-ups. Even though no significant differences existed among pretreatment methods, models using second derivatives performed better. The implementation of partial least squares (PLS), least squares support vector machines (LS...

Total Least Squares Approach to Modeling: A Matlab Toolbox

Abstract: This paper deals with a mathematical method known as total least squares or orthogonal regression or error-in-variables method. The mentioned method can be used for modeling of static and also dynamic processes. A wide area of other applications is in signal and image processing. We also present a Matlab toolbox which can solve basic problems related to the total least squares method in the modeling. Several illustrative examples are presented as well. In this paper we present the total least squares method (TLS), which is also known as error-in-variables method or orthogonal regression method. One could hardly name another method which is used as frequently as the method which is known as the least squares method. At the same time, it is difficult to name another method which was accompanied by such strong and long lasting controversy as the least squares method. It is also difficult to find another method, which is so easy and at the same time as artificial as the least squares method. The story of the birth of the least squares method is well covered in the literature and can be summarized as follows [4, 6, 12, 17]. The priority in publication definitely belongs to A. M. Legendre (1805), who also gave the method its famous name, but C. F. Gauss (1809) claimed that he knew and used this method much earlier, about 10 years before the Legendre's publications. Gauss's arguments for his priority were not perfect at all. His diaries with computations claimed to be made by the least squares method were lost. It is also known that when H. C. Schumacher suggested repeating Gauss's lost computations claimed to be done by the least squares method, Gauss totally rejected this idea with the words that such attempts would only suggest that he could not be trusted.

Genetic algorithm interval partial least squares regression combined successive projections algorithm for variable selection in near-infrared quantitative analysis of pigment in cucumber leaves.

Abstract: Variable (or wavelength) selection plays an important role in the quantitative analysis of near-infrared (NIR) spectra. A method based on a genetic algorithm interval partial least squares regression (GAiPLS) combined successive projections algorithm (SPA) was proposed for variable selection in NIR spectroscopy. GAiPLS was used to select informative interval regions among the spectrum, and then SPA was employed to select the most informative variables and to minimize collinearity between those variables in the model. The performance of the proposed method was compared with the full-spectrum model, conventional interval partial least squares regression (iPLS), and backward interval partial least squares regression (BiPLS) for modeling the NIR data sets of pigments in cucumber leaf samples. The multiple linear regression (MLR) model was obtained with eight variables for chlorophylls and five variables for carotenoids selected by SPA. When the SPA model was applied to the prediction of the validation set, the correlation coefficients of the predicted value by MLR and the measured value for the validation data set (rp) of chlorophylls and carotenoids were 0.917 and 0.932, respectively. Results show that the proposed method was able to select important wavelengths from the NIR spectra and makes the prediction more robust and accurate in quantitative analysis.

Convergence of a Regularized Euclidean Residual Algorithm for Nonlinear Least-Squares

Abstract: The convergence properties of the new regularized Euclidean residual method for solving general nonlinear least-squares and nonlinear equation problems are investigated. This method, derived from a proposal by Nesterov [Optim. Methods Softw., 22 (2007), pp. 469-483], uses a model of the objective function consisting of the unsquared Euclidean linearized residual regularized by a quadratic term. At variance with previous analysis, its convergence properties are here considered without assuming uniformly nonsingular globally Lipschitz continuous Jacobians nor an exact subproblem solution. It is proved that the method is globally convergent to first-order critical points and, under stronger assumptions, to roots of the underlying system of nonlinear equations. The rate of convergence is also shown to be quadratic under stronger assumptions.

Singular Spectrum Analysis Based on the Minimum Variance Estimator

Abstract: In recent years Singular Spectrum Analysis (SSA), used as a powerful technique in time series analysis, has been developed and applied to many practical problems. In this paper, the SSA technique based on the minimum variance estimator is introduced. The SSA technique based on the minimum variance and least squares estimators in reconstructing and forecasting time series is also considered. A well-known time series data set, namely, monthly accidental deaths in the USA time series, is used in examining the performance of the technique. The results are compared with several classical methods namely, Box–Jenkins SARIMA models, the ARAR algorithm and the Holt–Winter algorithm.

Optimization of inverse algorithm for estimating the optical properties of biological materials using spatially-resolved diffuse reflectance

Abstract: Determination of the optical properties from intact biological materials based on diffusion approximation theory is a complicated inverse problem, and it requires proper implementation of inverse algorithm, instrumentation and experiment. This article was aimed at optimizing the procedure of estimating the absorption and reduced scattering coefficients of turbid homogeneous media from spatially-resolved diffuse reflectance data. A diffusion model and the inverse algorithm were first validated by Monte Carlo simulations. Sensitivity analysis was performed to gain an insight into the relationship between the estimated parameters and the dependent variables in the inverse algorithm for improving the parameter estimation procedure. Three data transformation and the relative weighting methods were compared in the nonlinear least squares regression. It is found that the logarithm and integral data transformation and relative weighting methods greatly improve estimation accuracy with the relative errors of 10.4%...

Estimating Parameters of a Sine Wave by Separable Nonlinear Least Squares Fitting

Abstract: The four-parameter sine wave (FPSW) is commonly used to characterize noise, signal to noise, and the effective number of bits in analog-to-digital converters. An algorithm based on separable nonlinear least squares fitting is presented. In this algorithm, the frequency is the sole iterative variable. Numerical tests show that if the cycle in the record (CiR) is greater than 1, then the convergence is faster and is less dependent on the CiR and SNR. When the CiR is less than one, the convergence is significantly more sensitive to the CiR, the SNR, and the initial phase. Additionally, the lower the frequency, the more difficult the convergence.

Global Convergence of a New Hybrid Gauss-Newton Structured BFGS Method for Nonlinear Least Squares Problems

Abstract: In this paper, we propose a hybrid Gauss-Newton structured BFGS method with a new update formula and a new switch criterion for the iterative matrix to solve nonlinear least squares problems. We approximate the second term in the Hessian by a positive definite BFGS matrix. Under suitable conditions, global convergence of the proposed method with a backtracking line search is established. Moreover, the proposed method automatically reduces to the Gauss-Newton method for zero residual problems and the structured BFGS method for nonzero residual problems in a neighborhood of an accumulation point. A locally quadratic convergence rate for zero residual problems and a locally superlinear convergence rate for nonzero residual problems are obtained for the proposed method. Some numerical results are given to compare the proposed method with some existing methods.

Toward a solution of allocation in life cycle inventories: the use of least-squares techniques

Abstract: The matrix method for the solution of the so-called inventory problem in LCA generally determines the inventory vector related to a specific system of processes by solving a system of linear equations. The paper proposes a new approach to deal with systems characterized by a rectangular (and thus non-invertible) coefficients matrix. The approach, based on the application of regression techniques, allows solving the system without using computational expedients such as the allocation procedure. The regression techniques used in the paper are (besides the ordinary least squares, OLS) total least squares (TLS) and data least squares (DLS). In this paper, the authors present the application of TLS and DLS to a case study related to the production of bricks, showing the differences between the results accomplished by the traditional matrix approach and those obtained with these techniques. The system boundaries were chosen such that the resulting technology matrix was not too big and thus easy to display, but at the same time complex enough to provide a valid demonstrative example for analyzing the results of the application of the above-described techniques. The results obtained for the case study taken into consideration showed an obvious but not overwhelming difference between the inventory vectors obtained by using the least-squares techniques and those obtained with the solutions based upon allocation. The inventory vectors obtained with the DLS and TLS techniques are closer to those obtained with the physical rather than with the economic allocation. However, this finding most probably cannot be generalized to every inventory problem. Since the solution of the inventory problem in life cycle inventory (LCI) is not a standard forecasting problem because the real solution (the real inventory vector related to the investigated functional unit) is unknown, we are not able to compute a proper performance indicator for the implemented algorithms. However, considering that the obtained least squares solutions are unique and their differences from the traditional solutions are not overwhelming, this methodology is worthy of further investigation. In order to make TLS and DLS techniques a valuable alternative to the traditional allocation procedures, there is a need to optimize them for the very particular systems that commonly occur in LCI, i.e., systems with sparse coefficients matrices and a vector of constants whose entries are almost always all null but one. This optimization is crucial for their applicability in the LCI context.

Robust least squares support vector machine based on recursive outlier elimination

Abstract: To achieve robust estimation for noisy data set, a recursive outlier elimination-based least squares support vector machine (ROELS-SVM) algorithm is proposed in this paper. In this algorithm, statistical information from the error variables of least squares support vector machine is recursively learned and a criterion derived from robust linear regression is employed for outlier elimination. Besides, decremental learning technique is implemented in the recursive training–eliminating stage, which ensures that the outliers are eliminated with low computational cost. The proposed algorithm is compared with re-weighted least squares support vector machine on multiple data sets and the results demonstrate the remarkably robust performance of the ROELS-SVM.

Stability of Householder QR Factorization for Weighted Least Squares Problems

Abstract: For least squares problems in which the rows of the coefficient matrix vary widely in norm, Householder QR factorization (without pivoting) has unsatisfactory backward stability properties. Powell and Reid showed in 1969 that the use of both row and column pivoting leads to a desirable row-wise backward error result. We give a reworked backward error analysis in modern notation and prove two new results. First, sorting the rows by decreasing ∞-norm at the start of the factorization obviates the need for row pivoting. Second, row-wise backward stability is obtained for only one of the two possible choices of sign in the Householder vector.