Showing papers on "Non-linear least squares published in 2014"

Partial least squares discriminant analysis: taking the magic away

Abstract: Partial least squares discriminant analysis (PLS-DA) has been available for nearly 20 years yet is poorly understood by most users. By simple examples, it is shown graphically and algebraically that for two equal class sizes, PLS-DA using one partial least squares (PLS) component provides equivalent classification results to Euclidean distance to centroids, and by using all nonzero components to linear discriminant analysis. Extensions where there are unequal class sizes and more than two classes are discussed including common pitfalls and dilemmas. Finally, the problems of overfitting and PLS scores plots are discussed. It is concluded that for classification purposes, PLS-DA has no significant advantages over traditional procedures and is an algorithm full of dangers. It should not be viewed as a single integrated method but as step in a full classification procedure. However, despite these limitations, PLS-DA can provide good insight into the causes of discrimination via weights and loadings, which gives it a unique role in exploratory data analysis, for example in metabolomics via visualisation of significant variables such as metabolites or spectroscopic peaks. Copyright © 2014 John Wiley & Sons, Ltd.

A new perspective on least squares under convex constraint

Abstract: Consider the problem of estimating the mean of a Gaussian random vector when the mean vector is assumed to be in a given convex set. The most natural solution is to take the Euclidean projection of the data vector on to this convex set; in other words, performing "least squares under a convex constraint." Many problems in modern statistics and statistical signal processing theory are special cases of this general situation. Examples include the lasso and other high-dimensional regression techniques, function estimation problems, matrix estimation and completion, shape-restricted regression, constrained denoising, linear inverse problems, etc. This paper presents three general results about this problem, namely, (a) an exact computation of the main term in the estimation error by relating it to expected maxima of Gaussian processes (existing results only give upper bounds), (b) a theorem showing that the least squares estimator is always admissible up to a universal constant in any problem of the above kind and (c) a counterexample showing that least squares estimator may not always be minimax rate-optimal. The result from part (a) is then used to compute the error of the least squares estimator in two examples of contemporary interest.

Least squares algorithm for an input nonlinear system with a dynamic subspace state space model

Abstract: For a Hammerstein input nonlinear system with a subspace state space linear element, this paper transforms the system into a bilinear identification model by using the property of the shift operator to the state space model and presents a recursive and an iterative least squares algorithms to generate parameter estimates and state estimates by using the hierarchical identification principle and by replacing the unknown state variables with their estimates. The proposed approaches are computationally more efficient than the over-parameterization model based least squares method.

Least Squares Estimation

Abstract: The method of least squares is about estimating parameters by minimizing the squared discrepancies between observed data on the one hand, and their expected values on the other. It is commonly used in the context of a regression model, where one observes pairs of variables (X, Y), with X a covariable and Y a response variable. The expectation of the response variable Y given the covariable X is modeled as a function fβ(X) depending on parameters β. We present an explicit expression for the least squares estimator of β in the linear regression model, and also give its covariance matrix. The calculations are illustrated with a numerical example. The F statistic for testing linear hypotheses on β is also given. We conclude with some extensions. Keywords: least squares; linear hypothesis; regression

Sparse Recovery by Means of Nonnegative Least Squares

Abstract: This letter demonstrates that sparse recovery can be achieved by an L1-minimization ersatz easily implemented using a conventional nonnegative least squares algorithm. A connection with orthogonal matching pursuit is also highlighted. The preliminary results call for more investigations on the potential of the method and on its relations to classical sparse recovery algorithms.

Extracting the parameters of the double-dispersion Cole bioimpedance model from magnitude response measurements.

Abstract: In the field of bioimpedance measurements, the Cole impedance model is widely used for characterizing biological tissues and biochemical materials. In this work, a nonlinear least squares fitting is applied to extract the double-dispersion Cole impedance parameters from simulated magnitude response datasets without requiring the direct impedance data or phase information. The technique is applied to extract the impedance parameters from MATLAB simulated noisy magnitude datasets showing less than 1.2 % relative error when 60 dB SNR Gaussian white noise is present. This extraction is verified experimentally using apples as the Cole impedances showing less than 3 % relative error between simulated responses (using the extracted impedance parameters) and the experimental results over the entire dataset.

Iterative algorithm for weighted total least squares adjustment

Abstract: In this contribution, an iterative algorithm is developed for parameter estimation in a nonlinear measurement error model y2e5(A2EA)x, which is based on the complete description of the variance–covariance matrices of the observation errors e and of the coefficient matrix errors EA without any restriction, e.g. in the case that there are correlations among observations. This paper derives the weighted total least squares solution without applying Lagrange multipliers in a straightforward manner. The algorithm is simple in the concept, easy in the implementation, and fast in the convergence. The final exact solution can be achieved through iteration. Based on the similarity between the proposed algorithm and the ordinary least squares method, the estimate for the covariance matrix of the unknown parameters can be analogously computed by using the error propagation law. The efficacy of the proposed WTLS algorithm is demonstrated by solving three WTLS problems, i.e. a linear regression model, a planar similarity transformation and twodimensional affine transformation in the case of diagonal and fully populated covariance matrices in both start and transformed coordinate systems.

Analysis of the Gradient-Descent Total Least-Squares Adaptive Filtering Algorithm

Abstract: The gradient-descent total least-squares (GD-TLS) algorithm is a stochastic-gradient adaptive filtering algorithm that compensates for error in both input and output data. We study the local convergence of the GD-TLS algoritlun and find bounds for its step-size that ensure its stability. We also analyze the steady-state performance of the GD-TLS algorithm and calculate its steady-state mean-square deviation. Our steady-state analysis is inspired by the energy-conservation-based approach to the performance analysis of adaptive filters. The results predicted by the analysis show good agreement with the simulation experiments.

Robust non-convex least squares loss function for regression with outliers

Abstract: In this paper, we propose a robust scheme for least squares support vector regression (LS-SVR), termed as RLS-SVR, which employs non-convex least squares loss function to overcome the limitation of LS-SVR that it is sensitive to outliers Non-convex loss gives a constant penalty for any large outliers The proposed loss function can be expressed by a difference of convex functions (DC) The resultant optimization is a DC program It can be solved by utilizing the Concave–Convex Procedure (CCCP) RLS-SVR iteratively builds the regression function by solving a set of linear equations at one time The proposed RLS-SVR includes the classical LS-SVR as its special case Numerical experiments on both artificial datasets and benchmark datasets confirm the promising results of the proposed algorithm

New Bounds on Compressive Linear Least Squares Regression

Abstract: In this paper we provide a new analysis of compressive least squares regression that removes a spurious log N factor from previous bounds, where N is the number of training points. Our new bound has a clear interpretation and reveals meaningful structural properties of the linear regression problem that makes it solvable effectively in a small dimensional random subspace. In addition, the main part of our analysis does not require the compressive matrix to have the JohnsonLindenstrauss property, or the RIP property. Instead, we only require its entries to be drawn i.i.d. from a 0-mean symmetric distribution with finite first four moments.

Supervised Descent Method for Solving Nonlinear Least Squares Problems in Computer Vision

Abstract: Many computer vision problems (e.g., camera calibration, image alignment, structure from motion) are solved with nonlinear optimization methods. It is generally accepted that second order descent methods are the most robust, fast, and reliable approaches for nonlinear optimization of a general smooth function. However, in the context of computer vision, second order descent methods have two main drawbacks: (1) the function might not be analytically differentiable and numerical approximations are impractical, and (2) the Hessian may be large and not positive definite. To address these issues, this paper proposes generic descent maps, which are average "descent directions" and rescaling factors learned in a supervised fashion. Using generic descent maps, we derive a practical algorithm - Supervised Descent Method (SDM) - for minimizing Nonlinear Least Squares (NLS) problems. During training, SDM learns a sequence of decent maps that minimize the NLS. In testing, SDM minimizes the NLS objective using the learned descent maps without computing the Jacobian or the Hessian. We prove the conditions under which the SDM is guaranteed to converge. We illustrate the effectiveness and accuracy of SDM in three computer vision problems: rigid image alignment, non-rigid image alignment, and 3D pose estimation. In particular, we show how SDM achieves state-of-the-art performance in the problem of facial feature detection. The code has been made available at www.humansensing.cs.cmu.edu/intraface.

Polynomial Phase Estimation by Least Squares Phase Unwrapping

Abstract: Estimating the coefficients of a noisy polynomial phase signal is important in fields including radar, biology and radio communications. One approach attempts to perform polynomial regression on the phase of the signal. This is complicated by the fact that the phase is wrapped modulo 2π and must be unwrapped before regression can be performed. In this paper, we consider an estimator that performs phase unwrapping in a least squares manner. We call this the least squares unwrapping (LSU) estimator. The LSU estimator can be computed in a reasonable amount of time for data sets of moderate size using existing general purpose algorithms from algebraic number theory. Under mild conditions on the distribution of the noise we describe the asymptotic properties of this estimator, showing that it is strongly consistent and asymptotically normally distributed. A key feature is that the LSU estimator is accurate over a far wider range of parameters than many popular existing estimators. Monte-Carlo simulations support our theoretical results and demonstrate the excellent statistical performance of the LSU estimator when compared with existing state-of-the-art estimators.

Least squares approximated stability boundaries of milling process

Abstract: First and second order least squares methods are used in generating simple approximation polynomials for the state term of the model for regenerative chatter in the milling process. The least squares approximation of delayed state term and periodic term of the model does not go beyond first order. The resulting discrete maps are demonstrated to have same convergence rate as the discrete maps in other works that are based on the interpolation theory. The presented discrete maps are illustrated to be beneficial in terms of computational time (CT) savings that derive from reduction in the number of calculation needed for generation system monodromy matrix. This benefit is so much that computational time of second order least squares-based discrete map is noticeably shorter than that of first order interpolation-based discrete map. It is expected from analysis then verified numerically that savings in CT due to use of least squares theory relative to use of interpolation theory of same order rises with rise in order of approximation. The experimentally determined model parameters used for numerical calculations are extracted from literature.

Numerical Solution of Fractional Integro-Differential Equations by Least Squares Method and Shifted Chebyshev Polynomial

Abstract: We investigate the numerical solution of linear fractional integro-differential equations by least squares method with aid of shifted Chebyshev polynomial. Some numerical examples are presented to illustrate the theoretical results.

The Effects of Chance Correlations on Partial Least Squares Path Modeling

Abstract: Partial least squares path modeling (PLS) has been increasing in popularity as a form of or an alternative to structural equation modeling (SEM) and has currently considerable momentum in some mana...

Sampling Minimal Subsets with Large Spans for Robust Estimation

Abstract: When sampling minimal subsets for robust parameter estimation, it is commonly known that obtaining an all-inlier minimal subset is not sufficient; the points therein should also have a large spatial extent. This paper investigates a theoretical basis behind this principle, based on a little known result which expresses the least squares regression as a weighted linear combination of all possible minimal subset estimates. It turns out that the weight of a minimal subset estimate is directly related to the span of the associated points. We then derive an analogous result for total least squares which, unlike ordinary least squares, corrects for errors in both dependent and independent variables. We establish the relevance of our result to computer vision by relating total least squares to geometric estimation techniques. As practical contributions, we elaborate why naive distance-based sampling fails as a strategy to maximise the span of all-inlier minimal subsets produced. In addition we propose a novel method which, unlike previous methods, can consciously target all-inlier minimal subsets with large spans.

An Efficient Variable Projection Formulation for Separable Nonlinear Least Squares Problems

Abstract: We consider in this paper a class of nonlinear least squares problems in which the model can be represented as a linear combination of nonlinear functions. The variable projection algorithm projects the linear parameters out of the problem, leaving the nonlinear least squares problems involving only the nonlinear parameters. To implement the variable projection algorithm more efficiently, we propose a new variable projection functional based on matrix decomposition. The advantage of the proposed formulation is that the size of the decomposed matrix may be much smaller than those of previous ones. The Levenberg-Marquardt algorithm using finite difference method is then applied to minimize the new criterion. Numerical results show that the proposed approach achieves significant reduction in computing time.

A Recursive Restricted Total Least-Squares Algorithm

Abstract: We show that the generalized total least squares (GTLS) problem with a singular noise covariance matrix is equivalent to the restricted total least squares (RTLS) problem and propose a recursive method for its numerical solution. The method is based on the generalized inverse iteration. The estimation error covariance matrix and the estimated augmented correction are also characterized and computed recursively. The algorithm is cheap to compute and is suitable for online implementation. Simulation results in least squares (LS), data least squares (DLS), total least squares (TLS), and restricted total least squares (RTLS) noise scenarios show fast convergence of the parameter estimates to their optimal values obtained by corresponding batch algorithms.

Estimation of State of Charge for Lithium-Ion Battery Based on Finite Difference Extended Kalman Filter

Abstract: An accurate estimation of the state of charge (SOC) of the battery is of great significance for safe and efficient energy utilization of electric vehicles. Given the nonlinear dynamic system of the lithium-ion battery, the parameters of the second-order RC equivalent circuit model were calibrated and optimized using a nonlinear least squares algorithm in the Simulink parameter estimation toolbox. A comparison was made between this finite difference extended Kalman filter (FDEKF) and the standard extended Kalman filter in the SOC estimation. The results show that the model can essentially predict the dynamic voltage behavior of the lithium-ion battery, and the FDEKF algorithm can maintain good accuracy in the estimation process and has strong robustness against modeling error.