Showing papers on "Non-linear least squares published in 2016"
TL;DR: The proposed algorithm has lower computational cost than the existing over-parameterization model-based RLS algorithm and the simulation results indicate that the proposed algorithm can effectively estimate the parameters of the nonlinear systems.
Abstract: In this paper, we study the parameter estimation problem of a class of output nonlinear systems and propose a recursive least squares (RLS) algorithm for estimating the parameters of the nonlinear systems based on the model decomposition. The proposed algorithm has lower computational cost than the existing over-parameterization model-based RLS algorithm. The simulation results indicate that the proposed algorithm can effectively estimate the parameters of the nonlinear systems.
121 citations
01 Jan 2016
TL;DR: In this paper, the problem of estimating a step function in the presence of additive measurement noise is considered and the least squares estimators for the locations of the jumps and the levels of the step function are derived.
Abstract: Consider the problem of estimating a step function in the presence of additive measurement noise. In the case that the number of jumps is known, the least-squares estimators for the locations of the jumps and the levels of the step function are studied and their limiting distributions are derived. When the number of jumps is unknown, an estimator is proposed which is consistent under the condition that the number of jumps is not greater than a given upper bound.
100 citations
01 Jan 2016
TL;DR: In this paper, the authors introduce least squares fit by a line when the data represents measurements where the y-component is assumed to be functionally dependent on the x-component, and define E(A,B) = ∑m i=1[(Axi +B)− yi], which leads to a system of two linear equations in A and B which can be easily solved.
Abstract: This is the usual introduction to least squares fit by a line when the data represents measurements where the y–component is assumed to be functionally dependent on the x–component. Given a set of samples {(xi, yi)}i=1, determine A and B so that the line y = Ax + B best fits the samples in the sense that the sum of the squared errors between the yi and the line values Axi + B is minimized. Note that the error is measured only in the y–direction. Define E(A,B) = ∑m i=1[(Axi +B)− yi]. This function is nonnegative and its graph is a paraboloid whose vertex occurs when the gradient satistfies ∇E = (0, 0). This leads to a system of two linear equations in A and B which can be easily solved. Precisely,
90 citations
TL;DR: This paper proposes an online tracking algorithm based on a novel robust linear regression estimator that models the error term with the Gaussian-Laplacian distribution, which can be efficiently solved and provides insights on the relationships among the LSS problem, Huber loss function, and trivial templates.
Abstract: In this paper, we propose an online tracking algorithm based on a novel robust linear regression estimator. In contrast to existing methods, the proposed least soft-threshold squares (LSS) algorithm models the error term with the Gaussian–Laplacian distribution, which can be efficiently solved. For visual tracking, the Gaussian–Laplacian noise assumption enables our LSS model to handle the normal appearance change and outlier simultaneously. Based on the maximum joint likelihood of parameters, we derive an LSS distance metric to measure the difference between an observation sample and a dictionary of positive templates. Compared with the distance derived from ordinary least squares methods, the proposed metric is more effective in dealing with the outliers. In addition, we provide insights on the relationships among the LSS problem, Huber loss function, and trivial templates, which facilitate better understandings of the existing tracking methods. Finally, we develop a robust tracking algorithm based on the LSS distance metric with an update scheme and negative templates, and speed it up with a particle selection mechanism. Experimental results on numerous challenging image sequences demonstrate that the proposed tracking algorithm performs favorably than the state-of-the-art methods.
72 citations
TL;DR: This paper derives a Kalman filter based least squares iterative (KF- LSI) algorithm to estimate the parameters and states, and a model decomposition based KF-LSI algorithm to enhance computational efficiency.
71 citations
TL;DR: In this paper, the authors show how asymptotically valid inference in regression models based on the weighted least squares estimator can be obtained even when the model for reweighting the data is misspecified.
Abstract: This paper shows how asymptotically valid inference in regression models based on the weighted least squares (WLS) estimator can be obtained even when the model for reweighting the data is misspecified. Like the ordinary least squares estimator, the WLS estimator can be accompanied by heterokedasticty-consistent (HC) standard errors without knowledge of the functional form of conditional heteroskedasticity. First, we provide rigorous proofs under reasonable assumptions; second, we provide numerical support in favor of this approach. Indeed, a Monte Carly study demonstrates attractive finite-sample properties compared to the status quo, both in terms of estimation and making inference.
65 citations
TL;DR: This paper presents a filtering and auxiliary model based recursive least squares identification algorithm with finite measurement input–output data that can generate more accurate parameter estimates and has a higher computational efficiency because the dimensions of its covariance matrices become small.
Abstract: For dual-rate state space systems with time-delay, this paper combines the auxiliary model identification idea with the filtering technique, transforms the state space model into the identification model with different input and output sampling rates, and presents a filtering and auxiliary model based recursive least squares identification algorithm with finite measurement input–output data. Compared with the auxiliary model based recursive least squares algorithm, the proposed algorithm can generate more accurate parameter estimates and has a higher computational efficiency because the dimensions of its covariance matrices become small.
60 citations
TL;DR: This paper presents a quasi-optimal sample set for ordinary least squares (OLS) regression, and presents its efficient implementation via a greedy algorithm, along with several numerical examples to demonstrate its efficacy.
Abstract: In this paper we present a quasi-optimal sample set for ordinary least squares (OLS) regression. The quasi-optimal set is designed in such a way that, for a given number of samples, it can deliver the regression result as close as possible to the result obtained by a (much) larger set of candidate samples. The quasi-optimal set is determined by maximizing a quantity measuring the mutual column orthogonality and the determinant of the model matrix. This procedure is nonadaptive, in the sense that it does not depend on the sample data. This is useful in practice, as it allows one to determine, prior to the potentially expensive data collection procedure, where to sample the underlying system. In addition to presenting the theoretical motivation of the quasi-optimal set, we also present its efficient implementation via a greedy algorithm, along with several numerical examples to demonstrate its efficacy. Since the quasi-optimal set allows one to obtain a near optimal regression result under any affordable nu...
51 citations
20 Mar 2016
TL;DR: Numerical evaluations show that the proposed OLSTEC algorithm gives faster convergence per iteration comparing with the state-of-the-art online algorithms.
Abstract: We propose an online tensor subspace tracking algorithm based on the CP decomposition exploiting the recursive least squares (RLS), dubbed OnLine Low-rank Subspace tracking by TEnsor CP Decomposition (OLSTEC). Numerical evaluations show that the proposed OLSTEC algorithm gives faster convergence per iteration comparing with the state-of-the-art online algorithms.
51 citations
TL;DR: A fast bi-conjugate gradient stabilized method is developed that is superior in computational performance to Gaussian elimination and attains the same accuracy as the implicit difference method (IDM) for the direct problem.
Abstract: In this paper, we consider an inverse problem for identifying the fractional derivative indices in a two-dimensional space-fractional nonlocal model based on a generalization of the two-sided Riemann--Liouville formulation with variable diffusivity coefficients. First, we derive an implicit difference method (IDM) for the direct problem and the stability and convergence of the IDM are discussed. Second, for the implementation of the IDM, we develop a fast bi-conjugate gradient stabilized method (FBi-CGSTAB) that is superior in computational performance to Gaussian elimination and attains the same accuracy. Third, we utilize the Levenberg--Marquardt (L-M) regularization technique combined with the Armijo rule (the popular inexact line search condition) to solve the modified nonlinear least squares model associated with the parameter identification. Finally, we carry out numerical tests to verify the accuracy and efficiency of the IDM. Numerical investigations are performed with both accurate data and noisy...
48 citations
01 Jan 2016
TL;DR: The authors showed that the singular sets of LAD and LMS are at least as large as that of LS and often much larger than LAD, which casts doubt on the trustworthiness of these methods.
Abstract: Say that a regression method is "unstable" at a data set if a small change in the data can cause a relatively large change in the fitted plane. A well-known example of this is the instability of least squares regression (LS) near (multi)collinear data sets. It is known that least absolute deviation (LAD) and least median of squares (LMS) linear regression can exhibit instability at data sets that are far from collinear. Clear-cut instability occurs at a "singularity"-a data set, arbitrarily small changes to which can substantially change the fit. For example, the collinear data sets are the singularities of LS. One way to measure the extent of instability of a regression method is to measure the size of its "singular set" (set of singularities). The dimension of the singular set is a tractable measure of its size that can be estimated without distributional assumptions or asymptotics. By applying a general theorem on the dimension of singular sets, we find that the singular sets of LAD and LMS are at least as large as that of LS and often much larger. Thus, prima facie, LAD and LMS are frequently unstable. This casts doubt on the trustworthiness of LAD and LMS as exploratory regression tools.
TL;DR: In this article, a unified efficient algorithm for fitting SALES and establishing its theoretical properties is developed, and a COupled Sparse Asymmetric LEast Squares (COSALES) regression is proposed to detect heteroscedasticity in high-dimensional data.
Abstract: Asymmetric least squares regression is an important method that has wide applications in statistics, econometrics and finance. The existing work on asymmetric least squares only considers the traditional low dimension and large sample setting. In this paper, we systematically study the Sparse Asymmetric LEast Squares (SALES) regression under high dimensions where the penalty functions include the Lasso and nonconvex penalties. We develop a unified efficient algorithm for fitting SALES and establish its theoretical properties. As an important application, SALES is used to detect heteroscedasticity in high-dimensional data. Another method for detecting heteroscedasticity is the sparse quantile regression. However, both SALES and the sparse quantile regression may fail to tell which variables are important for the conditional mean and which variables are important for the conditional scale/variance, especially when there are variables that are important for both the mean and the scale. To that end, we further propose a COupled Sparse Asymmetric LEast Squares (COSALES) regression which can be efficiently solved by an algorithm similar to that for solving SALES. We establish theoretical properties of COSALES. In particular, COSALES using the SCAD penalty or MCP is shown to consistently identify the two important subsets for the mean and scale simultaneously, even when the two subsets overlap. We demonstrate the empirical performance of SALES and COSALES by simulated and real data.
TL;DR: It is shown that the matrix algorithm can solve the least squares problem within a finite number of iterations in the absence of roundoff errors and the descent property of the norm of residuals is obtained.
Abstract: This paper deals with the solution to the least squares problem min X ‖ ∑ i = 1 s A i XB i + ∑ j = 1 t C j X T D j − E ‖ , corresponding to the generalized Sylvester-transpose matrix equation. The conjugate gradient least squares (CGLS) method is extended to obtain a matrix algorithm for solving this problem. We show that the matrix algorithm can solve this problem within a finite number of iterations in the absence of roundoff errors. Also the descent property of the norm of residuals is obtained. Finally numerical results demonstrate the accuracy and robustness of the algorithm.
TL;DR: In this article, it was shown that for random corrections of the right hand side and Kaczmarz updates selected at random, the algorithm converges to the least square solution.
Abstract: In order to find the least squares solution of a very large and inconsistent system of equations, one can employ the extended Kaczmarz algorithm. This method simultaneously removes the error term, so that a consistent system is asymptotically obtained, and applies Kaczmarz iterations for the current approximation of this system. It has been shown that for random corrections of the right hand side and Kaczmarz updates selected at random, the algorithm converges to the least squares solution. In this work we consider deterministic strategies like the maximal-residual and the almost-cyclic control, and show convergence to a least squares solution.
TL;DR: The new least squares regression method is constructed under the orthogonal constraint which can preserve more discriminant information in the subspace and shows that a global optimal solution is obtained through the iterative algorithm even though the optimization problem is a non-convex problem.
TL;DR: Wang et al. as mentioned in this paper proposed a robust weighted total least squares (RWTLS) algorithm for the partial EIV model, which utilizes the standardized residuals to construct the weight factor function and employs the median method to obtain a robust estimator of the variance component.
Abstract: Total least squares (TLS) can solve the issue of parameter estimation in the errors-invariables (EIV) model, however, the estimated parameters are affected or even severely distorted when the observation vector and coefficient matrix are contaminated by gross errors. Currently, the use of existing robust TLS (RTLS) methods for the EIV model is unreasonable. Original residuals are directly used in most studies to construct the weight factor function, thus the robustness for the structure space is not considered. In this study, a robust weighted total least squares (RWTLS) algorithm for the partial EIV model is proposed based on Newton-Gauss method and the equivalent weight principle of general robust estimation. The algorithm utilizes the standardized residuals to construct the weight factor function and employs the median method to obtain a robust estimator of the variance component. Therefore, the algorithm possesses good robustness in both the observation and structure spaces. To obtain standardized residuals, we use the linearly approximate cofactor propagation law for deriving the expression of the cofactor matrix of WTLS residuals. The iterative procedure and precision assessment approach for RWTLS are presented. Finally, the robustness of RWTLS method is verified by two experiments involving line fitting and plane coordinate transformation. The results show that RWTLS algorithm possesses better robustness than the general robust estimation and the robust total least squares algorithm directly constructed with original residuals.
TL;DR: In this article, the authors describe the multivariate curve resolution-alternating least squares (MCR-ALS) algorithm for spectroscopic data analysis, paying special attention to applications to analyze spectroscopy data.
Abstract: The chapter describes the algorithm multivariate curve resolution-alternating least squares (MCR-ALS), paying special attention to applications to analyze spectroscopic data. The main application fields addressed are process and image analysis. A brief comment on the specific use of MCR for quantitative analysis is done. Finally, a small note on how MCR-ALS compares to similar bilinear or multilinear decomposition methods is written as a conclusion of this chapter.
TL;DR: The different ways in which a binary outcome may appear in a model are examined to distinguish those situations in which the outcome is indeed problematic versus those in which one can easily incorporate it into a PLS-SEM analysis.
Abstract: In this paper, we focus on PLS-SEM’s ability to handle models with observable binary outcomes. We examine the different ways in which a binary outcome may appear in a model and distinguish those situations in which a binary outcome is indeed problematic versus those in which one can easily incorporate it into a PLS-SEM analysis. Explicating such details enables IS researchers to distinguish different situations rather than avoid PLS-SEM altogether whenever a binary indicator presents itself. In certain situations, one can adapt PLS-SEM to analyze structural models with a binary observable variable as the endogenous construct. Specifically, one runs the PLS-SEM first stage as is. Subsequently, one uses the output for the binary variable and latent variable antecedents from this analysis in a separate logistic regression or discriminant analysis to estimate path coefficients for just that part of the structural model. We also describe a method—regularized generalized canonical correlation analysis (RGCCA)—from statistics, which is similar to PLS-SEM but unequivocally allows binary outcomes.
TL;DR: A new Ultra-Orthogonal Forward Regression (UOFR) algorithm is introduced for nonlinear system identification, which includes converting a least squares regression problem into the associated ultra-least squares problem and solving the ultra-leave squares problem using the orthogonal forward regression method.
Posted Content•
TL;DR: In this paper, the authors study distributed learning with the least squares regularization scheme in a reproducing kernel Hilbert space (RKHS) and show that the global output function of this distributed learning is a good approximation to the algorithm processing the whole data in one single machine.
Abstract: We study distributed learning with the least squares regularization scheme in a reproducing kernel Hilbert space (RKHS). By a divide-and-conquer approach, the algorithm partitions a data set into disjoint data subsets, applies the least squares regularization scheme to each data subset to produce an output function, and then takes an average of the individual output functions as a final global estimator or predictor. We show with error bounds in expectation in both the $L^2$-metric and RKHS-metric that the global output function of this distributed learning is a good approximation to the algorithm processing the whole data in one single machine. Our error bounds are sharp and stated in a general setting without any eigenfunction assumption. The analysis is achieved by a novel second order decomposition of operator differences in our integral operator approach. Even for the classical least squares regularization scheme in the RKHS associated with a general kernel, we give the best learning rate in the literature.
TL;DR: In this article, it is shown that good rational filter functions can be computed using (nonlinear least squares) optimization techniques as opposed to designing those functions based on a thorough understanding of complex analysis.
TL;DR: It is shown that the F-ML-RLS algorithm has a high computational efficiency with smaller sizes of its covariance matrices and can produce more accurate parameter estimates.
TL;DR: For the solution of large sparse nonnegative constrained linear least squares (NNLS) problems, a new iterative method is proposed which uses the CGLS method for the inner iterations and the modulus iterative process for the outer iterations to solve the linear complementarity problem resulting from the Karush--Kuhn--Tucker conditions.
Abstract: For the solution of large sparse nonnegative constrained linear least squares (NNLS) problems, a new iterative method is proposed which uses the CGLS method for the inner iterations and the modulus iterative method for the outer iterations to solve the linear complementarity problem resulting from the Karush--Kuhn--Tucker conditions of the NNLS problem. Theoretical convergence analysis including the optimal choice of the parameter matrix is presented for the proposed method. In addition, the method can be further enhanced by incorporating the active set strategy, which contains two stages; the first stage consists of modulus iterations to identify the active set, while the second stage solves the reduced unconstrained least squares problems only on the inactive variables, and projects the solution into the nonnegative region. Numerical experiments show the efficiency of the proposed methods compared to projection gradient--type methods with fewer iteration steps and less CPU time.
TL;DR: In this paper, a novel method for ultrasonic time-of-flight (TOF) estimation through envelope is proposed, where wavelet denoising technique is applied to the noisy echo to improve the estimation accuracy.
TL;DR: This paper considers the extension of the classical Levenberg--Marquardt algorithm to the scenarios where the linearized least squares subproblems are solved inexactly and/or the gradient model is noisy and accurate only within a certain probability.
Abstract: The Levenberg--Marquardt algorithm is one of the most popular algorithms for the solution of nonlinear least squares problems. Motivated by the problem structure in data assimilation, we consider in this paper the extension of the classical Levenberg--Marquardt algorithm to the scenarios where the linearized least squares subproblems are solved inexactly and/or the gradient model is noisy and accurate only within a certain probability. Under appropriate assumptions, we show that the modified algorithm converges globally to a first order stationary point with probability one. Our proposed approach is first tested on simple problems where the exact gradient is perturbed with a Gaussian noise or only called with a certain probability. It is then applied to an instance in variational data assimilation where stochastic models of the gradient are computed by the so-called ensemble methods.
TL;DR: An efficient iterative algorithm to orthogonalize a design matrix by adding new rows and then solve the original problem by embedding the augmented design in a missing data framework, which is considerably faster than competing methods when n is much larger than p.
Abstract: We introduce an efficient iterative algorithm, intended for various least squares problems, based on a design of experiments perspective. The algorithm, called orthogonalizing EM (OEM), works for ordinary least squares (OLS) and can be easily extended to penalized least squares. The main idea of the procedure is to orthogonalize a design matrix by adding new rows and then solve the original problem by embedding the augmented design in a missing data framework. We establish several attractive theoretical properties concerning OEM. For the OLS with a singular regression matrix, an OEM sequence converges to the Moore-Penrose generalized inverse-based least squares estimator. For ordinary and penalized least squares with various penalties, it converges to a point having grouping coherence for fully aliased regression matrices. Convergence and the convergence rate of the algorithm are examined. Finally, we demonstrate that OEM is highly efficient for large-scale least squares and penalized least squares proble...
TL;DR: In this paper, two penalized estimations of this model based on modifying the partial least squares criterion with roughness penalties for the weight functions are proposed, one introduces the penalty in the definition of the norm in the functional space, and the other one in the cross-covariance operator.
TL;DR: This paper proposes a novel method, the nested sub-sample search algorithm, which reduces the number of least squares operations drastically to O(log n) for large sample size, and demonstrates its speed and reliability via Monte Carlo simulation studies with finite samples.
Abstract: Threshold models have been popular for modelling nonlinear phenomena in diverse areas, in part due to their simple fitting and often clear model interpretation. A commonly used approach to fit a threshold model is the (conditional) least squares method, for which the standard grid search typically requires O(n) operations for a sample of size n; this is substantial for large n, especially in the context of panel time series. This paper proposes a novel method, the nested sub-sample search algorithm, which reduces the number of least squares operations drastically to O(log n) for large sample size. We demonstrate its speed and reliability via Monte Carlo simulation studies with finite samples. Possible extension to maximum likelihood estimation is indicated.
TL;DR: In this paper, a two-port coaxial probe is introduced to determine the permittivity tensor of a uniaxial material by minimizing the two-norm of the vector difference between the theoretical and measured scattering parameters via nonlinear least squares.
Abstract: A two-port coaxial probe is introduced to nondestructively determine the permittivity tensor of a uniaxial material. The proposed approach possesses several advantages over existing techniques, e.g., only a single sample is required, the sample does not need to be rotated, and only a single measurement system is needed. The derivation of the theoretical scattering parameters is shown. This is accomplished by applying Love’s equivalence theorem and the continuity of transverse magnetic fields to formulate a system of coupled integral equations. A necessary step in this approach is the derivation of the magnetic-current-excited uniaxial parallel-plate Green’s function. The development of this Green’s function is presented here using a new scalar potential formulation, which significantly reduces the difficulty of the probe’s theoretical development. The system of coupled integral equations is solved using the method of moments to yield the theoretical scattering parameters. The permittivity tensor is found by minimizing the two-norm of the vector difference between the theoretical and measured scattering parameters via nonlinear least squares. To validate the probe, measurement results of a uniaxial absorber are presented and compared to those obtained using a focused-beam (free-space) measurement system. The probe’s sensitivity to uncertainties in measured scattering parameters, sample thickness, and coaxial line properties is also investigated.
TL;DR: In this article, an alternative way to take correlations into account thanks to a diagonal covariance matrix is presented, which can be seen as a reduced version of the original matrix, its elements being simply the sums of the rows elements of the weighting matrix.
Abstract: Based on the results of Luati and Proietti (Ann Inst Stat Math 63:673–686, 2011) on an equivalence for a certain class of polynomial regressions between the diagonally weighted least squares (DWLS) and the generalized least squares (GLS) estimator, an alternative way to take correlations into account thanks to a diagonal covariance matrix is presented. The equivalent covariance matrix is much easier to compute than a diagonalization of the covariance matrix via eigenvalue decomposition which also implies a change of the least squares equations. This condensed matrix, for use in the least squares adjustment, can be seen as a diagonal or reduced version of the original matrix, its elements being simply the sums of the rows elements of the weighting matrix. The least squares results obtained with the equivalent diagonal matrices and those given by the fully populated covariance matrix are mathematically strictly equivalent for the mean estimator in terms of estimate and its a priori cofactor matrix. It is shown that this equivalence can be empirically extended to further classes of design matrices such as those used in GPS positioning (single point positioning, precise point positioning or relative positioning with double differences). Applying this new model to simulated time series of correlated observations, a significant reduction of the coordinate differences compared with the solutions computed with the commonly used diagonal elevation-dependent model was reached for the GPS relative positioning with double differences, single point positioning as well as precise point positioning cases. The estimate differences between the equivalent and classical model with fully populated covariance matrix were below the mm for all simulated GPS cases and below the sub-mm for the relative positioning with double differences. These results were confirmed by analyzing real data. Consequently, the equivalent diagonal covariance matrices, compared with the often used elevation-dependent diagonal covariance matrix is appropriate to take correlations in GPS least squares adjustment into account, yielding more accurate cofactor matrices of the unknown.