Showing papers on "Non-linear least squares published in 1989"
Book•
01 Sep 1989
TL;DR: This latent variable path modeling with partial least squares%0D will actually offer you the smart idea to be successful and will add even more expertise to life and also work much better.
Abstract: Partial Least Squares (PLS) is an estimation method and an algorithm for latent variable path (LVP) models. PLS is a component technique and estimates the latent variables as weighted aggregates. The implications of this choice are considered and compared to covariance structure techniques like LISREL, COSAN and EQS. The properties of special cases of PLS (regression, factor scores, structural equations, principal components, canonical correlation, hierarchical components, correspondence analysis, three-mode path and component analysis) are examined step by step and contribute to the understanding of the general PLS technique. The proof of the convergence of the PLS algorithm is extended beyond two-block models. Some 10 computer programs and 100 applications of PLS are referenced. The book gives the statistical underpinning for the computer programs PLS 1.8, which is in use in some 100 university computer centers, and for PLS/PC. It is intended to be the background reference for the users of PLS 1.8, not as textbook or program manual.
1,695 citations
TL;DR: Identification algorithms based on the well-known linear least squares methods of gaussian elimination, Cholesky decomposition, classical Gram-Schmidt, modified Gram- Schmidt, Householder transformation, Givens method, and singular value decomposition are reviewed.
Abstract: Identification algorithms based on the well-known linear least squares methods of gaussian elimination, Cholesky decomposition, classical Gram-Schmidt, modified Gram-Schmidt, Householder transformation, Givens method, and singular value decomposition are reviewed. The classical Gram-Schmidt, modified Gram-Schmidt, and Householder transformation algorithms are then extended to combine structure determination, or which terms to include in the model, and parameter estimation in a very simple and efficient manner for a class of multivariate discrete-time non-linear stochastic systems which are linear in the parameters.
1,620 citations
TL;DR: The algorithm implemented is an efficient and stable trust region (Levenberg-Marquardt) procedure that exploits the structure of the problem so that the computational cost per iteration is equal to that for the same type of algorithm applied to the ordinary nonlinear least squares problem.
Abstract: In this paper, we describe ODRPACK, a software package for the weighted orthogonal distance regression problem. This software is an implementation of the algorithm described in [2] for finding the parameters that minimize the sum of the squared weighted orthogonal distances from a set of observations to a curve or surface determined by the parameters. It can also be used to solve the ordinary nonlinear least squares problem. The weighted orthogonal distance regression procedure application to curve and surface fitting and to measurement error models in statistics. The algorithm implemented is an efficient and stable trust region (Levenberg-Marquardt) procedure that exploits the structure of the problem so that the computational cost per iteration is equal to that for the same type of algorithm applied to the ordinary nonlinear least squares problem. The package allows a general weighting scheme, provides for finite difference derivatives, and contains extensive error checking and report generating facilities.
221 citations
TL;DR: In this paper, a flexible least squares (FLS) solution is proposed to track linear, quadratic, sinusoidal, and regime shift motions in the true coefficients, despite noisy observations.
Abstract: Suppose noisy observations obtained on a process are assumed to have been generated by a linear regression model with coefficients which evolve only slowly over time, if at all. Do the estimated time-paths for the coefficients display any systematic time-variation, or is time-constancy a reasonably satisfactory approximation? A “flexible least squares” (FLS) solution is proposed for this problem, consisting of all coefficient sequence estimates which yield vector-minimal sums of squared residual measurement and dynamic errors conditional on the given observations. A procedure with FORTRAN implementation is developed for the exact sequential updating of the FLS estimates as the process length increases and new observations are obtained. Simulation experiments demonstrating the ability of FLS to track linear, quadratic, sinusoidal, and regime shift motions in the true coefficients, despite noisy observations, are reported. An empirical money demand application is also summarized.
139 citations
TL;DR: In this paper, the mathematics behind principal component analysis and partial least squares regression is presented in detail, starting from the appropriate extrema conditions, and the meaning of the resultant vectors and many of their mathematical interrelationships are also presented.
129 citations
TL;DR: In this paper, the Cramer-Rao bound for large-sample accuracy properties of two nonlinear least-squares estimators (NLSE) of sine-wave parameters, the basic NLSE which ignores the possible correlation of the noise, and the optimal NLSE, which also estimates the noise correlation, were established.
Abstract: This paper establishes the large-sample accuracy properties of two nonlinear least-squares estimators (NLSE) of sine-wave parameters: the basic NLSE, which ignores the possible correlation of the noise, and the optimal NLSE, which, besides the sine-wave parameters, also estimates the noise correlation (appropriately parametrized). It is shown that these two NLS estimators have thesame accuracy in large samples. This result provides complete justification for preferring the computationally less expensive basic NLSE over the “optimal” NLSE. Both estimators are shown to achieve the Cramer-Rao Bound (CRB) as the sample size increases. A simple explicit expression for the CRB matrix is provided, which should be useful in studying the performance of sine-wave parameter estimators designed to work in the colored-noise case.
97 citations
TL;DR: In this article, the first two moments of non-linear and least-squares estimators are studied and approximate expressions for the moments are derived and discussed, and the results are compared with those of Jeudy [1988].
Abstract: The first two moments of non-linear and least-squares estimators are studied. Approximate expressions for the moments are derived and discussed. The results are compared with those ofJeudy [1988].
56 citations
TL;DR: Software for the analysis of multivariate chemical data by principal components and partial least squares methods is included on disk and contains options for the graphical display of scores and loadings for interpretation of the results of analyses.
54 citations
TL;DR: In this article, the stability of the solutions of parameter estimation problems in their output least squares formulation is analyzed, and sufficient conditions for this property are proved for abstract elliptic equations for the estimation of the diffusion, convection, and friction coefficient in second-order ellipses.
Abstract: In this paper the stability of the solutions of parameter estimation problems in their output least squares formulation is analyzed. The concepts of output least squares stability (OLS stability) is defined and sufficient conditions for this property are proved for abstract elliptic equations. These results are applied to the estimation of the diffusion, convection, and friction coefficient in second-order elliptic equations inźn,n=2, 3. Results on Tikhonov regularization in a nonlinear setting are also given.
53 citations
TL;DR: The problem of finding the spatial correspondence between an object and the image of the object under perspective projection is investigated and a new technique is demonstrated, based on a geometrical description and a least squares solution of the resulting nonlinear equations.
46 citations
TL;DR: In this paper, it was shown that the solution to the normal equations is the solution of the rectangular, but consistent, linear system, which is the same as the solution in this paper.
Abstract: We have other methods that, while more costly, are more robust in the face of rounding errors. The other methods arrive at xLS by a different route. Recall that the normal equations were a result of requiring that b− Ax be orthogonal (normal) to the subspace S = ColSp(A). That is another way of saying that Ax is the orthogonal projection of b onto S. The solution to the normal equations is therefore the solution to the rectangular, but consistent, linear system
TL;DR: Two methods of analyzing individual pharmacokinetic data by nonlinear regression using a generalized least squares scheme for the case where the variance is assumed proportional to an unknown power of the mean are described.
Abstract: In the analysis of individual pharmacokinetic data by nonlinear regression it is important to allow for possible heterogeneity of variance in the response. Two common methods of doing this are weighted least squares with appropriate weights or data transformation using a suitable transform. With either approach it is appealing to let the data determine the appropriate choice of weighting scheme or transformation. This article describes two methods of doing this which are easy to compute using standard statistical software. The first method is a generalized least squares scheme for the case where the variance is assumed proportional to an unknown power of the mean. The second involves applying a power transformation to both sides of the regression equation. It is shown that both techniques may be implemented using only nonlinear regression routines. Sample code is provided for their implementation using the SAS software package. However, the proposed methods are feasible using any software package that incorporates a nonlinear least squares routine, and are thus well suited to routine use.
TL;DR: In this paper, a finite element like least square method is introduced for determining the density function in the Preisach hysteresis model from overdeterined measured data, and it is shown that the least squares error depends on the quality of the data and the best approximations to the analytic density.
Abstract: A finite element like least squares method is introduced for determining the density function in the Preisach hysteresis model from overdeterined measured data It is shown that the least squares error depends on the quality of the data and the best approximations to the analytic density For consistent data criteria are given for convergence of the approximate density and Preisach operator with increasing measurements
TL;DR: In this paper, the role of iterative generalized least squares as an algorithm for the computation of statistical estimators is discussed, and relationships between various algorithms, such as Newton-Raphson, Gauss-Newton, and scoring, are studied.
Abstract: This expository paper deals with the role of iterative generalized least squares as an algorithm for the computation of statistical estimators. Relationships between various algorithms, such as Newton-Raphson, Gauss-Newton, and scoring, are studied. A parallel is made between statistical properties of the model and the structure of the numerical algorithm employed to find parameter estimates. In particular a general linearizability property that extends the concept of link function in generalized linear models is considered and its computational meaning is discussed. Maximum quasilikelihood estimators are reinterpreted so that they may exist even when there is no quasilikelihood function.
TL;DR: In this paper, the inverse problem of magnetotellurics over a horizontally stratified earth is described, with emphasis on practical application, and the inversion is divided into basically two steps: the construction of some best solution and the analysis of that solution with regard to uncertainty and complexity.
Abstract: The inverse problem of magnetotellurics over a horizontally stratified earth is described, with emphasis on practical application. The inversion is divided into basically two steps. The construction of some best solution, and the analysis of that solution with regard to uncertainty and complexity. For the construction of best solutions a robust non-linear solver was developed, and for the estimation of parameter errors a modified eigenvalue-eigenvector analysis is performed to better describe non-linear effects. The choice of the number of layers is shown to be intimately connected with the structure of data errors and the misfit between model and data. An example from the Siljan impact structure in Sweden illustrates the power of the technique.
TL;DR: In this article, a modified method of the exponentially weighted recursive least-squares (WRLS) estimation using sliding window is presented, where two windowing techniques are simultaneously used to ensure that the estimator has good parameter tracking property and that the estimated parameters converge the true parameters.
Abstract: This letter presents a modified method of the exponentially weighted recursive least-squares (WRLS) estimation using sliding window. In the method, two windowing techniques are simultaneously used to ensure that the estimator has good parameter tracking property and that the estimated parameters converge the true parameters. Simulation shows that the proposed method tracks rapidly time-varying parameters more effectively than WRLS.
TL;DR: Results from this study generally support the theoretical interpretation of λ as a ratio of variances, when the model is constrained to recent survey biomass estimates, and estimates of the coefficient of variation of biomasses in earlier years are large.
Abstract: This paper presents a method for estimating the variances of biomass estimates made using the nonlinear least squares catch-at-age model. The estimates begin with the standard covariance matrix of ...
TL;DR: In this article, the closeness of the total least squares (TLS) and the classical least square (LS) problem is studied algebraically and interesting algebraic connections between their solutions, their residuals, their corrections applied to data fitting and their approximate subspaces are proven.
Abstract: In this paper the closeness of the total least squares (TLS) and the classical least squares (LS) problem is studied algebraically. Interesting algebraic connections between their solutions, their residuals, their corrections applied to data fitting and their approximate subspaces are proven.
All these relationships point out the parameters which mainly determine the equivalences and differences between the two techniques. These parameters also lead to a better understanding of the differences in sensitivity between both approaches with respect to perturbations of the data.
In particular, it is shown how the differences between both approaches increase when the equationsAX?B become less compatible, when the length ofB orX is growing or whenA tends to be rank-deficient. They are maximal whenB is parallel with the singular vector ofA associated with its smallest singular value. Furthermore, it is shown how TLS leads to a weighted LS problem, and assumptions about the underlying perturbation model of both techniques are deduced. It is shown that many perturbation models correspond with the same TLS solution.
Journal Article•
TL;DR: In this paper, the strong consistency of the least squares estimates for some diagonal bilinear model of arbitrary order is obtained, and the quasiminimal Markovian representation of the models is considered and developments on the notion of invertibility in the case of nonlinear models are given.
Abstract: The strong consistency of the least squares estimates for some diagonal bilinear model of arbitrary order is obtained. The quasiminimal Markovian representation of the models is considered and developments on the notion of invertibility in the case of non-linear models are given.
TL;DR: This paper investigates, via simulations, how perturbations on both A and B affect the accuracy of the TLS and LS solution in the presence of uncorrelated and equally sized errors.
TL;DR: The results indicate that the approach taken here is a viable alternative for least squares problems to the general nonlinear methods studied by Hock and Schittkowski.
Abstract: This paper is concerned with the development, numerical implementation, and testing of an algorithm for solving constrained nonlinear least squares problems. The algorithm is an adaptation of the least squares case of an exact penalty method for solving nonlinearly constrained optimization problems due to Coleman and Conn. It also uses the ideas of Nocedal and Overton for handling quasi-Newton updates of projected Hessians, those of Dennis, Gay, and Welsch for approaching the structure of nonlinear least squares Hessians, and those of Murray and Overton for performing line searches. This method has been tested on a selection of problems listed in the collection of Hock and Schittkowski. The results indicate that the approach taken here is a viable alternative for least squares problems to the general nonlinear methods studied by Hock and Schittkowski.
TL;DR: The application of the generalized-least-squares (GLS) method to the estimation of the frequencies of sinusoids in additive colored noise and an iterative algorithm is proposed, starting from the assumption that the parameter vector in the model of the sinusoid is symmetric.
Abstract: The application of the generalized-least-squares (GLS) method to the estimation of the frequencies of sinusoids in additive colored noise is discussed. An iterative algorithm is proposed, starting from the assumption that the parameter vector in the model of the sinusoids is symmetric. The algorithm utilizes an adaptive strategy for contracting the poles of the estimated signal model. Expressions for the probability limit and the asymptotic variance of the estimates are derived for the single sinusoid case. Possible convergence points and the asymptotic behavior of the algorithm in the case of low SNRs (signal-to-noise ratios) are analyzed. Extensive simulation results show that the algorithm represents a simple and reliable tool for practical applications. >
TL;DR: The early motivation for and development of diagonal increments to ease matrix inversion in least squares (LS) problems is discussed and the interplay among factors and the advent of ridge regression are considered in a historical and comparative framework.
Abstract: he early motivation for and development of diagonal increments to ease matrix inversion in least squares (LS) problems is discussed. It is noted that this diagonal incrementation evolved from three major directions: modification of existing methodology in nonlinear LS, utilization of additional information in linear regression, and improvement of the numerical condition of a matrix. The interplay among these factors and the advent of ridge regression are considered in a historical and comparative framework
01 Jan 1989
TL;DR: In this paper, the ability to track time varying frequency and damping parameters using on-line versions of seven time domain system identification algorithms; Least Squares, Double Least Square, Correlation Fit, Instrumental Variables and Instrumental Matrix with Delayed Observations.
Abstract: The ability to track time varying frequency and damping parameters using on-line versions of seven time domain system identification algorithms; Least Squares, Double Least Squares, Correlation Fit, Instrumental Variables, Instrumental Matrix with Delayed Observations, Extended Least Squares and Maximum Likelihood, is examined
TL;DR: In this paper, the extended Kalman filter and Marquardt's gradient expansion algorithm for nonlinear least squares are compared with respect to accuracy and precision of parameter extimates, computational burden, sensitivity to initial parameter estimates and ability to indicate model errors.
TL;DR: In this article, the optimum procedures for the statistical improvement, or adjustment, of an existing data evaluation are redeveloped from first principles, consistently employing a minimum-variance viewpoint.
Abstract: Optimum procedures for the statistical improvement, or adjustment, of an existing data evaluation are redeveloped from first principles, consistently employing a minimum-variance viewpoint. A set o...
TL;DR: In this paper, a comparison between least square regression and robust regression is made by means of Monte Carlo simulations, and the robust regression procedure is based on the Huber estimate and is computed by using the iteratively reweighted least square algorithm.
Abstract: By means of Monte Carlo simulations a comparison has been made between ordinary least squares regression and robust regression. The robust regression procedure is based on the Huber estimate and is computed by means of the iteratively reweighted least squares algorithm. The performance of both procedures has been evaluated for estimation of the parameters of a calibration function and for determination of the concentration of unknown samples. The influence of the distributional characteristics skewness and kurtosis has been studied, and the number of measurements used for constructing the calibration curve has also been taken into account. Under certain conditions robust regression offers an advantage over least squares regression.
01 Jan 1989
TL;DR: In this article, a characterization of the unique solution based on the comparison of averages is presented, which gives a first insight to the general approximation problem with unknown tree structure, and also provides a rigorous mathematical treatment of the underlying least squares approximation problem is missing.
Abstract: Several authors have proposed procedures for fitting an additive tree to a distance, and some of them make use of the least squares principle, e.g. Sattath & Tversky (1977), Carroll & Pruzansky (1980) and de Soete (1983). However, a rigorous mathematical treatment of the underlying least squares approximation problem is missing. We start to fill this gap by considering the approximation problem for a fixed tree structure. We present a characterization of the unique solution based on the comparison of averages. Some conclusions from this characterization give a first insight to the general approximation problem with unknown tree structure. Let X be a finite set of objects. Following Buneman (1971), a tree structure on X can be represented by a system of compatible splits: Each edge of the tree structure is represented by a bipartition {A, A c } of X, called a split, and two splits {A, A c } and {B, BC} are called compatible, if A ⊆B ⋁ A ⊆ B c ⋁ A c ⊆ ⋁ A c ⊆ B c holds (cp. fig. 1).