Showing papers on "Non-linear least squares published in 1994"
TL;DR: The Marquardt algorithm for nonlinear least squares is presented and is incorporated into the backpropagation algorithm for training feedforward neural networks and is found to be much more efficient than either of the other techniques when the network contains no more than a few hundred weights.
Abstract: The Marquardt algorithm for nonlinear least squares is presented and is incorporated into the backpropagation algorithm for training feedforward neural networks. The algorithm is tested on several function approximation problems, and is compared with a conjugate gradient algorithm and a variable learning rate algorithm. It is found that the Marquardt algorithm is much more efficient than either of the other techniques when the network contains no more than a few hundred weights. >
6,899 citations
TL;DR: In this article, the authors compared several methods (ordinary least squares, nonlinear least square, maximum likelihood in an error correction model, principal components, and canonical correlations) of estimating cointegrating vectors.
1,218 citations
TL;DR: In this article, the asymptotic conditional bias and variance of the estimator at points near the boundary of the support of the predictors were derived using weighted least squares matrix theory.
Abstract: Nonparametric regression using locally weighted least squares was first discussed by Stone and by Cleveland. Recently, it was shown by Fan and by Fan and Gijbels that the local linear kernel-weighted least squares regression estimator has asymptotic properties making it superior, in certain senses, to the Nadaraya-Watson and Gasser-Muller kernel estimators. In this paper we extend their results on asymptotic bias and variance to the case of multivariate predictor variables. We are able to derive the leading bias and variance terms for general multivariate kernel weights using weighted least squares matrix theory. This approach is especially convenient when analysing the asymptotic conditional bias and variance of the estimator at points near the boundary of the support of the predictors. We also investigate the asymptotic properties of the multivariate local quadratic least squares regression estimator discussed by Cleveland and Devlin and, in the univariate case, higher-order polynomial fits and derivative estimation.
1,082 citations
TL;DR: This article is to show how several different variants of the recursive least-squares algorithm can be directly related to the widely studied Kalman filtering problem of estimation and control.
Abstract: Adaptive filtering algorithms fall into four main groups: recursive least squares (RLS) algorithms and the corresponding fast versions; QR- and inverse QR-least squares algorithms; least squares lattice (LSL) and QR decomposition-based least squares lattice (QRD-LSL) algorithms; and gradient-based algorithms such as the least-mean square (LMS) algorithm. Our purpose in this article is to present yet another approach, for the sake of achieving two important goals. The first one is to show how several different variants of the recursive least-squares algorithm can be directly related to the widely studied Kalman filtering problem of estimation and control. Our second important goal is to present all the different versions of the RLS algorithm in computationally convenient square-root forms: a prearray of numbers has to be triangularized by a rotation, or a sequence of elementary rotations, in order to yield a postarray of numbers. The quantities needed to form the next prearray can then be read off from the entries of the postarray, and the procedure can be repeated; the explicit forms of the rotation matrices are not needed in most cases. >
470 citations
TL;DR: This work generalizes this approach to perspective projection and partial or uncertain tracks by using a nonlinear least squares technique and quickly converges to the desired optimal solution, even in the absence of a priori knowledge about the shape or motion.
349 citations
TL;DR: The iterated Kalman smoother is presented and shown to be a Gauss–Newton method for maximizing the likelihood function in the nonaffine case and to decompose a large least squares problem into a sequence of much smaller problems.
Abstract: The Kalman smoother is known to be the maximum likelihood estimator when the measurement and transition functions are affine; i.e., a linear function plus a constant. A new proof of this result is presented that shows that the Kalman smoother decomposes a large least squares problem into a sequence of much smaller problems. The iterated Kalman smoother is then presented and shown to be a Gauss–Newton method for maximizing the likelihood function in the nonaffine case. This method takes advantage of the decomposition obtained with the Kalman smoother.
166 citations
TL;DR: The elementary algorithm presented here fits readily in a first course in numerical linear algebra and is suitable for use in research on total least squares regression.
Abstract: Total least squares regression (TLS) fits a line to data where errors may occur in both the dependent and independent variables. In higher dimensions, TLS fits a hyperplane to such data. The elementary algorithm presented here fits readily in a first course in numerical linear algebra.
112 citations
TL;DR: Two implementations of these algorithms that are based on the normal equations and corrected seminormal equations approaches are described and it is shown that the method of normal equations should be employed in the implementation of the predictor-corrector algorithm.
Abstract: In this paper we discuss the use of block principal pivoting and predictor-corrector methods for the solution of large-scale linear least squares problems with nonnegative variables (NVLSQ). We also describe two implementations of these algorithms that are based on the normal equations and corrected seminormal equations (CSNE) approaches. We show that the method of normal equations should be employed in the implementation of the predictor-corrector algorithm. This type of approach should also be used in the implementation of the block principal pivoting method, but a switch to the CSNE method may be useful in the last iterations of the algorithm. Computational experience is also included in this paper and shows that both the predictor-corrector and the block principal pivoting algorithms are quite efficient to deal with large-scale NVLSQ problems.
97 citations
TL;DR: In this paper, an extension of the nonlinear least squares fitting technique of Vinas and Scudder (1986) (VS) is presented, which finds the physical and geometrical properties of nondissipational magnetohydrodynamic (MHD) shocks.
Abstract: This paper presents an extension of the nonlinear least squares fitting technique of Vinas and Scudder (1986) (VS), which finds the physical and geometrical properties of nondissipational magnetohydrodynamic (MHD) shocks. The new method incorporates plasma temperature observations in the form of normal momentum flux and energy density flux conservation as well as plasma density, velocity, and magnetic field data. The new technique is capable of using known standard deviations in the individual measurement points to properly weight the fitting procedure. The new fitting code is validated through the analysis of synthetic shocks with known physical and geometrical properties. Finally, it is compared to the original VS method and the preaveraged velocity coplanarity technique.
90 citations
TL;DR: In this article, a linearized version of the nonlinear least squares method combined with the maximum entropy spectral analysis method was proposed and applied to the annual sunspot number data from 1700 to 1991.
Abstract: A newly devised procedure of time series analysis, which is a linearized version of the nonlinear least squares method combined with the maximum entropy spectral analysis method, was proposed and applied to the annual sunspot number data from 1700 to 1991. Multiple periodicities of the temporal variation were elucidated in detail. The solar cycle of a 11.04-year period accompanied with the solar cycle multiplets, the periods of 50.41 years, the so-called "Yoshimura cycle", and 107.11 years corresponding to the cycle of century-scale minima, for example, were clearly observed. The optimum least squares fitting curve for the data was extended over the past two millennia and the next millennium. The past grand minima such as the Maunder minimum were confirmed in the past extrapolation curve, and the next centennial minimum was predicted to occur between 2000 and 2030.
88 citations
TL;DR: In this paper, a weighted-least-squares method using sensitivity-analysis technique is proposed for the estimation of parameters in water-distribution systems, where the parameters considered are the Hazen-Williams coefficients for the pipes.
Abstract: The weighted-least-squares method using sensitivity-analysis technique is proposed for the estimation of parameters in water-distribution systems. The parameters considered are the Hazen-Williams coefficients for the pipes. The objective function used is the sum of the weighted squares of the differences between the computed and the observed values of the variables. The weighted-least-squares method can elegantly handle multiple loading conditions with mixed types of measurements such as heads and consumptions, different sets and number of measurements for each loading condition, and modifications in the network configuration due to inclusion or exclusion of some pipes affected by valve operations in each loading condition. Uncertainty in parameter estimates can also be obtained. The method is applied for the estimation of parameters in a metropolitan urban water-distribution system in India.
TL;DR: In this article, the Cholesky factorization of the least square coefficient matrix without explicitly forming the normal equations is presented. But the method is based on QR factorizations of the original matrices $A$ and $B$.
Abstract: The general problem considered here is the least squares solution of $(A \otimes B)x = t$, where $A$ and $B$ are full rank, rectangular matrices, and $A \otimes B$ is the Kronecker product of $A$ and $B$. Equations of this form arise in areas such as digital image and signal processing, photogrammetry, finite elements, and multidimensional approximation. An efficient method of solution is based on QR factorizations of the original matrices $A$ and $B$. It is demonstrated how these factorizations can be used to obtain the Cholesky factorization of the least squares coefficient matrix without explicitly forming the normal equations. A similar approach based on singular value decomposition (SVD) factorizations also is indicated for the rank-deficient case.
TL;DR: A method is presented for joint analysis of survival proportions reported at multiple times in published studies to be combined in a meta-analysis, using generalized least squares to fit linear models including between-trial and within-trial covariates.
Abstract: A method is presented for joint analysis of survival proportions reported at multiple times in published studies to be combined in a meta-analysis. Generalized least squares is used to fit linear models including between-trial and within-trial covariates, using current fitted values iteratively to derive correlations between times within studies. Multi-arm studies and nonrandomized historical controls can be included with no special handling. The method is illustrated with data from two previously published meta-analyses. In one, an early treatment difference is detected that was not apparent in the original analysis.
14 Dec 1994
TL;DR: The author focuses on the extended Kalman filter, which may be viewed as an incremental version of the Gauss-Newton method, and provides a nonstochastic analysis of its convergence properties, and discusses variants aimed at accelerating its convergence.
Abstract: Proposes and analyzes nonlinear least squares methods, which process the data incrementally, one data block at a time. Such methods are well suited for large data sets and real time operation, and have received much attention in the context of neural network training problems. The author focuses on the extended Kalman filter, which may be viewed as an incremental version of the Gauss-Newton method. The author provides a nonstochastic analysis of its convergence properties, and discusses variants aimed at accelerating its convergence. >
TL;DR: In this paper, the Navier-Stokes equations are cast as a set of first-order equations involving viscous stresses as auxiliary variables and the primary and auxiliary variables are interpolated using equal-order C0 continuity, p-version hierarchical approximation functions.
Abstract: A p-version least squares finite element formulation for non-linear problems is applied to the problem of steady, two-dimensional, incompressible fluid flow. The Navier-Stokes equations are cast as a set of first-order equations involving viscous stresses as auxiliary variables. Both the primary and auxiliary variables are interpolated using equal-order C0 continuity, p-version hierarchical approximation functions. The least squares functional (or error functional) is constructed using the system of coupled first-order non-linear partial differential equations without linearization, approximations or assumptions. The minimization of this least squares error functional results in finding a solution vector {δ} for which the partial derivative of the error functional (integrated sum of squares of the errors resulting from individual equations for the entire discretization) with respect to the nodal degrees of freedom {δ} becomes zero. This is accomplished by using Newton's method with a line search. Numerical examples are presented to demonstrate the convergence characteristics and accuracy of the method.
TL;DR: In this article, non-linear least-squares methods have been coupled with high-resolution image simulation to determine the critical electron microscopic imaging parameters, such as thickness and defocus, from an experimental image.
TL;DR: A weighted nonlinear least squares method to solve the 8- or 16-term calibration problem for a 2-port vector network analyzer and generates "soft" bounds on the accuracy of the calibration.
Abstract: A weighted nonlinear least squares method to solve the 8- or 16-term calibration problem for a 2-port vector network analyzer is given. The method handles the connection repeatability problem, provides a test to verify whether the calibration problem can be solved with the available data and generates "soft" bounds on the accuracy of the calibration. The computational issues to obtain a fast and accurate implementation are stressed. >
TL;DR: A new algorithm for accurate downdating of least squares solutions is described and compared to existing algorithms and numerical test results are presented using the sliding window method.
Abstract: Solutions to a sequence of modified least squares problems, where either a new observation is added (updating) or an old observation is deleted (downdating), are required in many applications. Stable algorithms for downdating can be constructed if the complete QR factorization of the data matrix is available. Algorithms that only downdate $R$ and do not store $Q$ require less operations. However, they do not give good accuracy and may not recover accuracy after an ill-conditioned problem has occurred. The authors describe a new algorithm for accurate downdating of least squares solutions and compare it to existing algorithms. Numerical test results are also presented using the sliding window method, where a number of updatings and downdatings occur repeatedly.
TL;DR: In this article, the authors presented various novel and extended results on least squares based adaptive minimum variance control for linear stochastic systems, and established self-optimality, self-tuning property, and the best possible convergence rate of the control law in a variety of situations of interest.
Abstract: Based on the recently established results on self-tuning regulators originally proposed by Astrom and Wittenmark, this paper presents various novel and extended results on least squares based adaptive minimum variance control for linear stochastic systems. These results establish self-optimality, self-tuning property, and the best possible convergence rate of the control law in a variety of situations of interest.
TL;DR: In this paper, the authors examined TLS when a given value of A$ is nearly rank deficient by outlining its differences and similarities to the well-known truncated least squares (LS) method.
Abstract: The least squares (LS) and total least squares (TLS) methods are commonly used to solve the overdetermined system of equations $Ax \approx b$. The main objective of this paper is to examine TLS when $A$ is nearly rank deficient by outlining its differences and similarities to the well-known truncated LS method. It is shown that TLS may be viewed as a regularization technique much like truncated LS, even though the rank reduction depends on $b$. The sensitivity of LS and TLS approximate nullspaces to perturbations in the data is also examined. Some numerical simulations are included.
TL;DR: In this paper, a method of kinematic parameter estimation is proposed where a standard least-squares estimation procedure is replaced by weighted least squares, based on all the a priori available statistical information about the robot and the pose-measuring system.
Abstract: Significant attention has been paid recently to the topic of robot calibration. To improve the robot's accuracy, various approaches to the measurement of the robot's position and orientation (pose) and correction of its kinematic model have been proposed. Little attention, however, has been given to the method of estimation of the kinematic parameters from the measurement data. Typically, a least-squares solution method is used to estimate the corrections to the parameters of the model. In this paper, a method of kinematic parameter estimation is proposed where a standard least-squares estimation procedure is replaced by weighted least-squares. The weighting factors are calculated based on all the a priori available statistical information about the robot and the pose-measuring system. By giving greater weight to the measurements made where the standard deviation of the noise in the data is expected to be lower, a significant reduction in the error of the kinematic parameter estimates is made possible. The improvement in the calibration results was verified using a calibration simulation algorithm.
TL;DR: In this paper, an approach to analysis of internal reliability of linear least squares models is presented, based on the relationship between a single observational disturbance, i.e., a gross error or a blunder, and the model response being a certain pattern of distortions in the least squares residuals.
Abstract: An approach to analysis of internal reliability of linear least squares models is presented. It is based on the relationship between a single observational disturbance, i.e. a gross error or a blunder, and the model response being a certain pattern of distortions in the least squares residuals. Rigorous formulae describing this relationship in terms of internal reliability characteristics are derived both for the models with uncorrelated and correlated observations. A specific case of decorrelated observations is also taken into consideration. Finally, the criteria for the evaluation of the model internal reliability are proposed for all the above cases. It is worth mentioning that the criteria are obtained without resorting to any particular method of statistical testing. The theory is illustrated with two numerical examples, using simple measuring schemes.
TL;DR: The author uses the framework of structured secant methods to derive and investigate a new way to resize the approximation of the second-order term using some exact information, yielding a q-quadratic convergence rate for zero residual and aq-superlinear converge rate for nonzero residual problems.
Abstract: The problem of adjusting the second-order term in secant methods for nonlinear least squares problems with zero-residual is addressed. The author uses the framework of structured secant methods to derive and investigate a new way to resize the approximation of the second-order term using some exact information. The resulting method is a self-adjusting structured secant method for nonlinear least squares problems, yielding a q-quadratic convergence rate for zero residual and a q-superlinear convergence rate for nonzero residual problems.
TL;DR: In this article, the Stokes problem in 3D was studied and the mixed Galerkin method was used to solve the problem in the 3D case. But the analysis based on the results of Agmon, Douglis, Nirenberg, Wendland, Zienkiewicz, Owen, and Niles shows that this method is stable in the h-version.
Abstract: In this paper we are concerned with the Stokes problem in three dimensions (see recent works of the author and B. N. Jiang for the two-dimensional case). It is a linear system of four PDEs with velocity 11 and pressure p as unknowns. With the additional variable a = curl&, the second-order problem is reduced to a first-order system. Considering the compatibility condition d i v g = 0 , we have a system with eight first-order equations and seven unknowns. A least squares method is applied to this extended system, and also to the corresponding boundary conditions. The analysis based on works of Agmon, Douglis, and Nirenberg; Wendland; Zienkiewicz, Owen, and Niles; etc. shows that this method is stable in the h-version. For instance, if we choose continuous piecewise polynomials to approximate 11, a,and p , this method achieves optimal rates of convergence in the HI-norms. Let SZ be an open bounded and connected subset of IR3 with a smooth boundary T. Let be a given function representing the body f E [ L ~ ( R ) ] ~ force. The Stokes can be posed as where g , p with ( p , 1) = 0 , and v are respectively velocity, pressure, and kinematic viscosity (constant), all of which are assumed to be nondimensionalized. Over the past two decades many engineers and mathematicians have studied the above problem. The mixed Galerkin method solves this problem successfully. In most cases the elements are required to satisfy a saddle point condition [4, 5, 8, 9, 221, which is not necessary for our method. Received by the editor March 16, 1992 and, in revised form, October 12, 1992. 1991 Mathematics Subject Classification. Primary 65N30, 35F15. This work was performed when the author was an OAIJCWRU Summer Faculty fellow participant at NASA Lewis Research Center, Cleveland, OH (1991), and was revised at the University of Texas at Arlington (1992) and at Wright-Patterson Air Force Base, Dayton, OH (1993). @ 1994 American Mathematical Society 0025-5718194 $1.00 + $.25 per page
TL;DR: A new class of simplified low-cost analog artificial neural networks with on chip adaptive learning algorithms are proposed for solving linear systems of algebraic equations in real time.
Abstract: In this paper a new class of simplified low-cost analog artificial neural networks with on chip adaptive learning algorithms are proposed for solving linear systems of algebraic equations in real time. The proposed learning algorithms for linear least squares (LS), total least squares (TLS) and data least squares (DLS) problems can be considered as modifications and extensions of well known algorithms: the row-action projection-Kaczmarz algorithm and/or the LMS (Adaline) Widrow-Hoff algorithms. The algorithms can be applied to any problem which can be formulated as a linear regression problem. The correctness and high performance of the proposed neural networks are illustrated by extensive computer simulation results. >
TL;DR: In this article, the LMS/RLS method is extended to robust estimation of a distributed intercept model, and an algorithm for this situation is proposed, which is applied to estimation of earnings functions with different intercepts for industries, based on data from five countries.
TL;DR: In this paper, a new algorithm for the identification of continuous time nonlinear systems from sampled data records is derived based on state variable filters coupled with an orthogonal least squares estimator.
Abstract: A new algorithm for the identification of continuous time nonlinear systems from sampled data records is derived based on state variable filters coupled with an orthogonal least squares estimator. Delayed filtered inputs, outputs and associated higher order derivatives collected from the state variable filters are used for the identification of the unknown system parameters using an orthogonal least squares estimator. Because of the high dimensionality of general nonlinear systems the error reduction ratio derived from the orthogonal least squares estimator is used to detect the model structure. Plots of the unmodelled estimation errors against the state variables is also proposed as a means of investigating the nonlinear system characteristics. Simulation studies are included to illustrate the concepts.
TL;DR: A brief overview of robust alternatives to the least-squares mean is presented in this article, where a recent proposal called least median squares (LMS) is compared to least squares regression in an example involving the estimation of optical fiber geometry.
Abstract: Least squares is perhaps the most widely used technique for model fitting. In this article, we illustrate the poor performance of least squares when there are spurious values, or outliers, in a sequence of measurements. A brief overview of three well-known classes of robust alternatives to the least-squares mean is presented. For robust regression, a recent proposal called least median squares (LMS) is decribed. LMS regression is compared to least-squares regression in an example involving the estimation of optical fiber geometry. References are provided for software that is available for robust estimation techniques surveyed in this article.
TL;DR: In this paper, a p-version least square finite element formulation (LSFEF) was proposed for axisymmetric incompressible non-Newtonian fluid flow.
TL;DR: In this paper, the finite sample distributions of the least square estimator in a first order autoregressive model are computed numerically from an alternative approach for both stationary and nonstationary cases.