Showing papers on "Non-linear least squares published in 1987"

The Total Least Squares Problem: Computational Aspects and Analysis

Abstract: Introduction Basic principles of the total least squares problem Extensions of the basic total least squares problem Direct speed Improvement of the total least squares computations Iterative speed Improvement for solving slowly varying total least squares problems Algebraic Connections Between total least squares and least squares problems Sensitivity analysis of total least squares and least squares problems in the presence of errors in all data Statistical properties of the total least squares problem Algebraic connections between total least squares estimation and classical linear regression in Multicollinearity problems Conclusions.

Nonlinear two-dimensional elastic inversion of multioffset seismic data

Abstract: The treatment of multioffset seismic data as an acoustic wave field is becoming increasingly disturbing to many geophysicists who see a multitude of wave phenomena, such as amplitude-offset variations and shear-wave events, which can only be explained by using the more correct elastic wave equation. Not only are such phenomena ignored by acoustic theory, but they are also treated as undesirable noise when they should be used to provide extra information, such as S-wave velocity, about the subsurface. The problems of using the conventional acoustic wave equation approach can be eliminated via an elastic approach. In this paper, equations have been derived to perform an inversion for P-wave velocity, S-wave velocity, and density as well as the P-wave impedance, S-wave impedance, and density. These are better resolved than the Lame parameters. The inversion is based on nonlinear least squares and proceeds by iteratively updating the earth parameters until a good fit is achieved between the observed data and the modeled data corresponding to these earth parameters. The iterations are based on the preconditioned conjugate gradient algorithm. The fundamental requirement of such a least-squares algorithm is the gradient direction which tells how to update the model parameters. The gradient direction can be derived directly from the wave equation and it may be computed by several wave propagations. Although in principle any scheme could be chosen to perform the wave propagations, the elastic finite-difference method is used because it directly simulates the elastic wave equation and can handle complex, and thus realistic, distributions of elastic parameters. This method of inversion is costly since it is similar to an iterative prestack shot-profile migration. However, it has greater power than any migration since it solves for the P-wave velocity, S-wave velocity, and density and can handle very general situations including transmission problems. Three main weaknesses of this technique are that it requires fairly accurate a priori knowledge of the low-wavenumber velocity model, it assumes Gaussian model statistics, and it is very computer-intensive. All these problems seem surmountable. The low-wavenumber information can be obtained either by a prior tomographic step, by the conventional normal-moveout method, by a priori knowledge and empirical relationships, or by adding an additional inversion step for low wavenumbers to each iteration. The Gaussian statistics can be altered by preconditioning the gradient direction, perhaps to make the solution blocky in appearance like well logs, or by using large model variances in the inversion to reduce the effect of the Gaussian model constraints. Moreover, with some improvements to the algorithm and more parallel computers, it is hoped the technique will soon become routinely feasible.

A Stable and Efficient Algorithm for Nonlinear Orthogonal Distance Regression

Abstract: One of the most widely used methodologies in scientific and engineering research is the fitting of equations to data by least squares. In cases where significant observation errors exist in the independent variables as well as the dependent variables, however, the ordinary least squares (OLS) approach, where all errors are attributed to the dependent variable, is often inappropriate. An alternate approach, suggested by several researchers, involves minimizing the sum of squared orthogonal distances between each data point and the curve described by the model equation. We refer to this as orthogonal distance regression (ODR). This paper describes a method for solving the orthogonal distance regression problem that is a direct analog of the trust region Levenberg-Marquardt algorithm. The number of unknowns involved is the number of model parameters plus the number of data points, often a very large number. By exploiting sparsity, however, our algorithm has a computational effort per step which is of the same order as required for the Levenberg-Marquardt method for ordinary least squares. We prove our algorithm to be globally and locally convergent, and perform computational tests that illustrate some differences between ODR and OLS.

Total least squares approach for frequency estimation using linear prediction

Abstract: The resolution of the estimated closely spaced frequencies of the multiple sinusoids degrades as the signal-to-noise ratio (SNR) of the received signal becomes low. This resolution can be improved by using the total least squares (TLS) method in solving the linear prediction (LP) equation. This approach makes use of the singular value decomposition (SVD) of the augmented matrix for low rank approximation to reduce the noise effect from both the observation vector and the LP data matrix simultaneously. Comparison is made to the principle eigenvector (PE) method of Tufts and Kumaresan, both on theoretical and experimental grounds. The TLS algorithm exhibits superior performance over the PE method where low rank approximation is applied to the data matrix only.

Stokes profile analysis and vector magnetic fields. I. Inversion of photospheric lines

Abstract: Improvements are proposed for the Auer et al. (1977) method for the analytic inversion of Stokes profiles via nonlinear least squares. The introduction of additional physics into the Mueller absorption matrix (by including damping wings and magnetooptical birefringence, and by decoupling the intensity profile from the three-vector polarization profile in the analysis) is found to result in a more robust inversion method, providing more reliable and accurate estimates of sunspot vector magnetic fields without significant loss of economy. The method is applied to sunspot observations obtained with the High Altitude Observatory polarimeter. 29 references.

Computational Experience With Confidence Regions and Confidence Intervals for Nonlinear Least Squares

Abstract: We present the results of a Monte Carlo study of the leading methods for constructing approximate confidence regions and confidence intervals for parameters estimated by nonlinear least squares. We examine three variants of the linearization method, the likelihood method, and the lack-of-fit method. The linearization method is computationally inexpensive, produces easily understandable results, and is widely used in practice. The likelihood and lack-of-fit methods are much more expensive and more difficult to report. In our tests, both the likelihood and lack-of-fit methods perform very reliably. All three variants of the linearization method, however, often grossly underestimate confidence regions and sometimes significantly underestimate confidence intervals. The linearization method variant based solely on the Jacobian matrix appears preferable to the two variants that use the full Hessian matrix because it is less expensive, more numerically stable, and at least as accurate. The Bates and Watts curvat...

The spherical interpolation method of source localization

Abstract: A closed-form least squares approximate maximum likelihood method for localization of broad-band emitters from time-difference-of-arrival (TDOA) measurements, called the spherical interpolation (SI) method, is presented. The localization formula is derived from least squares "equation-error" minimization. Computer simulation results show that the SI method has variance approaching the Cramer-Rap lower bound.

A Comparison of Methods for Estimating the Functional Response Parameters of the Random Predator Equation

Abstract: (1) Simulations of functional response experiments were used to compare Rogers's linearization, nonlinear least squares, and a new nonparametric parameter space method as means of estimating parameters of the random predator equation. (2) Rogers's linearization gave highly biased estimates of both parameters. These estimates were consistently too low. (3) Nonlinear least squares using the implicit form of the equation, and the nonparametric parameter space method provided equally good estimates for data sets that departed only moderately from the assumptions of homogeneous, normally distributed error. For data sets with greater heteroscedasticity and more extreme nonnormality, the nonparametric procedure performed slightly better than nonlinear least squares. This was especially true for smaller data sets. (4) For both methods, actual frequencies of erroneous rejection of true null hypotheses were usually significantly greater than the nominal 0-05 for the cases studied. Even a moderate amount of heteroscedasticity seems to effect the probability of type I error for both methods. (5) The practice of analysing average values of number of prey eaten at each value of number of prey offered was compared to using individual data points. Use of average numbers eaten did not alter point estimates of parameters, but produced severe underestimates of S.E.s of parameter estimates. Use of average numbers eaten increased probability of type I error by 16 to 42%. (6) Simulation studies such as this may be generally useful to biologists as tools of applied statistics.

Improved least squares identification

Abstract: An improved recursive least squares algorithm for parameter estimation is presented which includes: on/off criteria to prevent parameter drift during periods of low excitation; a variable forgetting factor which maintains the trace of the covariance matrix at a user-specified value; data preprocessing and normalization to improve numerical accuracy; scaling of the regressor vector to minimize the condition number of the covariance matrix; plus independent estimation of the mean values of the input/output data which can be used to eliminate errors due to d.c. bias or slowly drifting elements in the regressor vector. The algorithm can also include parameter projection to constrain the estimates to a priori specified regions and retains the formal properties, such as convergence, of a true weighted least squares algorithm. The proposed algorithm is compared with other modifications suggested in the literature, and its advantages are demonstrated by a simulated example.

Hybrid Methods for Nonlinear Least Squares

Abstract: Etude de la convergence des methodes hybrides de resolution des problemes des moindres carres non lineaires. Conditions de convergenc lineaire ou superlineaire

Constrained total least squares

Abstract: The Total Least Squares (TLS) method is a generalized least square technique to solve an overdetermined system of equations Ax\simeqb . The TLS solution differs from the usual Least Square (LS) in that it tries to compensate for arbitrary noise present in both A and b . In certain problems the noise perturbations of A and b are linear functions of a common "noise source" vector. In this case we obtain a generalization of the TLS criterion called the Constrained Total Least Squares (CTLS) method by taking into account the linear dependence of the noise terms in A and b . If the noise columns of A and b are linearly related then the CTLS solution is obtained in terms of the largest eigenvalue and corresponding eigenvector of a certain matrix. The CTLS technique can be applied to problems like Maximum Likelihood Signal Parameter Estimation, Frequency Estimation of Sinusoids in white or colored noise by Linear Prediction and others.

ODRPACK Software for Weighted Orthogonal Distance Regression.

Abstract: : This paper describes ORDPACK, a software package for the Orthogonal Distance Regression (ODR) problem. This software implements the algorithm for finding the set of parameters that minimize the sum of the squared orthogonal distances from a set of observations to a curve determined by the parameters. It can also be used to solve the ordinary nonlinear least squares problem. The ODR procedure has application to curve fitting, and to the errors in variables problem in statistics. The algorithm implemented is an efficient and stable trust region (Levenberg-Marquardt) procedure that exploits the structure of the problem so that the computational cost per iteration is equal to that for the same type of algorithm applied to the ordinary nonlinear least squares problem. The package allows a general weighting scheme, provides for finite differences derivatives, and contains extensive error checking and report generating facilities.

Theis' Solution by Nonlinear Least-Squares and Finite-Difference Newton's Method

Abstract: A method using the nonlinear least-squares and finite-difference Newton's method to determine the aquifer parameters via a pumping test in a homogeneous and isotropic confined aquifer system is proposed. The nonlinear least-squares is used to find the values of transmissivity and storage coefficient such that the sum of the squares of differences between the predicted drawdowns and observed drawdowns is minimized. The finite-difference Newton's method is used to solve the system of nonlinear least-squares equations for transmissivity and storage coefficient. Comparisons of the results between the proposed method and graphical methods including the Theis, Cooper-Jacob, and Chow methods are discussed in detail, showing data of a 6-hour pumping test. The proposed method has the advantages of high accuracy and quick convergence for most initial guesses.

Advances in Econometrics: Identification and consistency in semi-nonparametric regression

Abstract: Nonlinear least squares is the prototypical problem for establishing the consistency of nonlinear econometric estimators in the sense that the analysis abstracts easily and the abstraction covers the standard methods of estimation in econometrics: instrumental variables, two- and three-stage least squares, full information maximum likelihood, seemingly unrelated regression, M -estimators, scale-invariant M -estimators, generalized method of moments, and so on (Burguete, Gallant, and Souza 1982; Gallant and White 1986). In this chapter, nonlinear least squares is adapted to a function space setting where the estimator is regarded as a point in a function space rather than a point in a finite-dimensional, Euclidean space. Questions of identification and consistency are analyzed in this setting. Least squares retains its prototypical status: The analysis transfers directly to both the above listed methods of inference on a function space and to semi-nonparametric estimation methods. Two semi-nonparametric examples, the Fourier consumer demand system (Gallant 1981) and semi-nonparametric maximum likelihood applied to nonlinear regression with sample selection (Gallant and Nychka 1987), are used to illustrate the ideas. Introduction The intent of a semi-nonparametric methodology is to endow parametric inference with the nonparametric property of asymptotic validity against any true state of nature. The idea is to set forth a sequence of finite dimensional, parametric models that can approximate any true state of nature in the limit with respect to an appropriately chosen norm. As sample size increases, one progresses along this sequence of models. The method is parametric.

OMNE: a new robust membership-set estimator for the parameters of nonlinear models.

Abstract: A new method for estimating parameters and their uncertainty is presented. Data are assumed to be corrupted by a noise whose statistical properties are unknown but for which bounds are available at each sampling time. The method estimates the set of all parameter vectors consistent with this hypothesis. Its results are compared with those of the weighted least squares, extended least squares, and biweight robust regression approaches on two data sets, one of which includes 33% outliers. On the basis of these preliminary results, the new method appears to have attractive properties of reliability and robustness.

Errors in variables in linear systems

Abstract: This paper extends the simple errors-in-variable bound to the setting of systems of equations. Both diagonal and nondiagonal measurement error covariance matrices are considered. In the nondiagonal case, the analogue of the simple errors-in-variable interval of estimates is an ellipsoid with diagonal equal to the line segment connecting the direct least squares with a two-stage least squares estimate. For the diagonal case, the set of estimates under some conditions must lie within the convex hull of 2k points.

On Large Scale Nonlinear Least Squares Calculations

Abstract: The nonlinear model fitting problem is analyzed in this paper, with special emphasis on the practical solution techniques when the number of parameters in the model is large. Classical approaches to small dimensional least squares calculations are reviewed and an extension of them to problems involving many variables is proposed. This extension uses the context of partially separable structures, which has already proved its applicability for large scale optimization. An adaptable algorithm is discussed, which chooses between various possible models of the objective function. Preliminary numerical experience is also presented, which shows that actual numerical solution of a large class of fitting problems involving several hundreds of nonlinear parameters is possible at a reasonable cost.

Recursive Least Squares

Abstract: There appears to be a substantial amount of criticism levelled these days at the deleterious effect that computers are having on the algebraic (especially manipulative) skills of students. Clearly computers do have an important part to play in a subject such as statistics but the article by Searle (1983) reminds us that sensible algebraic pre-planning can often improve the accuracy of estimation procedures many of which will be carried out by computer. There are a wide variety of problems in the real world (steering supertankers, short-term prediction of power loads and economic planning are typical examples) which students are happy to discuss and where there is a rapid awareness that model structures are updated regularly as new data becomes available. When the model structure is well understood and data becomes available a t regular intervals of time, and particularly where the requirement is for rapid updating, then recursive or online procedures are most useful. Many recursive algorithms require substantial algebraic manipulation to make them easy to implement on a computer. Once this manipulation is completed however, the updating can be carried out rapidly and data storage is minimal.

A new analysis technique for estimating zonal irregularity drifts and variability in the equatorial F region using spaced receiver scintillation data

Abstract: A new technique for analyzing spaced receiver measurements of equatorial scintillation is applied to VHF scintillation data. The technique is based on a model that includes both propagation effects and the statistical characteristics of scintillation-producing irregulatities. Nonlinear least squares fitting is used to fit the model to measured auto- and cross-correlation functions of the signal amplitude fading on spaced receivers. The results are compared with mean and “random” drift estimates obtained using the classical type of correlation analysis.

Exact least squares multi-step prediction from nonlinear autoregressive models

Abstract: . We obtain an integral equation recurrence relation for the optimal least squares predictor of a nonlinear autoregressive time series model. Numerical solutions are given for a first order threshold model up to three steps ahead.

The inverse problem for confined aquifer flow: Identification and estimation with extensions

Abstract: The contributions of this work are twofold. First, a methodology for estimating the elements of parameter matrices in the governing equation of flow in a confined aquifer is developed. The estimation techniques for the distributed-parameter inverse problem pertain to linear least squares and generalized least squares methods. The linear relationship among the known heads and unknown parameters of the flow equation provides the background for developing criteria for determining the identifiability status of unknown parameters. Under conditions of exact or overidentification it is possible to develop statistically consistent parameter estimators and their asymptotic distributions. The estimation techniques, namely, two-stage least squares and three stage least squares, are applied to a specific groundwater inverse problem and compared between themselves and with an ordinary least squares estimator. The three-stage estimator provides the closer approximation to the actual parameter values, but it also shows relatively large standard errors as compared to the ordinary and two-stage estimators. The estimation techniques provide the parameter matrices required to simulate the unsteady groundwater flow equation. Second, a nonlinear maximum likelihood estimation approach to the inverse problem is presented. The statistical properties of maximum likelihood estimators are derived, and a procedure to construct confidence intervals and do hypothesis testing is given. The relative merits of the linear and maximum likelihood estimators are analyzed. Other topics relevant to the identification and estimation methodologies, i.e., a continuous-time solution to the flow equation, coping with noise-corrupted head measurements, and extension of the developed theory to nonlinear cases are also discussed. A simulation study is used to evaluate the methods developed in this study.

A generalized Guass-Newton procedure for multi-response parameter estimation

Abstract: A method is presented for calculating the gradient and an approximate Hessian of the Box-Draper multi-response parameter estimation criterion using only the first-order derivatives of the model functions. This is an analogue of the Gauss-Newton iterative procedure for nonlinear least squares. We also describe an implementation based on a $QR$ decomposition of the residual matrix which allows incorporation of a regularization procedure similar to the Levenburg-Marquardt method in nonlinear least squares.The method incorporates a convergence criterion based on a comparison of the increment size to the statistical variability of the estimates.

Performance comparison of least squares and least mean squares algorithms as HF channel estimators

Abstract: In this paper a comparison is made between the convergence and tracking properties of Least Squares (LS) and Least Mean Squares (LMS) algorithms as high frequency (HF) channel estimators. Theoretical results are derived for the asymptotic error achieved by the LS algorithms under white-input conditions in the HF channel. This result is more accurate than previous analyses of LS algorithms in a nonstationary enviroment [5,8,9]. Utilising a state space definition of the channel model a minimum variance Kalman estimator is derived using the a-priori knowledge of the parameters which define the Markov process.