Showing papers on "Non-linear least squares published in 2002"

Penalized Spline Estimation for Partially Linear Single-Index Models

Abstract: Single-index models are potentially important tools for multivariate nonparametric regression. They generalize linear regression by replacing the linear combination α0Tx with a nonparametric component, η0(α0Tx), where η0(·) is an unknown univariate link function. By reducing the dimensionality from that of a general covariate vector x to a univariate index α0Tx, single-index models avoid the so-called “curse of dimensionality.” We propose penalized spline (P-spline) estimation of η0(·) in partially linear single-index models, where the mean function has the form η0(α0Tx) + β 0Tz. The P-spline approach offers a number of advantages over other fitting methods for single-index models. All parameters in the P-spline single-index model can be estimated simultaneously by penalized nonlinear least squares. As a direct least squares fitting method, our approach is rapid and computationally stable. Standard nonlinear least squares software can be used. Moreover, joint inference for η0(·), α0, and β0 is possible by...

An analysis of a least squares regression method for American option pricing

Abstract: Recently, various authors proposed Monte-Carlo methods for the computation of American option prices, based on least squares regression. The purpose of this paper is to analyze an algorithm due to Longstaff and Schwartz. This algorithm involves two types of approximation. Approximation one: replace the conditional expectations in the dynamic programming principle by projections on a finite set of functions. Approximation two: use Monte-Carlo simulations and least squares regression to compute the value function of approximation one. Under fairly general conditions, we prove the almost sure convergence of the complete algorithm. We also determine the rate of convergence of approximation two and prove that its normalized error is asymptotically Gaussian.

Consistency and efficiency of least squares estimation for mixed regressive, spatial autoregressive models

Abstract: Least squares estimation has casually been dismissed as an inconsistent estimation method for mixed regressive, spatial autoregressive models with or without spatial correlated disturbances. Although this statement is correct for a wide class of models, we show that, in economic spatial environments where each unit can be influenced aggregately by a significant portion of units in the population, least squares estimators can be consistent. Indeed, they can even be asymptotically efficient relative to some other estimators. Their computations are easier than alternative instrumental variables and maximum likelihood approaches.

Two highly efficient second-order algorithms for training feedforward networks

Abstract: We present two highly efficient second-order algorithms for the training of multilayer feedforward neural networks. The algorithms are based on iterations of the form employed in the Levenberg-Marquardt (LM) method for nonlinear least squares problems with the inclusion of an additional adaptive momentum term arising from the formulation of the training task as a constrained optimization problem. Their implementation requires minimal additional computations compared to a standard LM iteration. Simulations of large scale classical neural-network benchmarks are presented which reveal the power of the two methods to obtain solutions in difficult problems, whereas other standard second-order techniques (including LM) fail to converge.

Total Least Squares and Errors-in-Variables Modeling : Analysis, Algorithms and Applications

Abstract: Preface. Introduction to Total Least Squares and Errors-in-Variables Modeling S. Van Huffel, P. Lemmerling. Part I: Basic Concepts and Analysis in EIV Modeling. TLS and its Improvements by Semiparametric Approach S. Amari, M. Kawanabe. Unifying Least Squares, Total Least Squares and Data Least Squares C.C. Paige, Z. Strakos. Bounds for the Least Squares Residual Using Scaled TLS C.C. Paige, Z. Strakos. Part II: Total Least Squares Algorithms. Recent Developments in Rank Revealing and Lanczos Methods for Total Least Squares Related Problems R.D. Fierro, P.C. Hansen. A Regularized Total Least Squares Algorithm H. Guo, R.A. Renaut. The Parametric Quadratic Form Method for Solving TLS Problems with Elementwise Weighting A. Premoli, M.L. Rastello. Part III: Structured Total Least Squares Problems. Structured Total Least Squares: Analysis, Algorithms and Applications P. Lemmerling, S. Van Huffel. Fast Structured Total Least Squares Algorithms via Exploitation of the Displacement Structure N. Mastronardi, et al. The Extended STLS Algorithm for Minimizing the Extended LS Criterion A. Yeredor. Part IV: Nonlinear EIV Models and Statistical Estimators. Bayesian Smoothing for Measurement Error Problems S.M. Berry, et al. On the Polynomial Measurement Error Model Chi-Lun Cheng, H. Schneeweiss. On Consistent Estimators in Nonlinear Functional EIV Models A. Kukush, S. Zwanzig. On Consistent Estimators in Linear and Bilinear Multivariate Errors-in-Variables Models A. Kukush, et al. Identification of Semi-Linear Models Within an Errors-in-Variables Framework R. Pintelon, J. Schoukens. Cox's Proportional Hazards Model under Covariate Measurement Error T. Augustin, R. Schwarz. Part V: EIV Modeling with Bounded Uncertainties. State-Space Estimation with Uncertain Models A.H. Sayed, A. Subramanian. Models for Robust Estimation and Identification S. Chandrasekaran, K. Schubert. Robust Solutions to Linear Approximation Problems Under Ellipsoidal Uncertainty G.A. Watson. Part VI: Curve Fitting. QR Factorization of the Jacobian in Some Structured Nonlinear Least Squares Problems A. Bjorck. Neural Minor Component Analysis and TLS G. Cirrincione, M. Cirrincione. On the Structural Line Segment Model O. Davidov, et al. Model Fitting for Multiple Variables by Minimising the Geometric Mean Deviation C. Tofallis. Part VII: EIV Estimation in System Identification. Perspectives on Errors-in-Variables Estimation for Dynamic Systems T. Soderstrom, et al. Errors-in-Variables Filtering in Behavioural and State-Space Contexts, R. Guidorzi, et al. Weighted Total Least Squares, Rank Deficiency and Linear Matrix Structures B. De Moor. Frequency-Domain TLS and GTLS Algorithms for Modal Analysis Applications P. Verboven, et al. Part VIII: EIV Estimation in Signal Processing. A New Decimative Spectral Estimation Method with Unconstrained Model Order and Decimation Factor S.-E. Fotinea, et al. Modeling Audio with Damped Sinusoids Using Total Least Squares Algorithms W. Verhelst, et al. Real-Time TLS Algorithms in Gaussian and Impulse Noise Environments Da-Zeng Feng, et al. Part IX: EIV Applications in Other Fields. Efficient Computation of the Riemannian SVD in Total Least Squares Problems in Information Retrieval R.D. Fierro, M.W. Berry. Constrained Total Least Squares for Color Image Reconstructio

Application of equality constraints on variables during alternating least squares procedures

Abstract: We describe several methods of applying equality constraints while performing procedures that employ alternating least squares. Among these are mathematically rigorous methods of applying equality constraints, as well as approximate methods, commonly used in chemometrics, that are not mathematically rigorous. The rigorous methods are extensions of the methods described in detail in Lawson and Hanson's landmark text on solving least squares problems, which exhibit well-behaved least squares performance. The approximate methods tend to be easy to use and code, but they exhibit poor least squares behaviors and have properties that are not well understood. This paper explains the application of rigorous equality-constrained least squares and demonstrates the dangers of employing non-rigorous methods. We found that in some cases, upon initiating multivariate curve resolution with the exact basis vectors underlying synthetic data overlaid with noise, the approximate method actually results in an increase in the magnitude of residuals. This phenomenon indicates that the solutions for the approximate methods may actually diverge from the least squares solution. Copyright © 2002 John Wiley & Sons, Ltd.

Maximum likelihood fitting using ordinary least squares algorithms

Abstract: In this paper a general algorithm is provided for maximum likelihood fitting of deterministic models subject to Gaussian-distributed residual variation (including any type of non-singular covariance). By deterministic models is meant models in which no distributional assumptions are valid (or applied) on the parameters. The algorithm may also more generally be used for weighted least squares (WLS) fitting in situations where either distributional assumptions are not available or other than statistical assumptions guide the choice of loss function. The algorithm to solve the associated problem is called MILES (Maximum likelihood via Iterative Least squares EStimation). It is shown that the sought parameters can be estimated using simple least squares (LS) algorithms in an iterative fashion. The algorithm is based on iterative majorization and extends earlier work for WLS fitting of models with heteroscedastic uncorrelated residual variation. The algorithm is shown to include several current algorithms as special cases. For example, maximum likelihood principal component analysis models with and without offsets can be easily fitted with MILES. The MILES algorithm is simple and can be implemented as an outer loop in any least squares algorithm, e.g. for analysis of variance, regression, response surface modeling, etc. Several examples are provided on the use of MILES. Copyright © 2002 John Wiley & Sons, Ltd.

A unified approach to form error evaluation

Abstract: Evaluation of form error is a critical aspect of many manufacturing processes. Machines such as the coordinate measuring machine (CMM) often employ the technique of the least squares form fitting algorithms. While based on sound mathematical principles, it is well known that the method of least squares often overestimates the tolerance zone, causing good parts to be rejected. Many methods have been proposed in efforts to improve upon results obtained via least squares, including those, which result in the minimum zone tolerance value. However, these methods are mathematically complex and often computationally slow for cases where a large number of data points are to be evaluated. Extensive amount of data is generated where measurement equipment such as laser scanners are used for inspection, as well as in reverse engineering applications. In this report, a unified linear approximation technique is introduced for use in evaluating the forms of straightness, flatness, circularity, and cylindricity. Non-linear equation for each form is linearized using Taylor expansion, then solved as a linear program using software written in C++ language. Examples are taken from the literature as well as from data collected on a coordinate measuring machine for comparison with least squares and minimum zone results. For all examples, the new formulations are found to equal or better than the least squares results and provide a good approximation to the minimum zone tolerance.

Calibration of Jump-Diffusion Option Pricing Models: A Robust Non-Parametric Approach

Abstract: We present a non-parametric method for calibrating jump-diffusion models to a finite set of observed option prices. We show that the usual formulations of the inverse problem via nonlinear least squares are ill-posed and propose a regularization method based on relative entropy. We reformulate our calibration problem into a problem of finding a risk neutral jump-diffusion model that reproduces the observed option prices and has the smallest possible relative entropy with respect to a chosen prior model. Our approach allows to conciliate the idea of calibration by relative entropy minimization with the notion of risk neutral valuation in a continuous time model. We discuss the numerical implementation of our method using a gradient based optimization algorithm and show via simulation tests on various examples that the entropy penalty resolves the numerical instability of the calibration problem. Finally, we apply our method to datasets of index options and discuss the empirical results obtained.

An improved model function method for choosing regularization parameters in linear inverse problems

Abstract: This paper proposes a new model function iterative method, that improves our earlier work (Kunisch K and Zou J 1998 Inverse Problems 14 1247–64), on finding some reasonable regularization parameters in the widely used output least squares formulations of linear inverse problems, based on the Morozov and damped Morozov principles. The new algorithm updates the model parameters in a computationally more stable manner. In addition, the method can be rigorously shown to have global convergence, in particular, its convergence is carried on strictly monotone decreasingly. This property seems especially useful and important in real applications as it enables us to start with some larger regularization parameters, and thus with more stable least squares problems. Numerical experiments for one-and two-dimensional elliptic inverse problems and an inverse integral problem are presented to illustrate the efficiency of the proposed algorithm.

Concentration Residual Augmented Classical Least Squares (CRACLS): A Multivariate Calibration Method with Advantages over Partial Least Squares

Abstract: A significant extension to the classical least-squares (CLS) algorithm called concentration residual augmented CLS (CRACLS) has been developed. Previously, unmodeled sources of spectral variation have rendered CLS models ineffective for most types of problems, but with the new CRACLS algorithm, CLS-type models can be applied to a significantly wider range of applications. This new quantitative multivariate spectral analysis algorithm iteratively augments the calibration matrix of reference concentrations with concentration residuals estimated during CLS prediction. Because these residuals represent linear combinations of the unmodeled spectrally active component concentrations, the effects of these components are removed from the calibration of the analytes of interest. This iterative process allows the development of a CLS-type calibration model comparable in prediction ability to implicit multivariate calibration methods such as partial least squares (PLS) even when unmodeled spectrally active components are present in the calibration sample spectra. In addition, CRACLS retains the improved qualitative spectral information of the CLS algorithm relative to PLS. More importantly, CRACLS provides a model compatible with the recently presented prediction-augmented CLS (PACLS) method. The CRACLS/PACLS combination generates an adaptable model that can achieve excellent prediction ability for samples of unknown composition that contain unmodeled sources of spectral variation. The CRACLS algorithm is demonstrated with both simulated and real data derived from a system of dilute aqueous solutions containing glucose, ethanol, and urea. The simulated data demonstrate the effectiveness of the new algorithm and help elucidate the principles behind the method. Using experimental data, we compare the prediction abilities of CRACLS and PLS during cross-validated calibration. In combination with PACLS, the CRACLS predictions are comparable to PLS for the prediction of the glucose, ethanol, and urea components for validation samples collected when significant instrument drift was present. However, the PLS predictions required recalibration using nonstandard cross-validated rotations while CRACLS/PACLS was rapidly updated during prediction without the need for time-consuming cross-validated recalibration. The CRACLS/PACLS algorithm provides a more general approach to removing the detrimental effects of unmodeled components.

Estimating Lorenz Curves Using a Dirichlet Distribution

Abstract: The Lorenz curve relates the cumulative proportion of income to the cumulative proportion of population. When a particular functional form of the Lorenz curve is specified, it is typically estimated by linear or nonlinear least squares estimation techniques that have good properties when the error terms are independently and normally distributed. Observations on cumulative proportions are clearly neither independent nor normally distributed. This article proposes and applies a new methodology that recognizes the cumulative proportional nature of the Lorenz curve data by assuming that the income proportions are distributed as a Dirichlet distribution. Five Lorenz curve specifications are used to demonstrate the technique. Maximum likelihood estimates under the Dirichlet distribution assumption provide better fitting Lorenz curves than nonlinear least squares and another estimation technique that has appeared in the literature.

Estimation of input function and kinetic parameters using simulated annealing: application in a flow model

Abstract: Accurate determination of the input function is essential for absolute quantification of physiological parameters in positron emission tomography and single-photon emission computed tomography imaging, but it requires an invasive and tedious procedure of blood sampling that is impractical in clinical studies. We previously proposed a technique that estimates simultaneously kinetic parameters and the input function from the tissue impulse response functions and requires two blood samples. A nonlinear least squares method estimated all the parameters in the impulse response functions and the input function but failed occasionally due to high noise levels in the data, causing an ill-conditioned cost function. This paper investigates the feasibility of applying a Monte Carlo method called simulated annealing to estimate kinetic parameters in the impulse response functions and the input function. Time-activity curves of teboroxime, which is very sensitive to changes in the input function, were simulated based on published data obtained from a canine model. The equations describing the tracer kinetics in different regions were minimized simultaneously by simulated annealing and nonlinear least squares. We found that the physiological parameters obtained with simulated annealing are accurate, and the estimated input function more closely resembled the simulated curve. We conclude that simulated annealing reduces bias in the estimation of physiological parameters and determination of the input function.

Estimation of Rainfall Drop Size Distributions from Dual-Frequency Wind Profiler Spectra Using Deconvolution and a Nonlinear Least Squares Fitting Technique

Abstract: This paper compares two different analysis approaches for dual-frequency (VHF and UHF) wind profiler precipitation retrievals. The first technique is based on a general deconvolution process to remove broadening effects from the Doppler spectra due to turbulence and finite radar beamwidth. The second technique is based on a nonlinear least squares fitting approach, where an analytical function that describes the drop size distribution is convolved with a clear air distribution to model the broadening of the observed spectra. These techniques are tested with simulated data, where the true drop size distribution is known and represented by a gamma or exponential distribution, and with observations. The median volume diameter D0, which is based on the third moment of the drop size number distribution, is used to test these retrieval techniques. This parameter is closely related to the rain rate, which is approximately the 3.67th moment of the drop size number distribution. Methods for extracting D0 ...

Least Squares Solution of Matrix Equation $AXB^* + CYD^* = E$

Abstract: We present an efficient algorithm for the least squares solution (X, Y) of the matrix equation AXB* + CYD* = E with arbitrary coefficient matrices A, B, C, D and the right-hand side E. This method determines the least squares solution (X, Y) with the least norm. It relies on the SVD and generalized SVD of the coefficient matrices and has complexity proportional to the cost of these SVDs.

Time series prediction using support vector machines, the orthogonal and the regularized orthogonal least-squares algorithms

Abstract: Generalization properties of support vector machines, orthogonal least squares and zero-order regularized orthogonal least squares algorithms are studied using simulation. For high signal-to-noise ratios (40 dB), mixed results are obtained, but for a low signal-to-noise ratio, the prediction performance of support vector machines is better than the orthogonal least squares algorithm in the examples considered. However, the latter can usually give a parsimonious model with very fast training and testing time. Two new algorithms are therefore proposed that combine the orthogonal least squares algorithm with support vector machines to give a parsimonious model with good prediction accuracy in the low signal-to-noise ratio case.

Nonlinear calibration of a laser stripe profiler

Abstract: The calibration of a laser stripe profiler consisting of a laser stripe projector, camera, and linear motion table is considered. A nonlinear system model is used, which accommodates radial distortion in the camera lens. The way in which stripe data is extracted from the camera images leads to a natural formulation of the calibration problem as a nonlinear least squares problem. This can then be solved using standard techniques. The use of this nonlinear model reduces the error in the generated 3-D data by over an order of magnitude.

A note on least absolute deviation estimation of a threshold model

Abstract: This paper develops the limit law for the least absolute deviation estimator of the threshold parameter in linear regression. In this respect, we extend the literature of threshold models. The existing literature considers only the least squares estimation of the threshold parameter (see Chan, 1993, Annals of Statistics 21, 520–533; Hansen, 2000, Econometrica 68, 575–605). This result is useful because in the case of heavy-tailed errors there is an efficiency loss resulting from the use of least squares. Also, for the first time in the literature, we derive the limit law for the likelihood ratio test for the threshold parameter using the least absolute deviation technique.

Monotone approximation of aggregation operators using least squares splines

Abstract: The need for monotone approximation of scattered data often arises in many problems of regression, when the monotonicity is semantically important. One such domain is fuzzy set theory, where membership functions and aggregation operators are order preserving. Least squares polynomial splines provide great flexibility when modeling non-linear functions, but may fail to be monotone. Linear restrictions on spline coefficients provide necessary and sufficient conditions for spline monotonicity. The basis for splines is selected in such a way that these restrictions take an especially simple form. The resulting non-negative least squares problem can be solved by a variety of standard proven techniques. Additional interpolation requirements can also be imposed in the same framework. The method is applied to fuzzy systems, where membership functions and aggregation operators are constructed from empirical data.