Showing papers on "Non-linear least squares published in 2000"
TL;DR: In this paper, a method for assessing the uncertainty of the individual bilinear model parameters from two-block regression modelling by multivariate partial least squares regression (PLSR) is presented.
715 citations
28 May 2000
TL;DR: This paper investigates imposing sparseness by pruning support values from the sorted support value spectrum which results from the solution to the linear system.
Abstract: In least squares support vector machines (LS-SVMs) for function estimation Vapnik's /spl epsiv/-insensitive loss function has been replaced by a cost function which corresponds to a form of ridge regression. In this way nonlinear function estimation is done by solving a linear set of equations instead of solving a quadratic programming problem. The LS-SVM formulation also involves less tuning parameters. However, a drawback is that sparseness is lost in the LS-SVM case. In this paper we investigate imposing sparseness by pruning support values from the sorted support value spectrum which results from the solution to the linear system.
319 citations
TL;DR: It is shown that the algorithm of IEEE-STD-1057 provides accurate estimates for Gaussian and quantization noise and in the Gaussian scenario it provides estimates with performance close to the derived lower bound.
Abstract: The IEEE Standard 1057 (IEEE-STD-1057) provides algorithms for fitting the parameters of a sine wave to noisy discrete time observations. The fit is obtained as an approximate minimizer of the sum of squared errors, i.e., the difference between observations and model output. The contributions of this paper include a comparison of the performance of the four-parameter algorithm in the standard with the Cramer-Rao lower bound on accuracy, and with the performance of a nonlinear least squares approach. It is shown that the algorithm of IEEE-STD-1057 provides accurate estimates for Gaussian and quantization noise. In the Gaussian scenario it provides estimates with performance close to the derived lower bound. In severe conditions with noisy data covering only a fraction of a period, however, it is shown to have inferior performance compared with a one-dimensional search of a concentrated cost function.
183 citations
TL;DR: It is shown that substantial improvement over SOBI can be attained when the joint diagonalization is transformed into a properly weighted nonlinear least squares problem.
Abstract: Blind separation of Gaussian sources with different spectra can be attained using second-order statistics. The second-order blind identification (SOBI) algorithm, proposed by Belouchrani et al. (1997), uses approximate joint diagonalization. We show that substantial improvement over SOBI can be attained when the joint diagonalization is transformed into a properly weighted nonlinear least squares problem. We provide an iterative solution and derive the optimal weights for our weights-adjusted SOBI (WASOBI) algorithm. The improvement is demonstrated by analysis and simulations.
135 citations
01 May 2000
TL;DR: In this article, a method of determining the coefficients in a Prony series representation of a viscoelastic modulus from rate dependent data is presented, where load versus time test data for a sequence of different rate loading segments is least-squares fitted to a hereditary integral model of the material tested.
Abstract: In this study a method of determining the coefficients in a Prony series representation of a viscoelastic modulus from rate dependent data is presented. Load versus time test data for a sequence of different rate loading segments is least-squares fitted to a Prony series hereditary integral model of the material tested. A nonlinear least squares regression algorithm is employed. The measured data includes ramp loading, relaxation, and unloading stress-strain data. The resulting Prony series which captures strain rate loading and unloading effects, produces an excellent fit to the complex loading sequence.
133 citations
TL;DR: It is shown that the least squares and linear Taylor expansion based approach compares favorably with other analytic approaches, and that it is an efficient and economic alternative to the nonanalytic and computationally intensive bootstrap methods.
88 citations
TL;DR: The residuals method was found to be more sensitive than a simple t test, while not producing false-positive results, and it was showed that this method reliably differentiates changes in radioligand binding from the effects of changes in cerebral blood flow.
79 citations
TL;DR: In this article, it is shown that apart from a few square roots the problem is usually easily and robustly solved without iteration by employing standard techniques from linear algebra, and that in some cases it is more appropriate to formulate a nonlinear least squares problem in order to identify a ''best approximate solution''.
Abstract: The problem of determining the points of intersection of n spheres in R n has many applications. Examples in 3-D include problems in navigation, in positioning of specific atoms in crystal structures, in reconstructing torso geometries in experimental cardiology, in the `Pentacle Problem,' and in many other problems of distance geometry. The problem is easily formulated as a system of n nonlinear equations in the coordinates of the unknown point(s) of intersection and it is of interest to determine an efficient and reliable method of solution. It is shown that apart from a few square roots the problem is usually easily and robustly solved without iteration by employing standard techniques from linear algebra. In some applications, however, the radii of the spheres may not be known accurately and this can lead to difficulties, particularly when the required point is close to lying in the affine subspace defined by the n centres of the spheres. In such cases it is more appropriate to formulate a nonlinear least squares problem in order to identify a `best approximate solution.' The special structure of this nonlinear least squares problem allows a solution to be calculated through an efficient safeguarded Newton iteration.
72 citations
TL;DR: In this article, an optimal solution to the problem of determining both vehicle attitude and position using line-of-sight measurements is presented, which is derived from a generalized predictive filter for nonlinear systems.
Abstract: In this paper an optimal solution to the problem of determining both vehicle attitude and position using line-of-sight measurements is presented. The new algorithm is derived from a generalized predictive filter for nonlinear systems. This uses a one time-step ahead approach to propagate a simple kinematics model for attitude and position determination. The new algorithm is noniterative and is computationally efficient, which has significant advantages over traditional nonlinear least squares approaches. The estimates from the new algorithm are optimal in a probabilistic sense since the attitude/position covariance matrix is shown to be equivalent to the Cramer-Rao lower bound. Also, a covariance analysis proves that attitude and position determination is unobservable when only two line-of-sight observations are available. The performance of the new algorithm is investigated using line-of-sight measurements from a simulated sensor incorporating Position Sensing Diodes in the focal plane of a camera. Results indicate that the new algorithm provides optimal attitude and position estimates, and is robust to initial condition errors.
72 citations
TL;DR: Partial least squares regression has been widely adopted within some areas as a useful alternative to ordinary least square regression in the manner of other shrinkage methods such as principal components regression and ridge regression as discussed by the authors.
Abstract: Partial least squares regression has been widely adopted within some areas as a useful alternative to ordinary least squares regression in the manner of other shrinkage methods such as principal components regression and ridge regression. In this paper we examine the nature of this shrinkage and demonstrate that partial least squares regression exhibits some undesirable properties.
71 citations
TL;DR: A fast algorithm for the basic deconvolution problem is developed due to the low displacement rank of the involved matrices and the sparsity of the generators and Monte-Carlo simulations indicate the superior statistical performance of the structured total least squares estimator compared to other estimators such as the ordinary total least square estimator.
Abstract: In this paper we develop a fast algorithm for the basic deconvolution problem. First we show that the kernel problem to be solved in the basic deconvolution problem is a so-called structured total least squares problem. Due to the low displacement rank of the involved matrices and the sparsity of the generators, we are able to develop a fast algorithm. We apply the new algorithm on a deconvolution problem arising in a medical application in renography. By means of this example, we show the increased computational performance of our algorithm as compared to other algorithms for solving this type of structured total least squares problem. In addition, Monte-Carlo simulations indicate the superior statistical performance of the structured total least squares estimator compared to other estimators such as the ordinary total least squares estimator.
TL;DR: The combination of the value of the bivariate data set and their nonzero covariance justifies the need for maximum likelihood estimation over the simpler nonlinear least squares regression.
Abstract: A nonlinear regression technique for estimat- ing the Monod parameters describing biodegradation ki- netics is presented and analyzed. Two model data sets were taken from a study of aerobic biodegradation of the polycyclic aromatic hydrocarbons (PAHs), naphthalene and 2-methylnaphthalene, as the growth-limiting sub- strates, where substrate and biomass concentrations were measured with time. For each PAH, the parameters estimated were: qmax, the maximum substrate utilization rate per unit biomass; KS, the half-saturation coefficient; and Y, the stoichiometric yield coefficient. Estimating pa- rameters when measurements have been made for two variables with different error structures requires a tech- nique more rigorous than least squares regression. An optimization function is derived from the maximum likelihood equation assuming an unknown, nondiagonal covariance matrix for the measured variables. Because the derivation is based on an assumption of normally distributed errors in the observations, the error struc- tures of the regression variables were examined. Through residual analysis, the errors in the substrate concentration data were found to be distributed log- normally, demonstrating a need for log transformation of this variable. The covariance between ln C and X was found to be small but significantly nonzero at the 67% confidence level for NPH and at the 94% confidence level for 2MN. The nonlinear parameter estimation yielded unique values for qmax, KS, and Y for naphthalene. Thus, despite the low concentrations of this sparingly soluble compound, the data contained sufficient information for parameter estimation. For 2-methylnaphthalene, the val- ues of qmax and KS could not be estimated uniquely; however, qmax/KS was estimated. To assess the value of including the relatively imprecise biomass concentra- tion data, the results from the bivariate method were compared with a univariate method using only the sub- strate concentration data. The results demonstrated that the bivariate data yielded a better confidence in the estimates and provided additional information about the model fit and model adequacy. The combination of the value of the bivariate data set and their nonzero co- variance justifies the need for maximum likelihood esti- mation over the simpler nonlinear least squares regres- sion. © 2000 John Wiley & Sons, Inc. Biotechnol Bioeng 69: 160-170, 2000.
TL;DR: In this contribution, an extensive review of variable metric methods and their use in various optimization fields is given.
27 Jul 2000
TL;DR: Simulations of large scale classical neural network benchmarks are presented which reveal the power of the method to obtain solutions in difficult problems whereas other standard second order techniques (including LM) fail to converge.
Abstract: We present a highly efficient second order algorithm for the training of feedforward neural networks. The algorithm is based on iterations of the form employed in the Levenberg-Marquardt (LM) method for nonlinear least squares problems with the inclusion of an additional adaptive momentum term arising from the formulation of the training task as a constrained optimization problem. Its implementation requires minimal additional computations compared to a standard LM iteration which are compensated, however, from its excellent convergence properties. Simulations of large scale classical neural network benchmarks are presented which reveal the power of the method to obtain solutions in difficult problems whereas other standard second order techniques (including LM) fail to converge.
TL;DR: In this article, a nonlinear instantaneous least squares (NILS) estimator is proposed for signal parameter search. But the NILS estimator can be interpreted as an estimator based on the prediction error of a linear predictor.
Abstract: A novel method for signal parameter estimation is presented, termed the nonlinear instantaneous least squares (NILS) estimator. The basic idea is to use the observations in a sliding window to compute an instantaneous (short-term) estimate of the amplitude used in the separated nonlinear least squares (NLLS) criterion. The effect is a significant improvement of the numerical properties in the criterion function, which becomes well-suited for a signal parameter search. For small-sized sliding windows, the global minimum in the NLIS criterion function is wide and becomes easy to find. For maximum size windows, the NILS is equivalent to the NLLS estimator, which implies statistical efficiency for Gaussian noise. A "blind" signal parameter search algorithm that does not use any a priori information is proposed. The NILS estimator can be interpreted as a signal-subspace projection-based algorithm. Moreover, the NILS estimator can be interpreted as an estimator based on the prediction error of a (structured) linear predictor. Hereby, a link is established between NLLS, signal-subspace fitting, and linear prediction-based estimation approaches. The NILS approach is primarily applicable to deterministic signal models. Specifically, polynomial-phase signals are studied, and the NILS approach is evaluated and compared with other approaches. Simulations show that the signal-to-noise ratio (SNR) threshold is significantly lower than that of the other methods, and it is confirmed that the estimates are statistically efficient. Just as the NLLS approach, the NILS estimator can be applied to nonuniformly sampled data.
TL;DR: In this paper, the authors formulate and evaluate weighted least squares (WLS) and ordinary least square (OLS) procedures for estimating the parametric mean-value function of a nonhomogeneous Poisson process.
Abstract: We formulate and evaluate weighted least squares (WLS) and ordinary least squares (OLS) procedures for estimating the parametric mean-value function of a nonhomogeneous Poisson process. We focus the development on processes having an exponential rate function, where the exponent may include a polynomial component or some trigonometric components. Unanticipated problems with the WLS procedure are explained by an analysis of the associated residuals. The OLS procedure is based on a square root transformation of the "detrended" event (arrival) times - that is, the fitted mean-value function evaluated at the observed event times; and under appropriate conditions, the corresponding residuals are proved to converge weakly to a normal distribution with mean 0 and variance 0.25. The results of a Monte Carlo study indicate the advantages of the OLS procedure with respect to estimation accuracy and computational efficiency.
TL;DR: A Monte Carlo investigation of a number of variants of cross- validation for the assessment of performance of predictive models, including different values of k in leave-k-out cross-validation, and implementation either in a one-deep or a two-deep fashion.
Abstract: We describe a Monte Carlo investigation of a number of variants of cross-validation for the assessment of performance of predictive models, including different values of k in leave-k-out cross-validation, and implementation either in a one-deep or a two-deep fashion. We assume an underlying linear model that is being fitted using either ridge regression or partial least squares, and vary a number of design factors such as sample size n relative to number of variables p, and error variance. The investigation encompasses both the non-singular (i.e. n > p) and the singular (i.e. n ≤ p) cases. The latter is now common in areas such as chemometrics but has as yet received little rigorous investigation. Results of the experiments enable us to reach some definite conclusions and to make some practical recommendations.
TL;DR: In this article, robust schemes in regression are adapted to mean and covariance structure analysis, providing an iteratively reweighted least squares approach to robust structural equation modeling, which reduces to a standard distribution-free methodology if all cases are equally weighted.
Abstract: Robust schemes in regression are adapted to mean and covariance structure analysis, providing an iteratively reweighted least squares approach to robust structural equation modeling. Each case is properly weighted according to its distance, based on first and second order moments, from the structural model. A simple weighting function is adopted because of its flexibility with changing dimensions. The weight matrix is obtained from an adaptive way of using residuals. Test statistic and standard error estimators are given, based on iteratively reweighted least squares. The method reduces to a standard distribution-free methodology if all cases are equally weighted. Examples demonstrate the value of the robust procedure.
TL;DR: The proof given here is considerably more detailed than the original one and may well amount to overkill: there ought to be a more elementary way to show that dimFD(E) < m when E is the set on which rank FD' < m, especially in the situation where E is an algebraic variety.
Abstract: (2000). Sums of Squares of Polynomials. The American Mathematical Monthly: Vol. 107, No. 9, pp. 813-821.
TL;DR: In this article, an iterative target testing (TT) procedure is used to estimate rate constant estimates when the spectroscopic data is influenced by additional sources of variance, such as baseline drift or absorption shifts.
TL;DR: In this article, the shape restrictions are translated into linear inequality conditions on spline coefficients and the basis functions are selected in such a way that these conditions take a simple form, and the problem becomes non-negative least squares problem, for which effecitive and robust methods of solution exist.
Abstract: Least squares polynomial splines are an effective tool for data fitting, but they may fail to preserve essential properties of the underlying function, such as monotonicity or convexity. The shape restrictions are translated into linear inequality conditions on spline coefficients. The basis functions are selected in such a way that these conditions take a simple form, and the problem becomes non-negative least squares problem, for which effecitive and robust methods of solution exist. Multidimensional monotone approximation is achieved by using tensor-product splines with the appropriate restrictions. Additional inter polation conditions can also be introduced. The conversion formulas to traditional B-spline representation are provided.
TL;DR: In this paper, the perturbation analysis of two important problems for solving ill-conditioned or rank-deficient linear least squares problems is considered, namely the Tikhonov regularized problem and the minimum-norm problem.
Abstract: We consider the perturbation analysis of two important problems for solving ill-conditioned or rank-deficient linear least squares problems. The Tikhonov regularized problem is a linear least squares problem with a regularization term balancing the size of the residual against the size of the weighted solution. The weight matrix can be a non-square matrix (usually with fewer rows than columns). The minimum-norm problem is the minimization of the size of the weighted solutions given by the set of solutions to the, possibly rank-deficient, linear least squares problem. It is well known that the solution of the Tikhonov problem tends to the minimum-norm solution as the regularization parameter of the Tikhonov problem tends to zero. Using this fact and the generalized singular value decomposition enable us to make a perturbation analysis of the minimum-norm problem with perturbation results for the Tikhonov problem. From the analysis we attain perturbation identities for Tikhonov inverses and weighted pseudoinverses.
TL;DR: KaleidaGraph as discussed by the authors is a scientific data analysis and presentation tool used in the undergraduate physical chemistry teaching laboratory, and it is used for the analysis of a straight line through the origin and its transformation into a weighted average.
Abstract: A scientific data analysis and presentation program (KaleidaGraph, Synergy Software, available for both Macintosh and Windows platforms) is adopted as the main data analysis tool in the undergraduate physical chemistry teaching laboratory. The capabilities of this program (and others of this type) are illustrated with application to some common data analysis problems in the laboratory curriculum, most of which require nonlinear least-squares treatments. The examples include (i) a straight line through the origin, and its transformation into a weighted average; (ii) a declining exponential with a background, with application to first-order kinetics data; (iii) the analysis of vapor pressure data by both unweighted fitting to an exponential form and weighted fitting to a linear logarithmic relationship; (iv) the analysis of overlapped spectral lines as sums of Gaussians, with application to the H/D atomic spectrum; (v) the direct fitting of IR rotation-vibration spectral line positions (HCl); and (vi) a two...
01 Jan 2000
TL;DR: In this article, the authors perform canonical correlation analysis and partial least squares to find a subset of "latent variables" that summarize the linear dependencies between odour and wash/reference responses.
Abstract: The transient response of metal-oxide sensors exposed to mild odours can be oftentimes highly correlated with the behaviour of the array during the preceding wash and reference cycles. Since wash/reference gases are virtually constant overtime, variations in their transient response can be used to estimate the amount of sensor drift present in each experiment. We perform canonical correlation analysis and partial least squares to find a subset of "latent variables" that summarize the linear dependencies between odour and wash/reference responses. Ordinary least squares regression is then used to subtract these "latent variables" from the odour response. Experimental results on an odour database of four cooking spices, collected on a 10-sensor array over a period of three months, show significant improvements in predictive accuracy.
TL;DR: In this paper, generalized least squares methods are proposed to estimate potential mean structure parameters and evaluate whether the given model can be successfully augmented with a mean structure, and a simulation evaluates the performance of some alternative tests.
Abstract: The vast majority of structural equation models contains no mean structure, that is, the population means are estimated at the sample means and then eliminated from modeling consideration. Generalized least squares methods are proposed to estimate potential mean structure parameters and to evaluate whether the given model can be successfully augmented with a mean structure. A simulation evaluates the performance of some alternative tests. A method that takes variability due to the estimation of covariance structure parameters into account in the mean structure estimator, as well as in the weight matrix of the generalized least squares function, performs best. In small samples, the F test and Yuan-Bentler adjusted chi-square test perform best. For example, if there is interest in modeling whether arithmetic skills or vocabulary levels are increasing across time, as one would expect in school, an analysis of means is an essential modeling component.
TL;DR: It is shown that under mild conditions, identifiability of the parameters along with a finite CRB for the case of Gaussian noise is equivalent to the deterministic stability of the NLS problem.
Abstract: A number of problems of interest in signal processing can be reduced to nonlinear parameter estimation problems. The traditional approach to studying the stability of these estimation problems is to demonstrate finiteness of the Cramer-Rao bound (CRB) for a given noise distribution. We review an alternate, deterministic notion of stability for the associated nonlinear least squares (NLS) problem from the realm of nonlinear programming (i.e., that the global minimizer of the least squares problem exists and varies smoothly with the noise). Furthermore, we show that under mild conditions, identifiability of the parameters along with a finite CRB for the case of Gaussian noise is equivalent to the deterministic stability of the NLS problem. Finally, we demonstrate the application of our result, which is general, to the problems of multichannel blind deconvolution and sinusoid retrieval to generate new stability results for these problems with little additional effort.
23 Jul 2000
TL;DR: In this article, separable least squares (SLS) optimization methods are proposed as a means of simultaneously estimating both the linear and nonlinear elements, in an exact least squares framework.
Abstract: The Hammerstein cascade, a zero-memory nonlinearity followed by a linear filter, is often used to model nonlinear biological systems. Using this structure, some high-order nonlinear systems can be represented accurately using relatively few parameters. However, because the model output is not a linear function of its parameters, in general they cannot be estimated in closed form. Currently, an iterative technique, which alternates between estimating the linear element from a cross-correlation, and then fitting a polynomial to the nonlinearity via linear regression, is used to identify these cascades. In this paper, separable least squares (SLS) optimization methods are proposed as a means of simultaneously estimating both the linear and nonlinear elements, in an exact least squares framework. A SLS algorithm for the identification of Hammerstein cascades is developed and used to analyze stretch reflex EMG data from a spinal cord injured patient. Results are compared to those obtained using the traditional, iterative, algorithm.
TL;DR: In this article, the authors examined the reliability of non-linear regression methods for the estimation of the standard errors of the fitting parameters and their reliability of the methods used for estimation of these parameters.
Abstract: Two problems related to non-linear regression, the evaluation of the best set of fitting parameters and the reliability of the methods used for the estimation of the standard errors of these parameters, are examined. It is shown that a non-linear curve fitting routine, like the Microsoft Excel Solver, may give more than one solution for the same data set and a simple Monte Carlo routine is described for the evaluation of the bestfit. For standard errors, the reliability of two procedures based on the conventional curvature matrix method, four Jackknife techniques and the bootstrap method are examined by comparing their results to those obtained from a Monte Carlo simulation of the experimental data. It is shown that a fitting parameter may follow a nonnormal distribution when the equation to be fitted is complicated, even if the errors on the data are normally distributed. In this case only Monte Carlo methods of data simulation can give accurate information about the standard errors and the confidence intervals of these parameters.
TL;DR: This paper aims to find the best L"2-approximation over a d-simplex from polynomials of degree m given a polynomial p in d variables and of degree n.
TL;DR: The Curve Fitting with Least Squares (CFLS) curve fitting with least squares as discussed by the authors is a popular curve fitting technique in analytical chemistry. Critical Reviews in Analytical Chemistry: Vol. 30, No. 1, pp. 59-74.
Abstract: (2000). Curve Fitting with Least Squares. Critical Reviews in Analytical Chemistry: Vol. 30, No. 1, pp. 59-74.