Showing papers on "Non-linear least squares published in 2019"
TL;DR: This paper proposes to determine the regularization parameter using the weighted generalized cross-validation method at every iteration of ill-conditioned SNLLS problems based on the variable projection method to produce a consistent demand of decreasing at successive iterations.
Abstract: Separable nonlinear least-squares (SNLLS) problems arise frequently in many research fields, such as system identification and machine learning. The variable projection (VP) method is a very powerful tool for solving such problems. In this paper, we consider the regularization of ill-conditioned SNLLS problems based on the VP method. Selecting an appropriate regularization parameter is difficult because of the nonlinear optimization procedure. We propose to determine the regularization parameter using the weighted generalized cross-validation method at every iteration. This makes the original objective function changing during the optimization procedure. To circumvent this problem, we use an inequation to produce a consistent demand of decreasing at successive iterations. The approximation of the Jacobian of the regularized problem is also discussed. The proposed regularized VP algorithm is tested by the parameter estimation problem of several statistical models. Numerical results demonstrate the effectiveness of the proposed algorithm.
144 citations
TL;DR: In this paper, it is proposed to use OLS estimation of the MIDAS regression slope and intercept parameters combined with profiling the polynomial weighting scheme parameter(s) using the use of Beta polynomials.
54 citations
TL;DR: This procedure is shown to be the only one satisfying two properties, correctness in the consistent case, and invariance to a specific transformation on a triad, that is, the weight vector is not influenced by an arbitrary multiplication of matrix elements along a 3-cycle by a positive scalar.
49 citations
TL;DR: Two estimation problems for pipeline systems in which measurements of the compressible gas flowing through a network of pipes are affected by time-varying injections, withdrawals, and compression are formulated and a rapid, scalable computational method for performing a nonlinear least squares estimation is developed.
Abstract: We formulate two estimation problems for pipeline systems in which measurements of the compressible gas flowing through a network of pipes are affected by time-varying injections, withdrawals, and compression. We consider a state estimation problem that is then extended to a joint state and parameter estimation problem that can be used for data assimilation. In both formulations, the flow dynamics are described on each pipe by space- and time-dependent densities and mass flux which evolve according to a system of coupled partial differential equations, in which momentum dissipation is modeled using the Darcy–Wiesbach friction approximation. These dynamics are first spatially discretized to obtain a system of nonlinear ordinary differential equations on which state and parameter estimation formulations are given as nonlinear least squares problems. A rapid, scalable computational method for performing a nonlinear least squares estimation is developed. Extensive simulations and computational experiments on multiple pipeline test networks demonstrate the effectiveness of the formulations in obtaining state and parameter estimates in the presence of measurement and process noise.
44 citations
TL;DR: A discrete-time distributed algorithm developed by Euler’s method, converging exponentially to the least squares solution at the node states with suitable step size and graph conditions is developed.
33 citations
TL;DR: The trained CNN model is able to yield PK parameters which can better discriminate different brain tissues, including stroke regions, and generalizes well to new cases even if a subject specific arterial input function (AIF) is not available for the new data.
Abstract: Background and Purpose: The T1-weighted dynamic contrast enhanced (DCE)-MRI is an imaging technique that provides a quantitative measure of pharmacokinetic (PK) parameters characterizing microvasculature of tissues. For the present study, we propose a new machine learning (ML) based approach to directly estimate the PK parameters from the acquired DCE-MRI image-time series that is both more robust and faster than conventional model fitting. Materials and Methods: We specifically utilize deep convolutional neural networks (CNNs) to learn the mapping between the image-time series and corresponding PK parameters. DCE-MRI datasets acquired from 15 patients with clinically evident mild ischaemic stroke were used in the experiments. Training and testing were carried out based on leave-one-patient-out cross- validation. The parameter estimates obtained by the proposed CNN model were compared against the two tracer kinetic models: (1) Patlak model, (2) Extended Tofts model, where the estimation of model parameters is done via voxelwise linear and nonlinear least squares fitting respectively. Results: The trained CNN model is able to yield PK parameters which can better discriminate different brain tissues, including stroke regions. The results also demonstrate that the model generalizes well to new cases even if a subject specific arterial input function (AIF) is not available for the new data. Conclusion: A ML-based model can be used for direct inference of the PK parameters from DCE image series. This method may allow fast and robust parameter inference in population DCE studies. Parameter inference on a 3D volume-time series takes only a few seconds on a GPU machine, which is significantly faster compared to conventional non-linear least squares fitting.
32 citations
TL;DR: A fundamentally new representation of 3D cross fields based on Cartan's method of moving frames is introduced, finding that cross fields and ordinary frame fields are locally characterized by identical conditions on their Darboux derivative, and applies this representation to compute 3DCross fields that are as smooth as possible everywhere but on a prescribed network of singular curves.
Abstract: A basic challenge in field-guided hexahedral meshing is to find a spatially-varying frame that is adapted to the domain geometry and is continuous up to symmetries of the cube. We introduce a fundamentally new representation of such 3D cross fields based on Cartan's method of moving frames. Our key observation is that cross fields and ordinary frame fields are locally characterized by identical conditions on their Darboux derivative. Hence, by using derivatives as the principal representation (and only later recovering the field itself), one avoids the need to explicitly account for symmetry during optimization. At the discrete level, derivatives are encoded by skew-symmetric matrices associated with the edges of a tetrahedral mesh; these matrices encode arbitrarily large rotations along each edge, and can robustly capture singular behavior even on coarse meshes. We apply this representation to compute 3D cross fields that are as smooth as possible everywhere but on a prescribed network of singular curves---since these fields are adapted to curve tangents, they can be directly used as input for field-guided mesh generation algorithms. Optimization amounts to an easy nonlinear least squares problem that behaves like a convex program in the sense that it always appears to produce the same result, independent of initialization. We study the numerical behavior of this procedure, and perform some preliminary experiments with mesh generation.
28 citations
TL;DR: A light, flexible, and yet reliable multiparametric estimation algorithm for grid-connected voltage-source-converter-based systems that combines the nonlinear least squares (NLSQ) fitting algorithm and the steady-state extended harmonic domain (EHD) model to take into account the harmonic effects.
Abstract: This paper proposes a light, flexible, and yet reliable multiparametric estimation algorithm for grid-connected voltage-source-converter-based systems. This approach combines the nonlinear least squares (NLSQ) fitting algorithm and the steady-state extended harmonic domain (EHD) model to take into account the harmonic effects. The EHD model is used to take advantage of the harmonic content of the electrical signals to provide robustness and improved performance to the NLSQ estimation methodology. In addition, this harmonic formulation reduces the necessity of synchronized measurements. Details of the implementation are provided by an experimental case study, in which the grid equivalent (Thevenin voltage, inductance, and resistance), the ac-side filter (inductance and resistance), and the converter switching and conduction loss resistance are accurately estimated for three different operating conditions.
24 citations
TL;DR: Local convergence properties of the Levenberg–Marquardt method are considered, when applied to nonzero-residue nonlinear least-squares problems under an error bound condition, which is weaker than requiring full rank of the Jacobian in a neighborhood of a stationary point.
Abstract: The Levenberg–Marquardt method is widely used for solving nonlinear systems of equations, as well as nonlinear least-squares problems. In this paper, we consider local convergence properties of the method, when applied to nonzero-residue nonlinear least-squares problems under an error bound condition, which is weaker than requiring full rank of the Jacobian in a neighborhood of a stationary point. Differently from the zero-residue case, the choice of the Levenberg–Marquardt parameter is shown to be dictated by (i) the behavior of the rank of the Jacobian and (ii) a combined measure of nonlinearity and residue size in a neighborhood of the set of (possibly non-isolated) stationary points of the sum of squares function.
21 citations
TL;DR: This survey comprises some of the traditional and modern developed methods for nonlinear least-squares problems and suggests a few topics for further research.
Abstract: In this paper, we present a brief survey of methods for solving nonlinear least-squares problems. We pay specific attention to methods that take into account the special structure of the problems. Most of the methods discussed belong to the quasi-Newton family (i.e. the structured quasi-Newton methods (SQN)). Our survey comprises some of the traditional and modern developed methods for nonlinear least-squares problems. At the end, we suggest a few topics for further research.
21 citations
TL;DR: A model-based material decomposition algorithm which performs the reconstruction and decomposition simultaneously using a multienergy forward model is presented and demonstrated to produce accurate concentration estimates in challenging spatial/spectral sampling acquisitions.
Abstract: Spectral information in CT may be used for material decomposition to produce accurate reconstructions of material density and to separate materials with similar overall attenuation. Traditional methods separate the reconstruction and decomposition steps, often resulting in undesirable trade-offs (e.g. sampling constraints, a simplified spectral model). In this work, we present a model-based material decomposition algorithm which performs the reconstruction and decomposition simultaneously using a multienergy forward model. In a kV-switching simulation study, the presented method is capable of reconstructing iodine at 0.5 mg ml-1 with a contrast-to-noise ratio greater than two, as compared to 3.0 mg ml-1 for image domain decomposition. The presented method also enables novel acquisition methods, which was demonstrated in this work with a combined kV-switching/split-filter acquisition explored in simulation and physical test bench studies. This novel design used four spectral channels to decompose three materials: water, iodine, and gadolinium. In simulation, the presented method accurately reconstructed concentration value estimates with RMSE values of 4.86 mg ml-1 for water, 0.108 mg ml-1 for iodine and 0.170 mg ml-1 for gadolinium. In test-bench data, the RMSE values were 134 mg ml-1, 5.26 mg ml-1 and 1.85 mg ml-1, respectively. These studies demonstrate the ability of model-based material decomposition to produce accurate concentration estimates in challenging spatial/spectral sampling acquisitions.
TL;DR: In this article, the Akaike Information Criterion (AIC) is used to compare different weighting schemes as well as different models for hydrogen diffusion through an iron foil in a Devanathan-Stachurski cell.
TL;DR: A new calibration strategy by setting up a simple model structure for river water quality with unknown parameters of RWQM is demonstrated, which can be used as a relatively simple but robust calibration tool to support model application and data analysis.
TL;DR: The research results demonstrate that the MDM can more precisely characterize the rate-dependent hysteresis behaviors comparing with the CDM at high-frequency and high-amplitude excitations.
Abstract: Hysteresis behaviors are inherent characteristics of piezoelectric ceramic actuators. The classical Duhem model (CDM) as a popular hysteresis model has been widely used, but cannot precisely describe rate-dependent hysteresis behaviors at high-frequency and high-amplitude excitations. To describe such behaviors more precisely, this paper presents a modified Duhem model (MDM) by introducing trigonometric functions based on the analysis of the existing experimental data. The MDM parameters are also identified by using the nonlinear least squares method. Six groups of experiments with different frequencies or amplitudes are conducted to evaluate the MDM performance. The research results demonstrate that the MDM can more precisely characterize the rate-dependent hysteresis behaviors comparing with the CDM at high-frequency and high-amplitude excitations.
TL;DR: A modified structured secant relation is proposed to get a more accurate approximation of the second curvature of the least squares objective function and three scaled nonlinear conjugate gradient methods for nonlinear least squares problems are proposed.
Abstract: We propose a modified structured secant relation to get a more accurate approximation of the second curvature of the least squares objective function. Then, using this relation and an approach introduced by Andrei, we propose three scaled nonlinear conjugate gradient methods for nonlinear least squares problems. An attractive feature of one of the proposed methods is satisfication of the sufficient descent condition regardless of the line search and the objective function convexity. We establish that the three proposed algorithms are globally convergent, under the assumption of the Jacobian matrix having full column rank on the level set for one, and without such assumption for the other two. Numerical experiments are made on the collection of test problems, both zero-residual and nonzero-residual, using the Dolan–More performance profiles. They show that the outperformance of our proposed algorithms is more pronounced on nonzero-residual as well as large problems.
TL;DR: It is observed that local minima cease to be an issue as redundant measurements are added, and a bound is derived for the distance between the true solution and the nearest spurious local minimum of the nonconvex least-squares objective.
Abstract: The power system state estimation problem computes the set of complex voltage phasors given quadratic measurements using nonlinear least squares. This is a nonconvex optimization problem, so even in the absence of measurement errors, local search algorithms like Newton/Gauss–Newton can become “stuck” at local minima, which correspond to nonsensical estimations. In this paper, we observe that local minima cease to be an issue as redundant measurements are added. Posing state estimation as an instance of the low-rank matrix recovery problem, we derive a bound for the distance between the true solution and the nearest spurious local minimum. We use the bound to show that spurious local minima of the nonconvex least-squares objective become far-away from the true solution with the addition of redundant information.
TL;DR: An iterative fitting method is introduced, alternating between nonlinear least squares parameter optimization and an FE prestressing algorithm to obtain the correct constrained mixture material state during the mechanical test, which demonstrates a convergence towards constrained mixture compatible parameters, which differ significantly from classically obtained parameters.
Abstract: The constrained mixture theory is an elegant way to incorporate the phenomenon of residual stresses in patient-specific finite element models of arteries. This theory assumes an in vivo reference geometry, obtained from medical imaging, and constituent-specific deposition stretches in the assumed reference state. It allows to model residual stresses and prestretches in arteries without the need for a stress-free reference configuration, most often unknown in patient-specific modeling. A finite element (FE) model requires material parameters, which are classically obtained by fitting the constitutive model to experimental data. The characterization of arterial tissue is often based on planar biaxial test data, to which nonlinear elastic fiber-reinforced material parameters are fitted. However, the introduction of the constrained mixture theory requires an adapted approach to parameter fitting. Therefore, we introduce an iterative fitting method, alternating between nonlinear least squares parameter optimization and an FE prestressing algorithm to obtain the correct constrained mixture material state during the mechanical test. We verify the method based on numerically constructed planar biaxial test data sets, containing ground truth sets of material parameters. The results show that the method converges to the correct parameter sets in just a few iterations. Next, the iterative fitting approach is applied to planar biaxial test data of ovine pulmonary artery tissue. The obtained results demonstrate a convergence towards constrained mixture compatible parameters, which differ significantly from classically obtained parameters. We show that this new modeling approach yields in vivo wall stresses similar to when using classically obtained parameters. However, due to the numerous advantages of constrained mixture modeling, our fitting method is relevant to obtain compatible material parameters, that may not be confused with parameters obtained in a classical way.
TL;DR: In this article, the authors provide an a priori optimizability analysis of nonlinear least squares problems that are solved by local optimization algorithms and define attraction basins where the misfit functional is guaranteed to have only one local-and hence global-stationary point, provided the data error is below some tolerable error level.
Abstract: In this paper, we provide an a priori optimizability analysis of nonlinear least squares problems that are solved by local optimization algorithms. We define attraction (convergence) basins where the misfit functional is guaranteed to have only one local-and hence global-stationary point, provided the data error is below some tolerable error level. We use geometry in the data space (strictly quasiconvex sets) in order to compute the size of the attraction basin (in the parameter space) and the associated tolerable error level (in the data space). These estimates are defined a priori, i.e., they do not involve any least squares minimization problem, and only depend on the forward map. The methodology is applied to the comparison of the optimizability properties of two methods for the seismic inverse problem for a time-harmonic wave equation: the Full Waveform Inversion (FWI) and its Migration Based Travel Time (MBTT) reformulation. Computation of the size of attraction basins for the two approaches allows to quantify the benefits of the latter, which can alleviate the requirement of low-frequency data for the reconstruction of the background velocity model.
TL;DR: Experimental results using the shallow‐water equations demonstrate that the new big data method provides substantial performance improvement over the standard NLS‐4DVar method.
15 Mar 2019
TL;DR: This work considers three approaches for resolving the ill-conditioning in this sequence of linear inverse problems: 1) Laplacian regularization, 2) Bayesian formulation, and 3) resolution-filling gradients.
Abstract: This work presents refraction-corrected sound speed reconstruction techniques for transmission-based ultrasound computed tomography using a circular transducer array. Pulse travel times between element pairs can be calculated from slowness (the reciprocal of sound speed) using the eikonal equation. Slowness reconstruction is posed as a nonlinear least squares problem where the objective is to minimize the error between measured and forward-modeled pulse travel times. The Gauss-Newton method is used to convert this problem into a sequence of linear least-squares problems, each of which can be efficiently solved using conjugate gradients. However, the sparsity of ray-pixel intersection leads to ill-conditioned linear systems and hinders stable convergence of the reconstruction. This work considers three approaches for resolving the ill-conditioning in this sequence of linear inverse problems: 1) Laplacian regularization, 2) Bayesian formulation, and 3) resolution-filling gradients. The goal of this work is to provide an open-source example and implementation of the algorithms used to perform sound speed reconstruction, which is currently being maintained on Github: https://github.com/ rehmanali1994/refractionCorrectedUSCT.github.io
TL;DR: In this paper, the earth pressure balance shield machines are used in underground engineering to prevent ground deformation, even disastrous accidents, and to prevent earth deformation in the soil chamber.
Abstract: Background:Earth pressure balance shield machines are widely used in underground engineering. To prevent ground deformation even disastrous accidents, the earth pressure in soil chamber must be kep...
TL;DR: This work develops and demonstrates an approach for estimating reducible SDEs using standard nonlinear least squares or mixed-effects software, based on extending a known technique that converts maximum likelihood estimation for a Gaussian model with a nonlinear transformation of the dependent variable into an equivalent least-squares problem.
Abstract: Stochastic differential equations (SDEs) are increasingly used in longitudinal data analysis, compartmental models, growth modelling, and other applications in a number of disciplines. Parameter estimation, however, currently requires specialized software packages that can be difficult to use and understand. This work develops and demonstrates an approach for estimating reducible SDEs using standard nonlinear least squares or mixed-effects software. Reducible SDEs are obtained through a change of variables in linear SDEs, and are sufficiently flexible for modelling many situations. The approach is based on extending a known technique that converts maximum likelihood estimation for a Gaussian model with a nonlinear transformation of the dependent variable into an equivalent least-squares problem. A similar idea can be used for Bayesian maximum a posteriori estimation. It is shown how to obtain parameter estimates for reducible SDEs containing both process and observation noise, including hierarchical models with either fixed or random group parameters. Code and examples in R are given. Univariate SDEs are discussed in detail, with extensions to the multivariate case outlined more briefly. The use of well tested and familiar standard software should make SDE modelling more transparent and accessible.
01 Dec 2019
TL;DR: It is shown that bad convergence or divergence at a given local minimum can be a desirable property in the context of estimation problems with symmetric likelihood functions, because it avoids that the algorithm is attracted by statistically undesirable local minima.
Abstract: We analyze the convergence properties of two Newton-type algorithms for the solution of unconstrained nonlinear optimization problems with convex substructure: Generalized Gauss-Newton (GGN) and Sequential Convex Programming (SCP). While both algorithms are identical to the classical Gauss-Newton method for the special case of nonlinear least squares, they differ when applied to more general convex outer functions. We show under mild assumptions that GGN and SCP have locally linear convergence with the same contraction rate. The convergence or divergence rate can be characterized as the smallest scalar that satisfies two linear matrix inequalities. We further show that bad convergence or divergence at a given local minimum can be a desirable property in the context of estimation problems with symmetric likelihood functions, because it avoids that the algorithm is attracted by statistically undesirable local minima. Both algorithms and their convergence properties are illustrated with a numerical example.
10 Jul 2019
TL;DR: The results indicate improved identification with OED compared to LHS and point to the utility of the systematic approach, presented herein, for identifying the parameters of PEM fuel cell models.
Abstract: A methodology for parameterizing polymer electrolyte membrane (PEM) fuel cell models is presented. The procedure starts by optimal experimental design (OED) for parameter identification. This is done by exploring output sensitivities to parameter variations in the space of operating conditions. Once the optimal operating conditions are determined, they are used to gather synthetic experimental data. The synthetic data are then used to identify 7 model parameters in a step-by-step procedure that involves grouping the parameters for identification based on the preceding sensitivity analysis. Starting from the kinetic region of the polarization curve and continuing with the ohmic and mass transport regions, the parameters are identified in a cumulative fashion using a gradient-based nonlinear least squares algorithm. The impact of the OED for parameter identification is explored by comparing the results with another set of synthetic data obtained by Latin Hypercube Sampling (LHS) of the operating space. The results indicate improved identification with OED compared to LHS and point to the utility of the systematic approach, presented herein, for identifying the parameters of PEM fuel cell models.
TL;DR: In this article, two choices of structured spectral gradient methods for solving nonlinear least squares problems are presented, and the scalar multiple of identity approximation of the Hessian inverse is obtained by imposing the structured quasi-Newton condition.
Abstract: In this paper, we present two choices of structured spectral gradient methods for solving nonlinear least squares problems. In the proposed methods, the scalar multiple of identity approximation of the Hessian inverse is obtained by imposing the structured quasi-Newton condition. Moreover, we propose a simple strategy for choosing the structured scalar in the case of negative curvature direction. Using the nonmonotone line search with the quadratic interpolation backtracking technique, we prove that these proposed methods are globally convergent under suitable conditions. Numerical experiment shows that the methods are competitive with some recently developed methods.
TL;DR: In this article, the authors derived the complex valuedness of the DRT using the Hilbert integral transform (HT) and DRT properties for Kelvin-Voigt type RLC networks.
TL;DR: The results illustrate that the intelligent method can improve the accuracy of parametric estimation, and overcome the conventional algorithms' weakness of difficult identification of the nonlinear damping parameter in the roll model.
TL;DR: In this paper, the authors show that if damage is defined by sparse and non-negative vectors, such as the case for local stiffness reductions, then the nonnegative solution to the linearized inverse eigenvalue problem can be made unique with respect to a subset of eigenvalues significantly smaller than the number of potential damage locations.
Posted Content•
TL;DR: In this article, a C++/Python framework for solving non-linear least squares optimization problems represented by factor graphs is presented, which is called miniSAM, which has a full Python/NumPy API, which enables more agile development and easy binding with existing Python projects.
Abstract: Many problems in computer vision and robotics can be phrased as non-linear least squares optimization problems represented by factor graphs, for example, simultaneous localization and mapping (SLAM), structure from motion (SfM), motion planning, and control. We have developed an open-source C++/Python framework miniSAM, for solving such factor graph based least squares problems. Compared to most existing frameworks for least squares solvers, miniSAM has (1) full Python/NumPy API, which enables more agile development and easy binding with existing Python projects, and (2) a wide list of sparse linear solvers, including CUDA enabled sparse linear solvers. Our benchmarking results shows miniSAM offers comparable performances on various types of problems, with more flexible and smoother development experience.
TL;DR: The proposed approach is applied to estimate the position of a walking pedestrian sequentially based on wideband measurement data in an outdoor scenario and shows that the pedestrian can be localized throughout the scenario with an accuracy of 0.8 m at 90% confidence.
Abstract: This paper describes an approach to detect, localize, and track moving, non-cooperative objects by exploiting multipath propagation. In a network of spatially distributed transmitting and receiving nodes, moving objects appear as discrete mobile scatterers. Therefore, the localization of mobile scatterers is formulated as a nonlinear optimization problem. An iterative nonlinear least squares algorithm following Levenberg and Marquardt is used for solving the optimization problem initially, and an extended Kalman filter is used for estimating the scatterer location recursively over time. The corresponding performance bounds are derived for both the snapshot based position estimation and the nonlinear sequential Bayesian estimation with the classic and the posterior Cramer–Rao lower bound. Thereby, a comparison of simulation results to the posterior Cramer–Rao lower bound confirms the applicability of the extended Kalman filter. The proposed approach is applied to estimate the position of a walking pedestrian sequentially based on wideband measurement data in an outdoor scenario. The evaluation shows that the pedestrian can be localized throughout the scenario with an accuracy of 0 . 8 m at 90% confidence.