Showing papers on "Uncertainty quantification published in 2018"

Machine Learning a General-Purpose Interatomic Potential for Silicon

Abstract: The success of first-principles electronic-structure calculation for predictive modeling in chemistry, solid-state physics, and materials science is constrained by the limitations on simulated length scales and timescales due to the computational cost and its scaling. Techniques based on machine-learning ideas for interpolating the Born-Oppenheimer potential energy surface without explicitly describing electrons have recently shown great promise, but accurately and efficiently fitting the physically relevant space of configurations remains a challenging goal. Here, we present a Gaussian approximation potential for silicon that achieves this milestone, accurately reproducing density-functional-theory reference results for a wide range of observable properties, including crystal, liquid, and amorphous bulk phases, as well as point, line, and plane defects. We demonstrate that this new potential enables calculations such as finite-temperature phase-boundary lines, self-diffusivity in the liquid, formation of the amorphous by slow quench, and dynamic brittle fracture, all of which are very expensive with a first-principles method. We show that the uncertainty quantification inherent to the Gaussian process regression framework gives a qualitative estimate of the potential’s accuracy for a given atomic configuration. The success of this model shows that it is indeed possible to create a useful machine-learning-based interatomic potential that comprehensively describes a material on the atomic scale and serves as a template for the development of such models in the future.

Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detection

Abstract: To assure that an autonomous car is driving safely on public roads, its object detection module should not only work correctly, but show its prediction confidence as well. Previous object detectors driven by deep learning do not explicitly model uncertainties in the neural network. We tackle with this problem by presenting practical methods to capture uncertainties in a 3D vehicle detector for Lidar point clouds. The proposed probabilistic detector represents reliable epistemic uncertainty and aleatoric uncertainty in classification and localization tasks. Experimental results show that the epistemic uncertainty is related to the detection accuracy, whereas the aleatoric uncertainty is influenced by vehicle distance and occlusion. The results also show that we can improve the detection performance by 1%–5% by modeling the aleatoric uncertainty.

Model averaging in ecology: a review of Bayesian, information-theoretic, and tactical approaches for predictive inference

Abstract: In ecology, the true causal structure for a given problem is often not known, and several plausible models and thus model predictions exist. It has been claimed that using weighted averages of these models can reduce prediction error, as well as better reflect model selection uncertainty. These claims, however, are often demonstrated by isolated examples. Analysts must better understand under which conditions model averaging can improve predictions and their uncertainty estimates. Moreover, a large range of different model averaging methods exists, raising the question of how they differ in their behaviour and performance. Here, we review the mathematical foundations of model averaging along with the diversity of approaches available. We explain that the error in model‐averaged predictions depends on each model's predictive bias and variance, as well as the covariance in predictions between models, and uncertainty about model weights. We show that model averaging is particularly useful if the predictive error of contributing model predictions is dominated by variance, and if the covariance between models is low. For noisy data, which predominate in ecology, these conditions will often be met. Many different methods to derive averaging weights exist, from Bayesian over information‐theoretical to cross‐validation optimized and resampling approaches. A general recommendation is difficult, because the performance of methods is often context dependent. Importantly, estimating weights creates some additional uncertainty. As a result, estimated model weights may not always outperform arbitrary fixed weights, such as equal weights for all models. When averaging a set of models with many inadequate models, however, estimating model weights will typically be superior to equal weights. We also investigate the quality of the confidence intervals calculated for model‐averaged predictions, showing that they differ greatly in behaviour and seldom manage to achieve nominal coverage. Our overall recommendations stress the importance of non‐parametric methods such as cross‐validation for a reliable uncertainty quantification of model‐averaged predictions.

Neural Network-Based Uncertainty Quantification: A Survey of Methodologies and Applications

Abstract: Uncertainty quantification plays a critical role in the process of decision making and optimization in many fields of science and engineering. The field has gained an overwhelming attention among researchers in recent years resulting in an arsenal of different methods. Probabilistic forecasting and in particular prediction intervals (PIs) are one of the techniques most widely used in the literature for uncertainty quantification. Researchers have reported studies of uncertainty quantification in critical applications such as medical diagnostics, bioinformatics, renewable energies, and power grids. The purpose of this survey paper is to comprehensively study neural network-based methods for construction of prediction intervals. It will cover how PIs are constructed, optimized, and applied for decision-making in presence of uncertainties. Also, different criteria for unbiased PI evaluation are investigated. The paper also provides some guidelines for further research in the field of neural network-based uncertainty quantification.

Efficient Bayesian computation by proximal Markov chain Monte Carlo: when Langevin meets Moreau

Abstract: Modern imaging methods rely strongly on Bayesian inference techniques to solve challenging imaging problems. Currently, the predominant Bayesian computation approach is convex optimization, which scales very efficiently to high-dimensional image models and delivers accurate point estimation results. However, in order to perform more complex analyses, for example, image uncertainty quantification or model selection, it is necessary to use more computationally intensive Bayesian computation techniques such as Markov chain Monte Carlo methods. This paper presents a new and highly efficient Markov chain Monte Carlo methodology to perform Bayesian computation for high-dimensional models that are log-concave and nonsmooth, a class of models that is central in imaging sciences. The methodology is based on a regularized unadjusted Langevin algorithm that exploits tools from convex analysis, namely, Moreau--Yoshida envelopes and proximal operators, to construct Markov chains with favorable convergence properties. ...

High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach

Abstract: This paper considers the generation of prediction intervals (PIs) by neural networks for quantifying uncertainty in regression tasks. It is axiomatic that high-quality PIs should be as narrow as possible, whilst capturing a specified portion of data. We derive a loss function directly from this axiom that requires no distributional assumption. We show how its form derives from a likelihood principle, that it can be used with gradient descent, and that model uncertainty is accounted for in ensembled form. Benchmark experiments show the method outperforms current state-of-the-art uncertainty quantification methods, reducing average PI width by over 10%.

Bayesian approach to model-based extrapolation of nuclear observables

Abstract: Background: The mass, or binding energy, is the basis property of the atomic nucleus. It determines its stability and reaction and decay rates. Quantifying the nuclear binding is important for understanding the origin of elements in the universe. The astrophysical processes responsible for the nucleosynthesis in stars often take place far from the valley of stability, where experimental masses are not known. In such cases, missing nuclear information must be provided by theoretical predictions using extreme extrapolations. To take full advantage of the information contained in mass model residuals, i.e., deviations between experimental and calculated masses, one can utilize Bayesian machine-learning techniques to improve predictions. Purpose: To improve the quality of model-based predictions of nuclear properties of rare isotopes far from stability, we consider the information contained in the residuals in the regions where the experimental information exist. As a case in point, we discuss two-neutron separation energies S2n of even-even nuclei. Through this observable, we assess the predictive power of global mass models towards more unstable neutron-rich nuclei and provide uncertainty quantification of predictions. Methods: We consider 10 global models based on nuclear density functional theory with realistic energy density functionals as well as two more phenomenological mass models. The emulators of S2n residuals and credibility intervals (Bayesian confidence intervals) defining theoretical error bars are constructed using Bayesian Gaussian processes and Bayesian neural networks. We consider a large training dataset pertaining to nuclei whose masses were measured before 2003. For the testing datasets, we considered those exotic nuclei whose masses have been determined after 2003. By establishing statistical methodology and parameters, we carried out extrapolations toward the 2n dripline. Results: While both Gaussian processes and Bayesian neural networks reduce the root-mean-square (rms) deviation from experiment significantly, GP offers a better and much more stable performance. The increase in the predictive power of microscopic models aided by the statistical treatment is quite astonishing: The resulting rms deviations from experiment on the testing dataset are similar to those of more phenomenological models. We found that Bayesian neural networks results are prone to instabilities caused by the large number of parameters in this method. Moreover, since the classical sigmoid activation function used in this approach has linear tails that do not vanish, it is poorly suited for a bounded extrapolation. The empirical coverage probability curves we obtain match very well the reference values, in a slightly conservative way in most cases, which is highly desirable to ensure honesty of uncertainty quantification. The estimated credibility intervals on predictions make it possible to evaluate predictive power of individual models and also make quantified predictions using groups of models. Conclusions: The proposed robust statistical approach to extrapolation of nuclear model results can be useful for assessing the impact of current and future experiments in the context of model developments. The new Bayesian capability to evaluate residuals is also expected to impact research in the domains where experiments are currently impossible, for instance, in simulations of the astrophysical r process.

Probability-interval hybrid uncertainty analysis for structures with both aleatory and epistemic uncertainties: a review

Abstract: Traditional structural uncertainty analysis is mainly based on probability models and requires the establishment of accurate parametric probability distribution functions using large numbers of experimental samples. In many actual engineering problems, the probability distributions of some parameters can be established due to sufficient samples available, whereas for some parameters, due to the lack or poor quality of samples, only their variation intervals can be obtained, or their probability distribution types can be determined based on the existing data while some of the distribution parameters such as mean and standard deviation can only be given interval estimations. This thus will constitute an important type of probability-interval hybrid uncertain problem, in which the aleatory and epistemic uncertainties both exist. The probability-interval hybrid uncertainty analysis provides an important mean for reliability analysis and design of many complex structures, and has become one of the research focuses in the field of structural uncertainty analysis over the past decades. This paper reviews the four main research directions in this area, i.e., uncertainty modeling, uncertainty propagation analysis, structural reliability analysis, and reliability-based design optimization. It summarizes the main scientific problems, technical difficulties, and current research status of each direction. Based on the review, this paper also provides an outlook for future research in probability-interval hybrid uncertainty analysis.

Deep Directional Statistics: Pose Estimation with Uncertainty Quantification

Abstract: Modern deep learning systems successfully solve many perception tasks such as object pose estimation when the input image is of high quality. However, in challenging imaging conditions such as on low resolution images or when the image is corrupted by imaging artifacts, current systems degrade considerably in accuracy. While a loss in performance is unavoidable, we would like our models to quantify their uncertainty to achieve robustness against images of varying quality. Probabilistic deep learning models combine the expressive power of deep learning with uncertainty quantification. In this paper we propose a novel probabilistic deep learning model for the task of angular regression. Our model uses von Mises distributions to predict a distribution over object pose angle. Whereas a single von Mises distribution is making strong assumptions about the shape of the distribution, we extend the basic model to predict a mixture of von Mises distributions. We show how to learn a mixture model using a finite and infinite number of mixture components. Our model allows for likelihood-based training and efficient inference at test time. We demonstrate on a number of challenging pose estimation datasets that our model produces calibrated probability predictions and competitive or superior point estimates compared to the current state-of-the-art.

Review of Polynomial Chaos-Based Methods for Uncertainty Quantification in Modern Integrated Circuits

Abstract: Advances in manufacturing process technology are key ensembles for the production of integrated circuits in the sub-micrometer region. It is of paramount importance to assess the effects of tolerances in the manufacturing process on the performance of modern integrated circuits. The polynomial chaos expansion has emerged as a suitable alternative to standard Monte Carlo-based methods that are accurate, but computationally cumbersome. This paper provides an overview of the most recent developments and challenges in the application of polynomial chaos-based techniques for uncertainty quantification in integrated circuits, with particular focus on high-dimensional problems.

Quantification of Model Uncertainty in RANS Simulations: A Review

Abstract: In computational fluid dynamics simulations of industrial flows, models based on the Reynolds-averaged Navier--Stokes (RANS) equations are expected to play an important role in decades to come. However, model uncertainties are still a major obstacle for the predictive capability of RANS simulations. This review examines both the parametric and structural uncertainties in turbulence models. We review recent literature on data-free (uncertainty propagation) and data-driven (statistical inference) approaches for quantifying and reducing model uncertainties in RANS simulations. Moreover, the fundamentals of uncertainty propagation and Bayesian inference are introduced in the context of RANS model uncertainty quantification. Finally, the literature on uncertainties in scale-resolving simulations is briefly reviewed with particular emphasis on large eddy simulations.

Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Control

Abstract: A rise in popularity of Deep Neural Networks (DNNs), attributed to more powerful GPUs and widely available datasets, has seen them being increasingly used within safety-critical domains. One such domain, self-driving, has benefited from significant performance improvements, with millions of miles having been driven with no human intervention. Despite this, crashes and erroneous behaviours still occur, in part due to the complexity of verifying the correctness of DNNs and a lack of safety guarantees. In this paper, we demonstrate how quantitative measures of uncertainty can be extracted in real-time, and their quality evaluated in end-to-end controllers for self-driving cars. To this end we utilise a recent method for gathering approximate uncertainty information from DNNs without changing the network's architecture. We propose evaluation techniques for the uncertainty on two separate architectures which use the uncertainty to predict crashes up to five seconds in advance. We find that mutual information, a measure of uncertainty in classification networks, is a promising indicator of forthcoming crashes.

Belief Reliability for Uncertain Random Systems

Abstract: Measuring system reliability by a reasonable metric is a common problem in reliability engineering. Since real systems are usually uncertain random systems affected by both aleatory and epistemic uncertainties, existing reliability metrics are unreliable. This paper proposes a general reliability metric, called belief reliability metric, to cope with the problem. In this paper, the belief reliability is defined as the chance that a system state is within a feasible domain. Mathematically, the metric can degenerate to either probability theory-based reliability, which copes with aleatory uncertainty, or uncertainty theory-based reliability, which considers the effect of epistemic uncertainty. Based on the proposed metric, some commonly used belief reliability indexes, such as belief reliability distribution, mean time to failure, and belief reliable life, are introduced. We also develop system belief reliability formulas for different systems configurations. To further illustrate the formulas, a real case study is performed.

A model-independent iterative ensemble smoother for efficient history-matching and uncertainty quantification in very high dimensions

Abstract: An open-source, scalable and model-independent (non-intrusive) implementation of an iterative ensemble smoother has been developed to alleviate the computational burden associated with history-matching and uncertainty quantification of real-world-scale environmental models that have very high dimensional parameter spaces. The tool, named pestpp-ies, implements the ensemble-smoother form of the popular Gauss-Levenberg-Marquardt algorithm, uses the pest model-interface protocols and includes a built-in parallel run manager, multiple lambda testing and model run failure tolerance. As a demonstration of its capabilities, pestpp-ies is applied to a synthetic groundwater model with thousands of parameters and to a real-world groundwater flow and transport model with tens of thousands of parameters. pestpp-ies is shown to efficiently and effectively condition parameters in both cases and can provide means to estimate posterior forecast uncertainty when the forecasts depend on large numbers of parameters.

Uncertainty in Neural Networks: Bayesian Ensembling.

Abstract: Understanding the uncertainty of a neural network's (NN) predictions is essential for many applications. The Bayesian framework provides a principled approach to this, however applying it to NNs is challenging due to the large number of parameters and data. Ensembling NNs provides a practical and scalable method for uncertainty quantification. Its drawback is that its justification is heuristic rather than Bayesian. In this work we propose one modification to the usual ensembling process, that does result in Bayesian behaviour: regularising parameters about values drawn from a prior distribution. Hence, we present an easily implementable, scalable technique for performing approximate Bayesian inference in NNs.

Particle image velocimetry (PIV) uncertainty quantification using moment of correlation (MC) plane

Abstract: We present a new uncertainty estimation method for particle image velocimetry (PIV), that uses the correlation plane as a model for the probability density function (PDF) of displacements and calculates the second order moment of the correlation (MC). The cross-correlation between particle image patterns is the summation of all particle matches convolved with the apparent particle image diameter. MC uses this property to estimate the PIV uncertainty from the shape of the cross-correlation plane. In this new approach, the generalized cross-correlation (GCC) plane corresponding to a PIV measurement is obtained by removing the particle image diameter contribution. The GCC primary peak represents a discretization of the displacement PDF, from which the standard uncertainty is obtained by convolving the GCC plane with a Gaussian function. Then a Gaussian least-squares-fit is applied to the peak region, accounting for the stretching and rotation of the peak, due to the local velocity gradients and the effect of the convolved Gaussian. The MC method was tested with simulated image sets and the predicted uncertainties show good sensitivity to the error sources and agreement with the expected RMS error. Subsequently, the method was demonstrated in three PIV challenge cases and two experimental datasets and was compared with the published image matching (IM) and correlation statistics (CS) techniques. Results show that the MC method has a better response to spatial variation in RMS error and the predicted uncertainty is in good agreement with the expected standard uncertainty. The uncertainty prediction was also explored as a function of PIV interrogation window size. Overall, the MC method performance establishes itself as a valid uncertainty estimation tool for planar PIV.

Deep convolutional encoder-decoder networks for uncertainty quantification of dynamic multiphase flow in heterogeneous media

Abstract: Surrogate strategies are used widely for uncertainty quantification of groundwater models in order to improve computational efficiency. However, their application to dynamic multiphase flow problems is hindered by the curse of dimensionality, the saturation discontinuity due to capillarity effects, and the time-dependence of the multi-output responses. In this paper, we propose a deep convolutional encoder-decoder neural network methodology to tackle these issues. The surrogate modeling task is transformed to an image-to-image regression strategy. This approach extracts high-level coarse features from the high-dimensional input permeability images using an encoder, and then refines the coarse features to provide the output pressure/saturation images through a decoder. A training strategy combining a regression loss and a segmentation loss is proposed in order to better approximate the discontinuous saturation field. To characterize the high-dimensional time-dependent outputs of the dynamic system, time is treated as an additional input to the network that is trained using pairs of input realizations and of the corresponding system outputs at a limited number of time instances. The proposed method is evaluated using a geological carbon storage process-based multiphase flow model with a 2500-dimensional stochastic permeability field. With a relatively small number of training data, the surrogate model is capable of accurately characterizing the spatio-temporal evolution of the pressure and discontinuous CO2 saturation fields and can be used efficiently to compute the statistics of the system responses.