Showing papers on "Gaussian process published in 2021"

Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization

Abstract: Compared with global average pooling in existing deep convolutional neural networks (CNNs), global covariance pooling can capture richer statistics of deep features, having potential for improving representation and generalization abilities of deep CNNs. However, integration of global covariance pooling into deep CNNs brings two challenges: (1) robust covariance estimation given deep features of high dimension and small sample size; (2) appropriate usage of geometry of covariances. To address these challenges, we propose a global Matrix Power Normalized COVariance (MPN-COV) Pooling . Our MPN-COV conforms to a robust covariance estimator, very suitable for scenario of high dimension and small sample size. It can also be regarded as Power-Euclidean metric between covariances, effectively exploiting their geometry. Furthermore, a global Gaussian embedding network is proposed to incorporate first-order statistics into MPN-COV. For fast training of MPN-COV networks, we implement an iterative matrix square root normalization, avoiding GPU unfriendly eigen-decomposition inherent in MPN-COV. Additionally, progressive $1\times 1$ 1 × 1 convolutions and group convolution are introduced to compress covariance representations. The proposed methods are highly modular, readily plugged into existing deep CNNs. Extensive experiments are conducted on large-scale object classification, scene categorization, fine-grained visual recognition and texture classification, showing our methods outperform the counterparts and obtain state-of-the-art performance.

Distributed Variational Representation Learning

Abstract: The problem of distributed representation learning is one in which multiple sources of information $X_1,\ldots, X_K$ X 1 , ... , X K are processed separately so as to learn as much information as possible about some ground truth $Y$ Y . We investigate this problem from information-theoretic grounds, through a generalization of Tishby's centralized Information Bottleneck (IB) method to the distributed setting. Specifically, $K$ K encoders, $K \geq 2$ K ≥ 2 , compress their observations $X_1,\ldots, X_K$ X 1 , ... , X K separately in a manner such that, collectively, the produced representations preserve as much information as possible about $Y$ Y . We study both discrete memoryless (DM) and memoryless vector Gaussian data models. For the discrete model, we establish a single-letter characterization of the optimal tradeoff between complexity (or rate) and relevance (or information) for a class of memoryless sources (the observations $X_1,\ldots, X_K$ X 1 , ... , X K being conditionally independent given $Y$ Y ). For the vector Gaussian model, we provide an explicit characterization of the optimal complexity-relevance tradeoff. Furthermore, we develop a variational bound on the complexity-relevance tradeoff which generalizes the evidence lower bound (ELBO) to the distributed setting. We also provide two algorithms that allow to compute this bound: i) a Blahut-Arimoto type iterative algorithm which enables to compute optimal complexity-relevance encoding mappings by iterating over a set of self-consistent equations, and ii) a variational inference type algorithm in which the encoding mappings are parametrized by neural networks and the bound approximated by Markov sampling and optimized with stochastic gradient descent. Numerical results on synthetic and real datasets are provided to support the efficiency of the approaches and algorithms developed in this paper.

30 Years of space–time covariance functions

Abstract: In this article, we provide a comprehensive review of space–time covariance functions. As for the spatial domain, we focus on either the d‐dimensional Euclidean space or on the unit d‐dimensional sphere. We start by providing background information about (spatial) covariance functions and their properties along with different types of covariance functions. While we focus primarily on Gaussian processes, many of the results are independent of the underlying distribution, as the covariance only depends on second‐moment relationships. We discuss properties of space–time covariance functions along with the relevant results associated with spectral representations. Special attention is given to the Gneiting class of covariance functions, which has been especially popular in space–time geostatistical modeling. We then discuss some techniques that are useful for constructing new classes of space–time covariance functions. Separate treatment is reserved for spectral models, as well as to what are termed models with special features. We also discuss the problem of estimation of parametric classes of space–time covariance functions. An outlook concludes the paper.

Macroscopic traffic flow modeling with physics regularized Gaussian process: A new insight into machine learning applications in transportation

Abstract: Despite the wide implementation of machine learning (ML) technique in traffic flow modeling recently, those data-driven approaches often fall short of accuracy in the cases with a small or noisy training dataset. To address this issue, this study presents a new modeling framework, named physics regularized machine learning (PRML), to encode classical traffic flow models (referred as physics models) into the ML architecture and to regularize the ML training process. More specifically, leveraging the Gaussian process (GP) as the base model, a stochastic physics regularized Gaussian process (PRGP) model is developed and a Bayesian inference algorithm is used to estimate the mean and kernel of the PRGP. A physics regularizer, based on macroscopic traffic flow models, is also developed to augment the estimation via a shadow GP and an enhanced latent force model is used to encode physical knowledge into the stochastic process. Based on the posterior regularization inference framework, an efficient stochastic optimization algorithm is then developed to maximize the evidence lowerbound of the system likelihood. For model evaluations, this paper conducts empirical studies on a real-world dataset which is collected from a stretch of I-15 freeway, Utah. Results show the new PRGP model can outperform the previous compatible methods, such as calibrated traffic flow models and pure machine learning methods, in estimation precision and is more robust to the noisy training dataset.

Gaussian processes for autonomous data acquisition at large-scale synchrotron and neutron facilities

Abstract: The execution and analysis of complex experiments are challenged by the vast dimensionality of the underlying parameter spaces. Although an increase in data-acquisition rates should allow broader querying of the parameter space, the complexity of experiments and the subtle dependence of the model function on input parameters remains daunting owing to the sheer number of variables. New strategies for autonomous data acquisition are being developed, with one promising direction being the use of Gaussian process regression (GPR). GPR is a quick, non-parametric and robust approximation and uncertainty quantification method that can be applied directly to autonomous data acquisition. We review GPR-driven autonomous experimentation and illustrate its functionality using real-world examples from large experimental facilities in the USA and France. We introduce the basics of a GPR-driven autonomous loop with a focus on Gaussian processes, and then shift the focus to the infrastructure that needs to be built around GPR to create a closed loop. Finally, the case studies we discuss show that Gaussian-process-based autonomous data acquisition is a widely applicable method that can facilitate the optimal use of instruments and facilities by enabling the efficient acquisition of high-value datasets. Gaussian process regression (GPR) is a powerful, non-parametric and robust technique for uncertainty quantification and function approximation that can be applied to optimal and autonomous data acquisition. This Review introduces the basics of GPR and discusses several use cases from different fields.

Data-driven topology optimization with multiclass microstructures using latent variable Gaussian process

Abstract: The data-driven approach is emerging as a promising method for the topological design of multiscale structures with greater efficiency. However, existing data-driven methods mostly focus on a single class of microstructures without considering multiple classes to accommodate spatially varying desired properties. The key challenge is the lack of an inherent ordering or distance measure between different classes of microstructures in meeting a range of properties. To overcome this hurdle, we extend the newly developed latent-variable Gaussian process (LVGP) models to create multi-response LVGP (MR-LVGP) models for the microstructure libraries of metamaterials, taking both qualitative microstructure concepts and quantitative microstructure design variables as mixed-variable inputs. The MR-LVGP model embeds the mixed variables into a continuous design space based on their collective effects on the responses, providing substantial insights into the interplay between different geometrical classes and material parameters of microstructures. With this model, we can easily obtain a continuous and differentiable transition between different microstructure concepts that can render gradient information for multiscale topology optimization. We demonstrate its benefits through multiscale topology optimization with aperiodic microstructures. Design examples reveal that considering multiclass microstructures can lead to improved performance due to the consistent load-transfer paths for micro- and macro-structures.

Active Learning for Gaussian Process Considering Uncertainties With Application to Shape Control of Composite Fuselage

Abstract: In the machine learning domain, active learning is an iterative data selection algorithm for maximizing information acquisition and improving model performance with limited training samples. It is very useful, especially for industrial applications where training samples are expensive, time-consuming, or difficult to obtain. Existing methods mainly focus on active learning for classification, and a few methods are designed for regression, such as linear regression or Gaussian process (GP). Uncertainties from measurement errors and intrinsic input noise inevitably exist in the experimental data, which further affects the modeling performance. The existing active learning methods do not incorporate these uncertainties for GP. In this article, we propose two new active learning algorithms for the GP with uncertainties, which are variance-based weighted active learning algorithm and D-optimal weighted active learning algorithm. Through numerical study, we show that the proposed approach can incorporate the impact of uncertainties and realize better prediction performance. This approach has been applied to improving the predictive modeling for automatic shape control of composite fuselage. Note to Practitioners —This article was motivated by automatic shape control of composite fuselage. The main objective is to realize active learning for predictive analytics, which means maximizing information acquisition with limited experimental samples. This kind of need for active learning is very common in the industrial systems where it is expensive, time-consuming, or difficult to obtain experimental data. Existing approaches, from either a machine learning perspective or a statistics perspective, mainly focus on active learning for classification or regression models without incorporating impacts from intrinsic input uncertainties. However, intrinsic uncertainties widely exist in industrial systems. This article develops two active learning algorithms for the Gaussian process with uncertainties. The algorithms take variance-based information measure and Fisher information measure into consideration. The proposed algorithms can also be applied in other active learning scenarios, specifically for predictive models with multiple uncertainties.

Neural networks and quantum field theory

Abstract: We propose a theoretical understanding of neural networks in terms of Wilsonian effective field theory. The correspondence relies on the fact that many asymptotic neural networks are drawn from Gaussian processes, the analog of non-interacting field theories. Moving away from the asymptotic limit yields a non-Gaussian process and corresponds to turning on particle interactions, allowing for the computation of correlation functions of neural network outputs with Feynman diagrams. Minimal non-Gaussian process likelihoods are determined by the most relevant non-Gaussian terms, according to the flow in their coefficients induced by the Wilsonian renormalization group. This yields a direct connection between overparameterization and simplicity of neural network likelihoods. Whether the coefficients are constants or functions may be understood in terms of GP limit symmetries, as expected from 't Hooft's technical naturalness. General theoretical calculations are matched to neural network experiments in the simplest class of models allowing the correspondence. Our formalism is valid for any of the many architectures that becomes a GP in an asymptotic limit, a property preserved under certain types of training.

Online learning‐based model predictive control with Gaussian process models and stability guarantees

Abstract: Model predictive control allows to provide high performance and safety guarantees in the form of constraint satisfaction. These properties, however, can be satisfied only if the underlying model, used for prediction, of the controlled process is sufficiently accurate. One way to address this challenge is by data-driven and machine learning approaches, such as Gaussian processes, that allow to refine the model online during operation. We present a combination of an output feedback model predictive control scheme and a Gaussian process-based prediction model that is capable of efficient online learning. To this end, the concept of evolving Gaussian processes is combined with recursive posterior prediction updates. The presented approach guarantees recursive constraint satisfaction and input-to-state stability with respect to the model-plant mismatch. Simulation studies underline that the Gaussian process prediction model can be successfully and efficiently learned online. The resulting computational load is significantly reduced via the combination of the recursive update procedure and by limiting the number of training data points while maintaining good performance.

Comparative evaluation of supervised machine learning algorithms in the prediction of the relative density of 316L stainless steel fabricated by selective laser melting

Abstract: To find a robust combination of selective laser melting (SLM) process parameters to achieve the highest relative density of 3D printed parts, predicting the relative density of 316L stainless steel 3D printed parts was studied using a set of machine learning algorithms. The SLM process brings about the possibility to process metal powders and built complex geometries. However, this technology’s applicability is limited due to the inherent anisotropy of the layered manufacturing process, which generates porosity between adjacent layers, accelerating wear of the built parts when in service. To reduce interlayer porosity, the selection of SLM process parameters has to be properly optimized. The relative density of these manufactured objects is affected by porosity and is a function of process parameters, rendering it a challenging optimization task to solve. In this work, seven supervised machine learning regressors (i.e., support vector machine, decision tree, random forest, gradient boosting, Gaussian process, K-nearest neighbors, multi-layer perceptron) were trained to predict the relative density of 316L stainless steel samples produced by the SLM process. For this purpose, a total of 112 data sets were assembled from a deep literature review, and 5-fold cross-validation was applied to assess the regressor error. The accuracy of the predictions was evaluated by defining an index of merit, i.e., the norm of a vector whose components are the statistical metrics: root mean square error (RMSE), mean absolute error (MAE), and the coefficient of determination (R2). From this index of merit, it is established that the use of gradient boosting regressor shows the highest accuracy, followed by multi-layer perceptron and random forest regressor.

Parallel Gaussian Process Surrogate Bayesian Inference with Noisy Likelihood Evaluations

Abstract: We consider Bayesian inference when only a limited number of noisy log-likelihood evaluations can be obtained. This occurs for example when complex simulator-based statistical models are fitted to data, and synthetic likelihood (SL) method is used to form the noisy log-likelihood estimates using computationally costly forward simulations. We frame the inference task as a sequential Bayesian experimental design problem, where the log-likelihood function is modelled with a hierarchical Gaussian process (GP) surrogate model, which is used to efficiently select additional log-likelihood evaluation locations. Motivated by recent progress in the related problem of batch Bayesian optimisation, we develop various batch-sequential design strategies which allow to run some of the potentially costly simulations in parallel. We analyse the properties of the resulting method theoretically and empirically. Experiments with several toy problems and simulation models suggest that our method is robust, highly parallelisable, and sample-efficient.

Interaction-Aware Probabilistic Trajectory Prediction of Cut-In Vehicles Using Gaussian Process for Proactive Control of Autonomous Vehicles

Abstract: This paper presents a probabilistic trajectory prediction of cut-in vehicles exploiting the information of interacting vehicles. First, a probability distribution of behavioral parameters, which represents the characteristics of lane-change motion, is obtained via Gaussian Process Regression (GPR). For this purpose, Gaussian Process (GP) models are trained using real-world trajectories of lane-changing vehicles and adjacent vehicles. Subsequently, the future states of the lane-change vehicle are probabilistically estimated using a path-following model, which introduces virtual measurements based on the information of behavioral parameters. The proposed predictor is applied to the motion planning and control of autonomous vehicles. A Model Predictive Control (MPC) is designed to achieve predictive maneuvering of autonomous vehicles against cut-in preceding vehicles. The proposed predictor has been evaluated in terms of its prediction accuracy. Also, the performance of the proposed predictor-based control has been validated via computer simulations and autonomous driving vehicle tests. Compared to conventional prediction methods, it is shown that the interaction-aware proposed predictor provides improved prediction of cut-in vehicles’ motion in multi-vehicle scenarios. Furthermore, the control results indicate that the proposed predictor helps the autonomous vehicle to reduce the control effort and improve ride quality for passengers in cut-in scenarios, while guaranteeing safety.

Bayesian optimization using deep Gaussian processes with applications to aerospace system design

Abstract: Bayesian Optimization using Gaussian Processes is a popular approach to deal with optimization involving expensive black-box functions. However, because of the assumption on the stationarity of the covariance function defined in classic Gaussian Processes, this method may not be adapted for non-stationary functions involved in the optimization problem. To overcome this issue, Deep Gaussian Processes can be used as surrogate models instead of classic Gaussian Processes. This modeling technique increases the power of representation to capture the non-stationarity by considering a functional composition of stationary Gaussian Processes, providing a multiple layer structure. This paper investigates the application of Deep Gaussian Processes within Bayesian Optimization context. The specificities of this optimization method are discussed and highlighted with academic test cases. The performance of Bayesian Optimization with Deep Gaussian Processes is assessed on analytical test cases and aerospace design optimization problems and compared to the state-of-the-art stationary and non-stationary Bayesian Optimization approaches.

Optimal Steady-State Voltage Control Using Gaussian Process Learning

Abstract: In this article, an optimal steady-state voltage control framework is developed based on a novel linear voltage–power dependence deducted from Gaussian process (GP) learning. Different from other point-based linearization techniques, this GP-based linear relationship is valid over a subspace of operating points and, thus, suitable for a system with uncertainties such as those in power injections due to renewables. The proposed optimal voltage control algorithms, therefore, perform well over a wide range of operating conditions. Both centralized and distributed optimal control schemes are introduced in this framework. The least-squares estimation is employed to provide analytical forms of the optimal control, which offer great computational benefits. Moreover, unlike many existing voltage control approaches deploying fixed voltage references, the proposed control schemes not only minimize the control efforts but also optimize the voltage reference setpoints that lead to the least voltage deviation errors with respect to such setpoints. The control algorithms are also extended to handle uncertain power injections with robust optimal solutions, which guarantee compliance with the voltage regulation standards. As for the distributed control scheme, a new network partition problem is cast, based on the concept of effective voltage control source (EVCS), as an optimization problem which is further solved using convex relaxation. Various simulations on the IEEE 33-bus and 69-bus test feeders are presented to illustrate the performance of the proposed voltage control algorithms and EVCS-based network partition.

Small sample state of health estimation based on weighted Gaussian process regression

Abstract: -Battery state of health (SOH) estimation is essential for the safety and reliability of electric vehicles. Data-driven approaches are compelling in SOH estimation as they work effectively without human intervention and have excellent nonlinear approximation capabilities. Most studies assume that the training data is sufficient. However, in practical applications, data acquisition is often expensive and time-consuming. A novel weighted Gaussian process regression SOH estimation method is proposed to reduce the model's dependence on data through knowledge transfer. The squared exponential covariance function is introduced with a penalty mechanism to control the cross-battery knowledge transfer process. Experiments are carried out with battery cyclic aging data under different working conditions. Experimental results show that the proposed weighted Gaussian process SOH estimation model can obtain reliable prediction results, although the training data only accounts for 20% of the total dataset. 1

A Fast Kriging-Assisted Evolutionary Algorithm Based on Incremental Learning

Abstract: Kriging models, also known as Gaussian process models, are widely used in surrogate-assisted evolutionary algorithms (SAEAs). However, the cubic time complexity of the standard Kriging models limits their usage in high-dimensional optimization. To tackle this problem, we propose an incremental Kriging model for high-dimensional surrogate-assisted evolutionary computation. The main idea is to update the Kriging model incrementally based on the equations of the previously trained model instead of building the model from scratch when new samples arrive, so that the time complexity of updating the Kriging models can be reduced to quadratic. The proposed incremental learning scheme is very suitable for online SAEAs since they evaluate new samples in each one or several generations. The proposed algorithm is able to achieve competitive optimization results on the test problems compared with the standard Kriging-assisted evolutionary algorithm and is significantly faster than the standard Kriging approach. The proposed algorithm also shows competitive or better performances compared with four fast Kriging-assisted evolutionary algorithms and four state-of-the-art SAEAs. This work provides a fast way of employing Kriging models in high-dimensional surrogate-assisted evolutionary computation.

Towards a model-independent reconstruction approach for late-time Hubble data

Abstract: Gaussian processes offers a convenient way to perform nonparametric reconstructions of observational data assuming only a kernel which describes the covariance between neighbouring points in a data set. We approach the ambiguity in the choice of kernel in Gaussian processes with two methods -- (a) approximate Bayesian computation with sequential Monte Carlo sampling and (b) genetic algorithm -- and use the overall resulting method to reconstruct the cosmic chronometers and supernovae type Ia data sets. The results have shown that the Matern$\left( u = 5/2 \right)$ kernel emerges on top of the two-hyperparameter family of kernels for both cosmological data sets. On the other hand, we use the genetic algorithm in order to select a most naturally-fit kernel among a competitive pool made up of a ten-hyperparameters class of kernels. Imposing a Bayesian information criterion-inspired measure of the fitness, the results have shown that a hybrid of the Radial Basis Function and the Matern$\left( u = 5/2 \right)$ kernel best represented both data sets. The kernel selection problem is not totally closed and may benefit from further analysis using other strategies to resolve an optimal kernel for a particular data set.

Multi-point Gaussian States, Quadratic–Exponential Cost Functionals, and Large Deviations Estimates for Linear Quantum Stochastic Systems

Abstract: This paper is concerned with risk-sensitive performance analysis for linear quantum stochastic systems interacting with external bosonic fields. We consider a cost functional in the form of the exponential moment of the integral of a quadratic polynomial of the system variables over a bounded time interval. Such functionals are related to more conservative behaviour and robustness of systems with respect to statistical uncertainty, which makes the challenging problems of their computation and minimization practically important. To this end, we obtain an integro-differential equation for the time evolution of the quadratic–exponential functional, which is different from the original quantum risk-sensitive performance criterion employed previously for measurement-based quantum control and filtering problems. Using multi-point Gaussian quantum states for the past history of the system variables and their first four moments, we discuss a quartic approximation of the cost functional and its infinite-horizon asymptotic behaviour. The computation of the asymptotic growth rate of this approximation is reduced to solving two algebraic Lyapunov equations. Further approximations of the cost functional, based on higher-order cumulants and their growth rates, are applied to large deviations estimates in the form of upper bounds for tail distributions. We discuss an auxiliary classical Gaussian–Markov diffusion process in a complex Euclidean space which reproduces the quantum system variables at the level of covariances but has different fourth-order cumulants, thus showing that the risk-sensitive criteria are not reducible to quadratic–exponential moments of classical Gaussian processes. The results of the paper are illustrated by a numerical example and may find applications to coherent quantum risk-sensitive control problems, where the plant and controller form a fully quantum closed-loop system, and other settings with nonquadratic cost functionals.