Physics-informed neural networks (PINNs) have lately received great attention thanks to their flexibility in tackling a wide range of forward and inverse problems involving partial differential equations. However, despite their noticeable empirical success, little is known about how such constrained neural networks behave during their training via gradient descent. More importantly, even less is known about why such models sometimes fail to train at all. In this work, we aim to investigate these questions through the lens of the Neural Tangent Kernel (NTK); a kernel that captures the behavior of fully-connected neural networks in the infinite width limit during training via gradient descent. Specifically, we derive the NTK of PINNs and prove that, under appropriate conditions, it converges to a deterministic kernel that stays constant during training in the infinite-width limit. This allows us to analyze the training dynamics of PINNs through the lens of their limiting NTK and find a remarkable discrepancy in the convergence rate of the different loss components contributing to the total training error. To address this fundamental pathology, we propose a novel gradient descent algorithm that utilizes the eigenvalues of the NTK to adaptively calibrate the convergence rate of the total training error. Finally, we perform a series of numerical experiments to verify the correctness of our theory and the practical effectiveness of the proposed algorithms. The data and code accompanying this manuscript are publicly available at \url{this https URL}.

When and why PINNs fail to train: A neural tangent kernel perspective

NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations

\n Physics-informed neural networks (PINNs) have gained popularity across different engineering fields due to their effectiveness in solving realistic problems with noisy data and often partially missing physics. In PINNs, automatic differentiation is leveraged to evaluate differential operators without discretization errors, and a multitask learning problem is defined in order to simultaneously fit observed data while respecting the underlying governing laws of physics. Here, we present applications of PINNs to various prototype heat transfer problems, targeting in particular realistic conditions not readily tackled with traditional computational methods. To this end, we first consider forced and mixed convection with unknown thermal boundary conditions on the heated surfaces and aim to obtain the temperature and velocity fields everywhere in the domain, including the boundaries, given some sparse temperature measurements. We also consider the prototype Stefan problem for two-phase flow, aiming to infer the moving interface, the velocity and temperature fields everywhere as well as the different conductivities of a solid and a liquid phase, given a few temperature measurements inside the domain. Finally, we present some realistic industrial applications related to power electronics to highlight the practicality of PINNs as well as the effective use of neural networks in solving general heat transfer problems of industrial complexity. Taken together, the results presented herein demonstrate that PINNs not only can solve ill-posed problems, which are beyond the reach of traditional computational methods, but they can also bridge the gap between computational and experimental heat transfer.

Physics-Informed Neural Networks for Heat Transfer Problems

In recent years, the data-driven turbulence model has attracted widespread concern in fluid mechanics. The existing approaches modify or supplement the original turbulence model by machine learning based on the experimental/numerical data, in order to augment the capability of the present turbulence models. Different from the previous researches, this paper directly reconstructs a mapping function between the turbulent eddy viscosity and the mean flow variables by neural networks and completely replaces the original partial differential equation model. On the other hand, compared with the machine learning models for the low Reynolds (Re) number flows based on direct numerical simulation data, high Reynolds number flows around airfoils present the apparent scaling effects and strong anisotropy, which induce large challenges in accuracy and generalization capability for the machine learning algorithm. We mainly concentrate on the high Reynolds number turbulent flows around the airfoils and take the results calculated by the computational fluid dynamics solver with the Spallart-Allmaras (SA) model as training data to construct a high-dimensional data-driven network model based on machine learning. The radial basis function neural network and the auxiliary optimization methods are adopted, and the individual models are built separately for the flow fields of the near-wall region, wake region, and far-field region. The training data in this paper is extracted from only three subsonic flow fields of NACA0012 airfoil. The data-driven turbulence model can be applied to various airfoils and flow states, and the predicted eddy viscosity, lift/drag coefficients, and skin friction distributions are all in good agreement with the results of the original SA model. This research demonstrates the promising prospect of machine learning methods in future studies about turbulence modeling.

Machine learning methods for turbulence modeling in subsonic flows around airfoils

Abstract Physics-Informed Neural Networks (PINN) are neural networks (NNs) that encode model equations, like Partial Differential Equations (PDE), as a component of the neural network itself. PINNs are nowadays used to solve PDEs, fractional equations, integral-differential equations, and stochastic PDEs. This novel methodology has arisen as a multi-task learning framework in which a NN must fit observed data while reducing a PDE residual. This article provides a comprehensive review of the literature on PINNs: while the primary goal of the study was to characterize these networks and their related advantages and disadvantages. The review also attempts to incorporate publications on a broader range of collocation-based physics informed neural networks, which stars form the vanilla PINN, as well as many other variants, such as physics-constrained neural networks (PCNN), variational hp-VPINN, and conservative PINN (CPINN). The study indicates that most research has focused on customizing the PINN through different activation functions, gradient optimization techniques, neural network structures, and loss function structures. Despite the wide range of applications for which PINNs have been used, by demonstrating their ability to be more feasible in some contexts than classical numerical techniques like Finite Element Method (FEM), advancements are still possible, most notably theoretical issues that remain unresolved. 

/pdf/scientific-machine-learning-through-physics-informed-neural-3uhx9t2l.pdf

Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next

A data-driven model is proposed for the prediction of the velocity field around a cylinder by fusion convolutional neural networks (CNNs) using measurements of the pressure field on the cylinder. The model is based on the close relationship between the Reynolds stresses in the wake, the wake formation length, and the base pressure. Numerical simulations of flow around a cylinder at various Reynolds numbers are carried out to establish a dataset capturing the effect of the Reynolds number on various flow properties. The time series of pressure fluctuations on the cylinder is converted into a grid-like spatial-temporal topology to be handled as the input of a CNN. A CNN architecture composed of a fusion of paths with and without a pooling layer is designed. This architecture can capture both accurate spatial-temporal information and the features that are invariant of small translations in the temporal dimension of pressure fluctuations on the cylinder. The CNN is trained using the computational fluid dynami...

Prediction model of velocity field around circular cylinder over various Reynolds numbers by fusion convolutional neural networks based on pressure on the cylinder

Particle image velocimetry (PIV) has been extensively used in wind-tunnel test for flow-field measurement. However, the sampling frequency of traditional PIV is low and physics of flow field in high-frequency range is hard to capture. A data processing approach is proposed to obtain time-resolved flow field around a circular cylinder with PIV measurement data (high spatial precision) and wind speed measurement data using probes (high time resolution at discrete downstream locations). Proper orthogonal decomposition (POD) is used to extract the compact spatial representations of the flow field based on PIV measurement data, and bidirectional recurrent neural networks (RNNs) with gated recurrent units are designed to learn the time sequences of coefficients of the first few POD modes based on both PIV and probe measurement data. We analyze qualitatively the relationship between the velocity time sequence and the spatial distribution of velocity. Based on this qualitative relationship, an RNN with a unique “many-to-one” architecture is designed to learn the coefficients and make use of the intrinsic property that the velocity time sequence contains information on the spatial distribution of velocity. The inputs to the RNN are the sequential velocity-probe measurements with high sampling frequency, and the outputs are the coefficients of the first few POD modes. The proposed approach is validated by using both simulated datasets for two low Reynolds numbers (200 and 500) and experimental dataset for Reynolds number of 2.4 × 104. We also investigate the influence of velocity time length used in the inputs as well as the number and distribution of velocity probes on prediction accuracy. This study provides a feasible approach to get time-resolved flow field with high accuracy while low cost for all Reynolds numbers.

Time-resolved reconstruction of flow field around a circular cylinder by recurrent neural networks based on non-time-resolved particle image velocimetry measurements

Due to the strong nonlinear properties of the entire flutter process [including growth stages, decay stages and steady limit cycle oscillations (LCOs)], the generalization of the self-excited force model for the entire flutter process of bluff bodies is a very critical issue. This paper proposes a self-excited force model based on a novel long short-term memory (LSTM) neural network to simulate the entire flutter process for various leading-edge configurations. To obtain the dataset, the nonlinear flutter with the growth stages, decay stages and steady LCOs for different leading-edge aerodynamic configurations is investigated by a series of two-degrees-of-freedom spring-suspended sectional model tests. Based on the measured flutter responses and self-excited forces in the tests, a data-driven self-excited force model is proposed on the basis of a novel LSTM neural network. Considering that the nonlinear dynamic system is very sensitive to the accuracy of the model parameters, the Newmark-$$\beta $$ method is incorporated into the LSTM networks and forms a closed-loop process to restrain the error amplification effects and improve the model robustness, that is, the response calculated from the output of LSTM (the self-excited force) by the Newmark-$$\beta $$ method is set as the input of LSTM at the next step rather than the measured response. To validate the performance of the proposed model, the flutter critical wind speed, time history of oscillation and self-excited forces, steady-state oscillation amplitude and flutter development process predicted by the model are compared with the test results. The comparison indicates that the predicted results agree well with the test results, meaning that the proposed model has high accuracy, generalization and robustness in describing the nonlinear characteristics of the flutter for various aerodynamic configurations.

A novel long short-term memory neural-network-based self-excited force model of limit cycle oscillations of nonlinear flutter for various aerodynamic configurations

An intelligent data-driven method of modeling nonlinear dynamical systems named as LSTM network with output recurrence (OR-LSTM) is proposed, which can learn the inherent characteristics of the dynamical systems from data and predict the states of the systems given external excitation and initial conditions. For the proposed OR-LSTM architecture, the most important key point is output recurrence connections. In order to evaluate it, a teacher-forcing LSTM network without output recurrence connections (TF-LSTM) is designed for comparison. The differences between OR-LSTM and TF-LSTM from the perspective of gradient propagation and parameter updating are analyzed firstly. To verify the capability of the LSTM networks in dynamical systems modeling, the oscillation of a 42-story reinforced concrete frame-core tube building under severe and extreme seismic excitations are modeled. The results show that the OR-LSTM network performs well and maintains the long-term robustness in earthquake responses prediction, even only known displacements of 9 stories. However, the TF-LSTM network fails. In order to evaluate the extensive practicability of the OR-LSTM network, the famous Van der Pol dynamic system and Lorenz dynamic system with strong nonlinearity are also modeled by the proposed method. The results show that OR-LSTM network is competent in modeling most of nonlinear dynamic systems, while has limited in fully chaotic systems, but the predictable length of time sequence and prediction accuracy are improved. 

Xiaowei Jin

Papers

NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations

Prediction model of velocity field around circular cylinder over various Reynolds numbers by fusion convolutional neural networks based on pressure on the cylinder

Time-resolved reconstruction of flow field around a circular cylinder by recurrent neural networks based on non-time-resolved particle image velocimetry measurements

A novel long short-term memory neural-network-based self-excited force model of limit cycle oscillations of nonlinear flutter for various aerodynamic configurations

Intelligent modeling of nonlinear dynamical systems by machine learning