This paper focuses on the active flow control of a computational fluid dynamics simulation over a range of Reynolds numbers using deep reinforcement learning (DRL). More precisely, the proximal policy optimization (PPO) method is used to control the mass flow rate of four synthetic jets symmetrically located on the upper and lower sides of a cylinder immersed in a two-dimensional flow domain. The learning environment supports four flow configurations with Reynolds numbers 100, 200, 300, and 400, respectively. A new smoothing interpolation function is proposed to help the PPO algorithm learn to set continuous actions, which is of great importance to effectively suppress problematic jumps in lift and allow a better convergence for the training process. It is shown that the DRL controller is able to significantly reduce the lift and drag fluctuations and actively reduce the drag by ∼5.7%, 21.6%, 32.7%, and 38.7%, at Re = 100, 200, 300, and 400, respectively. More importantly, it can also effectively reduce drag for any previously unseen value of the Reynolds number between 60 and 400. This highlights the generalization ability of deep neural networks and is an important milestone toward the development of practical applications of DRL to active flow control.

/pdf/robust-active-flow-control-over-a-range-of-reynolds-numbers-34ujncfwa7.pdf

Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning

Reinforcement learning in dual-arm trajectory planning for a free-floating space robot

Deep reinforcement learning (DRL) has recently been adopted in a wide range of physics and engineering domains for its ability to solve decision-making problems that were previously out of reach due to a combination of non-linearity and high dimensionality. In the last few years, it has spread in the field of computational mechanics, and particularly in fluid dynamics, with recent applications in flow control and shape optimization. In this work, we conduct a detailed review of existing DRL applications to fluid mechanics problems. In addition, we present recent results that further illustrate the potential of DRL in Fluid Mechanics. The coupling methods used in each case are covered, detailing their advantages and limitations. Our review also focuses on the comparison with classical methods for optimal control and optimization. Finally, several test cases are described that illustrate recent progress made in this field. The goal of this publication is to provide an understanding of DRL capabilities along with state-of-the-art applications in fluid dynamics to researchers wishing to address new problems with these methods.

A review on Deep Reinforcement Learning for Fluid Mechanics

This paper focuses on the active flow control of a computational fluid dynamics simulation over a range of Reynolds numbers using deep reinforcement learning (DRL). More precisely, the proximal policy optimization (PPO) method is used to control the mass flow rate of four synthetic jets symmetrically located on the upper and lower sides of a cylinder immersed in a two-dimensional flow domain. The learning environment supports four flow configurations with Reynolds numbers 100, 200, 300 and 400, respectively. A new smoothing interpolation function is proposed to help the PPO algorithm to learn to set continuous actions, which is of great importance to effectively suppress problematic jumps in lift and allow a better convergence for the training process. It is shown that the DRL controller is able to significantly reduce the lift and drag fluctuations and to actively reduce the drag by approximately 5.7%, 21.6%, 32.7%, and 38.7%, at $Re$=100, 200, 300, and 400 respectively. More importantly, it can also effectively reduce drag for any previously unseen value of the Reynolds number between 60 and 400. This highlights the generalization ability of deep neural networks and is an important milestone to active flow control.

/pdf/robust-active-flow-control-over-a-range-of-reynolds-numbers-3ijf7ojb1v.pdf

Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning.

In recent years, artificial neural networks (ANNs) and deep learning have become increasingly popular across a wide range of scientific and technical fields, including fluid mechanics. While it will take time to fully grasp the potentialities as well as the limitations of these methods, evidence is starting to accumulate that point to their potential in helping solve problems for which no theoretically optimal solution method is known. This is particularly true in fluid mechanics, where problems involving optimal control and optimal design are involved. Indeed, such problems are famously difficult to solve effectively with traditional methods due to the combination of non linearity, non convexity, and high dimensionality they involve. By contrast, deep reinforcement learning (DRL), a method of optimization based on teaching empirical strategies to an ANN through trial and error, is well adapted to solving such problems. In this short review, we offer an insight into the current state of the art of the use of DRL within fluid mechanics, focusing on control and optimal design problems.

Deep reinforcement learning in fluid mechanics: A promising method for both active flow control and shape optimization

Aerodynamic shape optimization using a novel optimizer based on machine learning techniques

A computational-geometry-based 3-dimensional guidance law to control impact time and angle

Reachability-based cooperative strategy for intercepting a highly maneuvering target using inferior missiles

Dear editor, Aerodynamic design is usually a time-consuming process of four steps [1]. First, an initial design profile is obtained with designer’s domain knowledge. Second, the design profile is represented as a CAD model using softwares such as Catia or UG. Third, a CAE software, such as ICEM or Hypermesh, is applied to generate corresponding meshes. At last, a computational fluid dynamics (CFD) software, such as Fluent or CFX, is used to calculate the performance. Therefore, only numbered of design configurations could be evaluated. The design efficiency and results need to be improved desperately.

Missile aerodynamic design using reinforcement learning and transfer learning

To provide a feasible solution for a variable speed unmanned aerial vehicle (UAV) to home on a target with impact time and angle constraints, this paper presents a novel geometry-based guidance law composed of trajectory reshaping and tracking. A trajectory generation process using Bezier curves is introduced to satisfy the impact time and angle constraints under time-varying speed. The impact angle is satisfied by driving the UAV along a specified ending line. The impact time is satisfied by controlling the trajectory length, which is realized through adjusting one Bezier curve end point along the ending line. The adjustable range of this end point, along with the maximum trajectory curvature, is analyzed to ensure that the trajectory is flyable. Guidance command is generated using inverse dynamics. Numerical simulations under various scenarios are demonstrated to illustrate the performance and validate the effectiveness of the proposed method.

https://www.mdpi.com/2227-7390/8/6/1029/pdf

Yan Xinghui

Papers

Aerodynamic shape optimization using a novel optimizer based on machine learning techniques

A computational-geometry-based 3-dimensional guidance law to control impact time and angle

Reachability-based cooperative strategy for intercepting a highly maneuvering target using inferior missiles

Missile aerodynamic design using reinforcement learning and transfer learning

A Geometry-Based Guidance Law to Control Impact Time and Angle under Variable Speeds