Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle

https://dc.tsinghuajournals.com/cgi/viewcontent.cgi?article=1565&context=tsinghua-science-and-technology

Deep reinforcement learning based mobile robot navigation: A review

Existing mobile robots cannot complete some functions. To solve these problems, which include autonomous learning in path planning, the slow convergence of path planning, and planned paths that are not smooth, it is possible to utilize neural networks to enable to the robot to perceive the environment and perform feature extraction, which enables them to have a fitness of environment to state action function. By mapping the current state of these actions through Hierarchical Reinforcement Learning (HRL), the needs of mobile robots are met. It is possible to construct a path planning model for mobile robots based on neural networks and HRL. In this article, the proposed algorithm is compared with different algorithms in path planning. It underwent a performance evaluation to obtain an optimal learning algorithm system. The optimal algorithm system was tested in different environments and scenarios to obtain optimal learning conditions, thereby verifying the effectiveness of the proposed algorithm. Deep Deterministic Policy Gradient (DDPG), a path planning algorithm for mobile robots based on neural networks and hierarchical reinforcement learning, performed better in all aspects than other algorithms. Specifically, when compared with Double Deep Q-Learning (DDQN), DDPG has a shorter path planning time and a reduced number of path steps. When introducing an influence value, this algorithm shortens the convergence time by 91% compared with the Q-learning algorithm and improves the smoothness of the planned path by 79%. The algorithm has a good generalization effect in different scenarios. These results have significance for research on guiding, the precise positioning, and path planning of mobile robots.

/pdf/the-path-planning-of-mobile-robot-by-neural-networks-and-kvqwjhbmko.pdf

The Path Planning of Mobile Robot by Neural Networks and Hierarchical Reinforcement Learning.

The teleoperation and coordination of multiple industrial robots play an important role in today’s industrial internet-based collaborative manufacturing systems. The user-friendly teleoperation approach allows operators from different manufacturing domains to reduce redundant learning costs and intuitively control the robot in advance. Nevertheless, only a few preliminary works have been introduced very recently, let alone its effective implementation in the manufacturing scenarios. To address the gap, this research proposes a novel multi-robot collaborative manufacturing system with human-in-the-loop control by leveraging the cutting-edge augmented reality (AR) and digital twin (DT) techniques. In the proposed system, the DTs of industrial robots are firstly mapped to physical robots and visualize them in the AR glasses. Meanwhile, a multi-robot communication mechanism is designed and implemented, to synchronize the state of robots in the twin. Moreover, a reinforcement learning algorithm is integrated into the robot motion planning to replace the conventional kinematics-based robot movement with corresponding target positions. Finally, three interactive AR-assisted DT modes, including real-time motion control, planned motion control, and robot monitoring mode are generated, which can be readily switched by human operators. Two experimental studies are conducted on (1) a single robot with a commonly used peg-in-hole experiment, and (2) the motion planning of multi-robot collaborative tasks, respectively. From the experimental results, it can be found that the proposed system can well handle the multi-robot teleoperation tasks with high efficiency and owns great potentials to be adopted in other complicated manufacturing scenarios in the near future. 

AR-assisted digital twin-enabled robot collaborative manufacturing system with human-in-the-loop

Deep reinforcement learning (DRL) has been proved to be more suitable than reinforcement learning for path planning in large-scale scenarios. In order to more effectively complete the DRL-based collaborative path planning in crowd evacuation, it is necessary to consider the space expansion problem brought by the increase of the number of agents. In addition, it is often faced with complicated circumstances, such as exit selection and congestion in crowd evacuation. However, few existing works have integrated these two aspects jointly. To solve this problem, we propose a planning approach for crowd evacuation based on the improved DRL algorithm, which will improve evacuation efficiency for large-scale crowd path planning. First, we propose a framework of congestion detection-based multi-agent reinforcement learning, the framework divides the crowd into leaders and followers and simulates leaders with a multi-agent system, it considers the congestion detection area is set up to evaluate the degree of congestion at each exit. Next, under the specification of this framework, we propose the improved Multi-Agent Deep Deterministic Policy Gradient (IMADDPG) algorithm, which adds the mean field network to maximize the returns of other agents, enables all agents to maximize the performance of a collaborative planning task in our training period. Then, we implement the hierarchical path planning method, which upper layer is based on the IMADDPG algorithm to solve the global path, and lower layer uses the reciprocal velocity obstacles method to avoid collisions in crowds. Finally, we simulate the proposed method with the crowd simulation system. The experimental results show the effectiveness of our method.

/pdf/improved-multi-agent-deep-deterministic-policy-gradient-for-2z2bouw9gm.pdf

Improved Multi-Agent Deep Deterministic Policy Gradient for Path Planning-Based Crowd Simulation

Dynamic path planning of unknown environment has always been a challenge for mobile robots. In this paper, we apply double Q-network (DDQN) deep reinforcement learning proposed by DeepMind in 2016 to dynamic path planning of unknown environment. The reward and punishment function and the training method are designed for the instability of the training stage and the sparsity of the environment state space. In different training stages, we dynamically adjust the starting position and target position. With the updating of neural network and the increase of greedy rule probability, the local space searched by agent is expanded. Pygame module in PYTHON is used to establish dynamic environments. Considering lidar signal and local target position as the inputs, convolutional neural networks (CNNs) are used to generalize the environmental state. Q-learning algorithm enhances the ability of the dynamic obstacle avoidance and local planning of the agents in environment. The results show that, after training in different dynamic environments and testing in a new environment, the agent is able to reach the local target position successfully in unknown dynamic environment.

/pdf/dynamic-path-planning-of-unknown-environment-based-on-deep-1f548mt8hl.pdf

Peifang Dong

Papers

Dynamic Path Planning of Unknown Environment Based on Deep Reinforcement Learning