Showing papers on "Reinforcement learning published in 2022"
TL;DR: In this paper , a novel architecture for tokamak magnetic controller design that autonomously learns to command the full set of control coils is presented. But this approach has unprecedented flexibility and generality in problem specification and yields a notable reduction in design effort to produce new plasma configurations.
Abstract: Abstract Nuclear fusion using magnetic confinement, in particular in the tokamak configuration, is a promising path towards sustainable energy. A core challenge is to shape and maintain a high-temperature plasma within the tokamak vessel. This requires high-dimensional, high-frequency, closed-loop control using magnetic actuator coils, further complicated by the diverse requirements across a wide range of plasma configurations. In this work, we introduce a previously undescribed architecture for tokamak magnetic controller design that autonomously learns to command the full set of control coils. This architecture meets control objectives specified at a high level, at the same time satisfying physical and operational constraints. This approach has unprecedented flexibility and generality in problem specification and yields a notable reduction in design effort to produce new plasma configurations. We successfully produce and control a diverse set of plasma configurations on the Tokamak à Configuration Variable 1,2 , including elongated, conventional shapes, as well as advanced configurations, such as negative triangularity and ‘snowflake’ configurations. Our approach achieves accurate tracking of the location, current and shape for these configurations. We also demonstrate sustained ‘droplets’ on TCV, in which two separate plasmas are maintained simultaneously within the vessel. This represents a notable advance for tokamak feedback control, showing the potential of reinforcement learning to accelerate research in the fusion domain, and is one of the most challenging real-world systems to which reinforcement learning has been applied.
318 citations
TL;DR: The authors provides a taxonomy of automated driving tasks where deep reinforcement learning (DRL) methods have been employed, while addressing key computational challenges in real world deployment of autonomous driving agents and delineates adjacent domains such as behavior cloning, imitation learning, inverse reinforcement learning that are related but are not classical RL algorithms.
Abstract: With the development of deep representation learning, the domain of reinforcement learning (RL) has become a powerful learning framework now capable of learning complex policies in high dimensional environments. This review summarises deep reinforcement learning (DRL) algorithms and provides a taxonomy of automated driving tasks where (D)RL methods have been employed, while addressing key computational challenges in real world deployment of autonomous driving agents. It also delineates adjacent domains such as behavior cloning, imitation learning, inverse reinforcement learning that are related but are not classical RL algorithms. The role of simulators in training agents, methods to validate, test and robustify existing solutions in RL are discussed.
236 citations
TL;DR: Multi-task learning (MTL) as mentioned in this paper is a learning paradigm in machine learning and its aim is to leverage useful information contained in multiple related tasks to help improve the generalization performance of all the tasks.
Abstract: Multi-Task Learning (MTL) is a learning paradigm in machine learning and its aim is to leverage useful information contained in multiple related tasks to help improve the generalization performance of all the tasks. In this paper, we give a survey for MTL from the perspective of algorithmic modeling, applications and theoretical analyses. For algorithmic modeling, we give a definition of MTL and then classify different MTL algorithms into five categories, including feature learning approach, low-rank approach, task clustering approach, task relation learning approach and decomposition approach as well as discussing the characteristics of each approach. In order to improve the performance of learning tasks further, MTL can be combined with other learning paradigms including semi-supervised learning, active learning, unsupervised learning, reinforcement learning, multi-view learning and graphical models. When the number of tasks is large or the data dimensionality is high, we review online, parallel and distributed MTL models as well as dimensionality reduction and feature hashing to reveal their computational and storage advantages. Many real-world applications use MTL to boost their performance and we review representative works in this paper. Finally, we present theoretical analyses and discuss several future directions for MTL.
223 citations
TL;DR: In this article , the authors describe how they trained agents for Gran Turismo that can compete with the world's best e-sports drivers, and demonstrate the possibilities and challenges of using these techniques to control complex dynamical systems in domains where agents must respect imprecisely defined human norms.
Abstract: Many potential applications of artificial intelligence involve making real-time decisions in physical systems while interacting with humans. Automobile racing represents an extreme example of these conditions; drivers must execute complex tactical manoeuvres to pass or block opponents while operating their vehicles at their traction limits1. Racing simulations, such as the PlayStation game Gran Turismo, faithfully reproduce the non-linear control challenges of real race cars while also encapsulating the complex multi-agent interactions. Here we describe how we trained agents for Gran Turismo that can compete with the world's best e-sports drivers. We combine state-of-the-art, model-free, deep reinforcement learning algorithms with mixed-scenario training to learn an integrated control policy that combines exceptional speed with impressive tactics. In addition, we construct a reward function that enables the agent to be competitive while adhering to racing's important, but under-specified, sportsmanship rules. We demonstrate the capabilities of our agent, Gran Turismo Sophy, by winning a head-to-head competition against four of the world's best Gran Turismo drivers. By describing how we trained championship-level racers, we demonstrate the possibilities and challenges of using these techniques to control complex dynamical systems in domains where agents must respect imprecisely defined human norms.
128 citations
TL;DR: In this paper , the authors provide fundamental principles for interpretable ML and dispel common misunderstandings that dilute the importance of this crucial topic, and identify 10 technical challenge areas in interpretable machine learning and provide history and background on each problem.
Abstract: Interpretability in machine learning (ML) is crucial for high stakes decisions and troubleshooting. In this work, we provide fundamental principles for interpretable ML, and dispel common misunderstandings that dilute the importance of this crucial topic. We also identify 10 technical challenge areas in interpretable machine learning and provide history and background on each problem. Some of these problems are classically important, and some are recent problems that have arisen in the last few years. These problems are: (1) Optimizing sparse logical models such as decision trees; (2) Optimization of scoring systems; (3) Placing constraints into generalized additive models to encourage sparsity and better interpretability; (4) Modern case-based reasoning, including neural networks and matching for causal inference; (5) Complete supervised disentanglement of neural networks; (6) Complete or even partial unsupervised disentanglement of neural networks; (7) Dimensionality reduction for data visualization; (8) Machine learning models that can incorporate physics and other generative or causal constraints; (9) Characterization of the “Rashomon set” of good models; and (10) Interpretable reinforcement learning. This survey is suitable as a starting point for statisticians and computer scientists interested in working in interpretable machine learning.
115 citations
TL;DR: A service offloading (SOL) method with deep reinforcement learning, is proposed for DT-empowered IoV in edge computing, which leverages deep Q-network (DQN), which combines the value function approximation of deep learning and reinforcement learning.
Abstract: With the potential of implementing computing-intensive applications, edge computing is combined with digital twinning (DT)-empowered Internet of vehicles (IoV) to enhance intelligent transportation capabilities. By updating digital twins of vehicles and offloading services to edge computing devices (ECDs), the insufficiency in vehicles’ computational resources can be complemented. However, owing to the computational intensity of DT-empowered IoV, ECD would overload under excessive service requests, which deteriorates the quality of service (QoS). To address this problem, in this article, a multiuser offloading system is analyzed, where the QoS is reflected through the response time of services. Then, a service offloading (SOL) method with deep reinforcement learning, is proposed for DT-empowered IoV in edge computing. To obtain optimized offloading decisions, SOL leverages deep Q-network (DQN), which combines the value function approximation of deep learning and reinforcement learning. Eventually, experiments with comparative methods indicate that SOL is effective and adaptable in diverse environments.
107 citations
TL;DR: A coordination graph driven vehicular task offloading scheme, which minimizes offloading costs through efficiently integrating service matching exploitation and intelligent offloading scheduling in both digital twin and physical networks is proposed.
Abstract: Technological advancements of urban informatics and vehicular intelligence have enabled connected smart vehicles as pervasive edge computing platforms for a plethora of powerful applications. However, varies types of smart vehicles with distinct capacities, diverse applications with different resource demands as well as unpredictive vehicular topology, pose significant challenges on realizing efficient edge computing services. To cope with these challenges, we incorporate digital twin technology and artificial intelligence into the design of a vehicular edge computing network. It centrally exploits potential edge service matching through evaluating cooperation gains in a mirrored edge computing system, while distributively scheduling computation task offloading and edge resource allocation in an multiagent deep reinforcement learning approach. We further propose a coordination graph driven vehicular task offloading scheme, which minimizes offloading costs through efficiently integrating service matching exploitation and intelligent offloading scheduling in both digital twin and physical networks. Numerical results based on real urban traffic datasets demonstrate the efficiency of our proposed schemes.
105 citations
TL;DR: The bases, the derivation, and recent progresses of critic intelligence for discrete-time advanced optimal control design are presented with an emphasis on the iterative framework, and the so-called critic intelligence methodology is highlighted, which integrates learning approximators and the reinforcement formulation.
103 citations
TL;DR: The policy iterative algorithm proposed in this paper can solve the coupled algebraic Riccati equations corresponding to the multiplayer non-zero sum games.
95 citations
TL;DR: In this article , an optimal scheduling model for isolated micro-grids by using automated reinforcement learning-based multi-period forecasting of renewable power generation and loads is proposed to reduce the negative impact of the uncertainty of load and renewable energies outputs on microgrid operation.
Abstract: In order to reduce the negative impact of the uncertainty of load and renewable energies outputs on microgrid operation, an optimal scheduling model is proposed for isolated microgrids by using automated reinforcement learning-based multi-period forecasting of renewable power generations and loads. Firstly, a prioritized experience replay automated reinforcement learning (PER-AutoRL) is designed to simplify the deployment of deep reinforcement learning (DRL)-based forecasting model in a customized manner, the single-step multi-period forecasting method based on PER-AutoRL is proposed for the first time to address the error accumulation issue suffered by existing multi-step forecasting methods, then the prediction values obtained by the proposed forecasting method are revised via the error distribution to improve the prediction accuracy; secondly, a scheduling model considering demand response is constructed to minimize the total microgrid operating costs, where the revised forecasting values are used as the dispatch basis, and a spinning reserve chance constraint is set according to the error distribution; finally, by transforming the original scheduling model into a readily solvable mixed integer linear programming via the sequence operation theory (SOT), the transformed model is solved by using CPLEX solver. The simulation results show that compared with the traditional scheduling model without forecasting, this approach manages to significantly reduce the system operating costs by improving the prediction accuracy.
88 citations
TL;DR: In this paper , the latest deep reinforcement learning (RL) based traffic control applications are surveyed, specifically traffic signal control (TSC) applications based on (deep) RL, which have been studied extensively in the literature, are discussed in detail.
Abstract: Latest technological improvements increased the quality of transportation. New data-driven approaches bring out a new research direction for all control-based systems, e.g., in transportation, robotics, IoT and power systems. Combining data-driven applications with transportation systems plays a key role in recent transportation applications. In this paper, the latest deep reinforcement learning (RL) based traffic control applications are surveyed. Specifically, traffic signal control (TSC) applications based on (deep) RL, which have been studied extensively in the literature, are discussed in detail. Different problem formulations, RL parameters, and simulation environments for TSC are discussed comprehensively. In the literature, there are also several autonomous driving applications studied with deep RL models. Our survey extensively summarizes existing works in this field by categorizing them with respect to application types, control models and studied algorithms. In the end, we discuss the challenges and open questions regarding deep RL-based transportation applications.
TL;DR: In this article , a new reinforcement learning (RL)-based control approach that uses the Policy Iteration (PI) and a metaheuristic Grey Wolf Optimizer (GWO) algorithm to train the Neural Networks (NNs) is presented.
TL;DR: In this article , a model-free deep reinforcement learning-based distributed algorithm was proposed to minimize the expected long-term cost of task offloading in mobile edge computing systems. But the offloading decision of each mobile device was left to the edge nodes in a decentralized manner.
Abstract: In mobile edge computing systems, an edge node may have a high load when a large number of mobile devices offload their tasks to it. Those offloaded tasks may experience large processing delay or even be dropped when their deadlines expire. Due to the uncertain load dynamics at the edge nodes, it is challenging for each device to determine its offloading decision (i.e., whether to offload or not, and which edge node it should offload its task to) in a decentralized manner. In this work, we consider non-divisible and delay-sensitive tasks as well as edge load dynamics, and formulate a task offloading problem to minimize the expected long-term cost. We propose a model-free deep reinforcement learning-based distributed algorithm, where each device can determine its offloading decision without knowing the task models and offloading decision of other devices. To improve the estimation of the long-term cost in the algorithm, we incorporate the long short-term memory (LSTM), dueling deep Q-network (DQN), and double-DQN techniques. Simulation results show that our proposed algorithm can better exploit the processing capacities of the edge nodes and significantly reduce the ratio of dropped tasks and average delay when compared with several existing algorithms.
TL;DR: In this article , a reinforcement learning (RL)-based control approach that uses a combination of a deep Q-learning (DQL) algorithm and a metaheuristic Gravitational search algorithm (GSA) is presented.
TL;DR: In this paper, a novel reinforcement learning (RL)-based control approach that uses a combination of a deep Q-learning (DQL) algorithm and a metaheuristic Gravitational Search Algorithm (GSA) is employed to initialize the weights and the biases of the Neural Network (NN) involved in DQL in order to avoid the instability.
TL;DR: In this paper , the authors provide insight into the hierarchical motion planning problem and describe the basics of Deep Reinforcement Learning (DRL) for autonomous driving, including the modeling of the environment, the modeling abstractions, the description of the state and the perception models, the appropriate rewarding, and the realization of the underlying neural network.
Abstract: Academic research in the field of autonomous vehicles has reached high popularity in recent years related to several topics as sensor technologies, V2X communications, safety, security, decision making, control, and even legal and standardization rules. Besides classic control design approaches, Artificial Intelligence and Machine Learning methods are present in almost all of these fields. Another part of research focuses on different layers of Motion Planning, such as strategic decisions, trajectory planning, and control. A wide range of techniques in Machine Learning itself have been developed, and this article describes one of these fields, Deep Reinforcement Learning (DRL). The paper provides insight into the hierarchical motion planning problem and describes the basics of DRL. The main elements of designing such a system are the modeling of the environment, the modeling abstractions, the description of the state and the perception models, the appropriate rewarding, and the realization of the underlying neural network. The paper describes vehicle models, simulation possibilities and computational requirements. Strategic decisions on different layers and the observation models, e.g., continuous and discrete state representations, grid-based, and camera-based solutions are presented. The paper surveys the state-of-art solutions systematized by the different tasks and levels of autonomous driving, such as car-following, lane-keeping, trajectory following, merging, or driving in dense traffic. Finally, open questions and future challenges are discussed.
TL;DR: Simulation results obtained under heterogeneous home environments indicate the advantage of the proposed approach in terms of convergence speed, appliance energy consumption, and number of agents.
Abstract: This article proposesa novel federated reinforcement learning (FRL) approach for the energy management of multiple smart homes with home appliances, a solar photovoltaic system, and an energy storage system. The novelty of the proposed FRL approach lies in the development of a distributed deep reinforcement learning (DRL) model that consists of local home energy management systems (LHEMSs) and a global server (GS). Using energy consumption data, DRL agents for LHEMSs construct and upload their local models to the GS. Then, the GS aggregates the local models to update a global model for LHEMSs and broadcasts it to the DRL agents. Finally, the DRL agents replace the previous local models with the global model and iteratively reconstruct their local models. Simulation results obtained under heterogeneous home environments indicate the advantage of the proposed approach in terms of convergence speed, appliance energy consumption, and number of agents.
TL;DR: In this article , the authors summarized the recent deep learning and reinforcement learning sessions of the International Symposium on Artificial Intelligence and Brain Science (ISBWS) and discussed whether we can achieve comprehensive understanding of human intelligence based on the recent advances of deep learning algorithms.
TL;DR: This paper uses deep reinforcement learning to solve the optimization problem in the path planning and management of unmanned ships, and takes the waiting time at the corner of the path as the optimization goal to minimize the total travel time of unmanned ship passing through the path.
Abstract: Unmanned ship navigates on the water in an autonomous or semiautonomous way, which can be widely used in maritime transportation, intelligence collection, maritime training and testing, reconnaissance, and evidence collection. In this paper, we use deep reinforcement learning to solve the optimization problem in the path planning and management of unmanned ships. Specifically, we take the waiting time (phase and duration) at the corner of the path as the optimization goal to minimize the total travel time of unmanned ships passing through the path. We propose a new reward function, which considers the environment and control delay of unmanned ships at the same time, which can reduce the coordination time between unmanned ships at the same time. In the simulation experiment, through the quantitative and qualitative results of deep reinforcement learning of unmanned ship navigation and path angle waiting, the effectiveness of our solution is verified.
TL;DR: In this article , a multiuser offloading system is analyzed, where the QoS is reflected through the response time of services, and a service offloading (SOL) method with deep reinforcement learning, is proposed for DT-empowered Internet of vehicles in edge computing.
Abstract: With the potential of implementing computing-intensive applications, edge computing is combined with digital twinning (DT)-empowered Internet of vehicles (IoV) to enhance intelligent transportation capabilities. By updating digital twins of vehicles and offloading services to edge computing devices (ECDs), the insufficiency in vehicles’ computational resources can be complemented. However, owing to the computational intensity of DT-empowered IoV, ECD would overload under excessive service requests, which deteriorates the quality of service (QoS). To address this problem, in this article, a multiuser offloading system is analyzed, where the QoS is reflected through the response time of services. Then, a service offloading (SOL) method with deep reinforcement learning, is proposed for DT-empowered IoV in edge computing. To obtain optimized offloading decisions, SOL leverages deep Q-network (DQN), which combines the value function approximation of deep learning and reinforcement learning. Eventually, experiments with comparative methods indicate that SOL is effective and adaptable in diverse environments.
TL;DR: In this paper , a knowledge-based, multiphysics-constrained fast charging strategy for lithium-ion battery (LIB), with a consciousness of the thermal safety and degradation, is proposed, for the first time, to provide a LIB fast charging solution.
Abstract: Fast charging is an enabling technique for the large-scale penetration of electric vehicles. This article proposes a knowledge-based, multiphysics-constrained fast charging strategy for lithium-ion battery (LIB), with a consciousness of the thermal safety and degradation. A universal algorithmic framework combining model-based state observer and a deep reinforcement learning (DRL)-based optimizer is proposed, for the first time, to provide a LIB fast charging solution. Within the DRL framework, a multiobjective optimization problem is formulated by penalizing the over-temperature and degradation. An improved environmental perceptive deep deterministic policy gradient (DDPG) algorithm with priority experience replay is exploited to tradeoff smartly the charging rapidity and the compliance of physical constraints. The proposed DDPG-DRL strategy is compared experimentally with the rule-based strategies and the state-of-the-art model predictive controller to validate its superiority in terms of charging rapidity, enforcement of LIB thermal safety and life extension, as well as the computational tractability.
TL;DR: In this article , an adaptive optimal control approach based on reinforcement learning and adaptive dynamic programming is proposed to learn the optimal regulator with assured convergence rate for disturbed linear continuous-time systems.
TL;DR: In this paper , a vehicular edge computing network is designed to exploit potential edge service matching through evaluating cooperation gains in a mirrored edge computing system, while distributively scheduling computation task offloading and edge resource allocation in an multiagent deep reinforcement learning approach.
Abstract: Technological advancements of urban informatics and vehicular intelligence have enabled connected smart vehicles as pervasive edge computing platforms for a plethora of powerful applications. However, varies types of smart vehicles with distinct capacities, diverse applications with different resource demands as well as unpredictive vehicular topology, pose significant challenges on realizing efficient edge computing services. To cope with these challenges, we incorporate digital twin technology and artificial intelligence into the design of a vehicular edge computing network. It centrally exploits potential edge service matching through evaluating cooperation gains in a mirrored edge computing system, while distributively scheduling computation task offloading and edge resource allocation in an multiagent deep reinforcement learning approach. We further propose a coordination graph driven vehicular task offloading scheme, which minimizes offloading costs through efficiently integrating service matching exploitation and intelligent offloading scheduling in both digital twin and physical networks. Numerical results based on real urban traffic datasets demonstrate the efficiency of our proposed schemes.
TL;DR: Recently, a dizzying number of X-former models have been proposed as mentioned in this paper , which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency.
Abstract: Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision, and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of “X-former” models have been proposed—Reformer, Linformer, Performer, Longformer, to name a few—which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency . With the aim of helping the avid researcher navigate this flurry, this article characterizes a large and thoughtful selection of recent efficiency-flavored “X-former” models, providing an organized and comprehensive overview of existing work and models across multiple domains.
TL;DR: In this paper, the authors discuss the learning ability of buildings with a system-level perspective and present an overview of autonomous machine learning applications that make independent decisions for building energy management, and conclude that the buildings' adaptability to unpredicted changes can be enhanced at the system level through AI-initiated learning processes and by using digital twins as training environments.
TL;DR: The traditional DeepQ-Network is compared with the Deep Q-Network with prioritized experience replay, and the computational efficiency is improved by more than 70 % compared to the DP-based strategy.
TL;DR: This work proposed a real-time dynamic optimal energy management (OEM) based on deep reinforcement learning (DRL) algorithm based on a novel policy-based DRL algorithm with continuous state and action spaces, which includes two phases: offline training and online operation.
01 Dec 2022
TL;DR: In this article , an adaptive reference vector reinforcement learning (RVRL) approach was proposed to decomposition-based algorithms for industrial copper burdening optimization, where the RL operation treated the reference vector adaptation process as an RL task, where each reference vector learns from the environmental feedback and selects optimal actions for gradually fitting the problem characteristics.
Abstract: The performance of decomposition-based algorithms is sensitive to the Pareto front shapes since their reference vectors preset in advance are not always adaptable to various problem characteristics with no a priori knowledge. For this issue, this article proposes an adaptive reference vector reinforcement learning (RVRL) approach to decomposition-based algorithms for industrial copper burdening optimization. The proposed approach involves two main operations, that is: 1) a reinforcement learning (RL) operation and 2) a reference point sampling operation. Given the fact that the states of reference vectors interact with the landscape environment (quite often), the RL operation treats the reference vector adaption process as an RL task, where each reference vector learns from the environmental feedback and selects optimal actions for gradually fitting the problem characteristics. Accordingly, the reference point sampling operation uses estimation-of-distribution learning models to sample new reference points. Finally, the resultant algorithm is applied to handle the proposed industrial copper burdening problem. For this problem, an adaptive penalty function and a soft constraint-based relaxing approach are used to handle complex constraints. Experimental results on both benchmark problems and real-world instances verify the competitiveness and effectiveness of the proposed algorithm.
TL;DR: In this paper, a hybrid simulation and container orchestration framework is proposed to optimize Quality of Service (QoS) parameters in large-scale fog platforms, using a gradient-based optimization strategy using back-propagation of gradients with respect to input.
Abstract: Intelligent task placement and management of tasks in large-scale fog platforms is challenging due to the highly volatile nature of modern workload applications and sensitive user requirements of low energy consumption and response time. Container orchestration platforms have emerged to alleviate this problem with prior art either using heuristics to quickly reach scheduling decisions or AI driven methods like reinforcement learning and evolutionary approaches to adapt to dynamic scenarios. The former often fail to quickly adapt in highly dynamic environments, whereas the latter have run-times that are slow enough to negatively impact response time. Therefore, there is a need for scheduling policies that are both reactive to work efficiently in volatile environments and have low scheduling overheads. To achieve this, we propose a Gradient Based Optimization Strategy using Back-propagation of gradients with respect to Input (GOBI). Further, we leverage the accuracy of predictive digital-twin models and simulation capabilities by developing a Coupled Simulation and Container Orchestration Framework (COSCO). Using this, we create a hybrid simulation driven decision approach, GOBI*, to optimize Quality of Service (QoS) parameters. Co-simulation and the back-propagation approaches allow these methods to adapt quickly in volatile environments. Experiments conducted using real-world data on fog applications using the GOBI and GOBI* methods, show a significant improvement in terms of energy consumption, response time, Service Level Objective and scheduling time by up to 15, 40, 4, and 82 percent respectively when compared to the state-of-the-art algorithms.