scispace - formally typeset
Search or ask a question
Author

Shuqi Qin

Bio: Shuqi Qin is an academic researcher from Nanjing University of Posts and Telecommunications. The author has contributed to research in topics: Energy management & Reinforcement learning. The author has an hindex of 2, co-authored 4 publications receiving 12 citations.

Papers
More filters
Journal ArticleDOI
TL;DR: A comprehensive review of DRL for SBEM from the perspective of system scale is provided and the existing unresolved issues are identified and possible future research directions are pointed out.
Abstract: Global buildings account for about 30% of the total energy consumption and carbon emission, raising severe energy and environmental concerns. Therefore, it is significant and urgent to develop novel smart building energy management (SBEM) technologies for the advance of energy efficient and green buildings. However, it is a nontrivial task due to the following challenges. First, it is generally difficult to develop an explicit building thermal dynamics model that is both accurate and efficient enough for building control. Second, there are many uncertain system parameters (e.g., renewable generation output, outdoor temperature, and the number of occupants). Third, there are many spatially and temporally coupled operational constraints. Fourth, building energy optimization problems can not be solved in real time by traditional methods when they have extremely large solution spaces. Fifthly, traditional building energy management methods have respective applicable premises, which means that they have low versatility when confronted with varying building environments. With the rapid development of Internet of Things technology and computation capability, artificial intelligence technology find its significant competence in control and optimization. As a general artificial intelligence technology, deep reinforcement learning (DRL) is promising to address the above challenges. Notably, the recent years have seen the surge of DRL for SBEM. However, there lacks a systematic overview of different DRL methods for SBEM. To fill the gap, this article provides a comprehensive review of DRL for SBEM from the perspective of system scale. In particular, we identify the existing unresolved issues and point out possible future research directions.

99 citations

Posted Content
12 Aug 2020
TL;DR: This paper presents a comprehensive literature review on DRL for smart building energy management (SBEM) and introduces the fundamentals of DRL and provides the classification of D RL methods used in existing works related to SBEM.
Abstract: Global buildings consumed 30% of total energy and generated 28% of total carbon emission in 2018, which leads to economic and environmental concerns. Therefore, it is of great significance to reduce energy consumption, energy cost and carbon emission of buildings while maintaining user comfort. To this end, several challenges have to be addressed. Firstly, it is very challenging to develop a building thermal dynamics model that is both accurate and efficient enough for building control. Secondly, there are many kinds of uncertainties. Thirdly, there are many spatially and temporally operational constraints. Fourthly, building energy optimization problems may have extremely large solution spaces, which can not be solved in real-time by traditional methods. Fifthly, traditional building energy management methods have respective applicable premises, which means that they have low versatility when confronted with varying building environments. As a general artificial intelligence technology, deep reinforcement learning (DRL) has the potential of addressing the above challenges. Thus, this paper presents a comprehensive literature review on DRL for smart building energy management (SBEM). To be specific, we first introduce the fundamentals of DRL and provide the classification of DRL methods used in existing works related to SBEM. Then, we review the applications of DRL in a single building energy subsystem, multiple energy subsystems of buildings, and building microgrids, respectively. Furthermore, we identify the unsolved issues and point out the possible research directions of applying DRL. Finally, we summarize the lessons learned from this survey.

7 citations

Posted Content
30 Nov 2020
TL;DR: A scalable algorithm is proposed to solve the Markov game based on multi-agent deep reinforcement learning and prioritized experience replay, which can determine the optimal attacking line sequences and a defense strategy to decide the optimal defense line set.
Abstract: With the increase of connectivity in power grid, a cascading failure may be triggered by the failure of a transmission line, which can lead to substantial economic losses and serious negative social impacts. Therefore, it is very important to identify the critical lines under various types of attacks that may initiate a cascading failure and deploy defense resources to protect them. Since coordinated multistage line attacks can lead to larger negative impacts compared with a single-stage attack or a multistage attack without coordination, this paper intends to identify the critical lines under coordinated multistage attacks that may initiate a cascading failure and deploy limited defense resources optimally. To this end, we first formulate a total generation loss maximization problem with the consideration of multiple attackers and multiple stages. Due to the large size of solution space, it is very challenging to solve the formulated problem. To overcome the challenge, we reformulate the problem as a Markov game and design its components, e.g., state, action, and reward. Next, we propose a scalable algorithm to solve the Markov game based on multi-agent deep reinforcement learning and prioritized experience replay, which can determine the optimal attacking line sequences. Then, we design a defense strategy to decide the optimal defense line set. Extensive simulation results show the effectiveness of the proposed algorithm and the designed defense strategy.
DOI
25 Nov 2022
TL;DR: In this paper , an online algorithm based on prioritized multi-agent-attention-actor-critic (PMA3C) was designed to identify critical lines instantly under load uncertainty without searching the whole or partial solution space.
Abstract: The outages of transmission lines can trigger cascading failure in power grid, resulting in serious negative impacts. Identifying those critical lines can help to take precautionary measures and build robust power system. A group of existing works have developed approaches to identify those critical lines. However, the approaches haven’t consider the load uncertainty, which will lead to the changes of critical lines. In this paper, we investigate a critical line identification problem in smart grid considering load uncertainty. Specifically, we first formulate an optimal virtual attacking problem and its objective is to maximize the expected generation loss under the given attacking resources. Due to the existence of large solution space, unknown system dynamics model, and load uncertainty, solving the formulated problem is challenging. Thus, we design an online algorithm based on prioritized multi-agent-attention-actor-critic (PMA3C). Compared with optimization-based methods and evolutionary methods, the designed algorithm can identify critical lines instantly under load uncertainty without searching the whole or partial solution space. Simulation results indicate that the designed algorithm can identify critical lines efficiently.
Posted Content
22 Sep 2021
TL;DR: In this article, the authors investigated an optimal operation problem of a hydrogen-based multi-energy system with the consideration of building thermal dynamics and proposed an energy management algorithm to solve it based on multi-agent discrete actor-critic with rules (MADACR).
Abstract: Since hydrogen has many advantages (e.g., free pollution, extensive sources, convenient storage and transportation), hydrogen-based multi-energy systems (HMESs) have received wide attention. However, existing works on the optimal operation of HMESs neglect building thermal dynamics, which means that the flexibility of building thermal loads can not be utilized for reducing system operation cost. In this paper, we investigate an optimal operation problem of an HMES with the consideration of building thermal dynamics. Specifically, we first formulate an expected operational cost minimization problem related to an HMES. Due to the existence of uncertain parameters, inexplicit building thermal dynamics models, temporally coupled operational constraints related to three kinds of energy storage systems and indoor temperatures, as well as the coupling between electric energy subsystems and thermal energy subsystems, it is challenging to solve the formulated problem. To overcome the challenge, we reformulate the problem as a Markov game and propose an energy management algorithm to solve it based on multi-agent discrete actor-critic with rules (MADACR). Note that the proposed algorithm does not require any prior knowledge of uncertain parameters, parameter prediction, and explicit building thermal dynamics model. Simulation results based on real-world traces show the effectiveness of the proposed algorithm.

Cited by
More filters
Journal ArticleDOI
TL;DR: In this paper , the authors present a summary of carbon peak and carbon neutrality (CPCN) in buildings using a bibliometric approach and propose future research directions, which will enrich the research body of CPCN and overcome current limitations.
Abstract: Due to large energy consumption and carbon emissions (ECCE) in the building sector, there is huge potential for carbon emission reduction, and this will strongly influence peak carbon emissions and carbon neutrality in the future. To get a better sense of the current research situation and future trends and to provide a valuable reference and guidance for subsequent research, this study presents a summary of carbon peak and carbon neutrality (CPCN) in buildings using a bibliometric approach. Three areas are addressed in the review through the analysis of 364 articles published from 1990–2021: (1) Which countries, institutions, and individuals have conducted extensive and in-depth research on CPCN in buildings, and what is the status quo of their collaboration and contributions? (2) What subjects and topics have aroused wide interest and enthusiasm among scholars, and what are their time trajectories? (3) What journals and authors have grabbed the attention of many scholars, and what are the research directions related to them? Moreover, we propose future research directions. Filling these gaps will enrich the research body of CPCN and overcome current limitations by developing more methods and exploring other practical applications.

46 citations

Journal ArticleDOI
TL;DR: In this article, a review article taxonomically dives into the nitty-gritty of the mainstream DL-based PVPF methods while showcasing their strengths and weaknesses, and concludes with some research gaps and hints about future challenges and research directions in driving the further success of DL techniques to power forecasting.
Abstract: Deep learning (DL)-based PV Power Forecasting (PVPF) emerged nowadays as a promising research direction to intelligentize energy systems. With the massive smart meter integration, DL takes advantage of the large-scale and multi-source data representations to achieve a spectacular performance and high PV forecastability potential compared to classical models. This review article taxonomically dives into the nitty-gritty of the mainstream DL-based PVPF methods while showcasing their strengths and weaknesses. Firstly, we draw connections between PVPF and DL approaches and show how this relation might cross-fertilize or extend both directions. Then, fruitful discussions are conducted based on three classes: discriminative learning, generative learning, and deep reinforcement learning. In addition, this review analyzes recent automatic architecture optimization algorithms for DL-based PVPF. Next, the notable DL technologies are thoroughly described. These technologies include federated learning, deep transfer learning, incremental learning, and big data DL. After that, DL methods are taxonomized into deterministic and probabilistic PVPF. Finally, this review concludes with some research gaps and hints about future challenges and research directions in driving the further success of DL techniques to PVPF applications. By compiling this study, we expect to help aspiring stakeholders widen their knowledge of the staggering potential of DL for PVPF.

29 citations

Journal ArticleDOI
TL;DR: In this article, the authors proposed a DRL approach that uses a deep deterministic policy gradient algorithm for integrated control of HVAC and electric battery storage systems in the presence of on-site PV generation.

27 citations

Journal ArticleDOI
TL;DR: In this article , the authors present a review of previous studies that used interpretable machine learning techniques for building energy management to analyze how model interpretability is improved and discuss the future R&D needs for improving the interpretability of black-box models.
Abstract: Machine learning has been widely adopted for improving building energy efficiency and flexibility in the past decade owing to the ever-increasing availability of massive building operational data. However, it is challenging for end-users to understand and trust machine learning models because of their black-box nature. To this end, the interpretability of machine learning models has attracted increasing attention in recent studies because it helps users understand the decisions made by these models. This article reviews previous studies that adopted interpretable machine learning techniques for building energy management to analyze how model interpretability is improved. First, the studies are categorized according to the application stages of interpretable machine learning techniques: ante-hoc and post-hoc approaches. Then, the studies are analyzed in detail according to specific techniques with critical comparisons. Through the review, we find that the broad application of interpretable machine learning in building energy management faces the following significant challenges: (1) different terminologies are used to describe model interpretability which could cause confusion, (2) performance of interpretable ML in different tasks is difficult to compare, and (3) current prevalent techniques such as SHAP and LIME can only provide limited interpretability. Finally, we discuss the future R&D needs for improving the interpretability of black-box models that could be significant to accelerate the application of machine learning for building energy management.

21 citations

Journal ArticleDOI
TL;DR: In this paper , a real-time control algorithm based on attention-based multi-agent deep reinforcement learning is proposed to minimize the total energy consumption while maintaining a comfortable individual thermal environment for each occupant.

21 citations