Delft University of Technology
Residential demand response of thermostatically controlled loads using batch
Reinforcement Learning
Ruelens, F; Claessens, BJ; Vandael, S; De Schutter, Bart; Babuska, Robert; Belmans, R
DOI
10.1109/TSG.2016.2517211
Publication date
2017
Document Version
Accepted author manuscript
Published in
IEEE Transactions on Smart Grid
Citation (APA)
Ruelens, F., Claessens, BJ., Vandael, S., De Schutter, B., Babuska, R., & Belmans, R. (2017). Residential
demand response of thermostatically controlled loads using batch Reinforcement Learning.
IEEE
Transactions on Smart Grid
,
8
(5), 2149-2159. https://doi.org/10.1109/TSG.2016.2517211
Important note
To cite this publication, please use the final published version (if applicable).
Please check the document version above.
Copyright
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent
of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Takedown policy
Please contact us and provide details if you believe this document breaches copyrights.
We will remove access to the work immediately and investigate your claim.
This work is downloaded from Delft University of Technology.
For technical reasons the number of authors shown on this cover page is limited to a maximum of 10.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
1
Residential Demand Response of
Thermostatically Controlled Loads
Using Batch Reinforcement Learning
Frederik Ruelens, Bert J. Claessens, Stijn Vandael, Bart De Schutter, Senior Member, IEEE,
Robert Babuška, and Ronnie Belmans, Fellow, IEEE
Abstract—Driven by recent advances in batch Reinforcement
Learning (RL), this paper contributes to the application of batch
RL to demand response. In contrast to conventional model-
based approaches, batch RL techniques do not require a system
identification step, making them more suitable for a large-scale
implementation. This paper extends fitted Q-iteration, a stan-
dard batch RL technique, to the situation when a forecast of the
exogenous data is provided. In general, batch RL techniques do
not rely on expert knowledge about the system dynamics or the
solution. However, if some expert knowledge is provided, it can be
incorporated by using the proposed policy adjustment method.
Finally, we tackle the challenge of finding an open-loop sched-
ule required to participate in the day-ahead market. We propose
a model-free Monte Carlo method that uses a metric based on
the state-action value function or Q-function and we illustrate
this method by finding the day-ahead schedule of a heat-pump
thermostat. Our experiments show that batch RL techniques pro-
vide a valuable alternative to model-based controllers and that
they can be used to construct both closed-loop and open-loop
policies.
Index Terms—Batch reinforcement learning, demand response,
electric water heater, fitted Q-iteration, heat pump.
I. INTRODUCTION
T
HE INCREASING share of renewable energy sources
introduces the need for flexibility on the demand side
of the electricity system [1]. A prominent example of loads
that offer flexibility at the residential level are thermostati-
cally controlled loads, such as heat pumps, air conditioning
units, and electric water heaters. These loads represent about
20% of the total electricity consumption at the residential
level in the United States [2]. The market share of these
loads is expected to increase as a result of the electrifica-
tion of heating and cooling [2], making them an interesting
domain for demand response [1], [3]–[5]. Demand response
programs offer demand flexibility by motivating end users to
Manuscript received April 3, 2015; revised July 2, 2015 and September
29, 2015; accepted December 7, 2015. This work was supported by the
Flemish Institute for the Promotion of Scientific and Technological Research
in Industry (IWT). Paper no. TSG-00382-2015.
F. Ruelens, S. Vandael, and R. Belmans are with the Department of
Electrical Engineering, Katholieke Universiteit Leuven/EnergyVille, Leuven
3000, Belgium (e-mail: frederik.ruelens@esat.kuleuven.be).
B. J. Claessens is with the Energy Department, Flemish Institute for
Technological Research, Mol 2400, Belgium.
B. De Schutter and R. Babuška are with the Delft Center for Systems and
Control, Delft University of Technology, Delft 2600 AA, The Netherlands.
Digital Object Identifier 10.1109/TSG.2016.2517211
adapt their consumption profile in response to changing elec-
tricity prices or other grid signals. The forecast uncertainty
of renewable energy sources [6], combined with their limited
controllability, have made demand response the topic of an
extensive number of research projects [1], [7], [8] and scien-
tific papers [3], [5], [9]–[11]. The traditional control paradigm
defines the demand response problem as a model-based con-
trol problem [3], [7], [9], requiring a model of the demand
response application, an optimizer, and a forecasting tech-
nique. A critical step in setting up a model-based controller
includes selecting accurate models and estimating the model
parameters. This step becomes more challenging considering
the heterogeneity of the end users and their different patterns
of behavior. As a result, different end users are expected to
have different model parameters and even different models.
As such, a large-scale implementation of model-based con-
trollers requires a stable and robust approach that is able to
identify the appropriate model and the corresponding model
parameters. A detailed report of the implementation issues
of a model predictive control strategy applied to the heat-
ing system of a building can be found in [12]. Moreover, the
authors of [3] and [13] demonstrate a successful implementa-
tion of a model predictive control approach at an aggregated
level to control a heterogeneous cluster of thermostatically
controlled loads.
In contrast, Reinforcement Learning (RL) [14], [15]is
a model-free technique that requires no system identifica-
tion step and no a priori knowledge. Recent developments
in the field of reinforcement learning show that RL tech-
niques can either replace or supplement model-based tech-
niques [16]. A number of recent papers provide examples of
how a popular RL method, Q-learning [14], can be used for
demand response [4], [10], [17], [18]. For example in [10],
O’Neill et al. propose an automated energy management sys-
tem based on Q-learning that learns how to make optimal
decisions for the consumers. In [17], Henze and Schoenmann
investigate the potential of Q-learning for the operation of
commercial cold stores and in [4], Kara et al. use Q-learning to
control a cluster of thermostatically controlled loads. Similarly,
in [19] Liang et al. propose a Q-learning approach to minimize
the electricity cost of the flexible demand and the disutility of
the user. Furthermore, inspired by [20], Lee and Powell pro-
pose a bias-corrected form of Q-learning to operate battery
charging in the presence of volatile prices [18].
1949-3053
c
2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Accepted Author Manuscript. Link to published article IEEE Transactions on Smart Grid: http://dx.doi.org/10.1109/TSG.2016.2517211
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
2 IEEE TRANSACTIONS ON SMART GRID
Fig. 1. Building blocks of a model-free Reinforcement Learning (RL) agent (gray) applied to a Thermostatically Controlled Load (TCL).
While being a popular method, one of the fundamental
drawbacks of Q-learning is its inefficient use of data, given
that Q-learning discards the current data sample after every
update. As a result, more observations are needed to propagate
already known information through the state space. In order to
overcome this drawback, batch RL techniques [21]–[24] can
be used. In batch RL, a controller estimates a control pol-
icy based on a batch of experiences. These experiences can
be a fixed set [23] or can be gathered online by interacting
with the environment [25]. Given that batch RL algorithms
can reuse past experiences, they converge faster compared
to standard temporal difference methods like Q-learning [26]
or SARSA [27]. This makes batch RL techniques suitable
for practical implementations, such as demand response. For
example, the authors of [28] combine Q-learning with eligibil-
ity traces in order to learn the consumer and time preferences
of demand response applications. In [5], the authors use a
batch RL technique to schedule a cluster of electric water
heaters and in [29], Vandael et al. use a batch RL technique
to find a day-ahead consumption plan of a cluster of electric
vehicles. An excellent overview of batch RL methods can be
found in [25] and [30].
Inspired by the recent developments in batch RL, in particu-
lar fitted Q-iteration by Ernst et al. [16], this paper builds upon
the existing batch RL literature and contributes to the applica-
tion of batch RL techniques to residential demand response.
The contributions of our paper can be summarized as follows:
(1) we demonstrate how fitted Q-iteration can be extended
to the situation when a forecast of the exogenous data is pro-
vided; (2) we propose a policy adjustment method that exploits
general expert knowledge about monotonicity conditions of
the control policy; (3) we introduce a model-free Monte Carlo
method to find a day-ahead consumption plan by making use
of a novel metric based on Q-values.
This paper is structured as follows: Section II defines the
building blocks of our batch RL approach applied to demand
response. Section III formulates the problem as a Markov deci-
sion process. Section IV describes our model-free batch RL
techniques for demand response. Section V demonstrates the
presented techniques in a realistic demand response setting.
To conclude, Section VI summarizes the results and discusses
further research.
II. B
UILDING BLOCKS:MODEL-FREE APPROACH
Fig. 1 presents an overview of our model-free learning
agent applied to a Thermostatically Controlled Load (TCL),
where the gray building blocks correspond to the learning
agent.
At the start of each day the learning agent uses a batch
RL method to construct a control policy for the next day,
given a batch of past interactions with its environment. The
learning agent needs no a priori information on the model
dynamics and considers its environment as a black box.
Nevertheless, if a model of the exogenous variables, e.g., a
forecast of the outside temperature, or reward model is pro-
vided, the batch RL method can use this information to enrich
its batch. Once a policy is found, an expert policy adjust-
ment method can be used to shape the policy obtained with
the batch RL method. During the day, the learning agent
uses an exploration-exploitation strategy to interact with its
environment and to collect new transitions that are added
systematically to the given batch.
In this paper, the proposed learning agent is applied to
two types of TCLs. The first type is a residential electric
water heater with a stochastic hot-water demand. The dynamic
behavior of the electric water heater is modeled using a non-
linear stratified thermal tank model as described in [31]. Our
second TCL is a heat-pump thermostat for a residential build-
ing. The temperature dynamics of the building are modeled
using a second-order equivalent thermal parameter model [32].
This model describes the temperature dynamics of the indoor
air and of the building envelope. However, to develop a realis-
tic implementation, this paper assumes that the learning agent
cannot measure the temperature of the building envelope and
considers it as a hidden state variable.
In addition, we assume that both TCLs are equipped with
a backup controller that guarantees the comfort and safety
settings of its users. The backup controller is a built-in over-
rule mechanism that turns the TCL ‘on’ or ‘off’ depending on
the current state and a predefined switching logic. The oper-
ation and settings of the backup controller are assumed to be
unknown to the learning agent. However, the learning agent
can measure the overrule action of the backup controller (see
the dashed arrow in Fig. 1).
The observable state information contains sensory input data
of the state of the process and its environment. Before this
information is sent to the batch RL algorithm, the learning
agent can apply feature extraction [33]. This feature extrac-
tion step can have two functions namely to extract hidden
state information or to reduce the dimensionality of the state
vector. For example, in the case of a heat-pump thermostat,
this step can be used to extract a feature that represents the
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
RUELENS et al.: RESIDENTIAL DEMAND RESPONSE OF THERMOSTATICALLY CONTROLLED LOADS 3
hidden state information, e.g., the temperature of the building
envelope. Alternatively, a feature extraction mapping can be
used to find a low-dimensional representation of the sensory
input data. For example, in the case of an electrical water
heater, the observed state vector consists of the temperature
sensors that are installed along the hull of the buffer tank.
When the number of temperature sensors is large, it can be
interesting to map this high-dimensional state vector to a low-
dimensional feature vector. This mapping can be the result
of an auto-encoder network or principal component analysis.
For example in [34], Curran et al. indicate that when only a
limited number of observations are available, a mapping to a
low-dimensional state space can improve the convergence of
the learning algorithm.
In this paper, the learning agent is applied to two relevant
demand response business models: dynamic pricing and day-
ahead scheduling [1], [35]. In dynamic pricing, the learning
agent learns a control policy that minimizes its electricity cost
by adapting its consumption profile in response to an external
price signal. The solution of this optimal control problem is
a closed-loop control policy that is a function of the current
measurement of the state. The second business model relates to
the participation in the day-ahead market. The learning agent
constructs a day-ahead consumption plan and then tries to fol-
low it during the day. The objective of the learning agent is
to minimize its cost in the day-ahead market and to mini-
mize any deviation between the day-ahead consumption plan
and the actual consumption. In contrast to the solution of the
dynamic pricing scheme, the day-ahead consumption plan is
a feed-forward plan for the next day, i.e., an open-loop policy,
which does not depend on future measurements of the state.
III. M
ARKOV DECISION PROCESS FORMULATION
This section formulates the decision-making problem of the
learning agent as a Markov decision process. The Markov
decision process is defined by its d-dimensional state space
X ⊂ R
d
, its action space U ⊂ R, its stochastic discrete-time
transition function f, and its cost function ρ. The optimiza-
tion horizon is considered finite, comprising T ∈ N \{0} steps,
where at each discrete time step k, the state evolves as follows:
x
k+1
= f (x
k
, u
k
, w
k
) ∀k ∈{1,...,T − 1}, (1)
with w
k
a realization of a random disturbance drawn from
a conditional probability distribution p
W
(·|x
k
), u
k
∈ U the
control action, and x
k
∈ X the state. Associated with each
state transition, a cost c
k
is given by:
c
k
= ρ(x
k
, u
k
, w
k
) ∀k ∈{1,...,T}. (2)
The goal is to find an optimal control policy h
∗
: X → U that
minimizes the expected T-stage return for any state in the
state space. The expected T-stage return starting from x
1
and
following a policy h is defined as follows:
J
h
T
(x
1
) = E
w
k
∼p
W
(·|x
k
)
T
k=1
ρ(x
k
, h(x
k
), w
k
)
. (3)
A convenient way to characterize the policy h is by using a
state-action value function or Q-function:
Q
h
(x, u) = E
w∼p
W
(·|x)
ρ(x, u, w) + γ J
h
T
( f (x, u, w))
, (4)
where γ ∈ (0, 1) is the discount factor. The Q-function is the
cumulative return starting from state x, taking action u, and
following h thereafter.
The optimal Q-function corresponds the best Q-function that
can be obtained by any policy:
Q
∗
(x, u) = min
h
Q
h
(x, u). (5)
Starting from an optimal Q-function for every state-action pair,
the optimal policy is calculated as follows:
h
∗
(x) ∈ arg min
u∈U
Q
∗
(x, u), (6)
where Q
∗
satisfies the Bellman optimality equation [36]:
Q
∗
(x, u) = E
w∼p
W
(·|x)
ρ(x, u, w) + γ min
u
∈U
Q
∗
f (x, u, w), u
.
(7)
The next three paragraphs give a formal description of the state
space, the backup controller, and the cost function tailored to
demand response.
A. State Description
Following the notational style of [37], the state space
X is spanned by a time-dependent component X
t
, a con-
trollable component X
ph
, and an uncontrollable exogenous
component X
ex
:
X = X
t
× X
ph
× X
ex
. (8)
1) Timing: The time-dependent component X
t
describes the
part of the state space related to timing, i.e., it carries timing
information that is relevant for the dynamics of the system:
X
t
= X
q
t
× X
d
t
with X
q
t
=
{
1, ..., 96
}
, X
d
t
=
{
1, ..., 7
}
, (9)
where x
q
t
∈ X
q
t
denotes the quarter in the day, and x
d
t
∈ X
d
t
denotes the day in the week. The rationale is that most con-
sumer behavior tends to be repetitive and tends to follows a
diurnal pattern.
2) Physical Representation: The controllable component
X
ph
represents the physical state information related to the
quantities that are measured locally and that are influenced by
the control actions, e.g., the indoor air temperature or the state
of charge of an electric water heater:
x
ph
∈ X
ph
with x
ph
< x
ph
< x
ph
, (10)
where x
ph
and x
ph
denote the lower and upper bounds, set to
guarantee the comfort and safety of the end user.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
4 IEEE TRANSACTIONS ON SMART GRID
3) Exogenous Information: The state description of the
uncontrollable exogenous state is split into two components:
X
ex
= X
ph
ex
× X
c
ex
. (11)
When the random disturbance w
k+1
is independent of w
k
there is no need to include exogenous variables in the state
space. However, most physical processes, such as the outside
temperature and solar radiation, exhibit a certain degree of
autocorrelation, where the outcome of the next state depends
on the current state. For this reason we include an exoge-
nous component x
ph
ex
∈ X
ph
ex
in our state space description.
This exogenous component is related to the observable exoge-
nous information that has an impact on the physical dynamics
and that cannot be influenced by the control actions, e.g., the
outside temperature.
The second exogenous component x
c
ex
∈ X
c
ex
has no direct
influence on the dynamics, but contains information to calcu-
late the cost c
k
. This work assumes that a deterministic forecast
of the exogenous state information related to the cost ˆx
c
ex
and
related to the physical dynamics, i.e., outside temperature and
solar radiation, ˆx
ph
ex
is provided for the time span covering the
optimization problem.
B. Backup Controller
This paper assumes that each TCL is equipped with an over-
rule mechanism that guarantees comfort and safety constraints.
The backup function B : X × U −→ U
ph
maps the requested
control action u
k
∈ U taken in state x
k
to a physical control
action u
ph
k
∈ U
ph
:
u
ph
k
= B(x
k
, u
k
). (12)
The settings of the backup function B are unknown to the
learning agent, but the resulting action u
ph
k
can be measured
by the learning agent (see the dashed arrow in Fig. 1).
C. Cost Function
In general, RL techniques do not require a description of the
cost function. However, for most demand response business
models a cost function is available. This paper considers two
typical cost functions related to demand response.
1) Dynamic Pricing: In the dynamic pricing scenario an
external price profile is known deterministically at the start of
the horizon. The cost function is described as:
c
k
= u
ph
k
ˆx
c
k,ex
t, (13)
where ˆx
c
k,ex
is the electricity price at time step k and t is the
length of a control period.
2) Day-Ahead Scheduling: The objective of the second
business case is to determine a day-ahead consumption plan
and to follow this plan during operation. The day-ahead con-
sumption plan should be minimized based on day-ahead prices.
In addition, any deviation between the planned consumption
and actual consumption should be avoided. As such, the cost
function can be written as:
c
k
= u
k
ˆx
c
k,ex
t + α
u
k
t − u
ph
k
t
, (14)
Algorithm 1 Fitted Q-Iteration Using a Forecast of the
Exogenous Data (Extended FQI)
Input: F ={(x
l
, u
l
, x
l
, u
ph
l
)}
#F
l=1
, {(ˆx
ph
l,ex
, ˆx
c
l,ex
)}
#F
l=1
, T
1: let
Q
0
be zero everywhere on X × U
2: for l = 1,...,#F do
3: ˆx
l
← (x
q
l,t
, x
d
l,t
, x
l,ph
, ˆx
ph
l,ex
) replace the observed
exogenous part of the next state x
ph
l,ex
by its forecast ˆx
ph
l,ex
4: end for
5: for N = 1,...,T do
6: for l = 1,...,#F do
7: c
l
← ρ(ˆx
c
l,ex
, u
ph
l
)
8: Q
N,l
← c
l
+ min
u∈U
Q
N−1
(ˆx
l
, u)
9: end for
10: use regression to obtain
Q
N
from
T
reg
=
(x
l
, u
l
), Q
N,l
, l = 1,...,#F
11: end for
Ensure:
Q
∗
=
Q
T
where u
k
is the planned consumption, u
ph
k
is the actual con-
sumption, ˆx
c
k,ex
is the forecasted day-ahead price and α>0is
a penalty. The first part of (14) is the cost for buying energy
at the day-ahead market, whereas the second part penalizes
any deviation between the planned consumption and the actual
consumption.
D. Reinforcement Learning for Demand Response
When the description of the transition function and cost
function is available, techniques that make use of the Markov
decision process framework, such as approximate dynamic
programming [38] or direct policy search [30], can be used to
find near-optimal policies. However, in our implementation we
assume that the transition function f , the backup controller B,
and the underlying probability of the exogenous information w
are unknown. In addition, we assume that they are challenging
to obtain in a residential setting. For these reasons, we present
a model-free batch RL approach that builds on previous the-
oretical work on RL, in particular fitted Q-iteration [23],
expert knowledge [39], and the synthesis of artificial
trajectories [21].
IV. A
LGORITHMS
Typically, batch RL techniques construct policies based on
a batch of tuples of the form: F ={(x
l
, u
l
, x
l
, c
l
)}
#F
l=1
, where
x
l
= (x
q
l,t
, x
d
l,t
, x
l,ph
, x
ph
l,ex
) denotes the state at time step l and x
l
denotes the state at time step l+1. However, for most demand
response applications, the cost function ρ is given a priori, and
of the form ρ(ˆx
c
l,ex
, u
ph
l
). As such, this paper considers tuples
of the form (x
l
, u
l
, x
l
, u
ph
l
).
A. Fitted Q-Iteration Using a Forecast of
the Exogenous Data
Here we demonstrate how fitted Q-iteration [23] can be
extended to the situation when a forecast of the exogenous