Abstract: The restructuring of power systems with the introduction of electricity markets and decentralized structures increased the number of participating entities. This is particularly true in generation and retailing which are now provided under competition. Accordingly, it is important to develop models to simulate the behavior of these agents and to optimize their participation in electricity markets. Among them, it is essential to adequately model generation agents namely in countries having a large share of hydro stations. This paper describes an agent-based approach to model the day-ahead electricity market having a particular emphasis on hydro generation. Apart from the characterization of the agents, the paper details the introduction of the Q-Learning algorithm in the model as a way to enhance the performance of generation agents. This paper also presents some preliminary results taking the Portuguese generation system as an example.

Abstract: In the last decades power systems witnessed the implementation of an organizational and operational restructuring that lead to the introduction of competitive mechanisms in some activities of the value chain. This is the case of generation and retailing with the development of wholesale and retail markets. These developments together with a renewed emphasis on the adoption of more sustainable solutions while maintaining adequate security of supply levels contributed to increase the interest of generation companies for models enabling the optimization of the use of generation assets or for models and tools to help them to prepare and test bidding strategies to the day-ahead markets. Having in mind the increased complexity of the operation of power systems, Agent-Based Models, ABM, are been used to complement the traditional optimization and equilibrium models, taking advantage of the interaction between agents acting in a simulation environment. In this scope, this paper describes an ABM model that uses Q-learning to provide knowledge for the agents to behave in an optimal way. This model is designed to mimic the main features of the common electricity market between Portugal and Spain, the MIBEL. Apart from describing the developed model, this paper also includes preliminary results from its application to the MIBEL case.

4 citations

Cites methods from "Simulation of the operation of hydr..."

...In [15] we used a single set of states and actions....

[...]

...As mentioned in Section III we introduced in the model the Q-learning procedure detailed in [15]....

[...]

...In [15], and in order to simplify the problem we used 7 states (s1 to s7) as illustrated in Figure 3 to discretize the sigmoid function already described in Fig....

[...]

...This strategy is combined with the Q-learning approach as outlined in [15]....

[...]

...This algorithm is detailed in [15] and the parameters used in this study are similar to the ones used in [15]....

Abstract: The complexity of modeling electricity markets is increasing, due to for example increased penetration of renewable energy sources, electric vehicles and more active participation from the demand side. An analysis of a system with multiple market participants displaying certain behavior, could be difficult through an optimization problem. Due to several advantages, agent based approach can be followed to model the behavior of different market participants in the electricity markets. Lately, considerable amount of research work has been carried out on developing Agent-Based Models (ABMs) for electricity markets. This paper is aimed at providing a review of the literature on different applications of ABM in electricity markets. It is shown that ABMs have been applied to a wide range of studies of different parts of electricity trading. Some specific electricity markets modeled with ABMs have also been mentioned. According to the literature survey, the research gap is highlighted and future scope of work in this area is discussed.

TL;DR: An Agent-Based Model developed to simulate the Iberian Electricity Market is presented, with special focus on the modelling of hydro power plants, designed to simulate in a detailed way the hydro units that have a large impact in the electricity market common to Portugal and Spain.

Abstract: This paper presents the results of an Agent-Based Model developed to simulate the Iberian Electricity Market, with special focus on the modelling of hydro power plants. To simulate the agent’s dynamics in the day-ahead market, it was developed a bidding strategy based on a Q-Learning procedure. In the computation area, the recent years brought the discussion around artificial intelligence to a new upper level to complement traditional models, driven by the increased hardware computer capabilities, as well as new developments in the machine learning area. Reinforcement Learning models, as Q-Learning, are being widely used to represent complex systems such as electricity markets. The developed model is designed to simulate in a detailed way the hydro units that have a large impact in the electricity market common to Portugal and Spain. Apart from describing the developed model, this paper also includes results from its application to the Iberian Market case along 2018.

1 citations

Cites methods from "Simulation of the operation of hydr..."

...This algorithm is detailed in [20] and the parameters λ and γ used in this study are equal to 0....

[...]

...This strategy is combined with the Q-Learning as explained in [20]....

TL;DR: This book provides a clear and simple account of the key ideas and algorithms of reinforcement learning, which ranges from the history of the field's intellectual foundations to the most recent developments and applications.

Abstract: Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

32,257 citations

"Simulation of the operation of hydr..." refers methods in this paper

...Q learning is a reinforcement learning methodology [15] in which agents can learn a task by interacting with the environment through a trial and error search....

TL;DR: This paper presents and proves in detail a convergence theorem forQ-learning based on that outlined in Watkins (1989), showing that Q-learning converges to the optimum action-values with probability 1 so long as all actions are repeatedly sampled in all states and the action- values are represented discretely.

Abstract: \cal Q-learning (Watkins, 1989) is a simple way for agents to learn how to act optimally in controlled Markovian domains. It amounts to an incremental method for dynamic programming which imposes limited computational demands. It works by successively improving its evaluations of the quality of particular actions at particular states.
This paper presents and proves in detail a convergence theorem for \cal Q-learning based on that outlined in Watkins (1989). We show that \cal Q-learning converges to the optimum action-values with probability 1 so long as all actions are repeatedly sampled in all states and the action-values are represented discretely. We also sketch extensions to the cases of non-discounted, but absorbing, Markov environments, and where many \cal Q values can be changed each iteration, rather than just one.

Abstract: Q-learning (Watkins, 1989) is a simple way for agents to learn how to act optimally in controlled Markovian domains. It amounts to an incremental method for dynamic programming which imposes limited computational demands. It works by successively improving its evaluations of the quality of particular actions at particular states. This paper presents and proves in detail a convergence theorem for Q,-learning based on that outlined in Watkins (1989). We show that Q-learning converges to the optimum action-values with probability 1 so long as all actions are repeatedly sampled in all states and the action-values are represented discretely. We also sketch extensions to the cases of non-discounted, but absorbing, Markov environments, and where many Q values can be changed each iteration, rather than just one.

Abstract: The electricity market has since 1980s been gradually evolving from a monopoly market into a liberalized one for encouraging competition and improving efficiency. This brings the opportunity for generation companies (GenCos) to make more profits while embracing more risks of not being dispatched. Therefore, it has become a core interest for the GenCos to develop optimal bidding strategies to maximize the profits and minimize the risks while participating in such a competitive market. The literature pertaining to this issue has grown rapidly in recent years, and many different modeling approaches, such as mathematical programming, game theory, and agent-based models, have been investigated under the liberalized market environment. Meanwhile, along with the increasing penetration of renewable energy, the electricity market is facing more complexity and stochasticity from both uncertain generation and dynamic demands. The intermittent and unsteady nature of these renewable power sources motivates the GenCos to further optimize their bidding strategy by considering the new constraints. This paper presents a comprehensive literature analysis on the state-of-the-art research of bidding strategy modeling methods.

171 citations

"Simulation of the operation of hydr..." refers background in this paper

...These approaches can be organized in four main areas [10]:...