We develop a general reinforcement learning framework for mean field control (MFC) problems. Such problems arise for instance as the limit of collaborative multi-agent control problems when the number of agents is very large. The asymptotic problem can be phrased as the optimal control of a non-linear dynamics. This can also be viewed as a Markov decision process (MDP) but the key difference with the usual RL setup is that the dynamics and the reward now depend on the state's probability distribution itself. Alternatively, it can be recast as a MDP on the Wasserstein space of measures. In this work, we introduce generic model-free algorithms based on the state-action value function at the mean field level and we prove convergence for a prototypical Q-learning method. We then implement an actor-critic method and report numerical results on two archetypal problems: a finite space model motivated by a cyber security application and a continuous space model motivated by an application to swarm motion.

Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning

We investigate reinforcement learning for mean field control problems in discrete time, which can be viewed as Markov decision processes for a large number of exchangeable agents interacting in a mean field manner. Such problems arise, for instance when a large number of robots communicate through a central unit dispatching the optimal policy computed by minimizing the overall social cost. An approximate solution is obtained by learning the optimal policy of a generic agent interacting with the statistical distribution of the states of the other agents. We prove rigorously the convergence of exact and model-free policy gradient methods in a mean-field linear-quadratic setting. We also provide graphical evidence of the convergence based on implementations of our algorithms.

Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods.

Energy storage systems (ESSs) have been considered to be an effective solution to reduce the spatial and temporal imbalance between the stochastic energy generation and the demand. To effectively utilize an ESS, an approach of jointly sharing and operating an ESS has been proposed in a conceptual way. However, there is a lack of analytic approaches designed to optimize the operation of such a system considering interactions among users in a game theoretic perspective. In this study, we propose the energy capacity trading and operation (ECTO) game where each agent determines two actions, capacity trading, and the 24-hour ahead charging-discharging scheduling with the capacity that will be assigned, to minimize the energy operation cost. We then propose a distributed optimization strategy to find a generalized Nash equilibrium for the proposed ECTO game. Simulation studies show that when optimally operated, a shared ESS can decrease both the total energy operating cost and the peak-to-average ratio of the energy for the entire grid compared to the conventional ESS control strategy without ESS sharing.

Demand-Side Management With Shared Energy Storage System in Smart Grid

Here, we introduce a price formation model where a large number of small players can store and trade a commodity such as electricity. Our model is a constrained mean-field game (MFG) where the price is a Lagrange multiplier for the supply versus demand balance condition. We establish the existence of a unique solution using a fixed-point argument. In particular, we show that the price is well defined, and it is a Lipschitz function of time. Then, we study linear-quadratic models that can be solved explicitly and compare our model with real data.

/pdf/a-mean-field-game-approach-to-price-formation-2122v4wi4m.pdf

A Mean-Field Game Approach to Price Formation

In this paper, we deepen the analysis of continuous time Fictitious Play learning algorithm to the consideration of various finite state Mean Field Game settings (finite horizon, $\gamma$-discounted), allowing in particular for the introduction of an additional common noise. 
We first present a theoretical convergence analysis of the continuous time Fictitious Play process and prove that the induced exploitability decreases at a rate $O(\frac{1}{t})$. Such analysis emphasizes the use of exploitability as a relevant metric for evaluating the convergence towards a Nash equilibrium in the context of Mean Field Games. These theoretical contributions are supported by numerical experiments provided in either model-based or model-free settings. We provide hereby for the first time converging learning dynamics for Mean Field Games in the presence of common noise.

Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications

We consider a stylized model for a power network with distributed local power generation and storage. This system is modeled as a network connection of a large number of nodes, where each node is characterized by a local electricity consumption, has a local electricity production (photovoltaic panels for example) and manages a local storage device. Depending on its instantaneous consumption and production rate as well as its storage management decision, each node may either buy or sell electricity, impacting the electricity spot price. The objective at each node is to minimize energy and storage costs by optimally controlling the storage device. In a noncooperative game setting, we are led to the analysis of a nonzero sum stochastic game with N players where the interaction takes place through the spot price mechanism. For an infinite number of agents, our model corresponds to an extended mean field game. We are able to compare this solution to the optimal strategy of a central planner and in a linear quadratic setting, we obtain and explicit solution to the extended mean field game and we show that it provides an approximate Nash equilibrium for N-player game.

https://hal.archives-ouvertes.fr/hal-01740707/document

Imen Ben Taher

Papers

An Extended Mean Field Game for Storage in Smart Grids