scispace - formally typeset
Search or ask a question
Topic

Bellman equation

About: Bellman equation is a research topic. Over the lifetime, 5884 publications have been published within this topic receiving 135589 citations.


Papers
More filters
Journal ArticleDOI
TL;DR: The possibility for the immediate one-impulse strategy to be nonoptimal while both growth functions are monotonic is a surprising result and is illustrated with the help of numerical simulations.
Abstract: We consider the optimal control problem of feeding in minimal time a tank where several species compete for a single resource, with the objective being to reach a given level of the resource. We allow controls to be bounded measurable functions of time plus possible impulses. For the one-species case, we show that the immediate one-impulse strategy (filling the whole reactor with one single impulse at the initial time) is optimal when the growth function is monotonic. For nonmonotonic growth functions with one maximum, we show that a particular singular arc strategy (precisely defined in section 3) is optimal. These results extend and improve former ones obtained for the class of measurable controls only. For the two-species case with monotonic growth functions, we give conditions under which the immediate one-impulse strategy is optimal. We also give optimality conditions for the singular arc strategy (at a level that depends on the initial condition) to be optimal. The possibility for the immediate one-impulse strategy to be nonoptimal while both growth functions are monotonic is a surprising result and is illustrated with the help of numerical simulations.

61 citations

Journal ArticleDOI
TL;DR: Effective ways for verifying the Slater type conditions are investigated and other conditions which are based on lower semicontinuity of the optimal value function of the inner maximization problem of the DRO are introduced.
Abstract: A key step in solving minimax distributionally robust optimization (DRO) problems is to reformulate the inner maximization w.r.t. probability measure as a semiinfinite programming problem through Lagrange dual. Slater type conditions have been widely used for strong duality (zero dual gap) when the ambiguity set is defined through moments. In this paper, we investigate effective ways for verifying the Slater type conditions and introduce other conditions which are based on lower semicontinuity of the optimal value function of the inner maximization problem. Moreover, we propose two discretization schemes for solving the DRO with one for the dualized DRO and the other directly through the ambiguity set of the DRO. In the absence of strong duality, the discretization scheme via Lagrange duality may provide an upper bound for the optimal value of the DRO whereas the direct discretization approach provides a lower bound. Two cutting plane schemes are consequently proposed: one for the discretized dualized DRO and the other for the minimax DRO with discretized ambiguity set. Convergence analysis is presented for the approximation schemes in terms of the optimal value, optimal solutions and stationary points. Comparative numerical results are reported for the resulting algorithms.

61 citations

Journal ArticleDOI
TL;DR: A state and action discretization procedure for approximating the optimal value function and an optimal policy of the original control model is proposed and explicit bounds on the approximation errors are provided.

61 citations

Posted Content
TL;DR: This work introduces generic model-free algorithms based on the state-action value function at the mean field level and proves convergence for a prototypical Q-learning method for mean field control problems.
Abstract: We develop a general reinforcement learning framework for mean field control (MFC) problems. Such problems arise for instance as the limit of collaborative multi-agent control problems when the number of agents is very large. The asymptotic problem can be phrased as the optimal control of a non-linear dynamics. This can also be viewed as a Markov decision process (MDP) but the key difference with the usual RL setup is that the dynamics and the reward now depend on the state's probability distribution itself. Alternatively, it can be recast as a MDP on the Wasserstein space of measures. In this work, we introduce generic model-free algorithms based on the state-action value function at the mean field level and we prove convergence for a prototypical Q-learning method. We then implement an actor-critic method and report numerical results on two archetypal problems: a finite space model motivated by a cyber security application and a continuous space model motivated by an application to swarm motion.

61 citations

Journal ArticleDOI
TL;DR: In this article, an optimal control strategy for an open two-level system (a qubit in interaction with the electromagnetic field) controlled by a laser is derived using results from quantum filtering theory and methods from classical control theory.
Abstract: Using results from quantum filtering theory and methods from classical control theory, we derive an optimal control strategy for an open two-level system (a qubit in interaction with the electromagnetic field) controlled by a laser. The aim is to optimally choose the laser's amplitude and phase in order to drive the system into a desired state. The Bellman equations are obtained for the case of diffusive and counting measurements for vacuum field states. A full exact solution of the optimal control problem is given for a system with simpler, linear, dynamics. These linear dynamics can be obtained physically by considering a two-level atom in a strongly driven, heavily damped, optical cavity.

61 citations


Network Information
Related Topics (5)
Optimal control
68K papers, 1.2M citations
87% related
Bounded function
77.2K papers, 1.3M citations
85% related
Markov chain
51.9K papers, 1.3M citations
85% related
Linear system
59.5K papers, 1.4M citations
84% related
Optimization problem
96.4K papers, 2.1M citations
83% related
Performance
Metrics
No. of papers in the topic in previous years
YearPapers
2023261
2022537
2021369
2020411
2019348
2018353