scispace - formally typeset
Search or ask a question
Topic

Bellman equation

About: Bellman equation is a research topic. Over the lifetime, 5884 publications have been published within this topic receiving 135589 citations.


Papers
More filters
Journal ArticleDOI
TL;DR: In this article, the authors studied the Markov decision process under the maximization of the probability that total discounted rewards exceed a target level, and studied the dynamic programing equations of the model.
Abstract: The Markov decision process is studied under the maximization of the probability that total discounted rewards exceed a target level. We focus on and study the dynamic programing equations of the model. We give various properties of the optimal return operator and, for the infinite planning-horizon model, we characterize the optimal value function as a maximal fixed point of the previous operator. Various turnpike results relating the finite and infinite-horizon models are also given.

49 citations

Journal ArticleDOI
Endre Pap1
TL;DR: A unified method is given for solving some nonlinear differential (ordinary and partial), difference and optimization equations, using pseudo-superposition principle and pseudoLaplace transform.

49 citations

Journal ArticleDOI
TL;DR: This work considers power allocation for an access-controlled transmitter with energy harvesting capability based on causal observations of the channel fading state and proposes power allocation algorithms for both the finite- and infinite-horizon cases whose computational complexity is significantly lower than that of the standard discrete MDP method but with improved performance.
Abstract: We consider power allocation for an access-controlled transmitter with energy harvesting capability based on causal observations of the channel fading state. We assume that the system operates in a time-slotted fashion and the channel gain in each slot is a random variable which is independent across slots. Further, we assume that the transmitter is solely powered by a renewable energy source and the energy harvesting process can practically be predicted. With the additional access control for the transmitter and the maximum power constraint, we formulate the stochastic optimization problem of maximizing the achievable rate as a Markov decision process (MDP) with continuous state. To effi- ciently solve the problem, we define an approximate value function based on a piecewise linear fit in terms of the battery state. We show that with the approximate value function, the update in each iteration consists of a group of convex problems with a continuous parameter. Moreover, we derive the optimal solution to these con- vex problems in closed-form. Further, we propose power allocation algorithms for both the finite- and infinite-horizon cases, whose computational complexity is significantly lower than that of the standard discrete MDP method but with improved performance. Extension to the case of a general payoff function and imperfect energy prediction is also considered. Finally, simulation results demonstrate that the proposed algorithms closely approach the optimal performance.

49 citations

Journal ArticleDOI
TL;DR: It is shown that the optimal value function of an MDP is monotone with respect to appropriately defined stochastic order relations, and conditions for continuity withrespect to suitable probability metrics are found.
Abstract: The present work deals with the comparison of discrete time Markov decision processes MDPs, which differ only in their transition probabilities. We show that the optimal value function of an MDP is monotone with respect to appropriately defined stochastic order relations. We also find conditions for continuity with respect to suitable probability metrics. The results are applied to some well-known examples, including inventory control and optimal stopping.

49 citations

Journal ArticleDOI
TL;DR: In this paper, the authors established some elementary results on solutions to the Bellman equation without introducing any topological assumption, and applied these results to two optimal growth models: one with a discontinuous production function and the other with a roughly increasing return.
Abstract: We establish some elementary results on solutions to the Bellman equation without introducing any topological assumption. Under a small number of conditions, we show that the Bellman equation has a unique solution in a certain set, that this solution is the value function, and that the value function can be computed by value iteration with an appropriate initial condition. In addition, we show that the value function can be computed by the same procedure under alternative conditions. We apply our results to two optimal growth models: one with a discontinuous production function and the other with “roughly increasing” returns.

49 citations


Network Information
Related Topics (5)
Optimal control
68K papers, 1.2M citations
87% related
Bounded function
77.2K papers, 1.3M citations
85% related
Markov chain
51.9K papers, 1.3M citations
85% related
Linear system
59.5K papers, 1.4M citations
84% related
Optimization problem
96.4K papers, 2.1M citations
83% related
Performance
Metrics
No. of papers in the topic in previous years
YearPapers
2023261
2022537
2021369
2020411
2019348
2018353