Bellman equation

About: Bellman equation is a research topic. Over the lifetime, 5884 publications have been published within this topic receiving 135589 citations.

Variational properties of value functions

Abstract: Regularization plays a key role in a variety of optimization formulations of inverse problems. A recurring question in regularization approaches is the selection of regularization pa- rameters, and its eect on the solution and on the optimal value of the optimization problem. The sensitivity of the value function to the regularization parameter can be linked directly to the Lagrange multipliers. In this paper, we fully characterize the variational properties of the value functions for a broad class of convex formulations, which are not all covered by standard Lagrange multiplier theory. We also present an inverse function theorem that links the value functions of dierent regularization formulations (not necessarily convex). These results have implications for the selection of regularization parameters, and the development of specialized algorithms. We give numerical examples that illustrate the theoretical results.

Optimal Execution with Multiplicative Price Impact

Abstract: We consider the so-called optimal execution problem in algorithmic trading, which is the problem faced by an investor who has a large number of stock shares to sell over a given time horizon and whose actions have an impact on the stock price. In particular, we develop and study a price model that presents the stochastic dynamics of a geometric Brownian motion and incorporates a log-linear effect of the investor's transactions. We then formulate the optimal execution problem as a degenerate singular stochastic control problem. Using both analytic and probabilistic techniques, we establish simple conditions for the market to allow for no arbitrage or price manipulation and develop a detailed characterization of the value function and the optimal strategy. In particular, we derive an explicit solution to the problem if the time horizon is infinite.

A review of stochastic algorithms with continuous value function approximation and some new approximate policy iteration algorithms for multidimensional continuous applications

Abstract: We review the literature on approximate dynamic programming, with the goal of better understanding the theory behind practical algorithms for solving dynamic programs with continuous and vector-valued states and actions and complex information processes. We build on the literature that has addressed the well-known problem of multidimensional (and possibly continuous) states, and the extensive literature on model-free dynamic programming, which also assumes that the expectation in Bellman’s equation cannot be computed. However, we point out complications that arise when the actions/controls are vector-valued and possibly continuous. We then describe some recent research by the authors on approximate policy iteration algorithms that offer convergence guarantees (with technical assumptions) for both parametric and nonparametric architectures for the value function.

An incremental sampling-based algorithm for stochastic optimal control

Abstract: In this paper, we consider a class of continuous-time, continuous-space stochastic optimal control problems Building upon recent advances in Markov chain approximation methods and sampling-based algorithms for deterministic path planning, we propose a novel algorithm called the incremental Markov Decision Process (iMDP) to compute incrementally control policies that approximate arbitrarily well an optimal policy in terms of the expected cost The main idea behind the algorithm is to generate a sequence of finite discretizations of the original problem through random sampling of the state space At each iteration, the discretized problem is a Markov Decision Process that serves as an incrementally refined model of the original problem We show that with probability one, (i) the sequence of the optimal value functions for each of the discretized problems converges uniformly to the optimal value function of the original stochastic optimal control problem, and (ii) the original optimal value function can be computed efficiently in an incremental manner using asynchronous value iterations Thus, the proposed algorithm provides an anytime approach to the computation of optimal control policies of the continuous problem The effectiveness of the proposed approach is demonstrated on motion planning and control problems in cluttered environments in the presence of process noise

Conjugate points and shocks in nonlinear optimal control

Abstract: In this paper the authors use the method of characteristics to extend the Jacobi conjugate points theory to the Bolza problem arising in nonlinear optimal control. This yields necessary and sufficient optimality conditions for weak and strong local minima stated in terms of the existence of a solution to a corresponding matrix Riccati differential equation. The same approach allows to investigate as well smoothness of the value function.