Bellman equation

About: Bellman equation is a research topic. Over the lifetime, 5884 publications have been published within this topic receiving 135589 citations.

Sensitivity analysis of nonlinear programs and differentiability properties of metric projections

Abstract: This paper is concerned with a study of differentiability properties of the optimal value function and an associated optimal solution of a parametrized nonlinear program. Second order analysis is presented essentially under the Mangasarian-Fromovitz constraint qualification when the corresponding vector of Lagrange multipliers is not necessarily unique. It is shown that under certain regularity conditions the optimal value function possesses second order directional derivatives and the optimal solution mapping is directionally differentiable. The results obtained are applied to an investigation of metric projections in finite-dimensional spaces.

Optimal stochastic switching and the Dirichlet problem for the Bellman equation

Abstract: ABSTcRT. Let L' be a sequence of second order elliptic operators in a bounded n-dimensional domain Q, and let f' be given functions. Consider the problem of finding a solution u to the Bellman equation sup5(Lu f) 0 a.e. in Q, subject to the Dirichiet boundary condition u = 0 on M2. It is proved that, provided the leading coefficients of the L' are constants, there exists a unique solution u of this problem, belonging to W" '(Q) i W12',d(9). The solution is obtained as a limit of solutions of certain weakly coupled systems of nonlinear elliptic equations; each component of the vector solution converges to u. Although the proof is entirely analytic, it is partially motivated by models of stochastic control. We solve also certain systems of variational inequalities corresponding to switching with cost.

An Algorithm for Approximate Multiparametric Convex Programming

Abstract: For multiparametric convex nonlinear programming problems we propose a recursive algorithm for approximating, within a given suboptimality tolerance, the value function and an optimizer as functions of the parameters. The approximate solution is expressed as a piecewise affine function over a simplicial partition of a subset of the feasible parameters, and it is organized over a tree structure for efficiency of evaluation. Adaptations of the algorithm to deal with multiparametric semidefinite programming and multiparametric geometric programming are provided and exemplified. The approach is relevant for real-time implementation of several optimization-based feedback control strategies.

The Uncertainty Bellman Equation and Exploration.

Abstract: We consider the exploration/exploitation problem in reinforcement learning. For exploitation, it is well known that the Bellman equation connects the value at any time-step to the expected value at subsequent time-steps. In this paper we consider a similar \textit{uncertainty} Bellman equation (UBE), which connects the uncertainty at any time-step to the expected uncertainties at subsequent time-steps, thereby extending the potential exploratory benefit of a policy beyond individual time-steps. We prove that the unique fixed point of the UBE yields an upper bound on the variance of the posterior distribution of the Q-values induced by any policy. This bound can be much tighter than traditional count-based bonuses that compound standard deviation rather than variance. Importantly, and unlike several existing approaches to optimism, this method scales naturally to large systems with complex generalization. Substituting our UBE-exploration strategy for $\epsilon$-greedy improves DQN performance on 51 out of 57 games in the Atari suite.