Planning algorithms are impacting technical disciplines and industries around the world, including robotics, computer-aided design, manufacturing, computer graphics, aerospace applications, drug design, and protein folding. This coherent and comprehensive book unifies material from several sources, including robotics, control theory, artificial intelligence, and algorithms. The treatment is centered on robot motion planning but integrates material on planning in discrete spaces. A major part of the book is devoted to planning under uncertainty, including decision theory, Markov decision processes, and information spaces, which are the “configuration spaces” of all sensor-based planning problems. The last part of the book delves into planning under differential constraints that arise when automating the motions of virtually any mechanical system. Developed from courses taught by the author, the book is intended for students, engineers, and researchers in robotics, artificial intelligence, and control theory as well as computer graphics, algorithms, and computational biology.

Planning Algorithms: Introductory Material

This paper introduces the Point-Based Value Iteration (PBVI) algorithm for POMDP planning. PBVI approximates an exact value iteration solution by selecting a small set of representative belief points and then tracking the value and its derivative for those points only. By using stochastic trajectories to choose belief points, and by maintaining only one value hyper-plane per point, PBVI successfully solves large problems: we present results on a robotic laser tag problem as well as three test domains from the literature.

/pdf/point-based-value-iteration-an-anytime-algorithm-for-pomdps-479i4hs0kt.pdf

Point-based value iteration: an anytime algorithm for POMDPs

Planning for distributed agents with partial state information is considered from a decisiontheoretic perspective. We describe generalizations of both the MDP and POMDP models that allow for decentralized control. For even a small number of agents, the finite-horizon problems corresponding to both of our models are complete for nondeterministic exponential time. These complexity results illustrate a fundamental difference between centralized and decentralized control of Markov processes. In contrast to the MDP and POMDP problems, the problems we consider provably do not admit polynomialtime algorithms and most likely require doubly exponential time to solve in the worst case. We have thus provided mathematical evidence corresponding to the intuition that decentralized planning problems cannot easily be reduced to centralized problems and solved exactly using established techniques.

/pdf/the-complexity-of-decentralized-control-of-markov-decision-25tsias94m.pdf

The complexity of decentralized control of Markov decision processes

We consider decentralized control of Markov decision processes and give complexity bounds on the worst-case running time for algorithms that find optimal solutions. Generalizations of both the fully observable case and the partially observable case that allow for decentralized control are described. For even two agents, the finite-horizon problems corresponding to both of these models are hard for nondeterministic exponential time. These complexity results illustrate a fundamental difference between centralized and decentralized control of Markov decision processes. In contrast to the problems involving centralized control, the problems we considerprovably do not admit polynomial-time algorithms. Furthermore, assuming EXP ? NEXP, the problems require superexponential time to solve in the worst case.

The Complexity of Decentralized Control of Markov Decision Processes

Partially observable Markov decision processes (POMDPs) form an attractive and principled framework for agent planning under uncertainty. Point-based approximate techniques for POMDPs compute a policy based on a finite set of points collected in advance from the agent's belief space. We present a randomized point-based value iteration algorithm called PERSEUS. The algorithm performs approximate value backup stages, ensuring that in each backup stage the value of each point in the belief set is improved; the key observation is that a single backup may improve the value of many belief points. Contrary to other point-based methods, PERSEUS backs up only a (randomly selected) subset of points in the belief set, sufficient for improving the value of each belief point in the set. We show how the same idea can be extended to dealing with continuous action spaces. Experimental results show the potential of PERSEUS in large scale POMDP problems.

/pdf/perseus-randomized-point-based-value-iteration-for-pomdps-1zrr6uj72u.pdf

Perseus: randomized point-based value iteration for POMDPs

We present a real time non-invasive system that infers user stress level from evidences of different modalities. The evidences include physical appearance (facial expression, eye movements, and head movements) extracted from video via visual sensors, physiological conditions collected from an emotional mouse, behavioral data from user interaction activities with the computer, and performance measures. We provide a Dynamic Bayesian Network (DBN) framework to model the user stress and these evidences. We describe the computer vision techniques we used to extract the visual evidences, the DBN model for modeling stress and the associated factors, and the active sensing strategy to collect the most informative evidences for efficient stress inference. Our experiments show that the inferred user stress level by our system is consistent with that predicted by psychological theories.

/pdf/a-real-time-human-stress-monitoring-system-using-dynamic-5n0hkimyba.pdf

A Real-Time Human Stress Monitoring System Using Dynamic Bayesian Network

Partially observable Markov decision processes (POMDPs) have recently become popular among many AI researchers because they serve as a natural model for planning under uncertainty. Value iteration is a well-known algorithm for finding optimal policies for POMDPs. It typically takes a large number of iterations to converge. This paper proposes a method for accelerating the convergence of value iteration. The method has been evaluated on an array of benchmark problems and was found to be very effective: It enabled value iteration to converge after only a few iterations on all the test problems.

/pdf/speeding-up-the-convergence-of-value-iteration-in-partially-1i6mhmxkpy.pdf

Speeding up the convergence of value iteration in partially observable Markov decision processes

Partially Observable Markov Decision Process (POMDP) is a general sequential decision-making model where the effects of actions are nondeterministic and only partial information about world states is available. However, finding near optimal solutions for POMDPs is computationally difficult. Value iteration is a standard algorithm for solving POMDPs. It conducts a sequence of dynamic programming (DP) updates to improve value functions. 
Value iteration is inefficient for two reasons. First, a DP update is expensive due to the need of accounting for all belief states in a continuous belief space. Second, value iteration needs to conduct a large number of DP updates before its convergence. This thesis investigates two ways to accelerate value iteration. The work presented centers around the idea of conducting DP updates and therefore value iteration over a belief subspace, a subset of belief space. 
The first use of belief subspace is to reduce the number of DP updates for value iteration to converge. We design a computationally cheap procedure considering a belief subspace which consists of a finite number of belief states. It is used as an additional step for improving value functions. Due to additional improvements by the procedure, value iteration conducts fewer DP updates and therefore is more efficient. 
The second use of belief subspace is to reduce the complexity of DP updates. We establish a framework on how to carry out value iteration over a belief subspace determined by a POMDP model. Whether the belief subspace is smaller than the belief space is model dependent. If this is true for a POMDP, value iteration over the belief subspace is expected to be more efficient. 
Based on this framework, we study three POMDP classes with special problem characteristics and propose different value iteration algorithms for them. (1) An informative POMDP assumes that an agent always has a good idea about the world states. The subspace determined by the model is much smaller than the belief space. Value iteration over the belief subspace is more efficient for this POMDP class. (2) A near-discernible POMDP assumes that the agent can get a good idea about states once in a while if it executes some particular actions. For such a POMDP, the belief subspace determined by the model can be of the same size as the belief space. We propose an anytime value iteration algorithm which focuses the computations on a small belief subspace and gradually expand it. (3) A more general class than near-discernible POMDPs assumes that the agent can get a good idea about states with a high likelihood once in a while if it executes some particular actions. For such POMDPs, we adapt the anytime algorithm to conduct value iteration over a growing belief subspace.

Algorithms for partially observable markov decision processes

There is an increasing interest in developing intelligent human-computer interaction systems that can fulfill two functions-recognizing user affective states and providing the user with timely and appropriate assistance. In this paper, we present a general unified decision-theoretic framework based on influence diagrams for simultaneously modeling user affect recognition and assistance. Affective state recognition is achieved through active probabilistic inference from the available multi modality sensory data. User assistance is automatically accomplished through a decision-making process that balances the benefits of keeping the user in productive affective states and the costs of performing user assistance. We discuss three theoretical issues within the framework, namely, user affect recognition, active sensory action selection, and user assistance. Validation of the proposed framework via a simulation study demonstrates its capability in efficient user affect recognition as well as timely and appropriate user assistance. Besides the theoretical contributions, we build a non-invasive real-time prototype system to recognize different user affective states (stress and fatigue) from four-modality user measurements, namely physical appearance features, physiological measures, user performance, and behavioral data. The affect recognition component of the prototype system is subsequently validated through a real-world study involving human subjects.

https://interruptions.net/literature/Liao-IJHCS06.pdf

Toward a decision-theoretic framework for affect recognition and user assistance

An output of a photovoltaic (PV) device is predicted by applying Fourier analysis to historical data to obtain frequencies and a mean of the frequencies in the data. Regression analysis is applied to the data to obtain a regression coefficient. Then, the prediction is a sum of the mean at the time step and a deviation from the mean at a previous time step, wherein the means are represented and approximated by selected frequencies, and the deviation for the previous time step is weighted by the regression coefficient.

Weihong Zhang

Papers

A Real-Time Human Stress Monitoring System Using Dynamic Bayesian Network

Speeding up the convergence of value iteration in partially observable Markov decision processes

Algorithms for partially observable markov decision processes

Toward a decision-theoretic framework for affect recognition and user assistance

Method for predicting outputs of photovoltaic devices based on two-dimensional fourier analysis and seasonal auto-regression