This book introduces multiagent planning under uncertainty as formalized by decentralized partially observable Markov decision processes (Dec-POMDPs). The intended audience is researchers and graduate students working in the fields of artificial intelligence related to sequential decision making: reinforcement learning, decision-theoretic planning for single agents, classical multiagent planning, decentralized control, and operations research.

A Concise Introduction to Decentralized POMDPs

One approach to designing the decision making logic for an aircraft collision avoidance system is to frame the problem as Markov decision process and optimize the system using dynamic programming. The resulting strategy can be represented as a numeric table. This methodology has been used in the development of the ACAS X family of collision avoidance systems for manned and unmanned aircraft. However, due to the high dimensionality of the state space, discretizing the state variables can lead to very large tables. To improve storage efficiency, we propose two approaches for compressing the lookup table. The first approach exploits redundancy in the table. The table is decomposed into a set of lower-dimensional tables, some of which can be represented by single tables in areas where the lower-dimensional tables are identical or nearly identical with respect to a similarity metric. The second approach uses a deep neural network to learn a complex non-linear function approximation of the table. With the use of an asymmetric loss function and a gradient descent algorithm, the parameters for this network can be trained to provide very accurate estimates of values while preserving the relative preferences of the possible advisories for each state. As a result, the table can be approximately represented by only the parameters of the network, which reduces the required storage space by a factor of 1000. Simulation studies show that system performance is very similar using either compressed table representation in place of the original table. Even though the neural network was trained directly on the original table, the network surpasses the original table on the performance metrics and encounter sets evaluated here.

Policy compression for aircraft collision avoidance systems

Tactical decision making for autonomous driving is challenging due to the diversity of environments, the uncertainty in the sensor information, and the complex interaction with other road users. This article introduces a general framework for tactical decision making, which combines the concepts of planning and learning, in the form of Monte Carlo tree search and deep reinforcement learning. The method is based on the AlphaGo Zero algorithm, which is extended to a domain with a continuous state space where self-play cannot be used. The framework is applied to two different highway driving cases in a simulated environment and it is shown to perform better than a commonly used baseline method. The strength of combining planning and learning is also illustrated by a comparison to using the Monte Carlo tree search or the neural network policy separately.

Combining Planning and Deep Reinforcement Learning in Tactical Decision Making for Autonomous Driving

Transparency of algorithmic systems entails exposing system properties to various stakeholders for purposes that include understanding, improving, and/or contesting predictions. The machine learning (ML) community has mostly considered explainability as a proxy for transparency. With this work, we seek to encourage researchers to study uncertainty as a form of transparency and practitioners to communicate uncertainty estimates to stakeholders. First, we discuss methods for assessing uncertainty. Then, we describe the utility of uncertainty for mitigating model unfairness, augmenting decision-making, and building trustworthy systems. We also review methods for displaying uncertainty to stakeholders and discuss how to collect information required for incorporating uncertainty into existing ML pipelines. Our contribution is an interdisciplinary review to inform how to measure, communicate, and use uncertainty as a form of transparency.

Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty

POMDPs.jl is an open-source framework for solving Markov decision processes (MDPs) and partially observable MDPs (POMDPs). POMDPs.jl allows users to specify sequential decision making problems with minimal effort without sacrificing the expressive nature of POMDPs, making this framework viable for both educational and research purposes. It is written in the Julia language to allow flexible prototyping and large-scale computation that leverages the high-performance nature of the language. The associated JuliaPOMDP community also provides a number of state-of-the-art MDP and POMDP solvers and a rich library of support tools to help with implementing new solvers and evaluating the solution results. The most recent version of POMDPs.jl, the related packages, and documentation can be found at https://github.com/JuliaPOMDP/POMDPs.jl.

/pdf/pomdps-jl-a-framework-for-sequential-decision-making-under-qsbhp9vy8b.pdf

POMDPs.jl: a framework for sequential decision making under uncertainty

Many important problems involve decision making under uncertainty -- that is, choosing actions based on often imperfect observations, with unknown outcomes Designers of automated decision support systems must take into account the various sources of uncertainty while balancing the multiple objectives of the system This book provides an introduction to the challenges of decision making under uncertainty from a computational perspective It presents both the theory behind decision making models and algorithms and a collection of example applications that range from speech recognition to aircraft collision avoidance Focusing on two methods for designing decision agents, planning and reinforcement learning, the book covers probabilistic models, introducing Bayesian networks as a graphical model that captures probabilistic relationships between variables; utility theory as a framework for understanding optimal decision making under uncertainty; Markov decision processes as a method for modeling sequential problems; model uncertainty; state uncertainty; and cooperative decision making involving multiple interacting agents A series of applications shows how the theoretical concepts can be applied to systems for attribute-based person search, speech applications, collision avoidance, and unmanned aircraft persistent surveillance Decision Making Under Uncertainty unifies research from different communities using consistent notation, and is accessible to students and researchers across engineering disciplines who have some prior exposure to probability theory and calculus It can be used as a text for advanced undergraduate and graduate students in fields including computer science, aerospace and electrical engineering, and management science It will also be a valuable professional reference for researchers in a variety of disciplines

Decision Making Under Uncertainty: Theory and Application

This chapter contains sections titled: Half Title, MIT Lincoln Laboratory Series, Title, Copyright, Dedication, Table of Contents, Preface, About the Authors, Acknowledgments

/pdf/decision-making-under-uncertainty-dye2kx7z25.pdf

Decision Making under Uncertainty

Developing robust aircraft collision avoidance logic that reliably prevents midair collisions without excessive alerting is challenging because of sensor error and uncertainty in the future paths of the aircraft Over the past few years, research has focused on the use of a computational method known as dynamic programming for producing an optimized decision logic for airborne collision avoidance

Optimized Airborne Collision Avoidance

Cooperative Decision Making

An important application of unmanned aircraft is persistent surveillance over areas of interest. For example, a team of aircraft can be used to monitor a forest for biological activities, a disaster flooded area for water levels, or a battle theater for movement. In such scenarios, surveillance must be persistent over an extended duration, and team members may be geographically distributed.

Hayley J. Davison Reynolds

Papers

Decision Making Under Uncertainty: Theory and Application

Decision Making under Uncertainty

Optimized Airborne Collision Avoidance

Cooperative Decision Making

Multiagent Planning for Persistent Surveillance