http://user.it.uu.se/~jarst116/slides/week4.pdf

Graph-Based Algorithms for Boolean Function Manipulation

We propose a novel method for generating test scenarios for a black box autonomous system that demonstrate critical transitions in its performance modes. In complex environments it is possible for an autonomous system to fail at its assigned mission even if it complies with requirements for all subsystems and throws no faults. This is particularly true when the autonomous system may have to choose between multiple exclusive objectives. The standard approach of testing robustness through fault detection is directly stimulating the system and detecting violations of the system requirements. Our approach differs by instead running the autonomous system through full missions in a simulated environment and measuring performance based on high-level mission criteria. The result is a method of searching for challenging scenarios for an autonomous system under test that exercise a variety of performance modes. We utilize adaptive sampling to intelligently search the state space for test scenarios which exist on the boundary between distinct performance modes. Additionally, using unsupervised clustering techniques we can group scenarios by their performance modes and sort them by those which are most effective at diagnosing changes in the autonomous system's behavior.

Automated generation of diverse and challenging scenarios for test and evaluation of autonomous vehicles

The growth of scale and complexity of interactions between humans and robots highlights the need for new computational methods to automatically evaluate novel algorithms and applications. Exploring diverse scenarios of humans and robots interacting in simulation can improve understanding of the robotic system and avoid potentially costly failures in real-world settings. We formulate this problem as a quality diversity (QD) problem, where the goal is to discover diverse failure scenarios by simultaneously exploring both environments and human actions. We focus on the shared autonomy domain, where the robot attempts to infer the goal of a human operator, and adopt the QD algorithm MAP-Elites to generate scenarios for two published algorithms in this domain: shared autonomy via hindsight optimization and linear policy blending. Some of the generated scenarios confirm previous theoretical findings, while others are surprising and bring about a new understanding of state-of-the-art implementations. Our experiments show that MAP-Elites outperforms Monte-Carlo simulation and optimization based methods in effectively searching the scenario space, highlighting its promise for automatic evaluation of algorithms in human-robot interaction.

A Quality Diversity Approach to Automatically Generating Human-Robot Interaction Scenarios in Shared Autonomy

We present probabilistic models for autonomous agent search and retrieve missions derived from Simulink models for an Unmanned Aerial Vehicle (UAV) and show how probabilistic model checking and the probabilistic model checker PRISM can be used for optimal controller generation. We introduce a sequence of scenarios relevant to UAVs and other autonomous agents such as underwater and ground vehicles. For each scenario we demonstrate how it can be modelled using the PRISM language, give model checking statistics and present the synthesised optimal controllers. We conclude with a discussion of the limitations when using probabilistic model checking and PRISM in this context and what steps can be taken to overcome them. In addition, we consider how the controllers can be returned to the UAV and adapted for use on larger search areas.

/pdf/strategy-synthesis-for-autonomous-agents-using-prism-4ilnq8jp3m.pdf

Strategy Synthesis for Autonomous Agents Using PRISM

The growth of scale and complexity of interactions between humans and robots highlights the need for new computational methods to automatically evaluate novel algorithms and applications. Exploring diverse scenarios of humans and robots interacting in simulation can improve understanding of the robotic system and avoid potentially costly failures in real-world settings. We formulate this problem as a quality diversity (QD) problem, of which the goal is to discover diverse failure scenarios by simultaneously exploring both environments and human actions. We focus on the shared autonomy domain, in which the robot attempts to infer the goal of a human operator, and adopt the QD algorithms CMA-ME and MAP-Elites to generate scenarios for two published algorithms in this domain: shared autonomy via hindsight optimization and linear policy blending. Some of the generated scenarios confirm previous theoretical findings, while others are surprising and bring about a new understanding of state-of-the-art implementations. Our experiments show that the QD algorithms CMA-ME and MAP-Elites outperform Monte-Carlo simulation and optimization-based methods in effectively searching the scenario space, highlighting their promise for automatic evaluation of algorithms in human–robot interaction.

/pdf/evaluating-human-robot-interaction-algorithms-in-shared-sustiv6h.pdf

Evaluating Human–Robot Interaction Algorithms in Shared Autonomy via Quality Diversity Scenario Generation

An autonomous system has been widely applied for various civil/military research because of its versatile capability of understanding high-level intent and direction of a surrounding environment and targets of interest. However, as autonomous systems can be out of control to cause serious loss, injury, or death in the worst case, the verification of their functionalities has got increasing attention. For that reason, this study is focused on the verification of a heterogeneous multi-agent autonomous system. The thesis first presents an overview of formal methods, especially focuses on model checking for autonomous systems verification. Then, six case studies are presented to verify the decision making behaviours of multi-agent system using two basic scenarios: surveillance and convoy. The initial system considered in the surveillance mission consists of a ground control system and a micro aerial vehicle. Their decision-making behaviours are represented by means of Kripke model and computational tree logic is used to specify the properties of this system. For automatic verification, MCMAS (Model Checker for Multi-Agent Systems) is adopted due to its novel capability to accommodate the multi-agent system. After that, the initial system is extended to include a substitute micro aerial vehicle. These initial case studies are then further extended based on SEAS DTC exemplar 2 dealing with behaviours of convoy protection. This case study includes now a ground control system, an unmanned aerial vehicle, and an unmanned ground vehicle. The MCMAS successfully verifies the targeting behaviours of the team-level unmanned systems. Reversely, these verification results help retrospectively improve the design of decision-making algorithms by considering additional agents and behaviours during four steps of scenario modification. Consequently, the last scenario deals with the system composed of a ground control system, two unmanned aerial vehicles, and four unmanned ground vehicles with fault-tolerant and communications relay capabilities. In conclusion, this study demonstrates the feasibility of model checking algorithms as a verification tool of a multi-agent system in an initial design stage. Moreover, this research can be an important first step of the certification of multi-agent autonomous systems for the domains of robotics, aerospace and aeronautics. To My beloved father, mother, my handsome husband, and my lovely daughter

J. Choi

Papers

Model checking for decision making behaviour of heterogeneous multi-agent autonomous system