Interactive visualization for testing Markov Decision Processes

TLDR: The first visualization targeting MDP testing, MDPvis, is presented and it is shown the visualization's generality by connecting it to two reinforcement learning frameworks that implement many different MDPs of interest in the research community.

Abstract: Markov Decision Processes (MDPs) are a formulation for optimization problems in sequential decision making Solving MDPs often requires implementing a simulator for optimization algorithms to invoke when updating decision making rules known as policies The combination of simulator and optimizer are subject to failures of specification, implementation, integration, and optimization that may produce invalid policies We present these failures as queries for a visual analytic system (MDPVIS) MDPVIS addresses three visualization research gaps First, the data acquisition gap is addressed through a general simulator-visualization interface Second, the data analysis gap is addressed through a generalized MDP information visualization Finally, the cognition gap is addressed by exposing model components to the user MDPVIS generalizes a visualization for wildfire management We use that problem to illustrate MDPVIS and show the visualization's generality by connecting it to two reinforcement learning frameworks that implement many different MDPs of interest in the research community HighlightsMarkov decision processes (MDPs) formalize sequential decision optimization problemsComplex simulators often implement MDPs and are subject to a variety of bugsInteractive visualizations support testing MDPs and optimization algorithmsThe first visualization targeting MDP testing, MDPvis, is presented