scispace - formally typeset
Open AccessJournal ArticleDOI

The complexity of reachability in parametric Markov decision processes

Sebastian Junges, +3 more
- 01 Aug 2021 - 
- Vol. 119, pp 183-210
Reads0
Chats0
TLDR
The complexity of reachability decision problems for parametric Markov decision processes (pMDPs) is presented, and all known lower bounds are improved, and ETR-completeness results for distinct variants of this problem are provided.
About: 
This article is published in Journal of Computer and System Sciences.The article was published on 2021-08-01 and is currently open access. It has received 9 citations till now. The article focuses on the topics: Markov decision process & Decision problem.

read more

Figures
Citations
More filters
Proceedings ArticleDOI

On the Complexity of Reachability in Parametric Markov Decision Processes

Tobias Winkler, +3 more
TL;DR: In particular, the authors studies the complexity of finding values for these parameters such that the induced MDP satisfies some reachability constraints, and provides ETR-completeness results for distinct variants of this problem.
Book ChapterDOI

Fine-Tuning the Odds in Bayesian Networks

Bahare Salmani, +1 more
TL;DR: In this article, the authors propose new analysis techniques for Bayes networks in which conditional probability tables (CPTs) may contain symbolic variables, which are applicable to arbitrarily many, possibly dependent, parameters that may occur in multiple CPTs.
Book ChapterDOI

Gradient-Descent for Randomized Controllers Under Partial Observability

Book ChapterDOI

Tweaking the Odds in Probabilistic Timed Automata

Arnd Hartmanns, +4 more
TL;DR: In this paper, the authors prove that existing techniques to transform probabilistic timed automata into equivalent finite-state Markov decision processes (MDPs) remain correct in the parametric setting, using a systematic proof pattern.
Posted Content

Probabilistic Timed Automata with One Clock and Initialised Clock-Dependent Probabilities

Jeremy Sproston
- 07 Jun 2020 - 
TL;DR: In this paper, the authors consider the subclass of clock-dependent probabilistic timed automata that have one clock, that have clock dependencies described by affine functions, and satisfy an initialisation condition requiring that, at some point between taking edges with non-trivial clock dependencies, the clock must have an integer value.
References
More filters
Book

Artificial Intelligence: A Modern Approach

Stuart Russell, +1 more
TL;DR: In this article, the authors present a comprehensive introduction to the theory and practice of artificial intelligence for modern applications, including game playing, planning and acting, and reinforcement learning with neural networks.
Book

Markov Decision Processes: Discrete Stochastic Dynamic Programming

Martin L. Puterman
TL;DR: Puterman as discussed by the authors provides a uniquely up-to-date, unified, and rigorous treatment of the theoretical, computational, and applied research on Markov decision process models, focusing primarily on infinite horizon discrete time models and models with discrete time spaces while also examining models with arbitrary state spaces, finite horizon models, and continuous time discrete state models.
MonographDOI

Markov Decision Processes

P. Whittle, +1 more
- 15 Apr 1994 - 
TL;DR: Markov Decision Processes covers recent research advances in such areas as countable state space models with average reward criterion, constrained models, and models with risk sensitive optimality criteria, and explores several topics that have received little or no attention in other books.
Book

Principles of Model Checking

Christel Baier, +1 more
TL;DR: Principles of Model Checking offers a comprehensive introduction to model checking that is not only a text suitable for classroom use but also a valuable reference for researchers and practitioners in the field.
Book ChapterDOI

PRISM 4.0: verification of probabilistic real-time systems

Marta Kwiatkowska, +2 more
TL;DR: A major new release of the PRISMprobabilistic model checker is described, adding, in particular, quantitative verification of (priced) probabilistic timed automata.
Related Papers (5)

On the Complexity of Reachability in Parametric Markov Decision Processes

Tobias Winkler, +3 more
- 02 Apr 2019 - 

Probabilistic Reachability for Parametric Markov Models

Ernst Moritz Hahn, +2 more

Quantitative model-checking of controlled discrete-time Markov processes

Ilya Tkachev, +3 more
- 21 Jul 2014 - 

Controller Synthesis and Verification for Markov Decision Processes with Qualitative Branching Time Objectives

Tomáš Brázdil, +2 more

Symbolic Approximation of the Bounded Reachability Probability in Large Markov Chains

Markus N. Rabe, +4 more
Frequently Asked Questions (11)
Q1. What are the contributions in "On the complexity of reachability in parametric markov decision processes" ?

This paper studies parametric Markov decision processes ( pMDPs ), an extension to Markov decision processes ( MDPs ) where transitions probabilities are described by polynomials over a finite set of parameters. In particular, this paper studies the complexity of finding values for these parameters such that the induced MDP satisfies some reachability constraints. The authors discuss different variants depending on the comparison operator in the constraints and the domain of the parameter values. The authors improve all known lower bounds for this problem, and notably provide ETR-completeness results for distinct variants of this problem. Furthermore, the authors provide insights in the functions describing the induced reachability probabilities, and how pMDPs generalise concurrent stochastic reachability games. 

The authors focus ourselves on graph-preserving instantiations, as the analysis of pMDP M and PwdM corresponds to analysing constantly many pMDPsM′ on PgpM′ , cf. Rem. 

Its transition probability function PM is obtained by lettingPM(s, b, s′) = ∑ a∈As σ(a|s)P (s′|s, a, b) (6)for all s, s′ ∈ S and actions b ∈ Bs of player II. 

In the non-parametric case, a scheduler σ of an MDP is calledminimal if it minimises Prσ(♦T ), i.e. if σ ∈ argminσ′∈Σ Prσ ′ (♦T ). 

Essentially the polynomial f in mb4FEAS is constructed by taking the sum-of-squares of the quadratic polynomials, and further operations are adequatly shifting the polynomial. 

Robust strategies have been widely studied in the field of operations research (see, e.g., [40, 57]) and are the main focus of reinforcement learning [55]. 

The game then picks a successor state s′ according to a fixed probability distribution P (·|s, a, b) over S, and the play continues in s′. 

In particular, deciding membership for sentences with an a-priori fixed upper bound on the number of variables is in polynomial time. 

The authors denote the set of randomised schedulers with RΣ, and (deterministic) schedulers with Σ. For pMDP M = (S,X,Act, sι, P ) and σ ∈ RΣM, the induced pMC Mσ is defined as(S,X, sι, P ′) with P ′(s, s′) = ∑ a∈Act σ(s)(a) · P (s, a, s′). 

ETR denotes the complexity class [48] of problems with a polynomial-time many-one reduction to deciding membership in the existential theory of the reals. 

The gadget in Fig. 3 ensures that for any graph non-preserving instantiation, the probability to reach the target is 0, while it does not affect reachability probabilities for graph-preserving instantiations.