Verification of general Markov decision processes by approximate similarity relations and policy refinement
Reads0
Chats0
TLDR
It is shown that the new probabilistic similarity relations, inspired by a notion of simulation developed for finite-state models, can be effectively employed over general Markov decision processes for verification purposes, and specifically for control refinement from abstract models.Abstract:
In this work we introduce new approximate similarity relations that are shown to be key for policy (or control) synthesis over general Markov decision processes. The models of interest are discrete-time Markov decision processes, endowed with uncountably infinite state spaces and metric output (or observation) spaces. The new relations, underpinned by the use of metrics, allow, in particular, for a useful trade-off between deviations over probability distributions on states, and distances between model outputs. We show that the new probabilistic similarity relations, inspired by a notion of simulation developed for finite-state models, can be effectively employed over general Markov decision processes for verification purposes, and specifically for control refinement from abstract models.read more
Citations
More filters
Journal ArticleDOI
Compositional Construction of Infinite Abstractions for Networks of Stochastic Control Systems
TL;DR: In this article, a compositional approach for constructing infinite abstractions of interconnected discrete-time stochastic control systems is proposed, which is based on new notions of so-called storage functions.
Journal ArticleDOI
Compositional (In)Finite Abstractions for Large-Scale Interconnected Stochastic Systems
TL;DR: An approach to construct finite MDPs as finite abstractions of concrete models or their reduced-order versions satisfying an incremental input-to-state stability property is proposed and it is shown that for the particular class of nonlinear stochastic control systems, the aforementioned property can be readily checked by matrix inequalities.
Posted Content
Certified Reinforcement Learning with Logic Guidance.
TL;DR: This paper proposes the first model-free Reinforcement Learning (RL) framework to synthesise policies for unknown, and continuous-state Markov Decision Processes (MDPs), such that a given linear temporal property is satisfied.
Proceedings ArticleDOI
Formal Controller Synthesis for Continuous-Space MDPs via Model-Free Reinforcement Learning
TL;DR: A key contribution of the paper is to leverage the classical convergence results for reinforcement learning on finite MDPs and provide control strategies maximizing the probability of satisfaction over unknown, continuous-space MDPS while providing probabilistic closeness guarantees.
Proceedings ArticleDOI
From Dissipativity Theory to Compositional Construction of Finite Markov Decision Processes
TL;DR: In this article, a compositional approach for constructing finite Markov decision processes of interconnected discrete-time stochastic control systems is proposed, which leverages the interconnection topology and a notion of so-called storage functions describing joint dissipativity type properties of subsystems and their abstractions.
References
More filters
Book
Stochastic optimal control : the discrete time case
TL;DR: This research monograph is the authoritative and comprehensive treatment of the mathematical foundations of stochastic optimal control of discrete-time systems, including thetreatment of the intricate measure-theoretic issues.
Book
Lectures on the Coupling Method
TL;DR: In this article, the authors propose Discrete Theory Continuous Theory, Inequalities Intensity-Governed Processes Diffusions Appendix Frequently Used Notation References Index, Section 5.
Journal ArticleDOI
Bisimulation through probabilistic testing
Kim Guldstrand Larsen,Arne Skou +1 more
TL;DR: By using probabilistic transition systems as the underlying semantic model, it is shown how a testing algorithm can distinguish, with a probability arbitrarily close to one, between processes that are not bisimulation equivalent.
Dissertation
Modeling and verification of randomized distributed real-time systems
TL;DR: Theoretical work builds a new mathematical model of randomized distributed computation and develops techniques to prove the correctness of some property by reducing the problem to the verification of properties of non-randomized systems.
Journal ArticleDOI
Probabilistic reachability and safety for controlled discrete time stochastic hybrid systems
TL;DR: In this work, probabilistic reachability over a finite horizon is investigated for a class of discrete time stochastic hybrid systems with control inputs and it is revealed that it is amenable to two complementary interpretations, leading to dual algorithms for reachability computations.