This letter investigates the motion planning of autonomous dynamical systems modeled by Markov decision processes (MDP) with unknown transition probabilities over continuous state and action spaces. Linear temporal logic (LTL) is used to specify high-level tasks over infinite horizon, which can be converted into a limit deterministic generalized Buchi automaton (LDGBA) with several accepting sets. The novelty is to design an embedded product MDP (EP-MDP) between the LDGBA and the MDP by incorporating a synchronous tracking-frontier function to record unvisited accepting sets of the automaton, and to facilitate the satisfaction of the accepting conditions. The proposed LDGBA-based reward shaping and discounting schemes for the model-free reinforcement learning (RL) only depend on the EP-MDP states and can overcome the issues of sparse rewards. Rigorous analysis shows that any RL method that optimizes the expected discounted return is guaranteed to find an optimal policy whose traces maximize the satisfaction probability. A modular deep deterministic policy gradient (DDPG) is then developed to generate such policies over continuous state and action spaces. The performance of our framework is evaluated via an array of OpenAI gym environments.

Modular Deep Reinforcement Learning for Continuous Motion Planning With Temporal Logic

This paper investigates the motion planning of autonomous dynamical systems modeled by Markov decision processes (MDP) with unknown transition probabilities over continuous state and action spaces. Linear temporal logic (LTL) is used to specify high-level tasks over infinite horizon, which can be converted into a limit deterministic generalized Buchi automaton (LDGBA) with several accepting sets. The novelty is to design an embedded product MDP (EP-MDP) between the LDGBA and the MDP by incorporating a synchronous tracking-frontier function to record unvisited accepting sets of the automaton, and to facilitate the satisfaction of the accepting conditions. The proposed LDGBA-based reward shaping and discounting schemes for the model-free reinforcement learning (RL) only depend on the EP-MDP states and can overcome the issues of sparse rewards. Rigorous analysis shows that any RL method that optimizes the expected discounted return is guaranteed to find an optimal policy whose traces maximize the satisfaction probability. A modular deep deterministic policy gradient (DDPG) is then developed to generate such policies over continuous state and action spaces. The performance of our framework is evaluated via an array of OpenAI gym environments.

Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logic

The combination of data-driven learning methods with formal reasoning has seen a surge of interest, as either area has the potential to bolstering the other. For instance, formal methods promise to expand the use of state-of-the-art learning approaches in the direction of certification and sample efficiency. In this work, we propose a deep Reinforcement Learning (RL) method for policy synthesis in continuous-state/action unknown environments, under requirements expressed in Linear Temporal Logic (LTL). We show that this combination lifts the applicability of deep RL to complex temporal and memory-dependent policy synthesis goals. We express an LTL specification as a Limit Deterministic Buchi Automaton (LDBA) and synchronise it on-the-fly with the agent/environment. The LDBA in practice monitors the environment, acting as a modular reward machine for the agent: accordingly, a modular Deep Deterministic Policy Gradient (DDPG) architecture is proposed to generate a low-level control policy that maximises the probability of the given LTL formula. We evaluate our framework in a cart-pole example and in a Mars rover experiment, where we achieve near-perfect success rates, while baselines based on standard RL are shown to fail in practice.

Deep Reinforcement Learning with Temporal Logics

We may not be able to make you love reading, but tools and algorithms for the construction and analysis of systems third international workshop tacas 97 enschede the netherlands april 2 4 1997 proceedings will lead you to love reading starting from now. Book is the window to open the new world. The world that you want is in the better stage and level. World will always guide you to even the prestige stage of the life. You know, this is some of how reading will give you the kindness. In this case, more books you read more knowledge you know, but it can mean also the bore is full.

Proceedings of the 16th international conference on Tools and Algorithms for the Construction and Analysis of Systems

/pdf/secure-by-construction-synthesis-of-cyber-physical-systems-2utbr0zu.pdf

Secure-by-Construction Synthesis of Cyber-Physical Systems

This letter proposes a novel reinforcement learning method for the synthesis of a control policy satisfying a control specification described by a linear temporal logic formula. We assume that the controlled system is modeled by a Markov decision process (MDP). We convert the specification to a limit-deterministic generalized Buchi automaton (LDGBA) with several accepting sets that accepts all infinite sequences satisfying the formula. The LDGBA is augmented so that it explicitly records the previous visits to accepting sets. We take a product of the augmented LDGBA and the MDP, based on which we define a reward function. The agent gets rewards whenever state transitions are in an accepting set that has not been visited for a certain number of steps. Consequently, sparsity of rewards is relaxed and optimal circulations among the accepting sets are learned. We show that the proposed method can learn an optimal policy when the discount factor is sufficiently close to one.

Reinforcement Learning of Control Policy for Linear Temporal Logic Specifications Using Limit-Deterministic Generalized Büchi Automata

We propose an on-line supervisory control scheme for discrete event systems (DESs), where a control specification is described by a fragment of linear temporal logic. On the product automaton of the DES and an acceptor for the specification, we define a ranking function that returns the minimum number of steps required to reach an accepting state from each state. In addition, we introduce a permissiveness function that indicates a time-varying permissive level. At each step during the on-line control scheme, the supervisor refers to the permissiveness function as well as the ranking function in order to guarantee the control specification while handling the tradeoff between its permissiveness and acceptance of the specification. The proposed scheme is demonstrated in a surveillance problem for a mobile robot.

On-Line Permissive Supervisory Control of Discrete Event Systems for scLTL Specifications

Decentralized Supervision and Coordination of Concurrent Discrete Event Systems under LTL Constraints

Hierarchical Control of Concurrent Discrete Event Systems with Linear Temporal Logic Specifications

Khufu's Pyramid is one of the largest archaeological monument all over the world, which still holds many mysteries. In 2016 and 2017, the ScanPyramids team reported on several discoveries of previously unknown voids by cosmic-ray muon radiography that is a non-destructive technique ideal for the investigation of large-scale structures. Among these discoveries, a corridor-shaped structure has been observed behind the so-called Chevron zone on the North face, with a length of at least 5 meters. A dedicated study of this structure was thus necessary to better understand its function in relation with the enigmatic architectural role of this Chevron. Here we report on new measurements of excellent sensitivity obtained with nuclear emulsion films from Nagoya University and gaseous detectors from CEA, revealing a structure of about 9 m length with a transverse section of about 2.0 m by 2.0 m. 

/pdf/precise-characterization-of-a-corridor-shaped-structure-in-2aupe5jo.pdf

Ami Sakakibara

Papers

Reinforcement Learning of Control Policy for Linear Temporal Logic Specifications Using Limit-Deterministic Generalized Büchi Automata

On-Line Permissive Supervisory Control of Discrete Event Systems for scLTL Specifications

Decentralized Supervision and Coordination of Concurrent Discrete Event Systems under LTL Constraints

Hierarchical Control of Concurrent Discrete Event Systems with Linear Temporal Logic Specifications

Precise characterization of a corridor-shaped structure in Khufu’s Pyramid by observation of cosmic-ray muons