scholarly journals Counterexample-Guided Strategy Improvement for POMDPs Using Recurrent Neural Networks

2019 ◽  
Author(s):  
Steven Carr ◽  
Nils Jansen ◽  
Ralf Wimmer ◽  
Alexandru Serban ◽  
Bernd Becker ◽  
...  
Keyword(s):  
Formal Verification ◽  
Temporal Logic ◽  
State Of The Art ◽  
Decision Processes ◽  
Finite Markov Chain ◽  
Markov Decision ◽  
Novel Method ◽  
Finite Memory ◽  
Partially Observable ◽  
Verification Techniques

We study strategy synthesis for partially observable Markov decision processes (POMDPs). The particular problem is to determine strategies that provably adhere to (probabilistic) temporal logic constraints. This problem is computationally intractable and theoretically hard. We propose a novel method that combines techniques from machine learning and formal verification. First, we train a recurrent neural network (RNN) to encode POMDP strategies. The RNN accounts for memory-based decisions without the need to expand the full belief space of a POMDP. Secondly, we restrict the RNN-based strategy to represent a finite-memory strategy and implement it on a specific POMDP. For the resulting finite Markov chain, efficient formal verification techniques provide provable guarantees against temporal logic specifications. If the specification is not satisfied, counterexamples supply diagnostic information. We use this information to improve the strategy by iteratively training the RNN. Numerical experiments show that the proposed method elevates the state of the art in POMDP solving by up to three orders of magnitude in terms of solving times and model sizes.

scholarly journals Detecting Changes and Avoiding Catastrophic Forgetting in Dynamic Partially Observable Environments

2020 ◽  
Vol 14 ◽  
Author(s):  
Jeffery Dick ◽  
Pawel Ladosz ◽  
Eseoghene Ben-Iwhiwhu ◽  
Hideyasu Shimadzu ◽  
Peter Kinnell ◽  
...  
Keyword(s):  
Predictive Models ◽  
Probability Distributions ◽  
Statistical Tests ◽  
Decision Processes ◽  
The Novel ◽  
Predictive State Representations ◽  
Markov Decision ◽  
Novel Method ◽  
Two Phases ◽  
Partially Observable

The ability of an agent to detect changes in an environment is key to successful adaptation. This ability involves at least two phases: learning a model of an environment, and detecting that a change is likely to have occurred when this model is no longer accurate. This task is particularly challenging in partially observable environments, such as those modeled with partially observable Markov decision processes (POMDPs). Some predictive learners are able to infer the state from observations and thus perform better with partial observability. Predictive state representations (PSRs) and neural networks are two such tools that can be trained to predict the probabilities of future observations. However, most such existing methods focus primarily on static problems in which only one environment is learned. In this paper, we propose an algorithm that uses statistical tests to estimate the probability of different predictive models to fit the current environment. We exploit the underlying probability distributions of predictive models to provide a fast and explainable method to assess and justify the model's beliefs about the current environment. Crucially, by doing so, the method can label incoming data as fitting different models, and thus can continuously train separate models in different environments. This new method is shown to prevent catastrophic forgetting when new environments, or tasks, are encountered. The method can also be of use when AI-informed decisions require justifications because its beliefs are based on statistical evidence from observations. We empirically demonstrate the benefit of the novel method with simulations in a set of POMDP environments.

scholarly journals Task-Aware Verifiable RNN-Based Policies for Partially Observable Markov Decision Processes

2021 ◽  
Vol 72 ◽  
pp. 819-847
Author(s):  
Steven Carr ◽  
Nils Jansen ◽  
Ufuk Topcu
Keyword(s):  
Machine Learning ◽  
Markov Chain ◽  
Temporal Logic ◽  
Markov Decision Processes ◽  
Diagnostic Information ◽  
Decision Processes ◽  
Verification Methods ◽  
Markov Decision ◽  
Partially Observable Markov ◽  
Partially Observable

Partially observable Markov decision processes (POMDPs) are models for sequential decision-making under uncertainty and incomplete information. Machine learning methods typically train recurrent neural networks (RNN) as effective representations of POMDP policies that can efficiently process sequential data. However, it is hard to verify whether the POMDP driven by such RNN-based policies satisfies safety constraints, for instance, given by temporal logic specifications. We propose a novel method that combines techniques from machine learning with the field of formal methods: training an RNN-based policy and then automatically extracting a so-called finite-state controller (FSC) from the RNN. Such FSCs offer a convenient way to verify temporal logic constraints. Implemented on a POMDP, they induce a Markov chain, and probabilistic verification methods can efficiently check whether this induced Markov chain satisfies a temporal logic specification. Using such methods, if the Markov chain does not satisfy the specification, a byproduct of verification is diagnostic information about the states in the POMDP that are critical for the specification. The method exploits this diagnostic information to either adjust the complexity of the extracted FSC or improve the policy by performing focused retraining of the RNN. The method synthesizes policies that satisfy temporal logic specifications for POMDPs with up to millions of states, which are three orders of magnitude larger than comparable approaches.

scholarly journals IPOMDP-Net: A Deep Neural Network for Partially Observable Multi-Agent Planning Using Interactive POMDPs

2019 ◽  
Vol 33 ◽  
pp. 6062-6069 ◽  
Author(s):  
Yanlin Han ◽  
Piotr Gmytrasiewicz
Keyword(s):  
Neural Network ◽  
Network Architecture ◽  
State Of The Art ◽  
Neural Computing ◽  
Neural Network Architecture ◽  
Markov Decision ◽  
Planning Algorithm ◽  
Multi Agent ◽  
Partially Observable ◽  
Multi Agent Planning

This paper introduces the IPOMDP-net, a neural network architecture for multi-agent planning under partial observability. It embeds an interactive partially observable Markov decision process (I-POMDP) model and a QMDP planning algorithm that solves the model in a neural network architecture. The IPOMDP-net is fully differentiable and allows for end-to-end training. In the learning phase, we train an IPOMDP-net on various fixed and randomly generated environments in a reinforcement learning setting, assuming observable reinforcements and unknown (randomly initialized) model functions. In the planning phase, we test the trained network on new, unseen variants of the environments under the planning setting, using the trained model to plan without reinforcements. Empirical results show that our model-based IPOMDP-net outperforms the other state-of-the-art modelfree network and generalizes better to larger, unseen environments. Our approach provides a general neural computing architecture for multi-agent planning using I-POMDPs. It suggests that, in a multi-agent setting, having a model of other agents benefits our decision-making, resulting in a policy of higher quality and better generalizability.

Sign in / Sign up

Export Citation Format

Share Document