scholarly journals State Abstraction as Compression in Apprenticeship Learning

2019 ◽  
Vol 33 ◽  
pp. 3134-3142
Author(s):  
David Abel ◽  
Dilip Arumugam ◽  
Kavosh Asadi ◽  
Yuu Jinnai ◽  
Michael L. Littman ◽  
...  
Keyword(s):  
Rate Distortion ◽  
Sequential Decision Making ◽  
Sequential Decision ◽  
Rate Distortion Theory ◽  
Apprenticeship Learning ◽  
Information Bottleneck ◽  
State Abstraction ◽  
And Performance ◽  
Algorithmic Structure ◽  
Made In

State abstraction can give rise to models of environments that are both compressed and useful, thereby enabling efficient sequential decision making. In this work, we offer the first formalism and analysis of the trade-off between compression and performance made in the context of state abstraction for Apprenticeship Learning. We build on Rate-Distortion theory, the classic Blahut-Arimoto algorithm, and the Information Bottleneck method to develop an algorithm for computing state abstractions that approximate the optimal tradeoff between compression and performance. We illustrate the power of this algorithmic structure to offer insights into effective abstraction, compression, and reinforcement learning through a mixture of analysis, visuals, and experimentation.

scholarly journals Markov Information Bottleneck to Improve Information Flow in Stochastic Neural Networks

Entropy ◽  
2019 ◽  
Vol 21 (10) ◽  
pp. 976 ◽  
Author(s):  
Thanh Tang Nguyen ◽  
Jaesik Choi
Keyword(s):  
Neural Networks ◽  
Information Flow ◽  
Latent Variables ◽  
Rate Distortion ◽  
Relevant Information ◽  
Variational Approximation ◽  
Stochastic Neural Networks ◽  
Rate Distortion Theory ◽  
Information Bottleneck ◽  
Hidden Layer

While rate distortion theory compresses data under a distortion constraint, information bottleneck (IB) generalizes rate distortion theory to learning problems by replacing a distortion constraint with a constraint of relevant information. In this work, we further extend IB to multiple Markov bottlenecks (i.e., latent variables that form a Markov chain), namely Markov information bottleneck (MIB), which particularly fits better in the context of stochastic neural networks (SNNs) than the original IB. We show that Markov bottlenecks cannot simultaneously achieve their information optimality in a non-collapse MIB, and thus devise an optimality compromise. With MIB, we take the novel perspective that each layer of an SNN is a bottleneck whose learning goal is to encode relevant information in a compressed form from the data. The inference from a hidden layer to the output layer is then interpreted as a variational approximation to the layer’s decoding of relevant information in the MIB. As a consequence of this perspective, the maximum likelihood estimate (MLE) principle in the context of SNNs becomes a special case of the variational MIB. We show that, compared to MLE, the variational MIB can encourage better information flow in SNNs in both principle and practice, and empirically improve performance in classification, adversarial robustness, and multi-modal learning in MNIST.

scholarly journals Optimal Policies for Quantum Markov Decision Processes

2021 ◽  
Author(s):  
Ming-Sheng Ying ◽  
Yuan Feng ◽  
Sheng-Gang Ying
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Quantum Systems ◽  
Sequential Decision Making ◽  
Mathematical Framework ◽  
Sequential Decision ◽  
Learning Techniques ◽  
Optimal Policies ◽  
Markov Decision ◽  
Programming Algorithms

AbstractMarkov decision process (MDP) offers a general framework for modelling sequential decision making where outcomes are random. In particular, it serves as a mathematical framework for reinforcement learning. This paper introduces an extension of MDP, namely quantum MDP (qMDP), that can serve as a mathematical model of decision making about quantum systems. We develop dynamic programming algorithms for policy evaluation and finding optimal policies for qMDPs in the case of finite-horizon. The results obtained in this paper provide some useful mathematical tools for reinforcement learning techniques applied to the quantum world.

Sign in / Sign up

Export Citation Format

Share Document