scholarly journals Risk-sensitive inverse reinforcement learning via semi- and non-parametric methods

2018 ◽  
Vol 37 (13-14) ◽  
pp. 1713-1740 ◽  
Author(s):  
Sumeet Singh ◽  
Jonathan Lacotte ◽  
Anirudha Majumdar ◽  
Marco Pavone
Keyword(s):  
Reinforcement Learning ◽  
Cost Function ◽  
Risk Measure ◽  
Inverse Reinforcement Learning ◽  
Worst Case ◽  
Risk Sensitive ◽  
Wide Range ◽  
Risk Neutral ◽  
Parametric Algorithms ◽  
Non Parametric

The literature on inverse reinforcement learning (IRL) typically assumes that humans take actions to minimize the expected value of a cost function, i.e., that humans are risk neutral. Yet, in practice, humans are often far from being risk neutral. To fill this gap, the objective of this paper is to devise a framework for risk-sensitive (RS) IRL to explicitly account for a human’s risk sensitivity. To this end, we propose a flexible class of models based on coherent risk measures, which allow us to capture an entire spectrum of risk preferences from risk neutral to worst case. We propose efficient non-parametric algorithms based on linear programming and semi-parametric algorithms based on maximum likelihood for inferring a human’s underlying risk measure and cost function for a rich class of static and dynamic decision-making settings. The resulting approach is demonstrated on a simulated driving game with 10 human participants. Our method is able to infer and mimic a wide range of qualitatively different driving styles from highly risk averse to risk neutral in a data-efficient manner. Moreover, comparisons of the RS-IRL approach with a risk-neutral model show that the RS-IRL framework more accurately captures observed participant behavior both qualitatively and quantitatively, especially in scenarios where catastrophic outcomes such as collisions can occur.

scholarly journals State-Augmentation Transformations for Risk-Sensitive Reinforcement Learning

2019 ◽  
Vol 33 ◽  
pp. 4512-4519 ◽  
Author(s):  
Shuai Ma ◽  
Jia Yuan Yu
Keyword(s):  
Reinforcement Learning ◽  
Markov Processes ◽  
Value At Risk ◽  
Risk Measure ◽  
Reward Function ◽  
Current State ◽  
Risk Sensitive ◽  
State Augmentation ◽  
Reward Functions ◽  
Markov Reward

In the framework of MDP, although the general reward function takes three arguments—current state, action, and successor state; it is often simplified to a function of two arguments—current state and action. The former is called a transition-based reward function, whereas the latter is called a state-based reward function. When the objective involves the expected total reward only, this simplification works perfectly. However, when the objective is risk-sensitive, this simplification leads to an incorrect value. We propose three successively more general state-augmentation transformations (SATs), which preserve the reward sequences as well as the reward distributions and the optimal policy in risk-sensitive reinforcement learning. In risk-sensitive scenarios, firstly we prove that, for every MDP with a stochastic transition-based reward function, there exists an MDP with a deterministic state-based reward function, such that for any given (randomized) policy for the first MDP, there exists a corresponding policy for the second MDP, such that both Markov reward processes share the same reward sequence. Secondly we illustrate that two situations require the proposed SATs in an inventory control problem. One could be using Q-learning (or other learning methods) on MDPs with transition-based reward functions, and the other could be using methods, which are for the Markov processes with a deterministic state-based reward functions, on the Markov processes with general reward functions. We show the advantage of the SATs by considering Value-at-Risk as an example, which is a risk measure on the reward distribution instead of the measures (such as mean and variance) of the distribution. We illustrate the error in the reward distribution estimation from the reward simplification, and show how the SATs enable a variance formula to work on Markov processes with general reward functions.

Ensemble Inverse Reinforcement Learning from Semi-Expert Agents

2017 ◽  
Vol 137 (4) ◽  
pp. 667-673
Author(s):  
Shinji Tomita ◽  
Fumiya Hamatsu ◽  
Tomoki Hamagami
Keyword(s):  
Reinforcement Learning ◽  
Inverse Reinforcement Learning

Teaching AI Agents Ethical Values Using Reinforcement Learning and Policy Orchestration

2019 ◽  
Author(s):  
Ritesh Noothigattu ◽  
Djallel Bouneffouf ◽  
Nicholas Mattei ◽  
Rachita Chandra ◽  
Piyush Madan ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Ethical Values ◽  
Large Role ◽  
Learning To Learn ◽  
Inverse Reinforcement Learning ◽  
Time Step ◽  
Novel Approach

Autonomous cyber-physical agents play an increasingly large role in our lives. To ensure that they behave in ways aligned with the values of society, we must develop techniques that allow these agents to not only maximize their reward in an environment, but also to learn and follow the implicit constraints of society. We detail a novel approach that uses inverse reinforcement learning to learn a set of unspecified constraints from demonstrations and reinforcement learning to learn to maximize environmental rewards. A contextual bandit-based orchestrator then picks between the two policies: constraint-based and environment reward-based. The contextual bandit orchestrator allows the agent to mix policies in novel ways, taking the best actions from either a reward-maximizing or constrained policy. In addition, the orchestrator is transparent on which policy is being employed at each time step. We test our algorithms using Pac-Man and show that the agent is able to learn to act optimally, act within the demonstrated constraints, and mix these two functions in complex ways.

Portfolio Return, Risk and Market Timing: A Non-Parametric Approach

2015 ◽  
Vol 5 (2) ◽  
Author(s):  
Joyjit Dhar ◽  
Ram Pratap Sinha
Keyword(s):  
Risk Measure ◽  
Market Timing ◽  
Evaluation Framework ◽  
Data Envelopment ◽  
Portfolio Evaluation ◽  
Period 2 ◽  
Measure Of Performance ◽  
Standard Risk ◽  
Portfolio Return ◽  
Non Parametric

The present study extends the portfolio evaluation framework provided by Sharpe (1964) and Treynor (1965) by including the parameter of market timing with the help of a non-parametric framework. Data envelopment analysis has been used in the present exercise to evaluate the performance 79 mutual funds schemes operating in India for three different phases using two different models. Estimation of technical efficiency on the basis of both the models suggests that period 2 performance is substantially divergent from period 1 and 3. Also, higher moments framework gives a better measure of performance as it accounts not only the standard risk measure but also for skewness and kurtosis characteristics of returns.

scholarly journals Inverse reinforcement learning in contextual MDPs

2021 ◽  
Author(s):  
Stav Belogolovsky ◽  
Philip Korsunsky ◽  
Shie Mannor ◽  
Chen Tessler ◽  
Tom Zahavy
Keyword(s):  
Reinforcement Learning ◽  
Optimization Problem ◽  
Decision Processes ◽  
Inverse Reinforcement Learning ◽  
Convex Optimization Problem ◽  
Reward Function ◽  
Dynamic Treatment Regime ◽  
Markov Decision ◽  
Dynamic Treatment ◽  
Recorded Data

AbstractWe consider the task of Inverse Reinforcement Learning in Contextual Markov Decision Processes (MDPs). In this setting, contexts, which define the reward and transition kernel, are sampled from a distribution. In addition, although the reward is a function of the context, it is not provided to the agent. Instead, the agent observes demonstrations from an optimal policy. The goal is to learn the reward mapping, such that the agent will act optimally even when encountering previously unseen contexts, also known as zero-shot transfer. We formulate this problem as a non-differential convex optimization problem and propose a novel algorithm to compute its subgradients. Based on this scheme, we analyze several methods both theoretically, where we compare the sample complexity and scalability, and empirically. Most importantly, we show both theoretically and empirically that our algorithms perform zero-shot transfer (generalize to new and unseen contexts). Specifically, we present empirical experiments in a dynamic treatment regime, where the goal is to learn a reward function which explains the behavior of expert physicians based on recorded data of them treating patients diagnosed with sepsis.

scholarly journals Stackelberg pricing games with congestion effects

2021 ◽  
Author(s):  
Tobias Harks ◽  
Anja Schedel
Keyword(s):  
Cost Function ◽  
Stackelberg Game ◽  
Pure Strategy ◽  
Closed Form Expression ◽  
Social Optimum ◽  
Service Capacity ◽  
Worst Case ◽  
Congestion Cost ◽  
Pricing Games ◽  
Congestion Effects

AbstractWe study a Stackelberg game with multiple leaders and a continuum of followers that are coupled via congestion effects. The followers’ problem constitutes a nonatomic congestion game, where a population of infinitesimal players is given and each player chooses a resource. Each resource has a linear cost function which depends on the congestion of this resource. The leaders of the Stackelberg game each control a resource and determine a price per unit as well as a service capacity for the resource influencing the slope of the linear congestion cost function. As our main result, we establish existence of pure-strategy Nash–Stackelberg equilibria for this multi-leader Stackelberg game. The existence result requires a completely new proof approach compared to previous approaches, since the leaders’ objective functions are discontinuous in our game. As a consequence, best responses of leaders do not always exist, and thus standard fixed-point arguments á la Kakutani (Duke Math J 8(3):457–458, 1941) are not directly applicable. We show that the game is C-secure (a concept introduced by Reny (Econometrica 67(5):1029–1056, 1999) and refined by McLennan et al. (Econometrica 79(5):1643–1664, 2011), which leads to the existence of an equilibrium. We furthermore show that the equilibrium is essentially unique, and analyze its efficiency compared to a social optimum. We prove that the worst-case quality is unbounded. For identical leaders, we derive a closed-form expression for the efficiency of the equilibrium.

scholarly journals Dealing with multiple experts and non-stationarity in inverse reinforcement learning: an application to real-life problems

2021 ◽  
Author(s):  
Amarildo Likmeta ◽  
Alberto Maria Metelli ◽  
Giorgia Ramponi ◽  
Andrea Tirinzoni ◽  
Matteo Giuliani ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Real World ◽  
Real Life ◽  
User Preferences ◽  
Inverse Reinforcement Learning ◽  
Water Release ◽  
Reward Function ◽  
Model Free ◽  
Conflicting Objectives ◽  
Multiple Experts

AbstractIn real-world applications, inferring the intentions of expert agents (e.g., human operators) can be fundamental to understand how possibly conflicting objectives are managed, helping to interpret the demonstrated behavior. In this paper, we discuss how inverse reinforcement learning (IRL) can be employed to retrieve the reward function implicitly optimized by expert agents acting in real applications. Scaling IRL to real-world cases has proved challenging as typically only a fixed dataset of demonstrations is available and further interactions with the environment are not allowed. For this reason, we resort to a class of truly batch model-free IRL algorithms and we present three application scenarios: (1) the high-level decision-making problem in the highway driving scenario, and (2) inferring the user preferences in a social network (Twitter), and (3) the management of the water release in the Como Lake. For each of these scenarios, we provide formalization, experiments and a discussion to interpret the obtained results.

scholarly journals A Reinforcement Learning Framework for Spiking Networks with Dynamic Synapses

2011 ◽  
Vol 2011 ◽  
pp. 1-12 ◽  
Author(s):  
Karim El-Laithy ◽  
Martin Bogdan
Keyword(s):  
Reinforcement Learning ◽  
Spike Timing ◽  
Neural Representation ◽  
Model Parameters ◽  
Learning Framework ◽  
Reference Target ◽  
Wide Range ◽  
Spiking Network ◽  
Dynamic Synapses ◽  
Exclusive Or

An integration of both the Hebbian-based and reinforcement learning (RL) rules is presented for dynamic synapses. The proposed framework permits the Hebbian rule to update the hidden synaptic model parameters regulating the synaptic response rather than the synaptic weights. This is performed using both the value and the sign of the temporal difference in the reward signal after each trial. Applying this framework, a spiking network with spike-timing-dependent synapses is tested to learn the exclusive-OR computation on a temporally coded basis. Reward values are calculated with the distance between the output spike train of the network and a reference target one. Results show that the network is able to capture the required dynamics and that the proposed framework can reveal indeed an integrated version of Hebbian and RL. The proposed framework is tractable and less computationally expensive. The framework is applicable to a wide class of synaptic models and is not restricted to the used neural representation. This generality, along with the reported results, supports adopting the introduced approach to benefit from the biologically plausible synaptic models in a wide range of intuitive signal processing.

Sign in / Sign up

Export Citation Format

Share Document