After-Action Review for AI (AAR/AI)

2021 ◽  
Vol 11 (3-4) ◽  
pp. 1-35
Author(s):  
Jonathan Dodge ◽  
Roli Khanna ◽  
Jed Irvine ◽  
Kin-ho Lam ◽  
Theresa Mai ◽  
...  
Keyword(s):  
Ad Hoc ◽  
Modern Society ◽  
Sequential Decision ◽  
Military Organizations ◽  
Learning Agents ◽  
Model Based ◽  
After Action Review ◽  
Model Free ◽  
Wide Range ◽  
Explainable Ai

Explainable AI is growing in importance as AI pervades modern society, but few have studied how explainable AI can directly support people trying to assess an AI agent. Without a rigorous process, people may approach assessment in ad hoc ways—leading to the possibility of wide variations in assessment of the same agent due only to variations in their processes. AAR, or After-Action Review, is a method some military organizations use to assess human agents, and it has been validated in many domains. Drawing upon this strategy, we derived an After-Action Review for AI (AAR/AI), to organize ways people assess reinforcement learning agents in a sequential decision-making environment. We then investigated what AAR/AI brought to human assessors in two qualitative studies. The first investigated AAR/AI to gather formative information, and the second built upon the results, and also varied the type of explanation (model-free vs. model-based) used in the AAR/AI process. Among the results were the following: (1) participants reporting that AAR/AI helped to organize their thoughts and think logically about the agent, (2) AAR/AI encouraged participants to reason about the agent from a wide range of perspectives , and (3) participants were able to leverage AAR/AI with the model-based explanations to falsify the agent’s predictions.

scholarly journals Increased temporal discounting and reduced model-based control in problem gambling are not substantially modulated by exposure to virtual gambling environments.

2021 ◽  
Author(s):  
Luca Rene Bruder ◽  
Ben Wagner ◽  
David Mathar ◽  
Jan Peters
Keyword(s):  
Pathological Gambling ◽  
Electrodermal Activity ◽  
Temporal Discounting ◽  
Sports Betting ◽  
Reduced Model ◽  
Sequential Decision ◽  
Model Based Control ◽  
Model Based ◽  
Model Free ◽  
Environment Exposure

High-performance virtual reality (VR) technology has opened new possibilities for the examination of the reactivity towards addiction-related cues (cue-reactivity) in addiction. In this preregistered study (https://osf.io/4mrta), we investigated the subjective, physiological, and behavioral effects of gambling-related VR environment exposure in participants reporting frequent or pathological gambling (n=31) as well as non-gambling controls (n=29). On two separate days, participants explored two rich and navigable VR-environments (neutral: cafe vs. gambling-related: casino/sports-betting facility), while electrodermal activity and heart rate were continuously measured using remote sensors. Within VR, participants performed a temporal discounting task and a sequential decision-making task designed to assess model-based and model-free contributions to behavior. Replicating previous findings, we found strong evidence for increased temporal discounting and reduced model-based control in participants reporting frequent or pathological gambling. Although VR gambling environment exposure increased subjective craving, there was if anything inconclusive evidence for further behavioral or physiological effects. Instead, VR exposure substantially increased physiological arousal (electrodermal activity), across groups and conditions. VR is a promising tool for the investigation of context effects in addiction, but some caution is warranted since effects of real gambling environments might not generally replicate in VR. Future studies should delineate how factors such as cognitive load and ecological validity could be balanced to create a more naturalistic VR experience.

scholarly journals Ventral striatal dopamine reflects behavioral and neural signatures of model-based control during sequential decision making

2015 ◽  
Vol 112 (5) ◽  
pp. 1595-1600 ◽  
Author(s):  
Lorenz Deserno ◽  
Quentin J. M. Huys ◽  
Rebecca Boehme ◽  
Ralph Buchert ◽  
Hans-Jochen Heinze ◽  
...  
Keyword(s):  
Ventral Striatum ◽  
Behavioral Control ◽  
Striatal Dopamine ◽  
Quantitative Relation ◽  
Sequential Decision ◽  
Model Based Control ◽  
Model Based ◽  
Model Free ◽  
Neuropsychiatric Diseases ◽  
Neural Signatures

Dual system theories suggest that behavioral control is parsed between a deliberative “model-based” and a more reflexive “model-free” system. A balance of control exerted by these systems is thought to be related to dopamine neurotransmission. However, in the absence of direct measures of human dopamine, it remains unknown whether this reflects a quantitative relation with dopamine either in the striatum or other brain areas. Using a sequential decision task performed during functional magnetic resonance imaging, combined with striatal measures of dopamine using [18F]DOPA positron emission tomography, we show that higher presynaptic ventral striatal dopamine levels were associated with a behavioral bias toward more model-based control. Higher presynaptic dopamine in ventral striatum was associated with greater coding of model-based signatures in lateral prefrontal cortex and diminished coding of model-free prediction errors in ventral striatum. Thus, interindividual variability in ventral striatal presynaptic dopamine reflects a balance in the behavioral expression and the neural signatures of model-free and model-based control. Our data provide a novel perspective on how alterations in presynaptic dopamine levels might be accompanied by a disruption of behavioral control as observed in aging or neuropsychiatric diseases such as schizophrenia and addiction.

scholarly journals Hunger improves reinforcement-driven but not planned action

2021 ◽  
Author(s):  
Maaike M.H. van Swieten ◽  
Rafal Bogacz ◽  
Sanjay G. Manohar
Keyword(s):  
Decision Making ◽  
Sequential Decision ◽  
Context Sensitive ◽  
Decision Systems ◽  
Model Based ◽  
Model Free ◽  
Prevalence Of Obesity ◽  
Model Free Control ◽  
Decision Mode

AbstractHuman decisions can be reflexive or planned, being governed respectively by model-free and model-based learning systems. These two systems might differ in their responsiveness to our needs. Hunger drives us to specifically seek food rewards, but here we ask whether it might have more general effects on these two decision systems. On one hand, the model-based system is often considered flexible and context-sensitive, and might therefore be modulated by metabolic needs. On the other hand, the model-free system’s primitive reinforcement mechanisms may have closer ties to biological drives. Here, we tested participants on a well-established two-stage sequential decision-making task that dissociates the contribution of model-based and model-free control. Hunger enhanced overall performance by increasing model-free control, without affecting model-based control. These results demonstrate a generalised effect of hunger on decision-making that enhances reliance on primitive reinforcement learning, which in some situations translates into adaptive benefits.Significance statementThe prevalence of obesity and eating disorder is steadily increasing. To counteract problems related to eating, people need to make rational decisions. However, appetite may switch us to a different decision mode, making it harder to achieve long-term goals. Here we show that planned and reinforcement-driven actions are differentially sensitive to hunger. Hunger specifically affected reinforcement-driven actions, and did not affect the planning of actions. Our data shows that people behave differently when they are hungry. We also provide a computational model of how the behavioural changes might arise.

Model Based or Model Free? Comparing Adaptive Methods for Estimating Thresholds in Neuroscience

2021 ◽  
pp. 1-22
Author(s):  
Julien Audiffren ◽  
Jean-Pierre Bresciani
Keyword(s):  
Psychometric Function ◽  
Cumulative Distribution Function ◽  
Human Perception ◽  
Adaptive Methods ◽  
Cumulative Distribution ◽  
Function Model ◽  
Model Based ◽  
Model Free ◽  
Adaptive Procedures ◽  
Wide Range

The quantification of human perception through the study of psychometric functions Ψ is one of the pillars of experimental psychophysics. In particular, the evaluation of the threshold is at the heart of many neuroscience and cognitive psychology studies, and a wide range of adaptive procedures has been developed to improve its estimation. However, these procedures are often implicitly based on different mathematical assumptions on the psychometric function, and unfortunately, these assumptions cannot always be validated prior to data collection. This raises questions about the accuracy of the estimator produced using the different procedures. In the study we examine in this letter, we compare five adaptive procedures commonly used in psychophysics to estimate the threshold: Dichotomous Optimistic Search (DOS), Staircase, PsiMethod, Gaussian Processes, and QuestPlus. These procedures range from model-based methods, such as the PsiMethod, which relies on strong assumptions regarding the shape of Ψ, to model-free methods, such as DOS, for which assumptions are minimal. The comparisons are performed using simulations of multiple experiments, with psychometric functions of various complexity. The results show that while model-based methods perform well when Ψ is an ideal psychometric function, model-free methods rapidly outshine them when Ψ deviates from this model, as, for instance, when Ψ is a beta cumulative distribution function. Our results highlight the importance of carefully choosing the most appropriate method depending on the context.

scholarly journals Metacontrol of decision-making strategies in human aging

2019 ◽  
Author(s):  
Florian Bolenz ◽  
Wouter Kool ◽  
Andrea M.F. Reiter ◽  
Ben Eppinger
Keyword(s):  
Older Adults ◽  
Decision Making ◽  
Cognitive Abilities ◽  
Reduced Model ◽  
Task Structure ◽  
Costs And Benefits ◽  
Sequential Decision ◽  
Model Based Control ◽  
Model Based ◽  
Model Free

When making decisions, humans employ different strategies which are commonly formalized as model-free and model-based reinforcement learning. While previous research has reported reduced model-based control with aging, it remains unclear whether this is due to limited cognitive capacities or a reduced willingness to engage in an effortful strategy. Moreover, it is not clear how aging affects the metacontrol of decision making, i.e. the dynamic adaptation of decision-making strategies to varying situational demands. To this end, we tested younger and older adults in a sequential decision-making task that dissociates model-free and model-based control. In contrast to previous research, in this study we applied a task in which model-based control led to higher payoffs in terms of monetary reward. Moreover, we manipulated the costs and benefits associated with model-based control by varying reward magnitude as well as the stability of the task structure. Compared to younger adults, older adults showed reduced reliance on model-based decision making and less adaptation of decision-making strategies to varying costs and benefits of model-based control. Our findings suggest that aging affects the dynamic metacontrol of decision-making strategies and that reduced model-based control in older adults is due to limited cognitive abilities to represent the structure of the task.

scholarly journals Metacontrol of decision-making strategies in human aging

eLife ◽  
2019 ◽  
Vol 8 ◽  
Author(s):  
Florian Bolenz ◽  
Wouter Kool ◽  
Andrea MF Reiter ◽  
Ben Eppinger
Keyword(s):  
Older Adults ◽  
Decision Making ◽  
Cognitive Abilities ◽  
Cross Sectional Study ◽  
Reduced Model ◽  
Task Structure ◽  
Sequential Decision ◽  
Cross Sectional ◽  
Model Based ◽  
Model Free

Humans employ different strategies when making decisions. Previous research has reported reduced reliance on model-based strategies with aging, but it remains unclear whether this is due to cognitive or motivational factors. Moreover, it is not clear how aging affects the metacontrol of decision making, that is the dynamic adaptation of decision-making strategies to varying situational demands. In this cross-sectional study, we tested younger and older adults in a sequential decision-making task that dissociates model-free and model-based strategies. In contrast to previous research, model-based strategies led to higher payoffs. Moreover, we manipulated the costs and benefits of model-based strategies by varying reward magnitude and the stability of the task structure. Compared to younger adults, older adults showed reduced model-based decision making and less adaptation of decision-making strategies. Our findings suggest that aging affects the metacontrol of decision-making strategies and that reduced model-based strategies in older adults are due to limited cognitive abilities.

scholarly journals Identifying Model-Based and Model-Free Patterns in Behavior on Multi-Step Tasks

2016 ◽  
Author(s):  
Kevin J. Miller ◽  
Carlos D. Brody ◽  
Matthew M. Botvinick
Keyword(s):  
Model Fitting ◽  
Decision Task ◽  
Sequential Decision ◽  
Agent Based Model ◽  
Agent Based ◽  
Model Based ◽  
Model Free ◽  
Analysis Technique ◽  
Analyze Data

Recent years have seen a surge of research into the neuroscience of planning. Much of this work has taken advantage of a two-step sequential decision task developed by Daw et al. (2011), which gives the ability to diagnose whether or not subjects’ behavior is the result of planning. Here, we present simulations which suggest that the techniques most commonly used to analyze data from this task may be confounded in important ways. We introduce a new analysis technique, which suffers from fewer of these issues. This technique also presents a richer view of behavior, making it useful for characterizing patterns in behavior in a theory-neutral manner. This allows it to provide an important check on the assumptions of more theory-driven analysis such as agent-based model-fitting.

scholarly journals Individual differences in dopamine function underlying the balance between model-based and model-free control

2019 ◽  
Author(s):  
Ying Lee ◽  
Lorenz Deserno ◽  
Nils B. Kroemer ◽  
Shakoor Pooseh ◽  
Liane Oehme ◽  
...  
Keyword(s):  
Individual Differences ◽  
Control Individual ◽  
General Notion ◽  
Sequential Decision ◽  
Model Based ◽  
Model Free ◽  
Positron Emission ◽  
Model Free Control ◽  
Induced Changes ◽  
The Relationship

AbstractReinforcement learning involves a balance between model-free (MF) and model-based (MB) systems. Recent studies suggest that individuals with either pharmacologically enhanced levels of dopamine (DA) or higher baseline levels of DA exhibit more MB control. However, it remains unknown whether such pharmacological effects depend on baseline DA.Here, we investigated whether effects of L-DOPA on the balance of MB/MF control depend on ventral striatal baseline DA. Sixty participants had two functional magnetic resonance imaging (fMRI) scans while performing a two-stage sequential decision-making task under 150 mg L-DOPA or placebo (counterbalanced), followed by a 4-hour 18F-DOPA positron emission tomography (PET) scan (on a separate occasion).We found an interaction between baseline DA levels and L-DOPA induced changes in MB control. Individuals with higher baseline DA levels showed a greater L-DOPA induced enhancement in MB control. Surprisingly, we found a corresponding drug-by-baseline DA interaction on MF, but not MB learning signals in the ventromedial prefrontal cortex. We did not find a significant interaction between baseline DA levels and L-DOPA effects on MF control or MB/MF balance.In sum, our findings point to a baseline dependency of L-DOPA effects on differential aspects of MB and MF control. Individual differences in DA washout may be an important moderator of L-DOPA effects. Overall, our findings complement the general notion where higher DA levels is related to a greater reliance on MB control. Although the relationship between phasic DA firing and MF learning is conventionally assumed in the animal literature, the relationship between DA and MF control is not as straightforward and requires further clarification.

scholarly journals Hunger improves reinforcement-driven but not planned action

2021 ◽  
Author(s):  
Maaike M.H. van Swieten ◽  
Rafal Bogacz ◽  
Sanjay G. Manohar
Keyword(s):  
Decision Making ◽  
Learning Systems ◽  
Sequential Decision Making ◽  
Sequential Decision ◽  
Context Sensitive ◽  
Decision Systems ◽  
Model Based ◽  
Model Free ◽  
Model Free Control ◽  
Overall Performance

AbstractHuman decisions can be reflexive or planned, being governed respectively by model-free and model-based learning systems. These two systems might differ in their responsiveness to our needs. Hunger drives us to specifically seek food rewards, but here we ask whether it might have more general effects on these two decision systems. On one hand, the model-based system is often considered flexible and context-sensitive, and might therefore be modulated by metabolic needs. On the other hand, the model-free system’s primitive reinforcement mechanisms may have closer ties to biological drives. Here, we tested participants on a well-established two-stage sequential decision-making task that dissociates the contribution of model-based and model-free control. Hunger enhanced overall performance by increasing model-free control, without affecting model-based control. These results demonstrate a generalized effect of hunger on decision-making that enhances reliance on primitive reinforcement learning, which in some situations translates into adaptive benefits.

Representation, abstraction, and simple-minded sophisticates

2020 ◽  
Vol 43 ◽  
Author(s):  
Peter Dayan
Keyword(s):  
Decision Theory ◽  
Predictive Coding ◽  
Bayesian Decision Theory ◽  
Bayesian Decision ◽  
Model Based ◽  
Model Free

Abstract Bayesian decision theory provides a simple formal elucidation of some of the ways that representation and representational abstraction are involved with, and exploit, both prediction and its rather distant cousin, predictive coding. Both model-free and model-based methods are involved.

Sign in / Sign up

Export Citation Format

Share Document