Multiple sequential prediction errors during reward processing in the human brain

SUMMARYRecent developments in reinforcement learning, cognitive control, and systems neuroscience highlight the complimentary roles in learning of valenced reward prediction errors (RPEs) and non-valenced salience prediction errors (PEs) driven by the magnitude of surprise. A core debate in reward learning focuses on whether valenced and non-valenced PEs can be isolated in the human electroencephalogram (EEG). Here, we combine behavioral modeling and single-trial EEG regression revealing a sequence of valenced and non-valenced PEs in an interval timing task dissociating outcome valence, magnitude, and probability. Multiple regression across temporal, spatial, and frequency dimensions revealed a spatio-tempo-spectral cascade from valenced RPE value represented by the feedback related negativity event-related potential (ERP) followed by non-valenced RPE magnitude and outcome probability effects indexed by subsequent P300 and late frontal positivity ERPs. The results show that learning is supported by a sequence of multiple PEs evident in the human EEG.

Download Full-text

Single-trial modeling separates multiple overlapping prediction errors during reward processing in human EEG

Communications Biology ◽

10.1038/s42003-021-02426-1 ◽

2021 ◽

Vol 4 (1) ◽

Author(s):

Colin W. Hoy ◽

Sheila C. Steiner ◽

Robert T. Knight

Keyword(s):

Reward Processing ◽

Prediction Errors ◽

Single Trial ◽

Human Eeg ◽

Reward Positivity ◽

Outcome Valence ◽

Timing Task ◽

Value Effect ◽

Electroencephalogram Eeg ◽

Outcome Probability

AbstractLearning signals during reinforcement learning and cognitive control rely on valenced reward prediction errors (RPEs) and non-valenced salience prediction errors (PEs) driven by surprise magnitude. A core debate in reward learning focuses on whether valenced and non-valenced PEs can be isolated in the human electroencephalogram (EEG). We combine behavioral modeling and single-trial EEG regression to disentangle sequential PEs in an interval timing task dissociating outcome valence, magnitude, and probability. Multiple regression across temporal, spatial, and frequency dimensions characterized a spatio-tempo-spectral cascade from early valenced RPE value to non-valenced RPE magnitude, followed by outcome probability indexed by a late frontal positivity. Separating negative and positive outcomes revealed the valenced RPE value effect is an artifact of overlap between two non-valenced RPE magnitude responses: frontal theta feedback-related negativity on losses and posterior delta reward positivity on wins. These results reconcile longstanding debates on the sequence of components representing reward and salience PEs in the human EEG.

Download Full-text

The neural encoding of information prediction errors during non-instrumental information seeking

10.1101/179507 ◽

2017 ◽

Author(s):

Maja Brydevall ◽

Daniel Bennett ◽

Carsten Murawski ◽

Stefan Bode

Keyword(s):

Decision Making ◽

Information Seeking ◽

Prediction Error ◽

Reward Processing ◽

Event Related Potential ◽

Recent Theory ◽

Prediction Errors ◽

Neural Encoding ◽

New Information ◽

Human Participants

ABSTRACTIn a dynamic world, accurate beliefs about the environment are vital for survival, and individuals should therefore regularly seek out new information with which to update their beliefs. This aspect of behaviour is not well captured by standard theories of decision making, and the neural mechanisms of information seeking remain unclear. One recent theory posits that valuation of information results from representation of informative stimuli within canonical neural reward-processing circuits, even if that information lacks instrumental use. We investigated this question by recording EEG from twenty-three human participants performing a non-instrumental information-seeking task. In this task, participants could pay a monetary cost to receive advance information about the likelihood of receiving reward in a lottery at the end of each trial. Behavioural results showed that participants were willing to incur considerable monetary costs to acquire early but non-instrumental information. Analysis of the event-related potential elicited by informative cues revealed that the feedback-related negativity independently encoded both an information prediction error and a reward prediction error. These findings are consistent with the hypothesis that information seeking results from processing of information within neural reward circuits, and suggests that information may represent a distinct dimension of valuation in decision making under uncertainty.

Download Full-text

Supplemental Material for Reward Processing in Certain Versus Uncertain Contexts in Schizophrenia: An Event-Related Potential (ERP) Study

Journal of Abnormal Psychology ◽

10.1037/abn0000469.supp ◽

2019 ◽

Keyword(s):

Reward Processing ◽

Event Related Potential

Download Full-text

Dissociating the effect of reward uncertainty and timing uncertainty on neural indices of reward prediction errors: A reward positivity (RewP) event-related potential (ERP) study

Biological Psychology ◽

10.1016/j.biopsycho.2021.108121 ◽

2021 ◽

pp. 108121

Author(s):

Alexandra M. Muir ◽

Addison C. Eberhard ◽

Megan S. Walker ◽

Angus Bennion ◽

Mikle South ◽

...

Keyword(s):

Event Related Potential ◽

Prediction Errors ◽

Reward Positivity ◽

Reward Prediction ◽

Timing Uncertainty

Download Full-text

P.334 An event-related potential study of reward processing in subjects with schizophrenia and healthy controls

European Neuropsychopharmacology ◽

10.1016/j.euroneuro.2019.12.116 ◽

2020 ◽

Vol 31 ◽

pp. S72-S73

Author(s):

Ö. Akgül ◽

E. Fide ◽

F. Özel ◽

K. Alptekin ◽

G. Yener ◽

...

Keyword(s):

Reward Processing ◽

Event Related Potential ◽

Healthy Controls ◽

Potential Study

Download Full-text

Subsecond dopamine fluctuations in human striatum encode superposed error signals about actual and counterfactual reward

Proceedings of the National Academy of Sciences ◽

10.1073/pnas.1513619112 ◽

2015 ◽

Vol 113 (1) ◽

pp. 200-205 ◽

Cited By ~ 92

Author(s):

Kenneth T. Kishida ◽

Ignacio Saez ◽

Terry Lohrenz ◽

Mark R. Witcher ◽

Adrian W. Laxton ◽

...

Keyword(s):

Parkinson’S Disease ◽

Decision Making ◽

Parkinson's Disease ◽

Animal Models ◽

Large Body ◽

Reward Processing ◽

Dopamine Neurons ◽

Model Organisms ◽

Mammalian Brain ◽

Prediction Errors

In the mammalian brain, dopamine is a critical neuromodulator whose actions underlie learning, decision-making, and behavioral control. Degeneration of dopamine neurons causes Parkinson’s disease, whereas dysregulation of dopamine signaling is believed to contribute to psychiatric conditions such as schizophrenia, addiction, and depression. Experiments in animal models suggest the hypothesis that dopamine release in human striatum encodes reward prediction errors (RPEs) (the difference between actual and expected outcomes) during ongoing decision-making. Blood oxygen level-dependent (BOLD) imaging experiments in humans support the idea that RPEs are tracked in the striatum; however, BOLD measurements cannot be used to infer the action of any one specific neurotransmitter. We monitored dopamine levels with subsecond temporal resolution in humans (n = 17) with Parkinson’s disease while they executed a sequential decision-making task. Participants placed bets and experienced monetary gains or losses. Dopamine fluctuations in the striatum fail to encode RPEs, as anticipated by a large body of work in model organisms. Instead, subsecond dopamine fluctuations encode an integration of RPEs with counterfactual prediction errors, the latter defined by how much better or worse the experienced outcome could have been. How dopamine fluctuations combine the actual and counterfactual is unknown. One possibility is that this process is the normal behavior of reward processing dopamine neurons, which previously had not been tested by experiments in animal models. Alternatively, this superposition of error terms may result from an additional yet-to-be-identified subclass of dopamine neurons.

Download Full-text

Aberrant reward dynamics in trait anticipatory anhedonia

Social Cognitive and Affective Neuroscience ◽

10.1093/scan/nsz062 ◽

2019 ◽

Vol 14 (8) ◽

pp. 899-909

Author(s):

Shiyu Zhou ◽

Lu Nie ◽

Zhao Wang ◽

Mengyao Wang ◽

Ya Zheng

Keyword(s):

Clinical Sample ◽

Reward Processing ◽

Event Related Potential ◽

Behavioral Performance ◽

Current Event ◽

Disease States ◽

Monetary Incentive Delay ◽

Anticipatory Anhedonia ◽

Cardinal Feature

Abstract As a cardinal feature of several psychiatric disorders, anhedonia includes a consummatory component (deficits in hedonic response to rewards) and an anticipatory component (a reduced motivation to pursue them). Although being conceptualized as impairments of reward system, the neural characterization of reward processing in anhedonia is hampered by the enormous heterogeneity in the reward phase (‘wanting’ vs ‘liking’) and comorbidity (inherent to disease states). The current event-related potential (ERP) study examined the reward dynamics of anticipatory anhedonia in a non-clinical sample. Anticipatory and consummatory ERP components were assessed with a monetary incentive delay task in a high anticipatory anhedonia (HAA) group and a low anticipatory anhedonia (LAA) group. HAA vs LAA group showed a diminished reward-related speeding during behavioral performance and reported overall reduced positive affect during anticipation and receipt of outcomes. Importantly, neural dynamics underlying reward processing were negatively associated with anticipatory anhedonia across the anticipatory phase indexed by the contingent negative variation and the consummatory phase indexed by the feedback P3. Our results suggest that anticipatory anhedonia in non-clinical individuals is linked to a poor modulation during both anticipatory and consummatory phases of reward processing.

Download Full-text

Magnetoencephalography

10.1093/med/9780199688395.003.0013 ◽

2016 ◽

Author(s):

Paul L. Furlong ◽

Elaine Foley ◽

Caroline Witton ◽

Stefano Seri

Keyword(s):

Age Groups ◽

Seizure Freedom ◽

Additional Information ◽

Epileptogenic Lesion ◽

Recent Developments ◽

Paediatric Age ◽

Important Additional Information ◽

Movement Compensation ◽

Patients With Epilepsy ◽

Electroencephalogram Eeg

For presurgical assessments for resection of an epileptogenic lesion or zone, evaluations over the last 20 years have established magnetoencephalography (MEG) as a valuable tool in routine clinical practice in both adult and paediatric age groups. MEG can accurately localize both ictal and inter-ictal spike sources. MEG yields important additional information in around 30% of patients with epilepsy of suspected neocortical origin, aiding in the modification or extension of invasive measurements. Seizure freedom is most likely to occur when there is concordance between electroencephalogram (EEG) and MEG localization, and least likely to occur when these results are divergent. In some patients, invasive recordings may not be viable or repeatable. In these cases, MEG localization frequently provides additional information for planning surgery. Recent developments in technology for movement compensation and enhanced noise reduction provide optimism for continually improving outcomes of MEG-enhanced presurgical evaluations.

Download Full-text