scholarly journals An association between prediction errors and risk-seeking: Theory and behavioral evidence

2021 ◽  
Vol 17 (7) ◽  
pp. e1009213
Author(s):  
Moritz Moeller ◽  
Jan Grohn ◽  
Sanjay Manohar ◽  
Rafal Bogacz
Keyword(s):  
Risk Preferences ◽  
Recent Theory ◽  
Prediction Errors ◽  
Dynamic Learning ◽  
Risk Seeking ◽  
Learning Rates ◽  
Increased Risk ◽  
Behavioral Evidence ◽  
Behavioral Predictions ◽  
Value Learning

Reward prediction errors (RPEs) and risk preferences have two things in common: both can shape decision making behavior, and both are commonly associated with dopamine. RPEs drive value learning and are thought to be represented in the phasic release of striatal dopamine. Risk preferences bias choices towards or away from uncertainty; they can be manipulated with drugs that target the dopaminergic system. Based on the common neural substrate, we hypothesize that RPEs and risk preferences are linked on the level of behavior as well. Here, we develop this hypothesis theoretically and test it empirically. First, we apply a recent theory of learning in the basal ganglia to predict how RPEs influence risk preferences. We find that positive RPEs should cause increased risk-seeking, while negative RPEs should cause risk-aversion. We then test our behavioral predictions using a novel bandit task in which value and risk vary independently across options. Critically, conditions are included where options vary in risk but are matched for value. We find that our prediction was correct: participants become more risk-seeking if choices are preceded by positive RPEs, and more risk-averse if choices are preceded by negative RPEs. These findings cannot be explained by other known effects, such as nonlinear utility curves or dynamic learning rates.

scholarly journals A learning mechanism shaping risk preferences and a preliminary test of its relationship with psychopathic traits

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Takeyuki Oba ◽  
Kentaro Katahira ◽  
Hideki Ohira
Keyword(s):  
Decision Making ◽  
Personality Traits ◽  
Risk Preferences ◽  
Preliminary Test ◽  
Learning Task ◽  
Prediction Errors ◽  
Reflection Effect ◽  
Differential Learning ◽  
Learning Rates ◽  
Past Experiences

AbstractPeople tend to avoid risk in the domain of gains but take risks in the domain of losses; this is called the reflection effect. Formal theories of decision-making have provided important perspectives on risk preferences, but how individuals acquire risk preferences through experiences remains unknown. In the present study, we used reinforcement learning (RL) models to examine the learning processes that can shape attitudes toward risk in both domains. In addition, relationships between learning parameters and personality traits were investigated. Fifty-one participants performed a learning task, and we examined learning parameters and risk preference in each domain. Our results revealed that an RL model that included a nonlinear subjective utility parameter and differential learning rates for positive and negative prediction errors exhibited better fit than other models and that these parameters independently predicted risk preferences and the reflection effect. Regarding personality traits, although the sample sizes may be too small to test personality traits, increased primary psychopathy scores could be linked with decreased learning rates for positive prediction error in loss conditions among participants who had low anxiety traits. The present findings not only contribute to understanding how decision-making in risky conditions is influenced by past experiences but also provide insights into certain psychiatric problems.

scholarly journals Reward prediction errors induce risk-seeking

2020 ◽  
Author(s):  
Moritz Moeller ◽  
Jan Grohn ◽  
Sanjay Manohar ◽  
Rafal Bogacz
Keyword(s):  
Reinforcement Learning ◽  
Prediction Error ◽  
Risk Preferences ◽  
Behavioral Effect ◽  
Learning Task ◽  
Expected Value ◽  
Learning Theories ◽  
Prediction Errors ◽  
Risk Seeking ◽  
The Difference

AbstractReinforcement learning theories propose that humans choose based on the estimated values of available options, and that they learn from rewards by reducing the difference between the experienced and expected value. In the brain, such prediction errors are broadcasted by dopamine. However, choices are not only influenced by expected value, but also by risk. Like reinforcement learning, risk preferences are modulated by dopamine: enhanced dopamine levels induce risk-seeking. Learning and risk preferences have so far been studied independently, even though it is commonly assumed that they are (partly) regulated by the same neurotransmitter. Here, we use a novel learning task to look for prediction-error induced risk-seeking in human behavior and pupil responses. We find that prediction errors are positively correlated with risk-preferences in imminent choices. Physiologically, this effect is indexed by pupil dilation: only participants whose pupil response indicates that they experienced the prediction error also show the behavioral effect.

Respondents in Asian Cultures (e.g., Chinese) are More Risk-Seeking and More Overconfident than Respondents in Other Cultures (e.g., in United States) but the Reciprocal Predictions are in Total Opposition: How and Why?

2004 ◽  
Vol 4 (2) ◽  
pp. 263-292 ◽  
Author(s):  
Shu Li ◽  
Yongqing Fang
Keyword(s):  
United States ◽  
Social Network ◽  
Risk Taking ◽  
Risk Preferences ◽  
Risk Preference ◽  
Cross Cultural ◽  
Cultural Variations ◽  
Risk Seeking ◽  
The Social ◽  
Predicted Risk

AbstractTriggered by rather surprising findings that respondents in Asian cultures (e.g., Chinese) are more risk-seeking and more overconfident than respondents in other cultures (e.g., in United States) and that the reciprocal predictions are in total opposition, four experiments were designed to extend previous collective-culture oriented researches. Results revealed that (1) Singapore 21, which is a vision of Singapore in the 21st century and has highlighted the promotion of a collective culture, did not advocate greater risk-seeking but led to weaker overconfidence; (2) the knowledge of "financial help from social network" did not permit prediction of risk preference but the knowledge of "the value difference between possible outcomes" did; (3) the social network could be viewed not only as a positive "cushion" but also as a negative "burden" in both gain and loss domains of risky choices; (4) the predictions of the risk-as-value, risk-as-feelings and stereotype hypotheses were not consistent with the predicted risk preferences of others but the predictions of the economic-performance hypothesis were consistent with the predicted risk preferences as well as the predicted overconfidence of others. The implications for cross-cultural variations in overconfidence and for cross-cultural variations in risk-taking were discussed.

scholarly journals A nonlinear relationship between prediction errors and learning rates in human reinforcement learning

2019 ◽  
Author(s):  
Erdem Pulcu
Keyword(s):  
Reinforcement Learning ◽  
Nonlinear Relationship ◽  
Prediction Errors ◽  
Learning Rates ◽  
The Face ◽  
In The Wild ◽  
Actual Outcome ◽  
Update Rules ◽  
Different Sources

AbstractWe are living in a dynamic world in which stochastic relationships between cues and outcome events create different sources of uncertainty1 (e.g. the fact that not all grey clouds bring rain). Living in an uncertain world continuously probes learning systems in the brain, guiding agents to make better decisions. This is a type of value-based decision-making which is very important for survival in the wild and long-term evolutionary fitness. Consequently, reinforcement learning (RL) models describing cognitive/computational processes underlying learning-based adaptations have been pivotal in behavioural2,3 and neural sciences4–6, as well as machine learning7,8. This paper demonstrates the suitability of novel update rules for RL, based on a nonlinear relationship between prediction errors (i.e. difference between the agent’s expectation and the actual outcome) and learning rates (i.e. a coefficient with which agents update their beliefs about the environment), that can account for learning-based adaptations in the face of environmental uncertainty. These models illustrate how learners can flexibly adapt to dynamically changing environments.

scholarly journals Sensitivity to gains during risky decision-making differentiates chronic cocaine users from stimulant-naïve controls

2019 ◽  
Author(s):  
B. Kluwe-Schiavon ◽  
A. Kexel ◽  
G. Manenti ◽  
D.M. Cole ◽  
M.R. Baumgartner ◽  
...  
Keyword(s):  
Decision Making ◽  
Neuropsychological Test ◽  
Risky Decision Making ◽  
Risky Decision ◽  
Cocaine Use ◽  
Risk Seeking ◽  
Chronic Cocaine ◽  
Increased Risk ◽  
Cocaine Users ◽  
The Individual

AbstractBackgroundAlthough chronic cocaine use has been frequently associated with decision-making impairments that are supposed to contribute to the development and maintenance of cocaine addiction, it has remained unclear how risk-seeking behaviours observed in chronic cocaine users (CU) come about. Here we therefore test whether risky decision-making observed in CU is driven by alterations in individual sensitivity to the available information (gain, loss, and risk).MethodA sample of 96 participants (56 CU and 40 controls) performed the no-feedback (“cold”) version of the Columbia Card Task. Structured psychiatric interviews and a comprehensive neuropsychological test battery were additionally conducted. Current and recent substance use was objectively assessed by toxicological urine and hair analysis.ResultsCompared to controls, CU showed increased risk-seeking in unfavourable decision scenarios in which the risk was high and the returns were low, and a tendency for increased risk aversion in favourable decision scenarios. These differences arose from the fact that CU were less sensitive to gain, but similarly sensitive to loss and risk information in comparison to controls. Further analysis revealed that individual differences in sensitivity to loss and risk were related to cognitive performance and impulsivity.ConclusionThe reduced sensitivity to gain information in people with CU may contribute to their propensity for making risky decisions. While these alterations in the sensitivity to gain might be directly related to cocaine use per se, the individual psychopathological profile of CU might moderate their sensitivity to risk and loss impulsivity.

Abstract P190: Antihypertensive Monotherapy And Risk Of Depression Incidence

Hypertension ◽  
2020 ◽  
Vol 76 (Suppl_1) ◽  
Author(s):  
Anwar Alnakhli ◽  
Richard Shaw ◽  
Daniel Smith ◽  
Sandosh Padmanabhan
Keyword(s):  
Dose Response ◽  
Enzyme Inhibitor ◽  
Previous History ◽  
Cox Proportional Hazards ◽  
Recent Theory ◽  
Defined Daily Dose ◽  
Response Relationship ◽  
Dose Response Relationship ◽  
Antihypertensive Medications ◽  
Increased Risk

Background: Recent theory suggests that antihypertensive medications may be useful as repurposed treatments for mood disorders, however, empirical evidence is inconsistent Objective: We aimed to assess the risk of depression incidence as indicated by first-ever prescription of antidepressant in patients newly exposed to antihypertensive monotherapy and whether there is a dose-response relationship. Method: This study enrolled 2406 new users of antihypertensive monotherapy aged between 18 and 80 years with no previous history of antidepressant prescriptions. The exposure period (EP) to antihypertensive medication was fixed at one year starting from the first date of antihypertensive prescription between Jan 2005 and Mar 2012 and extended up to 12 months. Follow-up commence after the EP until March 2013. To test for dose-response relationship the cumulative defined daily dose (cDDD) of antihypertensive during the EP were stratified into tertiles. Cox proportional hazards models were used to estimate hazard ratios (HR) for depression incidence. Results: Among the five major classes of antihypertensive medications, calcium channel blocker (CCB) had the highest risk of developing depression after adjusting for covariates (HR = 1.40 95%CI 1.11,1.78) compared to angiotensin-converting enzyme inhibitor (ACEI). Angiotensin-receptor blocker (ARB) treatment showed higher risk of depression incidence with tertile 2(HR= 1.46, 95%CI 0.88,2.44) and tertile 3 (HR= 1.75, 95%CI 1.03,2.97) compared to tertile 1 of cDDD. Conclusion: Our findings confirmed previous evidence suggesting that CCB is associated with increased risk of depression incidence compared to ACEI. Risk of developing depression is also linked to ARB, though it might be dose dependent.

scholarly journals The Role of Memory in Distinguishing Risky Decisions from Experience and Description

2017 ◽  
Vol 70 (10) ◽  
pp. 2048-2059 ◽  
Author(s):  
Christopher R. Madan ◽  
Elliot A. Ludvig ◽  
Marcia L. Spetch
Keyword(s):  
Risk Preferences ◽  
Risk Preference ◽  
Risky Choice ◽  
Risk Averse ◽  
Memory Biases ◽  
Opposite Pattern ◽  
Decisions From Experience ◽  
Risk Seeking ◽  
Subject Design

People's risk preferences differ for choices based on described probabilities versus those based on information learned through experience. For decisions from description, people are typically more risk averse for gains than for losses. In contrast, for decisions from experience, people are sometimes more risk seeking for gains than losses, especially for choices with the possibility of extreme outcomes (big wins or big losses), which are systematically overweighed in memory. Using a within-subject design, this study evaluated whether this memory bias plays a role in the differences in risky choice between description and experience. As in previous studies, people were more risk seeking for losses than for gains in description but showed the opposite pattern in experience. People also more readily remembered the extreme outcomes and judged them as having occurred more frequently. These memory biases correlated with risk preferences in decisions from experience but not in decisions from description. These results suggest that systematic memory biases may be responsible for some of the differences in risk preference across description and experience.

scholarly journals When is the Time for a Change? Decomposing Dynamic Learning Rates

Neuron ◽  
2014 ◽  
Vol 84 (4) ◽  
pp. 662-664 ◽  
Author(s):  
Adrian G. Fischer ◽  
Markus Ullsperger
Keyword(s):  
Dynamic Learning ◽  
Learning Rates

scholarly journals Velocity estimation in reinforcement learning

2018 ◽  
Author(s):  
Carlos Velazquez ◽  
Manuel Villarreal ◽  
Arturo Bouzas
Keyword(s):  
Reinforcement Learning ◽  
Hierarchical Structure ◽  
Predictive Accuracy ◽  
Information Criterion ◽  
Individual Variability ◽  
Velocity Estimation ◽  
Hyperbolic Function ◽  
Prediction Errors ◽  
Signal To Noise ◽  
Learning Rates

The current work aims to study how people make predictions, under a reinforcement learning framework, in an environment that fluctuates from trial to trial and is corrupted with Gaussian noise. A computer-based experiment was developed where subjects were required to predict the future location of a spaceship that orbited around planet Earth. Its position was sampled from a Gaussian distribution with the mean changing at a variable velocity and four different values of variance that defined our signal-to-noise conditions. Three error-driven algorithms using a Bayesian approach were proposed as candidates to describe our data. The first is the standard delta-rule. The second and third models are delta rules incorporating a velocity component which is updated using prediction errors. The third model additionally assumes a hierarchical structure where individual learning rates for velocity and decision noise come from Gaussian distributions with means following a hyperbolic function. We used leave-one-out cross-validation and the Widely Applicable Information Criterion to compare the predictive accuracy of these models. In general, our results provided evidence in favor of the hierarchical model and highlight two main conclusions. First, when facing an environment that fluctuates from trial to trial, people can learn to estimate its velocity to make predictions. Second, learning rates for velocity and decision noise are influenced by uncertainty constraints represented by the signal-to-noise ratio. This higher order control was modeled using a hierarchical structure, which qualitatively accounts for individual variability and is able to generalize and make predictions about new subjects on each experimental condition.

scholarly journals Rare Rewards Amplify Dopamine Learning Responses

2019 ◽  
Author(s):  
Kathryn M. Rothenhoefer ◽  
Tao Hong ◽  
Aydin Alikaya ◽  
William R. Stauffer
Keyword(s):  
Probability Distributions ◽  
Reaction Times ◽  
Neural Mechanism ◽  
Dopamine Neurons ◽  
Point Estimate ◽  
Prediction Errors ◽  
Average Value ◽  
Learning Rates ◽  
Behavioral Choices ◽  
Predicted Values

AbstractDopamine neurons drive learning by coding reward prediction errors (RPEs), which are formalized as subtractions of predicted values from reward values. Subtractions accommodate point estimate predictions of value, such as the average value. However, point estimate predictions fail to capture many features of choice and learning behaviors. For instance, reaction times and learning rates consistently reflect higher moments of probability distributions. Here, we demonstrate that dopamine RPE responses code probability distributions. We presented monkeys with rewards that were drawn from the tails of normal and uniform reward size distributions to generate rare and common RPEs, respectively. Behavioral choices and pupil diameter measurements indicated that monkeys learned faster and registered greater arousal from rare RPEs, compared to common RPEs of identical magnitudes. Dopamine neuron recordings indicated that rare rewards amplified RPE responses. These results demonstrate that dopamine responses reflect probability distributions and suggest a neural mechanism for the amplified learning and enhanced arousal associated with rare events.

Sign in / Sign up

Export Citation Format

Share Document