Internal-Time Temporal Difference Model for Neural Value-Based Decision Making

2010 ◽  
Vol 22 (12) ◽  
pp. 3062-3106 ◽  
Author(s):  
Hiroyuki Nakahara ◽  
Sivaramakrishnan Kaveri
Keyword(s):  
Decision Making ◽  
Intertemporal Choice ◽  
Hyperbolic Discounting ◽  
Time Frame ◽  
Subjective Time ◽  
Temporal Difference ◽  
Internal Time ◽  
Learning Framework ◽  
Neural Activities ◽  
Choice Tasks

The temporal difference (TD) learning framework is a major paradigm for understanding value-based decision making and related neural activities (e.g., dopamine activity). The representation of time in neural processes modeled by a TD framework, however, is poorly understood. To address this issue, we propose a TD formulation that separates the time of the operator (neural valuation processes), which we refer to as internal time, from the time of the observer (experiment), which we refer to as conventional time. We provide the formulation and theoretical characteristics of this TD model based on internal time, called internal-time TD, and explore the possible consequences of the use of this model in neural value-based decision making. Due to the separation of the two times, internal-time TD computations, such as TD error, are expressed differently, depending on both the time frame and time unit. We examine this operator-observer problem in relation to the time representation used in previous TD models. An internal time TD value function exhibits the co-appearance of exponential and hyperbolic discounting at different delays in intertemporal choice tasks. We further examine the effects of internal time noise on TD error, the dynamic construction of internal time, and the modulation of internal time with the internal time hypothesis of serotonin function. We also relate the internal TD formulation to research on interval timing and subjective time.

scholarly journals Poverty and Economic Decision-Making: Evidence from Changes in Financial Resources at Payday

2016 ◽  
Vol 106 (2) ◽  
pp. 260-284 ◽  
Author(s):  
Leandro S. Carvalho ◽  
Stephan Meier ◽  
Stephanie W. Wang
Keyword(s):  
Decision Making ◽  
Cognitive Function ◽  
Low Income ◽  
Online Survey ◽  
Intertemporal Choice ◽  
Financial Resources ◽  
Economic Decision Making ◽  
Choice Tasks ◽  
Intertemporal Choices

We study the effect of financial resources on decision-making. Low-income US households are randomly assigned to receive an online survey before or after payday. The survey collects measures of cognitive function and administers risk and intertemporal choice tasks. The study design generates variation in cash, checking and savings balances, and expenditures. Before-payday participants behave as if they are more present-biased when making intertemporal choices about monetary rewards but not when making intertemporal choices about nonmonetary real-effort tasks. Nor do we find before-after differences in risk-taking, the quality of decision-making, the performance in cognitive function tasks, or in heuristic judgments. (JEL C83, D14, D81, D91, I32)

scholarly journals Waiting in intertemporal choice tasks affects discounting and subjective time perception.

2020 ◽  
Vol 149 (12) ◽  
pp. 2289-2313
Author(s):  
Ping Xu ◽  
Claudia González-Vallejo ◽  
Benjamin T. Vincent
Keyword(s):  
Time Perception ◽  
Intertemporal Choice ◽  
Subjective Time ◽  
Choice Tasks

scholarly journals Impulsivity as Bayesian inference under dopaminergic control

2020 ◽  
Author(s):  
John G. Mikhael ◽  
Samuel J. Gershman
Keyword(s):  
Decision Making ◽  
Bayesian Model ◽  
Opposite Effect ◽  
Intertemporal Choice ◽  
Contextual Information ◽  
Preference Reversals ◽  
Temporal Precision ◽  
Choice Tasks ◽  
Selection Of ◽  
Reward Delay

AbstractBayesian models successfully account for several of dopamine (DA)’s effects on contextual calibration in interval timing and reward estimation. In these models, DA controls the precision of stimulus encoding, which is weighed against contextual information when making decisions. When DA levels are high, the animal relies more heavily on the (highly precise) stimulus encoding, whereas when DA levels are low, the context affects decisions more strongly. Here, we extend this idea to intertemporal choice tasks, in which agents must choose between small rewards delivered soon and large rewards delivered later. Beginning with the principle that animals will seek to maximize their reward rates, we show that the Bayesian model predicts a number of curious empirical findings. First, the model predicts that higher DA levels should normally promote selection of the larger/later option, which is often taken to imply that DA decreases ‘impulsivity.’ However, if the temporal precision is sufficiently decreased, higher DA levels should have the opposite effect—promoting selection of the smaller/sooner option (more impulsivity). Second, in both cases, high enough levels of DA can result in preference reversals. Third, selectively decreasing the temporal precision, without manipulating DA, should promote selection of the larger/later option. Fourth, when a different post-reward delay is associated with each option, animals will not learn the option-delay contingencies, but this learning can be salvaged when the post-reward delays are made more salient. Finally, the Bayesian model predicts a correlation between behavioral phenotypes: Animals that are better timers will also appear less impulsive.Significance StatementDoes dopamine make animals more or less impulsive? Though impulsivity features prominently in several dopamine-related conditions, how dopamine actually influences impulsivity has remained unclear. In intertemporal choice tasks (ITCs), wherein animals must choose between small rewards delivered soon and large rewards delivered later, administering dopamine makes animals more willing to wait for larger/later rewards in some conditions (consistent with lower impulsivity), but less willing in others. We hypothesize that dopamine does not necessarily influence impulsivity at all, but rather gates the influence of contextual information during decision making. We show that this account explains an array of curious findings in ITCs, including the seemingly conflicting results above. Our work encourages a reexamination of ITCs as a method for assessing impulsivity.

scholarly journals Rats exhibit similar biases in foraging and intertemporal choice tasks

eLife ◽  
2019 ◽  
Vol 8 ◽  
Author(s):  
Gary A Kane ◽  
Aaron M Bornstein ◽  
Amitai Shenhav ◽  
Robert C Wilson ◽  
Nathaniel D Daw ◽  
...  
Keyword(s):  
Intertemporal Choice ◽  
Optimal Foraging Theory ◽  
Hyperbolic Discounting ◽  
Time Preferences ◽  
Common Mechanism ◽  
Similar Time ◽  
Short And Long Term ◽  
Choice Tasks ◽  
Exponential Discounting

Animals, including humans, consistently exhibit myopia in two different contexts: foraging, in which they harvest locally beyond what is predicted by optimal foraging theory, and intertemporal choice, in which they exhibit a preference for immediate vs. delayed rewards beyond what is predicted by rational (exponential) discounting. Despite the similarity in behavior between these two contexts, previous efforts to reconcile these observations in terms of a consistent pattern of time preferences have failed. Here, via extensive behavioral testing and quantitative modeling, we show that rats exhibit similar time preferences in both contexts: they prefer immediate vs. delayed rewards and they are sensitive to opportunity costs of delays to future decisions. Further, a quasi-hyperbolic discounting model, a form of hyperbolic discounting with separate components for short- and long-term rewards, explains individual rats’ time preferences across both contexts, providing evidence for a common mechanism for myopic behavior in foraging and intertemporal choice.

scholarly journals Time Unpacking Effect on Intertemporal Decision-Making: Does the Effect Change With Choice Valence?

2021 ◽  
Vol 12 ◽  
Author(s):  
Quan Yang ◽  
Xianmin Gong ◽  
Jinli Xiong ◽  
Shufei Yin
Keyword(s):  
College Students ◽  
Decision Making ◽  
Time Perception ◽  
Time Delays ◽  
Intertemporal Choice ◽  
Experimental Conditions ◽  
Effect Change ◽  
Gain Loss ◽  
Choice Tasks ◽  
Delayed Gains

People often feel that a period of time becomes longer when it is described in more detail or cut into more segments, which is known as the time unpacking effect. The current study aims to unveil how time unpacking manipulation impacts intertemporal decision making and whether the gain-loss valence of choices moderates such impacts. We recruited 87 college students (54 female) and randomly assigned them to the experimental conditions to complete a series of intertemporal choice tasks. The subjective values of the delayed choices were calculated for each participant and then analyzed. The results showed that participants perceived longer time delays and higher subjective values on the delayed gains (but not losses) in the time unpacking conditions than in the time packing conditions. These results suggest that time unpacking manipulation not only impacts time perception but also other factors, which in turn, influence the valuation of delayed outcomes and thereby intertemporal choices. The results are discussed in comparison to previous studies to highlight the complexity of the mechanism underlying the effect of time unpacking on intertemporal decision making.

scholarly journals Rats exhibit similar biases in foraging and intertemporal choice tasks

2018 ◽  
Author(s):  
Gary A. Kane ◽  
Aaron M. Bornstein ◽  
Amitai Shenhav ◽  
Robert C. Wilson ◽  
Nathaniel D. Daw ◽  
...  
Keyword(s):  
Intertemporal Choice ◽  
Optimal Foraging Theory ◽  
Hyperbolic Discounting ◽  
Time Preferences ◽  
Common Mechanism ◽  
Similar Time ◽  
Short And Long Term ◽  
Choice Tasks ◽  
Exponential Discounting

AbstractAnimals, including humans, consistently exhibit myopia in two different contexts: foraging, in which they harvest locally beyond what is predicted by optimal foraging theory, and intertemporal choice, in which they exhibit a preference for immediate vs. delayed rewards beyond what is predicted by rational (exponential) discounting. Despite the similarity in behavior between these two contexts, previous efforts to reconcile these observations in terms of a consistent pattern of time preferences have failed. Here, via extensive behavioral testing and quantitative modeling, we show that rats exhibit similar time preferences in both contexts: they prefer immediate vs. delayed rewards and they are sensitive to opportunity costs — delays to future decisions. Further, a quasi-hyperbolic discounting model, a form of hyperbolic discounting with separate components for short-and long-term rewards, explains individual rats’ time preferences across both contexts, providing evidence for a common mechanism for myopic behavior in foraging and intertemporal choice.

Temporal Discounting Across Adulthood: A Systematic Review and Meta-analysis

2020 ◽  
Author(s):  
Kendra Leigh Seaman ◽  
Sade J Abiodun ◽  
Zöe Fenn ◽  
Gregory Russell Samanez-Larkin ◽  
Rui Mata
Keyword(s):  
Age Differences ◽  
Intertemporal Choice ◽  
Meta Analysis ◽  
Empirical Work ◽  
Adult Life ◽  
Temporal Discounting ◽  
Total N ◽  
Adult Age ◽  
Adult Age Differences ◽  
Choice Tasks

A number of developmental theories have been proposed that make differential predictions about the links between age and temporal discounting; that is, the valuation of rewards at different points in time. Most empirical studies examining adult age differences in temporal discounting have relied on economic intertemporal choice tasks, which pit choosing a smaller, sooner monetary reward against choosing a larger, later one. Although initial studies using these tasks suggested older adults discount less than younger adults, follow-up studies provided heterogeneous, and thus inconclusive, results. Using an open science approach, we test the replicability of adult age differences in temporal discounting by conducting a preregistered systematic literature search and meta-analysis of adult age differences in intertemporal choice tasks. Across 37 cross-sectional studies (Total N = 104,736), we found no reliable relation between age and temporal discounting (r = -0.081, 95% CI [-0.185, 0.025]). We also found little evidence of publication bias or p-hacking. Exploratory analyses of moderators found no effect of experimental design (e.g., extreme-group vs. continuous age), incentives (hypothetical vs. rewards), amount of delay (e.g., days, weeks, months, or years), or quantification of discounting behavior (e.g., proportion of immediate choices vs. parameters from computational modeling). Additional analyses of 12 participant-level data sets found little support for a nonlinear relation between age and temporal discounting across adulthood. Overall, the results suggest that adult age is not reliably associated with individual differences in temporal discounting. We provide recommendations for future empirical work on temporal discounting across the adult life span.

Discounting Time and Time Discounting: Subjective Time Perception and Intertemporal Preferences

2009 ◽  
Vol 46 (4) ◽  
pp. 543-556 ◽  
Author(s):  
Gal Zauberman ◽  
B. Kyu Kim ◽  
Selin A. Malkoc ◽  
James R. Bettman
Keyword(s):  
Time Perception ◽  
Hyperbolic Discounting ◽  
Subjective Time ◽  
Multiple Time ◽  
Subjective Perceptions ◽  
Objective Time ◽  
Trade Offs ◽  
Intertemporal Preferences ◽  
Constant Rate ◽  
Within Subjects

Consumers often make decisions about outcomes and events that occur over time. This research examines consumers' sensitivity to the prospective duration relevant to their decisions and the implications of such sensitivity for intertemporal trade-offs, especially the degree of present bias (i.e., hyperbolic discounting). The authors show that participants' subjective perceptions of prospective duration are not sufficiently sensitive to changes in objective duration and are nonlinear and concave in objective time, consistent with psychophysical principles. More important, this lack of sensitivity can explain hyperbolic discounting. The results replicate standard hyperbolic discounting effects with respect to objective time but show a relatively constant rate of discounting with respect to subjective time perceptions. The results are replicated between subjects (Experiment 1) and within subjects (Experiments 2), with multiple time horizons and multiple descriptors, and with different measurement orders. Furthermore, the authors show that when duration is primed, subjective time perception is altered (Experiment 4) and hyperbolic discounting is reduced (Experiment 3).

Sign in / Sign up

Export Citation Format

Share Document