scholarly journals The best laid plans: Computational principles of ACC

2020 ◽  
Author(s):  
Clay B. Holroyd ◽  
Tom Verguts
Keyword(s):  
Reinforcement Learning ◽  
Anterior Cingulate Cortex ◽  
Hierarchical Model ◽  
Computational Models ◽  
Cingulate Cortex ◽  
Anterior Cingulate ◽  
Hierarchical Reinforcement Learning ◽  
The Past ◽  
Model Based ◽  
High Level

Despite continual debate for the past thirty years about the function of anterior cingulate cortex (ACC), its key contribution to neurocognition remains unknown. Here we review computational models that illustrate three core principles of ACC function (related to hierarchy, world models and cost), as well as four constraints on the neural implementation of these principles (related to modularity, binding, encoding and learning and regulation). These observations suggest a role for ACC in hierarchical model-based hierarchical reinforcement learning, which instantiates a mechanism for motivating the execution of high-level plans.

scholarly journals Anterior cingulate cortex represents action-state predictions and causally mediates model-based reinforcement learning in a two-step decision task

2017 ◽  
Author(s):  
Thomas Akam ◽  
Ines Rodrigues-Vaz ◽  
Ivo Marcelo ◽  
Xiangyu Zhang ◽  
Michael Pereira ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Anterior Cingulate Cortex ◽  
State Transition ◽  
Cingulate Cortex ◽  
Anterior Cingulate ◽  
State Transitions ◽  
Decision Task ◽  
Transition Structure ◽  
Model Based ◽  
Subsequent Choice

SummaryThe anterior cingulate cortex (ACC) is implicated in learning the value of actions, but it remains poorly understood whether and how it contributes to model-based mechanisms that use action-state predictions and afford behavioural flexibility. To isolate these mechanisms, we developed a multi-step decision task for mice in which both action-state transition probabilities and reward probabilities changed over time. Calcium imaging revealed ramps of choice-selective neuronal activity, followed by an evolving representation of the state reached and trial outcome, with different neuronal populations representing reward in different states. ACC neurons represented the current action-state transition structure, whether state transitions were expected or surprising, and the predicted state given chosen action. Optogenetic inhibition of ACC blocked the influence of action-state transitions on subsequent choice, without affecting the influence of rewards. These data support a role for ACC in model-based reinforcement learning, specifically in using action-state transitions to guide subsequent choice.HighlightsA novel two-step task disambiguates model-based and model-free RL in mice.ACC represents all trial events, reward representation is contextualised by state.ACC represents action-state transition structure, predicted states, and surprise.Inhibiting ACC impedes action-state transitions from influencing subsequent choice.

Deficits in Selective Attention Following Bilateral Anterior Cingulotomy

1991 ◽  
Vol 3 (3) ◽  
pp. 231-241 ◽  
Author(s):  
Kevin W. Janer ◽  
José V. Pardo
Keyword(s):  
Anterior Cingulate Cortex ◽  
Cingulate Cortex ◽  
Anterior Cingulate ◽  
Neuropsychological Measures ◽  
Positron Emission ◽  
Cognitive Operations ◽  
Before And After ◽  
Attentional System ◽  
Cortical Regions ◽  
High Level

Positron emission tomographic (PET) studies of normal humans undergoing specific cognitive activation paradigms have identified a region of the anterior cingulate cortex as a component of an anterior, midline attentional system involved in high-level processing selection. However, deficits in attention have not been demonstrated in patients following bilateral anterior cingulotomy, a procedure that results in lesions of adjacent anterior cingulate cortex. Task paradigms used in PET studies that recruit the anterior cingulate cortex were applied to normal, control subjects and to a patient before and after cingulotomy to provide highly sensitive and functionally targeted reaction time measures of attentional performance. In contrast to unchanged performance in several neuropsychological measures, this patient demonstrated specific deficits in attention during the subacute postoperative period, which resolved spontaneously several months after surgery. Such impairment is consistent with the evolving view of the anterior cingulate's involvement in high-level processing selection. These data show the feasibility of using information from PET activation studies of normals in the design of novel chronometric tasks useful for probing abnormalities in specific cognitive operations associated with discrete cortical regions.

scholarly journals Multicentric tracking of multiple agents by anterior cingulate cortex during pursuit and evasion

2021 ◽  
Vol 12 (1) ◽  
Author(s):  
Seng Bum Michael Yoo ◽  
Jiaxin Cindy Tu ◽  
Benjamin Yost Hayden
Keyword(s):  
Motor Control ◽  
Anterior Cingulate Cortex ◽  
Virtual World ◽  
Cingulate Cortex ◽  
Anterior Cingulate ◽  
Dorsal Anterior Cingulate Cortex ◽  
Pursuit Evasion ◽  
Pursuit And Evasion ◽  
High Level ◽  
Hippocampal Complex

AbstractSuccessful pursuit and evasion require rapid and precise coordination of navigation with adaptive motor control. We hypothesize that the dorsal anterior cingulate cortex (dACC), which communicates bidirectionally with both the hippocampal complex and premotor/motor areas, would serve a mapping role in this process. We recorded responses of dACC ensembles in two macaques performing a joystick-controlled continuous pursuit/evasion task. We find that dACC carries two sets of signals, (1) world-centric variables that together form a representation of the position and velocity of all relevant agents (self, prey, and predator) in the virtual world, and (2) avatar-centric variables, i.e. self-prey distance and angle. Both sets of variables are multiplexed within an overlapping set of neurons. Our results suggest that dACC may contribute to pursuit and evasion by computing and continuously updating a multicentric representation of the unfolding task state, and support the hypothesis that it plays a high-level abstract role in the control of behavior.

scholarly journals Pure Correlates of Exploration and Exploitation in the Human Brain

2017 ◽  
Author(s):  
Tommy C. Blanchard ◽  
Samuel J. Gershman
Keyword(s):  
Prefrontal Cortex ◽  
Reinforcement Learning ◽  
Human Brain ◽  
Anterior Cingulate Cortex ◽  
Fundamental Problem ◽  
Cingulate Cortex ◽  
Ventromedial Prefrontal Cortex ◽  
Anterior Cingulate ◽  
Exploration And Exploitation ◽  
Exploration Exploitation

AbstractBalancing exploration and exploitation is a fundamental problem in reinforcement learning. Previous neuroimaging studies of the exploration-exploitation dilemma could not completely disentangle these two processes, making it difficult to unambiguously identify their neural signatures. We overcome this problem using a task in which subjects can either observe (pure exploration) or bet (pure exploitation). Insula and dorsal anterior cingulate cortex showed significantly greater activity on observe trials compared to bet trials, suggesting that these regions play a role in driving exploration. A model-based analysis of task performance suggested that subjects chose to observe until a critical evidence threshold was reached. We observed a neural signature of this evidence accumulation process in ventromedial prefrontal cortex. These findings support theories positing an important role for anterior cingulate cortex in exploration, while also providing a new perspective on the roles of insula and ventromedial prefrontal cortex.Significance StatementSitting down at a familiar restaurant, you may choose to order an old favorite or sample a new dish. In reinforcement learning theory, this is known as the exploration-exploitation dilemma. The optimal solution is known to be intractable; therefore, humans must use heuristic strategies. Behavioral studies have revealed several candidate strategies, but identifying the neural mechanisms underlying these strategies is complicated due to the fact that exploration and exploitation are not perfectly dissociable in standard tasks. Using an “observe or bet” task, we identify for the first time pure neural correlates of exploration and exploitation in the human brain.

scholarly journals Model-based hierarchical reinforcement learning and human action control

2014 ◽  
Vol 369 (1655) ◽  
pp. 20130480 ◽  
Author(s):  
Matthew Botvinick ◽  
Ari Weinstein
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Hierarchical Model ◽  
Human Action ◽  
Action Control ◽  
Computational Framework ◽  
Hierarchical Reinforcement Learning ◽  
Model Based ◽  
Human Decision ◽  
Areas Of Interest

Recent work has reawakened interest in goal-directed or ‘model-based’ choice, where decisions are based on prospective evaluation of potential action outcomes. Concurrently, there has been growing attention to the role of hierarchy in decision-making and action control. We focus here on the intersection between these two areas of interest, considering the topic of hierarchical model-based control. To characterize this form of action control, we draw on the computational framework of hierarchical reinforcement learning, using this to interpret recent empirical findings. The resulting picture reveals how hierarchical model-based mechanisms might play a special and pivotal role in human decision-making, dramatically extending the scope and complexity of human behaviour.

scholarly journals Multicentric tracking of multiple agents by anterior cingulate cortex during pursuit and evasion

2019 ◽  
Author(s):  
Seng Bum Michael Yoo ◽  
Jiaxin Cindy Tu ◽  
Benjamin Yost Hayden
Keyword(s):  
Motor Control ◽  
Anterior Cingulate Cortex ◽  
Virtual World ◽  
Cingulate Cortex ◽  
Anterior Cingulate ◽  
Dorsal Anterior Cingulate Cortex ◽  
Pursuit Evasion ◽  
Pursuit And Evasion ◽  
High Level ◽  
Hippocampal Complex

SUMMARYSuccessful pursuit and evasion require rapid and precise coordination of navigation with adaptive motor control. We hypothesized that the dorsal anterior cingulate cortex (dACC), which communicates bidirectionally with both the hippocampal complex and premotor/motor areas, would serve a mapping role in this process. We recorded responses of dACC ensembles in two macaques performing a joystick-controlled continuous pursuit/evasion task. We found that dACC multiplexes two sets of signals, (1) world-centric variables that together form a representation of the position and velocity of all relevant agents (self, prey, and predator) in the virtual world, and (2) avatar-centric variables, i.e. self-prey distance and angle. Both sets of variables are multiplexed within an overlapping set of neurons. Our results suggest that dACC may contribute to pursuit and evasion by computing and continuously updating a multicentric representation of the unfolding task state, and support the hypothesis that it plays a high-level abstract role in the control of behavior.

Sign in / Sign up

Export Citation Format

Share Document