scholarly journals Offline Replay Supports Planning: fMRI Evidence from Reward Revaluation

2017 ◽  
Author(s):  
Ida Momennejad ◽  
A. Ross Otto ◽  
Nathaniel D. Daw ◽  
Kenneth A. Norman
Keyword(s):  
Machine Learning ◽  
Computational Cost ◽  
Decision Time ◽  
Anterior Cingulate ◽  
Prediction Errors ◽  
Activity Increase ◽  
Planning Methods ◽  
Goal Directed Behavior ◽  
The Brain

AbstractMaking decisions in sequentially structured tasks requires integrating distally acquired information. The extensive computational cost of such integration challenges planning methods that integrate online, at decision time. Furthermore, it remains unclear whether “offline” integration during replay supports planning, and if so which memories should be replayed. Inspired by machine learning, we propose that (a) offline replay of trajectories facilitates integrating representations that guide decisions, and (b) unsigned prediction errors (uncertainty) trigger such integrative replay. We designed a 2-step revaluation task for fMRI, whereby participants needed to integrate changes in rewards with past knowledge to optimally replan decisions. As predicted, we found that (a) multi-voxel pattern evidence for off-task replay predicts subsequent replanning; (b) neural sensitivity to uncertainty predicts subsequent replay and replanning; (c) off-task hippocampus and anterior cingulate activity increase when revaluation is required. These findings elucidate how the brain leverages offline mechanisms in planning and goal-directed behavior under uncertainty.

scholarly journals Offline replay supports planning in human reinforcement learning

eLife ◽  
2018 ◽  
Vol 7 ◽  
Author(s):  
Ida Momennejad ◽  
A Ross Otto ◽  
Nathaniel D Daw ◽  
Kenneth A Norman
Keyword(s):  
Machine Learning ◽  
Reinforcement Learning ◽  
Computational Cost ◽  
Decision Time ◽  
Anterior Cingulate ◽  
Prediction Errors ◽  
Activity Increase ◽  
Planning Methods ◽  
Goal Directed Behavior ◽  
The Brain

Making decisions in sequentially structured tasks requires integrating distally acquired information. The extensive computational cost of such integration challenges planning methods that integrate online, at decision time. Furthermore, it remains unclear whether ‘offline’ integration during replay supports planning, and if so which memories should be replayed. Inspired by machine learning, we propose that (a) offline replay of trajectories facilitates integrating representations that guide decisions, and (b) unsigned prediction errors (uncertainty) trigger such integrative replay. We designed a 2-step revaluation task for fMRI, whereby participants needed to integrate changes in rewards with past knowledge to optimally replan decisions. As predicted, we found that (a) multi-voxel pattern evidence for off-task replay predicts subsequent replanning; (b) neural sensitivity to uncertainty predicts subsequent replay and replanning; (c) off-task hippocampus and anterior cingulate activity increase when revaluation is required. These findings elucidate how the brain leverages offline mechanisms in planning and goal-directed behavior under uncertainty.

scholarly journals Precision-weighting of superior frontal cortex unsigned prediction error signals benefits learning, is mediated by dopamine, and is impaired in psychosis

2019 ◽  
Author(s):  
J. Haarsma ◽  
P.C. Fletcher ◽  
J.D. Griffin ◽  
H.J. Taverne ◽  
H. Ziauddeen ◽  
...  
Keyword(s):  
Prediction Error ◽  
Computational Modelling ◽  
Anterior Cingulate ◽  
Prediction Errors ◽  
Healthy Individuals ◽  
Cortical Function ◽  
Dopaminergic Modulation ◽  
Hierarchical Bayesian Inference ◽  
The Brain ◽  
Superior Frontal Cortex

AbstractRecent theories of cortical function construe the brain as performing hierarchical Bayesian inference. According to these theories, the precision of cortical unsigned prediction error (i.e., surprise) signals plays a key role in learning and decision-making, to be controlled by dopamine, and to contribute to the pathogenesis of psychosis. To test these hypotheses, we studied learning with variable outcome-precision in healthy individuals after dopaminergic modulation and in patients with early psychosis. Behavioural computational modelling indicated that precision-weighting of unsigned prediction errors benefits learning in health, and is impaired in psychosis. FMRI revealed coding of unsigned prediction errors relative to their precision in bilateral superior frontal gyri and dorsal anterior cingulate, which was perturbed by dopaminergic modulation, impaired in psychosis, and associated with task performance and schizotypy. We conclude that precision-weighting of cortical prediction error signals is a key mechanism through which dopamine modulates inference and contributes to the pathogenesis of psychosis.

scholarly journals Neural Representation of Costs and Rewards in Decision Making

2021 ◽  
Vol 11 (8) ◽  
pp. 1096
Author(s):  
Yixuan Chen
Keyword(s):  
Decision Making ◽  
Anterior Cingulate Cortex ◽  
Orbitofrontal Cortex ◽  
Internal State ◽  
Cingulate Cortex ◽  
Brain Regions ◽  
Anterior Cingulate ◽  
Neural Representation ◽  
Prediction Errors ◽  
The Brain

Decision making is crucial for animal survival because the choices they make based on their current situation could influence their future rewards and could have potential costs. This review summarises recent developments in decision making, discusses how rewards and costs could be encoded in the brain, and how different options are compared such that the most optimal one is chosen. The reward and cost are mainly encoded by the forebrain structures (e.g., anterior cingulate cortex, orbitofrontal cortex), and their value is updated through learning. The recent development on dopamine and the lateral habenula’s role in reporting prediction errors and instructing learning will be emphasised. The importance of dopamine in powering the choice and accounting for the internal state will also be discussed. While the orbitofrontal cortex is the place where the state values are stored, the anterior cingulate cortex is more important when the environment is volatile. All of these structures compare different attributes of the task simultaneously, and the local competition of different neuronal networks allows for the selection of the most appropriate one. Therefore, the total value of the task is not encoded as a scalar quantity in the brain but, instead, as an emergent phenomenon, arising from the computation at different brain regions.

scholarly journals Statistical and Machine Learning Link Selection Methods for Brain Functional Networks: Review and Comparison

2021 ◽  
Vol 11 (6) ◽  
pp. 735
Author(s):  
Ilinka Ivanoska ◽  
Kire Trivodaliev ◽  
Slobodan Kalajdziski ◽  
Massimiliano Zanin
Keyword(s):  
Machine Learning ◽  
Computational Cost ◽  
Learning Task ◽  
Functional Networks ◽  
Large Set ◽  
Selection Methods ◽  
Machine Learning Methods ◽  
Brain Functional Networks ◽  
The Cost ◽  
The Brain

Network-based representations have introduced a revolution in neuroscience, expanding the understanding of the brain from the activity of individual regions to the interactions between them. This augmented network view comes at the cost of high dimensionality, which hinders both our capacity of deciphering the main mechanisms behind pathologies, and the significance of any statistical and/or machine learning task used in processing this data. A link selection method, allowing to remove irrelevant connections in a given scenario, is an obvious solution that provides improved utilization of these network representations. In this contribution we review a large set of statistical and machine learning link selection methods and evaluate them on real brain functional networks. Results indicate that most methods perform in a qualitatively similar way, with NBS (Network Based Statistics) winning in terms of quantity of retained information, AnovaNet in terms of stability and ExT (Extra Trees) in terms of lower computational cost. While machine learning methods are conceptually more complex than statistical ones, they do not yield a clear advantage. At the same time, the high heterogeneity in the set of links retained by each method suggests that they are offering complementary views to the data. The implications of these results in neuroscience tasks are finally discussed.

scholarly journals A machine learning approach for efficient multi-dimensional integration

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Boram Yoon
Keyword(s):  
Machine Learning ◽  
Regression Model ◽  
Dimensional Space ◽  
Computational Cost ◽  
Gaussian Process Regression ◽  
Gradient Boosting ◽  
Prediction Errors ◽  
Integration Algorithm ◽  
Order Of Magnitude ◽  
Physics Problems

AbstractMany physics problems involve integration in multi-dimensional space whose analytic solution is not available. The integrals can be evaluated using numerical integration methods, but it requires a large computational cost in some cases, so an efficient algorithm plays an important role in solving the physics problems. We propose a novel numerical multi-dimensional integration algorithm using machine learning (ML). After training a ML regression model to mimic a target integrand, the regression model is used to evaluate an approximation of the integral. Then, the difference between the approximation and the true answer is calculated to correct the bias in the approximation of the integral induced by ML prediction errors. Because of the bias correction, the final estimate of the integral is unbiased and has a statistically correct error estimation. Three ML models of multi-layer perceptron, gradient boosting decision tree, and Gaussian process regression algorithms are investigated. The performance of the proposed algorithm is demonstrated on six different families of integrands that typically appear in physics problems at various dimensions and integrand difficulties. The results show that, for the same total number of integrand evaluations, the new algorithm provides integral estimates with more than an order of magnitude smaller uncertainties than those of the VEGAS algorithm in most of the test cases.

scholarly journals Frontal cortex function derives from hierarchical predictive coding

2016 ◽  
Author(s):  
William H Alexander ◽  
Joshua W Brown
Keyword(s):  
Machine Learning ◽  
Working Memory ◽  
Prefrontal Cortex ◽  
Frontal Cortex ◽  
Predictive Coding ◽  
Memory Storage ◽  
Prediction Errors ◽  
Learning Approaches ◽  
Hierarchical Groups ◽  
Goal Directed Behavior

The frontal lobes are essential for human volition and goal-directed behavior, yet their function remains unclear. While various models have highlighted working memory, reinforcement learning, and cognitive control as key functions, a single framework for interpreting the range of effects observed in prefrontal cortex has yet to emerge. Here we show that a simple computational motif based on predictive coding can be stacked hierarchically to learn and perform arbitrarily complex goal-directed behavior. The resulting Hierarchical Error Representation (HER) model simulates a wide array of findings from fMRI, ERP, single-units, and neuropsychological studies of both lateral and medial prefrontal cortex. Additionally, the model compares favorably with current machine learning approaches, learning more rapidly and with comparable performance, while self-organizing representations into efficient hierarchical groups and managing working memory storage. By reconceptualizing lateral prefrontal activity as anticipating prediction errors, the HER model provides a novel unifying account of prefrontal cortex function with broad implications both for understanding the frontal cortex and building more powerful machine learning applications.

scholarly journals Interneuron specific gamma synchronization indexes cue uncertainty and prediction errors in lateral prefrontal and anterior cingulate cortex

eLife ◽  
2021 ◽  
Vol 10 ◽  
Author(s):  
Kianoush Banaie Boroujeni ◽  
Paul Tiesinga ◽  
Thilo Womelsdorf
Keyword(s):  
Anterior Cingulate Cortex ◽  
Cingulate Cortex ◽  
Suppressive Effect ◽  
Anterior Cingulate ◽  
Inhibitory Interneurons ◽  
Prediction Errors ◽  
Gamma Band Activity ◽  
Winner Take All ◽  
High Degree ◽  
Goal Directed Behavior

Inhibitory interneurons are believed to realize critical gating functions in cortical circuits, but it has been difficult to ascertain the content of gated information for well characterized interneurons in primate cortex. Here, we address this question by characterizing putative interneurons in primate prefrontal and anterior cingulate cortex while monkeys engaged in attention demanding reversal learning. We find that subclasses of narrow spiking neurons have a relative suppressive effect on the local circuit indicating they are inhibitory interneurons. One of these interneuron subclasses showed prominent firing rate modulations and (35-45 Hz) gamma synchronous spiking during periods of uncertainty in both, lateral prefrontal cortex (LPFC) and in anterior cingulate cortex (ACC). In LPFC this interneuron subclass activated when the uncertainty of attention cues was resolved during flexible learning, whereas in ACC it fired and gamma-synchronized when outcomes were uncertain and prediction errors were high during learning. Computational modeling of this interneuron-specific gamma band activity in simple circuit motifs suggests it could reflect a soft winner-take-all gating of information having high degree of uncertainty. Together, these findings elucidate an electrophysiologically-characterized interneuron subclass in the primate, that forms gamma synchronous networks in two different areas when resolving uncertainty during adaptive goal-directed behavior.

Nanosecond Photodynamics Simulations of a Cis-Trans Isomerization Are Enabled by Machine Learning

2020 ◽  
Author(s):  
Jingbai Li ◽  
Patrick Reiser ◽  
André Eberhard ◽  
Pascal Friederich ◽  
Steven Lopez
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Excited State ◽  
Adaptive Sampling ◽  
Computational Cost ◽  
Ground Truth ◽  
Absolute Error ◽  
Photochemical Reactions ◽  
Computational Techniques ◽  
Full Potential

<p>Photochemical reactions are being increasingly used to construct complex molecular architectures with mild and straightforward reaction conditions. Computational techniques are increasingly important to understand the reactivities and chemoselectivities of photochemical isomerization reactions because they offer molecular bonding information along the excited-state(s) of photodynamics. These photodynamics simulations are resource-intensive and are typically limited to 1–10 picoseconds and 1,000 trajectories due to high computational cost. Most organic photochemical reactions have excited-state lifetimes exceeding 1 picosecond, which places them outside possible computational studies. Westermeyr <i>et al.</i> demonstrated that a machine learning approach could significantly lengthen photodynamics simulation times for a model system, methylenimmonium cation (CH<sub>2</sub>NH<sub>2</sub><sup>+</sup>).</p><p>We have developed a Python-based code, Python Rapid Artificial Intelligence <i>Ab Initio</i> Molecular Dynamics (PyRAI<sup>2</sup>MD), to accomplish the unprecedented 10 ns <i>cis-trans</i> photodynamics of <i>trans</i>-hexafluoro-2-butene (CF<sub>3</sub>–CH=CH–CF<sub>3</sub>) in 3.5 days. The same simulation would take approximately 58 years with ground-truth multiconfigurational dynamics. We proposed an innovative scheme combining Wigner sampling, geometrical interpolations, and short-time quantum chemical trajectories to effectively sample the initial data, facilitating the adaptive sampling to generate an informative and data-efficient training set with 6,232 data points. Our neural networks achieved chemical accuracy (mean absolute error of 0.032 eV). Our 4,814 trajectories reproduced the S<sub>1</sub> half-life (60.5 fs), the photochemical product ratio (<i>trans</i>: <i>cis</i> = 2.3: 1), and autonomously discovered a pathway towards a carbene. The neural networks have also shown the capability of generalizing the full potential energy surface with chemically incomplete data (<i>trans</i> → <i>cis</i> but not <i>cis</i> → <i>trans</i> pathways) that may offer future automated photochemical reaction discoveries.</p>

BAND NN: A Deep Learning Framework For Energy Prediction and Geometry Optimization of Organic Small Molecules

2019 ◽  
Author(s):  
Siddhartha Laghuvarapu ◽  
Yashaswi Pathak ◽  
U. Deva Priyakumar
Keyword(s):  
Machine Learning ◽  
Density Functional ◽  
Computational Cost ◽  
Geometry Optimization ◽  
Dft Methods ◽  
Energy Prediction ◽  
Machine Learning Model ◽  
Equilibrium Structures ◽  
High Level ◽  
Non Equilibrium

Recent advances in artificial intelligence along with development of large datasets of energies calculated using quantum mechanical (QM)/density functional theory (DFT) methods have enabled prediction of accurate molecular energies at reasonably low computational cost. However, machine learning models that have been reported so far requires the atomic positions obtained from geometry optimizations using high level QM/DFT methods as input in order to predict the energies, and do not allow for geometry optimization. In this paper, a transferable and molecule-size independent machine learning model (BAND NN) based on a chemically intuitive representation inspired by molecular mechanics force fields is presented. The model predicts the atomization energies of equilibrium and non-equilibrium structures as sum of energy contributions from bonds (B), angles (A), nonbonds (N) and dihedrals (D) at remarkable accuracy. The robustness of the proposed model is further validated by calculations that span over the conformational, configurational and reaction space. The transferability of this model on systems larger than the ones in the dataset is demonstrated by performing calculations on select large molecules. Importantly, employing the BAND NN model, it is possible to perform geometry optimizations starting from non-equilibrium structures along with predicting their energies.

A Review of Various Machine Learning Techniques for Brain Tumor Detection from MRI Images

2020 ◽  
Vol 16 (8) ◽  
pp. 937-945
Author(s):  
Aaishwarya Sanjay Bajaj ◽  
Usha Chouhan
Keyword(s):  
Machine Learning ◽  
Brain Tumor ◽  
Ct Scan ◽  
Imaging Techniques ◽  
Critical Evaluation ◽  
Tumor Detection ◽  
Machine Learning Techniques ◽  
X Rays ◽  
Detection Techniques ◽  
The Brain

Background: This paper endeavors to identify an expedient approach for the detection of the brain tumor in MRI images. The detection of tumor is based on i) review of the machine learning approach for the identification of brain tumor and ii) review of a suitable approach for brain tumor detection. Discussion: This review focuses on different imaging techniques such as X-rays, PET, CT- Scan, and MRI. This survey identifies a different approach with better accuracy for tumor detection. This further includes the image processing method. In most applications, machine learning shows better performance than manual segmentation of the brain tumors from MRI images as it is a difficult and time-consuming task. For fast and better computational results, radiology used a different approach with MRI, CT-scan, X-ray, and PET. Furthermore, summarizing the literature, this paper also provides a critical evaluation of the surveyed literature which reveals new facets of research. Conclusion: The problem faced by the researchers during brain tumor detection techniques and machine learning applications for clinical settings have also been discussed.

Sign in / Sign up

Export Citation Format

Share Document