Feeding the machine: Challenges to reproducible predictive modeling in resting-state connectomics

Abstract In this critical review, we examine the application of predictive models, e.g. classifiers, trained using Machine Learning (ML) to assist in interpretation of functional neuroimaging data. Our primary goal is to summarize how ML is being applied and critically assess common practices. Our review covers 250 studies published using ML and resting-state functional MRI (fMRI) to infer various dimensions of the human functional connectome. Results for hold-out (“lockbox”) performance was, on average, ~13% less accurate than performance measured through cross-validation alone, highlighting the importance of lockbox data which was included in only 16% of the studies. There was also a concerning lack of transparency across the key steps in training and evaluating predictive models. The summary of this literature underscores the importance of the use of a lockbox and highlights several methodological pitfalls that can be addressed by the imaging community. We argue that, ideally, studies are motivated both by the reproducibility and generalizability of findings as well as the potential clinical significance of the insights. We offer recommendations for principled integration of machine learning into the clinical neurosciences with the goal of advancing imaging biomarkers of brain disorders, understanding causative determinants for health risks, and parsing heterogeneous patient outcomes.

Download Full-text

Feeding the machine: challenges to reproducible predictive modeling in network neuroscience

10.31234/osf.io/raqcn ◽

2021 ◽

Author(s):

Andrew Cwiek ◽

Sarah Rajtmajer ◽

Brad Wyble ◽

Vasant Honavar ◽

Frank Hillary

Keyword(s):

Machine Learning ◽

Patient Outcomes ◽

Predictive Models ◽

External Validation ◽

Imaging Biomarkers ◽

Current Application ◽

Prediction Tools ◽

Functional Connectome ◽

Key Steps ◽

Network Neuroscience

Machine learning offers a promising set of prediction tools that have enjoyed more recent application in network neuroscience. In this NETN Perspectives, we examine the current application of predictive models, e.g., classifiers trained using machine learning (ML), within the clinical network neurosciences. Our review covers 118 studies published using ML and functional MRI (fMRI) to infer various dimensions of the human functional connectome. We identify several important methodological challenges in this literature. For example, more than half of the studies focused almost exclusively on maximizing the accuracy of classifying brain functional connectomes into one of several predetermined categories (e.g., disease versus healthy), with significantly less emphasis on reproducibility and generalizability of the findings.. . There was also a concerning lack of transparency across many of the key steps in training and evaluating predictive models using machine learning. The summary of this literature underscores the importance of external validation (i.e., lockbox or test-set data) and highlights several methodological pitfalls that can be addressed by the imaging community. We offer recommendations for the principled application of machine learning in the clinical neurosciences to advance imaging biomarkers, understand causative determinants for health risks and track the trajectory of heterogeneous patient outcomes.

Download Full-text

The neural features in the precentral gyrus predict the severity of internet game disorder: results from the multi-voxel pattern analyses

10.1101/2020.08.26.267989 ◽

2020 ◽

Author(s):

Shuer Ye ◽

Min Wang ◽

Qun Yang ◽

Haohao Dong ◽

Guang-Heng Dong

Keyword(s):

Machine Learning ◽

Predictive Models ◽

Resting State ◽

Internet Gaming Disorder ◽

Resting State Fmri ◽

Brain Regions ◽

Diagnostic Study ◽

Precentral Gyrus ◽

Gaming Disorder ◽

Internet Gaming

AbstractImportanceFinding the neural features that could predict internet gaming disorder severity is important in finding the targets for potential interventions using brain modulation methods.ObjectiveTo determine whether resting-state neural patterns can predict individual variations of internet gaming disorder by applying machine learning method and further investigate brain regions strongly related to IGD severity.DesignThe diagnostic study lasted from December 1, 2013, to November 20, 2019. The data were analyzed from December 31, 2019, to July 10, 2020.SettingThe resting-state fMRI data were collected at East China Normal University, Shanghai.ParticipantsA convenience sample consisting of 402 college students with diverse IGD severityMain Outcomes and MeasuresThe neural patterns were represented by regional homogeneity (ReHo) and the amplitude of low-frequency fluctuation (ALFF). Predictive model performance was assessed by Pearson correlation coefficient and standard mean squared error between the predicted and true IGD severity. The correlations between IGD severity and topological features (i.e., degree centrality (DC), betweenness centrality (BC), and nodal efficiency (NE)) of consensus highly weighted regions in predictive models were examined.ResultsThe final dataset consists of 402 college students (mean [SD] age, 21.43 [2.44] years; 239 [59.5%] male). The predictive models could significantly predict IGD severity (model based on ReHo: r = 0.11, p(r) = 0.030, SMSE = 3.73, p(SMSE) = 0.033; model based on ALFF: r=0.19, p(r) = 0.002, SMSE = 3.58, p(SMSE) = 0.002). The highly weighted brain regions that contributed to both predictive models were the right precentral gyrus and the left postcentral gyrus. Moreover, the topological properties of the right precentral gyrus were significantly correlated with IGD severity (DC: r = 0.16, p = 0.001; BC: r = 0.14, p = 0.005; NE: r = 0.15, p = 0.003) whereas no significant result was found for the left postcentral gyrus (DC: r = 0.02, p = 0.673; BC: r = 0.04, p = 0.432; NE: r = 0.02, p = 0.664).Conclusions and RelevanceThe machine learning models could significantly predict IGD severity from resting-state neural patterns at the individual level. The predictions of IGD severity deepen our understanding of the neural mechanism of IGD and have implications for clinical diagnosis of IGD. In addition, we propose precentral gyrus as a potential target for physiological treatment interventions for IGD.Key PointsQuestionCan machine learning algorithms predict internet gaming disorder (IGD) from resting-state neural patterns?FindingsThis diagnostic study collected resting-state fMRI data from 402 subjects with diverse IGD severity. We found that machine learning models based on resting-state neural patterns yielded significant predictions of IGD severity. In addition, the topological neural features of precentral gyrus, which is a consensus highly weighted region, is significantly correlated with IGD severity.MeaningThe study found that IGD is a distinctive disorder and its dependence severity could be predicted by brain features. The precentral gyrus and its connection with other brain regions could be view as targets for potential IGD intervention, especially using brain modulation methods.

Download Full-text

Disentangling data dependency using cross-validation strategies to evaluate prediction quality of cattle grazing activities using machine learning algorithms and wearable sensor data

Journal of Animal Science ◽

10.1093/jas/skab206 ◽

2021 ◽

Author(s):

Leonardo Augusto Coelho Ribeiro ◽

Tiago Bresolin ◽

Guilherme Jordão de Magalhães Rosa ◽

Daniel Rume Casagrande ◽

Marina de Arruda Camargo Danes ◽

...

Keyword(s):

Machine Learning ◽

Predictive Models ◽

Prediction Accuracy ◽

Cross Validation ◽

Machine Learning Algorithms ◽

Grazing Management ◽

Sensor Data ◽

Accelerometer Data ◽

Wearable Sensor ◽

Prediction Quality

Abstract Wearable sensors have been explored as an alternative for real-time monitoring of cattle feeding behavior in grazing systems. To evaluate the performance of predictive models such as machine learning (ML) techniques, data cross-validation (CV) approaches are often employed. However, due to data dependencies and confounding effects, poorly performed validation strategies may significantly inflate the prediction quality. In this context, our objective was to evaluate the effect of different CV strategies on the prediction of grazing activities in cattle using wearable sensor (accelerometer) data and ML algorithms. Six Nellore bulls (average live weight of 345 ± 21 kg) had their behavior visually classified as grazing or not-grazing for a period of 15 days. Elastic Net Generalized Linear Model (GLM), Random Forest (RF), and Artificial Neural Network (ANN) were employed to predict grazing activity (grazing or not-grazing) using 3-axis accelerometer data. For each analytical method, three CV strategies were evaluated: holdout, leave-one-animal-out (LOAO), and leave-one-day-out (LODO). Algorithms were trained using similar dataset sizes (holdout: n = 57,862; LOAO: n = 56,786; LODO: n = 56,672). Overall, GLM delivered the worst prediction accuracy (53%) compared to the ML techniques (65% for both RF and ANN), and ANN performed slightly better than RF for LOAO (73%) and LODO (64%) across CV strategies. The holdout yielded the highest nominal accuracy values for all three ML approaches (GLM: 59%, RF: 76%, and ANN: 74%), followed by LODO (GLM: 49%, RF: 61%, and ANN: 63%) and LOAO (GLM: 52%, RF: 57%, and ANN: 57%). With a larger dataset (i.e., more animals and grazing management scenarios), it is expected that accuracy could be increased. Most importantly, the greater prediction accuracy observed for holdout CV may simply indicate a lack of data independence and the presence of carry-over effects from animals and grazing management. Our results suggest that generalizing predictive models to unknown (not used for training) animals or grazing management may incur poor prediction quality. The results highlight the need for using management knowledge to define the validation strategy that is closer to the real-life situation, i.e., the intended application of the predictive model.

Download Full-text

Comparison of machine-learning algorithms to build a predictive model for detecting undiagnosed diabetes - ELSA-Brasil: accuracy study

Sao Paulo Medical Journal ◽

10.1590/1516-3180.2016.0309010217 ◽

2017 ◽

Vol 135 (3) ◽

pp. 234-246 ◽

Cited By ~ 20

Author(s):

André Rodrigues Olivera ◽

Valter Roesler ◽

Cirano Iochpe ◽

Maria Inês Schmidt ◽

Álvaro Vigo ◽

...

Keyword(s):

Machine Learning ◽

Logistic Regression ◽

Predictive Models ◽

Cross Validation ◽

Learning Algorithms ◽

Machine Learning Algorithms ◽

Undiagnosed Diabetes ◽

Generalization Testing ◽

Tenfold Cross Validation ◽

Using Data

ABSTRACT CONTEXT AND OBJECTIVE: Type 2 diabetes is a chronic disease associated with a wide range of serious health complications that have a major impact on overall health. The aims here were to develop and validate predictive models for detecting undiagnosed diabetes using data from the Longitudinal Study of Adult Health (ELSA-Brasil) and to compare the performance of different machine-learning algorithms in this task. DESIGN AND SETTING: Comparison of machine-learning algorithms to develop predictive models using data from ELSA-Brasil. METHODS: After selecting a subset of 27 candidate variables from the literature, models were built and validated in four sequential steps: (i) parameter tuning with tenfold cross-validation, repeated three times; (ii) automatic variable selection using forward selection, a wrapper strategy with four different machine-learning algorithms and tenfold cross-validation (repeated three times), to evaluate each subset of variables; (iii) error estimation of model parameters with tenfold cross-validation, repeated ten times; and (iv) generalization testing on an independent dataset. The models were created with the following machine-learning algorithms: logistic regression, artificial neural network, naïve Bayes, K-nearest neighbor and random forest. RESULTS: The best models were created using artificial neural networks and logistic regression. These achieved mean areas under the curve of, respectively, 75.24% and 74.98% in the error estimation step and 74.17% and 74.41% in the generalization testing step. CONCLUSION: Most of the predictive models produced similar results, and demonstrated the feasibility of identifying individuals with highest probability of having undiagnosed diabetes, through easily-obtained clinical data.

Download Full-text

Inter-subject pattern analysis: a straightforward and powerful scheme for group-level MVPA

10.1101/587899 ◽

2019 ◽

Author(s):

Qi Wang ◽

Bastien Cagna ◽

Thierry Chaminade ◽

Sylvain Takerkart

Keyword(s):

Cross Validation ◽

Pattern Analysis ◽

Functional Neuroimaging ◽

Individual Variability ◽

Group Level ◽

Multivariate Pattern Analysis ◽

Neuroimaging Data ◽

Partial Overlap ◽

Multivariate Pattern ◽

Level Analysis

AbstractMultivariate pattern analysis (MVPA) has become vastly popular for analyzing functional neuroimaging data. At the group level, two main strategies are used in the literature. The standard one is hierarchical, combining the outcomes of within-subject decoding results in a second-level analysis. The alternative one, inter-subject pattern analysis, directly works at the group-level by using, e.g, a leave-one-subject-out cross-validation. This study provides a thorough comparison of these two group-level decoding schemes, using both a large number of artificial datasets where the size of the multivariate effect and the amount of inter-individual variability are parametrically controlled, as well as two real fMRI datasets comprising respectively 15 and 39 subjects. We show that these two strategies uncover distinct significant regions with partial overlap, and that inter-subject pattern analysis is able to detect smaller effects and to facilitate the interpretation. The core source code and data are openly available, allowing to fully reproduce most of these results.

Download Full-text

An Evaluation of the Pipeline Framework for Ethical Considerations in Machine Learning Healthcare Applications: The Case of Prediction from Functional Neuroimaging Data

The American Journal of Bioethics ◽

10.1080/15265161.2020.1820110 ◽

2020 ◽

Vol 20 (11) ◽

pp. 56-58

Author(s):

Dawson J. Overton

Keyword(s):

Machine Learning ◽

Functional Neuroimaging ◽

Ethical Considerations ◽

Healthcare Applications ◽

Neuroimaging Data

Download Full-text

Examining early structural and functional brain alterations in postpartum depression through multimodal neuroimaging

Scientific Reports ◽

10.1038/s41598-021-92882-w ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Patricia Schnakenberg ◽

Lisa Hahn ◽

Susanne Stickel ◽

Elmar Stickeler ◽

Ute Habel ◽

...

Keyword(s):

Postpartum Depression ◽

Resting State ◽

Functional Neuroimaging ◽

Adjustment Disorder ◽

Depression Score ◽

Control Group ◽

Resting State Functional Connectivity ◽

Baby Blues ◽

Neuroimaging Data ◽

And Control

AbstractPostpartum depression (PPD) affects approximately 1 in 10 women after childbirth. A thorough understanding of a preexisting vulnerability to PPD will likely aid the early detection and treatment of PPD. Using a within-sample association, the study examined whether the brain’s structural and functional alterations predict the onset of depression. 157 euthymic postpartum women were subjected to a multimodal MRI scan within the first 6 days of childbirth and were followed up for 12 weeks. Based on a clinical interview 12 weeks postpartum, participants were classified as mentally healthy or having either PPD or adjustment disorder (AD). Voxel-based morphometry and resting-state functional connectivity comparisons were performed between the three groups. 13.4% of women in our study developed PPD (n = 21) and 12.1% (n = 19) adjustment disorder (AD). The risk factors for PPD were a psychiatric history and the experience and severity of baby blues and the history of premenstrual syndrome. Despite the different risk profiles, no differences between the PPD, AD and control group were apparent based on structural and functional neuroimaging data immediately after childbirth. At 12 weeks postpartum, a significant association was observed between Integrated Local Correlation (LCor) and the Edinburgh Postnatal Depression Score (EPDS). Our findings do not support the notion that the brain’s structural and resting-state functional alterations, if present, can be used as an early biomarker of PPD or AD. However, effects may become apparent if continuous measures of symptom severity are chosen.

Download Full-text

Two Distinct Neuronal Networks Mediate the Awareness of Environment and of Self

Journal of Cognitive Neuroscience ◽

10.1162/jocn.2010.21488 ◽

2011 ◽

Vol 23 (3) ◽

pp. 570-578 ◽

Cited By ~ 221

Author(s):

Audrey Vanhaudenhuyse ◽

Athena Demertzi ◽

Manuel Schabus ◽

Quentin Noirhomme ◽

Serge Bredart ◽

...

Keyword(s):

Healthy Volunteers ◽

Resting State ◽

Functional Neuroimaging ◽

A Priori ◽

External Input ◽

Anterior Cingulate ◽

Switching Frequency ◽

Brain Areas ◽

Neuroimaging Data ◽

Parietal Cortices

Evidence from functional neuroimaging studies on resting state suggests that there are two distinct anticorrelated cortical systems that mediate conscious awareness: an “extrinsic” system that encompasses lateral fronto-parietal areas and has been linked with processes of external input (external awareness), and an “intrinsic” system which encompasses mainly medial brain areas and has been associated with internal processes (internal awareness). The aim of our study was to explore the neural correlates of resting state by providing behavioral and neuroimaging data from healthy volunteers. With no a priori assumptions, we first determined behaviorally the relationship between external and internal awareness in 31 subjects. We found a significant anticorrelation between external and internal awareness with a mean switching frequency of 0.05 Hz (range: 0.01–0.1 Hz). Interestingly, this frequency is similar to BOLD fMRI slow oscillations. We then evaluated 22 healthy volunteers in an fMRI paradigm looking for brain areas where BOLD activity correlated with “internal” and “external” scores. Activation of precuneus/posterior cingulate, anterior cingulate/mesiofrontal cortices, and parahippocampal areas (“intrinsic system”) was linearly linked to intensity of internal awareness, whereas activation of lateral fronto-parietal cortices (“extrinsic system”) was linearly associated with intensity of external awareness.

Download Full-text

Examining early structural and functional brain alterations in postpartum depression through multimodal neuroimaging

10.31234/osf.io/w98cq ◽

2021 ◽

Author(s):

Patricia Schnakenberg ◽

Lisa Hahn ◽

Susanne Stickel ◽

Elmar Stickeler ◽

Ute Habel ◽

...

Keyword(s):

Postpartum Depression ◽

Resting State ◽

Functional Neuroimaging ◽

Adjustment Disorder ◽

Depression Score ◽

Control Group ◽

Resting State Functional Connectivity ◽

Baby Blues ◽

Neuroimaging Data ◽

And Control

Background: Postpartum depression (PPD) affects approximately 1 in 10 women after childbirth. A thorough understanding of a preexisting vulnerability for PPD will likely aid the early detection and treatment of PPD and help minimize its debilitating effects.Methods: Using a within-sample association, the study aimed to determine whether the brain’s structural and functional alterations predict the onset of depression. To that end, 157 euthymic postpartum women were subjected to a multimodal MRI scan within the first 6 days of childbirth, and were subsequently followed up for 12 weeks. Based on a clinical interview 12 weeks postpartum, participants were classified as mentally healthy or having either PPD or adjustment disorder (AD). Voxel-based morphometry and resting-state functional connectivity comparisons were performed between the three groups. Results: 13.4% of women in our study developed PPD (n=21) and 12.1% (n=19) adjustment disorder (AD). The risk factors for PPD were a psychiatric history and the experience and severity of baby blues and the history of premenstrual syndrome. Despite the different risk profiles, no differences between the PPD, AD and control group were apparent based on the structural and functional neuroimaging data. At 12 weeks postpartum, a significant association was observed between Integrated Local Correlation (LCor) and the Edinburgh Postnatal Depression Score (EPDS). Conclusion: Our findings do not support the notion that the brain’s structural and resting-state functional alterations, if present, can be used as an early biomarker of PPD or AD. However, effects may become apparent if continuous measures of symptom severity are chosen.

Download Full-text