Moving Beyond Processing and Analysis-Related Variation in Neuroscience
AbstractWhen fields lack consensus standards and ground truths for their analytic methods, reproducibility tends to be more of an ideal than a reality. Such has been the case for functional neuroimaging, where there exists a sprawling space of tools from which scientists can construct processing pipelines and draw interpretations. We provide a critical evaluation of the impact of differences observed in results across five independently developed functional MRI minimal preprocessing pipelines. We show that even when handling the same exact data, inter-pipeline agreement was only moderate, with the specific steps that contribute to the lack of agreement varying across pipeline comparisons. Using a densely sampled test-retest dataset, we show that the limitations imposed by inter-pipeline agreement mainly become appreciable when the reliability of the underlying data is high. We highlight the importance of comparison among analytic tools and parameters, as both widely debated (e.g., global signal regression) and commonly overlooked (e.g., MNI template version) decisions were each found to lead to marked variation. We provide recommendations for incorporating tool-based variability in functional neuroimaging analyses and a supporting infrastructure.