Task-based measures that capture neurocognitive processes can help bridge the gap between brain and behavior. To transfer tasks to clinical application, reliability is a crucial benchmark because it poses an upper bound to potential correlations with other variables (e.g., symptom or brain data). However, the reliability of many task-readouts is low. In this study, we scrutinized the re-test reliability of a probabilistic reversal learning task (PRLT) that is frequently used to characterize cognitive flexibility in psychiatric populations. We analyzed data from N=40 healthy subjects, who completed the PRLT twice. We focused on how individual metrics are derived, i.e., whether data was partially pooled across participants and whether priors were used to inform estimates. We compared the reliability of the resulting indices across sessions, as well as the internal consistency of a selection of indices. We find good to excellent reliability for behavioral indices as derived from mixed-effects models that include data from both sessions. The internal consistency was good to excellent. For indices derived from computational modelling, we find excellent reliability when using hierarchical estimation with empirical priors and including data from both sessions. Our results indicate that the PRLT is well equipped to measure individual differences of cognitive flexibility in reinforcement learning. However, this depends heavily on hierarchical modelling of the longitudinal data (whether sessions are modelled separately or jointly), on estimation methods, and the combination of parameters included in computational models. We discuss implications for the applicability of PRLT indices in psychiatric research and as diagnostic tools.
Brief Report: Feasibility of the Probabilistic Reversal Learning Task as an Outcome Measure in an Intervention Trial for Individuals with Autism Spectrum Disorder