Crowdsourced feature tagging for scalable and privacy-preserved autism diagnosis

ABSTRACT Standard medical diagnosis of mental health conditions often requires licensed experts who are increasingly outnumbered by those at risk, limiting reach. We test the hypothesis that a trustworthy crowd of non-experts can efficiently label features needed for accurate machine learning detection of the common childhood developmental disorder autism. We implement a novel process for creating a trustworthy distributed workforce for video feature extraction, selecting a workforce of 102 workers from a pool of 1,107. Two previously validated binary autism logistic regression classifiers were used to evaluate the quality of the curated crowd’s ratings on unstructured home videos. A clinically representative balanced sample (N=50 videos) of videos were evaluated with and without face box and pitch shift privacy alterations, with AUROC and AUPRC scores >0.98. With both privacy-preserving modifications, sensitivity is preserved (96.0%) while maintaining specificity (80.0%) and accuracy (88.0%) at levels that exceed classification methods without alterations. We find that machine learning classification from features extracted by a curated nonexpert crowd achieves clinical performance for pediatric autism videos and maintains acceptable performance when privacy-preserving mechanisms are applied. These results suggest that privacy-based crowdsourcing of short videos can be leveraged for rapid and mobile assessment of behavioral health.

Using mobile technology in assessment of entrustable professional activities in undergraduate medical education

Background The adoption of competency-based medical education requires objective assessments of a learner’s capability to carry out clinical tasks within workplace-based learning settings. This study involved an evaluation of the use of mobile technology to record entrustable professional activity assessments in an undergraduate clerkship curriculum. Approach A paper-based form was adapted to a mobile platform called eClinic Card. Students documented workplace-based assessments throughout core clerkship and preceptors confirmed accuracy via mobile phones. Assessment scores for the 2017–2018 academic year were collated and analyzed for all core rotations, and preceptors and students were surveyed regarding the mobile assessment experience. Evaluation The mobile system enabled 80 students and 624 preceptors to document 6850 assessment submissions across 47 clinical sites over a 48-week core clerkship curriculum. Students’ scores demonstrated progressive improvement across all entrustable professional activities with stage-appropriate levels of independence reported by end of core clerkship. Preceptors and students were satisfied with ease of use and dependability of the mobile assessment platform; however, students felt quality of formative coaching feedback could be improved. Reflection Our preliminary evaluation suggests the use of mobile technology to assess entrustable professional activity achievement across a core clerkship curriculum is a feasible and acceptable modality for workplace-based assessment. The use of mobile technology supported a programmatic assessment approach. However, meaningful coaching feedback, as well as faculty development and support, emerged as key factors influencing successful adoption and usage of entrustable professional activities within an undergraduate medical curriculum.

Frequent continuous mobile assessment of health behaviors in a multi-year study: acceptability, validity, and engagement with mNCANDA (Preprint)

BACKGROUND Longitudinal studies of many health behaviors often rely on infrequent self-report assessments. The measurement of psychoactive substance use among youth is expected to improve with more frequent mobile assessment, which can reduce recall bias. Researchers have used mobile devices for longitudinal research, but studies that last years and assess youth continuously at a fine-grained temporal level (e.g., weekly) are rare. A tailored mobile app (mNCANDA) and brief assessment protocol were designed specifically to provide a feasible platform to elicit responses to health behavior assessments in longitudinal studies, including the National Consortium on Alcohol and Neurodevelopment in Adolescence (NCANDA). OBJECTIVE This study aimed to determine if an acceptable mobile app system could provide repeatable and valid assessment of health behaviors for youth in different developmental stages over extended follow-up. METHODS Participants were recruited (N = 534, ages 17-28 years) from a larger longitudinal study of neurodevelopment. Participants used the mNCANDA mobile app to register reports of their behaviors for up to 18 months. Response rates as a function of time being measured with mNCANDA, and participant age were modeled using generalized estimating equations to evaluate response rate stability and age effects. mNCANDA-captured substance use reports were compared to responses from standardized interviews to assess concurrent validity. Reactivity was assessed by evaluating patterns of change in substance use after participants initiated weekly reports via mNCANDA. All participants were invited to complete a one-time questionnaire that assessed attitudes and perceptions related to mNCANDA. Qualitative interviews were conducted with a subset of participants who used the app for at least 1-month to obtain feedback on user experience, user-derived explanations of some quantitative results, and suggestions for system improvements. RESULTS mNCANDA protocol adherence was high (Mresponse rate = 82%, SD=27%) and stable over time, across age. Median time to complete each assessment was 51 seconds (Mresponse time= 1.14 min, SD= 1.03 min). Comparisons between mNCANDA and interview self-report on recent (prior 30 days) alcohol and cannabis use days demonstrate close agreement (e.g., within 1 day of reported use) for most observations. Models used to identify reactivity failed to detect changes in substance use patterns subsequent to enrolling in mNCANDA app assessments (P’s > .39). The majority (84%) of participants across the age range reported finding the mNCANDA system acceptable. Participants provided recommendations for improving the system (e.g., tailoring signaling times). CONCLUSIONS mNCANDA provides a feasible, multi-year, continuous fine-grained (e.g., weekly) assessment of health behaviors designed to minimize respondent burden and provides acceptable regimes for long-term self-report of health behaviors. Fine-grained characterization of variability in behaviors over relatively long periods (e.g., up to 18 months) may, through the use of mNCANDA, improve our understanding of the relationship between substance use exposure and neurocognitive development. CLINICALTRIAL

Cognition in Context: Understanding the Everyday Predictors of Cognitive Performance in a New Era of Measurement

Background Research suggests that variability in attention and working memory scores, as seen across time points, may be a sensitive indicator of impairment compared with a singular score at one point in time. Given that fluctuation in cognitive performance is a meaningful metric of real-world function and trajectory, it is valuable to understand the internal state-based and environmental factors that could be driving these fluctuations in performance. Objective In this viewpoint, we argue for the use of repeated mobile assessment as a way to better understand how context shapes moment-to-moment cognitive performance. To elucidate potential factors that give rise to intraindividual variability, we highlight existing literature that has linked both internal and external modifying variables to a number of cognitive domains. We identify ways in which these variables could be measured using mobile assessment to capture them in ecologically meaningful settings (ie, in daily life). Finally, we describe a number of studies that have already begun to use mobile assessment to measure changes in real time cognitive performance in people’s daily environments and the ways in which this burgeoning methodology may continue to advance the field. Methods This paper describes selected literature on contextual factors that examined how experimentally induced or self-reported contextual variables (ie, affect, motivation, time of day, environmental noise, physical activity, and social activity) related to tests of cognitive performance. We also selected papers that used mobile assessment of cognition; these papers were chosen for their use of high-frequency time-series measurement of cognition using a mobile device. Results Upon review of the relevant literature, it is evident that contextual factors have the potential to meaningfully impact cognitive performance when measured in laboratory and daily life environments. Although this research has shed light on the question of what gives rise to real-life variability in cognitive function (eg, affect and activity), many of the studies were limited by traditional methods of data collection (eg, involving retrospective recall). Furthermore, cognition has often been measured in one domain or in one age group, which does not allow us to extrapolate results to other cognitive domains and across the life span. On the basis of the literature reviewed, mobile assessment of cognition shows high levels of feasibility and validity and could be a useful method for capturing individual cognitive variability in real-world contexts via passive and active measures. Conclusions We propose that, through the use of mobile assessment, there is an opportunity to combine multiple sources of contextual and cognitive data. These data have the potential to provide individualized digital signatures that could improve diagnostic precision and lead to meaningful clinical outcomes in a wide range of psychiatric and neurological disorders.