BACKGROUND
Across eHealth intervention studies involving children, adolescents, and their parents, researchers have measured users’ experiences to assist with intervention development, refinement, and evaluation. To date, there are no widely agreed-on definitions or measures of ‘user experience’ to support a standardized approach for evaluation and comparison within or across interventions.
OBJECTIVE
We conducted a scoping review with subsequent Delphi consultation to (1) identify how user experience is defined and measured in eHealth research studies, (2) characterize the measurement tools used, and (3) establish working definitions for domains of user experience that could be used in future eHealth evaluations.
METHODS
We systematically searched electronic databases for published and gray literature available from January 1, 2005 to April 11, 2019. Studies assessing an eHealth intervention that targeted any health condition and was designed for use by children, adolescents, and their parents were eligible for inclusion. eHealth interventions needed to be web-, computer-, or mobile-based, mediated by the internet with some degree of interactivity. Studies were also required to report the measurement of ‘user experience’ as first-person experiences, involving cognitive and behavioural factors, reported by intervention users. Two reviewers independently screened studies for relevance and appraised the quality of user experience measures using published criteria: ‘well-established’, ‘approaching well-established’, ‘promising’, or ‘not yet established’. We conducted a descriptive analysis of how user experience was defined and measured in each study. Review findings subsequently informed the survey questions used in the Delphi consultations with eHealth researchers and adolescent users for how user experience should be defined and measured.
RESULTS
Of the 8,634 articles screened for eligibility, 129 and one erratum were included in the review. Thirty eHealth researchers and 27 adolescents participated in the Delphi consultations. Based on the literature and consultations, we proposed working definitions for six main user experience domains: acceptability, satisfaction, credibility, usability, user-reported adherence, and perceived impact. While most studies incorporated a study-specific measure, we identified ten well-established measures to quantify five of the six domains of user experience (all except for self-reported adherence). Our adolescent and researcher participants ranked perceived impact as one of the most important domains of user experience and usability as one of the least important domains. Rankings between adolescents and researchers diverged for other domains.
CONCLUSIONS
Findings highlight the various ways user experience has been defined and measured across studies and what aspects are most valued by researchers and adolescent users. We propose incorporating the working definitions and available measures of user experience to support consistent evaluation and reporting of outcomes across studies. Future studies can refine the definitions and measurement of user experience, explore how user experience relates to other eHealth outcomes, and inform the design and use of human-centred eHealth interventions.
Teach Your Children Well: (How and) Why Design Anthropology Speaks to Our Students
