Reducing bias in trials from reactions to measurement: the MERIT study including developmental work and expert workshop

Background Measurement can affect the people being measured; for example, asking people to complete a questionnaire can result in changes in behaviour (the ‘question–behaviour effect’). The usual methods of conduct and analysis of randomised controlled trials implicitly assume that the taking of measurements has no effect on research participants. Changes in measured behaviour and other outcomes due to measurement reactivity may therefore introduce bias in otherwise well-conducted randomised controlled trials, yielding incorrect estimates of intervention effects, including underestimates. Objectives The main objectives were (1) to promote awareness of how and where taking measurements can lead to bias and (2) to provide recommendations on how best to avoid or minimise bias due to measurement reactivity in randomised controlled trials of interventions to improve health. Methods We conducted (1) a series of systematic and rapid reviews, (2) a Delphi study and (3) an expert workshop. A protocol paper was published [Miles LM, Elbourne D, Farmer A, Gulliford M, Locock L, McCambridge J, et al. Bias due to MEasurement Reactions In Trials to improve health (MERIT): protocol for research to develop MRC guidance. Trials 2018;19:653]. An updated systematic review examined whether or not measuring participants had an effect on participants’ health-related behaviours relative to no-measurement controls. Three new rapid systematic reviews were conducted to identify (1) existing guidance on measurement reactivity, (2) existing systematic reviews of studies that have quantified the effects of measurement on outcomes relating to behaviour and affective outcomes and (3) experimental studies that have investigated the effects of exposure to objective measurements of behaviour on health-related behaviour. The views of 40 experts defined the scope of the recommendations in two waves of data collection during the Delphi procedure. A workshop aimed to produce a set of recommendations that were formed in discussion in groups. Results Systematic reviews – we identified a total of 43 studies that compared interview or questionnaire measurement with no measurement and these had an overall small effect (standardised mean difference 0.06, 95% confidence interval 0.02 to 0.09; n = 104,096, I 2 = 54%). The three rapid systematic reviews identified no existing guidance on measurement reactivity, but we did identify five systematic reviews that quantified the effects of measurement on outcomes (all focused on the question–behaviour effect, with all standardised mean differences in the range of 0.09—0.28) and 16 studies that examined reactive effects of objective measurement of behaviour, with most evidence of reactivity of small effect and short duration. Delphi procedure – substantial agreement was reached on the scope of the present recommendations. Workshop – 14 recommendations and three main aims were produced. The aims were to identify whether or not bias is likely to be a problem for a trial, to decide whether or not to collect further quantitative or qualitative data to inform decisions about if bias is likely to be a problem, and to identify how to design trials to minimise the likelihood of this bias. Limitation The main limitation was the shortage of high-quality evidence regarding the extent of measurement reactivity, with some notable exceptions, and the circumstances that are likely to bring it about. Conclusion We hope that these recommendations will be used to develop new trials that are less likely to be at risk of bias. Future work The greatest need is to increase the number of high-quality primary studies regarding the extent of measurement reactivity. Study registration The first systematic review in this study is registered as PROSPERO CRD42018102511. Funding Funded by the Medical Research Council UK and the National Institute for Health Research as part of the Medical Research Council–National Institute for Health Research Methodology Research Programme.

A systematic review and meta-analysis of studies of reactivity to objective digital measurement of health behaviour

Self-report measures of health behaviour have several limitations including measurement reactivity, i.e. changes in people’s behaviour, cognitions or emotions due to taking part in research. It is thus often recommended to use objective digital measurements instead; however, it is unclear whether they also induce reactivity. Therefore, this pre-registered systematic review and meta-analysis synthesised 31 studies, including 7 experimental studies with 18 effects. Most studies investigated reactivity to digital measurement of physical activity, reporting small but significant effects. Measurement reactivity may be amplified by (1) ease of changing the behaviour, (2) awareness of being measured and social desirability, and (3) resolving discrepancies between actual and desired behaviour through self-regulation. Measurement reactivity may be reduced by discarding the first days of measurement and by restricting visibility of the data. Studies on reactivity to measurement of other health behaviours were scarce, indicating the need for future rigorous experimental research.

Does Assessment Alter Responses? An Examination of Measurement Reactivity in an Ecological Momentary Assessment of Body Comparisons

Introduction: Body comparisons have been implicated in body dissatisfaction and eating disorder development. Ecological momentary assessment (EMA) has been used to capture body comparisons in everyday life; however, the potential for measurement reactivity this approach has on EMA responses has yet to be examined. The present study systematically evaluated measurement reactivity in EMA of body comparisons. Methods: Undergraduate women (N = 75) completed four surveys daily for 11 days; Days 1–4 did not include comparison assessments and Days 5–11 did. Changes were examined in related EMA measures between these time periods and individual difference measures pre- and post-EMA. Results: A within-person MANOVA showed no significant change in the individual difference measures. Multilevel analyses revealed that participants reported fewer comparisons as day of study increased. When they completed EMA with versus without comparison items, they reported fewer occasions of loss of control while eating and less pleasant social company at time of assessment. Discussion: Findings suggest limited concern for reactivity in this approach, though researchers examining body comparisons, loss of control eating, and perceptions of social company should be cautious when interpreting their findings and consider examining reactivity directly. Future research on body comparisons should also consider using run-in periods to improve data quality.

A Novel Combination of Accelerometry and Ecological Momentary Assessment for Post-Stroke Paretic Arm/Hand Use: Feasibility and Validity

Use of the paretic arm and hand is a key indicator of recovery and reintegration after stroke. A sound methodology is essential to comprehensively identify the possible factors impacting daily arm/hand use behavior. We combined ecological momentary assessment (EMA), a prompt methodology capturing real-time psycho-contextual factors, with accelerometry to investigate arm/hand behavior in the natural environment. Our aims were to determine (1) feasibility and (2) measurement validity of the combined methodology. We monitored 30 right-dominant, mild-moderately motor impaired chronic stroke survivors over 5 days (6 EMA prompts/day with accelerometers on each wrist). We observed high adherence for accelerometer wearing time (80.3%), EMA prompt response (84.6%), and generally positive user feedback upon exit interview. The customized prompt schedule and the self-triggered prompt option may have improved adherence. There was no evidence of EMA response bias nor immediate measurement reactivity. An unexpected small but significant increase in paretic arm/hand use was observed over days (12–14 min), which may be the accumulated effect of prompting that provided a reminder to choose the paretic limb. Further research that uses this combined methodology is needed to develop targeted interventions that effectively change behavior and enable reintegration post-stroke.

Measurement Reactivity of Accelerometer-Based Sedentary Behavior and Physical Activity in 2 Assessment Periods

Background: The purposes of this study were to examine accelerometer measurement reactivity (AMR) in sedentary behavior (SB), physical activity (PA), and accelerometer wear time in 2 measurement periods and to quantify AMR as a human-related source of bias for the reproducibility of SB and PA estimates. Methods: In total, 136 participants (65% women, mean age = 54.6 y) received 7-day accelerometry at the baseline and after 12 months. Latent growth models were used to identify AMR. Intraclass correlations were calculated to examine the reproducibility using 2-level mixed-effects linear regression analyses. Results: Within each 7-day accelerometry assessment, the participants increased their time spent in SB (b = 2.4 min/d; b = 3.8 min/d) and reduced their time spent in light PA (b = −2.0 min/d; b = −3.2 min/d), but did not change moderate to vigorous PA. The participants reduced their wear time (b = −5.2 min/d) only at the baseline. The intraclass correlations ranged from .42 for accelerometer wear time to .74 for SB. The AMR was not identified as a source of bias in any regression model. Conclusions: AMR may influence SB and PA estimates differentially. Although 7-day accelerometry seems to be a reproducible measure, our findings highlight accelerometer wear time as a crucial confounder in analyzing SB and PA data.

“Asking Too Much?”: Randomized N-of-1 Trial Exploring Patient Preferences and Measurement Reactivity to Frequent Use of Remote Multidimensional Pain Assessments in Children and Young People With Juvenile Idiopathic Arthritis

Background Remote monitoring of pain using multidimensional mobile health (mHealth) assessment tools is increasingly being adopted in research and care. This assessment method is valuable because it is challenging to capture pain histories, particularly in children and young people in diseases where pain patterns can be complex, such as juvenile idiopathic arthritis (JIA). With the growth of mHealth measures and more frequent assessment, it is important to explore patient preferences for the timing and frequency of administration of such tools and consider whether certain administrative patterns can directly impact on children’s pain experiences. Objective This study aimed to explore the feasibility and influence (in terms of objective and subjective measurement reactivity) of several time sampling strategies in remote multidimensional pain reporting. Methods An N-of-1 trial was conducted in a subset of children and young people with JIA and their parents recruited to a UK cohort study. Children were allocated to 1 of 4 groups. Each group followed a different schedule of completion of MPT for 8 consecutive weeks. Each schedule included 2 blocks, each comprising 4 different randomized time sampling strategies, with each strategy occurring once within each 4-week block. Children completed MPT according to time sampling strategies: once-a-day, twice-a-day, once-a-week, and as-and-when pain was experienced. Adherence to each strategy was calculated. Participants completed the Patient-Reported Outcomes Measurement Information System Pain Interference Scale at the end of each week to explore objective reactivity. Differences in pain interference scores between time sampling strategies were assessed graphically and using Friedman tests. Children and young people and their parents took part in a semistructured interview about their preferences for different time sampling strategies and to explore subjective reactivity. Results A total of 14 children and young people (aged 7-16 years) and their parents participated. Adherence to pain reporting was higher in less intense time sampling strategies (once-a-week=63% [15/24]) compared with more intense time sampling strategies (twice-a-day=37.8% [127/336]). There were no statistically significant differences in pain interference scores between sampling strategies. Qualitative findings from interviews suggested that children preferred once-a-day (6/14, 43%) and as-and-when pain reporting (6/14, 43%). Creating routine was one of the most important factors for successful reporting, while still ensuring that comprehensive information about recent pain was captured. Conclusions Once-a-day pain reporting provides rich contextual information. Although patients were less adherent to this preferred sampling strategy, once-a-day reporting still provides more frequent assessment opportunities compared with other less intense or overburdensome schedules. Important issues for the design of studies and care incorporating momentary assessment techniques were identified. We demonstrate that patient reporting preferences are key to accommodate and are important where data capture quality is key. Our findings support frequent administration of such tools, using daily reporting methods where possible.