Measuring implementation fidelity in a cluster-randomized pragmatic trial: development and use of a quantitative multi-component approach

Background In pragmatic trials, on-site partners, rather than researchers, lead intervention delivery, which may result in implementation variation. There is a need to quantitatively measure this variation. Applying the Framework for Implementation Fidelity (FIF), we develop an approach for measuring variability in site-level implementation fidelity. This approach is then applied to measure site-level fidelity in a cluster-randomized pragmatic trial of Music & MemorySM (M&M), a personalized music intervention targeting agitated behaviors in residents living with dementia, in US nursing homes (NHs). Methods Intervention NHs (N = 27) implemented M&M using a standardized manual, utilizing provided staff trainings and iPods for participating residents. Quantitative implementation data, including iPod metadata (i.e., song title, duration, number of plays), were collected during baseline, 4-month, and 8-month site visits. Three researchers developed four FIF adherence dimension scores. For Details of Content, we independently reviewed the implementation manual and reached consensus on six core M&M components. Coverage was the total number of residents exposed to the music at each NH. Frequency was the percent of participating residents in each NH exposed to M&M at least weekly. Duration was the median minutes of music received per resident day exposed. Data elements were scaled and summed to generate dimension-level NH scores, which were then summed to create a Composite adherence score. NHs were grouped by tercile (low-, medium-, high-fidelity). Results The 27 NHs differed in size, resident composition, and publicly reported quality rating. The Composite score demonstrated significant variation across NHs, ranging from 4.0 to 12.0 [8.0, standard deviation (SD) 2.1]. Scaled dimension scores were significantly correlated with the Composite score. However, dimension scores were not highly correlated with each other; for example, the correlation of the Details of Content score with Coverage was τb = 0.11 (p = 0.59) and with Duration was τb = − 0.05 (p = 0.78). The Composite score correlated with CMS quality star rating and presence of an Alzheimer’s unit, suggesting face validity. Conclusions Guided by the FIF, we developed and used an approach to quantitatively measure overall site-level fidelity in a multi-site pragmatic trial. Future pragmatic trials, particularly in the long-term care environment, may benefit from this approach. Trial registration Clinicaltrials.gov NCT03821844. Registered on 30 January 2019, https://clinicaltrials.gov/ct2/show/NCT03821844.

The Electronic Health Record as the Primary Data Source in a Pragmatic Trial: A Case Study

Introduction. Pragmatic clinical trials test interventions in patients representative of real-world medical practice and reduce data collection costs by using data recorded in the electronic health record (EHR) during usual care. We describe our experience using the EHR to measure the primary outcome of a pragmatic trial, hospital readmissions, and important clinical covariates. Methods. The trial enrolled patients recently discharged from the hospital for treatment of heart failure to test whether automated daily monitoring integrated into the EHR could reduce readmissions. The study team used data from the EHR and several data systems that drew on the EHR, supplemented by the hospital admissions files of three states. Results. Almost three-quarters of enrollees’ readmissions over the 12-mo trial period were captured by the EHRs of the study hospitals. State data, which took 7 mo to more than 2 y from first contact to receipt of first data, provided the remaining one-quarter. Considerable expertise was required to resolve differences between the 2 data sources. Common covariates used in trial analyses, such as weight and body mass index during the index hospital stay, were available for >97% of enrollees from the EHR. Ejection fraction, obtained from echocardiograms, was available for only 47.6% of enrollees within the 6-mo window that would likely be expected in a traditional trial. Discussion. In this trial, patient characteristics and outcomes were collected from existing EHR systems, but, as usual for EHRs, they could not be standardized for date or method of measurement and required substantial time and expertise to collect and curate. Hospital admissions, the primary trial outcome, required additional effort to locate and use supplementary sources of data. Highlights Electronic health records are not a single system but a series of overlapping and legacy systems that require time and expertise to use efficiently. Commonly measured patient characteristics such as weight and body mass index are relatively easy to locate for most trial enrollees but less common characteristics, like ejection fraction, are not. Acquiring essential supplementary data—in this trial, state data on hospital admission—can be a lengthy and difficult process.

Fatigue is associated with disease activity in some, but not all, patients living with rheumatoid arthritis: disentangling “between-person” and “within-person” associations

Background Previous research has shown an unclear and inconsistent association between fatigue and disease activity in patients with rheumatoid arthritis (RA). The aim of this study was to explore differences in “between-person” and “within-person” associations between disease activity parameters and fatigue severity in patients with established RA. Methods Baseline and 3-monthly follow-up data up to one-year were used from 531 patients with established RA randomized to stopping (versus continuing) tumor necrosis factor inhibitor treatment enrolled in a large pragmatic trial. Between- and within-patient associations between different indicators of disease activity (C-reactive protein [CRP], erythrocyte sedimentation rate [ESR], swollen and tender joint count [ SJC and TJC], visual analog scale general health [VAS-GH]) and patient-reported fatigue severity (Bristol RA Fatigue Numerical Rating Scale) were disaggregated and estimated using person-mean centering in combination with repeated measures linear mixed modelling. Results Overall, different indices of disease activity were weakly to moderately associated with fatigue severity over time (β’s from 0.121 for SJC to 0.352 for VAS-GH, all p’s < 0.0001). Objective markers of inflammation (CRP, ESR and SJC) were associated weakly with fatigue within patients over time (β’s: 0.104–0.142, p’s < 0.0001), but not between patients. The subjective TJC and VAS-GH were significantly associated with fatigue both within and between patients, but with substantially stronger associations at the between-patient level (β’s: 0.217–0.515, p’s < 0.0001). Within-person associations varied widely for individual patients for all components of disease activity. Conclusion Associations between fatigue and disease activity vary largely for different patients and the pattern of between-person versus within-person associations appears different for objective versus subjective components of disease activity. The current findings explain the inconsistent results of previous research, illustrates the relevance of statistically distinguishing between different types of association in research on the relation between disease activity and fatigue and additionally suggest a need for a more personalized approach to fatigue in RA patients. Trial registration Netherlands trial register, Number NTR3112.

A systematic review of the evidence supporting post-operative antithrombotic use following cardiopulmonary bypass in children with CHD

Objectives: To determine the optimal antithrombotic agent choice, timing of initiation, dosing and duration of therapy for paediatric patients undergoing cardiac surgery with cardiopulmonary bypass. Methods: We used PubMed and EMBASE to systematically review the existing literature of clinical trials involving antithrombotics following cardiac surgery from 2000 to 2020 in children 0–18 years. Studies were assessed by two reviewers to ensure they met eligibility criteria. Results: We identified 10 studies in 1929 children across three medications classes: vitamin K antagonists, cyclooxygenase inhibitors and indirect thrombin inhibitors. Four studies were retrospective, five were prospective observational cohorts (one of which used historical controls) and one was a prospective, randomised, placebo-controlled, double-blind trial. All included were single-centre studies. Eight studies used surrogate biomarkers and two used clinical endpoints as the primary endpoint. There was substantive variability in response to antithrombotics in the immediate post-operative period. Studies of warfarin and aspirin showed that laboratory monitoring levels were frequently out of therapeutic range (variably defined), and findings were mixed on the association of these derangements with bleeding or thrombotic events. Heparin was found to be safe at low doses, but breakthrough thromboembolic events were common. Conclusion: There are few paediatric prospective randomised clinical trials evaluating antithrombotic therapeutics post-cardiac surgery; most studies have been observational and seldom employed clinical endpoints. Standardised, validated endpoints and pragmatic trial designs may allow investigators to determine the optimal drug, timing of initiation, dosing and duration to improve outcomes by limiting post-operative morbidity and mortality related to bleeding or thrombotic events.

Persistent Barriers to Colorectal Cancer Screening Completion Amid Centralized Outreach: A Mixed Methods Study

Purpose To understand patient experiences and persistent barriers to colorectal cancer (CRC) screening amid centralized outreach at urban family medicine practices. Approach Following a pragmatic trial assessing mailed fecal immunochemical test (FIT) outreach, we invited a subset of participants to complete a semi-structured qualitative interview and structured questionnaire. Setting Single urban academic healthcare system. Participants Sixty patients who were eligible and overdue for CRC screening at the time of trial enrollment. Method Using Andersen’s Behavioral Model, we developed an interview guide to systematically assess factors shaping screening decisions and FIT uptake. Close-ended responses were analyzed using descriptive statistics. Qualitative data were analyzed using the constant comparative method. Results Most participants (82%) self-reported that they had ever completed any modality of CRC screening, and nearly half (43%) completed the mailed FIT during the trial. Most patients (60%) preferred FIT to colonoscopy due to its private, convenient, and non-invasive nature; however, persistent barriers related to screening beliefs including fear of test results and cancer treatment still prevented some patients from completing any form of CRC screening. Conclusions Mailed FIT can overcome many structural barriers to CRC screening, yet clear communication and follow-up amid centralized outreach are essential. For some patients, tailored outreach or navigation to address screening-related fears or other screening beliefs may be needed to ensure timely completion of CRC screening.

Engaging Stakeholders in a Pragmatic Trial of Home-Delivered Meals for Persons with Dementia

Gathering stakeholder feedback is essential to designing and implementing relevant and actionable research. Additionally, stakeholders, particularly those directly impacted by an intervention, bring unique insights and experiences. This paper presents the process and findings of a research endeavor to co-design a pragmatic clinical trial with a Stakeholder Advisory Panel (SAP) in an effort to understand facilitators and barriers to conducting the research and implementing study findings. The proposed trial compares the impact of frozen, drop-shipped meals versus daily home-delivered meals provided by Meals on Wheels (MOW) programs on the ability of individuals living with dementia to age in place. We recruited nine SAP members, who were compensated for their time. The SAP is composed of a) MOW clients with dementia, b) family members of MOW clients with dementia, c) paid or volunteer MOW drivers, and d) MOW staff. A research team member facilitated two 90-minute meetings with the SAP members via Zoom. The topics of the meetings included potential benefits and challenges with each mode of meal delivery, the importance of the primary outcome (time to nursing home placement), topics of interest to include in interviews with clients and caregivers, and how participants would explain the study to a friend. Audio of the Zoom meetings was transcribed, and meeting summaries were shared with the SAP. Benefits of forming and engaging a SAP, as well as key lessons learned from SAP members and how recommendations were reflected in changes to the study protocol will be discussed.

Pilot Testing of the UB-CAM Delirium Screening App

Systematic screening improves delirium detection among hospitalized older adults. This poster describes the development and pilot testing of an iOS-based app that incorporates the Ultra-Brief Confusion Assessment Method (UB-CAM), a two-step, delirium detection protocol that combines the UB-2 (2-item screener) and 3D-CAM. Previous work tested a RedCAP-based UB-CAM app in 527 patients with 399 physicians, nurses, and certified nursing assistants (CNAs) showing it can be successfully completed by all three disciplines in 97% of eligible patients in 80 seconds on average with over 85% accuracy relative to a gold standard. To improve accessibility to the clinical setting, our research team now collaborated with a computer scientist to develop and refine an iOS-based UB-CAM app for the iPhone and iPad through iterative “laboratory” testing. The app was piloted by non-clinician, research testers in hospitalized older adults (age x̄ =83, SD= 8.0) with dementia (Clinical Dementia Rating Scale x̄ =1.1, SD= .30); 64% were assessed to be delirium positive. The app demonstrated preliminary efficiency (90 seconds on average), high acceptability (100% satisfaction of users), and reliability (100% inter-rater). This project underscores the need for close collaboration between researchers, clinicians, and computer scientists with iterative testing of bedside-facing apps prior to testing with patients. Next steps include testing effectiveness in a pragmatic trial with clinician users (physicians, nurses, CNAs), integrating the UB-CAM app into the routine hospital care of all older patients. Having rapid, accurate bedside delirium detection has the potential to transform care.