Effective questionnaire design: How to use cognitive interviews to refine questionnaire items

BACKGROUND: Cognitive interviewing is a technique that can be used to improve and refine questionnaire items. We describe the basic methodology of cognitive interviewing and illustrate its utility through our experience using cognitive interviews to refine a questionnaire assessing parental understanding of concepts related to preterm birth. METHODS: Cognitive interviews were conducted using current best practices. Results were analyzed by the multidisciplinary research team and questionnaire items that were revealed to be problematic were revised. RESULTS: Revisions to the questionnaire items were made to improve clarity and to elicit responses that truly reflected the participants understanding of the concept. CONCLUSION: Cognitive interviewing is a useful methodology for improving validity of questionnaire items, we recommend researchers developing new questionnaire items design and complete cognitive interviews to improve their items and increase confidence in study conclusions.

Cognitive Interview

Cognitive interviewing helps to bring back the original meaning of pilotage in a case study. Cognitive processes in an interview start with the interpretation of the question and its constituent terms, include the stage of forming an opinion and developing a judgment about it, and ends with editing the answer. The objectives of the cognitive interview are to explore the concepts, words and concepts of everyday language; pull out meanings, associations and emotions associated with the subject of research; detect barriers blocking the respondent's presentation of his position; assess confidence in your opinion, as well as the willingness to correct your point of view; to form their understanding of the "ordinary theory" of the studied phenomenon, etc. For each task, you can build an original strategy for a cognitive interview, but it is important to understand what we are studying, what the result should be and what the research team will do with it.

Considerations for conducting and reporting digitally supported cognitive interviews with children and adults

Background Cognitive interviewing is a well-established qualitative method used to develop and refine PRO measures. A range of digital technologies including phone, web conferencing, and electronic survey platforms can be leveraged to support the conduct of cognitive interviewing in both children and adults. These technologies offer a potential solution to enrolling underrepresented populations, including those with rare conditions, functional limitations and geographic or socioeconomic barriers. In the aftermath of the COVID-19 pandemic, the use of digital technologies for qualitative interviewing will remain essential. However, there is limited guidance about adapting cognitive interviewing procedures to allow for remote data capture, especially with children. Methods Synthesizing the literature and our research experiences during the COVID-19 pandemic, we examine considerations for implementing digitally supported cognitive interviews with children, adolescents, and adults. We offer recommendations to optimize data quality and empirical rigor and illustrate the application of these recommendations in an ongoing cognitive interviewing study to develop and refine a new pediatric PRO measure. Results Good research practices must address participant and researcher preparation for study-related procedures and should anticipate and pre-emptively manage technological barriers. Field notes should detail interview context, audio/video cues, and any impact of technological difficulties on data quality. The approaches we recommend have been tested in an ongoing cognitive interviewing study that is enrolling children/adolescents with cGVHD ages 5–17 and their caregivers [NCT 04044365]. The combined use of telephone and videoconferencing to conduct cognitive interviews remotely is feasible and acceptable and yields meaningful data to improve the content validity of our new PRO measure of cGVHD symptom bother. Conclusion Digitally supported cognitive interviewing procedures will be increasingly employed. Remote data collection can accelerate accrual, particularly in multi-site studies, and may allow for interviewer personnel and data management to be centralized within a coordinating center, thus conserving resources. Research is needed to further test and refine techniques for remote cognitive interviewing, particularly in traditionally underrepresented populations, including children and non-English speakers. Expansion of international standards to address digitally supported remote qualitative data capture appears warranted.

Cognitive Interviewing to Evaluate the Content Validity of a Photo-Supported Preference Assessment

Photo-supported verbal assessments have shown to improve comprehension and expression of choices by older adults living with cognitive-communication challenges. The purpose of this study was to assess content validity (CV) of photographs used to supplement the Preferences for Everyday Living Inventory-Nursing Home (PELI-NH) from the perspective of older adults, using cognitive interviewing methods. Participants (N=21) were average age 75 (SD=5.67), mostly male (62%) and white (90%), living in residential communities (86%), with no known cognitive or communication deficits. Interview data was used to iteratively assess and revise photographs. A total of 46 photographs demonstrated CV; 26 demonstrated CV after revisions; 3 did not demonstrate CV after revisions. Content analysis revealed thematic codes describing participants’ photograph preferences including image quality, context, subject diversity, and relevance to long-term care. Discussion will include implications for clinicians and researchers on how to evaluate and improve CV of photo-supported verbal assessments.

ICF Linking and Cognitive Interviewing Are Complementary Methods for Optimizing Content Validity of Outcome Measures: An Integrated Methods Review

Content validity is a fundamental requirement of outcome measures. After reviewing operational needs and existing definitions, content validity we as defined as: the extent to which a measure provides a comprehensive and true assessment of the key relevant elements of a specified construct or attribute across a defined range, clearly and equitably for a stated target audience and context. ICF linkage rules from 2002, 2005, and 2019 have provide increasingly clear processes for describing and evaluating content of outcome measures. ICF Core Sets provide international reference standards of the core constructs of importance for different health conditions. Both are important as reference standards during content validation. To summarize their use as reference standards, the following summary indicators were proposed: (1) Measure to ICF linkage, (2) Measure to (Brief or Comprehensive) Core Set Absolute Linkage, (3) Measure to (Brief or Comprehensive) Core Set Unique Linkage, (4) Core Set Representation, and (5) Core Set Unique Disability Representation. Methods to assess how respondents engage with content are needed to complement ICF-linking. Cognitive interviewing is an ideal method since it used to explore how respondents interpret and calibrate response to individual items on an outcome measure. We proposed a framework for classifying these responses: Clarity/Comprehension, Relevance, Inadequate response definition, Reference Point, Perspective modification, and Calibration Across Items. Our analysis of 24 manuscripts that used ICF linking for content validation since updated linking rules were published found that authors typically used linking to validate existing measures, involved multiple raters, used 2005 linking rules, summarized content at a concept level (e.g., impairment, activity, participation) and/or use core sets as a reference standard. Infrequently, ICF linking was used to create item pools/conceptual frameworks for new measures, applied the full scope of the 2019 linking rules, used summary indicators, or integrated ICF-linking with qualitative methods like cognitive interviews. We conclude that ICF linkage is a powerful tool for content validity during development or validation of PROM. Best practices include use of updated ICF linking rules, triangulation of ICF linking with participant assessments of clarity and relevance preferably obtained using cognitive interview methods, and application of defined summary indicators.

1258Development of a valid evaluation tool assessing health behaviours among Māori & Pasifika young people

Background The Māori & Pasifika population experience high rates of health inequity, with a greater prevalence of obesity and the associated, life diminishing comorbidities. This is in part attributable to higher socioeconomic disadvantage, low health literacy and a lack of culturally tailored health services. Currently no validated tool exists to assess health behaviour change among Māori & Pasifika populations or success of tailored health interventions to tackle chronic disease. Methods The project incorporates a comprehensive validation and pilot testing process, including 1) cognitive interviewing and 2) test-retest reproducibility. Participants will include a representative sample of Māori & Pasifika young people and their parents/guardians. All research methodology is guided by cultural experts, specific to the Māori & Pasifika population. Results Qualitative data collected via cognitive interviewing will provide feedback regarding the readability, comprehension and content validity of the questionnaire items. Thematic analysis will inform improvements, optimising participant understandability. Cronbach’s alpha will be used to assess internal consistency and Pearson’s r will determine questionnaire test-retest reproducibility. Conclusions A validated questionnaire articulating with cultural values will provide meaningful data to researchers, health practitioners and government bodies regarding the lifestyle decisions of Māori & Pasifika peoples. Enhanced monitoring will determine the success of health initiatives to improve health outcomes of a priority population, across Australia and internationally. Key messages Lifestyle programs aiming to improve health outcomes are increasingly adapted to incorporate and acknowledge cultural values. Reliable measurement tools, including questionnaires, are fundamental to advancing future health research and tackling health inequity among priority populations.