The Value of Cognitive Pretesting: Improving Validity and Revealing Blind Spots through the Development of a Newborn Screening Parent Experiences Survey

Surveys are used to gather a range of data on newborn screening (NBS) processes. We describe the development of a survey about parents’ NBS experiences, in the United States, informed by cognitive pretest interviews among parents with varying NBS test results (true-positive, false-positive, normal). Cognitive pretest interviews were conducted following a semi-structured script and notes were taken to identify problematic survey items. The study team met weekly to discuss pretest feedback, draft changes, and generate revised items. Pretests indicated that parent experiences with NBS are varied and NBS screening procedures are not well understood. Substantial modifications were made to survey questions concerning NBS testing and result communication. Pretesters often associated NBS with other tests/exams/scales—APGAR scores, Ages and Stages questionnaires, and genetic testing during pregnancy. Some pretesters recalled receiving NBS blood spot results during their hospital admission, an uncommon practice, and few recalled knowing results would be provided to them or their pediatrician in the first few weeks of life. Thorough explanations regarding NBS procedures and expectations were embedded within the survey to enhance and improve interpretation of survey questions. Future NBS experience surveys should utilize cognitive pretesting to capture divergent experiences and improve response validity.

Question Order Effects in Cross-Cultural Web Probing: Pretesting Behavior and Attitude Questions

Cognitive pretesting is an essential method of piloting questionnaires and ensuring quality of survey data. Web probing has emerged as an innovative method of cognitive pretesting, especially for cross-cultural and web surveys. The order of presenting questions in cognitive pretesting can differ from the order of presentation in the later survey. Yet empirical evidence is missing whether the order of presenting survey questions influences the answers to open-ended probing questions. The present study examines the effect of question order on web probing in the United States and Germany. Results indicate that probe responses are not strongly impacted by question order. However, both content and consistency of probe responses may differ cross-culturally. Implications for cognitive pretesting are discussed.

Cognitive Pretesting and pretest of the STRAIN questionnaire to elaborate work-related stress of health care staff in Switzerland / Verständlichkeitsprüfung und Pretest des STRAIN-Fragebogens zur Erhebung der Arbeitsbelastung bei Gesundheitsfachpersonen in der Schweiz

The workforce shortage in health care is a major challenge worldwide. In Switzerland, this shortage is not only demographically caused but also due to workload. Adequate measures require a good data basis. The aim of this study is to test instrument specially developed for the health professions to measure work-related stress in practice. Cognitive pretesting was used for the evaluation. Overall, 18 semi-structured single-interviews with nurses and nursing assistants were conducted. In two rounds, the question-answer-process was analysed for 11 questions. The STRAIN questionnaire was then pretested by 17 health professionals. In the first round, 42 problems were identified based on the phases of the “Cognitive Aspects of Survey Methodology”. Of these, 57 % were comprehension problems. This was shown by the polysemantic usage or unknowingness of terms. Further 38 % were problems with the answer categories, such as a missing category or an inappropriate formulation. Further questions were modified through a final pretest. The uniform comprehensibility has been achieved. Due to the questionnaire length, an online tool with a save function should be used. The results of this study are similar to those of comparable studies, which also show the most common problems with understanding. This is due to changes of concepts’ meanings over time and culture. A cognitive pretest is therefore recommended, especially for a heterogeneous target group such as health professions.