Effective questionnaire design: How to use cognitive interviews to refine questionnaire items

BACKGROUND: Cognitive interviewing is a technique that can be used to improve and refine questionnaire items. We describe the basic methodology of cognitive interviewing and illustrate its utility through our experience using cognitive interviews to refine a questionnaire assessing parental understanding of concepts related to preterm birth. METHODS: Cognitive interviews were conducted using current best practices. Results were analyzed by the multidisciplinary research team and questionnaire items that were revealed to be problematic were revised. RESULTS: Revisions to the questionnaire items were made to improve clarity and to elicit responses that truly reflected the participants understanding of the concept. CONCLUSION: Cognitive interviewing is a useful methodology for improving validity of questionnaire items, we recommend researchers developing new questionnaire items design and complete cognitive interviews to improve their items and increase confidence in study conclusions.

Identifying and Minimizing Errors in the Measurement of Early Childhood Development: Lessons Learned from the Cognitive Testing of the ECDI2030

Challenges in measuring early childhood development (ECD) at scale have been documented, yet little is known about the specific difficulties related to questionnaire design and question interpretation. The purpose of this paper is to discuss the challenges of measuring ECD at scale in the context of household surveys and to show how to overcome them. The paper uses examples from the cognitive interviewing exercises that were conducted as part of the methodological work to develop a measure of ECD outcomes, the ECDI2030. It describes the methodological work carried out to inform the selection and improvement of question items and survey implementation tools as a fundamental step to reduce and mitigate systematic measurement error and improve data quality. The project consisted of a total of five rounds of testing, comprising 191 one-on-one, in-depth cognitive interviews across six countries (Bulgaria, India, Jamaica, Mexico, Uganda, and the USA). Qualitative data analysis methods were used to determine matches and mismatches between intention of items and false positives or false negative answers among subgroups of respondents. Key themes emerged that could potentially lead to systematic measurement error in population-based surveys on ECD: (1) willingness of child to perform task versus ability of child to perform task; (2) performing task versus performing task correctly; (3) identifying letters or numbers versus recognizing letters or numbers; (4) consistently performing task versus correctly performing task; (5) applicability of skills being asked versus observability of skills being asked; and (6) language production versus language comprehension. Through an iterative process of testing and subsequent revision, improvements were made to item wording, response options, and interviewer training instructions. Given the difficulties inherent in population-level data collection in the context of global monitoring, this study’s findings confirm the importance of cognitive testing as a crucial step in careful, culturally relevant, and sensitive questionnaire design and as a means to reduce response bias in cross-cultural contexts.

Measuring satisfaction with democracy: how good are different scales across countries and languages?

The Satisfaction With Democracy (SWD) indicator is very often used in social sciences’ research. However, while there is debate about which concept it measures, the discussion about the size of its measurement errors (how well it measures the underlying concept ‘satisfaction with the way democracy works’) is scarce. Nonetheless, measurement errors can affect the results and threaten comparisons across studies, countries and languages. Thus, in this paper, we estimated the measurement quality (complement of measurement errors) of the SWD indicator for 7 response scales across 38 country-language groups, using three multitrait-multimethod experiments from the European Social Survey. Results show that measurement errors explain from 16% (11-point scale) to 54% (4-point scale) of the variance in the observed responses. Additionally, we provide insights to improve questionnaire design and evaluate the indicator’s comparability across scales, countries and languages.

Ten years of the China migrants dynamic survey: retrospective and prospects

This paper reviews the background of China’s decision to launch the “China Migrants Dynamic Survey” (CMDS), and examines the ideas and characteristics of CMDSs carried out annually from 2009 through 2018. The paper considers such aspects as project design, questionnaire design, sampling design, and survey implementation, and summarizes why it is necessary to carry out CMDS and the survey’s performance. Based on the need for information that drives migration surveys and research in the new era, and taking into account the experience gained from previous surveys, this paper attempts to set forth several issues that need to be taken into consideration in the design and implementation of future surveys of migrants.

EXTENSION ACTIVITIES PROVIDED TO RICE FARMERS IN RAISING THE AWARENESS FOR COMPOST MANUFACTURE IN AL-DIWANIYAH PROVINCE

This study was aimed to determine extension activities provided to rice farmers in raising the awareness for compost manufacture in Al-Diwaniyah province and investigate the level of awareness of rice farmers in the field of compost manufacturing, as well as the problems facing farmers when manufacturing compost, mythology a questionnaire design to collect data from rice farmers during 2018 . The research society consisted of the agricultural departments of the directorate of Agriculture in Al Diwaniyah province. A random sample was taken from rice farmers related to the agricultural departments in the Directorate of Agriculture with their (632) farmers, The results of this study inculcated that the level of awareness for the farmers in the manufacture of compost, The average tends to weak in their awareness, The role of the agricultural extension has also just through extension seminars, which is considered one of the most extension activities were used to rice farmers of compost manufacturing, In addition, there is an absence of workers in the field of composting of rice residues, It could be to pay adequate attention to the rice crop as a strategic crop through the utilization of its waste and the elimination of environmental pollution that occurs as a result of burning those waste.