Open-ended versus Closed Probes: Assessing Different Formats of Web Probing

The method of web probing integrates cognitive interviewing techniques into web surveys and is increasingly used to evaluate survey questions. In a usual web probing scenario, probes are administered immediately after the question to be tested (concurrent probing), typically as open-ended questions. A second possibility of administering probes is in a closed format, whereby the response categories for the closed probes are developed during previously conducted qualitative cognitive interviews. Using closed probes has several benefits, such as reduced costs and time efficiency, because this method does not require manual coding of open-ended responses. In this article, we investigate whether the insights gained into item functioning when implementing closed probes are comparable to the insights gained when asking open-ended probes and whether closed probes are equally suitable to capture the cognitive processes for which traditionally open-ended probes are intended. The findings reveal statistically significant differences with regard to the variety of themes, the patterns of interpretation, the number of themes per respondent, and nonresponse. No differences in number of themes across formats by sex and educational level were found.

Download Full-text

Comparing Traditional and Crowdsourcing Methods for Pretesting Survey Questions

SAGE Open ◽

10.1177/2158244016671770 ◽

2016 ◽

Vol 6 (4) ◽

pp. 215824401667177 ◽

Cited By ~ 8

Author(s):

Jennifer Edgar ◽

Joe Murphy ◽

Michael Keating

Keyword(s):

Collective Intelligence ◽

Amazon Mechanical Turk ◽

Cognitive Interviewing ◽

Cognitive Interviews ◽

Quality Of Data ◽

The Public ◽

Survey Questions ◽

Participant Characteristics ◽

Cost Efficient

Cognitive interviewing is a common method used to evaluate survey questions. This study compares traditional cognitive interviewing methods with crowdsourcing, or “tapping into the collective intelligence of the public to complete a task.” Crowdsourcing may provide researchers with access to a diverse pool of potential participants in a very timely and cost-efficient way. Exploratory work found that crowdsourcing participants, with self-administered data collection, may be a viable alternative, or addition, to traditional pretesting methods. Using three crowdsourcing designs (TryMyUI, Amazon Mechanical Turk, and Facebook), we compared the participant characteristics, costs, and quantity and quality of data with traditional laboratory-based cognitive interviews. Results suggest that crowdsourcing and self-administered protocols may be a viable way to collect survey pretesting information, as participants were able to complete the tasks and provide useful information; however, complex tasks may require the skills of an interviewer to administer unscripted probes.

Download Full-text

Applicability of a Salient Belief Elicitation to Measure Abortion Beliefs

American Journal of Health Behavior ◽

10.5993/ajhb.45.1.7 ◽

2021 ◽

Vol 45 (1) ◽

pp. 81-94

Author(s):

Julie M. Maier ◽

Kristen N. Jozkowski ◽

Danny Valdez ◽

Brandon L. Crawford ◽

Ronna C. Turner ◽

...

Keyword(s):

Health Behavior ◽

Reasoned Action ◽

Behavioral Control ◽

Cognitive Interviewing ◽

Cognitive Interviews ◽

Reasoned Action Approach ◽

Belief Elicitation ◽

Survey Questions

Objectives: Salient belief elicitations (SBEs), informed by the Reasoned Action Approach (RAA), are used to identify 3 sets of beliefs – behavioral, control, and normative – that influence attitudes toward a health behavior. SBEs ask participants about their own beliefs through open-ended questions. We adapted a SBE by focusing on abortion, which is infrequently examined through SBEs; we also included a survey version that asked participants their views on what a hypothetical woman would do if contemplating an abortion. Given these deviations from traditional SBEs, the purpose of this study was to assess if the adapted SBE was understood by participants in English and Spanish through cognitive interviewing. Methods: We examined participants' interpretations of SBE items about abortion to determine if they aligned with the corresponding RAA construct. We administered SBE surveys and conducted cognitive interviews with US adults in both English and Spanish. Results: Participants comprehended the SBE questions as intended. Participants' interpretations of most questions were also in line with the respective RAA construct. Conclusions: SBE survey questions were comprehended well by participants. We discuss areas in which SBE questions can be modified to improve alignment with the underlying RAA construct to assess abortion beliefs.

Download Full-text

Design and Analysis of Cognitive Interviews for Comparative Multinational Testing

Field Methods ◽

10.1177/1525822x11414802 ◽

2011 ◽

Vol 23 (4) ◽

pp. 379-396 ◽

Cited By ~ 26

Author(s):

Kristen Miller ◽

Rory Fitzgerald ◽

José-Luis Padilla ◽

Stephanie Willson ◽

Sally Widdop ◽

...

Keyword(s):

Survey Data ◽

Cross Cultural ◽

Cognitive Testing ◽

Cognitive Interviewing ◽

Evidence Based ◽

Cognitive Interviews ◽

Survey Questions ◽

Method Bias

This article summarizes the work of the Comparative Cognitive Testing Workgroup, an international coalition of survey methodologists interested in developing an evidence-based methodology for examining the comparability of survey questions within cross-cultural or multinational contexts. To meet this objective, it was necessary to ensure that the cognitive interviewing (CI) method itself did not introduce method bias. Therefore, the workgroup first identified specific characteristics inherent in CI methodology that could undermine the comparability of CI evidence. The group then developed and implemented a protocol addressing those issues. In total, 135 cognitive interviews were conducted by participating countries. Through the process, the group identified various interpretive patterns resulting from sociocultural and language-related differences among countries as well as other patterns of error that would impede comparability of survey data.

Download Full-text

Use of Cognitive Interviews to Adapt PROMIS Measurement Items for Spanish Speakers Living with HIV

AIDS Research and Treatment ◽

10.1155/2016/8340863 ◽

2016 ◽

Vol 2016 ◽

pp. 1-12 ◽

Cited By ~ 1

Author(s):

R. Solorio ◽

N. C. Ayala ◽

E. Paez ◽

A. M. Skalicky ◽

L. S. Morales

Keyword(s):

Alcohol Use ◽

Patient Reported Outcomes ◽

Spanish Speakers ◽

Cognitive Interviewing ◽

Measurement Information ◽

Cognitive Interviews ◽

Patient Reported ◽

Measurement Information System ◽

Interviewing Techniques ◽

Living With Hiv

Purpose.To use cognitive interviewing techniques to assess comprehension of existing Patient-Reported Outcomes Measurement Information System (PROMIS) items among Latinos living with HIV and then refine items based on participant feedback.Methods.Latino monolingual Spanish speakers living with HIV (n=56) participated in cognitive interviews. Items from four PROMIS domains, including depression, anxiety, fatigue, and alcohol use, were assessed for comprehension. Audiotaped interviews and handwritten notes were subjected to content analysis to identify problems specific to each instrument for each domain.Results.The assessments from the cognitive interviews identified areas for improvement in each domain. We present data on the type of items that were difficult to comprehend and provide examples for how items were refined based on participants’ and PROMIS Statistical Coordinating Center (PSCC) feedback. Six out of 48 depression items, 7 out of the 61 anxiety items, 18 out of 42 fatigue items, and 7 out of 44 alcohol use items were found to have poor comprehension. These items were refined based on participant feedback; the items were then submitted to the PSCC for additional guidance on linguistics and grammar to improve comprehension.Conclusions.Cognitive interviews may be used to enhance comprehension of PROMIS items among Latinos.

Download Full-text

Item Bias Analysis of the Young Schema-Questionnaire for Psychopathology, Gender, and Educational Level

European Journal of Psychological Assessment ◽

10.1027/1015-5759/a000044 ◽

2011 ◽

Vol 27 (1) ◽

pp. 65-70 ◽

Cited By ~ 11

Author(s):

Marleen M. Rijkeboer ◽

Huub van den Bergh ◽

Jan van den Bout

Keyword(s):

Educational Level ◽

Cross Validation ◽

Item Bias ◽

Response Patterns ◽

Dutch Population ◽

Bias Analysis ◽

Young Schema Questionnaire ◽

Item Functioning ◽

Item Level ◽

Schema Questionnaire

This study examines the construct validity of the Young Schema-Questionnaire at the item level in a Dutch population. Possible bias of items in relation to the presence or absence of psychopathology, gender, and educational level was analyzed, using a cross-validation design. None of the items of the YSQ exhibited differential item functioning (DIF) for gender, and only one item showed DIF for educational level. Furthermore, item bias analysis did not identify DIF for the presence or absence of psychopathology in as much as 195 of the 205 items comprising the YSQ. Ten items, however, spread over the questionnaire, were found to yield relatively inconsistent response patterns for patients and nonclinical participants.

Download Full-text

Trust your Abilities More than the Stereotype: Effect of Gender-stereotype Threat and Task Difficulty on Word Production, Clustering, and Switching in Letter Fluency

Pertanika Journal of Social Sciences and Humanities ◽

10.47836/pjssh.28.4.05 ◽

2020 ◽

Vol 28 (4) ◽

Author(s):

Sri Wulandari Wulandari ◽

Donny Hendrawan

Keyword(s):

Stereotype Threat ◽

Cognitive Processes ◽

Educational Level ◽

Task Difficulty ◽

Gender Stereotype ◽

Cognitive Tasks ◽

Actual Performance ◽

Men And Women ◽

Undergraduate Men ◽

And Task

Gender-stereotype threat consistently accounts for underperformance phenomena experienced by women on male-stereotyped cognitive tasks. However, only a few studies have examined how the threat is affecting performance on female-stereotyped cognitive tasks, such as letter fluency. The present study examined whether variations in the cues to activate stereotype threat and the level of task difficulty would affect the letter fluency performance of undergraduate men and women (<em>n</em> = 168) and the underlying cognitive processes of this performance (i.e., switching, clustering). The results indicated participants held beliefs about women’s superiority in this task. However, threat-activation cues did not affect production of correct words, errors, clustering, or switching in men and women. Task difficulty affected the number of correct words, yet it did not interact with the stereotype threat-activation cues. Finally, participants’ actual performance was related to their self-rating perception about their ability instead of the stereotyping they perceived. The effect of self-efficacy, educational level, and individuals’ susceptibilities should be taken into account when studying the effects of stereotype threat.

Download Full-text