Qualitative Coding in the Computational Era: A Hybrid Approach to Improve Reliability and Reduce Effort for Coding Ethnographic Interviews

Sociologists have argued that there is value in incorporating computational tools into qualitative research, including using machine learning to code qualitative data. Yet standard computational approaches do not neatly align with traditional qualitative practices. The authors introduce a hybrid human-machine learning approach (HHMLA) that combines a contemporary iterative approach to qualitative coding with advanced word embedding models that allow contextual interpretation beyond what can be reliably accomplished with conventional computational approaches. The results, drawn from an analysis of 87 human-coded ethnographic interview transcripts, demonstrate that HHMLA can code data sets at a fraction of the effort of human-only strategies, saving hundreds of hours labor in even modestly sized qualitative studies, while improving coding reliability. The authors conclude that HHMLA may provide a promising model for coding data sets where human-only coding would be logistically prohibitive but conventional computational approaches would be inadequate given qualitative foci.

Evaluating QualiCO: an ontology to facilitate qualitative methods sharing to support open science

Qualitative science methods have largely been omitted from discussions of open science. Platforms focused on qualitative science that support open science data and method sharing are rare. Sharing and exchanging coding schemas has great potential for supporting traceability in qualitative research as well as for facilitating the reuse of coding schemas. In this study, we present and evaluate QualiCO, an ontology to describe qualitative coding schemas. Twenty qualitative researchers used QualiCO to complete two coding tasks. In our findings, we present task performance and interview data that focus participants’ attention on the ontology. Participants used QualiCO to complete the coding tasks, decreasing time on task, while improving accuracy, signifying that QualiCO enabled the reuse of qualitative coding schemas. Our discussion elaborates some issues that participants had and highlights how conceptual and prior practice frames their interpretation of how QualiCO can be used.

Development of a Codebook of Online Anti-Vaccination Rhetoric to Manage COVID-19 Vaccine Misinformation

Vaccine hesitancy (delay in obtaining a vaccine, despite availability) represents a significant hurdle to managing the COVID-19 pandemic. Vaccine hesitancy is in part related to the prevalence of anti-vaccine misinformation and disinformation, which are spread through social media and user-generated content platforms. This study uses qualitative coding methodology to identify salient narratives and rhetorical styles common to anti-vaccine and COVID-denialist media. It organizes these narratives and rhetorics according to theme, imagined antagonist, and frequency. Most frequent were narratives centered on “corrupt elites” and rhetorics appealing to the vulnerability of children. The identification of these narratives and rhetorics may assist in developing effective public health messaging campaigns, since narrative and emotion have demonstrated persuasive effectiveness in other public health communication settings.

Taxonomic categories for commissioning and startup hot spot strategies

Purpose The purpose of this study is to better understand distinct solution strategy types for common commissioning and startup problems (Hot Spots) in the construction of industrial facilities. The authors also sought to better understand which solution strategy types offer the best value for the effort required. Design/methodology/approach The authors used a method of qualitative coding of text-based data to identify themes, patterns and trends from a collection of 178 commissioning and startup (CSU) solution strategies for the CSU Hot Spots. Coding categories emerged after multiple iterations. The authors also mapped high-value, low-effort solution strategies across the categories. Chi-squared testing and analysis of proportion statistics help quantitatively justify this qualitative research. Findings The authors identified 12 distinct coding categories and showed that they follow a non-uniform distribution via statistical testing. Those strategy types which provide particularly good value for the effort required are identified (such as quality assurance and control strategies), as well as those strategy types that do not. Research limitations/implications Research is limited to CSU for the construction of industrial facilities. The findings are also limited to a subset of the most commonly problematic CSU activities. Many findings may be beneficial for heavy civil and commercial CSU as well. Practical implications Coding categories, definitions and descriptions provide a good overview of all 178 solution strategies for CSU project professionals. Implementing certain types of solutions or solution programs may allow CSU managers to prevent multiple Hot Spots from becoming problematic or to keep problems already occurring from becoming worse. Managers will also benefit from knowledge about which strategy types are more likely to give a higher value for lower effort. Originality/value Qualitative coding and analysis of solution strategies for common CSU problems have never been conducted so rigorously in any other CSU study. This method has yielded results distinct from other CSU studies which have used quantitative methods. Insights from findings have advanced the body of knowledge surrounding problem-solving in the commissioning and startup sub-discipline.

Towards More Meaningful Social Media Analysis: Case Study of Utilizing an Age Prediction Algorithm to Identify and Code Reddit Posts about E-cigarettes by Youth vs. Adults (Preprint)

BACKGROUND Previous qualitative studies and data science studies using Reddit for tobacco research are limited by the lack of available demographic information. Social media investigations are often limited to manual qualitative coding or machine learning classification in isolation. OBJECTIVE This study combines both machine learning methods and manual qualitative coding to provide contextual age nuance to social media analysis. By being able to predict a Redditor’s age using publicly available data, the most popular posts can be analyzed and qualitatively coded to provide nuanced comparisons on thematic topics by age group. METHODS The current study combines these two methods to 1) predict Reddit users’ age into two categories (13-20, 21-54) and 2) qualitatively code Electronic Nicotine Delivery System [ENDS] related Reddit posts within the two age groups. We identified Reddit posts on three topics: Vaping in General, Tobacco 21 Minimum Age Laws, and Flavor Restriction Policies. An age algorithm was used to predict Reddit users’ ages (13-20 or 21-54 year old users). The 25 posts with the highest karma score (number of upvotes minus number of downvotes) for each query and each predicted age group were qualitatively coded. RESULTS The top three, two of which were part of the query, out of nine, topics that emerged were “Flavor Restriction Policies”, “Tobacco 21 Policies”, and “Use”. Tobacco 21 and Flavor Restriction Policy posts were prominent coding categories. Opposition to flavor restriction policies was a prominent sub-category for both groups, but more common in the 21-54 group. The 13-20 group was more likely to discuss opposition to minimum age laws as well as access to flavored ENDS products. The 21-54 group more commonly mentioned general vaping use behavior. CONCLUSIONS Users predicted to be in the 13-20 age group posted about different ENDS-related topics on Reddit than users predicted to be in the 21-54 age group. Future studies could use these complementary methods with social media data to gain insights from target audiences.

Development of a Codebook of Online Anti-Vaccination Rhetoric to Manage COVID-19 Vaccine Misinformation

Vaccine hesitancy (delay in obtaining a vaccine, despite availability) represents a significant hurdle to managing the COVID-19 pandemic. Vaccine hesitancy is in part related to the prevalence of anti-vaccine misinformation and disinformation, which are spread through social media and user-generated content platforms. This study uses qualitative coding methodology to identify salient narratives and rhetorical styles common to anti-vaccine and COVID-denialist media. It organizes these narratives and rhetorics according to theme, imagined antagonist, and frequency. Most frequent were narratives centered on “corrupt elites” and rhetorics appealing to the vulnerability of children. The identification of these narratives and rhetorics may assist in developing effective public health messaging campaigns, since narrative and emotion have demonstrated persuasive effectiveness in other public health communication settings.

Evaluating QualiCO: An Ontology to Facilitate Qualitative Methods Sharing to Support Open Science

Qualitative science methods have largely been omitted from discussions of open science. Platforms focused on qualitative science that support open science data and method sharing are rare. Sharing and exchanging coding schemas has great potential for supporting traceability in qualitative research as well as for facilitating the re-use of coding schemas. In this study, we describe and evaluate QualiCO, an ontology for qualitative coding schemas. QualiCO is designed to describe a wide range of qualitative coding schemas. Twenty qualitative researchers used QualiCO to complete two coding tasks. In our findings, we present task performance and interview data that focus participants’ attention on the ontology. Participants used QualiCO to complete the coding tasks, decreasing time on task, while improving accuracy, signifying that QualiCO enabled the reuse of qualitative coding schemas. Our discussion elaborates some issues that participants had and highlights how conceptual and prior practice frames their interpretation of how QualiCo can be used.

Qualitative Coding in the Computational Era: A Hybrid Approach to Improve Reliability and Reduce Effort for Coding Ethnographic Interviews

Sociologists have argued that there is value in incorporating computational tools into qualitative research, including using machine learning to code qualitative data. Yet standard computational approaches do not neatly align with traditional qualitative practices. The authors introduce a hybrid human-machine learning approach (HHMLA) that combines a contemporary iterative approach to qualitative coding with advanced word embedding models that allow contextual interpretation beyond what can be reliably accomplished with conventional computational approaches. The results, drawn from an analysis of 87 human-coded ethnographic interview transcripts, demonstrate that HHMLA can code data sets at a fraction of the effort of human-only strategies, saving hundreds of hours labor in even modestly sized qualitative studies, while improving coding reliability. The authors conclude that HHMLA may provide a promising model for coding data sets where human-only coding would be logistically prohibitive but conventional computational approaches would be inadequate given qualitative foci.

Beyond a coefficient: an interactive process for achieving inter-rater consistency in qualitative coding

Though qualitative research has become more prevalent in practice over the last 30 years, there is still considerable uncertainty among researchers regarding how to ensure inter-rater consistency when teams are tasked with coding qualitative data. In this article, we offer an explanation of a methodology that our qualitative team used to achieve systematic coding of our dataset in a way that preserved the contextual, subjective nature of the data, lent itself to the deductive and inductive creation of a layered codebook, and ensured consistent application of the codebook to varied types of data. This methodology prepared us to draw logical and substantiated conclusions during subsequent analyses; hence, the process serves as a welcome addition to the literature on consistently coding qualitative data in a manner that honors its defining characteristics.