Methods for Classification of Text Data: Can the Potential of Quantitative Analysis Be Applied to Qualitative Research?

Text mining has developed rapidly in recent years. In this article we compare classification methods that are suitable for solving problems of predicting item nonresponse. The author builds reasoning about how the analysis of textual data can be implemented in a wider research field based on this material. The author considers a number of metrics adapted for textual analysis in the social sciences: accuracy, precision, recall, F1-score, and gives examples that can help a sociologist figure out which of them is worth paying attention depending on the task at hand (classify text data with equal accuracy, or more fully describe one of the classes of interest). The article proposes an analysis of results obtained by analyzing texts based on the materials of the European Social Survey (ESS).

Improving the Measurement of “Big Five” Personality Traits in a Brief Survey Instrument

The Ten Item Personality Inventory (TIPI) is the leading brief instrument for the “Big Five” personality measurement. However, TIPI’s design has suboptimal features: agree-disagree response options, numeric instead of verbal response labels, and multiple items per page. This paper presents a version of TIPI that addresses these problems. Using two nationally representative sample surveys, we compare the original and revised TIPIs on several dimensions: completion time, item nonresponse, paired item reliability, and validity based on relations to other variables. Completion time is the same and item nonresponse rates are low, while reliability and criterion validity for the revised TIPI is better than the original. The results show how better personality data can be obtained at no additional cost by optimizing questionnaire design.

Statistical Evaluation of Item Nonresponse Methods Using the World Bank’s 2015 Philippines Enterprise Survey

The main objective of the study was to evaluate item nonresponse procedures through a simulation study of different nonresponse levels or missing rates. A simulation study was used to explore how each of the response rates performs under a variety of circumstances. It also investigated the performance of procedures suggested for item nonresponse under various conditions and variable trends. The imputation methods considered were the cell mean imputation, random hotdeck, nearest neighbor, and simple regression. These variables are some of the major indicators for measuring productive labor and decent work in the country. For the purpose of this study, the researcher is interested in evaluating methods for imputing missing data for the number of workers and total cost of labor per establishment from the World Bank’s 2015 Enterprise Survey for the Philippines. The performances of the imputation techniques for item nonresponse were evaluated in terms of bias and coefficient of variation for accuracy and precision. Based on the results, the cell-mean imputation was seen to be most appropriate for imputing missing values for the total number of workers and total cost of labor per establishment. Since the study was limited to the variables cited, it is recommended to explore other labor indicators. Moreover, exploring choice of other clustering groups is highly recommended as clustering groups have great effect in the resulting estimates of imputation estimation. It is also recommended to explore other imputation techniques like multiple regression and other parametric models for nonresponse such as the Bayes estimation method. For regression based imputation, since the study is limited only in using the cluster groupings estimation, it is highly recommended to use other possible variables that might be related to the variable of interest to verify the results of this study.

Improving Data Quality with Sociodemographic Matching?: About the Effects and Implications of Age and Gender Matching in Face-to-Face Interviews

According to the theory of liking, data quality might be improved in face-to-face survey settings when there is a high degree of similarity between respondents and interviewers, for example, with regard to gender or age. Using two rounds of European Social Survey data from 25 countries including more than 70,000 respondents, this concept is tested for the dependent variables amount of item nonresponse, reluctance to answer, and the probability that a third adult person is interfering with the interview. The match between respondents and interviewers is operationalized using the variables age and gender and their statistical interactions to analyze how this relates to the outcomes. While previous studies can be corroborated, overall effect sizes are small. In general, item nonresponse is lower when a male interviewer is conducting the interview. For reluctance, there are no matching effects at all. Regarding the presence of other adults, only female respondents profit from a gender match, while age is without any effect. The results indicate that future surveys should weigh the costs and benefits of sociodemographic matching as advantages are probably small.

Design and Methodology of the Study of Associated Risks of Stillbirth (SOARS) in Utah

Objectives The Utah Study of Associated Risks of Stillbirth (SOARS) collects data about stillbirths that are not included in medical records or on fetal death certificates. We describe the design, methods, and survey response rate from the first year of SOARS. Methods The Utah Department of Health identified all Utah women who experienced a stillbirth from June 1, 2018, through May 31, 2019, via fetal death certificates and invited them to participate in SOARS. The research team based the study protocol on the Pregnancy Risk Assessment Monitoring System surveillance of women with live births and modified it to be sensitive to women’s recent experience of a stillbirth. We used fetal death certificates to examine survey response rates overall and by maternal characteristics, gestational age of the fetus, and month in which the loss occurred. Results Of 288 women invited to participate in the study, 167 (58.0%) completed the survey; 149 (89.2%) responded by mail and 18 (10.8%) by telephone. A higher proportion of women who were non-Hispanic White (vs other races/ethnicities), were married (vs unmarried), and had ≥high school education (vs <high school education) responded to the survey. Differences between responders and nonresponders by maternal age, gestational age of the fetus, or month of delivery were not significant. Among responders, item nonresponse rates were low (range, 0.6%-5.4%). The question about income (4.8%) and the questions about tests offered and performed during the hospital stay had the highest item nonresponse rates. Conclusions The response rate suggests that a mail- and telephone-based survey can be successful in collecting self-reported information about risk factors for stillbirths not currently included in medical records or fetal death certificates.

Методы машинного обучения в социологическом исследовании: предсказание частичного неответа с использованием наивного байесовского классификатора

Пропущенные данные в социологических исследованиях могут быть связаны с различными причинами, и в данной статье рассматриваются те из них, что появляются в результате незнания, нежелания или затруднения с поиском ответа на отдельные вопросы анкеты у респондента, — частичные неответы (item nonresponse). Остро стоит вопрос о предсказании частичных неответов, решение которого позволило бы сократить вероятность появления пропусков в собираемых данных. В статье показано, как возникновение частичного неответа можно прогнозировать с помощью современных методов текст-майнинга и машинного обучения на примере данных Европейского социального исследования (European Social Survey) по Великобритании. Для решения поставленной задачи использовался метод наивного байесовского классификатора (Naive Bayes Classifier) — популярный метод предсказания класса зависимой переменной на основе текстовых данных. С опорой на научную литературу показываем, как работает этот метод. Мы подготовили базу данных, объединяющую полные формулировки вопросов, ответов, инструкций и результатов опросов исследования European Social Survey по Великобритании. Нами показано, как отдельные модели для предсказания появления частичных неответов были обучены с помощью метода наивного байесовского классификатора на основе частот слов и метрики важности слов TF-IDF, процессу расчета которых мы также приводим подробное описание. Каждая из моделей предсказания частичного неответа оценивалась нами с точки зрения частоты возникновения ошибок при получении прогнозов с их помощью. Мы получили списки слов, наличие в вопросах которых статистически чаще сопровождается или не сопровождается частичными неответами. Наши результаты показали, что респонденты менее охотно отвечают на сенситивные вопросы, а некоторые слова, имеющие отношение к процедуре получения ответа на вопрос, статистически чаще пропускаются респондентами.

Identifying Outliers in Response Quality Assessment by Using Multivariate Control Charts Based on Kernel Density Estimation

When monitoring industrial processes, a Statistical Process Control tool, such as a multivariate Hotelling T 2 chart is frequently used to evaluate multiple quality characteristics. However, research into the use of T 2 charts for survey fieldwork–essentially a production process in which data sets collected by means of interviews are produced–has been scant to date. In this study, using data from the eighth round of the European Social Survey in Belgium, we present a procedure for simultaneously monitoring six response quality indicators and identifying outliers: interviews with anomalous results. The procedure integrates Kernel Density Estimation (KDE) with a T 2 chart, so that historical “in-control” data or reference to the assumption of a parametric distribution of the indicators is not required. In total, 75 outliers (4.25%) are iteratively removed, resulting in an in-control data set containing 1,691 interviews. The outliers are mainly characterized by having longer sequences of identical answers, a greater number of extreme answers, and against expectation, a lower item nonresponse rate. The procedure is validated by means of ten-fold cross-validation and comparison with the minimum covariance determinant algorithm as the criterion. By providing a method of obtaining in-control data, the present findings go some way toward a way to monitor response quality, identify problems, and provide rapid feedbacks during survey fieldwork.

No-respuesta al ítem en una encuesta de hogares en Colombia

Item nonresponse occurs when sample units do not provide information on a particular variable, problem that may affect the representativeness of the sample and the reliability of the estimates. Efforts to reduce the item nonresponse rate do not necessarily improve the quality of the information. Besides the nonresponse rate, representativeness indicators can be used to measure the quality of the collected data. This paper analyzes the wage nonresponse mechanism of a household survey in Colombia and evaluates the quality of the wage information in two different periods of time (2008:4 and 2017:4). The results show a low but increasing nonresponse rate whose behavior does not seem to be associated with the set of observables considered. This result is of value since the selection of the adequate imputation method relies on the assumptions on the missing data mechanism.