Modelling Mode Effects for a Panel Survey in Transition

2021 ◽  
pp. 63-88
Author(s):  
Paul P. Biemer ◽  
Kathleen Mullan Harris ◽  
Dan Liao ◽  
Brian J. Burke ◽  
Carolyn Tucker Halpern
Keyword(s):  
Data Collection ◽  
Data Quality ◽  
Mixed Mode ◽  
Good News ◽  
Longitudinal Change ◽  
Nonresponse Bias ◽  
Two Phase ◽  
Mode Effects ◽  
Analytical Results ◽  
Weighting Adjustments

Funding reductions combined with increasing data-collection costs required that Wave V of the USA’s National Longitudinal Study of Adolescent to Adult Health (Add Health) abandon its traditional approach of in-person interviewing and adopt a more cost-effective method. This approach used the mail/web mode in Phase 1 of data collection and in-person interviewing for a random sample of nonrespondents in Phase 2. In addition, to facilitate the comparison of modes, a small random subsample served as the control and received the traditional in-person interview. We show that concerns about reduced data quality as a result of the redesign effort were unfounded based on findings from an analysis of the survey data. In several important respects, the new two-phase, mixed-mode design outperformed the traditional design with greater measurement accuracy, improved weighting adjustments for mitigating the risk of nonresponse bias, reduced residual (or post-adjustment) nonresponse bias, and substantially reduced total-mean-squared error of the estimates. This good news was largely unexpected based upon the preponderance of literature suggesting data quality could be adversely affected by the transition to a mixed mode. The bad news is that the transition comes with a high risk of mode effects for comparing Wave V and prior wave estimates. Analytical results suggest that significant differences can occur in longitudinal change estimates about 60 % of the time purely as an artifact of the redesign. This begs the question: how, then, should a data analyst interpret significant findings in a longitudinal analysis in the presence of mode effects? This chapter presents the analytical results and attempts to address this question.

scholarly journals Mode Effects on Data Quality and Measurement in a Mixed-Mode Time-Use Survey

2021 ◽  
Author(s):  
Stella Chatzitheochari ◽  
Elena Mylona
Keyword(s):  
Data Collection ◽  
Data Quality ◽  
Time Use ◽  
Mixed Mode ◽  
Large Scale ◽  
New Technologies ◽  
Quality Data ◽  
Mode Effects ◽  
Time Use Survey ◽  
Time Diaries

Recent years have witnessed an increasing interest in the use of new technologies for time-use data collection, driven by their potential to reduce survey administration costs and improve data quality. However, despite the steady growth of studies that employ web and app time diaries, there is little research on their comparability with traditional paper-administered diaries that have long been regarded as the “gold standard” for measurement in time-use research. This paper rectifies this omission by investigating diary mode effects on data quality and measurement, drawing on data from a mixed-mode large-scale time diary study of adolescents in the United Kingdom. After controlling for selection effects, we find that web and app diaries yield higher quality data than paper diaries, which attests to the potential of new technologies in facilitating diary completion. At the same time, our analysis of broad time-use domains does not find substantial mode effects on measurement for the majority of daily activity categories. We conclude by discussing avenues for future methodological research and implications for time-use data collection.

scholarly journals Using New Technologies for Time Diary Data Collection: Instrument Design and Data Quality Findings from a Mixed-Mode Pilot Survey

2017 ◽  
Vol 137 (1) ◽  
pp. 379-390 ◽  
Author(s):  
Stella Chatzitheochari ◽  
Kimberly Fisher ◽  
Emily Gilbert ◽  
Lisa Calderwood ◽  
Tom Huskinson ◽  
...  
Keyword(s):  
Data Collection ◽  
Data Quality ◽  
Mixed Mode ◽  
New Technologies ◽  
Instrument Design ◽  
Pilot Survey ◽  
Time Diary ◽  
Diary Data ◽  
Data Collection Instrument

Comparing Two Web/Mail Mixed-mode Contact Protocols to a Unimode Mail Survey

Field Methods ◽  
2017 ◽  
Vol 29 (3) ◽  
pp. 281-298 ◽  
Author(s):  
Milton G. Newberry ◽  
Glenn D. Israel
Keyword(s):  
Mixed Mode ◽  
Survey Methodology ◽  
Mail Survey ◽  
Response Rates ◽  
Nonresponse Bias ◽  
Future Research ◽  
Similar Response ◽  
Treatment Groups ◽  
Mode Effects ◽  
Utilization Data

Recent research has shown mixed-mode surveys are advantageous for organizations to use in collecting data. Previous research explored web/mail mode effects for four-contact waves. This study explores the effect of web/mail mixed-mode systems over a series of contacts on the customer satisfaction data from the Florida Cooperative Extension Service during 2012–2013. The experimental design involved a group of clients who provided e-mail and mail contact information randomly assigned to two mixed-mode treatment groups and a mail-only control. Demographic and service utilization data were compared to assess response rates and nonresponse bias. Logistic regression showed the treatment groups had similar response rates and nonresponse bias. The fifth contact was statistically significant in increasing response rates but did not reduce nonresponse bias. Future research should continue exploring optimizing the number of contacts in mixed-mode survey methodology.

scholarly journals Surveying Ourselves

2010 ◽  
Vol 34 (4) ◽  
pp. 448-463 ◽  
Author(s):  
Kristen A. Matteson ◽  
Britta L. Anderson ◽  
Stephanie B. Pinto ◽  
Vrishali Lopes ◽  
Jay Schulkin ◽  
...  
Keyword(s):  
Data Collection ◽  
Data Quality ◽  
Private Practice ◽  
Mixed Mode ◽  
National Sample ◽  
Survey Mode ◽  
Web Based ◽  
Minimal Reduction ◽  
The Web

A survey was distributed, using a sequential mixed-mode approach, to a national sample of obstetrician–gynecologists. Differences between responses to the web-based mode and the on-paper mode were compared to determine if there were systematic differences between respondents. Only two differences in respondents between the two modes were identified. University-based physicians were more likely to complete the web-based mode than private practice physicians. Mail respondents reported a greater volume of endometrial ablations compared to online respondents. The web-based mode had better data quality than the paper-based mailed mode in terms of less missing and inappropriate responses. Together, these findings suggest that, although a few differences were identified, the web-based survey mode attained adequate representativeness and improved data quality. Given the metrics examined for this study, exclusive use of web-based data collection may be appropriate for physician surveys with a minimal reduction in sample coverage and without a reduction in data quality.

scholarly journals Do Sequential Mixed-Mode Surveys Decrease Nonresponse Bias, Measurement Error Bias, and Total Bias? An Experimental Study

2019 ◽  
Vol 7 (4) ◽  
pp. 545-571 ◽  
Author(s):  
Joseph W Sakshaug ◽  
Alexandru Cernat ◽  
Trivellore E Raghunathan
Keyword(s):  
Measurement Error ◽  
Data Collection ◽  
Mixed Mode ◽  
Traffic Accidents ◽  
Nonresponse Bias ◽  
Bias Estimation ◽  
Design Decisions ◽  
Second Mode ◽  
Error Bias

Abstract Mixing multiple modes of survey data collection has become standard practice in survey research. Mixed-mode surveys are faced with a slew of design decisions regarding which types of modes to administer and which sequence to administer them in. Such decisions are largely based on administrative objectives, such as minimizing costs and maximizing response rates. However, just as important to these mixed-mode decisions is their impact on nonresponse bias, measurement error bias, and total bias, which are understudied issues in the mixed-mode literature. In this article, we report on a sequential mixed-mode experiment of young adult drivers randomized to one of two mode sequences: an interviewer-administered (telephone) mode with self-administered (mail) follow-up, or the reverse sequence. Using a mix of direct and indirect bias estimation strategies, we find support for the notion that implementing a second mode of data collection can reduce nonresponse and measurement error bias, but the sequence in which the modes are administered makes a difference: the mail-telephone sequence minimizes bias to a greater extent than the telephone-mail sequence, relative to the starting mode and overall. However, a backfiring effect was found: despite reducing both nonresponse and measurement error bias, switching from mail to telephone increased the total bias in a key estimate of traffic accidents. A discussion of these findings and their implications for survey practice are provided in conclusion.

scholarly journals Risk of Nonresponse Bias and the Length of the Field Period in a Mixed-Mode General Population Panel

2021 ◽  
Author(s):  
Bella Struminskaya ◽  
Tobias Gummer
Keyword(s):  
Panel Data ◽  
General Population ◽  
Data Collection ◽  
Mixed Mode ◽  
Nonresponse Bias ◽  
German Population ◽  
Coefficients Of Variation ◽  
Using Data ◽  
Day By Day

Abstract Survey researchers are often confronted with the question of how long to set the length of the field period. Longer fielding time might lead to greater participation yet requires survey managers to devote more of their time to data collection efforts. With the aim of facilitating the decision about the length of the field period, we investigated whether a longer fielding time reduces the risk of nonresponse bias to judge whether field periods can be ended earlier without endangering the performance of the survey. By using data from six waves of a probability-based mixed-mode (online and mail) panel of the German population, we analyzed whether the risk of nonresponse bias decreases over the field period by investigating how day-by-day coefficients of variation develop during the field period. We then determined the optimal cut-off points for each mode after which data collection can be terminated without increasing the risk of nonresponse bias and found that the optimal cut-off points differ by mode. Our study complements prior research by shifting the perspective in the investigation of the risk of nonresponse bias to panel data as well as to mixed-mode surveys, in particular. Our proposed method of using coefficients of variation to assess whether the risk of nonresponse bias decreases significantly with each additional day of fieldwork can aid survey practitioners in finding the optimal field period for their mixed-mode surveys.

scholarly journals Deployment of a Mixed-Mode Data Collection Strategy Does Not Reduce Nonresponse Bias in a General Population Health Survey

2012 ◽  
Vol 47 (4) ◽  
pp. 1739-1754 ◽  
Author(s):  
Timothy J. Beebe ◽  
Donna D. McAlpine ◽  
Jeanette Y. Ziegenfuss ◽  
Sarah Jenkins ◽  
Lindsey Haas ◽  
...  
Keyword(s):  
General Population ◽  
Data Collection ◽  
Population Health ◽  
Health Survey ◽  
Mixed Mode ◽  
Nonresponse Bias ◽  
Data Collection Strategy

scholarly journals Estimating Nonresponse Bias and Mode Effects in a Mixed-Mode Survey

2011 ◽  
Vol 53 (5) ◽  
pp. 669-686 ◽  
Author(s):  
Peter Lugtig ◽  
Gerty J.L.M. Lensvelt-Mulders ◽  
Remco Frerichs ◽  
Assyn Greven
Keyword(s):  
Propensity Score ◽  
Mixed Mode ◽  
Sample Selection ◽  
Nonresponse Bias ◽  
Selection Effects ◽  
Separate Sample ◽  
Mode Effects ◽  
Internet Surveys ◽  
Internet Sample ◽  
Study Mode

In mixed-mode surveys, it is difficult to separate sample selection differences from mode-effects that can occur when respondents respond in different interview settings. This paper provides a framework for separating mode effects from selection effects by matching very similar respondents from different survey modes using propensity score matching. The answer patterns of the matched respondents are subsequently compared. We show that matching can explain differences in nonresponse and coverage in two Internet samples. When we repeat this procedure for a telephone and Internet sample however, differences persist between the samples after matching. This indicates the occurrence of mode effects in telephone and Internet surveys. Mode effects can be problematic; hence we conclude with a discussion of designs that can be used to explicitly study mode effects.

Sign in / Sign up

Export Citation Format

Share Document