Comparing Two Web/Mail Mixed-mode Contact Protocols to a Unimode Mail Survey

Field Methods ◽  
2017 ◽  
Vol 29 (3) ◽  
pp. 281-298 ◽  
Author(s):  
Milton G. Newberry ◽  
Glenn D. Israel
Keyword(s):  
Mixed Mode ◽  
Survey Methodology ◽  
Mail Survey ◽  
Response Rates ◽  
Nonresponse Bias ◽  
Future Research ◽  
Similar Response ◽  
Treatment Groups ◽  
Mode Effects ◽  
Utilization Data

Recent research has shown mixed-mode surveys are advantageous for organizations to use in collecting data. Previous research explored web/mail mode effects for four-contact waves. This study explores the effect of web/mail mixed-mode systems over a series of contacts on the customer satisfaction data from the Florida Cooperative Extension Service during 2012–2013. The experimental design involved a group of clients who provided e-mail and mail contact information randomly assigned to two mixed-mode treatment groups and a mail-only control. Demographic and service utilization data were compared to assess response rates and nonresponse bias. Logistic regression showed the treatment groups had similar response rates and nonresponse bias. The fifth contact was statistically significant in increasing response rates but did not reduce nonresponse bias. Future research should continue exploring optimizing the number of contacts in mixed-mode survey methodology.

scholarly journals The effect of a web-push survey on physician survey response rates: a randomized experiment.

2019 ◽  
Author(s):  
Cristine D. Delnevo ◽  
Binu Singh
Keyword(s):  
Survey Methodology ◽  
Response Rate ◽  
Mail Survey ◽  
Cost Effective ◽  
Response Rates ◽  
National Sample ◽  
High Response Rate ◽  
Future Research ◽  
Effective Delivery ◽  
The Web

Abstract Background: Achieving a high response rate for physicians has been challenging and with response rates declining in recent years, innovative methods are needed to increase rates. An emerging concept in survey methodology has been web-push survey delivery. In this delivery method, contact is made by mail to request a response by web. This study explored the feasibility of a web-push survey on a national sample of physicians. Methods: 1000 physicians across six specialties were randomly assigned to a mail only or web-push survey delivery. Each mode consisted of four contacts including an initial mailing, reminder postcard, and two additional follow-ups. Response rates were calculated using AAPOR’s response rate 3 calculation. Data collection occurred between Febuary – April 2018 and analyzed March 2019. Results: Overall reponse rates for the mail only vs. web-push survey delivery were comparable (51.2% vs. 52.8%). Higher response rates across all demographics were seen in the web-push delivery with the exception of pulmonary/critical care and physicians over the age of 65. The web-push survey yielded a greater response after the first mailing requiring fewer follow-up contacts resulting in a more cost-effective delivery. Conclusions: A web-push mail survey is effective in achieveing a comparable response rate to traditional mail only delivery for physicians. The web-push survey was more efficient in terms of cost and in receiving responses in a more timely manner. Future research should explore the efficiency of a web-push survey delivery across various health care provider populations.

scholarly journals The Effects of a Delayed Incentive on Response Rates, Response Mode, Data Quality, and Sample Bias in a Nationally Representative Mixed Mode Study

Field Methods ◽  
2016 ◽  
Vol 29 (3) ◽  
pp. 221-237 ◽  
Author(s):  
Katherine A. McGonagle ◽  
Vicki A. Freedman
Keyword(s):  
Data Quality ◽  
Mixed Mode ◽  
Panel Study ◽  
Response Rates ◽  
Response Mode ◽  
Future Research ◽  
Sample Bias ◽  
Treatment Conditions ◽  
Nationally Representative ◽  
The Impact

This article describes the results of an experiment designed to examine the impact of the use and amount of delayed unconditional incentives in a mixed mode (push to web) supplement on response rates, response mode, data quality, and sample bias. The supplement was administered to individuals who participate in the U.S. Panel Study of Income Dynamics, the longest running national household panel in the world. After 10 weeks of data collection, individuals who had not yet completed the interview were sent a final survey request and randomly assigned to one of three treatment conditions: no incentive, US$5, and US$10. The impact of the incentives on response rates and mode, effects on data quality, and sample bias are described. The implications for the use of incentives in mixed mode surveys and directions for future research are discussed.

scholarly journals Every Door Direct Mail in US survey research: An anonymous census approach to mail survey sampling

2019 ◽  
Vol 12 (2) ◽  
pp. 205979911986210 ◽  
Author(s):  
Emily Grubert
Keyword(s):  
Survey Methodology ◽  
Lower Cost ◽  
Low Cost ◽  
Mail Survey ◽  
Response Rates ◽  
Direct Mail ◽  
Postal Service ◽  
United States Postal Service ◽  
Household Level

Mail surveys remain a popular method of eliciting attitudinal information, but declining response rates motivate inquiry into new, lower cost methods of contacting potential respondents. This work presents methodological findings from a medium-sized (~12,000 addresses) mail survey testing a United States Postal Service direct mail product called Every Door Direct Mail as a low-cost approach to anonymous mail survey distribution. The results suggest that under certain conditions, Every Door Direct Mail can be a useful approach for mail survey distribution, with response rates similar to those observed with analogous first-class mailing approaches but lower cost per response. As a tool for postal carrier-route saturation mailing that does not use names or addresses, Every Door Direct Mail is potentially useful for researchers who work in small, specific geographies or value or require anonymity. The results from this work suggest good performance on demographics and socially undesirable answers for Every Door Direct Mail relative to addressed mailings. The major disadvantages include an inability to conduct household-level probability sampling, an inability to customize nonresponse follow-up, and minimum mailing sizes associated with the postal carrier route saturation requirement. Every Door Direct Mail is unlikely to become a major tool for survey researchers, but it could be useful in niche applications. This study introduces Every Door Direct Mail to the survey methodology literature and presents empirical data intended to help researchers considering using Every Door Direct Mail.

scholarly journals Mode and Frame Matter: Assessing the Impact of Survey Mode and Sample Frame in Choice Experiments

2019 ◽  
Vol 39 (7) ◽  
pp. 827-841 ◽  
Author(s):  
Verity Watson ◽  
Terry Porteous ◽  
Tim Bolt ◽  
Mandy Ryan
Keyword(s):  
Mail Survey ◽  
Response Rates ◽  
Choice Experiments ◽  
Future Research ◽  
Computer Assisted ◽  
Health State ◽  
Public Preferences ◽  
Internet Surveys ◽  
The Uk ◽  
The Impact

Background. Choice experiments (CE) are applied in health economics to elicit public preferences and willingness to pay (WTP). CEs are frequently administered as Internet-based surveys. Internet surveys have recognized advantages, but concerns exist about the representativeness of Internet samples, data quality, and the impact on elicited values. Aim. We conducted the first study in health comparing an Internet-based CE survey with the more traditional general population mail survey. We also compared the Internet-based and mail CE surveys with computer-assisted personal interviews (CAPIs), which are commonly used to elicit health state valuations. Methods. Two separate samples were drawn from 2 United Kingdom (UK) volunteer Internet panels (IPs), CAPIs were undertaken with respondents sampled from UK Census Output Areas, and mail surveys were sent to UK households drawn from the postcode address file (PAF). Each mode received more than 1000 respondents. We compared modes and frames using objective measures (response rate, sample representativeness of the UK population, elicited values, theoretical validity, and cost per response) and subjective/self-reported measures (time taken to complete the study, perceived study consequentiality, and stated attribute nonattendance). This study intentionally confounded the survey modes and sample frame by choosing sample frames that are typically used by researchers for each mode. Results. Estimated WTP differs across mode-frame pairs. On most measures, CAPIs dominated. They are more expensive, however. On all measures, except response rates, Internet surveys dominated the mail survey. They were also cheaper. Conclusion. Researchers using IPs should pay attention to response rates and be aware that the quality of IPs differs. Given the importance of perceived consequentiality and attribute attendance in CEs, future research should address their impact across modes and frames.

Modelling Mode Effects for a Panel Survey in Transition

2021 ◽  
pp. 63-88
Author(s):  
Paul P. Biemer ◽  
Kathleen Mullan Harris ◽  
Dan Liao ◽  
Brian J. Burke ◽  
Carolyn Tucker Halpern
Keyword(s):  
Data Collection ◽  
Data Quality ◽  
Mixed Mode ◽  
Good News ◽  
Longitudinal Change ◽  
Nonresponse Bias ◽  
Two Phase ◽  
Mode Effects ◽  
Analytical Results ◽  
Weighting Adjustments

Funding reductions combined with increasing data-collection costs required that Wave V of the USA’s National Longitudinal Study of Adolescent to Adult Health (Add Health) abandon its traditional approach of in-person interviewing and adopt a more cost-effective method. This approach used the mail/web mode in Phase 1 of data collection and in-person interviewing for a random sample of nonrespondents in Phase 2. In addition, to facilitate the comparison of modes, a small random subsample served as the control and received the traditional in-person interview. We show that concerns about reduced data quality as a result of the redesign effort were unfounded based on findings from an analysis of the survey data. In several important respects, the new two-phase, mixed-mode design outperformed the traditional design with greater measurement accuracy, improved weighting adjustments for mitigating the risk of nonresponse bias, reduced residual (or post-adjustment) nonresponse bias, and substantially reduced total-mean-squared error of the estimates. This good news was largely unexpected based upon the preponderance of literature suggesting data quality could be adversely affected by the transition to a mixed mode. The bad news is that the transition comes with a high risk of mode effects for comparing Wave V and prior wave estimates. Analytical results suggest that significant differences can occur in longitudinal change estimates about 60 % of the time purely as an artifact of the redesign. This begs the question: how, then, should a data analyst interpret significant findings in a longitudinal analysis in the presence of mode effects? This chapter presents the analytical results and attempts to address this question.

Testing the Effectiveness of Alternative Foot-in-the-Door Manipulations

1980 ◽  
Vol 17 (3) ◽  
pp. 359-364 ◽  
Author(s):  
Robert A. Hansen ◽  
Larry M. Robinson
Keyword(s):  
Response Times ◽  
Short Form ◽  
Statistical Significance ◽  
Response Rates ◽  
Future Research ◽  
Business Research ◽  
Control Groups ◽  
Treatment Groups ◽  
Managerial Implications ◽  
Single Contact

Recently the foot-in-the-door principle was applied in a typical business research setting by Reingen and Kernan. Results of this single-contact or nondelay foot application were mixed; compliance rates for the foot treatment groups lacked statistical significance when compared with those of appropriate control groups. A field experiment undertaken to explain these results yields evidence that the specific nature of the foot manipulation in part determines its effectiveness. The study compares the effectiveness of high and low involvement foot manipulations in generating compliance with both short and long forms of a mail questionnaire. The high and the low involvement foot manipulations generate significantly higher response rates and faster response times than a noncontact control situation. In addition, the short form questionnaires generate a higher return rate. High involvement foot treatments generate higher response rates than low involvement foot treatments. Managerial implications of the results and suggestions for future research are presented.

Rethinking Response Rates: New Evidence of Little Relationship Between Survey Response Rates and Nonresponse Bias

2018 ◽  
Vol 43 (5) ◽  
pp. 307-330 ◽  
Author(s):  
Richard Hendra ◽  
Aaron Hill
Keyword(s):  
Survey Methodology ◽  
Response Rate ◽  
Evaluation Research ◽  
Response Rates ◽  
High Response Rate ◽  
Survey Response ◽  
High Response ◽  
High Rate ◽  
Nonresponse Bias ◽  
Survey Response Rate

Background: Federally funded evaluation research projects typically strive for an 80% survey response rate, but the increasing difficulty and expense in reaching survey respondents raises the question of whether such a threshold is necessary for reducing bias and increasing the accuracy of survey estimates. Objectives: This analysis focuses on a particular component of survey methodology: the survey response rate and its relationship to nonresponse bias. Following a review of the literature, new analysis of data from a large, multisite random assignment experiment explores the relationship between survey response rates and measured nonresponse bias. Research Design: With detailed survey disposition data, the analysis simulates nonresponse bias at lower response rates. The subjects included 12,000 individuals who were fielded for 16 identical surveys as part of the Employment Retention and Advancement evaluation. Results: The results suggest scant relationship between survey nonresponse bias and response rates. The results also indicate that the pursuit of high response rates lengthens the fielding period, which can create other measurement problems. Conclusions: The costly pursuit of a high response rate may offer little or no reduction of nonresponse bias. Achieving such a high rate of response requires considerable financial resources that might be better applied to methods and techniques shown to have a greater effect on the reduction of nonresponse bias.

Response Rates for Mixed-Mode Surveys Using Mail and E-mail/Web

2008 ◽  
Vol 29 (1) ◽  
pp. 99-107 ◽  
Author(s):  
Patrick D. Converse ◽  
Edward W. Wolfe ◽  
Xiaoting Huang ◽  
Frederick L. Oswald
Keyword(s):  
Mixed Mode ◽  
Design Method ◽  
Mail Survey ◽  
Response Rates ◽  
Initial Contact ◽  
Web Survey ◽  
Web Based ◽  
Tailored Design ◽  
E Mail

This study examines response rates for mixed-mode survey implementation involving mail and e-mail/Web components. Using Dillman's Tailored Design Method, 1,500 participants were sent a survey either (a) via mail with a follow-up contact via e-mail that directed them to a Web-based questionnaire or (b) via e-mail that directed them to a Web-based questionnaire with a follow-up contact via mail. Results indicate that these mixed-mode procedures produce moderately high response rates. However, the mail survey tended to be more effective than the e-mail/Web survey, when serving either as the initial contact or as the follow-up contact. These results suggest that survey implementation involving mail followed by e-mail/Web, or even mail-only approaches, may result in larger samples than implementation involving e-mail/Web followed by mail.

scholarly journals Face-to-Face or Sequential Mixed-Mode Surveys Among Non-Western Minorities in the Netherlands: The Effect of Different Survey Designs on the Possibility of Nonresponse Bias

2015 ◽  
Vol 31 (1) ◽  
pp. 1-30 ◽  
Author(s):  
Johannes W.S. Kappelhof
Keyword(s):  
The Netherlands ◽  
Mixed Mode ◽  
Minority Groups ◽  
Survey Design ◽  
Age Groups ◽  
Response Rates ◽  
Nonresponse Bias ◽  
Auxiliary Variables ◽  
Survey Estimates

Abstract This article compares the quality of response samples based on a single mode CAPI survey design with the quality of response samples based on a sequential mixed-mode (CAWI-CATICAPI) survey design among four non-Western minority ethnic groups in the Netherlands. The quality is assessed with respect to the representativity of the response samples and the estimated potential for nonresponse bias in survey estimates based on auxiliary variables and the response rate. This article also investigates if these designs systematically enhance response rates differently among various sociodemographic subgroups based on auxiliary variables. Also, costs and cost-related issues particular to this sequential mixed-mode design are discussed. The results show that sequential mixed mode surveys among non-Western ethnic minorities in the Netherlands lead to less representative response samples and show more potential for nonresponse bias in survey estimates. Furthermore, the designs lead to systematic differences in response rates among various sociodemographic subgroups, such as older age groups. Both designs also cause some of the same sociodemographic subgroups to be systematically underrepresented among all non-Western ethnic minority groups. Finally, the results show that in this instance the cost savings did not outweigh the reduction in quality.

scholarly journals Estimating Nonresponse Bias and Mode Effects in a Mixed-Mode Survey

2011 ◽  
Vol 53 (5) ◽  
pp. 669-686 ◽  
Author(s):  
Peter Lugtig ◽  
Gerty J.L.M. Lensvelt-Mulders ◽  
Remco Frerichs ◽  
Assyn Greven
Keyword(s):  
Propensity Score ◽  
Mixed Mode ◽  
Sample Selection ◽  
Nonresponse Bias ◽  
Selection Effects ◽  
Separate Sample ◽  
Mode Effects ◽  
Internet Surveys ◽  
Internet Sample ◽  
Study Mode

In mixed-mode surveys, it is difficult to separate sample selection differences from mode-effects that can occur when respondents respond in different interview settings. This paper provides a framework for separating mode effects from selection effects by matching very similar respondents from different survey modes using propensity score matching. The answer patterns of the matched respondents are subsequently compared. We show that matching can explain differences in nonresponse and coverage in two Internet samples. When we repeat this procedure for a telephone and Internet sample however, differences persist between the samples after matching. This indicates the occurrence of mode effects in telephone and Internet surveys. Mode effects can be problematic; hence we conclude with a discussion of designs that can be used to explicitly study mode effects.

Sign in / Sign up

Export Citation Format

Share Document