Rethinking Response Rates: New Evidence of Little Relationship Between Survey Response Rates and Nonresponse Bias

Background: Federally funded evaluation research projects typically strive for an 80% survey response rate, but the increasing difficulty and expense in reaching survey respondents raises the question of whether such a threshold is necessary for reducing bias and increasing the accuracy of survey estimates. Objectives: This analysis focuses on a particular component of survey methodology: the survey response rate and its relationship to nonresponse bias. Following a review of the literature, new analysis of data from a large, multisite random assignment experiment explores the relationship between survey response rates and measured nonresponse bias. Research Design: With detailed survey disposition data, the analysis simulates nonresponse bias at lower response rates. The subjects included 12,000 individuals who were fielded for 16 identical surveys as part of the Employment Retention and Advancement evaluation. Results: The results suggest scant relationship between survey nonresponse bias and response rates. The results also indicate that the pursuit of high response rates lengthens the fielding period, which can create other measurement problems. Conclusions: The costly pursuit of a high response rate may offer little or no reduction of nonresponse bias. Achieving such a high rate of response requires considerable financial resources that might be better applied to methods and techniques shown to have a greater effect on the reduction of nonresponse bias.

Download Full-text

Influence of Paper Color and a Monetary Incentive on Response Rate

Psychological Reports ◽

10.2466/pr0.95.2.432-434 ◽

2004 ◽

Vol 95 (2) ◽

pp. 432-434 ◽

Cited By ~ 2

Author(s):

Keith A. King ◽

Jennifer L. Vaughan

Keyword(s):

Eating Disorders ◽

Response Rate ◽

Athletic Trainers ◽

Response Rates ◽

Monetary Incentive ◽

Survey Response ◽

Green Paper ◽

Survey Response Rate

This study examined whether survey response rate differed based on the color of the paper the survey was printed on (blue vs green) and presence of a monetary incentive. A 4-page survey on eating disorders was mailed to Division 1A and 1AA college head athletic trainers ( N = 223) with half of the surveys on blue paper and half on green paper. Half of the athletic trainers ( n = 111) received a $1.00 monetary incentive, and half ( n = 112) received no monetary incentive. A total of 166 (71%) athletic trainers returned completed surveys. Response rates did not differ based on survey color but did differ based on presence of a monetary incentive. Athletic trainers who received a monetary incentive were significantly more likely than those who did not to return completed surveys (86% vs 63%, respectively).

Download Full-text

The effect of a web-push survey on physician survey response rates: a randomized experiment.

10.21203/rs.2.12242/v1 ◽

2019 ◽

Cited By ~ 1

Author(s):

Cristine D. Delnevo ◽

Binu Singh

Keyword(s):

Survey Methodology ◽

Response Rate ◽

Mail Survey ◽

Cost Effective ◽

Response Rates ◽

National Sample ◽

High Response Rate ◽

Future Research ◽

Effective Delivery ◽

The Web

Abstract Background: Achieving a high response rate for physicians has been challenging and with response rates declining in recent years, innovative methods are needed to increase rates. An emerging concept in survey methodology has been web-push survey delivery. In this delivery method, contact is made by mail to request a response by web. This study explored the feasibility of a web-push survey on a national sample of physicians. Methods: 1000 physicians across six specialties were randomly assigned to a mail only or web-push survey delivery. Each mode consisted of four contacts including an initial mailing, reminder postcard, and two additional follow-ups. Response rates were calculated using AAPOR’s response rate 3 calculation. Data collection occurred between Febuary – April 2018 and analyzed March 2019. Results: Overall reponse rates for the mail only vs. web-push survey delivery were comparable (51.2% vs. 52.8%). Higher response rates across all demographics were seen in the web-push delivery with the exception of pulmonary/critical care and physicians over the age of 65. The web-push survey yielded a greater response after the first mailing requiring fewer follow-up contacts resulting in a more cost-effective delivery. Conclusions: A web-push mail survey is effective in achieveing a comparable response rate to traditional mail only delivery for physicians. The web-push survey was more efficient in terms of cost and in receiving responses in a more timely manner. Future research should explore the efficiency of a web-push survey delivery across various health care provider populations.

Download Full-text

Encouraging Physicians to Respond to Surveys through the Use of Fax Technology

Evaluation & the Health Professions ◽

10.1177/01632780022034642 ◽

2000 ◽

Vol 23 (3) ◽

pp. 348-359 ◽

Cited By ~ 13

Author(s):

Shelly Y. Lensing ◽

Stephen R. Gillaspy ◽

Pippa M. Simpson ◽

Stacie M. Jones ◽

John M. James ◽

...

Keyword(s):

Observational Study ◽

General Practitioners ◽

Response Rate ◽

Pediatric Asthma ◽

Family Physicians ◽

Response Rates ◽

Survey Response ◽

Telephone Surveys ◽

Survey Response Rate ◽

Survey Participation

High response rates to surveys of physicians are difficult to achieve. One possible strategy to improve physicians’ survey participation is to offer the option of receiving and returning the survey by fax. This study describes the success of the option of fax communication in a survey of general practitioners, family physicians, and pediatricians in Arkansas with regard to pediatric asthma. Eligible physicians were given the choice of receiving the survey by telephone, mail, or fax. In this observational study, physicians’ preferences, response rates, and biases for surveys administered by fax were compared with mail and telephone surveys. The overall survey response rate was 59%. For the 96 physicians completing an eligibility screener survey, the largest percentage requested to be surveyed by fax (47%) rather than by telephone (28%) or mail (25%). Faxing may be one strategy to add to the arsenal of tools to increase response rates in surveying physicians.

Download Full-text

Mail Survey Response Rate and Bias: The Effect of Home versus Work Address

Journal of Marketing Research ◽

10.1177/002224377701400316 ◽

1977 ◽

Vol 14 (3) ◽

pp. 383-384 ◽

Cited By ~ 2

Author(s):

Michael A. Mcginnis ◽

Charles J. Hollon

Keyword(s):

Response Rate ◽

Mail Survey ◽

Response Rates ◽

Survey Response ◽

Questionnaire Item ◽

Survey Response Rate ◽

Professional Employees ◽

At Home

A study was undertaken to ascertain whether the response rate and responses of a group of technically trained professional employees differed between those who received the questionnaire at work and those who received it at home. Response rates were found to be independent of address, and the frequency of questionnaire item significance was not significantly different from chance.

Download Full-text

Effect of Respondent Code Location on Survey Response Rate

Psychological Reports ◽

10.2466/07.pr0.109.6.718-722 ◽

2011 ◽

Vol 109 (3) ◽

pp. 718-722

Author(s):

Keith A. King ◽

Rebecca A. Vidourek

Keyword(s):

Health Education ◽

Response Rate ◽

Response Rates ◽

Faculty Members ◽

Survey Response ◽

Numerical Code ◽

Program Coordinators ◽

Survey Response Rate ◽

Health Education Faculty ◽

Return Envelope

Survey response rates were examined for differences based on the location of the respondent code, a numerical code placed on the survey versus a numerical code placed on the return envelope. A four-page survey on criteria and procedures used when hiring health education faculty members was mailed to program coordinators of university health education programs ( N = 223). Half of the coordinators ( n = 112) received coded surveys (surveys that had a written overt three-digit code at the top right corner of the first page of the survey). The other half of the coordinators ( n = 111) received coded return envelopes (return envelopes with a six-digit code typed directly under the return mailing address). A total of 132 coordinators returned completed surveys (62%). Response rate did not differ significantly based on placement of respondent code. Survey researchers are thus recommended to devote their efforts to other inducement strategies to increase response rates.

Download Full-text

Response Rates and Perceived Questionnaire Length in Mail Surveys

Journal of Marketing Research ◽

10.1177/002224377901600318 ◽

1979 ◽

Vol 16 (3) ◽

pp. 429-431 ◽

Cited By ~ 12

Author(s):

Terry L. Childers ◽

O. C. Ferrell

Keyword(s):

Interaction Effect ◽

Response Rate ◽

Mail Survey ◽

Response Rates ◽

Survey Response ◽

Factorial Experiment ◽

Mail Surveys ◽

Survey Response Rate ◽

Significant Difference ◽

The Impact

A 2 × 2 factorial experiment was designed to test the impact on mail survey response rate resulting from variations in paper trim size and number of printed pages in the questionnaire. ANOVA findings suggest 8½ × 11″ paper trim size produces a better response rate than an 8½ × 14″ paper trim size. Use of a one-sheet (front and back) versus a two-sheet (front only) questionnaire did not cause a significant difference in response rate; a hypothesized interaction effect was not found to be statistically significant.

Download Full-text

A $2 Bill or Two $1 Bills: An Experiment that Challenges Standard Protocol

Field Methods ◽

10.1177/1525822x19844307 ◽

2019 ◽

Vol 31 (3) ◽

pp. 230-240

Author(s):

Sarah Mills

Keyword(s):

Statistical Difference ◽

Response Rate ◽

Mail Survey ◽

Response Rates ◽

Survey Response ◽

The Other ◽

Standard Protocol ◽

Survey Response Rate ◽

Survey Participation ◽

To Receive

The $2 bill is commonly used as a prepaid incentive in survey administration. Despite the widespread use of this incentive, there is no peer-reviewed research into how providing a single $2 bill rather than two $1 bills affects survey response rate. The results are from a mail survey of 2,970 rural landowners in Michigan with identical survey protocols. Half were randomly assigned to receive a single $2 bill and the other half to receive two $1 bills accompanying the questionnaire. I found no statistical difference in the response rates of the two groups. While there may be logistical advantages to using a single $2 bill compared to two $1 bills, this research suggests that the $2 bill is not more effective at inducing survey participation.

Download Full-text

Pre-Printed Versus Handwritten Post-It Notes’ Impact on Survey Response in Hard to Recruit Survey Recipients: A Randomized Controlled Trial in Veterans with PTSD Disability

10.31234/osf.io/k4dx7 ◽

2019 ◽

Author(s):

Maureen Murdoch ◽

Michele Spoont ◽

Eileen Harwood ◽

Barbara Clothier ◽

Ann Bangerter ◽

...

Keyword(s):

United States ◽

Randomized Controlled Trial ◽

Response Rate ◽

Controlled Trial ◽

Response Rates ◽

Survey Response ◽

Intention To Treat ◽

Randomized Controlled ◽

Survey Response Rate ◽

Mailed Survey

Our goal was to see if affixing pre-printed Post-it note messages to a mailed survey would generate response rates similar to when handwritten Post-it notes were used in hard to recruit survey recipients. The study was a nested, randomized controlled trial of 575 individuals from a cohort of 2,100 United States Veterans with PTSD disability. Analysis was intention-to-treat. Survey response rate was 38.5% in the pre-printed Post-it note arm and 44.9% in the handwritten Post-it note arm (p = 0.09). In non-inferiority testing, pre-printed notes’ response rate exceeded the 10% margin of inferiority but was less than the 15% margin when compared to handwritten notes.

Download Full-text