Using dual frame designs to reduce nonresponse in telephone surveys

TLDR: In this article, Traugott et al. report on the results of a series of experiments designed to improve response rates for telephone surveys and assess the cost and error components of these designs.

Abstract: This article reports on the results of a series of experiments designed to improve response rates for telephone surveys. In three surveys telephone households were selected using both standard random digit dialing (RDD) techniques and lists of telephone numbers purchased from a commercial firm. In the RDD portions of the samples "cold contact" interviewing methods were used; in the list frame portions advance letters were mailed, and the listed household name was used in the introduction. Experiments were designed to test the effects on response rates of the advance letters and use of the listed household name as a means of establishing rapport. The advance letters increased response rates, but no difference could be attributed to the use of names. The mixture of RDD and list sampling techniques is also used to evaluate the effects of relative response rates on substantive findings. The cost consequences of these dual frame designs are assessed along a number of dimensions, and the cost and error components of these designs are discussed. Survey nonresponse error is partially a function of achieved response rates. The researcher's ability to increase response rates, in turn, depends on a number of survey design features-the topic of the survey, MICHAEL W. TRAUGOTT is Research Scientist in the Center for Political Studies, ROBERT M. GROVES is Associate Research Scientist in the Survey Research Center, and JAMES M. LEPKOWSKI is Assistant Research Scientist in the Survey Research Center, all at the University of Michigan. The data utilized in this paper were collected in conjunction with a contract with the Detroit News. Support for consideration of dual frame telephone survey designs was obtained from the U.S. Bureau of the Census. However, all of the analyses and interpretations presented here are the sole responsibility of the authors. The research assistance of Judy Connor and Kim Fridkin Kahn is gratefully acknowledged, as is the computing support of the University of Michigan. A version of this paper was presented at the annual meeting of the American Association for Public Opinion Research, St. Petersburg, Florida, 16-18 May 1986. Public Opinion Quarterly Volume 51:522-539 ? 1987 by the Amerncan Association for Public Opinion Research Published by The University of Chicago Press / 0033-362X/87/0051-04(1)/$2.50 This content downloaded from 157.55.39.180 on Mon, 25 Apr 2016 06:57:17 UTC All use subject to http://about.jstor.org/terms Dual Frame Designs to Reduce Nonresponse 523 the population studied, the efforts at refusal conversion, the duration of the interviewing period, information known about the sample persons, and a host of other factors. Inevitably, decisions regarding response rate goals require a balancing of the expected costs of these design attributes and the likely nonresponse error reduction which may result from them. When names, addresses, and telephone numbers are available on a frame, two techniques can be used to improve cooperation-advance letters or telephone calls informing sample persons that they have been selected for a study and indicating the need for cooperation in order to achieve accurate results. Brunner and Carroll (1967) found that an advance telephone call for a personal interview actually led to lower response rates. Groves and Magilavy (1981) observed that advance telephone calls for a telephone survey had no effect on final response rates in an RDD survey. Dillman, Gallegos, and Frey (1976) found that overall response rates were increased by about five percentage points using an advance letter, but that the content of the letter had little influence on the magnitude of the response rate increase. Sykes and Hoinville (1985) found no effect in a telephone survey in Great Britain. Brunner and Carroll (1969) found large increases in response rates to a telephone survey (30 percentage points) with a letter sent on university stationery, but a decline in response rates (6 percentage points) using one from a market research company. Response rates are also sensitive to the amount of effort used to contact sample households, but even this may be affected by characteristics of the sampling frame. The use of callbacks is a minimum condition for increasing response rates (Traugott, 1987), and refusal conversion techniques are also important for this purpose. In some surveys the number of callbacks is a fixed design feature, determined prior to the interviewing period. In other designs the maximum number of callbacks is determined by the length of the survey period. Since the time spent screening ineligible sample units detracts from time available for interviewing, the percentage of units on the sampling frame that are eligible can also affect response rates. In this way, the choice of sampling frame can have a consequence for nonresponse error. The achievement of high response rates is a goal linked not only to improved estimation of descriptive statistics but also increased confidence in measured relationships. While there has been considerable research on increasing response rates, there has been little on their effects on measured relationships between variables. This is ordinarily complicated by an inability to measure relationships for nonrespondents. The literature that does exist compares relationships between variables among cooperative and reluctant respondents (O'Neil, 1979; Smith, 1984). This content downloaded from 157.55.39.180 on Mon, 25 Apr 2016 06:57:17 UTC All use subject to http://about.jstor.org/terms 524 M. W. Traugott, R. M. Groves, and J. M. Lepkowski Telephone surveys based upon random digit dialed (RDD) samples have offered substantial advantages in reducing data collection costs relative to personal interview surveys, in large part because travel costs can be eliminated and the number of interviewers required to complete a given sample size is often smaller. The RDD method has, however, also resulted in a reduction in survey response rates (Groves and Kahn, 1979). In some part these reduced rates may be attributed to "cold contacts" with respondents about whom the interviewers know little. In addition, the lower response rates may be attributed to features of the RDD sampling frame, including the relatively high percentages of nonworking and nonresidential numbers that are part of this frame. The time requirements for screening out such sample numbers reduces the time available to pursue eligible numbers. This article summarizes a series of experiments conducted across three telephone surveys designed to measure cost and nonresponse advantages of a sampling frame based on telephone directories (termed the "list frame") in comparison with RDD samples.1 In contrast to prior uses of directories as the sole frame or as a "seeding" mechanism for RDD work (Sudman, 1973), these experiments explored their use in a dual frame design, jointly employing RDD and directory based methods. The research reported here focuses not on the statistical design issues involved in dual frame telephone surveys but on those properties of the list frame that make it a desirable companion to the RDD frame. The major test described in this paper involved a comparison of the response rate among list frame households sent an advance letter with the rate for those in which cold contacts were made. Following previous results it was hypothesized that the use of the letter would increase response rates. A subsidiary experimental treatment involved the use upon initial contact of the name in the directory listing for a portion of the households that were sent the letter. Assuming that personalized approaches emphasize the unique attributes of the sample person, it was hypothesized that the use of names in conjunction with the letter would further increase response rates. Because of the anticipated higher proportions of working household numbers on the list frame, and possible differences in cooperation 1. The first two of these three surveys used the same basic dual frame sample design, while the third used a two-phase design described in Lepkowski and Groves (1986b). There were other elements of each study which differed. The first survey was conducted between 18 October and 10 November 1985, involved 753 interviews which averaged 23 minutes in length, and achieved an overall response rate of 61%. The second survey was conducted between 14 February and 3 March 1986, involved 668 interviews which averaged 13 minutes in length, and achieved an overall response rate of 61%. The third survey, conducted between 24 May and 15 June 1986, involved 789 interviews which averaged 22 minutes in length, and achieved an overall response rate of 63%. This content downloaded from 157.55.39.180 on Mon, 25 Apr 2016 06:57:17 UTC All use subject to http://about.jstor.org/terms Dual Frame Designs to Reduce Nonresponse 525 among respondents with listed and unlisted numbers, it was hypothesized that response rates would be higher in the list frame than in the RDD frame, for an equivalent level of effort at contacting respondents. The effects of these design features on survey costs, measured as efficiency in obtaining interviews, were evaluated under controlled conditions in a fixed field period and achievement of a designated final response rate. Examples are also presented of differences in observed bivariate relationships among respondents who were sent a letter (the "high response" sample) and those who were not (the "low response" sample). Elements of the Dual Frame Design for Telephone