Part I
Survey Data



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues Part I Survey Data

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues This page in the original is blank.

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues 1 Designing Surveys Acknowledging Nonresponse Robert M.Groves and Mick P.Couper THE NATURE OF NONRESPONSE ERROR IN SURVEY STATISTICS Sample surveys used to describe low-income populations are effective only when several things go “right.” The target population must be defined well, having the geographical and temporal extents that fit the goals of the survey. The sampling frame, the materials used to identify the population, must include the full target population. The measurement instrument must be constructed in a way that communicates the intent of the research question to the respondents, ideally in their nomenclature and within their conceptual framework. The sample design must give known, nonzero chances of selection to each low-income family/person in the sampling frame. All sample persons must be contacted and measured, eliminating nonresponse error. Finally, the administration of the measurement instrument must be conducted in a manner that fulfills the design. Rarely does everything go exactly right. Because surveys are endeavors that are (1) customized to each problem, and (2) constructed from thousands of detailed decisions, the odds of imperfections in survey statistics are indeed large. As survey methodology, the study of how alternative survey designs affect the quality of statistics, matures, it is increasingly obvious that errors are only partially avoidable in surveys of human populations. Instead of having the goal of eliminating errors, survey researchers must learn how to reduce them “within reason and budget” and then attempt to gain insight into their impacts on key statistics in the survey. This paper is a review of a large set of classic and recent findings in the study of survey nonresponse, a growing concern about survey quality. It begins with a

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues review of what nonresponse means and how it affects the quality of survey statistics. It notes that nonresponse is relevant to simple descriptive statistics as well as measures of the relationship between two attributes (e.g., length of time receiving benefits and likelihood of later job retention). It then reviews briefly what survey statisticians can do to reduce the impact of nonresponse after the survey is complete, through various changes in the analysis approach of the data. After this brief overview of the basic approaches to reducing the impacts of nonresponse on statistical conclusions from the data concludes, the paper turns to reducing the problem of nonresponse. It reviews current theoretical viewpoints on what causes nonresponse as well as survey design features that have been found to be effective in reducing nonresponse rates. Nonresponse Rates and Their Relationship to Error Properties Sample surveys often are designed to draw inferences about finite populations by measuring a subset of the population. The classical inferential capabilities of the survey rest on probability sampling from a frame covering all members of the population. A probability sample assigns known, nonzero chances of selection to every member of the population. Typically, large amounts of data from each member of the population are collected in the survey. From these variables, hundreds or thousands of different statistics might be computed, each of which is of interest to the researcher only if it describes well the corresponding population attribute. Some of these statistics describe the population from which the sample was drawn; others stem from using the data to test causal hypotheses about processes measured by the survey variables (e.g., how length of time receiving welfare payments affects salary levels of subsequent employment). One example statistic is the sample mean as an estimator of the population mean. This is best described by using some statistical notation in order to be exact in our meaning. Let one question in the survey be called the question, “Y,” and the answer to that question for a sample member, say the ith member of the population, be designated by Yi. Then we can describe the population, mean by (1) where N is the number of units in the target population. The estimator of the population mean is often (2) where r is the number of respondents in the sample and wi is the reciprocal of the probability of selection of the ith respondent. (For readers accustomed to equal probability samples, as in a simple random sample, the wi is the same for all cases in the sample and the computation above is equivalent to ∑yi/n.)

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues One problem with the sample mean as calculated here is that is does not contain any information from the nonrespondents in the sample. However, all the desirable inferential properties of probability sample statistics apply to the statistics computed on the entire sample. Let’s assume that in addition to the r respondents to the survey, there are m (for “missing”) nonrespondents. Then the total sample size is n=r+m. In the computation mentioned we miss information on the m missing cases. How does this affect our estimation of the population mean, Y¯? Let’s make first a simplifying assumption. Assume that everyone in the target population is either, permanently and forevermore, a respondent or a nonrespondent. Let the entire target population, thereby, be defined as N=R+M, where the capital letters denote numbers in the total population. Assume that we are unaware at the time of sample selection about which stratum each person occupies. Then in drawing our sample of size n, we will likely select some respondents and some nonrespondents. They total n in all cases, but the actual number of respondents and nonrespondents in any one sample will vary. We know that in expectation that the fraction of sample cases that are respondents should be equal to the fraction of population cases that lie in the respondent stratum, but there will be sampling variability about that number. That is, E(r)=fR, where f is the sampling fraction used to draw the sample from the population. Similarly, E(m)=fM. For each possible sample we could draw, given the sample design, we could express a difference between the full sample mean, n, and the respondent mean, in the following way: (3) which, with a little manipulation, becomes (4) RESPONDENT MEAN-TOTAL SAMPLE MEAN=(NONRESPONSE RATE)* (DIFFERENCE BETWEEN RESPONDENT AND NONRESPONDENT MEANS) This shows that the deviation of the respondent mean from the full sample mean is a function of the nonresponse rate (m/n) and the difference between the respondent and nonrespondent means. Under this simple expression, what is the expected value of the respondent mean over all samples that could be drawn given the same sample design? The answer to this question determines the nature of the bias in the respondent mean, where “bias” is taken to mean the difference between the expected value (over all possible samples given a specific design) of a statistic and the statistic computed on the target population. That is, in cases of equal probability samples of fixed size, the bias of the respondent mean is approximately

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues (5) BIAS(RESPONDENT MEAN)=(NONRESPONSE RATE IN POPULATION)* (DIFFERENCE IN RESPONDENT AND NONRESPONDENT POPULATION MEANS) where the capital letters denote the population equivalents to the sample values. This shows that the larger the stratum of nonrespondents, the higher the bias of the respondent mean, other things being equal. Similarly, the more distinctive the nonrespondents are from the respondents, the larger the bias of the respondent mean. These two quantities, the nonresponse rate and the differences between respondents and nonrespondents on the variables of interest, are key issues to surveys of the welfare population. Figures 1–1a to 1–1d through show four alternative frequency distributions for respondents and nonrespondents on a hypothetical variable, y, measured on all cases in some target population. The area under the curves is proportional to the size of the two groups, respondents and nonrespondents. These four figures correspond to the four rows in Table 1–1 that show response rates, means of respondents and nonrespondents, bias, and percentage bias for each of the four cases. The first case reflects a high response rate survey and one in which the nonrespondents have a distribution of y values quite similar to that of the respon- FIGURE 1–1a High response rate, nonrespondents similar to respondents. SOURCE: Groves and Couper (1998). NOTE: y=outcome variable of interest.

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues FIGURE 1–1b High response rate, nonrespondents different from respondents. SOURCE: Groves and Couper (1998). NOTE: y=outcome variable of interest. FIGURE 1–1c Low response rate, nonrespondents similar to respondents. SOURCE: Groves and Couper (1998). NOTE: y=outcome variable of interest.

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues FIGURE 1–1d Low response rate, nonrespondents different from respondents SOURCE: Groves and Couper (1998). NOTE: y=outcome variable of interest. dents. This is the lowest bias case; both factors in the nonresponse bias are small. For example, assume the response rate is 95 percent, the respondent mean for reported expenditures on clothing for a quarter is $201.00, and the mean for nonrespondents is $228.00. Then the nonresponse error is .05($201.00-$228.00) =-$1.35. The second case, like the first, is a low nonresponse survey, but now the nonrespondents tend to have much higher y values than the respondents. This means that the difference term, (), is a large negative number, meaning the respondent mean underestimates the full population mean. However, the size of the bias is small because of the low nonresponse rate. Using the same example as above, with a nonrespondent mean now of $501.00, the bias is .05($201.00-$501.00)=-$15.00. The third case shows a very high nonresponse rate (the area under the respondent distribution is about 50 percent greater than that under the nonrespondent—a nonresponse rate of 40 percent). However, as in the first graph, the values on y of the nonrespondents are similar to those of the respondents. Hence, the respondent mean again has low bias due to nonresponse. With the same example as mentioned earlier, the bias is .40($201.00-$228.00)=[-$10.80]. The fourth case is the most perverse, exhibiting a large group of nonrespondents who have much higher values in general on y than the respondents. In this case, both m/n is large (judging by the area under the nonrespondent curve) and () is large in absolute terms. This is the case of large non-

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues TABLE 1–1 Bias and Percentage Bias in Respondent Mean Relative to Total Sample Mean for Four Situations in Figures 1–1a–1 to 1d and Sample Size of Nonrespondents Needed to Detect the Nonresponse Bias Response Rate Difference Response Rate Percentage Respondent Mean Nonrespondent Mean Total Sample Mean Bias Bias Percentage Required Sample Size of Nonrespondents High Small 95 $201 $228 $202 $1.35 -0.7 20,408 High Large 95 $201 $501 $216 $15.00 -6.9 210 Low Small 60 $201 $228 $212 $10.80 -5.1 304 Low Large 60 $201 $501 $321 $120.00 -37.4 7

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues response bias. Using the previous example, the bias is .40($201.00–$501.00)=−$120.00, a relative bias of 37 percent compared to the total sample mean! These four very different situations also have implications for studies of nonrespondents. Let’s imagine we wish to mount a special study of nonrespondents in order to test whether the respondent mean is biased. The last column of Table 1–1 shows the sample size of nonrespondents required to obtain the same stability for a bias ratio estimate (assuming simple random sampling and the desire to estimate a binomial mean statistic with a population value of .50). The table shows that such a nonresponse study can be quite small (n=7) and still be useful to detect the presence of nonresponse bias in a low-response-rate survey with large differences between respondents and nonrespondents (the fourth row of the table). However, the required sample size to obtain the same precision for such a nonresponse bias test in the high-response-rate case is very large (n=20,408, in the first row). Unfortunately, prior to a study being fielded, it is not possible to have much information on the size of the likely nonresponse bias. Nonresponse Error on Different Types of Statistics The discussion in the previous section focused on the effect of nonresponse on estimates of the population mean, using the sample mean. This section briefly reviews effects of nonresponse on other popular statistics. We examine the case of an estimate of a population total, the difference of two subclass means, and a regression coefficient. The Population Total Estimating the total number of some entity is common in federal, state, and local government surveys. For example, most countries use surveys to estimate the total number of unemployed persons, the total number of new jobs created in a month, the total retail sales, and the total number of criminal victimizations. Using similar notation as previously, the population total is ∑Yi, which is estimated by a simple expansion estimator, ∑wiyi, or by a ratio expansion estimator, X(∑wiyi/∑wixi), where X is some auxiliary variable, correlated with Y, for which target population totals are known. For example, if y were a measure of the length of first employment spell of a welfare leaver, and x were a count of sample welfare leavers, X would be a count of the total number of welfare leavers. For variables that have nonnegative values (like count variables), simple expansion estimators of totals based only on respondents always underestimate the total. This is because the full sample estimator is (6)

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues FULL SAMPLE ESTIMATE OF POPULATION TOTAL=RESPONDENT-BASED ESTIMATE+NONRESPONDENT-BASED ESTIMATE Hence, the bias in the respondent-based estimator is (7) It is easy to see, thereby, that the respondent-based total (for variables that have nonnegative values) always will underestimate the full sample total, and thus, in expectation, the full population total. The Difference of Two Subclass Means Many statistics of interest from sample surveys estimate the difference between the means of two subpopulations. For example, the Current Population Survey often estimates the difference in the unemployment rate for black and nonblack men. The National Health Interview Survey estimates the difference in the mean number of doctor visits in the past 12 months between males and females. Using the expressions above, and using subscripts 1 and 2 for the two subclasses, we can describe the two respondent means as (8) (9) These expressions show that each respondent subclass mean is subject to an error that is a function of a nonresponse rate for the subclass and a deviation between respondents and nonrespondents in the subclass. The reader should note that the nonresponse rates for individual subclasses could be higher or lower than the nonresponse rates for the total sample. For example, it is common that nonresponse rates in large urban areas are higher than nonresponse rates in rural areas. If these were the two subclasses, the two nonresponse rates would be quite different. If we were interested in as a statistic of interest, the bias in the difference of the two means would be approximately (10) Many survey analysts are hopeful that the two terms in the bias expression cancel. That is, the bias in the two subclass means is equal. If one were dealing with two subclasses with equal nonresponse rates that hope is equivalent to a

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues experienced interviewers acquired a wider repertoire of persuasion techniques, but they are also better able to select the most appropriate approach for each situation. Maintaining interaction. The introductory contact of the interviewer and householder is a small conversation. It begins with the self-identification of the interviewer, contains some descriptive matter about the survey request, and ends with the initiation of the questioning, a delay decision, or the denial of permission to continue. There are two radically different optimization targets in developing an introductory strategy—maximizing the number of acceptances per time unit (assuming an ongoing supply of contacts), and maximizing the probability of each sample unit accepting. The first goal is common to some quota sample interviewing (and to sales approaches). There, the supply of sample cases is far beyond that needed for the desired number of interviews. The interviewer behavior should be focused on gaining speedy resolution of each case. An acceptance of the survey request is preferred to a denial, but a lengthy, multicontact preliminary to an acceptance can be as damaging to productivity as a denial. The system is driven by number of interviews per time unit. The second goal, maximizing the probability of obtaining an interview from each sample unit, is the implicit aim of probability sample interviewing. The amount of time required to obtain cooperation on each case is of secondary concern. Given this, interviewers are free to apply the “tailoring” over several turns in the contact conversation. How to tailor the appeal to the householder is increasingly revealed as the conversation continues. Hence, the odds of success are increased as the conversation continues. Thus, the interviewer does not maximize the likelihood of obtaining a “yes” answer in any given contact, but minimizes the likelihood of a “no” answer over repeated turntaking in the contact. We believe the techniques of tailoring and maintaining interaction are used in combination. Maintaining interaction is the means to achieve maximum benefits from tailoring, for the longer the conversation is in progress, the more cues the interviewer will be able to obtain from the householder. However, maintaining interaction is also a compliance-promoting technique in itself, invoking the commitment principle as well as more general norms of social interaction. That is, as the length of the interaction grows, it becomes more difficult for one actor to summarily dismiss the other. Figure 1–9 is an illustration of these two interviewer strategies at work. We distinguish between the use of a general compliance-gaining strategy (e.g., utilizing the principle of authority) and a number of different (verbal and nonverbal) arguments or tactics within each strategy (e.g., displaying the ID badge prominently, emphasizing the sponsor of the survey). The successful application of tailoring depends on the ability of the interview to evaluate the reaction of the householder to his or her presence, and the effectiveness of the arguments pre-

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues FIGURE 1–9 Interviewer behavior during interaction with householders. SOURCE: Groves and Couper (1998).

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues sented. Note that the interviewer’s initial goal is to maintain interaction (avoiding pushing for the interview) as long as the potential respondent’s reaction remains neutral or noncommittal. An interviewer will continue to present different arguments until the householder is clearly receptive to an interview request, or there are no more arguments to present. For inexperienced interviewers the latter may occur before the former, forcing the interviewer to (prematurely in some cases) initiate the interview request. There is some support from training procedures that the “maintaining interaction” model operates as theorized. First, interviewers typically are warned against unintentionally leading the householder into a quick refusal. If the person appears rushed or preoccupied by some activity in the household (e.g., fighting among children), the interviewer should seek another time to contact the unit. A common complaint concerning inexperienced interviewers is that they create many “soft refusals” (i.e., cases easily converted by an experienced interviewer) by pressing the householder into a decision prematurely. Unfortunately, only rarely do interviewer recruits receive training in the multiturn repartee inherent in maximizing the odds of a “yes” over all contacts. Instead, they are trained in stock descriptors of the survey leading to the first question of the interview. We note how similar the goals of a quota sample interviewer are to those of any salesperson, but how different are those of the probability sample interviewer. Given this, it is not surprising that many attempts to use sales techniques in probability sample surveys have not led to large gains in cooperation. The focus of the salesperson is on identifying and serving buyers. The “browser” must be ignored when a known buyer approaches. In contrast, the probability sample interviewer must seek cooperation from both the interested and uninterested. At the same time that the interviewer is exercising skills regarding tailoring and maintaining interaction, the householder is engaged in a very active process of determining whether there has been prior contact with the interviewer, what is the intent of the interviewer’s call, whether a quick negative decision is warranted, or whether continued attention to the interviewer’s speech is the right decision. Figure 1–10 describes this process. The process has various decision points at which the householder can make positive or negative decisions regarding participation in the survey. These arise because the householder misinterprets the visit as involving some unpleasant nonsurvey request; that is, the householder chooses the wrong script. They arise if there are very high opportunity costs for the householder to continue the interaction with the interviewer. They arise if any of the heuristics point to the wisdom of a negative or positive decision. DESIGNING SURVEYS ACKNOWLEDGING NONRESPONSE The previous discussions review various theoretical perspectives on nonresponse. These theoretical perspectives have two implications for survey design:

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues FIGURE 1–10 Householder behavior during interactions with the interviewer. SOURCE: Groves and Couper (1998).

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues (1) contact and interviewer protocols should be chosen to be consistent with the diverse influences, and (2) no single survey design will achieve 100-percent response rates and defenses to nonresponse error should be built into the chosen survey design. The Value of Rich Sampling Frames The list of the target population (or the materials used to construct a list) is a tool to assure that a probability sample will offer a useful description of the full population. When the designer acknowledges that nonresponse inevitably will occur in the survey, the frame takes on new value. When the designer has a choice of frames (e.g., a list frame from a social welfare agency containing data about the person’s prior income and employment experience, an area frame, a random digit dial frame), evaluation of the frame must include both coverage and nonresponse issues. Coverage, the extent to which the frame includes all target population elements and nothing else, is an important attribute. Sampling frames that contain information beyond simple identifiers can help reduce nonresponse error. If the frames include data on prior addresses, then those with a history of moves might be identified as likely movers, with higher than expected locating effort. If frames contain data on use of agency services in the past, the data might be used to customize approaches to sample persons in an effort to address potential interests and concerns about survey participation (e.g., having interviewers explain the importance of the survey to measuring the wellbeing of former food stamp recipients). Sometimes data exists that are correlates of key survey variables (e.g., participation in types of programs [higher correlated with current statuses like work training programs]). Such data might be useful in assessing nonresponse errors and building weighting adjustment or imputation models. Collecting Additional Information to Enhance Data Collection Sometimes interviewers can observe that sample persons have certain attributes that are related to certain concerns about survey participation. Followup efforts to persuade the sample person to be interviewed can use this information creatively to improve response rates. This has included treating sample numbers generating answering machine responses as candidates for calling at different times of the day, attempting to avoid times when the machine is activated. It includes interviewer observation about any concerns regarding the legitimacy of the survey request, followed by special mailings or communications demonstrating the sponsorship and purpose of the survey. It includes, in face-to-face surveys, observations of housing units for entrance impediments (e.g., locked apartment buildings, locked gates, security guards), leading to more intensive calling patterns on those sample units versus others.

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues Collecting Information Valuable in Postsurvey Adjustment Several studies (Purdon et al, 1999; Groves and Couper, 1998; Brehm, 1993) now demonstrate that the utterances of sample persons during their interactions with interviewers contain some information regarding motivations for their reaction to the survey request and the likelihood of eventual cooperation with the survey request. The evidence comes more from face-to-face surveys than from telephone surveys, although Couper and Groves (1995) find some support for links between the utterances and the final outcome in telephone surveys as well. These become useful predictors in response-propensity models sometimes used in postsurvey adjustment. Another set of variables involves fixed attributes of the housing unit, best observed in face-to-face surveys. For example, the Consumer Expenditure Survey uses observations from the interviewer about whether the unit is owned or rented in postsurvey adjustments based on the belief that the consumption patterns are relatively homogeneous in the two groups. Similarly, observations of multiple-person households (through records on who answered the telephone) the presence of children, etc. are possible in some designs. These too can be useful in forming postsurvey adjustment weighting classes. Two-Phase Sampling to Acquire Information About Nonrespondents When survey data are used in legal or policy settings, the credibility of results is sometimes enhanced by mounting separate studies concerning nonresponse. There are two possible foci: experimental comparisons of different protocols and two-phase sample surveys of nonrespondents. An example of the first study is a mixed-mode design based on a list frame sample of prior recipients, one mode using telephone matching and telephone survey requests; and the other uses address locating and face-to-face interviews. For cost reasons the face-to-face mode might use a smaller sample size than the telephone mode. The telephone mode is likely to have lower response rates than the face-to-face mode. The sample sizes might be fixed to determine the magnitude of mode differences at some prior specified standard error. The total cost of the survey per unit measured lies between the telephone and face-to-face modes, but the additional information purchased with the mixed-mode design is protection against large-mode effects on key survey conclusions. A two-phase sample design for nonresponse studies begins after the main survey has completed its work. The intent under perfect conditions is that a probability subsample of nonrespondents to the first phase of the survey can yield evidence regarding the likelihood of large nonresponse errors in the first-phase estimates. The “perfect” conditions yield 100 percent response rates on the second-phase cases, thus providing unbiased estimates of the characteristics of the nonrespondent pool. Although such designs have a long history (Deming, 1953;

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues Hansen and Hurwitz, 1958), they never inevitably achieve the perfect conditions in practice. They are used, however, when some information on the nonrespondents is judged to be of crucial importance. For example, a second-phase sample of nonrespondents was taken on the National Survey of American Families, using a radically reduced telephone interview, relaxed respondent rules, and an incentive offer. Among the nonrespondent cases to the first-phase effort (spanning many months and repeated refusal conversion efforts), 36 percent of screener nonrespondents and 58 percent of full interview nonrespondents complied with the second-phase request (Groves et al., 1999). Those responding were found not to have large socioeconomic status differences from the respondent group (what differences did exist suggested higher income households were more likely to be nonrespondents). JUDGMENTS REGARDING DESIRABLE DESIGN FEATURES FOR SURVEYS OF THE U.S. LOW-INCOME POPULATION As survey methodology matures, it is increasingly finding that the process of survey participation is subject to diverse causes across different subgroups. In short, what “works” for some groups does not for others. Furthermore, in free societies 100 percent compliance is not to be expected; survey designers should incorporate nonresponse concerns into every aspect of their designs. What follows is a listing of the top 10 lessons from the survey methodology literature regarding nonresponse in studies of the low-income population. These are current judgments of the authors of this paper based on experience and study of the field. 1. No record system is totally accurate or complete. Using a record system as a sampling frame generally asks more of the record than it was designed to provide. Surveys demand accurate, up-to-date, personal identifiers. They demand that the person sampled can be located. 2. Choose sample sizes that permit adequate locating, contacting, and recruitment efforts. Sample surveys suffer from the tyranny of the measurable, with sampling errors dominating design decisions because they can be measured more easily than nonresponse errors. It is tempting to assume the absence of nonresponse error and to maximize sample size to achieve low reported sampling errors. It is important to note that the larger the sample size, the greater the proportion of total error likely to come from nonresponse bias, other things being equal. (Sampling errors can be driven down to a trivial amount, but nonresponse biases may remain the same.)

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues 3. Assume nonresponse will occur; prepare for it. In practice no sample survey avoids nonresponse completely. Assuming at the design stage that it will not occur leaves the researcher unprepared to deal with it at the estimation stage. Whenever possible use interviewers to collect information that can be used either to reduce nonresponse (e.g., utterances of the sample person suggesting reasons for nonresponse, useful later in tailoring refusal conversion protocol) or to adjust for nonresponse (e.g., observations about respondents and nonrespondents related to propensities to respond). 4. Consider relationships with the sponsoring agency as sources of nonresponse error. Sample persons with prior experiences or relationships with the sponsoring agency for the survey make decisions based partially on how they evaluate those relationships. This may underlie the tendency for those persons dependent on programs to respond at higher levels. It also underlies the findings of those with relatively low trust in government to respond at lower rates to some government surveys. Mixed-mode designs and alternative sponsoring organizations may act to reduce these sources of differential nonresponse. 5. Do not script interviewers; use flexible interviewer behaviors. The research literature is increasingly strong on the conclusion that effective interviewers need to be trained to deliver information relevant to a wide variety of concerns that different sample persons may have. Stock phrases and fixed approaches defeat the need to address these diverse concerns. Once interviewers can classify the sample person’s utterances into a class of concerns, identify a relevant piece of information to convey to the person, and deliver it in the native language of the sample person, cooperation rates can be higher. 6. Consider incentives, especially for the reluctant. Incentives have been shown to have disproportionately large effects on those who have no other positive influence to respond. Although not completely clear from the literature, the value of a given incentive may be dependent on relative income/assets of the sample person. If greater effects pertain to low-income populations, then incentives might be more attractive to studies of that population.

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues 7. Give separate attention to location, noncontact, refusal; each has different causes and impacts on error. Sample persons not interviewed because of failure to locate are disproportionately movers. All the correlates of residential mobility (rental status, small households, relative youth, few extended family ties), if relevant to the survey measures, make nonlocation nonresponse a source of error. Noncontacts and refusals may have very different patterns of correlates. Treating nonresponse rates as an undifferentiated source of nonresponse error is thus naive. Separate tracking of these nonresponse rates is needed. 8. Mount special studies of nonrespondents. The higher the nonresponse rate, the higher the risk of nonresponse error, other things being equal. With higher than desired nonresponse rates, the investigators have an obligation to assure themselves that major nonresponse errors are not present, damaging their ability to draw conclusions from the respondent-based statistics. Special studies of nonrespondents are appropriate in these cases, using auxiliary data from records, followback attempts at samples of respondents, and other strategies. 9. Perform sensitivity analyses on alternative postsurvey adjustments. Postsurvey adjustments (weighting and imputation) entail explicit or implicit assumptions about the relationships between propensity to respond to the survey and survey variables. Insight is sometimes gained into the dependence on nonresponse adjustments of substantive conclusions by varying the assumptions, using different postsurvey adjustments, and comparing their impact on conclusions. 10. Involve the target population. Using focus groups and other intensive qualitative investigations can offer insights into how the target population might receive the survey request. Such insights are rarely native to research investigators who are members of different subcultures. REFERENCES Abelson, R.P. 1981 Psychological status of the script concept. American Psychologist 36(7):715–729. Berk, M.L., N.A.Mathiowetz, E.P.Ward, and A.A.White 1987 The effect of prepaid and promised incentives: Results of a controlled experiment. Journal of Official Statistics 3(4):449–457.

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues Brehm, J. 1993 The Phantom Respondents: Opinion Surveys and Political Representation. Ann Arbor: University of Michigan Press. Cialdini, R.B. 1984 Influence: The New Psychology of Modern Persuasion. New York: Quill. 1988 Influence: Science and Practice. Glenview, IL: Scott, Foresman and Company. Cochran, W.G. 1977 Sampling Techniques. Third Edition, New York: Wiley and Sons. Couper, M.P. 1997 Survey introductions and data quality. Public Opinion Quarterly 61(2):317–338. Couper, M.P., and R.M.Groves 1995 Introductions in Telephone Surveys and Nonresponse. Unpublished paper presented at the Workshop on Interaction in the Standardized Survey Interview, Amsterdam. De Heer, W.F., and A.Z.Israëls 1992 Response Trends in Europe. Unpublished paper presented at the Joint Statistical Meetings of the American Statistical Association, Boston. Deming, W.E. 1953 On a probability mechanism to attain an economic balance between the resultant error of response and the bias of nonresponse. Journal of the American Statistical Association 48(264):743–772. Eagly, A.H., and S.Chaiken 1984 Cognitive theories of persuasion. Pp. 267–359 in Advances in Experimental Social Psychology Volume 17, L.Berkowitz, ed. San Diego: Academic Press. Ekholm, A., and S.Laaksonen 1991 Weighting via response modeling in the Finnish household budget survey. Journal of Official Statistics 7(3):325–337. Ford, Barry 1983 An overview of hot-deck procedures. Pp. 185–209 in Incomplete Data in Sample Surveys, W.Madow, I.Olkin, and D.Rubin, eds. New York: Academic Press. Freedman, J.L., and S.C.Fraser 1966 Compliance without pressure: The foot-in-the-door technique. Journal of Personality and Social Psychology 4:195–202. Groves, R.M., R.B.Cialdini, and M.P.Couper 1992 Understanding the decision to participate in a survey. Public Opinion Quarterly 56(4):475– 495. Groves, R.M., and M.P.Couper 1998 Nonresponse in Household Interview Surveys. New York: Wiley and Sons. Groves, R.M., and R.L.Kahn. 1979 Surveys by Telephone: A National Comparison with Personal Interviews. New York: Academic Press. Groves, R.M., D.Wissoker, and E.Singer 1999 Early Nonresponse Studies of the 1997 National Survey of America’s Families. Washington, DC: Urban Institute. Hansen, M.H., and V.M.Hurwitz 1958 The problem of nonresponse in sample surveys. Journal of the American Statistical Association (December) 517–529. Hill, D.H. 1978 Home production and the residential electric load curve. Resources and Energy 1:339– 358.

OCR for page 11
Studies of Welfare Populations: Data Collection and Research Issues House, J.S., and S.Wolf 1978 Effects of urban residence on interpersonal trust and helping behavior. Journal of Personality and Social Psychology 36(9), 1029–1043. Keeter, S., C.Miller, A.Kohut, R.Groves, and S.Presser 2000 Consequences of reducing nonresponse in a national telephone survey. Public Opinion Quarterly, 64, 125–148. Little, R., and D.B.Rubin 1987 Statistical Analysis with Missing Data. New York: Wiley and Sons. Oksenberg, L., L.Coleman, and C.F.Cannell 1986 Interviewers’ voices and refusal rates in telephone surveys. Public Opinion Quarterly 50(1):97–111. Petty, R.E., and J.T.Caccioppo 1986 Communication and Persuasion: Central and Peripheral Routes to Attitude Change. New York: Springer-Verlag. Purdon, S., P.Campanelli, and P.Sturgis 1999 Interviewers’ calling strategies on face-to-face interview surveys. Journal of Official Statistics 15(2):199–216. Rubin, D.B. 1987 Multiple Imputation for Nonresponse in Surveys. New York: Wiley and Sons. Singer, E., and L.Kohnke-Aguirre 1979 Interviewer expectation effects: A replication and extension. Public Opinion Quarterly 43(2):245–260. Singer, E., N.Gebler, T.Raghunathan, J.Van Hoewyk, and K.McGonagle 1996 The Effects of Incentives on Response Rates in Personal, Telephone, and Mixed-Mode Surveys: Results from a Meta Analysis. Unpublished paper presented at the Annual Conference of the American Association of Public Opinion Research, Salt Lake City. Tuckel, P., and B.M.Feinberg 1991 The answering machine poses many questions for telephone survey researchers. Public Opinion Quarterly 55(2):200–217. Tuckel, P., and H.O’Neil 1995 A profile of answering machine owners and screeners. Pp. 1157–1162. In Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association.