Click for next page ( 24


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 23
3 Sampling Frames Graham Kalton, moderator for the session, described the presentations as a discussion of the potential uses of sampling frames to aid in particular surveys and the multiple sources for these frames. Given the costs associated with frame development, some of the questions to consider are whether there are any economies that can be achieved with the current sampling frames and what are the difficulties related to implementing them. USING LARGE SURVEYS TO ASSIST IN FRAME DEVELOPMENT FOR SMALLER SURVEYS James Lepkowski (University of Michigan) began his talk on using large surveys as frames for smaller surveys with examples of cases in which this is currently being done and a discussion of the issues associated with these approaches. The first example described the Current Population Survey (CPS) and the American Time Use Survey (ATUS). The CPS is a well-established, rotating panel, continuous survey of the noninstitutionalized population in the United States ages 15 and older. A joint effort of the Census Bureau and the Bureau of Labor Statistics, the CPS is the primary source of information about characteristics of the U.S. labor force. It uses independent samples in each state and the District of Columbia and oversamples the Hispanic population. Since the 1940s, it has used probability sampling and has produced national as well as state-level estimates. The ATUS uses a sample of households from a CPS panel that is rotating out of the survey. There are three stages of the ATUS sample design. From 23

OCR for page 23
24 THE FUTURE OF FEDERAL HOUSEHOLD SURVEYS the sample of households (in the third and final stage of the sample design), one person age 15 or older is randomly selected for interview by telephone and becomes the ATUS “designated person.” Nontelephone households are contacted by mail, given a phone number, and requested to call in, with a $40 incentive that is awarded at the completion of the survey. Lepkowski said that one of the major challenges in using the CPS as a frame for the ATUS is timing. Although most of the CPS sample becomes avail- able to the ATUS within three months, the sample is still spread out over time due to the interviewing and processing schedule. Similar challenges related to timing have led some survey organizations to abandon second-phase samples. Another challenge in the context of the CPS and the ATUS is that the CPS is a household survey, which must then be transformed into a person- level sample for the ATUS. Finally, it is possible that ATUS response rates are adversely affected by previous participation in several prior CPS interviews, but it is difficult to determine conclusively the potential magnitude of this effect. Overall, the telephone response rates are in the mid-50 percent range. The second example Lepkowski described is the case of the National Health Interview Survey (NHIS) and the Medical Expenditure Panel Survey (MEPS). The NHIS is the primary source of data about the U.S. household population’s health and health care utilization. The survey is conducted by the Census Bureau and sponsored by the National Center for Health Statistics (NCHS), although other agencies also fund supplements, a situation that can be an important factor that influences an organization’s ability to share sample efficiently. The NHIS is a continuous, multistage, national probability survey with oversamples of black, Hispanic, and Asian populations. Response rates vary depending on the type of interview, generally ranging between 65 and 80 percent. The MEPS, sponsored by the Agency for Healthcare Research and Qual - ity (AHRQ), uses completed NHIS interviews as a sampling frame for the household component of the survey (there is also a medical provider com - ponent and an insurance component). The goal of the survey is to produce national and regional estimates of health care utilization and expenditures. Approximately 15,000 households are included annually, with occasional oversamples for additional policy-relevant subgroups. The MEPS also utilizes the oversampling performed for the NHIS. Rather than a cross-sectional design like the NHIS, the MEPS uses a panel design. The MEPS response rates are also affected by the response rates to the NHIS. Response rates for recent NHIS surveys have typically been in the upper 80s, and the MEPS nonresponse rate is compounded by the nonresponse in the first phase. In addition, the NHIS sample sizes can vary from year to year, changing the proportion of the sample the MEPS takes from the NHIS to meet its own sample size designations. One of the main advantages of using one survey as the sampling frame

OCR for page 23
25 SAMPLING FRAMES for another is the cost efficiency that can be achieved by the second survey. The cost savings can be realized in the form of efficiencies in sample design, data collection, screening, and data processing. For example, the ATUS has a list of items that are nearly identical to those in the CPS, and going through the same processing system saves the cost of system development. Although typically the efficiencies benefit the second survey, Lepkowski observed that when the sample sharing is a long-term arrangement, there has to be some sharing of the cost burden as well. He pointed it out that there are several challenges related to these designs as well. Nonresponse rates can be affected not only by the fact that respondents’ willingness to participate sometimes declines by the time of the second-phase survey, but also because of increased difficulties related to locating sample persons by the time of the follow-up. Although drawing a sample based on another survey also presents a unique opportunity to estimate nonresponse bias based on responses to the first survey, this is often leveraged to some extent, but perhaps not as much as it could be. A related concern is the measurement bias that can potentially be introduced into the second-phase survey as a result of participation in previous surveys, even if respondents are willing to participate (also known as time-in-panel bias). The quality of any stratification performed for the second-phase survey depends on the quality of the data collected in the first survey. For example, if the second-phase survey is stratified on income and this information is mis - reported in the first survey, the misclassification will lead to inefficiencies in selection. Capacity issues are often another consideration. The first survey has to provide adequate sample to meet the needs of the second-phase survey. Some of this is driven by disproportionate allocation in the second phase, which may use up a large proportion of a particular subgroup, which can also preclude the first-phase sample’s use by other surveys. Small-area estimation is another hurdle for second-phase samples. All of these factors lead to a set of administrative challenges that have been briefly mentioned in the context of the examples provided but are worth acknowledging more generally, Lepkowski said. One such challenge involves funding, particularly deciding on how the second-phase survey can share some of the costs of the first-phase survey (e.g., the costs related to screening or list - ing). Another challenge is related to the changes in sample size and the logistics associated with adapting to these changes. Second-phase surveys tend to be administered after the first survey, although concurrent designs are also pos - sible, and these represent a separate set of administrative challenges. The use of some sample frames, such as the Master Address File (MAF), has limitations that impose restrictions on second-phase survey operations. Something that is not typical of currently existing two-phase surveys is a conscious effort to design them as true two-phase surveys from the outset.

OCR for page 23
26 THE FUTURE OF FEDERAL HOUSEHOLD SURVEYS Instead, second-phase surveys seem to occur on the basis of arising opportuni - ties for collaboration between agencies and an after-the-fact recognition that there is a possibility to save on costs across two or more organizations. A joint design from the outset would allow for optimal allocation across phases and better input into units of selection. Two-phase surveys could also be more successful at reducing nonresponse if the goals and designs of both surveys were kept in mind. This would allow for the planning of a more comprehensive incentive structure, as well as tracking and follow-up procedures. There is also tremendous opportunity to use paradata and a responsive design for utilizing first-phase data to predict what will happen in the second phase. Prediction models compared with what actually transpired in the second phase can then be used, improving the ability to intervene and improve response properties. THE POTENTIAL ROLE OF THE AMERICAN COMMUNITY SURVEY IN SAMPLING RARE POPULATIONS Keith Rust (Westat) began by saying that he added the word “potential” to the title of his presentation to illustrate that some of the ideas presented are in development or are under consideration, rather than already in progress. He then proceeded with an overview of the American Community Survey (ACS). Conducted by the Census Bureau, the ACS surveys approximately 250,000 households each month by mail, or 3 million households per year. The ques- tionnaire contains 48 questions about each individual in the household and 21 questions on housing. Nonrespondents to the mailed questionnaire receive a telephone follow-up whenever possible (when a phone number is available). The remaining nonrespondents for whom there is no phone number or who did not respond by phone are eligible to be in the sample for follow-up by an in-person interview using computer-assisted personal interviewing (CAPI) technology. The in-person follow-up obtains interviews from about one-third of the 48 percent of nonrespondents who do not respond by mail or telephone. But the CAPI subsample rate does vary by population group. The overall weighted response rate to the ACS is very high at 97-98 per- cent, but due to CAPI subsampling for follow-up, the data actually obtained are about two-thirds of the original sample. Therefore, data are obtained for approximately 2 million households per year. Differential sampling also affects the total final count of respondents. The sampling for the ACS is complex, but, as an example, there is an initial oversample of small governmental units. This works out to about 15 percent of the sample, which covers 5 percent of the population in these units. Also, since nonresponse CAPI subsampling yields about one-quarter of the sample that is obtained through CAPI, these interviews get three times the weight of the remainder. This suggests that the effective sample size due to the differential weighting is closer to 1.5 million

OCR for page 23
27 SAMPLING FRAMES household interviews per year, although the design effects due to weighting could vary among subgroups. As with any survey collected by mail, there is item nonresponse. There are a lot of questions on the ACS, and some of them are open-ended responses that must be coded (e.g., industry, occupation, field of degree). There is also the issue of response error, particularly when it comes to reporting income. Some questions involve a challenging recall task, such as the question about employment. Each of these factors can contribute to item nonresponse and response error. It is in this context that the use of the ACS as a frame for sampling rare populations should be considered, Rust said. Issues to keep in mind with sam- pling rare populations are cost and burden of sampling, timeliness of the data available, the sample size available, the amount of cumulation that is needed (from the ACS), the effects of differential weighting, coverage issues, response error, the quality of the contact information, sampling error estimation, and confidentiality and human subjects concerns. One of the most obvious benefits to using the ACS as a frame for other surveys is the reduction in the cost and burden associated with smaller surveys. Cost is reduced for the smaller survey by not having to screen a large initial sample in order to identify a subpopulation of interest. Respondent burden is reduced by not having to participate in a screening survey. Furthermore, there is the ability to fine-tune sample allocation for different population subgroups. Sample size can also be controlled precisely because the sampling done is from a frame of people known to be in the population of interest. Finally, it is possible to orchestrate the release of sample in waves or replicates in order to fine-tune yield. As Lepkowski mentioned in the previous presentation, the timeliness of data available for use as another survey’s sampling frame is also a consideration, Rust said. In this case, what proportion of people will have a status change that might cause them to move into or out of the population of interest? As an extreme example, the ACS would be of no use as a frame in the case of new - borns, very recent immigrants, or the recently unemployed. Another question is what constitutes a sufficient sample for the rarest group of interest. If cumula - tion of data over many months or years is required, then issues of timeliness are exacerbated. Furthermore, the differential representation in the ACS sample may lead to large weighting design effects in a rare population, although some of this may be offset with subsampling—if there is enough sample to do this. Like most surveys, the ACS probably undercovers certain groups (potentially the groups of interest) in the population. Data from the census undercover new- borns; it is likely that the ACS does as well. Household surveys tend to under- cover young adult black men, so it seems likely that the ACS would, too. The ACS weighting adjustments can help address undercoverage for estimates, but it is unknown how useful this will be for the subsampled rare population group.

OCR for page 23
28 THE FUTURE OF FEDERAL HOUSEHOLD SURVEYS Misclassification as it relates to rare population status can result in substan- tial undercoverage and wasted sample, Rust went on. Any survey of a rare pop- ulation that uses screener identification will have this problem. Furthermore, in the case of the ACS, which is largely a mail survey, there is no interviewer who can follow up with probes to ascertain that a respondent is answering a particular question correctly. The quality of the contact information that is available on the ACS is another issue to consider, Rust observed. Is the address information on the ACS accurate enough for follow-up by mail, telephone, or in-person contact? The ACS does not ask for address corrections or clarifications on its form. This could be a potentially significant issue, particularly for multiunit structures, he said. If the contact information is sufficient for a subsample, there is the related issue of confidentiality and human subjects protection issues. The ACS response is required by law; respondents are told that their responses are confi - dential and will be used for statistical purposes only. Title 13 of the U.S. Code, which authorizes collection of personally identifiable information, requires that follow-up surveys must be conducted by the Census Bureau because the infor- mation collected in the ACS is confidential. Thus, access to this information cannot be shared outside the agency. The ACS sample is a rolling sample, with a new sample produced every month. Could this be utilized to design rolling samples for rare populations? It may be possible to draw sample from the ACS every quarter, but, for reporting subgroups, data can be cumulated across quarters to get a continuous rolling sample. This could be used to measure trends, Rust said. Another question that arises is whether the ACS in its own right is suf - ficient to identify a rare population of interest. This suggests the possibility of adding questions to the ACS to be used as a screener for identifying a rare population. This leads further to what kind and how many questions can be asked, as well as who is responsible for the quality of the data from these ques - tions. He said it is important to distinguish screener questions from those that will be tabulated along with other ACS data. How will the effect of adding questions to the ACS on response rates be evaluated? He observed that this may not be the right time to add questions, given suggestions that the ACS should be cancelled altogether, or at least made voluntary, because of claims that the survey is too intrusive. Rust noted that a couple of examples can be used as case studies of smaller surveys using the ACS for sample creation. One is the National Science Foundation’s National Survey of College Graduates (NSCG). This survey, conducted by the Census Bureau in the past, measures the number and characteristics of people with science and engineering degrees. Formerly the frame for the NSCG was the census long-form sample. Since the long- form sample no longer exists, the ACS will be used as a frame instead. A “field-of-degree” question was added to the ACS specifically for that purpose

OCR for page 23
29 SAMPLING FRAMES (although it is also of interest in its own right). The benefit of adding the ques - tion is that it permits oversampling of people with science and engineering degrees. However, several years of ACS data will be required to achieve what has previously been the desired sample for some of the groups. Still, this is a vital question for targeting the sample of persons with science and engineer- ing degrees, and getting that information from the ACS greatly decreases screening costs. The field-of-degree question does have its problems, he said; it is an open-ended question and therefore requires extensive coding. And in 2009 there was 9 percent item nonresponse. There are most likely issues of data quality and coverage. And this also raises the question of whether the NSCG could benefit from using a rolling sample, at least for a component. The second case study describes a test of the feasibility of using the ACS for the National Immunization Survey (NIS). The NIS produces annual vacci - nation rates for children ages 19 to 35 months, plus a component for teenagers ages 13-17 years. It produces data at multiple levels of geography, including 78 areas known as Immunization Action Plan Areas. The NIS currently uses a list-assisted random digit dialing (RDD) sample—a methodology with high screening costs, because only 5 percent of households have infants. And the sample size is quite large: 26,000 infants per year and 31,000 teens. Rust observed that this survey, like others, experiences many of the prob- lems associated with telephone surveys: low response rates and undercoverage, to name just two. To help combat these problems, the proposal was to consider using the ACS as a frame for the NIS. The ACS certainly offers the possibility to overcome many of the current deficiencies in the NIS sample, and the idea of a rolling sample would integrate naturally into the NIS design. There are also rich data on respondents that could be used for adjustment and bias analyses. The ACS probably undercovers persons under 1 year of age, so there are probably coverage problems. The immunization surveys are interested in children ages 19 months and older, but because of the time lag, those under 1 year of age would need to be selected from the ACS. Moreover, the NIS would need to be in the field within 19 months of the ACS response to cover 19-month-olds. The Census Bureau and the Centers for Disease Control and Prevention jointly conducted a one-state trial with children ages 19-35 months using ACS data for the period 2006-2008. They found that although the response rate was good, in-person interviewing was vital. A provider check was included in the survey, in which respondents gave contact information for those who pro- vided the immunization. Generally, respondents gave good information about the provider, but confidentiality issues were raised related to the fact that the respondents were identified on the basis of the ACS. As a work-around, Rust said, providers were given special sworn status by the Census Bureau. Although this appeared to work for the trial, it may be an issue for surveys that want to use the ACS as a frame. The ACS has the potential to greatly reduce screening costs and reduce

OCR for page 23
30 THE FUTURE OF FEDERAL HOUSEHOLD SURVEYS undercoverage for surveys of special populations. It is also rich in respondent information, which is quite useful for enhancing estimation. These benefits may be overstated, however, as the evidence suggests that a significant amount of in-person interviewing will be needed. Other issues include timing, cover- age, data quality, sample size, and differential weighting. Using the ACS data as the basis for a person’s inclusion in a future survey raises issues of consent and confidentiality. Issues related to including additional questions on the ACS (such as how many additional questions and who decides on the questions) are other hurdles that any survey using the ACS as a frame must overcome. Given current Title 13 restrictions, the Census Bureau must conduct the survey and maintain the data. Rust ended by saying that, although the ACS appears to offer opportunities for use as a sampling frame for other surveys, it is not a panacea, and there are real risks of abuse. SAMPLING FRAMES FOR FEDERAL HOUSEHOLD SURVEYS: A VISION FOR THE FUTURE Frederick Scheuren (National Opinion Research Center) began his talk on sampling frames by saying that the goal of the workshop should be to identify ways of supporting an information society, not just individual agency missions. The focus should be on multimode and multiagency sampling frames. He noted that even the concept of “household” survey frames is too narrow and unable to describe many new developments, such as the spread of cell phones. Scheuren said that government agencies do not typically cooperate well, except in times of crises. But there are some common challenges across the federal statistical system that need to be addressed: survey costs are too high and the delivery of information is too slow. Referring back to the presentation by Rust, he gave the example of outdated sampling frames that can be so old by the time they are shared with another agency that they are no longer useful. Scheuren argued that government data collections will become increasingly difficult to conduct in a data-rich world, with information becoming available from many competing data sources. This means that government agencies will have to learn to adapt or they will risk irrelevance. Some of the examples from other countries discussed earlier are cheaper, faster, and more responsive, and they should not be ignored, he said, even if the same approaches cannot be implemented in the United States. A possible new paradigm in a data-rich world could be characterized by emphasis on paradata, both design and model-based estimation, and quick, simulated outcomes, instead of traditional estimators. Statistical systems can no longer afford large samples, so small sample properties also have to be stressed in this context. Unified sampling frames are an important consideration for the future. These could be assembled starting with geographic addresses, which would then be

OCR for page 23
31 SAMPLING FRAMES linked to sampling frames based on other modes of contact, such as telephone (both landline and cell phone) and web. An ideal unified sampling frame would also integrate information available from such sources as the census, the Ameri- can Community Survey, the Internal Revenue Service, the Social Security Admin- istration, and the U.S. Postal Service. The key benefit to adding the administrative data is that they enable stratification of the sample on variables of interest. Scheuren argued that cooperation related to work on sampling frames is important, and that in his view all federal survey contracts should require data sharing for frame construction. He acknowledged that there are many barriers to implementing this type of mandatory data sharing, including legislative and regulatory restrictions, the lack of political and bureaucratic will, the fact that it violates precedents, and the need for a long-term commitment. Confidential - ity concerns could be addressed if access is through a data enclave—in other words, a secure environment that provides authorized researchers access to confidential microdata. Access would have to be limited in purpose to frame construction only and subject to oversight by a neutral entity, such as the Office of Management and Budget. Given the large opportunity costs, it is important to consider whether implementing a unified sampling frame is worth it, said Scheuren. Such a system could improve data quality and enable faster delivery times. However, maintaining the frame will be expensive, and depositing the data in the enclave will take time, which means that at least some of the information will be out - dated. He noted that although an obvious benefit of the work on a unified sampling frame would be the development of a cooperative structure in the federal statistical system, a large investment in sampling frames could turn out not to be the right long-term investment. Scheuren reiterated that the world is becoming more data dense, and gov - ernment statistical agencies now have strong competitors. This could mean that, in the future, surveys and censuses will have a smaller role, and emphasis will shift to increasing reliance on administrative data and to combining informa - tion, which is more than just combining data. The federal statistical system will be ready for this new reality if agencies invest in becoming more cooperative. DISCUSSION Frauke Kreuter (University of Maryland) asked whether Rust had a sense of what the cost savings would be for the NIS if it were to use the ACS as a sam - pling frame. Rust responded that although he did not know, he did not think cost savings were a particular consideration; this initiative was probably driven by quality concerns and dissatisfaction with how the survey is conducted now. Marcie Cynamon (National Center for Health Statistics) added that there were concerns about the NIS coverage related to the percentage of the children who were in the age range through the RDD and cell phone components.

OCR for page 23
32 THE FUTURE OF FEDERAL HOUSEHOLD SURVEYS Related to the ACS as a frame for the NSCG, Robert Kominski suggested that the argument for including a screener question on the ACS for the NSCG was the cost savings of $3 to $6 million. There was a good chance that, in many cases, the addresses in the NSCG frame were no longer applicable for persons who held at least a bachelor’s degree. However, even with the delay in ACS processing, the data would still be more current than alternatives. In addition, getting data more frequently makes the sample to draw from much larger, even though the sampling rate is much smaller. Cynamon noted that the National Health Interview Survey gives half of its sample to the MEPS, and it is not a cherry-picked half. Trena Ezzati-Rice (Agency for Healthcare Research and Quality) discussed the screening efficiencies that are gained from the integration of the NHIS and the MEPS. The integration has been extremely helpful in the benchmarking of the survey estimates, she said, and this has been useful for both surveys. The end result of the NHIS-MEPS integra- tion is a very rich frame of auxiliary variables that can be used for nonresponse adjustments. Further research has found that incorporation of some of the health variables from the NHIS reduces bias in the MEPS estimates. Lepkowski added that, if there is this rich set of data and methods for doing model-based estimates on something like the MEPS and the extension of the frame to the other half of the NHIS sample, then why are the only esti - mates produced based on the MEPS sample? The estimates could be of higher precision, but currently these resources are not being utilized, which is part of the failure of two-phase designs. Sondik added that, beyond the use of a survey as a sampling frame for another survey, the potential is also there for using substantive data from the first survey, although implementing this link would require substantial resources. Ezzati-Rice commented that the integration of the NHIS and the MEPS has provided an additional data point—beyond the two years of the MEPS data—for longitudinal analysis. There has not been as much mining of these data as there perhaps could be, but they have been looked at, specifically the transitions in health insurance coverage from the NHIS to the two panels of the MEPS. There is also an interagency agreement to look at cancer survivors from the NHIS supplement and then to look at issues related to health care access and costs for these respondents in the MEPS. On recycling sample, Kominski said that currently an ACS case can be used in only one secondary survey based on the primary survey, but this is not necessarily the case in the commercial sector, and other scenarios could be considered. A participant from the National Agricultural Statistics Service added that they have this problem with the Agricultural Resource Management Survey: first, there is a screening survey, and then another survey uses up the data, so they cannot be used for the next survey. A used case is taken from the frame, and the weight of everything else is increased proportionally. The ATUS adjusts its weights similarly.

OCR for page 23
33 SAMPLING FRAMES Another speaker commented on the importance of developing a process of selection for who gets to use what portion of a sample. What are the terms of the agreement? And how does this get resolved? Citro said that the ACS has the benefit of containing questions that can be used for screening and weighting, but the available sample is limited. The MAF does not contain survey responses, but the efficiencies associated with sharing it would be a major step forward for the statistical system. She added that although the sharing of the MAF is limited by Title 13 of the U.S. Code, it is conceivable to think of enabling the sharing of some version of the MAF. For example, one of the concerns is that, as part of its work, the Census Bureau discovers sensitive information, such as a building that contains more housing units than it is supposed to. However, this type of data could be collapsed across cases. Under the Confidential Information Protection and Statistical Efficiency Act (CIPSEA), a “slightly sanitized” version of the MAF could be entrusted to the statistical system as a whole, and federal agencies could col - laborate, perhaps in the context of a data enclave, on building, validating, and using the MAF for sampling frames. As part of the data enclave, access could potentially be also extended to data collection contractors who work on federal surveys. Kalton commented that the opportunity costs associated with Scheuren’s vision are high, and the approach described by Citro is more manageable. It is not only the statistical system that could benefit from access to the MAF, but the MAF would also benefit from more agencies working on improving its quality. Sondik asked Scheuren about his vision for the federal statistical system and the idea of increasingly fuzzier data and methods: How will it be decided what constitutes a benchmark? Scheuren responded that it is critical to take advantage of opportunities that are already available: administrative records, business frames, modern methods, sharing mechanisms, and the knowledge of those who move around and have worked in other countries—especially smaller ones forced to use more economical means to obtain data. Finally, he said, the cooperation and common culture that existed in the Census Bureau and unified the system in the 1940s and 1950s could be resurrected.

OCR for page 23