Reengineering the Survey of Income and Program Participation (2009)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Summary T he Survey of Income and Program Participation (SIPP) began 25 years ago to fill gaps in the available information on the short- term ­dynamics of income, household composition, employment, and eligibility for and participation in government assistance programs experi- enced by families in America. At present, SIPP follows samples of household members (panels) for 3-4 years, interviewing them every 4 months in order to obtain as accurate monthly information as possible and starting up a new panel when a previous panel ends. Beginning in 2006, the Census Bureau embarked on a program to reengineer SIPP to reduce its costs and improve data quality and timeliness to the extent possible by such means as making greater use of administra- tive records, moving to annual interviews in which event history calendars would be used to ascertain monthly information, and modernizing the SIPP data collection and processing systems. The bureau also requested the Committee on National Statistics of the National Academies to establish a study panel to address specific aspects of the reengineering program. The panel was asked to consider the advantages and disadvantages of strategies for linking administrative records and survey data, taking account of the accessibility of relevant administrative records, the operational feasibility of linking, the quality and usefulness of the linked data, and the ability to provide access to the linked data while protecting the confidentiality of   Event history calendars are customized calendars that show the reference period, such as a year, and contain timelines for different domains, such as residence history, household compo- sition history, work history, and other areas, that might aid a respondent’s memory.  

 REENGINEERING THE SURVEY individual respondents. The panel also was charged to consider alternative uses of administrative records for a reengineered SIPP that do not require actual data linking (for example, to evaluate SIPP data quality). In addi- tion, the panel could consider aspects of the reengineered SIPP survey with regard to interview periodicity, mode of data collection, and sample source and size. The panel addressed the charge by first examining the history of SIPP to inform its deliberations about the survey’s purpose, value, strengths, and weaknesses (Chapter 2). We then reviewed alternative uses of administra- tive records in a reengineered SIPP (Chapter 3) and, finally, considered innovations in SIPP design and data collection, including the proposed use of annual interviews with an event history calendar (Chapter 4). The panel’s conclusions and recommendations from each chapter follow. HISTORICAL PERSPECTIVE Conclusion 2-1: The Survey of Income and Program Participation is a unique source of information for a representative sample of household members on the intrayear dynamics of income, employment, and pro- gram eligibility and participation, together with related demographic and socioeconomic characteristics. This information remains as vital today for evaluating and improving government programs addressed to social and economic needs of the U.S. population as it did when the survey began 25 years ago. Conclusion 2-2: The Survey of Income and Program Participation’s (SIPP) history of forward movement followed by setbacks has contrib- uted to the survey’s falling short of its original promise with regard to t ­ imeliness, usability, and maintenance of data quality. With the Census Bureau’s planned SIPP reengineering program, there is an opportunity to put the survey on a much firmer foundation for the future. It is essential that the Census Bureau’s program to reengineer SIPP address its problems and retain and build on its unique value and strengths. No survey can be all things to all users. In reengineering SIPP, the focus should be on improving the content and design features of the survey that make possible its unique contribution. Recommendation 2-1: To guide the design of a reengineered Survey of Income and Program Participation, the Census Bureau should consider the primary goal of the survey to be to provide data for policy analysis and research on the short-run (intrayear) dynamics of economic well-being for families and households, including employment, earnings, other income, and program eligibility and participation. 

SUMMARY  Recommendation 2-2: The Census Bureau’s reengineering program for the Survey of Income and Program Participation should explicitly evaluate each proposed innovative feature, such as the use of administrative records or an event history calendar, on the extent to which a feature contributes to the survey’s ability to measure short-term changes in economic well-being with improved quality and timeliness. The Role of Administrative Records in a Reengineered SIPP Conclusion 3-1: In reengineering the Survey of Income and Program Participation (SIPP) to provide policy-relevant information on the short- run dynamics of economic well-being for families and households, the Census Bureau must continue to use survey interviews as the primary data collection vehicle. Administrative records from federal and state agencies cannot replace SIPP, primarily because they do not provide information on people who are eligible for—but do not participate in—government assistance programs and, more generally, because they do not provide all of the detail that is needed for SIPP to serve its primary goal. Many records are also difficult to acquire and use because of legal restrictions on data sharing, and some of the information they contain may be erroneous. Nonetheless, information from administrative records that is relevant to SIPP and likely to improve the quality of SIPP reports of program par- ticipation and income receipt in particular can and should be used in a reengineered SIPP. Conclusion 3-2: The Census Bureau has made excellent progress with the Statistical Administrative Records System and related systems, such as the person validation system, in building the infrastructure to support widespread use of administrative records in its household survey programs. The bureau’s administrative records program, both now and in the future as it adds new sets of records and analysis capabilities, will be an important resource for applications of administrative records in a reengineered Survey of Income and Program Participation. Acquisition of Records Conclusion 3-3: Many relevant federal administrative records are read- ily available to the Census Bureau for use in a reengineered Survey of Income and Program Participation (SIPP). However, most state administra- tive data are not available for use in a reengineered SIPP at this time and could be difficult to obtain. 

 REENGINEERING THE SURVEY Recommendation 3-1: The Census Bureau should seek to acquire addi- tional federal records that are relevant to the Survey of Income and Pro- gram Participation, which could include records from the U.S. Department of Veterans Affairs and the Office of Child Support Enforcement. Recommendation 3-2: The Census Bureau, in close consultation with data users, should develop a strategy for acquiring selected state adminis- trative records, recognizing that it will be costly and probably unfeasible to acquire all relevant records from all or even most states. The bureau’s acquisition strategy should be guided by such criteria as the importance of the income source for lower income households, particularly in times of economic distress, and the relative ease of acquiring the records. Unemploy­ ment insurance benefit records should be a high priority for the Census Bureau to acquire on both of these counts, and the bureau should inves- tigate whether it is possible to acquire these records from the National Directory of New Hires, which would eliminate the need to negotiate with individual states. Indirect Uses of Records Conclusion 3-4: Indirect uses of administrative records are those uses, such as evaluation of data quality and improvement of imputation models for missing data, in which the administrative data are never recorded on survey records. They are advantageous for a reengineered Survey of Income and Program Participation (SIPP) in that they should have little or no adverse effects on timeliness or the needed level of confidentiality protection of SIPP data products. Recommendation 3-3: The Census Bureau, in close cooperation with knowledgeable staff from program agencies, should conduct regular, fre- quent assessments of Survey of Income and Program Participation (SIPP) data quality by comparison with aggregate counts of recipients and income and benefit amounts from appropriate administrative records. When fea- sible, the bureau should also evaluate reporting errors for income sources— both underreporting and overreporting—by exact-match studies that link SIPP records with the corresponding administrative records. The Census Bureau should use the results of aggregate and individual-level comparisons to identify priority areas for improving SIPP data quality. Recommendation 3-4: The Census Bureau should move to replace hot-deck imputation routines for missing data in the Survey of Income and Program Participation with modern model-based imputations, implemented multiple times to permit estimating the variability due to imputation. Impu- 

SUMMARY  tation models for income and program participation should make use of program eligibility criteria and characteristics of beneficiaries from admin- istrative records so that the imputed values reflect as closely as possible what is known about the beneficiary population. Before implementation, new imputation models should be evaluated to establish their superiority to the imputation routines they are to replace. Recommendation 3-5: The Census Bureau should request the ­Statistical and Science Policy Office in the U.S. Office of Management and ­Budget to establish an interagency working group on uses of administrative records in the Survey of Income and Program Participation (SIPP). The group would include technical staff from relevant agencies who have deep knowledge of assistance programs and income sources along with Census Bureau SIPP staff. The group would facilitate regular comparisons of SIPP data with administrative records counts of income recipients and amounts (see R ­ ecommendation 3-3) and advise the Census Bureau on priorities for acquiring additional federal and selected state administrative records, how best to ­tailor imputation models for different sources of income and pro- gram benefits, and other matters related to the most effective ways to use administrative records in SIPP. The Census Bureau should regularly report on its progress in implementing priority actions identified by the group. Direct Uses of Records Conclusion 3-5: Direct uses of administrative records in a reengineered Survey of Income and Program Participation (SIPP), which include substi- tuting administrative values for missing survey responses, adjusting survey responses for net underreporting, using administrative values instead of asking survey questions, and appending additional administrative data, potentially offer significant improvements in the quality of SIPP data on income and program participation. They also raise significant concerns about increased risks of disclosure and delays in the release of SIPP data products. Recommendation 3-6: In the near term, the Census Bureau should give priority to indirect uses of administrative records in a reengineered Survey of Income and Program Participation (SIPP). At the same time and working closely with data users and agencies with custody of relevant administrative records, the bureau should identify feasible direct uses of administrative records in SIPP to be implemented in the medium and   See Recommendation 4-5 regarding an advisory group of outside researchers and policy analysts. 

 REENGINEERING THE SURVEY longer terms. Social Security and Supplemental Security Income benefit records, which are available to the Census Bureau on a timely basis, are prime candidates for research and development on ways to use the admin- istrative values directly—either to adjust survey responses for catego- ries of beneficiaries or to replace survey questions (which would reduce respondent burden)—in ways that protect confidentiality. Recommendation 3-7: When considering the addition to the Survey of Income and Program Participation (SIPP) of administrative records v ­ alues for variables that have never been ascertained in the survey itself, the Census Bureau should ensure that the benefits from the added variables are worth the costs, such as additional steps to protect confidentiality. The bureau should consult closely with users to be sure that the added variables are central to SIPP’s purpose to provide information on the short- run dynamics of economic well-being and that their inclusion does not compromise the ability to release public-use microdata files that accurately represent the survey data. Confidentiality Protection and Data Access Conclusion 3-6: Multiple strategies for confidentiality protection and data access are necessary for a survey as rich in data as the Survey of Income and Program Participation. Public-use microdata files, which are available on a timely basis and in which confidentiality protection tech- niques do not unduly distort the relationships in the data, are the preferred mode of data release. Some uses may require access to confidential data that at present can be provided only at one of the Census Bureau’s Research Data Centers. Recommendation 3-8: The Census Bureau should develop confidenti- ality protection techniques and restricted access modes for the Survey of Income and Program Participation (SIPP) that are as user-friendly as pos- sible, consistent with the bureau’s duty to minimize disclosure risk. In this regard, the bureau should develop partial synthesis techniques for SIPP p ­ ublic-use microdata files that, based on evaluation results, are found to preserve the research utility of the information. For SIPP data that cannot be publicly released, the Census Bureau should give high priority to developing a secure remote access system that does not require visiting a Research Data Center to use the information. The bureau should also deposit SIPP files of linked survey and administrative records data (with identifiers removed) at all Research Data Centers in order to expand the opportunities for research that contributes to scientific knowledge and informed public policy. 

SUMMARY  INNOVATION IN DESIGN AND DATA COLLECTION Event History Calendar Approach Conclusion 4-1: The Survey of Income and Program Participation (SIPP) is the only national survey that provides information on the short- term dynamics of employment, income, program participation, and other family characteristics, and its monthly time frame is essential for many applications. The Census Bureau’s plans to move SIPP to an annual sur- vey, filling in intrayear dynamics using event history calendars, potentially affects—perhaps positively, perhaps negatively—SIPP’s single most impor- tant feature. Conclusion 4-2: The panel is not aware of conclusive evidence that a 12-month event history calendar (EHC) framework is capable (or not) of generating accurate monthly information on income, program participa- tion, and other topics that are covered in the Survey of Income and Program Participation (SIPP). The lack of evidence about the ability of an EHC to collect monthly data places considerable pressure on the Census Bureau, not only to design an effective pretesting program for the EHC methodol- ogy, but also to make its survey reengineering plans for SIPP sufficiently flexible so that it can modify its plans if the pretesting reveals unanticipated, negative evidence on the likely success of the proposed methodology in providing high-quality monthly information. Conclusion 4-3: Understanding transitions at the seam between inter- views in a reengineered Survey of Income and Program Participation (SIPP) using the event history calendar approach will require data from at least two annual interviews. Moreover, not enough is yet known about the f ­ actors driving seam bias in the traditional SIPP. Conclusion 4-4: A parallel traditional Survey of Income and Program Participation (SIPP) panel that provides 2 or more years of data is a neces- sary component of a thorough evaluation of the reengineered SIPP using the event history approach. The recently completed paper test is of limited value for this purpose. The Census Bureau’s planned electronic prototype test is promising but, as a single test, is unlikely to provide conclusive findings. Recommendation 4-1: The Census Bureau should engage in a major program of experimentation and evaluation of the event history approach for developing suitable data on the short-run dynamics of household compo- sition, income, employment, and program participation from a reengineered Survey of Income and Program Participation (SIPP). The details of the 

 REENGINEERING THE SURVEY C ­ ensus Bureau’s plans should be disseminated to SIPP stakeholders for com- ment and suggestions for improvement. If the experimental results indicate that the quality of data on income and program dynamics is significantly worse under the event history calendar approach than in the traditional SIPP, the Census Bureau should return to a more frequent interview schedule, say, every 6 months, devise other methods to improve data on short-run d ­ ynamics, or revert to the traditional SIPP with 4-month interviews using standard questionnaires. Recommendation 4-2: To ensure not only adequate evaluation of a reengineered Survey of Income and Program Participation (SIPP), but also a bridge between data collected under the new and old methods, the ­Census Bureau should conduct traditional and reengineered SIPP panels to pro- vide at least 2 years of comparable data. If the new design works, then the ­parallel traditional panel provides a bridge. If the new design does not work, then the parallel panel provides a backup for the continued collection of SIPP data while the new design is modified as appropriate. Recommendation 4-3: Because the reengineered Survey of Income and Program Participation (SIPP) should be compared with the first year of a traditional SIPP panel in order to minimize attrition bias, the Census Bureau should begin a new traditional SIPP panel in February 2012. If the costs of fielding two concurrent national longitudinal surveys appear prohibitive, the 2012 traditional SIPP panel could be smaller than previous SIPP panels without substantially diminishing its scientific value. Length and Frequency of Interviews and Panels Conclusion 4-5: Design features for a reengineered Survey of Income and Program Participation (SIPP) that are important to evaluate in terms of their effects on respondent burden, survey costs, data quality, and opera- tional complexity include the length and frequency of interviews, the length of panels, and whether successive panels overlap. With regard to interviews, there is no evidence that a 12-month event history calendar strikes the opti- mal balance between respondent burden, costs, and data quality in com- parison to the traditional SIPP design of 4-month interviews. With regard to panels, there is evidence that nonoverlapping panels have adverse effects on cross-sectional estimates of trends over time, yet they are advantageous in terms of larger sample sizes per panel and operational feasibility. Recommendation 4-4: The Census Bureau should study the trade- offs in survey quality and respondent burden in comparison to survey costs between longer but less frequent event history-based interviews in a 

SUMMARY  r ­ eengineered Survey of Income and Program Participation (SIPP) and more frequent interviews in the traditional SIPP. The Census Bureau’s research and evaluation program for SIPP should also improve understanding of panel bias and how it grows over time. Because overlapping panels remain the best way to document the extent of panel bias across the full range of variables collected in SIPP, they should be on the research agenda for possible implementation at a future time. Due to technical demands and c ­ apacity issues that arise in launching the reengineered SIPP, the initial design plans should not include overlapping panels. Content Conclusion 4-6: The Census Bureau has done an exemplary job in reaching out to the Survey of Income and Program Participation user com- munity with “content matrices” and other efforts to identify critical por- tions of the core questionnaire and topical modules for data users. Recommendation 4-5: The Census Bureau should expand the scope of the reconstituted Survey of Income and Program Participation (SIPP) Working Group or establish a new SIPP advisory group with members from academic institutions and policy research organizations that would meet periodically to assist the Census Bureau in its efforts to continually improve the quality and relevance of the SIPP survey content. This group, which could include government members from the recommended interagency working group on uses of administrative records in SIPP (see Recommen- dation 3-5), would review the Census Bureau’s use of cognitive and other methods to evaluate and improve survey question wording and improve response rates (or, when that is not possible, either dropping the question or seeking an alternate data source); assist in benchmarking survey responses against external, reliable sources; and advise the bureau on ways to improve imputation and editing procedures. The group would provide a sounding board for the Census Bureau’s plans to develop appropriate survey content in a reengineered SIPP and advise the bureau on appropriate modifications to survey content as policy developments occur, such as health care and immigration reform. Timeliness Conclusion 4-7: The release of Survey of Income and Program Partici- pation (SIPP) data is often not timely. Data from the 2004 SIPP panel were generally released more than 2 years after being collected. Other panel sur- veys have more timely data release, often within a year of data collection, which enhances their usefulness to external users. 

10 REENGINEERING THE SURVEY Recommendation 4-6: The Census Bureau should release Survey of Income and Program Participation data within 1 year of data collection. Management and Budget Conclusion 4-8: Unlike other surveys of people and households that the Census Bureau conducts, the Survey of Income and Program Participation (SIPP) does not have a government client outside the Census Bureau or a federally mandated set of reports that are based on the survey. Not having an external client, such as the Bureau of Labor Statistics (which has a col- laborative and financial stake in the monthly Current Population Survey), or a set of regular reporting requirements, as with the decennial census and the American Community Survey, has contributed to setbacks in the devel- opment of SIPP. The value of the survey has also been diminished over its history by sample cutbacks necessitated by cutbacks in funding. We agree with an earlier Committee on National Statistics panel (National Research Council, 1993) that SIPP would benefit from a project director with full management and budget authority for design, evaluation, and operations. The budget should always include adequate research and development funding, since SIPP is a major ongoing survey that requires regular evaluation and improvement.