2
SIPP’s History, Strengths, and Weaknesses

This chapter briefly reviews the history of the Survey of Income and Program Participation (SIPP) from the perspective of its original goals and summarizes plans for reengineering the survey.1 It describes SIPP’s strengths under its current design—strengths that a new design needs to maintain. It also describes SIPP’s weaknesses, which a new design needs to ameliorate to the extent possible. Conclusions and recommendations are provided at the end of the chapter.

HISTORY

From its earliest days to the present, SIPP has exhibited a pattern of a forward movement, followed by a setback, followed by another forward movement, another setback, and so on. This pattern has adversely affected the usefulness, quality, and cost-effectiveness of the data at various times. Yet, overall, the survey has shown a marked resilience and has earned the support of users who find the SIPP data indispensable for important kinds of policy analysis and research.

1

Principal sources of information for this chapter include National Research Council (1993); Citro (2007); and presentations by David Johnson, chief of the Census Bureau’s Housing and Household Economic Statistics Division, in 2006 and 2007 (available at http://www.census.gov/sipp/dews.html; see also http://www.census.gov/sipp/).



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 17
2 SIPP’s History, Strengths, and Weaknesses T his chapter briefly reviews the history of the Survey of Income and Program Participation (SIPP) from the perspective of its origi- nal goals and summarizes plans for reengineering the survey.1 It describes SIPP’s strengths under its current design—strengths that a new design needs to maintain. It also describes SIPP’s weaknesses, which a new design needs to ameliorate to the extent possible. Conclusions and recommendations are provided at the end of the chapter. HISTORy From its earliest days to the present, SIPP has exhibited a pattern of a forward movement, followed by a setback, followed by another forward movement, another setback, and so on. This pattern has adversely affected the usefulness, quality, and cost-effectiveness of the data at various times. Yet, overall, the survey has shown a marked resilience and has earned the support of users who find the SIPP data indispensable for important kinds of policy analysis and research. 1 Principal sources of information for this chapter include National Research Council (1993); Citro (2007); and presentations by David Johnson, chief of the Census Bureau’s Housing and Household Economic Statistics Division, in 2006 and 2007 (available at http://www.census. gov/sipp/dews.html; see also http://www.census.gov/sipp/). 

OCR for page 17
 REENGINEERING THE SURVEY Origins and goals The origins of SIPP date to the late 1960s, when policy makers trying to implement antipoverty programs under the War on Poverty expressed dissatisfaction with the quality and detail of data on income and welfare program participation available from the Current Population Survey (CPS) March Income Supplement.2 In 1975 the then U.S. Department of Health, Education and Welfare (HEW) established the Income Survey Development Program (ISDP). Responsibility for designing and analyzing a new survey was shared between the Office of the Assistant Secretary for Planning and Evalua- tion (ASPE) and the Social Security Administration (SSA), both in HEW at the time. The U.S. Census Bureau was charged with collecting the survey data. The ISDP conducted experiments at five test sites in 1977. Next, a 1978 ISDP research panel followed members of about 2,340 original sample households through several interviews. Finally, a 1979 ISDP research panel followed members of about 9,500 original sample households over 6 inter- views every 3 months, for a total of 18 months. The interviews asked about monthly employment, income, and program participation; asset income was ascertained once every 6 months. At about the same time, when plans were well along to implement a new survey to be called SIPP, an interagency memorandum was drawn up in 1980 stating the survey’s goals (see Kasprzyk, 1988). Signed by representa- tives of SSA, ASPE, and the Census Bureau, the memorandum stipulated that SIPP’s goals were to 1. extend the scope and precision of policy analyses for a wide range of federal and state tax and social welfare programs; 2. improve current estimates of income and income change, including annual and subannual estimates, by source of income; and 3. broadly assess the economic well-being of the population. First Crisis SIPP’s first crisis occurred at the moment when it was officially sup- posed to begin. The transition from the ISDP to SIPP was scheduled for 1981, with operational control of the survey transferred from ASPE to SSA. While ASPE and the Census Bureau were to remain as partners in 2 The CPS March Income supplement was renamed the CPS Annual Social and Economic Supplement (ASEC) when the sample for the supplement was expanded to improve the reli- ability of state estimates of children’s health insurance coverage and some of the cases were interviewed in February and April (most cases are still interviewed in March). This sample expansion was first implemented in the 2002 CPS. Hereafter, we use “the CPS” when discussing income and program participation information from the supplement and “the monthly CPS” when discussing the core data on labor force participation that are collected every month.

OCR for page 17
 SIPP’S HISTORY, STRENGTHS, AND WEAKNESSES the survey, the bulk of the funding was in the SSA budget. The election of Ronald Reagan as president, however, brought new policy priorities to the federal government. These new priorities caused the new administration and Congress to cancel SIPP. In 1982 SIPP experienced the first of many last-minute reprieves. Bruce Chapman, the new director of the Census Bureau, convinced the White House to restore its funding. He argued that because SIPP would record more income than the CPS (based on the ISDP tests), it would produce a lower poverty rate, compared with the official poverty rate computed from the CPS. In restoring SIPP, full funding went to the Census Bureau, rather than being funneled through user agencies, such as ASPE and SSA, as originally planned. The First Decade (1983-1993) The first SIPP panel (the 1984 panel) began in October 1983. It origi- nally included about 21,000 sample households, whose adult members age 15 and older the Census Bureau attempted to follow for 8 or 9 waves of interviews conducted every 4 months. However, 7 percent of the sample had to be dropped after Wave 4 because of budget cuts. Original sample members who moved within the United States were interviewed at their new address, unless they moved into an institution or became homeless. People in institutional settings and homeless people were not part of the sample, nor were people who moved outside the United States. Children under age 15 and adults who moved in with an original sample member after the first interview wave were included in the data collection so long as they resided with the original sample member.3 The SIPP design called for sample members to be interviewed every 4 months in order to increase the accuracy of answers to core questions on income amounts, participation in social programs, employment status, and health insurance coverage on a month-by-month basis compared with interviews at longer time intervals. Experiments conducted in the ISDP sup- ported the use of 3-month interviews compared with 6-month interviews (Ycas and Lininger, 1983:28), and 4 months was a compromise. The core of each interview also included questions on key background characteristics, such as education, family composition, and ages of household members. In addition to the core questions, SIPP included one or more topical modules on important issues related to well-being and social policy. Questions in the topical modules, which covered a wide range of subjects (see Box 2-1), were asked only once or twice in a single panel. 3 The 1993 SIPP panel followed children under age 15 even if they no longer resided with an original sample adult; however, the practice was abandoned in subsequent panels because so few such children were actually located.

OCR for page 17
0 REENGINEERING THE SURVEY BOX 2-1 Topical Modules in SIPP Panels, 1984-2004 Child care and support modules • Child Care (once or twice in every panel) • Child Support Agreements (once or twice in every panel beginning in 1985) • Child Support Paid (2-4 times in 1996, 2001, 2004 panels) • Informal Care-Giving (once in 2004 panel) • Support for Nonhousehold Members (once or twice in every panel) • Welfare History and Child Support (once in 1984 panel) Disability and health care utilization modules • Disability Status of Children (once or twice in 1985-1989 panels) • Employer-Provided Health Benefits (once in 1996, 2001, 2004 panels) • Functional Limitations and Disability (once or twice in 1990-1991 panels); separate modules for adults and children (once or twice in 1992, 1993, 1996, 2001, 2004 panels) • Health and Disability (once in 1984 panel) • Health Status and Utilization of Health Care Services (1-3 times in every panel beginning in 1985) • Home Health Care (once in 1988-1989 panels) • Long-Term Care (once or twice in 1985-1989 panels) • Medical Expenses and Work Disability (once in 1987-1992 panels; 2-4 times in 1993, 1996, 2001, 2004 panels) • Work Disability History (once early in every panel beginning in 1986) Education modules • Education and Training and Education and Work History (once each in 1984 panel) • Education and Training History (once early in every panel beginning in 1986) • School Enrollment and Financing (once or twice in every panel through 1996) Employment modules • Employment History (once early in every panel beginning in 1986) • Home-Based Self-Employment/Size of Firm (once in 1992-1993 panels) • Job Offers (once in 1985-1986 panels) • Reasons for Not Working/Reservation Wage (once in 1984 panel) • Time Spent Outside Workforce (once in 1990 panel) • Work Expenses (once or twice in 1984-1987 panels; 2-4 times in 1996, 2001, 2004 panels) • Work Schedule (once or twice in every panel beginning in 1987)

OCR for page 17
 SIPP’S HISTORY, STRENGTHS, AND WEAKNESSES Family background modules • Family Background (once in 1986-1988 panels) • Fertility History (once early in every panel) • Household Relationships (once early in every panel) • Marital History (once early in every panel) • Migration History (once early in every panel) Financial modules • Annual Income and Retirement Accounts (once or twice in every panel; 3 times in 1996 panel) • Assets and Liabilities (once or twice in every panel; 3-4 times in 1996, 2001 panels) • Housing Costs, Conditions, and Energy Usage (once in 1984 panel) • Retirement Expectations and Pension Plan Coverage (once in most panels) • Selected Financial Assets (once in selected panels) • Shelter Costs and Energy Usage (once in 1986-1987 panels) • Taxes (once or twice in every panel) Program participation modules • Real Estate Property and Vehicles (once or twice in most panels—for deter- mining program eligibility) • Real Estate, Shelter Costs, Dependent Care, and Vehicles (once or twice in selected panels—for determining program eligibility) • Recipiency History (early in every panel beginning in 1986) • Welfare Reform (once in 1996, 2001, 2004 panels) Well-being modules • Adult Well-Being (once in 1993, 1996, 2001 panels) • Basic Needs (once in 1993 panel) • Child Well-Being (1-3 times in 1993, 1996, 2001, 2004 panels) • Extended Measures of Well-Being (once in 1991-1992 panels) NOTE: Over the history of SIPP, the content of some topical modules changed with no change in title or the title changed with little change in content. Sometimes two topical modules with different titles have had similar content. There were no topi- cal modules in Waves 9-12 of the 2004 panel. The actual questions are provided with the microdata technical documentation for the SIPP public-use files from the Census Bureau. SOURCE: See http://www.census.gov/sipp/top_mod/top_mods_chart.html.

OCR for page 17
 REENGINEERING THE SURVEY Building on the 4-month interval between interviews, a SIPP sample is divided into four equally sized rotation groups, which are interviewed in successive months. In addition to distributing the survey fieldwork uni- formly over time, this rotation group structure ensures that the survey esti- mates for a given calendar month represent an average of responses given 1, 2, 3, and 4 months later.4 Thus, any response bias associated with the reference month—for example, a decline in accuracy with distance from the interview—will affect all calendar months equally. New SIPP panels began every February from 1985 through 1993. These panels were designed to overlap in time, so that samples from two different panels could be combined to provide representative cross-sectional estimates for a given year of the poverty rate and other characteristics, whereas a single panel would provide longitudinal information on intrayear transitions in employment, poverty status, and other characteristics for a sample of people followed over 2-3 years. However, because of lack of time and resources, the Census Bureau did not combine panels for analytical use, although it did provide factors to apply to the panel weights so that users could produce estimates from two overlapping panels. The sample design for each panel was a multistage clustered probability sample of the population in the 50 states and the District of Columbia that excluded only inmates of institutions and those members of the armed forces living on base without their families. There was no oversampling of specific population groups in SIPP in the 1984-1993 panels, except that the 1990 panel included about 3,800 extra households continued from the 1989 panel, most of them selected because they were headed by blacks, Hispanics, or female single parents at the first wave of the 1989 panel. Original sample sizes for the 1984 through 1993 panels ranged from 12,400 to 21,800 households, and the number of interview waves ranged from 3 to 10 (see Table 2-1, which also includes information for the 1996, 2001, and 2004 panels). In the early years of the survey, SIPP interviewers conducted in-person interviews of sample members using paper and pencil questionnaires. Telephone interviewing was tested in the 1986 panel (Gbur, Cantwell, and Petroni, 1990) and first used on a production basis in Febru- ary 1992 in Wave 7 of the 1990 panel and Wave 4 of the 1991 panel. In the 1992 and 1993 panels, SIPP interviewers conducted in-person interviews for Waves 1, 2, and 6 and telephone interviews to the maximum extent possible for the other waves. In the 1984, 1985, and 1986 panels, SIPP did not collect all of the information, such as shelter costs and medical expenses, that was necessary 4 The rotation group design is incorporated into the SIPP survey weights as well. All cross- sectional and longitudinal weights are calculated so that each rotation group represents one- quarter of the population.

OCR for page 17
 SIPP’S HISTORY, STRENGTHS, AND WEAKNESSES TABLE 2-1 Characteristics of SIPP Panels, 1984-2004: Number of Waves, Original Sample Size, Reduced Sample Size (if applicable), and Cumulative Sample Loss by Wave Reduced Overall Original Sample Loss Cumulative Sample Loss at Wave No. of Sample Size (in (Final Wave)d Panel Waves Size Wave x) 1 3 6 9 12 (8)a 1984 9 20,900 19,500 (5) 4.9 12.3 19.4 22.3 N.A. 22.3 (9) 8 (7)b 1985 14,300 13,500 (4) 6.7 13.2 19.7 N.A. N.A. 20.8 (8) 7 (6)b 1986 12,400 7.3 15.2 20.0 N.A. N.A. 20.7 (7) 1987 7 12,500 6.7 14.2 18.9 N.A. N.A. 19.0 (7) 1988 6 12,700 7.5 14.7 18.3 N.A. N.A. 18.3 (6) 1989 3 12,900 7.6 13.8 N.A. N.A. N.A. 13.8 (3) 1990c 8 19,800 7.3 14.4 20.2 N.A. N.A. 21.0 (8) 1991 8 15,600 8.4 16.1 20.3 N.A. N.A. 21.4 (8) 1992 10 21,600 9.3 16.4 21.6 26.2 N.A. 26.6 (10) 1993 9 21,800 8.9 16.2 22.2 26.9 N.A. 26.9 (9) 1996 12 40,200 8.4 17.8 27.4 32.8 35.5 35.5 (12) 2001 9 40,500 30,500 (2) 13.3 24.7 28.2 31.9 N.A. 31.9 (9) 2004 12 51,400 21,300 (9) 14.9 25.6 31.2 34.0 36.6 36.6 (12) NOTES: N.A. = Not applicable. Original and reduced sample sizes are rounded to the nearest hundred households. Original sample sizes are the number of households eligible to be interviewed at the start of Wave 1. Reduced sample sizes are reductions in original sample sizes due to budget cuts (the wave in which the cut took effect is indicated in parentheses). Reduced sample sizes do not reflect reduction in sample sizes due to attrition; nor do they reflect growth (or decline) in sample sizes due to changes in household composition because original sample people moved out of or back into an original sample household. Sample loss rates consist of cumulative noninterview rates adjusted for unobserved growth in the noninterviewed units (created by household splits after Wave 1). There are some differ- ences in the calculation of sample loss between the 1984-1993 and 1996 panels, which allowed nonresponding households to drop out of a panel permanently if they missed a specific number of waves, and the 2001and 2004 panels, which kept all nonresponding households after Wave 1 in the sample. aTwo rotation groups in the 1984 panel received nine interview waves; the other two groups received eight waves—one group skipped Wave 2, and another group skipped Wave 8 in order to align the timing of collection of income tax information. bOne rotation group in each of the 1985 and 1986 panels received one fewer wave than the other three groups in order to collect income tax information in approximately the same time period. cSample loss rates are for the nationally representative portion of the sample; they exclude about 3,800 extra households headed by blacks, Hispanics, or single parents that were con- tinued from the 1989 panel. dThe last wave of interviewing in a panel is indicated in parentheses. SOURCE: Tabulations provided by Census Bureau staff.

OCR for page 17
 REENGINEERING THE SURVEY to simulate eligibility for assistance programs, such as food stamps and Aid to Families with Dependent Children. Eligibility determination is essential to understand trends in program take-up rates—for example, an increase in the number of program participants could be due to an expansion of eligibility that did not alter the take-up rate, or to an increase in the take-up rate among already-eligible participants who decided to apply for benefits, or to both factors. In response to user requests, Wave 6 of the 1987 panel collected information on selected financial assets and medical expenses and disability that allowed eligibility simulations to begin. These modules were asked once in each of the 1988, 1990, 1991, and 1992 panels and twice in the 1993 panel. Beginning with Wave 7 of the 1993 panel, the eligibility modules were combined with the assets and liabilities module, and the com- bined modules were asked annually in the 1996 and subsequent panels. Budget shortfalls necessitated a reduction in sample size in the 1984 and 1985 SIPP panels. Budget constraints also limited the sample size and number of interview waves in all panels initiated between 1985 and 1989 (see Table 2-1). These reductions and fluctuations in panel size and length made it difficult to plan for either fieldwork or analysis with confidence. Some of these panels were so short as to be effectively useless, except for cross-sectional analyses. Consequently, the U.S. Office of Management and Budget (OMB) requested an evaluation of SIPP, and in 1990 the Census Bureau asked the Committee on National Statistics (CNSTAT) for a report on the future of SIPP (National Research Council, 1993).5 Funding was also secured to boost somewhat the sample sizes and length of the 1990-1993 panels. The Future of the Survey of Income and Program Participation, the CNSTAT panel’s report, appeared in 1993 and contained a long list of recommendations for improving the content, design, and operation of the survey. After the CNSTAT panel report and an internal evaluation, SIPP underwent a major redesign that became effective with the 1996 panel. To maximize sample size for longitudinal analysis with a single panel and so reduce the need to combine panels for analysis purposes, as well as to reduce the burden on the field interviewers, the practice of introducing a new panel every year was dropped. The CNSTAT panel had recommended introducing new panels every 2 years and continuing them for 4 years, so that two panels would be in the field at any one time. The Census Bureau decided on a design that ran a panel through to completion before starting another panel. 5 The full-scale study was preceded by an interim evaluation (National Research Council, 1989).

OCR for page 17
 SIPP’S HISTORY, STRENGTHS, AND WEAKNESSES The Second Decade (1996-2006) The 1996 panel began with 40,200 original sample households, of which about 36,800 (92 percent) were interviewed in Wave 1 and their members followed for as many waves as possible through 12 waves (4 years). Households in a high-poverty stratum based on their 1990 census characteristics were oversampled relative to other households. SIPP also kept track of original sample members who moved into institutions and resumed interviews with them when and if they rejoined the household population. An effort to field a new 4-year panel in 2000 was aborted because of the need for staff to devote full attention to the 2000 census. The 2001 panel began with about 40,500 households, of which about 35,100 were interviewed in Wave 1 (87 percent); the sample was reduced by 25 percent for budgetary reasons in Wave 2, and the members of the remaining sample households were followed for as many waves as possible through nine waves. The 2004 panel began with about 51,400 households, of which about 43,700 (85 percent) were interviewed in Wave 1; the mem- bers of these households were followed for as many as 12 waves, with a 58 percent sample reduction in the last four waves to free up resources to reengineer the survey (see “Reengineering (2006-Present),” below). Both the 2001 and 2004 panels oversampled the low-income population based on census characteristics (the 1990 census for the 2001 panel and the 2000 census for the 2004 panel). The interviews from these various SIPP panels have yielded important data on a range of policy-related issues.6 For example, Vaughan (2007) ana- lyzed data from the first year of the 1996 SIPP panel to identify factors that would be likely to facilitate or impede the ability of participants in the Aid to Families with Dependent Children (AFDC) to make the transition from welfare to work subsequent to implementation of the Public Responsibility and Work Opportunity Reconciliation Act of 1996 (PRWORA, or welfare reform). He found that nearly half of AFDC recipients possessed two or more attributes (such as a disability) that impeded work in the period of transition to the new regime, in which work was the primary emphasis of the program. Only 30 percent of these participants held a job in 1996. In contrast, 41 percent of recipients possessed three or more attributes that facilitated work, and 68 percent of them held a job during 1996. A find- ing of note was that the age of the participants’ children did not seem to represent a substantial barrier to work. The Congressional Budget Office (CBO, 2003) analyzed data on health insurance coverage from the SIPP 1996 panel, the 1998 and 1999 Medical 6 A recent estimate puts the number of publications based on SIPP at over 2,000 books, articles, reports, and other written products issued through 2006 (see http://www.census. gov/sipp/aboutbib.html).

OCR for page 17
 REENGINEERING THE SURVEY Expenditure Panel Survey (MEPS), and the 1998 and 1999 CPS. From the SIPP and MEPS data, CBO estimated that 21 to 31 million people lacked coverage for an entire year in 1998 compared with the widely cited CPS estimate of about 44 million. CBO also estimated from SIPP and MEPS that about 41 million people lacked health insurance coverage at a specific point in time in 1998, while about 58 million lacked coverage at some point during the year. Looking at the duration of spells without coverage experienced by nonelderly people using 11 waves of the SIPP 1996 panel, CBO estimated that 45 percent of the uncovered spells that began between July 1996 and June 1997 lasted only 4 months, while 26 percent lasted 5-12 months, and 29 percent lasted more than a year. These different mea- sures of health insurance coverage and estimates of duration of spells have important implications for the design of more effective health care coverage in the United States. A companion Survey of Program Dynamics (SPD) is part of SIPP’s his- tory in its second decade. The SPD was mandated by PRWORA; it followed households that completed the 1992 and 1993 SIPP panels annually from 1997 through 2002. The SPD core instrument asked about employment, income, program participation, health insurance and utilization, child well- being, marital relationships, and parents’ depression. SPD topical modules included a self-administered adolescent questionnaire asked in 1998 and 2001, additional child-related questions asked in 1999 and 2002, and resi- dential histories of children asked in 2000. The SPD experienced some of the same problems as the main SIPP (see “Strengths and Weaknesses of SIPP Data” below), including not only data processing delays, but also high attri- tion rates until additional efforts and incentives were used to bring house- holds back into the survey. (For more information, see http://www.census. gov/spd/overview.html; http://www.census.gov/spd/reports/pu02strp.html.) Crisis in 2006 Like many events in the nation’s capital, SIPP’s most recent crisis resulted from a threatened cutoff of funds. In its Budget for Fiscal Year 00 (delivered to Congress in February 2006), the Bush administration planned for the Census Bureau to cut $40 million from its budget as part of a larger set of proposed reductions in domestic spending. Given the bureau’s need to prepare for the 2010 decennial census, its choices were limited. Essentially, the cut could be accomplished either by taking pieces away from several different programs, making each of them less effective, or by eliminating one program and allowing the remainder to keep to their planned budgets and schedules. This time, rather than continue a strategy of “death by a thousand cuts” in several programs, the Census Bureau decided that it would be

OCR for page 17
 SIPP’S HISTORY, STRENGTHS, AND WEAKNESSES preferable simply to drop one whole program. SIPP was chosen as the pro- gram to drop for a number of reasons. First, and perhaps most important, SIPP had no outside agency sponsor or legal mandate. The Census Bureau cannot unilaterally choose to terminate most of its surveys, because other departments and agencies depend on their output and frequently contribute to their budgets. The monthly CPS, for example, is sponsored by the Bureau of Labor Statistics (BLS), even though the data are collected by Census Bureau field staff. The monthly CPS is used to calculate monthly unemploy- ment statistics, and the need to keep producing those statistics would pre- vent the Census Bureau from making major changes in the survey on its own initiative. Similarly, various economic surveys produce data used in computing the National Income and Product Accounts, the source of data on the gross domestic product. Changing any of these surveys would pro- voke protests from the Bureau of Economic Analysis, the Federal Reserve Board, and other agencies that use their data, and so would be far harder to accomplish than changing or dropping SIPP. A second issue in the decision to drop SIPP was concern about its quality and usability. At the time of the proposed cut and even before, many SIPP users believed that the survey had developed serious problems, completely apart from the funding crisis (although some may have been related to previous economies in data collection and processing). The prob- lems included sample attrition (explicitly cited by the Census Bureau in its announcement about dropping SIPP),7 underreporting of both income and participation in various government programs, a lengthy lag between data collection and the availability of public-use files because of an outdated and cumbersome data editing and processing system, and the difficulty many users had in working with SIPP data files because of their complex structure and inadequate documentation (see “Strengths and Weaknesses of SIPP Data” below). All of these problems had long been recognized and indeed were exam- ined in detail in the 1993 report The Future of the Survey of Income and Program Participation. Although the Census Bureau had continued its attempts to address the problems and to act on the recommendations pro- vided by the CNSTAT report, these efforts were at best only partially suc- cessful, and many of the earlier recommendations were essentially ignored, in large part because funding was not available to carry them out. Thus, with or without a funding crisis, the Census Bureau needed to perform nontrivial surgery on SIPP, and it was working on redesigning the data processing system to implement when a new panel began following the 2004 panel. 7 E-mail memorandum from Carole Popoff to the Census Bureau’s electronic mailing list for SIPP users, February 6, 2006.

OCR for page 17
 REENGINEERING THE SURVEY The bureau’s proposal for fiscal 2007 retained only $9.2 million of SIPP’s full funding of about $44 million.8 Of this amount, $3.6 million was to support continued data collection for the 2004 panel, although the Census Bureau said that either it would need additional funding from other agencies or else the 2004 panel would have to be terminated in September 2006 (the original plan was to continue it through the end of 2007). The remaining $5.6 million in the bureau’s proposed budget for fiscal 2007 would be used to design a new program to collect longitudinal information, dubbed shortly thereafter the Dynamics of Economic Well- being System (DEWS). The new system would rely much more heavily on administrative data for information on program participation and would markedly scale back the survey component. One thought was that an existing survey, such as the CPS, might provide baseline data for a sample cohort: current and retrospective income and program data would be obtained from administrative records, and additional information would be obtained from follow-up interviews at annual intervals. The original charge to our panel was to evaluate the plans to use administrative records for this new program. When the Census Bureau announced in early 2006 that SIPP would be terminated and replaced with DEWS, the user community, led by researchers at the Center for Economic and Policy Research, reacted with unexpected speed and forcefulness. SIPP advocates sent letters to Congress arguing that SIPP was crucial to policy research and should be continued and that the Bush administration’s recommended cuts in SIPP should not be imple- mented. One public letter, signed by over 400 researchers, including two Nobel laureates, was sent to Congress on March 1, 2006. The letter attested that SIPP “provides a constant stream of in-depth data that enables govern- ment, academic, and independent researchers to evaluate the effectiveness and improve the efficiency of several hundred billion dollars in spending on social programs” and that cutting the survey would lose the investment made over the years in collecting and using the data for important policy analysis and applied social science research.9 The letters were accompanied by effective lobbying of Congress, especially of staff and members on the appropriations committees in the House and Senate that control the Census Bureau’s budget. The lobbying campaign was assisted by a surprising level of coverage in the media, including an editorial in the New York Times on March 4, 2006, recommending that SIPP be retained. 8 E-mail memorandum from Carole Popoff to the Census Bureau’s electronic mailing list for SIPP users, February 6, 2006. SIPP had been receiving about $34 million in annual appro- priations plus another $10 million originally appropriated for the completed SPD, which was reallocated to SIPP. 9 Available at http://www.ceprdata.org/savesipp/resletter-name.pdf; see also Glenn (2006).

OCR for page 17
 SIPP’S HISTORY, STRENGTHS, AND WEAKNESSES Reengineering (2006-Present) Congress passed a fiscal 2007 budget in February 2006, in which it refused to accept the administration’s proposal to terminate SIPP, although it did not restore full funding. Instead, Congress cut SIPP funding by about 25 percent, from about $44 million to $32.6 million (including $10 million from the appropriation that originally provided for the SPD). The Census Bureau in turn cut the 2004 SIPP panel sample size by over 50 percent, from 45,700 original sample households still eligible for the survey (this number includes new households formed by panel members after Wave 1) to 21,300 original sample households for the last four waves of the panel; it also eliminated the topical modules for these waves. This reduction allowed the agency to reduce SIPP spending of about $44 million annually to $25.4 mil- lion and to use part of the savings to continue disseminating data to users from earlier waves of the 2004 panel. The Census Bureau planned to use the remaining $7 million of the 2007 appropriation to work on developing the new DEWS program to replace SIPP. In effect both the advocates who wanted SIPP to continue and the Census Bureau and a portion of the user community who wanted to redesign SIPP got part of what they wanted: SIPP was continued, albeit with a reduced sample, and the Census Bureau continued work on developing the DEWS program. With the restoration of funds for SIPP in the 2007 budget and again in the 2008 budget, the Census Bureau in September 2008 began a new panel under the existing design and processing system with a sample of about 45,000 households. In addition, at the instigation of Congress and data users, the bureau abandoned the DEWS concept of using administrative records in place of most survey content and instead embarked on a redesign or reengineering of SIPP. Thus, the report of the House Appropriations Committee on the Commerce, Justice, Science, and Related Agencies Appro- priations Bill, 2008, issued July 19, 2007, directed “the Bureau of the Census to suspend activity on the DEWS survey development” and, instead, “to work with stakeholders to reengineer the SIPP to develop a more accurate and timely survey to capture the economic dynamics of the country.” The currently available funding is sufficient to continue the 2008 panel with a full sample. This level of funding also allows for work to go forward on reengineering the current SIPP, including the following components: • improvements in the data collection instrument and processing system to achieve greater efficiency of operations and timeliness of data products, such as converting the current DOS-based soft- ware that supports computer-assisted interviewing for SIPP to a Windows-based system called BLAISE;

OCR for page 17
0 REENGINEERING THE SURVEY • development and evaluation of an event history calendar to facili- tate collection of monthly core data in annual interviews; • evaluation of administrative records data to supplement and evalu- ate the survey data; and • development of survey content and use of reimbursable supple- ments, through interactions with stakeholders. The goal is to implement the first 3- or 4-year panel under the new design in 2013. If the testing program supports it, the new design for SIPP panels will consist of three (or four) annual interview waves, each of which will collect data for the previous calendar year (using an event history calendar), with content similar to that collected in the current SIPP core questionnaire, plus some previously topical module content moved into the core. There will be no topical modules as such, but agencies can obtain additional information by paying for supplemental questions, which are most likely to be asked between the core interviews. STRENgTHS AND WEAKNESSES OF SIPP DATA Ideally, a reengineered SIPP would preserve or even enhance the sur- vey’s strengths while ameliorating many of its weaknesses. SIPP’s principal strengths include • its unique and extensive monthly data on employment, earnings, program participation, and household composition; • the information collected on assets, shelter costs, medical expenses, and other items in its periodic topical modules that is necessary to simulate program eligibility and take-up rates; • the detailed information collected on an array of subject areas related to socioeconomic well-being in its periodic topical modules; and • the overall quality of the information collected on program par- ticipants and the low-income population generally relative to other household surveys. SIPP’s major weaknesses include • a marked decline in the quality of income data as income rises; • misplaced and erroneous transitions in income receipt, program participation, and health insurance coverage; • possible biases arising from attrition and an underrepresentation of new entrants to the population (such as births, immigrants from abroad, and people moving from group quarters to household residences);

OCR for page 17
 SIPP’S HISTORY, STRENGTHS, AND WEAKNESSES • a lack of timeliness in the release of data files; and • until the late 1990s when the first edition of the SIPP Users’ Guide was published, inadequate documentation to assist users in work- ing with the complex SIPP public-use microdata files.10 SIPP panels are also shorter in length than panels in most other longitudi- nal data sets, which limits the usefulness of the information from SIPP for modeling long-run dynamics. SIPP’s unique value SIPP stands alone among nationally representative household surveys in collecting income and program participation by month on a recurrent basis, and it does so at the person level for an extensive array of sources. Because of this feature, SIPP is uniquely able to support monthly estimates of participation in and eligibility for many federal and even state programs, although eligibility simulations still require imputation of components (such as assets, shelter costs, child care expenses, and other employment-related expenses) that are either not collected in the SIPP or are collected at times other than the month being estimated. SIPP is also unique in its ability to support models of short-term dynamics over a wide range of character- istics, including models of earnings dynamics based on its monthly data on employers and wages. The household component of the continuous MEPS—see http://www.meps.ahrq.gov/mepsweb/—also collects data on short-term dynamics of employment and health insurance coverage, in 5 interviews over a 2.5-year period for each panel, providing 2 calendar years of data. However, income data are collected only twice in MEPS panels, using a calendar-year reference period, and MEPS has a markedly smaller sample size than SIPP, even when two overlapping panels are combined for calendar-year estimates. MEPS also covers a shorter span of time than SIPP (2 years versus 3 or 4 years), which limits analysis of transitions that are experienced by only a small proportion of the population in a given year. SIPP’s topical modules expand the survey’s content to include types of data that few other surveys collect—such as wealth, child care and housing expenditures, and marital and immigration histories. SIPP’s topical mod- ule data on disability have become the model of excellence for disability measurement. 10 The first edition of the SIPP Users’ Guide covered the 1984-1993 panels; it was updated through the 1996 panel in 2001 and is currently partially updated through the 2008 panel (see http://www.census.gov/sipp/usrguid.html).

OCR for page 17
 REENGINEERING THE SURVEY Overall Quality of SIPP Income Data11 Assessments of data quality in a national survey such as SIPP typically rely on comparisons with other surveys or, for certain types of data, admin- istrative records. Unless a particular other survey has been established as the gold standard in a given area—as is true, for example, of the Survey of Consumer Finances for the measurement of wealth—comparisons across surveys may indicate only where surveys differ and not which is best. Com- pounding the difficulty of evaluating SIPP data is the general uniqueness of SIPP’s monthly estimates among surveys. The survey’s great strength lies in collecting data that are not obtained elsewhere, but this limits how fully SIPP data can be evaluated. No survey matches program administrative totals with respect to total recipients or, especially, aggregate dollars, but among the major national surveys SIPP performs best overall. For programs with high turnover, such as Medicaid, SIPP finds as many participants in a typical month as the CPS finds over a calendar year (Czajka and Denmead, 2008). This suggests that SIPP’s superiority may be a direct result of its frequent interviews and short reference period, underscoring the challenge that the Census Bureau faces in planning to reduce three interviews per year to just one, with a 12-month reference period. Compared with the CPS, the official source of income and poverty statistics for the United States, SIPP captures more income from families in the bottom quintile of the family income distribution, finds more sources of income and less reliance on Social Security among the elderly, and finds a somewhat smaller proportion of the population in poverty. Except for self-employment and entitlement programs, however, SIPP’s superiority in the measurement of income is restricted to the bottom quintile. Overall, SIPP captures only 89 percent of the aggregate income recorded in the CPS, which in turn underestimates total household income in comparison to administrative records. The American Community Survey (ACS), which uses a mailout/mailback questionnaire to collect data from about half of its respondents, obtains 98 percent as much total income as the CPS (Czajka and Denmead, 2008). Data Quality Shortcomings With the monthly data collected in SIPP, users can estimate transi- tions involving a wide range of phenomena, including labor force activity, program participation, health insurance coverage, and family composition. 11 A n extended discussion of SIPP data quality, including additional citations, appears in Appendix A.

OCR for page 17
 SIPP’S HISTORY, STRENGTHS, AND WEAKNESSES Estimates of the timing of transitions and the duration of spells created by transitions are affected by various types of reporting error that can generate a pronounced seam bias—that is, a tendency for transitions to fall dispro- portionately at the seams between waves rather than within the surround- ing reference periods. In SIPP, transitions can occur between months 1 and 2, 2 and 3, or 3 and 4 of a 4-month reference period or between month 4 of one reference period and month 1 of the next reference period. SIPP’s rota- tion group structure distributes interviews uniformly by calendar month, so changes in such characteristics as program participation, employment, and health insurance coverage should occur with the same frequency between any consecutive pair of reference months within or between survey waves. Instead, such transitions are more likely to be reported between month 4 of one wave and month 1 of the next wave than between any pair of months within the same wave. The extent of seam bias varies widely across characteristics but is particularly strong for health insurance coverage and program participation in general. For example, in one recent analysis of the 2001 SIPP panel, between 83 and 100 percent of transitions into or out of the major sources of health insurance coverage were reported at the seam between interviews (Czajka and Mabli, 2009). While the likely causes of seam bias in panel surveys are many and varied (Callegaro, 2008), the principal source of seam bias in reported health insurance coverage in SIPP appears to be a tendency for respondents to report that they or other household members were covered by a par- ticular source for either all 4 months or no months of the reference period. This phenomenon has a pronounced impact on distributions of duration. Excluding persons who were uninsured for all 36 months, 64 percent of the nonelderly adults who were uninsured for some portion of the 2001 panel were reported as uninsured for a multiple of 4 months (Czajka and Mabli, 2009). While seam bias may pose a serious problem for longitudinal analysis with SIPP, its impact on cross-sectional estimates is muted by SIPP’s rotation group design, which ensures that seams are distributed uniformly across calendar months. Monthly estimates will reflect any net reporting bias, but the bias for any survey wave will be distributed uniformly across the calendar months of the reference period. This is important for estimates of monthly program eligibility and participation. Too Many Transitions? Health insurance coverage estimates from SIPP illustrate the general problem of overstated transitions and their implications for longitudinal analysis. Average monthly estimates of health insurance coverage from SIPP compare closely with estimates of health insurance coverage obtained

OCR for page 17
 REENGINEERING THE SURVEY in the National Health Interview Survey, which measures coverage at the time of the interview (Czajka and Denmead, 2008; Davern et al., 2007). However, changes in coverage in SIPP occur with a frequency that strains belief—particularly among children. Among both adults and children, per- sons who experience changes in coverage often revert back to their original coverage at the start of the next wave, suggesting that reporting error may play an important role (Czajka and Mabli, 2009). Attrition Attrition is the bane of panel surveys, as more and more cases drop out because they move and cannot be found or refuse to stay in the survey. While SIPP enjoyed initial response rates at Wave 1 above 90 percent prior to the 2001 panel, the Wave 1 response rate dropped to 87 percent in the 2001 panel and 85 percent in the 2004 panel (see Table 2-1). Moreover, cumulative attrition has always been appreciable. In the 1996 panel, by the end of Wave 12, the cumulative sample loss—including the 8.4 percent initial Wave 1 nonresponse—exceeded 35 percent. With the discontinua- tion of a practice of terminating households that missed two consecutive interviews after Wave 1, the Census Bureau reduced the cumulative attrition rate at Wave 9 by 1 percentage point between the 1996 and 2001 panels. Nevertheless, cumulative attrition remains high (Czajka, Mabli, and Cody, 2008), and indeed increased for the 2004 panel. It should be noted that attrition is increasing over time with all household surveys. Even more than its impact on sample size, attrition raises concerns because of its potential biasing effect. There is ample evidence from com- parisons of characteristics measured in the initial waves of panel sur- veys that attriters differ from stayers. However, evidence using matched administrative records, which are not subject to differential reporting error between attriters and stayers, indicates that differences between the two groups diminish over time (Vaughan and Scheueren, 2002). In a long panel, even with no adjustment for differential attrition in the survey weights, cross-sectional bias will be reduced by this phenomenon, but the amount of change over time will be underestimated. Another study using the same sources of administrative records found that there were negli- gible differences between the stayers—reweighted to represent the Wave 1 universe—and the full, initial sample on annual earnings reported to the Internal Revenue Service (IRS), Social Security income and type of recipi- ency, and benefit amounts from the Supplemental Security Income (SSI) program (Czajka, Mabli, and Cody, 2008). Even estimates of change over time showed little evidence of bias. While limited to a small set of variables, these findings suggest that when respondents leaving the survey universe are handled appropriately and the Census Bureau’s weighting adjustments

OCR for page 17
 SIPP’S HISTORY, STRENGTHS, AND WEAKNESSES are taken into account, the evidence of attrition bias in the SIPP is not as strong as is commonly assumed. Nevertheless, as long as attrition remains high, there is always reason to be concerned that the remaining sample cases may over or underrepresent particular types of people, events, or temporal phenomena—especially those associated with disruptions in per- sonal circumstances. Other Bias Concerns Although SIPP is a panel survey, cross-sectional uses may be more com- mon than longitudinal analyses of SIPP data. Evidence that cross-sectional estimates of poverty show trends that deviate from trends recorded in the CPS suggests a panel bias that should caution users against reliance on cross-sectional estimates from later waves (Czajka, Mabli, and Cody, 2008). If attrition is not the principal cause, then renewed efforts to under- stand the sources of the problem would benefit the survey redesign. A pos- sible contributor to the problem of panel bias in cross-sectional estimates is SIPP’s underrepresentation of persons who join the population after the initial interview (Czajka and Mabli, 2009). Recent panel estimates show an appreciable reduction in poverty between Waves 1 and 2, yet little change over the next waves. Seeking an explanation in the first two waves, Czajka, Mabli, and Cody (2008) com- pared poverty status between the first two waves of the 2004 panel and found that changes in recorded poverty among persons present in both waves, rather than excess attrition among the Wave 1 poor, accounted for 87 percent of the net reduction in the number of poor. Did the experience of the Wave 1 interview make the respondents better reporters of income in Wave 2 (an example of time-in-sample bias), or is this nothing more than a classic regression to the mean? Whatever the cause or causes, the possibility that Wave 1 data behave differently from subsequent waves becomes a matter of greater concern if Wave 1 becomes the first of only 3 or 4 annual interviews rather than 1 of 12 part-year interviews. Lack of Timeliness One commonly articulated problem with SIPP data is the lag between when the data are collected and when they are released. For example, Wave 1 interviews of the 2004 SIPP panel were conducted between Febru- ary and May 2004. Data collected in the core instrument were not released until late April 2006, an interval of nearly 2 years, with a re-release of the data to correct minor errors a few months later. Wave 2 core data were not released until March 2007, or 30 months after the interviews were completed. Even by Wave 6 of the 2004 panel, the lag between collection

OCR for page 17
 REENGINEERING THE SURVEY and release remained well over 2 years. Certainly, it should be recognized that the 2004 panel incorporated several changes over the previous (2001) panel that contributed to these delays. Nevertheless, the delays associated with the 2004 panel have been the norm for SIPP panels more often than the exception. CONCLuSIONS AND RECOMMENDATIONS Conclusion 2-1: The Survey of Income and Program Participation is a unique source of information for a representative sample of household members on the intrayear dynamics of income, employment, and pro- gram eligibility and participation, together with related demographic and socioeconomic characteristics. This information remains as vital today for evaluating and improving government programs addressed to social and economic needs of the U.S. population as it did when the survey began 25 years ago. Conclusion 2-2: The Survey of Income and Program Participation’s (SIPP) history of forward movement followed by setbacks has contributed to the survey’s falling short of its original promise with regard to timeli- ness, usability, and maintenance of data quality. With the Census Bureau’s planned SIPP reengineering program, there is an opportunity to put the survey on a much firmer foundation for the future. It is essential that the Census Bureau’s program to reengineer SIPP address its problems and retain and build on its unique value and strengths. No survey can be all things to all users. In reengineering SIPP, the focus should be on improving the content and design features of the survey that make possible its unique contribution. Recommendation 2-1: To guide the design of a reengineered Survey of Income and Program Participation, the Census Bureau should consider the primary goal of the survey to be to provide data for policy analysis and research on the short-run (intrayear) dynamics of economic well-being for families and households, including employment, earnings, other income, and program eligibility and participation. Recommendation 2-2: The Census Bureau’s reengineering program for the Survey of Income and Program Participation should explicitly evaluate each proposed innovative feature, such as the use of administrative records or an event history calendar, on the extent to which a feature contributes to the survey’s ability to measure short-term changes in economic well-being with improved quality and timeliness.