Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 50
Evaluating Welfare Reform: A Framework and Review of Current Work 3 ASPE Leaver Studies and Other Current Research on Welfare Reform Part of the charge to the panel is to assess the designs of 14 state-level 1 welfare ''leaver'' studies funded by the Office of the Assistant Secretary for Planning and Evaluation (ASPE) in the Department of Health and Human Services (DHHS). This chapter reports such an assessment. These studies are designed to track those who leave welfare programs, through the development of administrative data, survey data, and linkages of these two. The first part of the chapter briefly summarizes the proposed study plans. The second part discusses some conceptual and technical issues raised by the proposed studies, including issues raised during a workshop held with researchers and administrators from the states and counties. Here we also use the principles of good evaluation design reviewed in Chapter 2 to draw lessons for leaver studies. The final section of the chapter goes briefly beyond these leaver studies. A large number of leaver studies around the country have been completed already, and in the third part of this chapter we summarize some of these other leaver studies. In addition, DHHS is sponsoring other major welfare reform evaluation studies, and many other government agencies, private institutes and researchers are funding and conducting such studies as well. The fourth part of this chapter briefly reviews some of the other major studies evaluating the effects of welfare reform. 1 These 14 studies include two counties and one group of counties (see below), but, for simplicity, we often refer to all of them as state-level studies throughout the report.
OCR for page 51
Evaluating Welfare Reform: A Framework and Review of Current Work ASPE-FUNDED LEAVER STUDIES In an effort to provide support to states who are conducting studies of families who leave welfare programs, ASPE in May 1998 invited states to submit proposals to study the outcomes of at least one of three groups: those individuals and families who once participated in TANF but who stopped (leavers), those who applied for TANF but were diverted from participating, or those who appeared to be eligible for TANF but did not apply to receive it. In September 1998, 14 jurisdictions—10 states, the District of Columbia, and 3 counties or groups of counties—were awarded a total of $2.9 million to conduct studies.2 The average grant award was just over $250,000; the studies will last from 1 to 1 1/2 years. All grantees must produce interim reports in the first year of the grant period and final reports at the end of the grant period. The grantees are: Arizona, the District of Columbia, Florida, Georgia, Illinois, Massachusetts, Missouri, New York, South Carolina, Washington, Wisconsin, Cuyahoga County, Ohio (Cleveland), Los Angeles County, and a group of counties in Northern California (Santa Clara, Santa Cruz, and San Mateo counties—the San Mateo county group).3 Table A-1 in Appendix A summarizes key study components for the 14 jurisdictions. As that table shows, and as we discuss further below, most grantees plan to study the first group mentioned by ASPE—leavers. At this writing, not all state grantees' interim reports were finalized. Our summary and review of the issues in these studies are therefore based on two sources: the proposals and, when possible, the finalized interim reports submitted by the grantees, and a workshop held with the grantees and the panel in November 1998. This summary discussion of the state plans considers the outcomes proposed for study, the study populations, the proposed methodologies, and the data sources, in turn. Outcomes ASPE gave the grantees much discretion in choosing the outcomes to be studied, except that one requirement of all grantees was to study employment and earnings outcomes. Table A-2 in Appendix A summarizes the outcomes studied by the grantees and the sources of data for the outcomes. Employment outcomes 2 Additional funding for these studies was provided by the Department of Agriculture, the Department of Labor, and the Administration for Children and Families in the Department of Health and Human Services. The award to one of the states, South Carolina, was made under a previous grant announcement. 3 ASPE issued a second Request for Applications in spring 1999 but the grant recipients have not yet been announced. This request called for studies of the status of applicants and potential applicants to TANF and families entering TANF, in addition to further studies of leavers, including, perhaps, extensions of some of the 14 studies.
OCR for page 52
Evaluating Welfare Reform: A Framework and Review of Current Work that states and counties proposed to study include: employment status, hourly wage, hours worked, quarterly earnings, attachment to the labor force, types of jobs held, receipt of fringe benefits by those who are employed, and receipt of education and training. Other adult and family outcomes the studies plan to investigate (not all studies plan to measure all of these outcomes, but rather combinations of some of these outcomes) include: economic well-being (overall economic status, food security, housing security, child support receipt, other sources of support such as support from emergency shelters or support from other family members, and health insurance coverage); program participation (reason for leaving TANF, recidivism, reason for returning to TANF, and other assistance program participation such as food stamps or public housing); family structure, family formation and family functioning; barriers to self-sufficiency (lack of child care, mental health barriers, drug usage or illiteracy); and, finally, attitudes and awareness of public assistance programs. Child outcomes are also of interest to the grantees, including: availability, usage, and quality of child care; health; school attendance; behavior patterns; living arrangements; and abuse and neglect. Study Populations The major population of interest to the grantees are families who once received TANF but have stopped receiving it because their earnings were too high, they were sanctioned for not complying with requirements, they reached the end of their time limit, they voluntarily withdrew from the program, their children reached the age limit, they thought they were no longer eligible for the program, or for any other reason. This is the so-called leavers population. What it means to leave TANF varies across jurisdictions, but the majority of states count families who do not receive cash assistance for at least 2 consecutive months as leavers. For all states, to be eligible to leave TANF, cases only had to be open for 1 month. The 2-month criterion was used to avoid administrative "churning" and to avoid the inclusion of cases that were erroneously classified as having left for 1 month, which would presumably be correctly reported after 2 months. Within the group defined as leavers, further distinctions are made. First, some states have partial case sanctions, which often means that a client's benefit is reduced (usually for failing to meet work requirements). In some states these benefit reductions are small. In others, the reductions fully sanction the adult(s) in the case so that the remaining grant theoretically covers only the client's child(ren). Another distinction is made between these "partially sanctioned" child-only cases and other child-only cases, which are often cases in which the child lives with foster parents or grandparents or has immigrant parents who may not be eligible for assistance. The adults in the other child-only cases are not subject to work requirements; in some states, the adult in the partially sanctioned child-only case is still subject to work requirements and may face a more severe
OCR for page 53
Evaluating Welfare Reform: A Framework and Review of Current Work benefit reduction for continued noncompliance.4 In the 14 leaver studies, five states have proposed to count the partially sanctioned child-only cases as leavers (Georgia, Illinois, Missouri, New York, and the San Mateo County group5). Only Georgia and New York have decided to include all types of child-only closed cases (child-only for any reason, including foster care or immigrant status) as leavers. New York will follow this type of child-only case with administrative data only. The unit of observation in all of the state and county proposals is the closed case, including children and other family members who are living in the same household as the client at the time of closure. For child outcomes, some states have proposed tracking the outcomes of all children in a recipient's family, while other states plan to follow one randomly selected child per family. Several of the grantees also proposed to study families who were diverted from enrolling in the TANF program (or its predecessor, AFDC) through a formal diversion program or who were informally diverted, which some states have defined as families who begin the application process but withdraw the application before it is completed or complete the application but never enroll in the program. Washington has a formal diversion program and will be studying those who participate in it. In Jackson County (Kansas City), Missouri, those who apply for TANF and who are participating in Work First, a job placement program, will also be studied. Florida, the San Mateo County group, and South Carolina will study families who started the application process but never finished or who finished the application process but then voluntarily withdrew without receiving any cash assistance. The San Mateo County group will also study families who applied but whose applications were denied for nonmonetary reasons. Florida, South Carolina, and Washington also plan to study the outcomes of families who appear to be eligible for, but are not receiving, cash assistance. The proposed methodology to reach this population is to identify those who received food stamps or Medicaid (and so met means-tested requirements for these programs) but who did not apply to receive TANF grants. A similar effort is being undertaken in Milwaukee, Wisconsin, as part of Wisconsin's welfare leaver study grant. This part of the study, being conducted by the Institute for Research on Poverty at the University of Wisconsin-Madison, is examining three types of applicants for Wisconsin Works (W-2) in Milwaukee: families who apply and later receive assistance, families who apply for assistance and are determined to be ineligible for assistance, and families who apply and appear to be eligible for assistance but do not participate. 4 Eventually, with increasingly severe benefit reductions, some states completely close a case, while other states fully sanction only the adults in a case, but still provide benefit coverage for the child (see Gallagher et al., 1998, for state-by-state sanction rules). 5 Cuyahoga County will do some analysis of child-only cases.
OCR for page 54
Evaluating Welfare Reform: A Framework and Review of Current Work Study Methods: Definition of Cohorts and Comparison Groups The majority of states followed ASPE's suggestion for defining two separate cohorts of families who were once receiving cash assistance but have since stopped. Most of the plans define the first, pre-PRWORA, cohort as those who left cash assistance in the last quarter of 1996 or the first two quarters of 1997. The second, post-PRWORA, cohort of leavers is typically defined as those who left in a quarter as early as 1997 or as late as 1999. Although most of the studies plan to simply monitor the outcomes of these cohorts, some jurisdictions have proposed to compare outcomes of the two cohorts of welfare leavers. (See below for a discussion of the complications of such a comparison.) A couple of states have also proposed to compare the outcomes of the sample of leavers to a sample of families who stayed in the program. Several subgroup comparisons were proposed by states and counties; Table A-1 in Appendix A lists the subgroups that each grantee hopes to examine. Some states and counties are conducting qualitative analyses from data collected through focus groups or ethnographic studies. Both the Cuyahoga County, Ohio, and Los Angeles County projects are part of Manpower Demonstration Research Corporation's Project on Devolution and Urban Change, which includes ethnographic studies. The District of Columbia will be conducting focus group interviews of the first cohort of welfare leavers in its study. South Carolina is conducting "semi-structured" case-study interviews (in-depth personal stories) with 40 family heads who will also be interviewed in the survey component of its study. Data Sources All the grantees proposed to use administrative data supplemented with survey data for their studies. Most states have proposed using only administrative data to track the first cohort of welfare leavers and to use both administrative data and survey data to track the second cohort. In addition, the four studies that are collecting qualitative data through in-depth ethnographic studies or focus groups will use these data to complement the quantitative data. Administrative data will provide an inexpensive way for states to collect data on program participation, earnings, and employment outcomes. All the states proposed linking administrative data on earnings and employment to the records of closed cases. Most of them proposed using employment and earnings records from unemployment insurance (UI) reporting by employers to the states. A couple of them proposed using state Department of Revenue administrative records to track a case's employment, earnings, and income status. In order to obtain data on other well-being measures, such as income security, food security, housing security, mental health, and child support, and to track dependency of
OCR for page 55
Evaluating Welfare Reform: A Framework and Review of Current Work cash assistance leavers on other social welfare programs, states plan to link administrative records from programs in housing, education, child welfare services, Medicaid, food stamps, Supplemental Security Income, and child support. The period for which most jurisdictions plan to track leavers with administrative data varies widely. Some states have extensive data and can track individual cases as far back as 1988. Most proposed to track leavers for at least 1 year after their program participation ended. Some states proposed using administrative data for all leavers defined in a quarter, while others proposed using administrative data for only a subsample of the leavers in a quarter. In order to obtain information about outcomes not available in administrative records, all the states proposed conducting surveys of a subsample of the leavers, most often a subsample of the second cohort of leavers. Surveys will cover questions not typically available on administrative records, such as living arrangements and marital status, earnings and income support not on administrative records, fringe benefits covered by employers, health insurance coverage, use of child care, barriers to self-sufficiency, food insecurity, housing insecurity, rent and other expenses, use of emergency services, and, finally, child well-being. Not all the states plan a stratified sample, but most surveys will stratify the subsample of closed cases in a particular quarter of the year. The characteristics used for stratification are reason for leaving assistance, urban/rural, and region of state or county. Most of the states will use mixed mode surveys—telephone interviews followed by in-person interviews for those who do not respond to the telephone survey. There is a wide range in the sample sizes of the surveys: the smallest proposed sample is 350, while the largest is 15,000. All the studies hope to achieve a response rate of around 70 percent; many of the states will provide incentives for completing the survey. Three states have proposed to interview a case more than once after leaving welfare, but most proposed only one interview with each case. The proposed timing of the surveys varies: most states plan to interview the individuals between 6 months to 1 year after leaving TANF, although the time frames for interviews range from the month the client left the rolls up to almost 2 years after leaving. Each project plans to link all the administrative data sets. Most states proposed using a common identification code across all records in order to link the data (most often the client's Social Security number). A few states proposed using probabilistic matching to do the linking. The linked administrative data sets will also be linked to the survey data collected on each individual. The grants require that the data be made available for public use, so the states also have plans for how access will be provided and how confidentiality of the cases will be maintained.
OCR for page 56
Evaluating Welfare Reform: A Framework and Review of Current Work RESEARCH ISSUES FOR THE ASPE LEAVER STUDIES The 14 leaver studies funded by ASPE have many strengths, and they show considerable promise for providing new information relevant to welfare reform when their final reports are completed. The panel believes their most important strength is their contribution to the building of data infrastructure at the state level, which will provide many benefits in the future as the study of welfare reform continues. The studies will also be of generally higher quality and have more cross-state comparability than welfare leaver studies that have been completed to date. At the same time, using the general principles of welfare reform evaluations we outline in Chapter 2, there are several issues raised by the grantees' plans that the panel believes should be addressed for the studies to yield their maximum potential. Cross-Study Comparability One issue that cuts across all the individual topic areas (outcome variables, populations, etc.) is that of cross-study comparability. Such comparability is not the sole objective of the studies, for a goal of each study is an adequate description of its own leaver population and the special characteristics of that population in terms of the outcomes most important to that state. Nevertheless, cross-study comparability would assist the states in comparing their own programs to those of other states, and it would assist Congress and the administration, which will need some national-level assessment of the effect of welfare reform. In the absence of randomized trials or well-designed and credible nonexperimental analyses within each individual state, cross-state comparisons of outcomes and policies are very important. ASPE has made some attempts at cross-state comparability among the studies. However, many data barriers to cross-state comparability in the studies remain. Although a consensus was reached on the length of time off the rolls required in order to be considered a leaver (a notable achievement),6 there is still variation in defining leavers in the presence of partial sanctions, child-only cases, and transitional benefits besides cash assistance. These differences arise naturally because different states officially terminate cases at different points in the process—sometimes immediately after a sanction has been imposed, sometimes only after an appeal process has been exhausted, sometimes only after the family has had a certain length of time to come into compliance, and so on. For administrative data, there are considerable differences across states in concepts used for the definition of a case, types of cases delineated, and the universe of administrative 6 Consensus among the grantees on the length of time for which a case is considered a leaver was achieved primarily through discussion at a technical assistance workshop that ASPE held with grantees and through the ASPE-sponsored Internet-based Welfare Reform Outcomes Grantees list serve.
OCR for page 57
Evaluating Welfare Reform: A Framework and Review of Current Work data sets. For example, one state (Wisconsin) defines a welfare leaver as someone who leaves all assistance programs (both cash and noncash benefits), while other states define those who leave cash assistance only as leavers, meaning the recipients could still be receiving other benefits.7 In principle, survey data should be capable of standardization across states. Because new surveys are being undertaken in the leaver studies, comparability would be possible, but it requires attention to detail in making survey questions and constructs similar and in making accounting and tracking time periods the same. Another aspect of comparability across states (which relates to in-state issues, as well) is the need for a characterization of the recipient population in terms of the key variables discussed in Chapter 2. Documenting the distribution of cases across long-termers, cyclers, and short-termers in each state and reporting the findings on outcomes separately for the three groups would allow a crude, but useful means of standardization across states and go a long way toward eliminating differences among the compositions of caseloads. Similarly, stratification of the state samples by household type, education, health, and other fundamental determinants of well-being would assist in comparability. For example, individuals with more education or who are in better health when they leave the program may have very different outcomes than those who left with less education or bad health. Stratifying the analysis by groups defined by these types of characteristics does not need to be justified by cross-state comparability, however, but rather simply by the importance of subgroup analysis. The states have plans for some subgroup analyses (see Appendix A, Table A-1), but the groups are largely defined by programmatic status (time-limit leavers, sanctioned cases) or by race or number of children. These subgroup categories are important because time limits differ and are enforced differently across states. This is an obvious reason for variability in exit rates and the outcomes of leavers. However, characterizing leavers by these subgroup categories does not capture the basic characteristics of individual earnings capacity or the capability for self-sufficiency. As noted in Chapter 2, characterizing women by their work history, welfare history, and their history of health and related social problems should be important in predicting the likelihood of success or lack of success after leaving TANF. To do so clearly has implications for data collection and subsequent analysis. Another analytic approach that would both be informative to a state and assist cross-state comparability is a comparison of the characteristics of families who are leavers with the characteristics of families who are not leavers ("stayers"). With such an analysis included in all state reports, an assessment could be made about whether differences in the characteristics of leavers across states is a result 7 Wisconsin will present results defining a leaver in a manner the other state's do, as well as under its own definition of a welfare leaver.
OCR for page 58
Evaluating Welfare Reform: A Framework and Review of Current Work of differences in the types of families that leave the rolls or of differences in the types of families on the rolls. It would also be helpful to compare the characteristics of some of the families in the in-between categories mentioned above (partial-sanctions, child-only cases, transitional cases, etc.) to the characteristics of both stayer and leaver families. Such comparisons require that data be collected on stayers as well as leavers, but many states do not plan their analyses in that way.8 However, administrative data could be used to compare characteristics of stayers to characteristics of leavers with relatively low time and cost efforts. Monitoring Versus Evaluation A second issue raised by the state studies is the relative importance of monitoring and evaluation. The studies, as they have evolved thus far, have been primarily focused on monitoring. This is proper and the best use of resources in the short term. Setting up state data bases to be able to track recipients and nonrecipients over time, for example, is a valuable exercise in data building and data infrastructure that is needed for long-term efforts to study the effects of welfare reform. Determining the employment rates of leavers and, more generally, their well-being is also important, as is documenting the percentage of leavers who can be characterized as in distress. An understanding of particular barriers to self-sufficiency faced by particular kinds of families can be gained from the monitoring studies as well. Monitoring can also help states understand what transitional services former recipients are using, the extent to which they are using them, and what other social programs the recipients use (such as food stamps), in order to plan future programs and policies and to allocate resources. For such an assessment, it is important to know the characteristics of those leaving TANF, especially their past history of welfare receipt and employment, and their education level, since those who are relatively better off in terms of these characteristics may not need as many transitional services after they leave TANF as those who are relatively worse off. As was stressed in Chapter 2, however, a monitoring study is not capable of generating an estimate of the effect of the policy change by itself, because no comparison group or counterfactual is present. Determining the employment rates, earnings, and general well-being outcomes of leavers is of interest. However, unless it can be determined whether those outcomes are different than the outcomes that would have occurred in the absence of the policy change, it cannot be known whether the actual outcomes are a result of the policy change. The states are collecting data on a pre-PRWORA cohort implicitly for evalu- 8 More generally, it would be helpful for cross-state comparability if states describe the characteristics (education, work history, recipiency history, demographic structure, etc.) of the caseload as a whole, not just for leavers.
OCR for page 59
Evaluating Welfare Reform: A Framework and Review of Current Work ation purposes. In either their current or future reports, they will be analyzing the two cohorts and comparing their outcomes along dimensions for which data are available in both periods. Unfortunately, many outcomes (such as those obtainable only through household surveys) will generally not be available on the pre-PWRORA cohorts. When this stage of the analysis is reached, all the important issues discussed in Chapter 2 will need to be addressed: that is, the possible threats to the cohort design will have to be studied. In particular, states will have to assess whether the pre-PRWORA cohorts are similar to the post-PRWORA cohorts in their characteristics. For example, a key issue in all states will be the effect of the socioeconomic environment on outcomes because the economy improved over the period of the two cohorts in most states. This improvement will lead the outcomes for leavers in the post-PRWORA cohort to be more favorable than those in the pre-PRWORA cohort, even if there had been no policy change. To address that threat to the study designs, states will have to make an estimate of the effect of the economy on the outcomes of leavers and "net out" that effect when arriving at an estimate of the effect of their policies. Yet, as also discussed in Chapter 2, declining caseloads may alter the composition of the caseload, leaving a disproportionate number of long-termers on the rolls, with corresponding low leaving rates and worse outcomes when they leave (some of whom will be forced to leave because of the time limit provisions). This effect works in the opposite direction to the effect of the economy, because in this case it could appear that outcomes of leavers have worsened over time. However, this change in leaver outcomes would be a result entirely of the caseload composition, not an effect of the legislation. To address that threat, states will need to characterize both study cohorts in terms of their status (long-termers, cycler, and short-termers), their work histories and general job skills, and other characteristics related to potential success off the rolls. Both of these threats will also affect a comparison of the rates of reentry of leavers in the pre-PRWORA and post-PRWORA cohorts. The improvement in the economy will tend to make reentry rates lower in the post-PRWORA cohorts, while the change in the composition of the caseload will tend to make reentry rates higher in those cohorts. Both of these effects are independent of a state's policies. A major barrier to assessing comparability of the pre-PRWORA and post-PRWORA cohorts is, once again, data availability. Although almost all states are linking welfare recipiency data to other administrative data sets, few have proposed to track welfare recipiency history for more than 1 year prior to the dates for which the cohorts are defined (see Appendix A, Table A-1). Yet data on welfare recipiency history over longer periods of time are needed to assess changes in caseload composition. It might also be helpful to understand the use of other social programs (such as food stamps, Welfare to Work, Medicaid, and job training programs) by the pre-PRWORA cohort, which must necessarily rely on administrative rather than survey data. Because of the lack of survey data, it
OCR for page 60
Evaluating Welfare Reform: A Framework and Review of Current Work will be difficult to measure income, family structure, and other characteristics and outcomes for the pre-PRWORA cohort. It is not clear whether the states have data collection and analysis plans that can assess the effects of the economy on outcomes for leavers or characterize other changes in the programmatic and policy environment (changes in child care, child support, Medicaid, and other programs) that may have changed between the pre-PRWORA and post-PRWORA cohorts. In fact, if a state had a waiver policy at the time of the pre-PRWORA cohort, the programmatic environment would not be AFDC in its pre-reform structure. In such a case, it may be necessary to use time-series caseload modeling, which imposes a separate and very different set of data requirements on the analysis, or comparisons of different counties in the state with different unemployment rates. In the absence of any formal empirical analysis of the effect of the economy and other programs, states should carefully document what those changes have been, to at least allow policy makers and analysts to judge the likelihood that they changed outcomes in a particular direction. The use of a cohort comparison design implies that the relevant counterfactual is what went before PRWORA, which will generally be the AFDC program. Most states had Section 1115 waivers at the time their pre-PRWORA cohorts were drawn. These waivers had been implemented at various times, from 1993 until right before PRWORA was passed. Some waivers covered only a few counties or an experimental group in a state, while some covered an entire state except for a small control group. 9 This variation of rules in the pre-PRWORA cohort complicates any conclusions drawn from the comparison. As noted in Chapter 2, future evaluations may wish to examine as a counter-factual an incremental altering of welfare reform provisions within the general framework of a state's PRWORA program. Changes in time-limit policies that affect particular groups of recipients who are thought to be in particular need, provisions of extra services to the hard-to-serve population, testing the effects of alternative child care reimbursement policies, and other provisions might be important in the not-so-distant future. While estimating the total effect of a program bundle rightly deserves first priority, states should begin thinking of ways in which modifications and incremental reforms could be tested and evaluated as well. TANF Entry Rates A third issue raised by the state grantee studies is the need for an examination of the TANF entry rates of those who apply and, more broadly, of the entire 9 Nine states are continuing waiver plans unmodified under PRWORA. Twelve states are modifying components of their waivers (including some of the nine states with unmodified waivers that had more than one waiver).
OCR for page 68
Evaluating Welfare Reform: A Framework and Review of Current Work Study Iowa:Limited Benefit Plan Study (LBP) Indiana Maryland South Carolina 2nd Quarter 1997 Cohort South Carolina 3nd Quarter 1997 Cohort Income After Leaving Welfare 40% had increase; 49% had decline; 11% no change 43% of all leavers had income >> $1,000/month; 40% had income between $500 and $1,000 For those working, first quarter earnings after leaving averaged $2,384 66% said income was up 59% said income was up Percent Returned to Welfare Not available Not available 19% within 3 months; 23% within 12 months Not available Not available a Waiver provisions include: removal of 100-hour work cap, higher earnings and asset disregards, must participate in PROMISE JOBS employment and training program, if not, assigned to LBP. For the cases in this study period, LBP recipients received 3 months of full benefits, 3 months of limited benefits, and then no benefits for a period of 6 months, after which they are eligible to receive benefits again. b Waiver provisions for experimental group include: signing a Personal Responsibility Agreement, children must attend school and be immunized, family benefit cap, disregard of child support and earnings in first 6 months, work or training requirements, time limit of 24 months if subject to work requirements. c No further policy details provided. d TANF policy includes: 24-month time limit on economic assistance over 10 years and 60 months over lifetime, full family sanction for not meeting requirements, spouses eligible to participate in program. e Only closed cases that met the following three requirements were included in the study: (1) household received at least one check for cash assistance; (2) household received no subsequent cash assistance; (3) at least one household member was required to seek work or voluntarily sought work. f A leaver is defined as someone on welfare during the study period, but who was not on welfare at the time of the survey. A leaver may have left welfare, returned to welfare, and left again before the survey was conducted. All results are reported for those not on welfare at the time of the survey. SOURCES: Data from U.S. Department of Health and Human Services (1998); Fraker, et al. (1997); Fein (1997); Fein, et al. (1997); Maryland School of Social Work (1998); South Carolina Department of Social Services (1998).
OCR for page 69
Evaluating Welfare Reform: A Framework and Review of Current Work and short-termers and whether different caseload compositions in this respect across the states might explain some of the observed differences.15 In addition, there is no information on whether the leaving rates measured in these studies were higher or lower than at any time in the past, nor whether the improvement in the economy affected the leaving rates.16 The studies did not report the unemployment rates in the states during the period of study. However, the time periods covered by these studies range from May 1995 to as late as September 1997, and the state of the economies across these time periods and across these areas could be quite different. A final reason that the studies are difficult to compare is because the policies in effect in each state differ. Both the Iowa and Indiana studies were conducted before PRWORA and the policies in effect in both states were waiver policies. Both of these studies were part of a larger evaluation that included policy implementation studies which provided details on the policies in place during the study periods. Although these studies had some similar policy components, such as work requirements and asset and earnings disregards, each study had policy components that differed significantly, making comparisons difficult. The Iowa study focuses only on those cases that are sanctioned for not participating in an employment and training program, and so are assigned to the Limited Benefit Program, which gives the recipient a full benefit for 3 months, then a limited benefit for 3 months, and, finally, no benefit for 6 months. After these 6 months, the recipient may be eligible for benefits again. In the Indiana study, half of the sample of leavers were part of an experimental group that was subject to family caps, time limits, and work requirements, among other things, and half the sample consisted of those in a control group that were subject to the old AFDC rules. These differences make it difficult to compare and interpret outcomes from the two studies. The South Carolina and Maryland studies were conducted after TANF was implemented in each state. While the policies in these states may be similar to each other and may even be similar to the pre-PRWORA policies for the Iowa and Indiana studies, neither study documents the policies in place in sufficient detail to make such a comparison possible. The South Carolina study reported a few components of the state's TANF policies, and the Maryland study provided no detail on its policies. Because the populations studied, the economic conditions, and the policy regimes in place across the states in these studies likely differ so much, what can be learned from these studies is quite modest. 15 The Maryland study did characterize the welfare recipiency history of leavers and whether or not leavers had past work experience. However, the only outcome that was compared across these characteristics of leavers was whether or not the leaver returned to welfare. The Indiana study also compared leavers to stayers by past welfare receipt and work experience histories of recipients, but did not compare employment outcomes or any other outcomes of leavers by past welfare receipt or work history characteristics. 16 A separate component of the Indiana study did show trends in AFDC caseloads and state unemployment rates for several years before and after the study.
OCR for page 70
Evaluating Welfare Reform: A Framework and Review of Current Work There is also considerable variation in the time that the outcomes are measured compared to when the leavers stopped receiving cash assistance, ranging from 2 months to almost 2 years. Recidivism is one outcome that is difficult to assess since the number of leavers who eventually return to welfare can only increase with time. The GAO and B&L reviews both only considered studies that achieved an adequate response rate. Several of the leaver studies conducted by states were eliminated because they had such low response rates. Four of the studies in Table 3-1 use survey data (Iowa, Indiana, and the two cohorts from South Carolina). Each of these studies achieved good response rates. Both the Iowa and South Carolina studies describe nonresponse in their surveys and how they defined their response rates. These two studies also compared demographic characteristics of respondents and nonrespondents. The Iowa study showed few differences across respondent status. The South Carolina second-quarter cohort survey showed that those who left because their earned income levels were too high were more likely to respond to the survey, and the third-quarter cohort survey showed that sanctioned leavers were more likely to not respond to the survey than other types of leavers. This may be cause for concern since sanctioned leavers may be worse off than those leavers who left for other reasons, and those with higher earned incomes may be better off, meaning the survey results may not reflect the worse outcomes. Neither studies supplemented their surveys with administrative data to attempt to characterize responders and nonresponders by outcomes of interest, such as employment and wage status. The brief report of the Indiana study we used to gather information about this study did not show how the response rate of the survey was defined or if responders differed from nonresponders. The Maryland study uses only administrative data, which, as we discussed in Chapter 2, does not cover the entire population of interest, as many welfare leavers may not show up in any administrative data sets. Such an exclusion is potentially biasing because leavers who are missing may be those who have moved out of state or who are not participating in any other program in the state with an available administrative database. This study does not provide detailed information on whether any observations had missing data for the outcome measures or whether those with missing data were dropped from the study. Such reporting should be standard practice in reporting results. Although the range of employment rates in the leaver studies is between 53 and 88 percent, one could view this range as narrow instead of broad and a range that adds up to a fairly consistent story, given the differences in methods and lack of comparability across the studies in Table 3-1 and in the B&L and GAO reviews that we have noted. Moreover, subgroup analysis, which the leaver studies have mostly neglected, may show a narrower range of employment outcomes. That the range of employment rates for leavers is not wider in these studies may suggest that the differences in policies, in socioeconomic environments, and in data comparability are not that great. Or it may merely reflect a
OCR for page 71
Evaluating Welfare Reform: A Framework and Review of Current Work strong national economy with low unemployment so that economy wide effects dominate all other effects. Thus, much more progress in leaver studies is needed before one can be confident of their general conclusions. OTHER MAJOR WELFARE REFORM PROJECTS There are a large number of other welfare reform studies under way around the country. In this section we review some of the other efforts to evaluate welfare reform that are being conducted by the ASPE and other offices in DHHS, as well as the efforts of researchers in academia and at private research organizations. We first summarize several projects whose researchers addressed the panel during its initial meeting in September 1998. There are many more evaluations under way as well; a more comprehensive list is included in Table B-1 in Appendix B. Office of the Assistant Secretary for Planning and Evaluation In addition to the grants to the 14 jurisdictions to study welfare leavers, ASPE has several other research projects it is funding or conducting internally. Among the external projects, the largest study is an evaluation of the welfare-to-work grantees that the Department of Labor funded to provide services for welfare recipients and noncustodial parents of children on TANF who are the hardest to employ. ASPE has contracted with a team of researchers from the Urban Institute and Mathematica Policy Research to evaluate the net impact and cost-effectiveness of the program at up to 10 sites and to conduct a process evaluation of how the projects are being implemented at several sites. To the extent possible, net impacts will be assessed using a random assignment design. External evaluations of child and youth well-being funded by ASPE include projects on abstinence education programs, child welfare, child support enforcement, and child care policies for low-income families. ASPE and the Administration for Children and Families are jointly sponsoring a project, Measurement of the Impacts on Children in Evaluations of State Welfare Reforms, which seeks to improve states' abilities to measure child outcomes for welfare reform evaluations. ASPE is also funding several smaller studies that focus on special populations of welfare recipients, such as Native Americans, individuals with disabilities, individuals who are victims of domestic violence, and child-only welfare cases. ASPE will also be funding a second round of grants to states and large counties to examine the diversion of applicants and potential applicants for cash assistance. Applicants who eventually do not enroll because they are ineligible for nonfinancial reasons, participate in formal diversion programs, participate in job search activities prior to enrolling, or never finish the application process are
OCR for page 72
Evaluating Welfare Reform: A Framework and Review of Current Work of particular interest. ASPE will also consider funding projects that focus on individuals and families entering or leaving TANF. One major internal ASPE project is creating an historical baseline of data on welfare receipt, starting with AFDC and including TANF. Other internal projects include: analyzing administrative data from the Food Stamp Program to learn more about low-income individuals not receiving welfare, assessing the research capabilities of the New Hires Database, matching data from the Social Security Administration with national-level survey data, and working on strategies for ensuring that former welfare recipients and persons diverted from TANF are enrolled in Medicaid when they are eligible. Finally, ASPE is working with the Census Bureau in developing the Survey of Program Dynamics, which is summarized below. Administration for Children and Families The Administration for Children and Families (ACF) in DHHS also funds and conducts research on welfare reform. One of ACF's major projects is the evaluation of nine states with Track 1 welfare reform demonstration waivers, which were implemented before TANF and have not been modified or have been only slightly modified since. These studies, almost all of which use an experimental design, compare the outcomes of individuals in a control group subject to the old set of AFDC rules to the outcomes of individuals in the experimental group, who were subject to the demonstration rules, which had many TANF-like components. Five of these states were also given grants to do comprehensive and systematic measurements of family processes and child outcomes. ACF is also funding 13 Track 2, or modified state welfare reform demonstration evaluations in 12 states that modified or replaced their welfare reform demonstrations implemented prior to TANF. Some of the 13 sites use an experimental design; some of the projects include process analyses on the implementation of policies; and some of the projects include impact analyses. Another study funded by ACF is the Employment Retention and Advancement Initiative, which involves planning grants to states to examine job retention and job advancement program development and subsequent outcomes. There are two phases to this project: a planning and design phase and the provision of technical assistance through a contract with the Lewin Group. The purpose of this contract is to help states refine their program interventions and develop evaluation designs. In conjunction with ASPE, ACF is also finishing evaluations of the former JOBS program, now called the National Evaluation of Welfare-to-Work Strategies. The evaluations began in 1989 and will end in 2000. There are 7 sites and 11 different programs.
OCR for page 73
Evaluating Welfare Reform: A Framework and Review of Current Work Survey of Program Dynamics The Survey of Program Dynamics (SPD) is being conducted by the Census Bureau under a requirement of the 1996 PRWORA legislation (see Weinberg et al., 1998). The purpose of the survey is to collect longitudinal data on the demographic, social, and economic characteristics of a nationally representative sample of the U.S. population so that overall evaluations of welfare reforms can be conducted. Congress mandated that the 1992 and 1993 panels of the Survey of Income and Program Participation (SIPP) continue to be followed so that the pre-reform characteristics and well-being of families would be understood. The data from the 1992 and 1993 SIPP panels give 3 years of a longitudinal baseline before the reforms in 1996 (1992–1995 for half the sample and 1993–1995 for the other half). SPD will follow the 1992 and 1993 panels of SIPP participants over the years 1996–2001, meaning that, combined, SIPP and SPD provide 10 years of panel data. The 1997 SPD Bridge Survey attempted to interview all sample persons in the 38,000 households that completed all waves of the 1992 and 1993 SIPP panels (76% of the original sample). The Census Bureau interviewed 82 percent of the households in the Bridge Survey (about 30,000), using a modified version of the March 1997 Current Population Survey (CPS) questionnaire. A new core SPD questionnaire was developed for the 1998 survey (with the assistance of Child Trends, Inc.). This survey included a self-administered adolescent questionnaire and retrospective questions on the core topics of jobs, income, and program participation for all persons over the age of 15. The sample size for the 1998 SPD was reduced to approximately 18,500 because of budget constraints. The 1998 SPD interviewed 89 percent of eligible households. The 2001 SPD will repeat the 1998 adolescent questionnaire. A topical module on child well-being will be included in the 1999 and 2001 SPD, and a children's residential history module will be included in 2000. The 1998 SPD data were released in February 1999 as a research file. The 1998 SPD data are expected to be released by the end of summer 1999. One criticism of the SPD is the cumulative attrition rate, which, from the beginning of the 1992 and 1993 SIPP panels through the 1998 SPD is approaching 50 percent. The Census Bureau is taking steps to address this attrition problem by investigating the use of incentives to induce former SIPP respondents to return to the SPD sample to offset additional attrition expected. Plans to link Social Security Administration earnings records to SPD households to assess any effects of attrition and to look at employer–side variables have also been made. ASPE is contributing funds for the Social Security records and SPD/SIPP analysis. Urban Institute: Assessing the New Federalism Project The Assessing the New Federalism Project (ANF) of the Urban Institute has a broad focus, based on the assumption that the shift of responsibility for social
OCR for page 74
Evaluating Welfare Reform: A Framework and Review of Current Work welfare programs from the federal to the state level will affect many families in the low-income population, not just those receiving welfare (Kondratas, Weil, and Goldstein 1998). The project is not specifically designed as a welfare reform evaluation; rather, it is a study of all social support systems: health, income, housing, food, and cash assistance. The project looks at the effect of changes in the nation's social safety net on the well being of families. Intensive analysis is under way in 13 states that cover half the population of the United States. The 13 states are geographically diverse, vary in their fiscal capacities, and have different policy approaches to social and health care services. In addition, the project has established a state database with more than 900 variables covering the program rules and policies for all 50 states. The ANF project is primarily a monitoring project, although it also includes policy analysis. The primary source of individual-level data (the unit of analysis is families) is through the Urban Institute's National Survey of America's Families (NSAF). The initial survey of a cross-section of families in the 13 states, as well as in the nation as a whole, was conducted between February and September 1997. A follow-up survey of another cross-section is currently (mid-1999) being fielded. The survey interviewed over 48,000 households: about 28,000 parents with children under 18 and 20,000 households without children. Half of the sample is of households with incomes below 200 percent of the official poverty line. The survey was conducted by telephone. The survey of the first cross-section achieved a 65 percent response rate for interviews about children and a 62 percent response rate for interviews about nonelderly adults (Urban Institute, 1999). Topics covered in the survey include economic security, health and health care, child education and cognitive development, child social and positive development, child behavior problems, family environment, and community environment. The Urban Institute has made public-use data files from the survey available on-line at its Web site. Manpower Demonstration Research Corporation: Project on Devolution and Urban Change The Project on Devolution and Urban Change is a study of welfare reform in four urban counties: Philadelphia County, Cuyahoga County (Cleveland), Ohio, Dade County (Miami), Florida, and Los Angeles County (see Manpower Demonstration Research Council, 1998, for a full account of the project). The study has five components: an individual-level impact study, a neighborhood indicators study, an implementation study, an ethnographic study, and an institutional study. The individual-level study includes, for each county, a sample of households that used food stamps, AFDC/TANF, or, possibly, Medicaid at any time between 1992 and 1999 and whose head is under age 65. From this universe, cohorts of AFDC/TANF recipients and nonrecipients will be selected and tracked over time with administrative data from unemployment insurance records and survey data.
OCR for page 75
Evaluating Welfare Reform: A Framework and Review of Current Work The surveys will be conducted in person and last approximately 90 minutes. The goal is to obtain 1,000 completed surveys in each county. Two cohorts will be surveyed: single mothers between the ages of 18 and 55 who received AFDC benefits in May 1995 and single mothers who received TANF during May 1997. The two cohorts in each area will be compared in order to estimate the effects of the changes in welfare programs. The implementation component involves visiting three welfare offices in each county several times a year to study how the policies are being implemented. The ethnographic study consists of a series of in-depth interviews with 30–40 families in each of the four counties over several years. The institutional component includes interviews with nonprofit and for-profit community institutions that provide general services and emergency services for the poor. Each of these components of the studies will provide descriptive assessments of outcomes. The neighborhood indicators study will collect data on the social and economic conditions of the neighborhoods in the counties that are being studied. The characteristics of these neighborhoods will be described over time, and neighborhoods will also be compared with one another at a given time. Changes in neighborhood-level measures will be compared among poor neighborhoods and between poor and nonpoor neighborhoods. Welfare, Children, and Families: A Three-City Study The Three-City Study covers Boston, Chicago, and San Antonio and is being conducted by researchers at Harvard University, Johns Hopkins University, Northwestern University, Pennsylvania State University, and the University of Texas (Angel et al., 1998). The study has three major components. The first is a survey that plans to sample 2,800 low-income, primarily single-mother families from poor and moderate-income blocks in the central urban areas of each of the three cities beginning in 1999. Half of the families interviewed will be TANF recipients and half will not. Only families with children between birth and age 4 or between the ages of 4 and 14 will be included, reflecting a focus on the effects of welfare reform on young children and adolescents as well as adults. The longitudinal survey will interview families and collect information on adult and family well-being, employment and welfare outcomes, and child outcomes and the home environment, three times over a span of 4 years. Over the study period, information about children of almost all ages will be collected as the children grow into older age ranges. The second component is an embedded developmental study, which designates 800 families with children aged 2 to 4 for whom more detailed parent and child interaction data will be gathered. Parent and child interactions will be videotaped and coded for further analysis. This part of the study will also interview out-of-home child care providers as well as the child's father. A time diary of the child's activities will also be collected. The 800 families will be drawn
OCR for page 76
Evaluating Welfare Reform: A Framework and Review of Current Work from the 2,800 families in the main survey, and will be interviewed on the same schedule over 4 years. The third study component involves ethnographic studies of 170 families in the three cities. It will track how welfare policies affect the daily lives and neighborhood resources of poor families. In-depth interviews will be conducted over the course of 2 years and will cover such topics as the respondent's life history and daily routines of life. This component also includes diary studies and observations of the participant when she goes to a social services office for assistance. Neighborhood data and indicators will also be collected under this component of the study. In the third year of the project, a second cohort of families will be drawn from the same cities, neighborhoods, and population strata as the first cohort. Both TANF recipients and nonrecipients will again be included and the sampling frame will be identical in other respects to that used to draw the first cohort sample. The second cohort will be compared to the first to assess how the progress of welfare reform over time affects succeeding cohorts of families. Fragile Families and Child Well-Being Study The Fragile Families and Child Well-Being Study is a study of parents and newborns being conducted by a team of researchers from Columbia University and Princeton University (see McLanahan and Garfinkel, 1999). The study's focus is on unwed parents of newborns, although a sample of married parents of newborns will also be drawn. The purpose of the study is to understand the capabilities and situations of the fathers of the children, the dynamics of the relationships between the fathers and mothers, the well-being of the children in early life, and the effects of welfare and family policy changes on each of these areas. The study is drawing a cohort sample of 4,000 children born to unwed parents and 1,000 children born to married parents. These samples are being drawn from 20 cities randomly selected from all cities with populations over 200,000. Mothers of the children are interviewed in the hospital after the births. If a father is not in the hospital for a child's birth, the interviewers ask the mother how the father can be contacted. The interviews are 30 minutes long for the mother and 40 minutes long for the father. Follow-up interviews are planned for each parent every year for the next 4 years. Interviews in the first 3 years will be conducted by telephone and will last 45 minutes. The fourth-year interview will be conducted in person. Other Studies There are many other studies of welfare reform in progress around the country which we have not separately described (see Table B-1 in Appendix B).
OCR for page 77
Evaluating Welfare Reform: A Framework and Review of Current Work Many, if not most, are studies at the state level and represent some type of evaluation of a local program. Many are directly conducted by the state, while others are being conducted by national research organizations (Abt Associates, Manpower Demonstration Research Corporation, Mathematica Policy Research, Inc., etc.). Not all are focused directly on PRWORA or related waiver programs, however. Some are focused on programs now defunct (the JOBS evaluation), while others are evaluations of different types of reform programs that have not been implemented on a large scale (New Hope). Assessment The number and variety of welfare reform studies under way around the country is large. The majority of studies have welfare recipients, current or former, as their main population of interest, but a few have included nonrecipients as well (SPD, ANF-NSAF, Three-City Study, Fragile Families). Most studies use traditional household survey data for their databases, but some have significant ethnographic components in addition (New Hope, Three-City Study, Urban Change). Some studies have a strong focus on children, (ASPE Child Impacts State Studies, Three-City Study, SPD topical supplements), while others are focused on adults and their barriers to employment and self-sufficiency (the Michigan Women's Employment Study). Most are focused on individual families, but a few have neighborhoods and communities as their major unit of analysis and focus of interest (RAND Community Survey). Most studies are focused on single-parent families, but one has as its major goal the inclusion of absent fathers (Fragile Families). Almost all are focused on individual cities, counties, or states; only one is completely national in scope (SPD). With respect to the general principles for welfare reform evaluation discussed in Chapter 2, these studies must be judged to be only partially successful in fulfilling those principles. All of the studies necessarily study the effects of a single bundled set of program services; there are none which have, as a major focus, the identification of the effects of the individual provisions in the reform bundle. Most studies that have welfare recipients as their population of interest necessarily do not address issues of diversion and entry; only a few of the studies incorporate those effects. For the key issue of evaluation methodology, only a small number of evaluations have an experimental design (New Hope, the ACF studies); the rest are nonexperimental. In the latter category, only a few have explicitly outlined comparison groups (Urban Change, Three-City Study), and those studies will need to assess the threats to cohort comparison designs that we have discussed. Other studies (ANF-NSAF, Fragile Families, SPD) will have sufficient cross-state variation to permit an evaluation based on cross-sectional or cross-sectional-panel designs, and these studies will face both the problem of measuring what the different state policies are—a fundamental data problem we identified in
OCR for page 78
Evaluating Welfare Reform: A Framework and Review of Current Work Chapter 2 and above—as well as controlling for other state differences that might confound the effects of welfare policy. To a great extent, the importance of these issues will be clear only when the results of the studies are available. If the diverse populations, cities, and evaluation designs yield a reasonably coherent and consistent set of findings that add up to a credible overall picture of the consequences of welfare reform, the issues and potential difficulties we have noted will be of only modest importance. In this case, the great diversity of the studies will work to the advantage of the overall national evaluation effort by demonstrating the robustness and consistency of estimates across a wide range of approaches and across different populations for study. If, on the other hand, the studies yield results that vary greatly and in ways that are not easily explainable by obvious differences, a mixed message will result and some assessment of the relative reliability of the different approaches will have to be undertaken. This could be a formidable challenge. CONCLUSIONS The state grantee studies funded by ASPE have made a good start in what is expected to be a longer term effort to assess the effects of welfare reform policies on the low-income population. To a great extent, the grantee studies should be viewed as initial investments in building data capacity at the state level, leading to improved administrative and survey data. However, much remains to be done for these data sets to fulfill their potential. As monitoring studies, the ASPE grantee designs have significant interest and should yield interesting findings on the outcomes of welfare leavers. While the cross-state comparability in these 14 studies will be much greater than among the welfare leaver studies conducted previously, thus strengthening their findings as a whole, many improvements in such comparability need to be made. The grantee designs also have thus far not proposed to identify the most important subgroups in the welfare recipient population, namely, long-term and short-term recipients and those with strong and weak work histories. As evaluation studies, the ASPE grantee designs face considerable challenges in demonstrating the credibility of a recipient-based cohort comparison design. Finally, the grantee designs thus far concentrate almost exclusively on welfare leavers; they do not include an examination of nonapplicants and divertees or of families who are still on the rolls (stayers). The other welfare leavers studies that have been completed around the country have the same strengths and weaknesses, and the ASPE grantee studies promise to be of higher general quality. There are many other welfare reform projects under way around the country as well, both within and outside of government, and these will, when completed, yield a large number of assessments of welfare reform using different approaches and different data. Most of these projects also face significant evaluation challenges, which we have identified. Whether the set of findings that will emerge from these diverse studies will yield a consistent and coherent picture of the effects of welfare reform remains to be seen.
Representative terms from entire chapter: