Evaluating Welfare Reform A Framework and Review of Current Work, Interim Report (1999) / Chapter Skim
Currently Skimming:
2 Framework, Principles, and Designs for Evaluation
2 Framework, Principles, and Designs for Evaluation
Pages 16-49
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 16... ...
concern the outcomes of a program on recipients, such as the effects on individual employment, earnings, and family income. Process evaluations (sometimes called implementation evaluations)
|
From page 17... ...
The types of reforms that are of interest and the geographic level at which these effects are assessed are major issues in the research community. One key distinction, for example, is whether interest centers on the effect of an entire "bundle" of reforms that is, a package containing provisions for work requirements, sanctions, time limits, a particular set of support services, and other features or whether one is interested in the effects of each component separately, holding the others fixed.
|
From page 18... ...
In addition to these issues of the inherent questions of interest, there are several practical questions about the feasibility of estimating the effects of individual components, rather than the bundle. We discuss these issues when we discuss alternative evaluation designs below.
|
From page 19... ...
While these debates occur at the federal level, at the state level there is more interest in knowing the effect of a state's own specific reforms. Because evaluation, as well as operations, have shifted so heavily toward the states and away from Washington, welfare reform analysis in the current environment is much more state focused than it has previously been.
|
From page 20... ...
A before-andafter design uses roughly the same data strategy as a monitoring study, namely, the collection of data on outcomes before and after a policy change. However, in a before-and-after design the family and individual outcomes in the "after" phase are intended to be causally related to the policy.
|
From page 21... ...
Even though a monitoring study may carefully note that it has not established any cause-andeffect conclusions, the results may nevertheless be incorrectly labeled by others as demonstrating the effect of policy changes. This often occurs because many monitoring studies do not explicitly state the purpose of the study as monitoring, making the results easily interpreted as the results of a before-and-after evaluation.
|
From page 22... ...
Many welfare reform studies are even narrower in their focus, concentrating instead on the population of program participants, usually those who are receiving benefits at a particular time or at two or three different times. Such a focus comes naturally because participants are those actually receiving program benefits and services.
|
From page 23... ...
These characteristics include a recipient's level of education, work experience, physical and mental health status, history of drug abuse, past history of nonmarital childbearing, and family background and how well it has prepared the recipient for adulthood; the family, social, and community networks available to the recipient; the neighborhood environment from which the recipient comes; her exposure to others with social difficulties; and related factors. Among the types of recipients, short-termers are typically the best off, with relatively good educational and work backgrounds and a relative lack of severe health problems, and who come from better-off family and neighborhood backgrounds than other recipients.
|
From page 24... ...
Implications What are the implications of a caseload dynamics perspective for the study of welfare reform policy changes? The major implication is that, while it is easy for a study to define its population of interest as recipients at one particular time, the resulting estimates of the policy effects on that population may not generalize to any other time or any other place.
|
From page 25... ...
Second, some agencies have implemented formal diversion programs, which commonly offer a lump-sum payment or support services, such as job search support or transportation support, in exchange for not enrolling for the cash assistance program. Furthermore, some agencies are directly or indirectly sending signals to potential clients that the emphasis of welfare is now on employment and self-sufficiency and that more will be expected of them if they enroll in the cash assistance program, a sort of informal diversion program.
|
From page 26... ...
Dependence on other government programs (such as food stamps) implies that families are not 4The PRWORA policy changes targeted at specific subpopulations are too numerous to explain in detail.
|
From page 27... ...
There is also the behavior of the children themselves, such as nonmarital childbearing, drug abuse, and illegal activities; their physical health; socioemotional development, especially for young children; cognitive outcomes, such as test scores and performance on standardized scales; and attitudinal changes, mental health problems, and the like. The time frame for studying outcomes is also an important feature of any welfare reform study, and time frames often differ across studies, which can introduce noncomparability.
|
From page 28... ...
The extent to which these policy changes are successful in reducing nonmarital childbearing may require a relatively long time frame because demographic outcomes are heavily influenced by custom and social acceptability, and these may change slowly. However, it is also possible that the widespread attention and debate over welfare reform and nonmarital childbearing may have already affected childbear 1 · .
|
From page 29... ...
studies, for there is less need to collect background and retrospective information on the individuals involved in order to control for their differences; differences between the experimental and control groups have already been eliminated, at a first approximation, by the randomization. Nor is there need to collect data across a number of states and localities to ensure that sufficient programmatic variation is obtained, because programmatic variation is built into the experimental design and is thus "forced" on the environment.5 Despite these advantages, randomized trials are generally not being used in current welfare reform evaluations, except in a few areas in which pre-1996 waiver evaluations are being continued with experimental methodologies.
|
From page 30... ...
Individuals followed over time as policy changes and affects different individuals differently. Within areas: individuals subject to different requirements; across areas: individuals subject to different policy rules.
|
From page 31... ...
An example of this design would be a comparison of the earnings outcomes of program participants in a state that has a 5-year time limit, lenient sanctions, and weak work requirements, with the outcomes of program participants in another state that has a different bundle of reforms for
|
From page 32... ...
The major threat to this type of design is that not all differences across areas in the economic, social, or programmatic environments have been controlled for, and there are, therefore, alternative explanations for any differing outcomes among the study populations. A closely related threat to this design occurs if the study populations themselves are different across areas, an issue which is best understood in the context of our previous discussion of differing caseload compositions.
|
From page 33... ...
Relative to a pure cross-sectional comparison of the states over a period in which policy is not changing, this design has the advantage of permitting a comparison of the composition of the recipient populations before the policy change in order to ascertain the differences that may confound efforts to estimate the true effect; once these differences are controlled for, the isolation of the post-policy change difference across the states that results from the change itself is more credibly achieved. The major threat to this design is the danger that the changes in outcomes across areas differ for reasons other than the difference in the policy change.
|
From page 34... ...
Assuming cohorts are defined in this or some related way, one can compare series of cohorts across areas in which policies are changing differentially. Whether cohorts are defined by recipiency, eligibility, or birth year, all cohort designs face the additional threat of changes over time in the economic and programmatic environment that will affect outcomes independently of those induced by the policy change, just as in a before-and-after study.
|
From page 35... ...
Because before-and-after and cohort designs are the most common within-state evaluation methodologies, it is very difficult to estimate the effects of individual program components on data from a single state. In contrast, cross-section designs and combined crosssection and before-and-after designs, which use cross-state variation in policy to estimate welfare reform effects, are more amenable to estimating the effect of individual components because the variation in bundles across states sometimes permits an indirect assessment of individual component effects.
|
From page 36... ...
Administrative data include information gathered from welfare records of all kinds (TANF, Food Stamp Program, Medicaid, etc.) , as well as information gathered from nonwelfare sources, such as information on earnings from the records of the unemployment insurance system or information on fertility from birth records.
|
From page 37... ...
TANF records typically indicate the months of receipt by a family, a list of the persons included on the grant, the benefit paid, and various characteristics of the persons relevant to eligibility and to the grant amount, such as earned and unearned income, assets, and ages of children. With the more complex types of welfare reforms that have been implemented over the last decade, administrative data have come to include information on participation in work programs, sanctioning status, and related indicators of program treatment.
|
From page 38... ...
For example, a family that has stopped receiving cash assistance and is no longer in the welfare data system may still be receiving food stamps, Medicaid, or public housing benefits. A major issue in current welfare reform data discussions is whether administrative data from, say, non-TANF welfare programs provides adequate coverage of TANF leavers or TANF-eligible nonparticipants.
|
From page 39... ...
Consequently, information on education, occupation, marital status, and other basic characteristics is usually not available from administrative data. However, linking administrative data sets can improve the coverage of socioeconomic characteristics of individuals and families because different programs need different information about a potential recipient to judge eligibility or compliance.
|
From page 40... ...
. Unfortunately, data from administrative records are often not comparable because of variations in the definition of what a case is, what a program is, and how a case is tracked with administrative data.
|
From page 41... ...
A frequently used alternative in welfare evaluations is to gather administrative data from welfare or other programs to generate a sample of current or former welfare recipients. The major disadvantage to such list frames is their partial coverage of the population, because many families who are not receiving welfare benefits will not be included in such administrative data.
|
From page 42... ...
One of the main ways of detecting response errors is, in fact, the use of the administrative data discussed above, through cross-checking information with survey data. For example, TANF receipt information from survey data can be cross-checked with administrative records.
|
From page 43... ...
Linked administrative data can provide information on the services recipients receive while they are on welfare, such as work supports under Welfare to Work, job training, and job search services. Linked administrative data can also be used to track a recipient's or former recipient's dependency on other social welfare programs, such as public housing, food stamps, and others.
|
From page 44... ...
As we explain in Chapter 3, the Census Bureau, with support from ASPE, is looking into the feasibility of matching social security records to the SIPP and SPD data. Privacy and confidentiality are significant concerns for the development and linkage of administrative data sets and for survey data sets linked to administrative data sets.
|
From page 45... ...
Without reliable information on the programs enacted by the states and how they are changing over time, at a level of detail permitting accurate comparisons of how different states have approached the various major categories of reform policy (time limits, work requirements, sanctions, diversion, family caps, and so on) , it is unlikely that credible cross-state comparisons will be possible.
|
From page 46... ...
Process evaluations describe how program rules are operationalized and how the services are actually delivered. Implementation information is gathered by
|
From page 47... ...
Process evaluations can be used for administrative purposes, such as assessing caseworker and administrator performance, determining whether the intended policies are actually being implemented, or as an example of how services are provided in one area. Process evaluations can also be used in conjunction with outcome evaluations by linking the exposure individuals had to the program to the effects of policies on individuals.
|
From page 48... ...
AFDC was an entitlement program in which caseworkers were basically charged only with determining eligibility and benefit levels, and there were quality control measures taken to ensure that eligibility and benefit calculations were implemented consistently. Now, however, local welfare offices are increasingly becoming integrated with other social program offices so that caseworkers serve as gatekeepers to a variety of services (job training, job search, transportation benefits, and child care benefits, all in addition to cash assistance)
|
From page 49... ...
Household survey data sets, which are rare at the state level, are more plentiful at the national level but suffer from small sample sizes, a lack of key variables, and the relative unavailability of comparable policy measures across states. State-level administrative data sets, which have traditionally been used for management rather than research purposes, are still at an early stage of development and need much more work before they can fulfill their potential.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.