National Academies Press: OpenBook

Studies of Welfare Populations: Data Collection and Research Issues (2002)

Chapter: 12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process

« Previous: 11 The Right (Soft) Stuff: Qualitative Methods and the Study of Welfare Reform
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

Part IV
Welfare Leavers and Welfare Dynamics

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
This page in the original is blank.
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

12
Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process

Gregory Acs and Pamela Loprest

In August 1996, President Clinton signed the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA), making sweeping changes in the system of cash assistance for poor families and creating the Temporary Assistance for Needy Families (TANF) program. Four years after the passage of PRWORA, policy makers, practitioners, and the public continue to ask the ill-defined question, “Did welfare reform work?” Although cash assistance caseloads have dropped dramatically, from 4.4 million in August 1996 to 2.4 million in December, 1999, declining caseloads are not the sole criterion for a successful reform. Indeed, there is concern about the well-being of families who have left welfare: Are families leaving cash assistance postreform worse off than leavers prereform? Are they worse off than they were while receiving aid? To this end, many states and policy researchers, some with federal funding, have conducted and continue to conduct studies of families who have left the welfare rolls, often referred to as “leaver studies.”

Given the proliferation of these studies, this paper attempts to provide guidance for authors and consumers of leaver studies on how to best use and create these studies. Our goals are threefold:

  • To review the methods used in leaver studies;

  • To identify preferred practices for those planning to conduct a leaver study; and

  • To provide guidance to readers in assessing study results and making comparisons across studies.

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

To this end, we have examined 49 studies of welfare leavers, including 13 studies funded by the U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation (ASPE).1 They are listed in Table 12–1. Although we have made every attempt to review the body of work on families leaving welfare, these studies are by no means an exhaustive list of research in this area. Although most are explicitly studies of welfare leavers, some are studies of specific state welfare programs and reforms. We include these latter studies because they provide significant amounts of information on welfare leavers. Several of the studies present ongoing work; their findings are preliminary.

This paper is organized into three sections. First, we discuss the value of leaver studies as well as their limitations. Next we discuss what leaver studies should measure, which addresses the question of how to measure economic well-being and how some studies have done so. Finally, we examine methods for conducting a leaver study. This section describes important issues around defining leavers, positives and limitations of administrative and survey data, and how to assess the quality of data used. We hope that information in all these sections will be valuable to both future authors of leaver studies and those who are using them to understand how former welfare recipients are faring.

THE VALUE OF LEAVER STUDIES

Leaver studies can be valuable tools for monitoring the well-being of families who have been exposed to TANF and have left the rolls. Indeed, they can tell policy makers if families who have left welfare are facing problems that can be addressed by policy changes regardless of whether these problems arose as the result of past reforms. Furthermore, although leaver studies may provide only limited information about welfare reform in 1996, the ongoing capacity built by states and the research community will provide a baseline for evaluating future reforms.

Policy researchers and some policy makers also may wish to compare findings across leaver studies; after all, it is tempting to compare the status of leavers across states taking different approaches to welfare reform in order to assess the relative effectiveness of various policies. However, any such comparisons should be made with great caution for two main reasons. First, as we discuss in detail, leaver studies can have important methodological differences. These differences

1  

Throughout this report, the term “welfare leaver” refers to someone exiting the Aid to Families with Dependent Children (AFDC) or TANF programs. Note that the 13 ASPE studies cover only 11 study locations because 2 of the locations report findings from different data sources in separate reports.

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

include the time period studied, the type of data used, the exact wording and ordering of survey questions, and even the definition of a leaver. Indeed, some leaver studies focus on families leaving welfare in the early to mid-1990s while other report findings from the late 1990s. Findings may differ or differences may be obscured simply because the studies analyze different historical periods. Similarly, some studies focus on the well-being of leavers shortly after they exit welfare while others examine their status several years later.

Second, differences between states, such as in economic opportunities or even the characteristics of welfare recipients themselves, may be even more important than policy differences in accounting for differences in the status of welfare leavers. It would not be surprising to find that leavers in areas where jobs are plentiful fare better than leavers in areas with slack economies regardless of the state’s policy choices. Similarly, differences in the characteristics of state caseloads can affect the status of families leaving welfare. For example, if a state’s welfare recipients are more disadvantaged than those in another state, then its leavers may be more likely to face difficulties after exiting. Finally, if a state pursues policies aimed at encouraging work among current welfare recipients rather than encouraging exits from welfare—for example, through generous earned-income disregards—then leaver studies could miss an important impact of reform: More families are mixing welfare and work. Such families would be ignored in leaver studies because they are still on welfare.

Nevertheless, as long as one keeps in mind these limitations in leaver studies, a well-done leaver study can help policy makers understand the process families go through as they leave welfare and the factors that help them make a successful and long-term transition. Furthermore, leaver studies can help identify challenges faced by leavers and the direction for subsequent policy interventions.

WHAT LEAVER STUDIES SHOULD MEASURE

The primary role of leaver studies is to assess and track the well-being of welfare leavers; associating changes in the well-being of welfare leavers to changes in welfare policy plays a secondary role. Thus, an assessment of leaver studies requires us to address the following questions:

  • What do we mean by well-being?

  • How do we measure well-being?

When assessing a family’s overall well-being, policy makers and researchers generally consider five areas: (1) income security, (2) employment, (3) health, (4) living arrangements, and (5) quality of life or hardships. Although one can be “rich and miserable” or “poor and happy,” a family’s financial resources, especially a lack of resources, are an important indicator of well-being. Thus, leaver

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

TABLE 12–1 List of Leaver Studies by State

State

Title

General Leaver Studies

 

Arizona-1*

Arizona Cash Assistance Exit Study: First Quarter 1998 Cohort-Final Report

Arizona-2*

Arizona Cash Assistance Exit Study: Cases Exiting Fourth Quarter 1996

California-Los Angeles County*

Employment and Earnings of Single-Parent AFDC Leavers: Quarter 3 1996 Leavers: PRELIMINARY REPORT

California-San Mateo County*

Examining Circumstances of Individuals and Families who Leave TANF: Assessing the Validity of Administrative Data

District of Columbia*

The Status of TANF Leavers in the District of Columbia — Final Report

Florida

The Family Transition Program: Implementation and Three-Year Impacts of Florida’s Initial Time-Limited Welfare Program

Georgia-1

Transition from Welfare to Work: Findings for the First Year of Temporary Assistance for Needy Families

Georgia-2*

Outcomes for Single-Parent Leavers by Cohort Quarter for Jan-Mar 99: Quarterly Progress Report: PRELIMINARY REPORT

Idaho-1

Project Self-Reliance: TAFI Participant Closure Study (II)

Idaho-2

Differences Between a Surveyed Closed TAFI Case Population and Its “Unreachable” Subpopulation

Illinois-1

How are TANF Leavers Faring? Early Results from the Illinois TANF Closed Case Project

Illinois-2*

Illinois Study of Former TANF Clients: Interim Report

Indiana

The Indiana Welfare Reform Evaluation: Who is On and Who is Off? Comparing Characteristics and Outcomes for Current and Former TANF Recipients

Kentucky

From Welfare to Work: Welfare Reform in Kentucky

Maryland-1

Life After Welfare: An Interim Report

Maryland-2 Life After Welfare: Second Interim Report

Maryland-3

Life After Welfare Reform: Third Interim Report

Massachusetts

How are They Doing? A Longitudinal Study Tracking Households Leaving Welfare Under Massachusetts Reform

Mississippi

Tracking of TANF Clients: First Report of a Longitudinal Study

Missouri-1*

Preliminary Outcomes for 1996 Fourth Quarter AFDC Leavers: Revised Interim Report

Missouri-2*

Chapters 1–4: MRI Project No. 1033–1

Montana

Montana’s Welfare Reform Project: Families Achieving Independence in Montana

New Mexico

Survey of the New Mexico Case Closed AFDC Recipients

New York-1

Leaving Welfare: Findings from a Survey of Former New York City Welfare Recipients

New York-2*

After Welfare: A Study of Work and Benefit in New York State After Case Closing

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

Author(s)

Date

Data Used

Karen L.Westra and John Routley

Jan-00

Survey/ Administrative

Karen L.Westra and John Routley

Jul-99

Administrative

 

Jan-99

Administrative

Anne Moses and David Mancuso

May-99

Administrative

Gregory Acs and Pamela Loprest

Oct-99

Survey/ Administrative

Dan Bloom, Mary Farell, James J.Kemple, and Nandita Verma

Apr-99

Administrative

Georgia Department of Human Resources

Jan-98

Administrative

E.Michael Foster

 

Administrative

Idaho Department of Health and Welfare

Spring 1998

Survey

Idaho Department of Health and Welfare

Winter 1998

Survey

Steve Anderson, George Julnes, Anthony Halter, David Gruenenfelder, and Linda Brumleve

Aug-99

Survey

George Julnes and Anthony Halter

Mar 00

Survey/ Administrative

David J.Fein

Sep-97

Survey

Scott Cummings and John P.Nelson

Jan-98

Survey

University of Maryland- School of Social Work

Sep-97

Administrative

University of Maryland- School of Social Work

Mar-98

Administrative

University of Maryland- School of Social Work

Mar-98

Administrative

Massachusetts Department of Transitional Assistance

Apr-99

Survey

Jesse D.Beeler, Bill M.Brister, Sharon Chambry, and Anne L.McDonald

Jan-99

Survey/ Administrative

Sharon Ryan

Sep-99

Administrative

Midwest Research Institute

Jun-00

Survey

Montana Department of Public Health and Human Services

Feb-98

Survey

University of New Mexico-Bureau of Business and Economic Research

Sep-97

Survey

Andrew S.Bush, Swati Desai, and Lawrence M.Mead

Sep-98

Survey

Rockefeller Institute

Dec-99

Administrative

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

State

Title

North Carolina-1

Evaluation of the North Carolina Work First Program: Initial Analysis of Administrative Data

North Carolina-2

Evaluation of the North Carolina Work First Program: Status of Families Leaving Work First After Reaching the 24-Month Time Limit

Ohio-1 Cuyahoga County

Work After Welfare: Employment in the 1996 Exit Cohort, Cuyahoga County

Ohio-2 Cuyahoga County*

Employment and Return to Public Assistance Among Single, Female Headed Families Leaving AFDC in the Third Quarter, 1996, Cuyahoga County, Ohio

Oklahoma

Family Health and Well-Being In Oklahoma: An Exploratory Analysis of TANF Cases Closed and Denied October 1996-November 1997

Pennsylvania

TANF Closed-Case Telephone Survey

South Carolina-1

Former Clients of South Carolina’s New Welfare Program: Trends and Issues in Surveys to Date

South Carolina-2

Survey of Former Family Independence Program Clients: Cases Closed During April Through June 1997

South Carolina-3

Survey of Former Family Independence Program Clients: Cases Closed During July Through September 1997

Tennessee

Summary of Surveys of Welfare Recipients Employed or Sanctioned for Noncompliance

Texas

Texas Families in Transition: The Impacts of Welfare Reform Changes in Texas: Early Findings

Virginia

Fairfax Welfare Reform Evaluation Study

Washington-1

Conversations with 65 Families

Washington-2

Washington’s TANF Single-Parent Families Shortly After Welfare

Washington-3

Washington’s TANF Single-Parent Families After Welfare

Washington-4*

A Study of Washington State TANF Leavers and TANF Recipients

Washington-5*

A Study of Washington State TANF Leavers and TANF Recipients

Wisconsin-1

Post-Exit Earnings and Benefit Receipt Among Those Who Left AFDC in Wisconsin

Wisconsin-2

Employment and Earnings of Milwaukee County Single Parent AFDC Families: Establishing Benchmarks for Measuring Employment Outcomes

Wisconsin-3

Survey of Those Leaving AFDC or W-2: January to March 1998 Preliminary Report

Wyoming

A Survey of Power Recipients

Sanctioned Leavers

 

Iowa

Iowa’s Limited Benefit Plan: Summary Report

Michigan

A Study of AFDC Case Closures Due to JOBS Sanctions: April 1996 AFDC Case Closures

New Jersey

Survey of WFNJ/TANF Case Closed to Sanction

*Assistant Secretary for Planning and Evaluation (ASPE) funded study.

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

Author(s)

Date

Data Used

Maximus

May-99

Administrative

Maximus

May-99

Survey

Claudia Coulton, Marilyn Su, Neil Bania, and Edward Wang

 

Administrative

Claudia Coulton and Nandita Verma

May-99

Administrative

Lynda Williams

Sep-98

Survey

Pennsylvania Bureau of Program Evaluation

Feb-98

Survey

Donald M.Klos

 

Survey

South Carolina Department of Social Services

12-Jun-98

Survey

South Carolina Department of Social Services

9-Oct-98

Survey

Center for Manpower Studies

Mar-98

Survey

Texas Department of Human Services

Dec-98

Survey

Carole Kuhns, Danielle Hollar, and Renee Loeffler

 

Survey

City of Seattle Department of Housing and Human Services

Mar-98

Survey

Washington Department of Social and Health Services

Jul-98

Survey

Washington Department of Social and Health Services

Jan-99

Survey

Jay Ahn

Feb-00

Administrative

Debra Fogerty and Shon Kraley

Feb-00

Survey

Marcia Cancian, Robert Haveman, Thomas Kaplan, and Barbara Wolfe

Oct-98

Administrative

University of Wisconsin- Milwaukee, Employment and Training Institute

 

Administrative

Institute for Research on Poverty-University of Wisconsin

13-Jan-99

Survey

Western Management Services

May-98

Survey

Thomas M.Fraker

May-97

Survey

Laura Colville, Gerry Moore, Laura Smith, and Steve Smucker

May-97

Survey

New Jersey Division of Family Development, Bureau of Quality Control

Mar-98

Survey

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

studies should collect and present information on a family’s income.2 In addition to earned income, the studies should consider cash from friends and family, including child support payments, as well as public assistance in the form of cash and near-cash aid such as food stamps.

Because a central goal of PRWORA is to move families from welfare to work, it is also important to consider their employment situation. Employment should be measured at a point in time as well as over a period of time. For example, there can be a great deal of difference in how many leavers are working in a specific month compared to how many have worked at any point over the past year. Having both sets of data allows for broader understanding of employment among leavers.

Leaver studies also should collect data on the number of hours that leavers work and how much their jobs pay. Additional information about jobs is also beneficial, including whether their jobs have regular hours or schedules, whether adult leavers hold multiple jobs, what noncash benefits they receive, what the costs of working are (transportation, child care, job-related expenses such as work clothes or uniforms), and what skills are required for their jobs.

Health status and access to health insurance and health care also are important indicators of well-being. In addition to ascertaining the health status of adult leavers and their children, it is also important to ask whether the members of a leaver’s family have health insurance coverage and what the sources of that coverage are (public programs such as Medicaid, employer-sponsored health plans, or other sources). Although insurance is generally a good indicator of access to health care, it is also useful to directly determine if a leaver can obtain medical attention when needed.

One goal of welfare reform is to foster stable families, but the strain of balancing a job and child care may be profound on low-income single mothers. Thus, it is also important to understand if leavers’ families are breaking up, with children being sent off to live with friends or relatives. Similarly, leavers may struggle to maintain independent households, so a leaver study also should determine whether leavers are “crowding in” with friends or relatives. Alternatively, leavers may be forming stable two-adult households either through marriage or cohabitation.

It is also important to assess if leavers are facing hardships that cannot be captured by examining income alone. Thus, leaver studies also should consider whether leavers must struggle to meet their families’ nutritional needs, pay their bills, or live in substandard housing. In addition, policy makers are concerned

2  

Collecting reliable income information can be challenging. Generally the only way to obtain information on income is to ask people a detailed series of questions which was done for the March supplements to the Current Population Survey, and few leaver studies do this. In fact, only four of the studies we examine provide information on total family income (Arizona, Illinois, Missouri, and Washington).

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

about the impact of welfare reform on children. To assess child well-being, leaver studies could gather information about children’s school performance and behavioral problems, for example. Some studies also have gathered information on leaver families’ involvement in the child welfare system.

Furthermore, leaver studies can examine how a leaver’s status changes over time. This information helps to answer the question of whether a leaver’s situation is improving during the transition off welfare and whether he or she is achieving self-sufficiency. Specifically, studies should try to learn whether leavers experience earnings growth over time and whether their use of public program benefits wanes over time.

Finally, it is also useful for leaver studies to fit their findings into a broader context. For example, even if leavers report high incidences of hardships, it is important to be able to know whether they are worse off since leaving welfare than before leaving welfare. Another approach is to compare leavers’ outcomes to other groups, such as current welfare recipients or other low-income families who never received welfare, to better interpret how well they are faring.

Taken together, these five areas—income security, employment, health, living arrangements, and quality of life or hardships—can describe the well-being of TANF leavers. In addition, states should think about how to tailor their leaver studies to garner information that is of specific interest to them.

LEAVER STUDY METHODS

Defining Welfare Leavers

The first issue all leaver studies must address is, “Who is a leaver?” A leaver clearly is someone who was receiving welfare and then stopped receiving welfare, but precisely how to define this term can vary.

It is not uncommon for a welfare case to be closed for administrative reasons—for example, the adult in the unit failed to appear for a recertification meeting. Sometimes cases closed for this reason reopen within a matter of weeks. These “leavers” were neither trying to exit welfare nor were they “forced off by a formal sanction. To avoid including these “administrative closures,” studies can require that a case remains closed for a certain period of time before the case is considered to be a leaver. Many studies follow a definition that requires closure for 2 months before inclusion in the sample of leavers. Others require only 1 month. One might expect that studies using a 1-month definition would have higher returns to welfare and lower employment than those using 2-month definitions, all else equal. Interestingly, we find no clear pattern across the two definitions, (as shown in Table 12–2). This could be because all else is not equal, and there are many other differences across these studies that could affect outcomes. Only Arizona-1 actually provides outcome numbers for both definitions in the same data. Although this is only one study, it does show that first-quarter returns

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

TABLE 12–2 Leaver Population Studied

State

Definition of Leavera

All Leaversb

Continuous Leaversc

Sanctioned Leavers

Child Only Cases Excluded

Arizona-1

1 month

x

 

x

x

Arizona-2

2 months

x

 

x

California-Los Angeles Co.

2 months

x

 

California-San Mateo Co.

2 months

x

 

District of Columbia

1 month

x

 

Florida

 

x

 

Georgia-1

2 months

x

 

Georgia-2

2 months

x

 

Idaho-1

 

x

 

Idaho-2

 

x

 

Illinois-1

2 months

x

 

x

Iliinois-2

2 months

x

 

x

Indiana

 

x

 

Iowa

 

x

 

Kentucky

 

x

 

Maryland-1

 

x

x

x

 

Maryland-2

 

x

x

x

 

Maryland-3

 

x

x

x

 

Massachusetts

 

x

 

Michigan

Sanctioned for 1 year

 

x

 

Mississippi

 

x

 

Missouri-1

2 months

x

 

Missouri-2

2 months

x

 

x

Montana

 

x

 

New Jersey

 

x

 

New Mexico

 

x

 

New York-1

 

x

 

New York-2

2 months

x

 

x

North Carolina-1

1 month

x

 

North Carolina-2

 

x

 

Ohio-Cuyahoga Co. 1

2 months

x

x

 

x

Ohio-Cuyahoga Co. 2

2 months

x

x

 

x

Oklahoma

 

x

x

x

 

Pennsylvania

 

x

 

South Carolina-1

 

x

 

South Carolina-2

 

x

 

South Carolina-3

 

x

 

Tennessee

 

x

 

x

 

Texas

6 months

x

 

Virginia

 

x

 

Washington-1

 

x

 

Washington-2

 

x

 

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

State

Definition of Leavera

All Leaversb

Continuous Leaversc

Sanctioned Leavers

Child Only Cases Excluded

Washington-3

1 month

 

x

 

Washington-4

2 months

x

x

 

Washington-5

2 months

 

 

 

x

Wisconsin-1

2 months

x

x

 

Wisconsin-2

 

x

 

Wisconsin-3

6 to 9 months

 

x

 

x

Wyoming

 

x

 

NOTE: The notation x means that the study included a special focus on continuous or sanctioned leavers.

aIf a cell in the leaver definition column is blank, then the study did not specifically define the term.

bIf “all leavers” is marked, the study includes continuous leavers and sanctioned leavers. If the two subsequent categories are not marked, then the study does not include a special focus of either continuous or sanctioned leavers.

cContinuous leavers refers to individuals who did not return to cash assistance.

to welfare are higher using the 1-month definition of leaver. Employment is approximately the same.

In addition to defining the number of months a case is closed before being included as a leaver, studies must also define the period of time over which to “collect” the leaver sample. Studies usually include all who meet the leaver definition for a specific month, a quarter, or a longer period. Table 12–3 shows the specific calendar time period over which studies define their leaver sample, with results ranging up to a year. How the length of the time period chosen affects results depends on the extent to which the environment is changing. In an area where the context is rapidly changing, combining a group of leavers who left over a long time period can make results less easy to interpret. Many of the studies have chosen to define their leaver study cohort over a 3-month period.

The specific calendar time period chosen for defining the leaver sample also will likely affect results. Some of the studies examined here are based on cohorts from 1996 and others are based on cohorts from 1999. In addition to other differences across areas that make comparisons difficult, readers should keep in mind the specific time period the study is addressing.

Although most studies are interested in how all families that left welfare are faring, some studies also include information on families that remain off welfare for an extended period of time. We refer to such leavers as continuous leavers. For some studies, this is a subset of all leavers defined using a 1- or 2-month closure period. A few studies focus solely on leavers who remain off welfare for

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

TABLE 12–3 Time Period Covered by Leaver Studies

State/Study

Exit Cohort

Follow-up Period

Arizona-1

1Q98

Administrative: 1 year; Survey: 12–18 months

Arizona-2

4Q96

1 year

California-Los Angeles Co.

3Q96

1 year

California-San Mateo Co.

1997

1 year

District of Columbia

4Q97, 4Q98

Administrative data: 18 months; Survey: 1 year

Florida

*

3 years

Georgia-1

1997

1 year

Georgia-2

1Q97

1 yaer

Idaho-1

3rd and 4th Q97

6 months

Idaho-2

3rd and 4th Q97

10 months

Illinois-1

December 1997 or June 1998

4–11 months

Illinois-2

Adminstrative: 3Q97–4Q98: Survey: Dec 1998

Administrative: One year; Survey: 6–8 months

Indiana

*

n.a.

Iowa

*

n.a.

Kentucky

January–November 1997

1–11 months

Maryland-1

October 1996–September 1997

One year

Maryland-2

October 1996–September 1997

Two years

Maryland-3

October 1996–March 1998

18 months

Massachusetts

1st and 2nd Q97

3 months**

Michigan

April 1996

12 months

Mississippi

1Q98

6 months

Missouri-1

4Q96

2 years

Missouri-2

4Q98

30 months

Montana

March 1996–September 1997

1–18 months

New Jersey

February-October 1998

n.a.

New Mexico

July 1996–June 1997

n.a.

New York-1

November 1997

6 months

New York-2

1Q97

One year

North Carolina-1

September 1996

30 months

North Carolina-2

July 1998

5 months

Ohio-Cuyahoga Co. 1

1996

One year

Ohio-Cuyahoga Co. 2

3Q96

One year

Oklahoma

October 1996–November 1997

2–20 months

Pennsylvania

March 1997–January 1998

1–11 months

South Carolina-1

n.a.

n.a.

South Carolina-2

2Q97

One year

South Carolina-3

3Q97

One year

Tennessee

n.a.

n.a.

Texas

November 1997

6 months

Virginia

n.a.

n.a.

Washington-1

n.a.

n.a.

Washington-2

December 1997–March 1998

12–18 months

Washington-3

*

n.a.

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

State/Study

Exit Cohort

Follow-up Period

Washington-4

4Q97

Two years

Washington-5

October 1998

6 months

Wisconsin-1

July 1995–1996

15 months

Wisconsin-2

n.a.

n.a.

Wisconsin-3

1Q98

6–9 months

Wyoming

n.a.

n.a.

*These studies took a random sample of people who began receiving benefits when Temporary Assistance for Needy Families (TANF) was implemented in the state. At the time of the survey, these recipients may or may not have been receiving TANF benefits. These are caseload tracking studies, not leaver studies.

**This study surveyed respondents every 3 months for a year. The study includes the results of the interviews at months 3 and 12.

a more extended period of time, defining leaver as a case being closed from 6 months to a year.

Information on continuous leavers is valuable because those who return to welfare most likely have lower rates of employment, and higher participation in other programs such as the Food Stamps Program and Medicaid. For example, if we examine all leavers, we might find that the share receiving food stamps remains constant over time. But this approach might mask two countervailing trends: As time goes by, one group of leavers returns to welfare, thereby increasing food stamp participation, while another group of leavers, continuous leavers, has declining food stamp participation. Consequently, examining continuous leavers can be extremely useful. Note, however, that presenting results solely for continuous leavers (without information on returns to welfare) biases results toward positive outcomes when a significant portion of the caseload returns. Indeed, results from the studies using administrative data reveal that returns to welfare 1 year after exit range from 13 percent to 40 percent. Thus, presentation of results for all leavers and continuous leavers is preferred.

Another important subgroup to consider is families that were terminated from welfare by a sanction. Nine of the studies reviewed examine sanctioned cases (see Table 12–2). Because sanctioned leavers may behave differently or have different characteristics than nonsanctioned leavers, separation of these results can be important, especially in areas where a significant portion of a given leaver group left due to sanctions. Results for all leavers in such an area could potentially mask negative results for the subset of sanctioned leavers.

Most studies are interested in how the adults in a welfare case fare after they leave welfare; however, a growing portion of welfare cases are “child only”

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

cases. Ten of the studies we review explicitly exclude “child only” cases from their leaver studies. Because many of the outcomes examined in leaver studies involve parental employment, we suspect that most leaver studies, in fact, exclude such cases. Furthermore, when an adult leaves a welfare assistance unit but her children become a “child only” case, some studies consider that adult to be a welfare leaver while others consider the case to remain open. Finally, some studies focus exclusively on single parent cases while others combine information on one- and two-parent families. Providing information for all leavers as well as separately for one- and two-parent cases is preferred especially in locations with a high proportion of two-parent cases.

Data Used in Leaver Studies

Studies of welfare leavers rely heavily on two types of data: state administrative records and direct surveys of welfare leavers.3 Each source can provide valuable but limited information about some aspects of the well-being of welfare leavers.

Administrative Data

Twenty-one of the 49 leaver studies we review use administrative data as shown in Table 12–4. States have data systems used in administering programs, such as TANF, and these databases can be used in conducting leaver studies. Typically state welfare program data can provide information on the timing of receipt of welfare benefits, the value of the grant, the number of people (adults and children) in the case, as well as some demographic characteristics of recipients, usually race, age, number and ages of children, and whether a case is single parent or two parent. Of course, availability of TANF data is critical to conducting a leaver study because the data allow one to define who is a leaver. In addition, this information can be used to determine who among a group of leavers returns to welfare and to develop some basic characteristics for conducting subgroup analysis. One also can examine records on participation prior to the month of exit to assemble a history of receipt.4 This information also can be used to analyze subgroups based on being a long-term or short-term recipient, although none of the studies we review have carried out such a subgroup analysis.

3  

Ethnographic interviews and focus groups with welfare leavers also could provide valuable information on leavers; however, none of the studies reviewed here relied on this of type of data.

4  

In some states, the ability to assemble records of past welfare receipt may be limited because under AFDC, such information was not vital for program administration. Under TANF with its lifetime limit, it is imperative that state data systems contain lengthy historic information on receipt for each case.

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

TABLE 12–4 Studies Using Administrative Data

State

Exit Cohort

Period of Follow up After Exit

Programs Covereda

Arizona-1

1Q98

1 year

Employment, Temporary Assistance for Needy Families (TANF), Food Stamps, childcare subsidy, child support, child welfare

Arizona-2

4Q96

1 year

Employment, TANF, Food Stamps, Medicaid

California-Los Angeles Co.

3Q96

1 year

Employment

California-San Mateo Co.

1997

1 year

Employment, TANF, Food Stamps, Medicaid

District of Columbia

4Q98

18 months

TANF, Food Stamps, Medicaid

Florida

c

3 years

Employment, TANF, Food Stamps

Georgia-1

1997

1 year

Employment, TANF

Georgia-2

1Q97

1 year

Employment, TANF

Illinois-2

3Q97–4Q98

1 year

Employment, TANF, Food Stamps, Medicaid, the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC), childcare subsidy, family case management services, drug and alcohol treatment services, child support, child welfareb

Maryland-1

October 1996– September 1997

1 year

Employment, TANF

Maryland-2

October 1996– September 1997

2 years

Employment, TANF

Maryland-3

October 1996– March 1998

18 months

Employment, TANF

Mississippi

1Q98

6 months

Employment, TANF

Missouri-1

4Q96

2 years

Employment, TANF, Food Stamps, Medicaid

New York-2

1Q97

1 year

Employment, TANF, Food Stamps, Medicaid

North Carolina-1

September 1996d

30 months

Employment, TANF, Food Stamps

Ohio-Cuyahoga Co. 1

1996

1 year

TANF

Ohio-Cuyahoga Co. 2

3Q96

1 year

Employment, TANF, Food Stamps, Medicaid

Washington-4

4Q97

2 years

Employment, TANF, Food Stamps, Medicaid, childcare subsidy, child support programs, child welfareb

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

State

Exit Cohort

Period of Follow up After Exit

Programs Covereda

Wisconsin-1

July 1995–1996

15 months

Employment, TANF, Food Stamps, Medicaid

Wisconsin-2

n.a.

n.a.

Employment, TANF, Food Stamps, Medicaid

aTANF refers to cash assistance. For studies that predate the implementation of TANF, the use of the term TANF in the table indicates Aid to Families with Dependent Children (AFDC) cash assistance.

bChild abuse and neglect referrals and out-of-home placements. Women, Infants, and Children (WIC).

cThe AFDC Component exited in February 1995.

dFlorida uses a TANF Cohort instead of an exit cohort. The study chose a random sample of people who began receiving TANF benefits with the implementation of TANF. The study tracks their employment, TANF, and Food Stamp history over three years.

State program data also can include information on participation in the Food Stamp and Medicaid programs linked to TANF program data. Table 12–4 shows that the majority of study areas (9 out of 15) have a study that includes both of these sources of data. Other types of program data also may be available to be linked to TANF data. Only three of the studies listed here have made use of additional program data. Examples of the types of data they examine include childcare subsidies, receipt of child support payments, and involvement in the child welfare system. Information from such programs provides a richer description of the well-being of leavers.

By their nature, program data do not contain information on families who no longer receive program benefits. Consequently, there is no way to determine if leavers who do not return to the caseload and are not participating in other programs from which data are available are finding jobs. To address this problem, many leaver studies use additional administrative data, linking their welfare program records to data from state unemployment insurance (UI) systems. If a leaver is working for an employer that reports wages to the state UI system, then these linked records can reveal whether a leaver is working in a given quarter and how much that leaver earned. Because the employment and earnings of welfare leavers are a key outcome for policy makers and researchers, linking administrative data from the welfare system with data from the state UI system is vital. Nineteen of the 21 studies link their program data with state UI data.

Note that using administrative data to assess the status of welfare leavers often requires researchers to link information across various data systems. In

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

general, researchers use Social Security numbers to link information on welfare leavers with information from other sources such as UI earnings records. If there is a discrepancy in an individual’s Social Security number across data systems, then no match can be made. Goerge and Lee (this volume: Chapter 7) provide a detailed discussion of techniques that can be used to improve the quality of matched data between administrative data systems.

Overall, the greatest strength of administrative data is that they provide accurate information on program participation for all leavers who continue to reside in the state. Information on employment and earnings from UI records also is reliable; however, leavers who work outside the state or in jobs that do not generate UI wage reports5 will not be picked up in a state’s UI system.6 Thus, administrative data on employment probably understate employment among leavers. The greatest weakness of administrative data is their failure to provide information on many aspects of well-being and changes in family structure. Thus, they provide a limited picture of the status of TANF leavers.

Survey Data

Surveys of welfare leavers are particularly good at obtaining information that is beyond the scope of administrative data systems. For example, in addition to employment and wage information, a survey can obtain data on job characteristics—nonwage benefits, training, and work-related expenses. Surveys also can elicit information on changes in a leaver’s personal characteristics and household composition as well as what sort of hardships the leavers have faced. Furthermore, leavers can be surveyed even if they have moved across state lines. Thirty-two of the 49 leaver studies we review use data collected from surveys of welfare leavers. Features of the 32 studies are listed in Table 12–5.

Surveys of welfare leavers generally collect information on a sample of families who left TANF during a specific timeframe by interviewing them months after their exit. The choice of how long after exit to interview respondents has advantages and disadvantages. The sooner the time period is to the exit from welfare, the more able a recipient is to recall information on the circumstances around leaving, such as reason for leaving and specifics of his or her first job. The later the interview takes place from the exit, the more information about a family’s transition can be gathered. The actual range of time of the interview after exiting in these studies varies from 3 months (Massachusetts) to 30 months (Missouri).

Most studies gather survey information using telephone interviews, but many also conduct some in-person interviews. This combination method ensures that

5  

Most jobs are reported to a state’s UI system. Some exceptions include certain jobs in agriculture, self-employed workers, and household employees whose employers often fail to meet reporting requirements.

6  

Missouri is the only study to examine UI data from a neighboring state (Kansas).

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

TABLE 12–5 Leaver Studies Using Surveys

State/Study

Exit Cohort

Timing of Survey Postexit

Sample Size

Response Rate (%)

Type of Survey

Respondents Paid

Leaver Studies:

 

Arizona

1Q98

12–18 months

821

72

Phone/in person

Yes

District of Columbia

4Q98

1 year

277

61

Phone/in person

Yes

Idaho-1

3rd and 4th Q97

6 months

477

17

Mail

No

Idaho-2

3rd and 4th Q97

10 months

53

47

Mail

No

Illinois-1

December 1997 or June 1998

4–11 months

427

31

Phone

Yes

Illinois-2

December 1998

6–8 months

514

51

Phone/in person

Yes

Kentucky

January-November 1997

1–11 months

560

17

Phone

No

Massachusetts

1st and 2nd Q97

3 monthsa

341

53

In person

Yes

Michigan

July 1998

12 months

126

85

In person

No

Mississippi

1Q98

6 months

405

87

Phone/mail/in person

No

Missouri-2

4Q98

30 months

878

75

Phone/in person

Yes

Montana

March 1996–September 1997

1–18 months

208

c

Phone

No

North Carolina-2

July 1998

5 months

315

77

Phone

Yes

New Jersey

February-October 1998

n.a.

453

45

In person

No

New Mexico

July 1996–June 1997

n.a.

88

12

Mail

No

New York-1

November 1997

6 months

126

22

Phone

No

Oklahoma

October 1996–November 1997

2–20 months

292

53

Phone

Yes

Pennsylvania

March 1997–January 1998

1–11 months

169

47

Phone

No

South Carolina-1

n.a.

n.a.

2,002b

77

Phone/in person

No

South Carolina-2

2Q97

1 year

391

76

Phone/in person

No

South Carolina-3

3Q97

1 year

403

76

Phone/in person

No

Tennessee

n.a.

n.a.

2,500

51

Phone

No

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

Texas

November 1997

6 months

1,396

42

Phone/mail

No

Virginia

n.a.

n.a.

171

46

Phone

No

Washington-1

 

n.a.

65

c

In person

No

Washington-2

December 1997–March 1998

12–18 months

560

31

Phone

No

Washington-5

October 1998

6 months

987

72

Phone/in person

Yes

Wisconsin-3

1Q98

6–9 months

375

69

d

No

Wyoming

n.a.

n.a.

200

32

Phone

No

Caseload Studies:e

 

Indiana

n.a.

n.a.

847

71

Phone/in person

No

Iowa

n.a.

n.a.

162

85

In person

No

Washington-3

n.a.

n.a.

592

52

Phone

No

aThis study surveyed respondents every 3 months for a year. The study includes the results of the interviews at months 3 and 12.

bThis study is an analysis of five surveys performed in South Carolina. These five surveys have a total sample of 2002 cases. The overall response rate of these five surveys was 77 percent.

cResponse rate not reported.

dSurvey mode not described.

eThese studies took a random sample of people who began receiving benefits when Temporary Assistance for Needy Families (TANF) was implemented in the state. At the time of the survey, these recipients may or may not have been receiving TANF benefits.

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

leavers without telephones are included in the study. Three studies (two from Idaho and one from New Mexico) used mail surveys; this method is not recommended because the common problems with all surveys (described as follows) are magnified in mail surveys.

Overall, the strength of survey data is the breadth of information they contain. However, survey data have their own shortcomings. First, surveys rely on respondents to answer questions accurately and truthfully.7 Second, survey data are collected for only a sample of welfare leavers; therefore, any assessment of the well-being of leavers based on surveys is subject to sampling error. Finally, and potentially most seriously, even if the sample of leavers accurately reflects all leavers, not all sampled families will respond to the survey. That is, a researcher only will be able to contact and interview a subset of the original sample. If the leavers who respond to the survey are very different from the nonrespondents, then the survey data will suffer from nonresponse bias and not accurately represent the status of leavers. The best way to reduce nonresponse bias is to have a high response rate. A large literature is available on increasing response rates (see Cantor and Cunningham, this volume: Chapter 2; Singer and Kulka, this volume: Chapter 4, and Weiss and Bailar, this volume: Chapter 3). (See Table 12–5 for response rates in the leaver studies examined here.)

Getting the Most Out of a Leaver Study

Both administrative and survey data have their shortcomings, but combining data from these two sources provides a rich description of the overall well-being of leavers. As Table 12–1 shows, eight studies use both survey and administrative data to study the same cohort of leavers.8 In the following sections, we describe steps researchers can take to examine the accuracy of employment information from administrative data and assess the accuracy and representativeness of survey data. None of these techniques can completely address the potential shortcomings in the data, but if they are employed, they can help readers weigh the findings reported in any given leaver study.

Do UI Records Understate Employment by Welfare Leavers?

With the exception of Missouri, all leaver studies using UI wage records to examine employment only link into a single state’s UI system. Consequently, leavers that move out of state or work outside of their home state will not appear

7  

For a discussion of measurement in error in surveys of low-income populations, see Mathiowetz et al. (this volume: Chapter 6).

8  

Four states (Arizona, DC, Illinois, and Mississippi) present findings from both survey and administrative data in the same report; another four states (Missouri, North Carolina, Washington, and Wisconsin) present their findings from these two data sources in separate reports.

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

TABLE 12–6 Employment of Welfare Leavers: Comparison of Administrative and Survey Data

 

Employment Rate (%)

State/Study

Exit Cohort

Timing of Survey

Survey Data

Administrative Data*

Arizona

1Q98

12–18 months

57.0

50.0

District of Columbia

4Q98

12 months

60.3

n.a.

Illinois

December 1998

6–8 months

63.2

55.0

Missouri

4Q98

30 months

65.0

58.0

Washington

October 1998

6–8 months

59.0

57.0

*Based on employment rate from the fourth postexit quarter.

SOURCE: See Appendix B for a complete listing of the leavers studies referenced.

in the data.9 Furthermore, not all jobs are covered by state UI systems so there will be no record of work for a leaver who works in an uncovered job. If a leaver study uses both administrative and survey data and has asked surveyed leavers about their employment status, one can assess the extent of this potential underreporting.

Five jurisdictions use surveys of TANF leavers to ask the leavers themselves about their current employment status. The responses of leavers generally refer to employment about 6 months to a year after exit. Table 12–6 compares these self-reported employment rates with fourth quarter post exit employment rates computed from administrative data. The surveys consistently find higher employment rates than those reported in UI wage records; in general they are about 7 percentage points higher. The Illinois survey presents some instructive information. In its administrative records, Illinois finds that 30 percent of leavers never worked over the first four postexit quarters. In its survey, Illinois finds that only 15 percent of leavers say they have never worked since exiting TANF.

Further, a supplemental study by Wisconsin’s Department of Workforce Development (1998) examines how much employment is missed using UI wage records by comparing administrative and survey data on families leaving welfare in the first quarter of 1998. This study finds that out of the 375 surveyed leavers, 85 percent reported employment information consistent with administrative

9  

It may be possible to obtain employment and earnings information on leavers who work “out of state” by matching program data to UI data from neighboring states, but this may be too costly and time consuming for the expected benefit. Alternatively, several researchers and states have contemplated using data on the National Directory of New Hires maintained by the Office of Child Support Enforcement (OCSE) of the U.S. Department of Health and Human Services. This database contains information on the employment and earnings of all newly hired workers in the United States. To this date, however, OCSE has not allowed anyone to use these data for research purposes.

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

records. Among the leavers who reported that they had worked in the survey but did not show up in Wisconsin’s UI data, 38 percent claimed to be working in temporary jobs that may not be reported to the UI system. Another 32 percent worked as housekeepers, childcare workers, farmhands, or in other jobs in which they may be considered self-employed and/or for which employers may not file UI reports. Ten percent explicitly stated they were self-employed and 17 percent had left the state.

Are Respondents Answering Survey Questions Accurately?

Survey data are based on self-reported information from respondents. If respondents intentionally or unwittingly provide inaccurate information, the survey findings may not reflect the well-being of leavers. When surveys gather information that duplicates information available through administrative sources, it is possible to compare a respondent’s answer to the administrative report to assess accuracy. For example, a survey may ask, “In the year since you exited welfare, have you ever received food stamps?” Because this information is reported in administrative data, it is possible to see if survey respondents are providing reliable information. In general, studies that compare survey and administrative findings on common areas find fairly close agreement, as shown in Table 12–7 . Finding similar results using survey and administrative data does not guarantee that all other survey responses are accurate; however, if the findings were different, it would undermine the confidence one would have in the survey results.

Of course, the real value of surveys is their ability to obtain information unavailable in administrative records, and for such items it is not possible to obtain external validation. This can be particularly challenging when trying to determine whether a leaver is better off since exit than before. For example, a welfare leaver interviewed 9 months after exit may not recall the trouble he or she had paying the rent prior to leaving welfare. One way to examine the importance of recall problems is to supplement a leaver study with a survey of families still on welfare. The Washington state study is the only study we review that conducts a “stayer” analysis. Surprisingly, while other surveys (Arizona and Illinois) that ask about food security find that leavers generally report the same or lower levels of insecurity prior to exit than after exiting, Washington finds that current recipients actually report higher rates of food insecurity than leavers.

How Representative Are Survey Respondents of Leavers in General?

As we discussed, nonresponse bias is a potentially significant problem for surveys of welfare leavers. Indeed, if the leavers who did not respond to the survey (either because they could not be located or because they refused to participate) are appreciably different from respondents, then survey data will

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

TABLE 12–7 Post-Temporary Assistance for Needy Families (TANF) Exit Program Participation: Comparing Administrative and Survey Data Findings

 

Point in Time

Since Exit

State

Administrative (%)

Survey (%)

Administrative (%)

Survey (%)

Welfare

 

District of Columbiaa

18.8

18.8

21.1

24.6

Illinois-2c

17.5

13.7

28.9

18.5

Missouri-1b

20.5

14.0

44.0

31.0

Washington-4c

16.0

19.0

23.4

n.a.

Food Stamps

 

District of Columbiaa

37.9

40.8

n.a.

55.2

Illinois-2c

34.2

32.9

56.0

44.1

Missouri-1b

40.1

47.0

81.0

83.0

Washington-4c

40.0

n.a.

n.a.

50.0

Medicaidd

 

Arizona-1a

36.9

39.0

71.7

n.a.

District of Columbiaa

47.5

53.8

n.a.

n.a.

Illinois-2c

47.4

46.9

68.8

n.a.

Missouri-1b

n.a.

33.0

n.a.

n.a

Washington-4c

39.6

53.3

n.a.

n.a.

aThe periods of follow-up for Arizona and the District of Columbia’s survey data are 12–18 months and 12 months, respectively. The administrative data are reported for the fourth quarter after exit.

bThe period of follow-up for Missouri’s survey is 30 months. However, only 12 months of administrative data are available. The administrative data reported are for the fourth quarter after exit.

cThe period of follow-up for Illinois’s and Washington’s survey data is 6–8 months. The administrative data reported are for the third quarter after exit.

dData reported for adults.

paint a misleading picture of the well-being of TANF leavers. In general, the higher the response rate to a survey, the less concerned one is about its representativeness. (Table 12–4 shows response rates.)

Differences in response rates can affect outcomes for welfare leavers as measured by surveys. We report these results separately for surveys with high, moderate, and low response rates. In general, we would expect respondents to lead more stable lives than nonrespondents and to be more eager to share good news with survey takers. To the extent that nonresponse bias is a problem in these surveys, we would expect surveys with lower response rates to generally show that welfare leavers are better off. Note, however, that even in a survey with a 75-percent response rate, the nonresponse bias may be profound.

Table 12–8 shows employment and earnings information from survey data by response rate. Out of the nine surveys with high response rates, seven report

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

TABLE 12–8 Employment Earnings of Employed Welfare Leavers: Survey Data Findings by Survey Response Rate

State

Hours Worked

Earnings

Panel A: Response Rate Greater Than 70%

Arizona-1

#

Average wage: $7.52

Indiana

61% worked 35 or more hours a week

40.7% earned $7 or more an hour

Michigan

#

53.2% earned $400 or more a month

Mississippi

Average number of hours worked: 35

Average wage: $5.77

Missouri-2

Average number of hours worked: 39

##

North Carolina

37.9% worked 40 or more hours

Median monthly salary: $849.76

South Carolina-2

Average number of hours worked: 36

Average wage: $6.44

South Carolina-3

Average number of hours worked: 36

Average wage: $6.45

Washington-5

Average number of hours worked: 36

Average wage: $7.70

Panel B: Response Rate Between 50% and 70%

District of Columbia

Average number of hours worked: 36

Average wage: $8.74

Illinois-2

Median number of hours worked: 37

Median wage: $7.42

Massachusetts

#

63.3% income $250 or more a weeka

Oklahoma

Average number of hours worked: 34

Average wage: $6.15

Tennessee

35% worked full time

Average wage: $5.67

Washington-3

Average number of hours worked: 36

Average wage: $8.09

Wisconsin-3

57% worked 40 or more hours a week

Average wage: $7.42

Panel C: Response Rate Less Than 50%

Idaho-1

40% worked 30 or more hours a week

21% earned $7 or more an hour

Illinois-1

Average number of hours worked: 35.8

Median wage: $7.11

Kentucky

73.5% worked 35 or more hours

40.9% earned $7 an hour or more

Montana

47% worked 21 or more hours

##

New Mexico

74.6% worked 30 or more hours

29% earned $7 or more an hour

New York-1

40% worked 35 or more hours

##

Pennsylvania

62% worked 30 or more hours

59% earned $6.50 or more an hour

Texas

Average numbers of hours worked: 34

Average wage: $6.28

Virginia

#

Median monthly salary: $1,160

Washington-2

Average hours worked: 34

Average wage: $8.42

Wyoming

#

83% earned $7.50 or more an hour

aAverage weekly earning for full-time work is $305.

# Hours worked not reported.

## Earnings not reported.

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

information on hours worked, with five reporting the average number of hours worked by employed leavers. These five studies find that leavers work an average of 35 to 39 hours per week. Five studies report average hourly earnings: They range from $5.77 to $7.70. Among the studies with response rates of between 50 and 70 percent, four report average or median hours worked per week, and they show that employed leavers work between 34 and 37 hours per week. Among low-response-rate studies, three report average hours, and they, too, find an average of about 35 hours per week. The range of hourly wage rates reported in low-and moderate-response-rate studies runs from a low of $5.67 in Tennessee to a high of $8.74 in the District of Columbia.

Researches use two relatively straightforward techniques to assess the extent of nonresponse bias in surveys of welfare leavers. The first technique involves using administrative data on the entire survey sample and comparing respondents to nonrespondents. The second involves using the survey data to compare the characteristics of easily located and interviewed leavers with those of leavers that were “hard to find.”10

First, consider how administrative data can help uncover potentially important non-response bias in survey data. Three studies, the District of Columbia (DC), Missouri, and South Carolina, have compared administrative information on survey respondents and nonrespondents to see if nonrespondents appear to be very different from respondents. Missouri (Dunton, 1999) finds that nonrespondents tend to have less education and lower quarterly earnings than respondents. South Carolina (Edelhoch and Martin, 1999) compares the reasons for TANF exit for survey respondents and nonrespondents and finds that respondents are significantly less likely to have their cases closed because of a sanction and significantly more likely to have their cases closed because of earned income than nonrespondents. These comparisons suggest that findings from these studies may present too sunny a picture of the status of welfare leavers. On the other hand, DC’s leaver study finds that nonrespondents are slightly younger, have younger children, and have had shorter spells of receipt than nonrespondents. Overall, however, DC finds that respondents are fairly similar to nonrespondents.

Another technique to gauge the importance and potential biases of nonresponse involves examining differences among respondents, comparing survey responses from respondents who were easy to contact and quickly agreed to be surveyed to the responses of hard-to-contact and reluctant responders.11 This

10  

One also can attempt to do an ex post facto study of nonrespondents. This is rather costly and involves painstaking efforts to locate nonrespondents to the initial survey and interviewing them. None of the studies reviewed here attempt this; however, Mathematica Policy Research is conducting such a nonrespondent study in Iowa. The organization’s goal is to locate and interview 15 nonrespondents.

11  

Groves and Wissoker (1999) use a similar approach for examining nonresponse bias in the National Survey of America’s Families.

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

approach is based on the idea that “hard to interview” cases fall on a continuum between the “easy to interview” and nonrespondents. If the hard to interview are very different from the easy to interview in ways that are important to the study, it is likely that nonrespondents are even more different, and nonresponse bias is likely to be a big problem.

Only DC explicitly uses this technique. DC finds that hard-to-interview cases are neither clearly better nor worse off than the easy-to-interview cases; rather, their experiences are more diverse. For example, easy-to-interview cases are slightly more likely to work than hard-to-interview cases but among those who work, the hard-to-interview have higher hourly wages. In a supplementary study, Missouri (1999) compares employment and earnings among survey respondents in the Kansas City area based on the timing of response. Missouri finds that respondents among the final third of completed interviews are slightly less likely to work than respondents in the first two-thirds of completed interviews (88.5 versus 91.4 percent). The harder to interview also have lower monthly incomes ($935 versus $1,094).

Although we have described several techniques researchers can use to assess the potential for nonresponse bias in leaver studies, the best way to guard against nonresponse bias is to have a high response rate. Even though these techniques cannot rule out the possibility of significant nonresponse bias, they do provide readers with a sense of the potential size and direction of the bias. Interestingly, however, we find that surveys with moderate response rates (50 to 70 percent) report findings that are fairly similar to those with higher response rates (more than 70 percent).

CONCLUSION

Leaver studies are useful tools for monitoring the well-being of families that have been exposed to TANF and have left the rolls. They can help policy makers identify the problems that families who have left welfare are facing, and the ongoing capacity built by states and the research community will provide a baseline for formulating and evaluating future reforms.

This paper examines the methodologies used in a large set of leaver studies, identifies preferred practices for conducting such studies, and discusses the implications of research methods for the interpretations of the findings reported in these studies.

Leaver studies rely on two types of data: (1) linked administrative records from welfare programs, other low-income assistance programs, and state unemployment insurance systems, and (2) survey data. The quality of the information garnered from administrative data depends on how well the data systems are linked as well as the coverage of these systems. In general, leaver studies do not describe the methods they used to link data from multiple sources. Furthermore, although the employment of former welfare recipients is an important outcome,

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

this information comes from state UI records. Even with a perfect match to welfare program data, state UI records will understate the level of employment of welfare leavers because a nontrivial portion of jobs are not reported to the state’s UI system (jobs out of state, self-employment, as well as some domestic and agricultural work).

Surveys of leavers provide a broader set of information than administrative data on the well-being of families that have left welfare. However, the quality of survey data depend on the accuracy of the information garnered from respondents and the representativeness of the completed survey sample. Indeed, it is reasonable to expect that leavers who can be located and who choose to respond to a survey may be better off than other leavers.

Leaver studies that examine the same cohort of leavers using both administrative and survey data present a more complete picture of the status of leavers than studies that rely on only a single source. Although both sources have their limitations, combining information from the two sources can help researchers and policy makers to better assess the findings. For example, it is useful to obtain information on employment and program participation in surveys that is also available in administrative data. The survey data can be used to assess the extent of underreporting of employment in UI wage records, while the administrative data on program participation can be used to assess if respondents are responding accurately to survey questions.

In addition, nonresponse bias is potentially an important problem in leaver studies. By using administrative data available for both survey respondents and nonrespondents, researchers can gauge the extent to which respondents differ from leavers in general. In addition, one can also obtain a sense of the extent of nonresponse bias by comparing the responses of easily interviewed cases with those of cases that were hard to locate or initially refused to respond.

Finally, states can build on these studies by repeating them for new cohorts of leavers or by following existing cohorts over time. Studying new cohorts allows comparison of whether the status of leavers is changing as policies become more fully implemented and time limits are reached. Reinterviewing or analyzing administrative data for the same cohort of leavers as time passes provides information on whether employment is becoming more stable, earnings are rising, and economic hardship is decreasing—in short, whether the well-being of leavers is improving over time.

REFERENCES

Dunton, Nancy 1999 Non-Response Analysis: Missouri Leavers Survey. Unpublished tables and presentation at the Fall 1999 Outcomes Grantee Meeting of the U.S. Department of Health and Human Services Office of the Assistant Secretary for Planning and Evaluation, Washington, DC, October 25–26.

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×

Edelhoch, Marilyn, and Linda Martin 1999 Analysis of Response Rates and Non-Response Bias in Surveys. Unpublished tables and presentation at the Fall 1999 Welfare Outcomes Grantee Meeting of the U.S. Department of Health and Human Services Office of the Assistant Secretary for Planning and Evaluation, Washington, DC, October 25–26.


Groves, Robert, and Douglas Wissoker 1999 No. 7: Early Nonresponse Studies of the 1997 National Survey of America’s Families. National Survey of America’s Families Methodology Working Paper. Washington, DC: The Urban Institute.


Wisconsin Department of Workforce Development 1998 Differences between AFDC and W-W Leavers Survey Data for January–March 1998 and Wisconsinís UI Wage Records for 1998. Department of Workforce Development MEP Folio Brief 09–99, October 19.

Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 385
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 386
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 387
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 388
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 389
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 390
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 391
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 392
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 393
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 394
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 395
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 396
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 397
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 398
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 399
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 400
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 401
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 402
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 403
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 404
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 405
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 406
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 407
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 408
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 409
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 410
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 411
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 412
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 413
Suggested Citation:"12 Studies of Welfare Leavers: Data, Methods, and Contributions to the Policy Process." National Research Council. 2002. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academies Press. doi: 10.17226/10206.
×
Page 414
Next: 13 Preexit Benefit Receipt of Employment Histories and Postexit Outcomes of Welfare Leavers »
Studies of Welfare Populations: Data Collection and Research Issues Get This Book
×
Buy Paperback | $74.00 Buy Ebook | $59.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This volume, a companion to Evaluating Welfare Reform in an Era of Transition, is a collection of papers on data collection issues for welfare and low-income populations. The papers on survey issues cover methods for designing surveys taking into account nonresponse in advance, obtaining high response rates in telephone surveys, obtaining high response rates in in-person surveys, the effects of incentive payments, methods for adjusting for missing data in surveys of low-income populations, and measurement error issues in surveys, with a special focus on recall error. The papers on administrative data cover the issues of matching and cleaning, access and confidentiality, problems in measuring employment and income, and the availability of data on children. The papers on welfare leavers and welfare dynamics cover a comparison of existing welfare leaver studies, data from the state of Wisconsin on welfare leavers, and data from the National Longitudinal Survey of Youth used to construct measures of heterogeneity in the welfare population based on the recipient's own welfare experience. A final paper discusses qualitative data.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!