trated in the lower income groups in the sample. Finally, timeliness of data release and availability is a major barrier to the analysis of welfare reform, a problem particularly acute for SIPP, which has released very little post-PRWORA data at the time of this writing.

State-level surveys are in much worse conditions, primarily because states are only beginning to conduct such surveys. Most such surveys are conducted by telephone and response rates are often very low. Currently, they cannot be a major resource for the study of welfare reform.

State-level administrative data hold more promise in an era of devolution because they provide fairly large samples at the state level. But their use as research tools, rather than as management information systems, is still in its infancy. Administrative data sets are usually quite complex and usable only by those who are expert in understanding the coding of the data, which makes the data difficult to use by any wide set of evaluators or researchers. The data are sometimes not available over time, because states have had no reason to save them in the past. The lack of historical information on individuals in data sets puts limits on their use for evaluation and monitoring. Evaluation efforts are also hindered by the lack of cross-state comparability in data items and variable definitions, which makes cross-area methods of evaluation problematic with these data. An inherent limitation of administrative data is, of course, that they are only available for those who receive welfare.

Data describing state policies and programs have improved in quality and quantity in the last year. However, the development of databases with this critical information will need to continue and will need significant support from DHHS. Furthermore, the databases need to be significantly widened to cover more than TANF program rules, while at the same time maintaining historical dimensions with regularly recorded rules for all the states going back several years. Collecting information about the actual implementation of the official rules continues to be a very difficult problem.

Qualitative data in the form of ethnographic information on families is an underused source of information in program evaluation on social welfare reform. Neither administrative nor survey data can fully characterize the complexity of individual families’ lives and the way different types of families respond to welfare reform. However, researchers working with qualitative data need to continue to develop standardized protocols for collecting data and documenting how the data were collected, and they need to extend the data to cover a more representative set of areas and population groups. To date, these data have not played a major role in welfare reform evaluation despite their potential.

There is a critical need to address the data barriers hindering good evaluation and monitoring as these data barriers limit what is known about the effects of PRWORA and welfare reform. As we note in our previous two chapters, ASPE has an important role to play in addressing these barriers. ASPE has already



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement