largest efforts to document state policy implementations. The State Capacity Study covers program implementation in 20 states and the Front-line Management study will cover, among other things, program implementation in 12 local areas of four states. In addition, implementation studies are being conducted in conjunction with monitoring and outcome studies in each of the four metropolitan areas of Urban Change Project, and in the 13 states of the Assessing the New Federalism Project.
However, there is no a systematic effort to conduct process evaluations of how states and local areas have implemented TANF policies, and process studies are not routinely used in conjunction with outcome evaluations. Although it would be impossible to conduct process studies for every local area, a more comprehensive effort is needed. Studies of implementations at both the state and local levels should be sponsored in as many areas as possible and across areas with a wide range of characteristics: for example, across regions of the country; urban and rural areas; areas with different approaches to TANF policies; areas with different special subpopulations such as immigrants; areas with different macroeconomic conditions; or perhaps on a random sample of sites. The panel believes that such an effort will be valuable for understanding how states and local areas have used their block grant funds and leverage for program design to implement programs, information that is now only being provided on a limited basis.
Recommendation 4.2 The panel recommends that U.S. Department of Health and Human Services sponsor process research in a number of service delivery areas to better understand how service delivery administrations have implemented new welfare programs and the benefits and services families and children are receiving under these new programs.
Despite the growing potential for process studies to provide needed information about what is happening in program implementation, methodological improvements in how process studies are conducted are needed for them to be maximally effective. Currently, there are no standard protocols for methods to conduct process studies and, as a result, they are often of uneven quality. Part of this lack of protocols stems from the nature of process studies in that they typically rely on qualitative and subjective data sources (e.g., caseworker interviews concerning problems that arise in delivering services). But too often process studies are conducted on an informal basis, do not carefully design their studies, visit only a few convenient sites or talk to only a few key administrators and as a result, are not reproducible. Carefully designed and credible process studies that use such techniques as formal fieldwork protocols for observation and data collection, repeat visitation or data collection at one site, and data collection across multiple sites are needed.