Violence in Families Assessing Prevention and Treatment Programs (1998) / Chapter Skim
Currently Skimming:
3 IMPROVING EVALUATIONS
3 IMPROVING EVALUATIONS
Pages 59-92
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 59... ...
exploring new evaluation methods to assess comprehensive community initiatives. The emerging emphasis on integrated, multifaceted, community-based approaches to treatment and prevention services, in particular, presents a new dilemma in evaluating family violence interventions: comprehensive interventions are particularly difficult, if not impossible, to implement as well as study using experimental or quasi-experimental designs.
|
From page 60... ...
Nor can it be improved simply by urging that evaluation studies be introduced in the early stages of the planning and design of interventions. Specific attention is needed to the hierarchy of study designs, the developmental stages of evaluation research and interventions, the marginal role of research in service settings, and
|
From page 61... ...
. Some study designs involve multiple treatment groups who receive modified forms of the services that are the subject of the evaluation; other studies use one subject group, but sample the group at multiple times prior to and after the intervention to determine whether the measures immediately before and after the program are a continuation of earlier patterns or whether they indicate a decisive change that endures after the cessation of services; this is called a time-series design (Campbell and Stanley, 1966, and Weiss, 1972, are two comprehensive primers on the basic principles and designs of program assessment and evaluation research)
|
From page 62... ...
Over the last few decades, evaluation research has developed a general consensus about the relative strength of various study designs used to assess the effectiveness (or net effects) of interventions (Figure 3-1; see also Green and Byar, 1984)
|
From page 63... ...
Experimental Designs 7. Simple randomized controls.
|
From page 64... ...
. Similarly, a methodological review of intensive family preservation services programs excluded 36 of 46 identified program evaluations because they contained no comparison groups (Heneghan et al., 1996)
|
From page 65... ...
The Marginal Role of Research in Service Settings Most research on family violence interventions is concentrated in social service settings, in which researchers have comparatively easy access to clients and can exert greater control over the service implementation process that accom
|
From page 66... ...
It is important to reiterate that the distribution of the evaluation studies reviewed in this report does not match the history of the programs and the interventions themselves. Some interventions, such as home visitation and family preservation services, are comparatively new and employ innovative service strategies.
|
From page 67... ...
. Several program evaluations conducted over several years made it possible for the study panel to examine the pattern of evidence about the effectiveness of the needle exchange program.
|
From page 68... ...
. Similarly, a review of intensive family preservation services programs indicated that, in a major Illinois evaluation of the intervention, the 2,000 families included were distributed across six sites that administered significantly different types of programs and services (Rossi, 1992)
|
From page 69... ...
IMPROVING EVALUATION 69 TABLE 3-1 Interventions by Type of Strategy and Relevant QuasiExperimental Evaluations, 1980-1996 Intervention Quasi-Experimental Evaluations Parenting practices and family Barth et al., 1988 support services 4A-1 Barth, 1991 Brunk et al., 1987 Burch and Mohr, 1980 Egan, 1983 Gaudin et al., 1991 Hornick and Clarke, 1986 Lutzker et al., 1984 National Center on Child Abuse and Neglect, 1983a,b Reid et al., 1981 Resnick, 1985 Schinke et al., 1986 Wesch and Lutzker, 1991 Whiteman et al., 1987 School-based sexual abuse Conte et al., 1985 prevention 4A-2 Fryer et al., 1987 Harvey et al., 1988 Hazzard et al., 1991 Kleemeier et al., 1988 Kolko et al., 1989 McGrath et al., 1987 Miltenberger and Thiesse-Duffy, 1988 Peraino, 1990 Randolph and Gold, 1994 Saslawsky and Wurtele, 1986 Wolfe et al., 1986 Wurtele et al., 1986, 1991 Child protective services investigation and casework 4A-3 Intensive family preservation AuClaire and Schwartz, 1986 services 4A-4 Barton, 1994 Bergquist et al., 1993 Dennis-Small and Washburn, 1986 Feldman, 1991 Halper and Jones, 1981 Pecora et al., 1992 Schuerman et al., 1994 Schwartz et al., 1991 Szykula and Fleischman, 1985 Walton et al., 1993 Walton, 1994 Wood et al., 1988 Yuan et al., 1990 continued on next page
|
From page 70... ...
70 VIOLENCE IN FAMILIES TABLE 3-1 (Continued) Intervention Quasi-Experimental Evaluations Child placement services 4A-5 Chamberlain et al., 1992 Elmer, 1986 Runyan and Gould, 1985 Wald et al., 1988 Individualized service Clark et al., 1994 programs 4A-6 Hotaling et al., undated Jones, 1985 Shelters for battered women 4B-1 Berk et al., 1986 Peer support groups for battered women 4B-2 Advocacy services for battered Sullivan and Davidson, 1991 women 4B-3 Tan et al., 1995 Domestic violence prevention Jaffe et al., 1992 programs 4B-4 Jones, 1991 Krajewski et al., 1996 Lavoie et al., 1995 Adult protective services 4C-1 Training for caregivers 4C-2 Scogin et al., 1989 Advocacy services to prevent Filinson, 1993 elder abuse 4C-3 Mandatory reporting requirements 5A-1 Child placement by the courts 5A-2 Court-mandated treatment for Irueste-Montes and Montes, 1988 child abuse offenders 5A-3 Wolfe et al., 1980 Treatment for sexual abuse Lang et al., 1988 offenders 5A-4 Marshall and Barbaree, 1988 Criminal prosecution of child abuse offenders 5A-5 Improving child witnessing 5A-6 Evidentiary reforms 5A-7
|
From page 71... ...
IMPROVING EVALUATION 71 TABLE 3-1 (Continued) Intervention Quasi-Experimental Evaluations Procedural reforms 5A-8 Reporting requirements 5B-1 Protective orders 5B-2 Arrest procedures 5B-3 Berk et al., 1992a Dunford et al., 1990 Ford and Regoli, 1993 Hirschel and Hutchison, 1992 Pate and Hamilton, 1992 Sherman and Berk, 1984a,b Sherman et al., 1992a,b Steinman 1988, 1990 Court-mandated treatment for Chen et al., 1989 domestic violence offenders 5B-4 Dutton, 1986 Edleson and Grusznski, 1989 Edleson and Syers, 1990 Hamberger and Hastings, 1988 Harrell, 1992 Palmer et al., 1992 Tolman and Bhosley, 1989 Criminal prosecution 5B-5 Ford and Regoli, 1993 Specialized courts 5B-6 Systemic approaches 5B-7 Davis and Taylor, 1995 Gamache et al., 1988 Training for criminal justice personnel 5B-8 Reporting requirements 5C-1 Protective orders 5C-2 Education and legal counseling 5C-3 Guardians and conservators 5C-4 Arrest, prosecution, and other litigation 5C-5 Identification and screening 6A-1 Brayden et al., 1993 continued on next page
|
From page 72... ...
72 VIOLENCE IN FAMILIES TABLE 3-1 (Continued) Intervention Quasi-Experimental Evaluations Mental health services for child Culp et al., 1991 victims of physical abuse and Fantuzzo et al., 1987 neglect 6A-2 Fantuzzo et al., 1988 Kolko, 1996a,b Mental health services for child Berliner and Saunders, 1996 victims of sexual abuse 6A-3 Cohen and Mannarino, 1996 Deblinger et al., 1996 Downing et al., 1988 Oates et al., 1994 Verleur et al., 1986 Wollert, 1988 Mental health services for children Jaffe et al., 1986b who witness domestic violence 6A-4 Wagar and Rodway, 1995 Mental health services for adult Alexander et al., 1989, 1991 survivors of child abuse 6A-5 Home visitation and family Larson, 1980 support programs 6A-6 Marcenko and Spence, 1994 National Committee to Prevent Child Abuse, 1996 Olds, 1992 Olds et al., 1986, 1988, 1994, 1995 Scarr and McCartney, 1988 Domestic violence screening, McLeer et al., 1989 identification, and medical care McLeer and Anwar, 1989 responses 6B-1 Olson et al., 1996 Tilden and Shepherd, 1987 Mental health services for domestic Bergman and Brismar, 1991 violence victims 6B-2 Cox and Stoltenberg, 1991 Harris et al., 1988 O'Leary et al., 1994 Elder abuse identification and screening 6C-1 Hospital multidisciplinary teams 6C-2 Hospital-based support groups 6C-3 SOURCE: Committee on the Assessment of Family Violence Interventions, National Research Council and Institute of Medicine, 1998.
|
From page 73... ...
4. Child Carroll et al., An evaluation of school curricula 18 programs abuse 1992 child sexual abuse prevention program evaluations.
|
From page 74... ...
14. Child Heneghan et al., A review of 46 family preservation 10 studies abuse 1996 services program evaluations, including 5 randomized trials and 5 quasi experimental studies.
|
From page 75... ...
Child Pecora et al., 1992 Review of studies of family-based and 12 studies abuse intensive family preservation programs.
|
From page 76... ...
. The knowledge base also provides insight into the difficulties associated with implementing innovative service designs in various institutional settings and the range of variations associated with clients who receive family violence interventions in health care, social service, and law enforcement settings.
|
From page 77... ...
But expanding the time frame increases the possibility that the intervention may be revised or changes may occur in the subject population or the community. Lessons from Nonexperimental Studies Limitations in the calibre of study designs, the weak statistical power of the majority of studies, and inconsistencies in the reliability and validity of measures suggest that little firm knowledge can be extracted from the existing literature about program effects of family violence interventions.
|
From page 78... ...
notes that, although usual care circumstances generally involve less service than is delivered in intensive family preservation programs, families who do not receive the intensive services rarely receive no service at all. Without knowing more precisely the magnitude and quality of the differences between the treatment and control conditions, estimates of effects are difficult to interpret.
|
From page 79... ...
. Program Implementation A second point of collaboration between researchers and service providers occurs in developing knowledge about the nature and stages of implementation of the intervention being tested.
|
From page 80... ...
Science-based efforts to maintain rigorous and consistent eligibility criteria can sometimes result in an incomplete picture of the characteristics of the general client base, which are often familiar to the community of service providers. When clients who have multiple problems (such as substance abuse, chronic health or mental health disorders, family emergencies, or criminal histories)
|
From page 81... ...
. Such studies can also help identify factors in the sociopolitical context of family violence interventions that influence service design and implementation.
|
From page 82... ...
. For example, in 10 evaluation studies of intensive family preservation services reported by Heneghan et al.
|
From page 83... ...
However, the service providers and program developers often share a conceptual framework that reflects their common understanding of the origins of the problem and the reasons why a specific configuration of services should remedy it. Extracting and characterizing the theory of change that guides an intervention can be a useful tool in evaluating community-based programs (Weiss, 1995)
|
From page 84... ...
; the federally supported and optional Statewide Automated Child Welfare Information Systems (SACWIS) ; and the National Center on Child Abuse and Neglect's National Child Abuse and Neglect Data System (NCANDS)
|
From page 85... ...
For front-line service providers, evaluation research may take time and resources away from the provision of services. The limited financial resources available to many agencies have created situations in which finances directed toward evaluation seem to absorb funds that might support additional services for clients or program staff.
|
From page 86... ...
86 VIOLENCE IN FAMILIES TABLE 3-3 Outcome Measures Used in Evaluations of Family Violence Interventions Subject of Outcome Measure Type of Violence Instrument Victim Perpetrator Other Child Abuse Adolescent-Family Inventory Events X Adult/Adolescent Parenting Inventory X Chemical Measurement Package X Child Abuse Potential Inventory X Child and Family Well-Being Scales X X Child Behavior Checklist X Conflict Tactics Scales X X Coopersmith Self-Opinion Inventory X Coping Health Inventory X X Family Adaptability and Cohesion Scales X X X Family Assessment Form X X X Family Environment Scale X X Family Inventory of Life Events and Changes X X X Family Systems Outcome Measures X Home Observation X Kent Infant Development Scale X Maternal Characteristics Scale (Wright) X Minnesota Child Development Inventories X Parent Outcome Interview X Parenting Stress Index X Domestic Adult Self-Expression Scale X X Violence Conflict Tactics Scales X X Depression Scale CES-D X X Quality of Life Measure X Rosenberg Self-Esteem Scale X X Rotter Internal-External Locus of Control Scale X X Social Support Scale X Elder Abuse Anger Inventory X Brief Symptom Inventory X X Rosenberg Self-Esteem Scale X X SOURCE: Committee on the Assessment of Family Violence Interventions, National Research Council and Institute of Medicine, 1998.
|
From page 87... ...
Rather than waiting for a deadline, service providers and researchers need to support a plan for assessing the effects of interventions early in their development. Viable collaborations also involve the consumers of the service at every step of the evaluation process.
|
From page 88... ...
Safety and Ethics Safety concerns related to evaluation of family violence interventions are complex and multifaceted. Research confidentiality can conflict with legal reporting requirements, and concern for the safety of victims must be paramount.
|
From page 89... ...
The increasing prevalence of cross-problems, such as substance abuse and family violence or child abuse and domestic violence, has encouraged the use of comprehensive services to address multiple risk factors associated with a variety of social problems. This has prompted some analysts to argue that the community, rather then individual clients, is the proper unit of analysis in assessing the effectiveness of family violence interventions.
|
From page 90... ...
If all communities willing to volunteer for such an experiment have initiated such efforts, the amount of programmatic difference between communities is likely to be too small to allow for the detection of effects. Using nonrandomly assigned comparison communities engenders similar problems and adds another -- selectivity bias -- that is particularly difficult to account for using conventional statistical procedures (Hollister and Hill, 1995)
|
From page 91... ...
CONCLUSION Evaluation studies in the area of family violence are usually small in scale, likely to be underpowered, and subject to a long list of rival interpretations because of study designs that include threats to validity, such as the lack of appropriate control groups, small study samples, unreliable research measures that have not been tested across diverse social classes and ethnic and cultural groups, short follow-up periods, and inconsistencies in program content and service delivery. Limited evidence exists in this field about what works, for whom, and under what conditions.
|
From page 92... ...
The use of community measures and the role of social context, including the impact of social class and culture, deserve further analysis in evaluating family violence interventions.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.