Improving America's Schools The Role of Incentives (1996) / Chapter Skim
Currently Skimming:
10 Value-Added Indicators of School Performance
10 Value-Added Indicators of School Performance
Pages 197-224
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 197... ...
Reliance on such indicators is largely the result of a growing demand to hold schools accountable for their performance, defined in terms of outcomes, such as standardized test scores, rather than inputs, such as teacher qualifications, class size, and the number of books in a school's library. Unfortunately, most schools and districts have not developed and implemented entirely suitable performance indicators.
|
From page 198... ...
The simulation results reported by Meyer indicate that changes over time in average test scores could be negatively correlated with actual changes in school performance. The purpose of this chapter is to consider the class of educational indicators referred to as value-added indicators that satisfy the third criterion discussed above.
|
From page 199... ...
It is not widely appreciated that properly constructed school performance indicators differ greatly from simple aggregate indicators such as average test scores, in part because test vendors have tended to focus attention on measuring student achievement rather than school performance. Increasingly, however, schools, states, and other groups are interested in assessing the performance of schools as well as students through standardized tests.
|
From page 200... ...
Posttestis and Pretestis represent student achievement for a given individual in second grade and first grade, respectively; StudCharis represents a set of individual and family characteristics assumed to determine growth in student achievement (and a constant term) ; εis captures the unobserved student-level determinants of achievement growth; θ and α are model parameters that must be estimated; and ηs is a schoollevel effect that must be estimated.6 The model has been structured such that the 4See Bryk and Raudenbush (1992)
|
From page 201... ...
s = {θ PretestBENCHMARK + α StudCharBENCHMARK} + ηs, (2) where PretestBENCHMARK and StudCharBENCHMARK represent the mean values of Pretest and StudChar in the benchmark year and the term in brackets is a fixed number for all schools.7 For those accustomed to interpreting test scores reported on a given scale, it may be easier to interpret the total performance indicator when it is reported as a predicted mean than as an indicator centered around zero.8 In the remainder of this chapter the benchmark predicted mean form of the indicator is used.
|
From page 202... ...
where ηs is the school effect (and total school performance indicator) for school s from the level-one equation; Externals and Internals represent all observed school-level characteristics assumed to determine growth in student achievement plus a constant term; us is the unobserved determinant of total school performance; and δ1 and δ2 are parameters that must be estimated.
|
From page 203... ...
The intrinsic performance indicator can be interpreted in more than one way, depending on the types of variables that are considered external in the model and on which individuals and institutions are responsible for determining a school's policies and inputs at a given grade level. Let us assume that the external variables are limited to community characteristics and school-level student characteristics, such as average parental income and education.
|
From page 204... ...
In the case of value-added indicators, in contrast, the possibilities for replication are essentially eliminated because the objective typically is to estimate the performance of a single school. As a result, steps need to be taken to ensure that school performance indicators meet acceptable criteria for reliability -- for example, aggregation of indicators across multiple grade levels and subject areas within a given school.
|
From page 205... ...
(θ, α, δ1, and δ2) are estimated by using appropriate statistical procedures.12 These coefficients reflect the contributions of prior achievement, individual characteristics, and school-level factors to growth in student achievement and thus can be used to adjust for the contributions of these factors to average differences across schools in student achievement growth.
|
From page 206... ...
This implies that the average test score overstates the true performance of schools that have high adjustment factors and understates the true performance of those that have low adjustment factors. The intrinsic performance indicator has a similar interpretation.
|
From page 207... ...
The data are based on a highly simplified model of growth in student achievement that contains only a single control variable -- prior achievement.17 As a result, the data can conveniently be displayed on a two-dimensional graph. For simplicity, the discussion here is limited to the total performance indicator; analysis for the intrinsic performance indicator would be similar.
|
From page 208... ...
Table 10.1 reports the total performance indicator in these two alternative formats. The total performance indicator is higher for school A than school B, even though the average test scores tend to be higher for school B
|
From page 209... ...
One might interpret the predicted levels of performance in these cases as indicative of the level that a school could reach in the long run, after it has adjusted to the particular needs of new students.19 This implies that parents should be wary of selecting schools solely on the basis of their predicted total performance if, in the past, the schools have not served students with similar individual and family characteristics. To assist school choice therefore, it is important to convey information about the composition of alternative schools.
|
From page 210... ...
Researchers equipped with more extensive data have demonstrated that parental education and income, family attitudes toward education, and other variables also are powerful determinants of student achievement. The consequence of failing to control adequately for these and other student, family, and community characteristics is that feasible real-world value-added indicators are apt to be biased, if only slightly, because they absorb differences across schools in average unmeasured student, family, and community characteristics.
|
From page 211... ...
For accountability purposes, for example, a school district might prefer to focus attention on the combined performance of all grade levels at a given school (e.g., grades K–6, 6–8, or 9–12)
|
From page 212... ...
As mentioned 23As in Figures 10.1 and 10.2, the data are based on a highly simplified model of achievement growth that contains only a single control variable -- prior achievement.
|
From page 213... ...
It is likely that comparisons across schools of average test scores primarily reflect these differences rather than genuine differences in total or intrinsic school performance. Average test scores are highly biased against schools that serve disproportionately higher numbers of academically disadvantaged students.
|
From page 214... ...
To allow educators to react to assessment results in a timely and responsible fashion, performance indicators presumably must reflect information that is current. Third, average test scores at the school, district, and state levels tend to be highly contaminated because of student mobility in and out of different schools.
|
From page 215... ...
of gradual improvement. Figure 10.4 illustrates that the average tenth-grade test score provides a totally misleading view of the effectiveness of the hypothetical academic reforms implemented in 1992.
|
From page 216... ...
. Unfortunately, the NAEP is not structured in such a way that it is possible to construct a value-added measure of school performance,25 so we compare average test scores with the simple average growth in achievement from one test period to the next for the same cohort of students.
|
From page 217... ...
As a result, it is not possible to estimate the parameters of the achievement growth model. This weakness in NAEP data could be remedied by switching to a survey design that is at least partially longitudinal, although a number of technical problems would need to be addressed.
|
From page 218... ...
The eleventh-grade data, by themselves, are fully consistent with the premise that academic reforms in the early and mid1980s generated substantial gains in student achievement. In fact, an analysis of the data based on a more appropriate indicator than average test scores suggests the opposite conclusion.
|
From page 219... ...
For example, schools sometimes create an environment that is relatively inhospitable to academically disadvantaged students, provide course offerings that predominantly address the needs of academically advantaged students, fail to work aggressively to prevent students from dropping out of high school, err on the side of referring "problem" students to alternative schools, err on the side of classifying students as special education students where the latter are exempt from statewide testing, or make it difficult for low-scoring students to participate in statewide exams. These practices are designed to improve average test scores in a school, not by improving school quality but by catering to high-scoring students while ignoring or alienating low-scoring ones.
|
From page 220... ...
Value-Added Indicators: Data Requirements In view of the problems associated with average test scores and other level indicators, is it appropriate to consider using value-added indicators as the core of school district, state, and national performance indicator/accountability systems? There are at least two reasons to be optimistic in this regard.
|
From page 221... ...
Average test scores fail to localize school performance to the classroom or grade level, aggregate information on school performance that tends to be grossly out of date, are contaminated by student mobility, and fail to distinguish the distinct value-added contribution of schools to growth in student achievement from the contributions of student, family, and community factors. Average test scores are a weak, if not counterproductive, instrument of public accountability.
|
From page 222... ...
In particular, it would be helpful to know more about the empirical differences between total and intrinsic school performance indicators. Finally, in order to improve the reliability of estimates of school performance, particularly intrinsic performance, small and medium-sized school districts should consider combining their data to create school performance indicator systems that serve multiple districts.
|
From page 223... ...
1992. "Uses and abuses of achievement test scores." Educational Measurement: Issues and Practice 11:9–15.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.