In examining the effectiveness of curricula, student outcome measures are of critical importance. One must keep in mind that the outcomes of curricular use should be the documentation of the growth of mathematical thinking and knowledge over time. We sought to identify the primary factors that would influence the perceived effectiveness of curricula based on those measures.

Most curricula evaluators use tests as the primary tool for measuring curricular effectiveness. Commonly used tests are large-scale assessments from state accountability systems; national standardized tests such as the SAT, Iowa Test of Basic Skills (ITBS), or AP exams or the National Assessment of Educational Progress (NAEP); or international tests such as items from the Third International Mathematics and Science Study (TIMSS). Among these tests, there are different choices in terms of selecting norm-referenced measures that produce national percentile rankings or criteria-referenced measures that are also subject to scaling influences. At a more specific level, assessments are used that measure particular cognitive skills such as problem solving or computational fluency or are designed to elicit likely errors or misconceptions. At other times, evaluators report outcomes using the tests and assessments included with the curriculum’s materials, or develop their own tests to measure program components.

For evaluation studies, there can be significant problems associated with the selection and use of outcome measures. We use the term “curricular validity of measures” to refer to the use of outcome measures sensitive to the curriculum’s stated goals and objectives. We believe that effectiveness must be judged relative to curricular validity of measures as a standard of scientific rigor.¹ In contrast, local decision makers may wish to gauge how well a curricular program supports positive outcomes on measures that facilitate student progress in the system, such as state tests, college entrance exams, and future course taking. We refer to this as “curricular alignment with systemic factors.”²

One should not conflate curricular validity of measures with curricular alignment with systemic factors in evaluation studies. Additionally, whereas the use of measures demonstrating curricular validity is obligatory to determine effectiveness, curricular alignment with systemic factors may also be advised.