The committee emphasizes that it was not charged with and therefore did not:
Evaluate the curriculum materials directly; or
Rate or rank specific curricular programs.
In addressing its charge, the committee held fast to a single commitment: that our greatest contribution would be to clarify the proper elements of an array of evaluation studies designed to judge the effectiveness of mathematics curricula and clarify what standards of evidence would need to be met to draw conclusions on effectiveness.
The committee began by systematically identifying and examining the large array of evaluation studies available on these 19 curricula. In all, 698 studies were found. The first step in our process was to eliminate studies that were clearly not evaluations of effectiveness—those lacking relevance or adequacy for the task (e.g., product descriptions, editorials) (n=281), and those classified as providing background information, historical perspective, or a project update (n=225). We then categorized the remaining (192) studies into the four major evaluation methodologies—content analyses (n=36), comparative studies (n=95), case studies (n=45), and syntheses (n=16). Criteria by which to judge methodological adequacy, specific to each methodology, were then used to decide whether studies should be retained for further examination by the committee.
Content analyses focus almost exclusively on examining the content of curriculum materials; these analyses usually rely on expert review and judgments about such things as accuracy, depth of coverage, or the logical sequencing of topics. For the 36 studies classified as content analyses, the committee drew on the perspectives of eight prominent mathematicians and mathematics educators, in addition to applying the criteria of requiring full reviews of at least one year of curricular material. All 36 studies of this type were retained for further analysis by the committee.
Comparative studies involve the selection of pertinent variables on which to compare two or more curricula and their effects on student learning over significant time periods. For the 95 comparative studies, the committee stipulated that they had to be “at least minimally methodologically adequate,” which required that a study:
Include quantifiably measurable outcomes such as test scores, responses to specified cognitive tasks of mathematical reasoning, performance evaluations, grades, and subsequent course taking; and