method around feasible, valid, and reliable ways to evaluate the quality of evidence on effectiveness.

The committee proceeded in a systematic way to accumulate the array of studies on these curricula, categorizing those studies into four major methods that could shed light on the determination of effectiveness: content analyses, comparative studies, case studies, and synthesis studies. Other studies and submitted reports provided valuable information on the background or the emerging constructs for curricular implementation, but were not sufficiently relevant to our charge. Within these four categories, subcommittees again scrutinized the evaluations and identified the studies that met adequate standards for that methodology. This task required committee members to articulate those standards in the context of mathematics curricula. Each subcommittee compiled their findings, which are based on a careful review of the evaluation studies. Finally, these findings were submitted to the whole committee for review. Then, the committee as a whole drew relationships among those findings, connected those reviews to the framework, and crafted the conclusions and recommendations.

These 19 curricular projects essentially have been experiments. We owe them a careful reading on their effectiveness. Demands for evaluation may be cast as a sign of failure, but we would rather stress that this examination is a sign of the success of these programs to engage a country in a scholarly debate on the question of curricular effectiveness and the essential underlying question, What is most important for our youth to learn in their studies in mathematics? To summarily blame national decline on a set of curricula whose use has a limited market share lacks credibility. At the same time, to find out if a major investment in an approach is successful and worthwhile is a prime example of responsible policy. In experimentation, success and worthiness are two different measures of experimental value. An experiment can fail and yet be worthy. The experiments that probably should not be run are those in which it is either impossible to determine if the experiment has failed or it is ensured at the start, by design, that the experiment will succeed. The contribution of the committee is intended to help us ascertain these distinctive outcomes.

The charge to the committee was “to assess the quality of studies about the effectiveness of 13 sets of mathematics curriculum materials developed through NSF support and six sets of commercially generated curriculum materials.” Based on our activities, the final product of our work was to present “the criteria and framework for reviewing the evidence, and indicating whether the currently available data are sufficient for evaluating the efficacy of these materials.” Finally, if these data were not sufficiently