On Evaluating Curricular Effectiveness
JUDGING THE QUALITY OF K-12 MATHEMATICS EVALUATIONS
THE NATIONAL ACADEMIES PRESS
Washington, D.C. www.nap.edu
THE NATIONAL ACADEMIES PRESS
500 Fifth Street, N.W. Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study/publication was supported by Contract/Grant No. ESI-0102582 between the National Academy of Sciences and the National Science Foundation. Additional funding was provided by an award from the Presidents’ Committee of the National Academies. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
Library of Congress Cataloging-in-Publication Data
On evaluating curricular effectiveness : judging the quality of K-12 mathematics evaluations / Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials, Mathematical Sciences Education Board, Center for Education, Division of Behavioral and Social Sciences and Education.
p. cm.
Includes bibliographical references.
ISBN 0-309-09242-6 (pbk.)—ISBN 0-309-53287-6 (pdf)
1. Mathematics—Study and teaching—Evaluation. I. National Research Council (U.S.). Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials.
QA11.2.O5 2004
510'.71—dc22
2004015000
Additional copies of this report are available from the
National Academies Press,
500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu.
Printed in the United States of America.
Copyright 2004 by the National Academy of Sciences. All rights reserved.
National Research Council. (2004). On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials. Mathematical Sciences Education Board, Center for Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council.
COMMITTEE FOR A REVIEW OF THE EVALUATION DATA ON THE EFFECTIVENESS OF NSF-SUPPORTED AND COMMERCIALLY GENERATED MATHEMATICS CURRICULUM MATERIALS
JERE CONFREY (Chair),
Department of Education, Washington University in St. Louis, MO
CARLOS CASTILLO-CHAVEZ,
Department of Mathematics and Statistics, Arizona State University
DOUGLAS A. GROUWS,
Department of Learning, Teaching, and Curriculum, University of Missouri
CAROLYN MAHONEY,
Division of Academic Affairs, Elizabeth City State University, NC
DONALD SAARI,
Institute for Mathematical Behavioral Sciences, University of California, Irvine
WILLIAM SCHMIDT,
College of Education, Michigan State University
PATRICK W. THOMPSON,
Department of Teaching and Learning, Vanderbilt University, TN
WILLIAM VELEZ,
Department of Mathematics, University of Arizona
CAROLE B. LACAMPAGNE, Study Director
VICKI STOHL, Research Associate
DIONNA J. WILLIAMS, Senior Program Assistant
MATHEMATICAL SCIENCES EDUCATION BOARD 2004
JOAN LEITZEL (Chair), President Emerita,
University of New Hampshire
JERE CONFREY (Vice Chair),
Department of Education, Washington University in St. Louis, MO
THOMAS BANCHOFF,
Department of Mathematics, Brown University, RI
JAN DE LANGE,
Freudenthal Institute, Utrecht University, The Netherlands
LOUIS GOMEZ,
School of Education and Social Policy, Northwestern University, IL
DOUGLAS A. GROUWS,
Department of Learning, Teaching, and Curriculum, University of Missouri
ARTHUR JAFFE,
Department of Mathematics, Harvard University, MA
ERIC JOLLY,
Science Museum of Minnesota
JIM LEWIS,
Department of Mathematics and Statistics, University of Nebraska-Lincoln
GEORGE MCSHAN,
National School Boards Association, TX
KAREN MICHALOWICZ,
Mathematics Department, The Langley School, VA
JUDITH MUMME,
WestEd, CA
CASILDA PARDO,
Teaching and Learning Systems, Valle Vista Elementary School, NM
SUE PARSONS, Instructor,
Cerritos College, CA
MARGE PETIT,
The National Center for the Improvement of Educational Assessment, VT
DONALD SAARI,
Institute for Mathematical Behavioral Sciences, Universityof California, Irvine
RICHARD SCHEAFFER, Professor Emeritus,
University of Florida
FRANCIS SULLIVAN,
Center for Computing Sciences, MD
HUNG HSI WU,
Department of Mathematics, University of California, Berkeley
Foreword
The Mathematical Sciences Education Board (MSEB) of the National Research Council was established in 1985 to provide national leadership and guidance for policies, programs, and practices supporting the improvement of mathematics education at all levels. Curriculum materials for grades K-12 play a central role in what mathematics topics are taught in our schools, how the topics are sequenced and presented to students, what levels of understanding are expected, what skills students will develop and when. Schools, practitioners, policy makers, and the public depend on evaluations of materials undertaken during their development and implementation in making decisions about the appropriate uses of the materials. The MSEB recognized that the nature and quality of the evidence used to judge claims of success and failure are critical elements in enabling the community to make sound judgments. This report presents a synthesis of the evidence used in the evaluations of several sets of recently developed curriculum materials, provides a framework for the design of evaluation studies of curriculum materials, and gives conclusions and recommendations to guide future efforts in evaluating curriculum materials.
The report was prepared by a committee of experts who devoted their time, skills, and scholarship to this project over the past two years. On behalf of the MSEB, I want to thank each of them for their commitment to the important and difficult set of issues this study comprised. I especially want to commend Jere Confrey and extend deepest appreciation to her for her extraordinary leadership and commitment as chair of this project. In addition to her leadership of the committee, Jere played an extensive role
in drafting and redrafting the report through the final stages of committee consultation and the intensive review process. Her dedication to maintaining the highest standards of scholarship and the full engagement of the committee, despite her many other professional obligations, was exemplary. The report bears the imprint of her commitment to intellectual and empirical rigor; the field of mathematics education is the fortunate beneficiary.
Joan R. Leitzel
Chair, MSEB
Acknowledgments
This report reflects the efforts of many people, and the committee is most grateful for their contributions. First, we wish to acknowledge and thank our sponsor, the National Science Foundation’s Directorate for Education and Human Resources, for their support of this study. The National Academies’ Presidents’ Fund also provided some additional support for the report.
The committee was aided greatly by the individuals who participated in our evidence-gathering workshops. They helped us understand the nuances of curriculum development and implication and ways to evaluate the efficacy of such curricula once out in the field. The first evidence gathering workshop was held in September 2002 and focused on how to define or evaluate “effectiveness” of school mathematics curricula. The committee gained valuable insights from the experts who presented at that workshop: Johanna Birckmayer, Pacific Institute of Research and Evaluation; David Francis, University of Houston; Pendred Noyce, The Noyce Foundation; Andrew Porter, Wisconsin Center for Education Research; Frank Wang, Saxon Publishers; Jan Mokros, TERC; Thomas Romberg, University of Wisconsin–Madison; Sheila Sconiers, ARC Implementation Center; Andrew Isaacs, University of Chicago; Judith Zawojewski, Illinois Institute of Technology; Solomon Garfunkel, COMAP, Inc; Norman Webb, Wisconsin Center for Education Research; Richard Askey, University of Wisconsin–Madison; Roger Howe, Yale University; William McCallum, University of Arizona; M. Kathleen Heid, Pennsylvania State University; Richard Lehrer, Vanderbilt University; Richard Lesh, Purdue University; Mark Jenness,
Western Michigan University; June Mark, Education Development Center, Inc.; Maria Santos, Achieve, Inc; Richard Stanley, University of California, Berkeley; Terri Dahl, Charles M. Russell High School, MT; Eric Gutstein, University of Illinois, Chicago; Mark St. John, Inverness Research Associates; Timothy Wierenga, Naperville Community Unit School District #203, IL; James Stigler, University of California, Los Angeles; and William Tate, Washington University, St. Louis. The second evidence-gathering workshop was held in October 2002 and focused on how commercial publishers evaluate their text series. The committee gained valuable insight from Arthur Block, McGraw-Hill; Karen Usiskin, Scott Foresman; Steve Rasmussen, Key Curriculum Press; Denise McDowell, Houghton Mifflin; Paula Gustafson, Texas Education Agency; and Lisa Brady-Gill, Texas Instruments. Much was learned from those who presented at that these workshops which proved of great value to the committee as it wrote this report.
The committee would like to acknowledge the kindness of the principal investigators of the NSF-supported curriculum projects, of the NSF Implementation Centers, and of textbook publishers for supplying us with texts and evaluation studies. The committee is also most grateful to the graduate students who labored through reading and entering data on the almost 700 evaluation studies that we accumulated: Kathleen M. Clark, University of Maryland; Timothy P. Fukawa-Connelly, University of Maryland; Michelle Caren Massey, American University; and Francesca Palik, George Washington University. Among these graduate students, special thanks go to Erica Slate Young, University of Texas, Austin, who worked closely with the chair of the committee. Sibel Kazak and Dustin Mitchell, Washington University in St. Louis also provided assistance to the chair. In addition special thanks go to Alan Maloney, for the long hours he dedicated (including his birthday) to helping the chair prepare the final revised draft of the report.
Special thanks go to the NRC staff who worked on this report. Research associate Vicki Stohl worked tirelessly to develop a data base, assemble materials for the committee, and produce part of the preliminary draft of the document. Senior program assistant Dionna Williams worked efficiently under great time pressure to engineer the workshops and other committee meetings and to put the many drafts of this report into good form. Carole Lacampagne provided leadership for the project, Jay Labov gave sage advice along the way, and Patricia Morison was instrumental in bringing this report to successful closure. Martin Orland, director of the Center for Education, also provided leadership and support for the project.
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC’s RRC. The purpose of this independent review is to provide candid and critical comments that will assist the institu-
tion in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Douglas Carnine, National Center to Improve the Tools of Educators, University of Oregon; Mary M. Lindquist, Department of Mathematics Emeritus, Columbus State University, Columbus, GA; R. James Milgram, Department of Mathematics, Stanford University; Leona Schauble, Teaching and Learning Department, Vanderbilt University; Alan H. Schoenfeld, School of Education, University of California, Berkeley; Norman Webb, Wisconsin Center for Education Research, University of Wisconsin–Madison; and Iris R. Weiss, President, Horizon Research, Inc., Chapel Hill, NC.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations nor did they see the final draft of the report before its release. The review of this report was overseen by Adam Gamoran, University of Wisconsin–Madison, and William G. Howard, Jr., Independent Consultant, Scottsdale, AZ, appointed by the National Research Council, who were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility of the final content of this report, however, rests entirely with the authoring committee and the NRC.