National Academies Press: OpenBook
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

On Evaluating Curricular Effectiveness

JUDGING THE QUALITY OF K-12 MATHEMATICS EVALUATIONS

Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials

Mathematical Sciences Education Board

Center for Education

Division of Behavioral and Social Sciences and Education

NATIONAL RESEARCH COUNCIL OF THE NATIONAL ACADEMIES

THE NATIONAL ACADEMIES PRESS
Washington, D.C. www.nap.edu

Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

THE NATIONAL ACADEMIES PRESS
500 Fifth Street, N.W. Washington, DC 20001

NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.

This study/publication was supported by Contract/Grant No. ESI-0102582 between the National Academy of Sciences and the National Science Foundation. Additional funding was provided by an award from the Presidents’ Committee of the National Academies. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.

Library of Congress Cataloging-in-Publication Data

On evaluating curricular effectiveness : judging the quality of K-12 mathematics evaluations / Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials, Mathematical Sciences Education Board, Center for Education, Division of Behavioral and Social Sciences and Education.

p. cm.

Includes bibliographical references.

ISBN 0-309-09242-6 (pbk.)—ISBN 0-309-53287-6 (pdf)

1. Mathematics—Study and teaching—Evaluation. I. National Research Council (U.S.). Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials.

QA11.2.O5 2004

510'.71—dc22

2004015000

Additional copies of this report are available from the
National Academies Press,
500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu.

Printed in the United States of America.

Copyright 2004 by the National Academy of Sciences. All rights reserved.

National Research Council. (2004). On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials. Mathematical Sciences Education Board, Center for Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

THE NATIONAL ACADEMIES

Advisers to the Nation on Science, Engineering, and Medicine

The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.

The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering.

The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.

The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council.

www.national-academies.org

Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

This page intentionally left blank.

Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

COMMITTEE FOR A REVIEW OF THE EVALUATION DATA ON THE EFFECTIVENESS OF NSF-SUPPORTED AND COMMERCIALLY GENERATED MATHEMATICS CURRICULUM MATERIALS

JERE CONFREY (Chair),

Department of Education, Washington University in St. Louis, MO

CARLOS CASTILLO-CHAVEZ,

Department of Mathematics and Statistics, Arizona State University

DOUGLAS A. GROUWS,

Department of Learning, Teaching, and Curriculum, University of Missouri

CAROLYN MAHONEY,

Division of Academic Affairs, Elizabeth City State University, NC

DONALD SAARI,

Institute for Mathematical Behavioral Sciences, University of California, Irvine

WILLIAM SCHMIDT,

College of Education, Michigan State University

PATRICK W. THOMPSON,

Department of Teaching and Learning, Vanderbilt University, TN

WILLIAM VELEZ,

Department of Mathematics, University of Arizona

CAROLE B. LACAMPAGNE, Study Director

VICKI STOHL, Research Associate

DIONNA J. WILLIAMS, Senior Program Assistant

Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

MATHEMATICAL SCIENCES EDUCATION BOARD 2004

JOAN LEITZEL (Chair), President Emerita,

University of New Hampshire

JERE CONFREY (Vice Chair),

Department of Education, Washington University in St. Louis, MO

THOMAS BANCHOFF,

Department of Mathematics, Brown University, RI

JAN DE LANGE,

Freudenthal Institute, Utrecht University, The Netherlands

LOUIS GOMEZ,

School of Education and Social Policy, Northwestern University, IL

DOUGLAS A. GROUWS,

Department of Learning, Teaching, and Curriculum, University of Missouri

ARTHUR JAFFE,

Department of Mathematics, Harvard University, MA

ERIC JOLLY,

Science Museum of Minnesota

JIM LEWIS,

Department of Mathematics and Statistics, University of Nebraska-Lincoln

GEORGE MCSHAN,

National School Boards Association, TX

KAREN MICHALOWICZ,

Mathematics Department, The Langley School, VA

JUDITH MUMME,

WestEd, CA

CASILDA PARDO,

Teaching and Learning Systems, Valle Vista Elementary School, NM

SUE PARSONS, Instructor,

Cerritos College, CA

MARGE PETIT,

The National Center for the Improvement of Educational Assessment, VT

DONALD SAARI,

Institute for Mathematical Behavioral Sciences, Universityof California, Irvine

RICHARD SCHEAFFER, Professor Emeritus,

University of Florida

FRANCIS SULLIVAN,

Center for Computing Sciences, MD

HUNG HSI WU,

Department of Mathematics, University of California, Berkeley

Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

Foreword

The Mathematical Sciences Education Board (MSEB) of the National Research Council was established in 1985 to provide national leadership and guidance for policies, programs, and practices supporting the improvement of mathematics education at all levels. Curriculum materials for grades K-12 play a central role in what mathematics topics are taught in our schools, how the topics are sequenced and presented to students, what levels of understanding are expected, what skills students will develop and when. Schools, practitioners, policy makers, and the public depend on evaluations of materials undertaken during their development and implementation in making decisions about the appropriate uses of the materials. The MSEB recognized that the nature and quality of the evidence used to judge claims of success and failure are critical elements in enabling the community to make sound judgments. This report presents a synthesis of the evidence used in the evaluations of several sets of recently developed curriculum materials, provides a framework for the design of evaluation studies of curriculum materials, and gives conclusions and recommendations to guide future efforts in evaluating curriculum materials.

The report was prepared by a committee of experts who devoted their time, skills, and scholarship to this project over the past two years. On behalf of the MSEB, I want to thank each of them for their commitment to the important and difficult set of issues this study comprised. I especially want to commend Jere Confrey and extend deepest appreciation to her for her extraordinary leadership and commitment as chair of this project. In addition to her leadership of the committee, Jere played an extensive role

Page viii Cite
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

in drafting and redrafting the report through the final stages of committee consultation and the intensive review process. Her dedication to maintaining the highest standards of scholarship and the full engagement of the committee, despite her many other professional obligations, was exemplary. The report bears the imprint of her commitment to intellectual and empirical rigor; the field of mathematics education is the fortunate beneficiary.

Joan R. Leitzel

Chair, MSEB

Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

Acknowledgments

This report reflects the efforts of many people, and the committee is most grateful for their contributions. First, we wish to acknowledge and thank our sponsor, the National Science Foundation’s Directorate for Education and Human Resources, for their support of this study. The National Academies’ Presidents’ Fund also provided some additional support for the report.

The committee was aided greatly by the individuals who participated in our evidence-gathering workshops. They helped us understand the nuances of curriculum development and implication and ways to evaluate the efficacy of such curricula once out in the field. The first evidence gathering workshop was held in September 2002 and focused on how to define or evaluate “effectiveness” of school mathematics curricula. The committee gained valuable insights from the experts who presented at that workshop: Johanna Birckmayer, Pacific Institute of Research and Evaluation; David Francis, University of Houston; Pendred Noyce, The Noyce Foundation; Andrew Porter, Wisconsin Center for Education Research; Frank Wang, Saxon Publishers; Jan Mokros, TERC; Thomas Romberg, University of Wisconsin–Madison; Sheila Sconiers, ARC Implementation Center; Andrew Isaacs, University of Chicago; Judith Zawojewski, Illinois Institute of Technology; Solomon Garfunkel, COMAP, Inc; Norman Webb, Wisconsin Center for Education Research; Richard Askey, University of Wisconsin–Madison; Roger Howe, Yale University; William McCallum, University of Arizona; M. Kathleen Heid, Pennsylvania State University; Richard Lehrer, Vanderbilt University; Richard Lesh, Purdue University; Mark Jenness,

Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

Western Michigan University; June Mark, Education Development Center, Inc.; Maria Santos, Achieve, Inc; Richard Stanley, University of California, Berkeley; Terri Dahl, Charles M. Russell High School, MT; Eric Gutstein, University of Illinois, Chicago; Mark St. John, Inverness Research Associates; Timothy Wierenga, Naperville Community Unit School District #203, IL; James Stigler, University of California, Los Angeles; and William Tate, Washington University, St. Louis. The second evidence-gathering workshop was held in October 2002 and focused on how commercial publishers evaluate their text series. The committee gained valuable insight from Arthur Block, McGraw-Hill; Karen Usiskin, Scott Foresman; Steve Rasmussen, Key Curriculum Press; Denise McDowell, Houghton Mifflin; Paula Gustafson, Texas Education Agency; and Lisa Brady-Gill, Texas Instruments. Much was learned from those who presented at that these workshops which proved of great value to the committee as it wrote this report.

The committee would like to acknowledge the kindness of the principal investigators of the NSF-supported curriculum projects, of the NSF Implementation Centers, and of textbook publishers for supplying us with texts and evaluation studies. The committee is also most grateful to the graduate students who labored through reading and entering data on the almost 700 evaluation studies that we accumulated: Kathleen M. Clark, University of Maryland; Timothy P. Fukawa-Connelly, University of Maryland; Michelle Caren Massey, American University; and Francesca Palik, George Washington University. Among these graduate students, special thanks go to Erica Slate Young, University of Texas, Austin, who worked closely with the chair of the committee. Sibel Kazak and Dustin Mitchell, Washington University in St. Louis also provided assistance to the chair. In addition special thanks go to Alan Maloney, for the long hours he dedicated (including his birthday) to helping the chair prepare the final revised draft of the report.

Special thanks go to the NRC staff who worked on this report. Research associate Vicki Stohl worked tirelessly to develop a data base, assemble materials for the committee, and produce part of the preliminary draft of the document. Senior program assistant Dionna Williams worked efficiently under great time pressure to engineer the workshops and other committee meetings and to put the many drafts of this report into good form. Carole Lacampagne provided leadership for the project, Jay Labov gave sage advice along the way, and Patricia Morison was instrumental in bringing this report to successful closure. Martin Orland, director of the Center for Education, also provided leadership and support for the project.

This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC’s RRC. The purpose of this independent review is to provide candid and critical comments that will assist the institu-

Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

tion in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Douglas Carnine, National Center to Improve the Tools of Educators, University of Oregon; Mary M. Lindquist, Department of Mathematics Emeritus, Columbus State University, Columbus, GA; R. James Milgram, Department of Mathematics, Stanford University; Leona Schauble, Teaching and Learning Department, Vanderbilt University; Alan H. Schoenfeld, School of Education, University of California, Berkeley; Norman Webb, Wisconsin Center for Education Research, University of Wisconsin–Madison; and Iris R. Weiss, President, Horizon Research, Inc., Chapel Hill, NC.

Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations nor did they see the final draft of the report before its release. The review of this report was overseen by Adam Gamoran, University of Wisconsin–Madison, and William G. Howard, Jr., Independent Consultant, Scottsdale, AZ, appointed by the National Research Council, who were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility of the final content of this report, however, rests entirely with the authoring committee and the NRC.

Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

This page intentionally left blank.

Page xiii Cite
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×


On Evaluating Curricular Effectiveness

Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

This page intentionally left blank.

Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page R1
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page R2
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page R3
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page R4
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page R5
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page R6
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page R7
Page viii Cite
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page R8
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page R9
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page R10
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page R11
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page R12
Page xiii Cite
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page R13
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page R14
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page R15
Suggested Citation:"Front Matter." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page R16
Next: Executive Summary »
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations Get This Book
×
Buy Paperback | $59.00 Buy Ebook | $47.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This book reviews the evaluation research literature that has accumulated around 19 K-12 mathematics curricula and breaks new ground in framing an ambitious and rigorous approach to curriculum evaluation that has relevance beyond mathematics. The committee that produced this book consisted of mathematicians, mathematics educators, and methodologists who began with the following charge:

  • Evaluate the quality of the evaluations of the thirteen National Science Foundation (NSF)-supported and six commercially generated mathematics curriculum materials;
  • Determine whether the available data are sufficient for evaluating the efficacy of these materials, and if not;
  • Develop recommendations about the design of a project that could result in the generation of more reliable and valid data for evaluating such materials.

The committee collected, reviewed, and classified almost 700 studies, solicited expert testimony during two workshops, developed an evaluation framework, established dimensions/criteria for three methodologies (content analyses, comparative studies, and case studies), drew conclusions on the corpus of studies, and made recommendations for future research.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!