NATIONAL ACADEMY PRESS
2101 Constitution Avenue, NW Washington, DC 20418
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study was supported by Contract No. ED-00-00-0038 between the National Academy of Sciences and the U.S. Department of Education National Educational Research Policy and Priorities Board. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
Library of Congress Control Number 2002101615
International Standard Book Number 0-309-08291-9
Additional copies of this report are available from
National Academy Press,
2101 Constitution Avenue, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu
Printed in the United States of America
Copyright 2002 by the National Academy of Sciences. All rights reserved.
Suggested citation: National Research Council. (2002). Scientific research in education. Committee on Scientific Principles for Education Research. Shavelson, R.J., and Towne, L., Editors. Center for Education. Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.
National Academy of Sciences
National Academy of Engineering
Institute of Medicine
National Research Council
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Wm. A. Wulf are chairman and vice chairman, respectively, of the National Research Council.
COMMITTEE ON SCIENTIFIC PRINCIPLES FOR EDUCATION RESEARCH
Richard J. Shavelson (Chair),
School of Education, Stanford University
Donald I. Barfield,
WestEd, San Francisco
Robert F. Boruch,
Graduate School of Education, Wharton School Department of Statistics, and Fels Center for Government, University of Pennsylvania, Philadelphia
Jere Confrey,
Department of Curriculum and Instruction, University of Texas at Austin
Rudolph Crew,
Stupski Family Foundation, Mill Valley, California
Robert L. DeHaan,
Department of Cell Biology, Emory University, Atlanta, Georgia
Margaret Eisenhart,
School of Education, University of Colorado at Boulder
Jack McFarlin Fletcher,
Department of Pediatrics, University of Texas, Houston
Eugene E. Garcia,
Graduate School of Education, University of California, Berkeley
Norman Hackerman,
Robert A. Welch Foundation, Houston, Texas
Eric Hanushek,
Hoover Institution, Stanford University
Robert Hauser,
Center for Demography of Health and Aging, University of Wisconsin-Madison
Paul W. Holland,
Educational Testing Service, Princeton, New Jersey
Ellen Condliffe Lagemann,
The Spencer Foundation, Chicago, Illinois, and New York University, New York, New York
Denis C. Phillips,
School of Education, Stanford University
Carol H. Weiss,
Graduate School of Education, Harvard University
Lisa Towne, Study Director
Tina Winters, Research Assistant
Linda DePugh, Senior Project Assistant
Dedication
Lee J. Cronbach (1916-2001)
Lee Cronbach was the Vida Jacks Professor of Education Emeritus at Stanford University and member of the National Academy of Sciences and the National Academy of Education. His career spanned five decades and yielded seminal contributions to psychometrics, instruction, and program evaluation. His ideas and high standards for rigorous scientific inquiry greatly influenced educational and social science research and the deliberations of our committee.
Foreword
The National Academies have been in the business of bringing science to bear on pressing problems since 1863. Our operating arm— the National Research Council (NRC)—has produced hundreds of reports that synthesize scientific knowledge in a wide range of areas that affect the public interest. Most of this work involves scientists acting to promote rational decision making in matters of public policy. Less often, our reports explicitly comment on the nature of the scientific enterprise itself. This report is such an example. Its authoring committee was assembled amid vibrant debate about quality and rigor in scientific education research. In the course of its work, the committee revisited long-standing philosophies about the nature of science, so as to place them in the context of modern education research.
Because in many ways this report is itself a product of scientific work, it had to live up to its own depiction of what constitutes good science. The authoring committee has applied rigorous reasoning to its scrutiny of evidence and ideas, considered alternative perspectives, and presented its findings and conclusions in a language that invites constructive discussion.
I hope that Scientific Research in Education will advance the current dialogue in at least two respects. First, it offers a comprehensive perspective of “scientifically-based” education research for the policy communities who are increasingly interested in its utilization for improving education policy and practice. Second, the report shows that, within the diverse field of education, researchers who often disagree along philosophical and
methodological lines nonetheless share much common ground about the definition and pursuit of quality. This report should therefore be useful for researchers, as well as for those who use research.
This effort continues a series of recent institutional changes and initiatives within the NRC designed to elevate the role of education research in improving policy and practice. In 1999, we created the Center for Education to integrate and strengthen our already substantial portfolio of work in education and education research. In addition, a major NRC initiative called the Strategic Education Research Partnership focuses on developing the capacity and infrastructure to systematically link education research and practice. Taken together, these and future efforts are intended to help transform education into an increasingly evidence-based field—one of the most important goals of my presidency.
Like any good scholarly work, this book will no doubt incite debate and discussion, invite critique and commentary, and claim its defenders and detractors. As the authors argue in the pages that follow, this kind of professional, constructive discourse is precisely what characterizes a healthy scientific community. We welcome the dialogue to come.
Bruce Alberts
President, National Academy of Sciences
Acknowledgments
This report could not have been produced without support from a number of people, and the committee is grateful for their contributions. First, we wish to acknowledge our sponsor, the U.S. Department of Education’s National Educational Research Policy and Priorities Board, and in particular Kenji Hakuta, Thelma Leenhouts, Mary Grace Lucier, Alba Ortiz, and Rafael Valdivieso.
The committee was aided greatly by individuals who participated in our meetings and helped us understand the complex issues involved in examining the nature of scientific inquiry in education. At our first meeting in December 2000, Tom Glennan of RAND, Jane Oates of the office of U.S. Senator Kennedy, D’Arcy Philps formerly of the Committee on Education and the Workforce of the U.S. House of Representatives, and Alexandra (Sandy) Wigdor of the National Research Council all provided helpful presentations about the context of our work.
In March 2001 the committee hosted a workshop on science, evidence, and inference in education. We are particularly grateful for the contributions of the workshop speakers and panelists: Michael Agar, University of Maryland and Ethknoworks; Norman Bradburn, U.S. National Science Foundation; Glen Cain, University of Wisconsin; Susan Chipman, U.S. Office of Naval Research; Christopher T. Cross, Council for Basic Education; Larry Hedges, University of Chicago; Jeremy Kilpatrick, University
of Georgia; David Klahr, Carnegie Mellon University; Sharon Lewis, Council of Great City Schools; Reid Lyon, U.S. National Institute of Child Heath and Human Development; Kent McGuire, Manpower Demonstration Research Corporation; Robert Mislevy, University of Maryland; William Morrill, Caliber Associates; William Quinn, North Central Regional Educational Laboratory; Diane Ravitch, New York University and Brookings Institution; Sally Rockey, Cooperative State Research, Education, and Extension Service, U.S. Department of Agriculture; Steven Ross, University of Memphis; Nancy Songer, University of Michigan; Judith Sunley, U.S. National Science Foundation; Richard Suzman, U.S. National Institute on Aging; Peter Tillers, Cardozo Law School, Yeshiva University, and Yale Law School; and Maris Vinovskis, University of Michigan. We also want to thank all of the workshop participants, whose active engagement over the course of the 2-day workshop significantly enhanced the dialogue.
In the months following the workshop, Norman Bradburn, Reid Lyon, Judith Sunley, and Richard Suzman continued to be of tremendous assistance as we collected more data on the research activities of their federal agencies. Special thanks go to Martin Orland at the U.S. Department of Education’s Office of Educational Research and Improvement; his help in the development of our data collection instrument was invaluable, as were the extensive data he provided.
The committee also benefited from discussion with and ideas from several National Academy of Sciences and Institute of Medicine members during our deliberations: Arthur Goldberger, Eleanor Maccoby, James March, Neil Smelser, and Patrick Suppes. Their willingness to share their insights with the committee at a formative time of our deliberations was extraordinarily generous.
Many others have supported this project. The staff at the library and archives of the Educational Testing Service provided considerable assistance in putting together the material on the history of assessment. Several staff members at the National Research Council (NRC) Center for Education provided informal guidance to staff throughout the entire process, especially Naomi Chudowsky. The center’s director, Michael Feuer, was very generous of his time; his advice throughout the course of this project
was invaluable. We also wish to thank Kirsten Sampson Snyder for guiding us through the report review process and Eugenia Grohman for her expert editing of the manuscript. The committee is especially grateful for the project staff support of Linda DePugh, who provided logistical and administrative assistance throughout the project, and Tina Winters, who managed the post-workshop data collection and provided assistance with manuscript preparation. Their contributions were essential to producing this volume.
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Deborah Lowenberg Ball, University of Michigan; Hyman Bass, University of Michigan; M. Susan Burns, George Mason University; Thomas Cook, Northwestern University; David S. Cordray, Vanderbilt University; Michael Doyle, Research Corporation; James Duderstadt, University of Michigan; Arthur S. Goldberger, University of Wisconsin-Madison; Jane Hannaway, The Urban Institute; William Reese, University of Wisconsin-Madison; Marshall Smith, William and Flora Hewlett Foundation; and John Willinsky, University of British Columbia.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations nor did they see the final draft of the report before its release. The review of this report was overseen by Robert L. Linn, University of Colorado, and Lyle V. Jones, The University of North Carolina at Chapel Hill. Appointed by the NRC, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.
Finally, we must thank the members of our marvelous committee. It was no small task to agree on and articulate the nature of scientific research in education, particularly in less than a year’s time. It was their dedication, perseverance, and collective expertise that made it possible. And it was their candid, professional approach that made the effort rewarding and fun.
Richard J. Shavelson, Chair
Lisa Towne, Study Director