National Academies Press: OpenBook
Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

Keeping Score for All

The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments

Committee on Participation of English Language Learners and Students with Disabilities in NAEP and Other Large-Scale Assessments

Judith Anderson Koenig and Lyle F. Bachman, Editors

Board on Testing and Assessment Center for Education

Division of Behavioral and Social Sciences and Education

NATIONAL RESEARCH COUNCIL OF THE NATIONAL ACADEMIES

THE NATIONAL ACADEMIES PRESS
Washington, D.C. www.nap.edu

Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

THE NATIONAL ACADEMIES PRESS
500 Fifth Street, N.W. Washington, DC 20001

NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.

This study was supported by Contract/Grant No. R215U990016 between the National Academy of Sciences and the U.S. Department of Education. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.

Library of Congress Cataloging-in-Publication Data

National Research Council (U.S.). Committee on Participation of English Language Learners and Students with Disabilities in NAEP and Other Large-Scale Assessments.

Keeping score for all : the effects of inclusion and accommodation policies on large-scale educational assessments / Committee on Participation of English Language Learners and Students with Disabilities in NAEP and Other Large-Scale Assessments ; Judith Anderson Koenig and Lyle F. Bachman, editors.

p. cm.

Includes bibliographical references.

ISBN 0-309-09253-1 (pbk.)—ISBN 0-309-53317-1 (pdf)

1. Educational tests and measurements—United States—Evaluation. 2. Inclusive education—United States—Evaluation. 3. Students with disabilities—Education—United States. 4. Limited English-proficient students—Education—United States. 5. National Assessment of Educational Progress (Project) I. Koenig, Judith A. II. Bachman, Lyle F., 1944- III. Title.

LB3060.83.N295 2004

371.26’4—dc22

2004014364

Additional copies of this report are available from the
National Academies Press,
500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu.

Printed in the United States of America

Copyright 2004 by the National Academy of Sciences. All rights reserved.

Suggested citation: National Research Council. (2004). Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Committee on Participation of English Language Learners and Students with Disabilities in NAEP and Other Large-Scale Assessments. Judith A. Koenig and Lyle F. Bachman, Editors. Board on Testing and Assessment, Center for Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

THE NATIONAL ACADEMIES

Advisers to the Nation on Science, Engineering, and Medicine

The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.

The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering.

The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.

The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council.

www.national-academies.org

Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

This page intentionally left blank.

Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

COMMITTEE ON PARTICIPATION OF ENGLISH LANGUAGE LEARNERS AND STUDENTS WITH DISABILITIES IN NAEP AND OTHER LARGE-SCALE ASSESSMENTS

Lyle F. Bachman (Chair*),

Department of Applied Linguistics, University of California, Los Angeles

Jonathan G. Dings,

Boulder Valley School District, Boulder, CO

Judy L. Elliott,

Long Beach Unified School District, San Pedro, CA

Margaret J. McLaughlin,

Institute for the Study of Exceptional Children and Youth, University of Maryland, College Park

Mark D. Reckase (Chair**),

Measurement and Quantitative Methods, Michigan State University

Lourdes C. Rovira,

Miami-Dade County Public Schools, Miami, FL

María Medina Seidner,

Texas Education Agency, Austin, TX

Rebecca Zwick,

Gevirtz Graduate School of Education, University of California, Santa Barbara

Judith A. Koenig, Study Director

Alexandra Beatty, Senior Program Officer

Michael DeCarmine, Senior Program Assistant

*  

Beginning November 2003.

**  

Stepped down from the Committee in November 2003, when he was appointed to the National Assessment Governing Board.

Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

BOARD ON TESTING AND ASSESSMENT

Eva L. Baker (Chair),

Center for the Study of Evaluation, University of California, Los Angeles

Lorraine McDonnell (Vice Chair),

Department of Political Science, University of California, Santa Barbara

Lauress L. Wise (Vice Chair),

Human Resources Research Organization, Alexandria, VA

Christopher F. Edley, Jr.,

Harvard Law School

Emerson J. Elliott, Independent Consultant,

Arlington, VA

Milton D. Hakel,

Department of Psychology, Bowling Green State University, OH

Robert M. Hauser,

Institute for Research on Poverty, Center for Demography, University of Wisconsin–Madison

Paul W. Holland,

Educational Testing Service, Princeton, NJ

Daniel M. Koretz,

Graduate School of Education, Harvard University

Edward P. Lazear,

Graduate School of Business, Stanford University

Richard Light,

Graduate School of Education, John F. Kennedy School of Government, Harvard University

Robert J. Mislevy,

Department of Measurement, Statistics, and Evaluation, University of Maryland, College Park

James W. Pellegrino,

Department of Psychology, University of Illinois, Chicago

Lorrie Shepard,

Department of Education, University of Colorado, Boulder

Stuart W. Elliott, Director

Lisa D. Alston, Administrative Associate

Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

Preface

The Committee on the Participation of Students with Disabilities and English Language Learners in NAEP and Other Large-Scale Assessments was formed under the auspices of the National Research Council (NRC) with the support of the National Center for Education Statistics (NCES). Its report addresses critical issues in the assessment of students with disabilities and English language learners, an issue that has come to the forefront of conversations about the effects of the No Child Left Behind Act of 2001. The NRC’s Board on Testing and Assessment (BOTA) has focused considerable attention during its ten-year history on the challenges and questions presented by the need to include these students in assessment and accountability programs. At present these students make up approximately 20 percent of the nation’s 46 million public school students, and while the responsibility for monitoring their progress is not new either to the National Assessment of Educational Progress (NAEP) or to states and districts, the provisions of the No Child Left Behind Act of 2001 have made that responsibility more public, more complex, and more urgent.

A November 2001 BOTA workshop, also sponsored by NCES, focused on the reporting and interpretation of test results for students with disabilities and English language learners who receive test accommodations. Discussions at the workshop made clear that several key issues merited more in-depth examination. The first set of concerns pertained to the way decisions regarding both the inclusion of students with special needs in assessments and the identification of appropriate accommodations for them are made. It was clear from the workshop not only that there is considerable variability across states and districts in the way these decisions are made but also that this variability can affect NAEP results in

Page viii Cite
Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

significant ways. The second set of concerns pertained to the research that has been conducted on the effects of accommodations on test performance. The 2001 workshop stimulated considerable discussion about the conclusions that could be drawn from the existing literature base and about appropriate research approaches for evaluating the effects of accommodations on test performance.

Therefore, at the request of NCES, the Committee on the Participation of Students with Disabilities and English Language Learners in NAEP and Other Large-Scale Assessments was formed and given a three-part charge, to (1) synthesize research findings about the effects of accommodations on test performance, (2) review the procedures used for making inclusion and accommodation decisions for large-scale assessment programs, and (3) determine the implications of these findings for NAEP inclusions and accommodation policies. The work of this committee was intended to build on the discussions from the 2001 workshop (and other earlier NRC projects).

Thus, the report from the 2001 workshop served as the starting point for the committee’s work. In addition, researcher Stephen Sireci was commissioned to conduct a review and critique of the research literature on the effects of accommodations on test performance. This literature review focused on empirical studies conducted between 1990 and 2003 and was commissioned and completed in time to be ready for the committee’s first meeting in March 2003. The workshop report and this literature review served as background information for the committee as it began its in-depth examination of the relevant policies and practices in effect around the country and the state of the research in this vital area.

The resulting report is designed to set the committee’s findings and recommendations in the context of current policy and practice with regard to the inclusion and accommodation of students with disabilities and English language learners. In this report, the committee discusses the meaning of scores from accommodated assessments and the kinds of evidence that are needed to support inferences made from these scores. It is BOTA’s hope that this report will be of use to both the officials who oversee NAEP and those who oversee state and local assessments as they work to make their assessments as inclusive as possible, and to make them yield results that accurately reflect the knowledge and skills of all students.

The committee is very grateful to the many individuals who have helped with this project from its inception. It takes particular note of the contribution of Mark Reckase of Michigan State University, who served as its chair through most of its work, until his appointment to the National Assessment Governing Board required him to step down. The committee also sincerely appreciates Lyle Bachman’s willingness to assume the responsibility of chair for the completion of the project.

The committee extends its heartfelt thanks to Peggy Carr of NCES for her interest in this important topic and her willingness to fund the project. The committee also thanks Arnold Goldstein of NCES for his constant and prompt support

Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

and quick answers to all of the committee’s questions. The committee also appreciates the assistance of Debra Hollinger Martinez, formerly of NCES, who provided valuable materials and information. Nancy Caldwell of Westat provided information about NAEP’s sampling procedures and responded to committee questions about NAEP’s administrative procedures for accommodating students with disabilities and English language learners. Jim Carlson of NAGB provided the committee with information on NAEP policies on accommodation as well as on research on the effects of accommodations on NAEP results. Mary Crovo of NAGB spoke with the committee about the constructs assessed by NAEP, and the committee is grateful to all of these busy officials for their time and assistance.

Many others assisted the committee as well. Researchers Martha Thurlow of the National Center on Education Outcomes and Charlene Rivera of the Center for Equity and Excellence provided the committee with background about states’ policies; John Olsen of the Council of Chief State School Officers (CCSSO) made a presentation about the results of the CCSSO’s annual survey of states’ practices. Presentations by Wayne Camara of the College Board and Robert Ziomek of ACT, Inc., on their research on the effects of accommodations on admissions test performance were particularly informative. The committee is grateful to Ed Haertel of Stanford University for providing an advance copy of his paper on evidentiary arguments and the comparability of scores from standard and nonstandard test administrations. Eric Hansen of Educational Testing Service briefed the committee on his work on the evidence-centered design approach and prepared a commissioned paper that was very helpful to the committee. The committee is also indebted to Stephen Sireci of the University of Massachusetts, Amherst and his colleagues Stanley Scarpati and Shuhong Li for their work on the commissioned review and critique of the literature.

The committee also wishes to thank the NRC staff who have supported the project. Study director Judith Koenig has offered leadership and support in countless ways and has guided the committee through some quite complicated territory. The committee also thanks Pasquale DeVito for his initial work on the project, and Stuart Elliott and Patricia Morison, who read and made valuable comments on several versions of the report. Michael DeCarmine’s calm and able administrative assistance throughout the project is much appreciated as well. The committee is also grateful for Alexandra Beatty’s expert writing ability. She provided invaluable assistance in drafting portions of the report and editing it so that it read with one voice. NRC editor Chris McShane provided valuable editing and smoothing in the final stages as well. Finally, the committee is indebted to Kirsten Sampson Snyder for ably guiding the report through the NRC review process.

This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC’s RRC. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its

Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

published report as sound as possible and ensuring that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. I thank the following individuals for their review of this report: Jamal Abedi, Technical Projects, Center for the Study of Evaluation, University of California, Los Angeles; Susan A. Agruso, Office of Assessment, Charlotte-Mecklenburg School District, Charlotte, NC; Richard Duran, Gevirtz Graduate School of Education, University of California, Santa Barbara; Shelley Loving-Ryder, Assistant Superintendent for Assessment and Reporting, Virginia Department of Education; Diana Pullin, Education Law and Public Policy, Boston College; and Martha Thurlow, National Center on Educational Outcomes, University of Minnesota, Minneapolis.

Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by Lyle V. Jones, L.L. Thurstone Psychometric Laboratory, University of North Carolina. Appointed by the National Research Council, he was responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.

Eva L. Baker

Chair, Board on Testing and Assessment

Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

This page intentionally left blank.

Page xiii Cite
Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×


Keeping Score for All

The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments

Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

This page intentionally left blank.

Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page R1
Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page R2
Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page R3
Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page R4
Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page R5
Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page R6
Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page R7
Page viii Cite
Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page R8
Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page R9
Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page R10
Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page R11
Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page R12
Page xiii Cite
Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page R13
Suggested Citation:"Front Matter." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page R14
Next: Executive Summary »
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments Get This Book
×
Buy Paperback | $49.00 Buy Ebook | $39.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

U.S. public schools are responsible for educating large numbers of English language learners and students with disabilities. This book considers policies for including students with disabilities and English language learners in assessment programs. It also examines the research findings on testing accommodations and their effect on test performance.

Keeping Score for All discusses the comparability of states’ policies with each other and with the National Assessment of Educational Progress (NAEP) policies and explores the impact of these differences on the interpretations of NAEP results. The book presents a critical review of the research literature and makes suggestions for future research to evaluate the validity of test scores obtained under accommodated conditions. The book concludes by proposing a new framework for conceptualizing accommodations. This framework would be useful both for policymakers, test designers, and practitioners in determining appropriate accommodations for specific assessments and for researchers in planning validity studies.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!