7
Background and Methodology

Karen S. Hollweg

One of the main goals of this project was to produce a bibliography of the available literature and completed research regarding the influence of the National Science Education Standards (NSES). The purpose of this chapter is to document the methodology used in creating the bibliography (published in Chapter 8) and to give an overview of the information contained in it.

There has been an increasing interest internationally in using research evidence to inform the development of policy and practice. Researchers in the United Kingdom at the Cochrane Collaboration, the Centre for Reviews and Dissemination in York, and the Institute of Education at the University of London have become known for their high-quality systematic reviews of research relevant to education. In 2001, the Evidence Informed Policy and Practice Information and Co-ordination Centre (EPPI-Centre) at the University of London wrote a Review Group Manual to guide the work of individuals interested in participating in their production and dissemination of systematic reviews in education. The manual, available online at http://eppi.ioe.ac.uk, and the Framework, presented as Figure 1-1 in Chapter 1 and first published in Investigating the Influence of Standards (National Research Council [NRC], 2002), served as starting points for this project. The Steering Committee and staff drew on both documents as they designed the literature search and the guidelines for the work of the commissioned authors, described below.

THE LITERATURE SEARCH

The project Steering Committee and staff wanted to locate as much of the research that addressed the charge as possible and at the same time avoid bias in the search. To make the search as rigorous, exhaustive, and replicable as possible given the limited resources available, two basic strategies were employed: (1) electronic searches of bibliographic databases, journals, and federally funded agencies and institutions, and (2) searches of Web sites of numerous organizations and agencies actively involved in science education research and analysis of standards-based policies and practices.

First, the NRC library staff performed the research using the following databases: ERIC, NTIS, PAIS, PsycINFO, and Sociological Abstracts. In simultaneous searches of these databases, the librarian created a large base set consisting of documents produced between 1993 and 2001 by the journals, federal agencies, and



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 121
7 Background and Methodology Karen S. Hollweg One of the main goals of this project was to produce a bibliography of the available literature and completed research regarding the influence of the National Science Education Standards (NSES). The purpose of this chapter is to document the methodology used in creating the bibliography (published in Chapter 8) and to give an overview of the information contained in it. There has been an increasing interest internationally in using research evidence to inform the development of policy and practice. Researchers in the United Kingdom at the Cochrane Collaboration, the Centre for Reviews and Dissemination in York, and the Institute of Education at the University of London have become known for their high-quality systematic reviews of research relevant to education. In 2001, the Evidence Informed Policy and Practice Information and Co-ordination Centre (EPPI-Centre) at the University of London wrote a Review Group Manual to guide the work of individuals interested in participating in their production and dissemination of systematic reviews in education. The manual, available online at http://eppi.ioe.ac.uk, and the Framework, presented as Figure 1-1 in Chapter 1 and first published in Investigating the Influence of Standards (National Research Council [NRC], 2002), served as starting points for this project. The Steering Committee and staff drew on both documents as they designed the literature search and the guidelines for the work of the commissioned authors, described below. THE LITERATURE SEARCH The project Steering Committee and staff wanted to locate as much of the research that addressed the charge as possible and at the same time avoid bias in the search. To make the search as rigorous, exhaustive, and replicable as possible given the limited resources available, two basic strategies were employed: (1) electronic searches of bibliographic databases, journals, and federally funded agencies and institutions, and (2) searches of Web sites of numerous organizations and agencies actively involved in science education research and analysis of standards-based policies and practices. First, the NRC library staff performed the research using the following databases: ERIC, NTIS, PAIS, PsycINFO, and Sociological Abstracts. In simultaneous searches of these databases, the librarian created a large base set consisting of documents produced between 1993 and 2001 by the journals, federal agencies, and

OCR for page 121
BOX 7-1 Literature Search Targets Journals American Educational Research Journal Educational Evaluation & Policy Analysis Educational Leadership Educational Researcher Harvard Educational Review International Journal of Science Education Journal for Research in Mathematics Education Journal of Research in Science Teaching Journal of Science Education & Technology Journal of Science Teacher Education Journal of Teacher Education Phi Delta Kappan Research in Science & Technological Education Research in Science Education Review of Educational Research Review of Research in Education School Science & Mathematics Science Educator Science Scope Science Teacher Scientia Paedogogica Experimentalis Teachers College Record Teaching & Teacher Education Federal Agencies National Education Goals Panel National Science Foundation U.S. Department of Education Office of Adult and Vocational Education Office of Educational Research and Improvement Office of Elementary and Secondary Education Planning and Evaluation Services Organizations AAAS/Project 2061 American Federation of Teachers American Institutes for Research (AIR) Brookings Institution Carnegie Corporation Carnegie Foundation Consortium for Policy Research in Education (CPRE) Cosmos Council of Chief State School Officers (CCSSO) Education Commission of the States Education Development Center (EDC) Eisenhower National Clearinghouse (ENC) ERIC Clearinghouse (ERIC) Fund for Improvement of Education Hoover Institution/Stanford Horizon Research, Inc. Inverness Research Associates National Center for Research on Evaluation, Standards, and Student Testing (CRESST) The National Commission on Teaching and America’s Future National Institute for Science Education RAND Corporation Research Triangle Institute (RTI) SRI TERC Thomas B. Fordham Foundation The Urban Institute Westat Wisconsin Center for Education Research organizations listed in Box 7-1. This base set was then cross-searched using the keywords in Box 7-2. Both of these lists were generated by a combination of suggestions from members of the Steering Committee, the Committee on Science Education K-12, staff, and others consulted by staff. The goal was to search multiple sources representing the full range of large and small entities involved in standards-based science education work and to include the work of groups having different philosophical and political perspectives. Keyword searches were supplemented by “free text” searches—that is, looking through titles and abstracts for key words and phrases. To prevent exclusion of potentially useful studies, the searches were intentionally overinclusive (e.g., including full text, rather than just titles and abstracts) and encompassed everything from January 1993 (the year in which the Benchmarks were published) through October 2001.

OCR for page 121
BOX 7-2 Key Words and Phrases Used to Identify Studies AAAS 2061 AAAS Benchmarks accountability + science assessment + science association + science education Benchmarks + science benchmarks for science literacy business + science education classroom assessment + science college entrance + science college placement + science curriculum + science district assessment + science district curriculum + science district standards + science Education standards + science industry + science education instructional materials + science instructional materials development + science local systemic change local systemic change initiative local systemic initiative LSC materials selection + science National Educational Goals Panel National Science Education Standards national standards + science NBPTS + science policy + science policy makers + science education policymakers + science education politicians + science education preservice + science Professional association + science education professional development + science professional organization + science education public + science education RSI rural systemic initiative Science education SSI standards + education + science standards-based + science standards-based reform + science state assessment + science state curriculum + science state standards + science state systemic initiative statewide systemic initiative student learning + science teacher certification + science teacher development + science teacher preparation + science teachers + science teaching + science teaching credential + science teaching practice + science text + science textbook + science textbooks + science urban systemic initiative USI Knowing that research regarding the NSES and Benchmarks was ongoing, the project staff also attempted to collect “gray” or “fugitive” literature that had not yet been published in journals or other hard copy formats. The primary strategy for this search was accessing and scanning items posted on the Web sites of the agencies and organizations listed in Box 7-1. In addition, science education researchers and officials responsible for managing government-funded research and evaluation programs were contacted and asked to suggest additional sources of material for consideration. When duplicates were deleted, these searches resulted in several hundred items concerning the NSES and the Benchmarks for Science Literacy.1 1   Subsequently in this chapter, reference to the National Science Education Standards is meant to imply both the NSES and the Benchmarks. The two are not distinguished because of their overlap.

OCR for page 121
FIGURE 7-1 Bibliography worksheet. IDENTIFYING ITEMS FOR INCLUSION The next step was to identify the items from this large collection that would provide evidence to address the research question: What does the research tell us about the influence of the NSES on various facets of the educational system, on opportunities for all students to learn, and on student learning? Explicit criteria for inclusion were defined and applied to each study to verify that the study actually addressed the research question. Only studies that met inclusion criteria were to be included in the bibliography and provided to the commissioned authors. To reduce bias in this process, a Bibliography Worksheet was created that defined explicit criteria for inclusion (Figure 7-1). Full documents were obtained for all items that included reference to the NSES or Benchmarks and one or more other key words used in the search, and a copy of the inclusion criteria chart was

OCR for page 121
attached to each. Either the project director, program officer, or a project intern (graduate student) scanned each document, noted the study design (i.e., criteria 2) and component of the system addressed (criteria 3) by highlighting the pertinent section(s) of the Worksheet, and categorized each document/study as I (meeting the criteria—documents that met at least one of the criteria in 2 and in 3), II (questionable, unclear), or III (not meeting criteria—for documents that did not meet at least one of the criteria in 2 and in 3). The project director reviewed all IIs and assigned them to the I or III categories, erring on the side of overinclusion to prevent exclusion of potentially useful studies. Many of the items categorized as III were fact sheets and classroom activities keyed to the NSES as opposed to studies that assess the NSES as a means of inducing change or that focused on outcomes of standards-based interventions (see criteria 2). The resulting 245 items included implementation or outcome studies that focused on one or more of the elements in the Framework shown on the Bibliography Worksheet.

OCR for page 121
THE ANNOTATIONS Each of the commissioned authors was sent copies of the papers that the staff had categorized as relevant to that author’s topic. In addition, staff assigned to each author the responsibility for annotating a number of papers. In general, the paper was assigned to the author for whom the study was most relevant, but studies addressing multiple components of the education system were distributed to equalize the load among authors. The commissioned authors agreed to evaluate the bibliographic entries relevant to their topics and to write each annotation to include the following: A statement regarding the nature of the work, whether the paper describes conceptual or experimental research, and the type(s) of data used by the researcher(s) The overall purpose of the paper, including methods the researchers used to collect and evaluate that data; The methodological rigor of the research enterprise; The inferences that were drawn; A statement regarding the findings in terms of the areas of influence listed in the inclusion criteria. Authors were encouraged to add other studies with which they were familiar to the original set of 245 items identified so that the project could provide a more comprehensive bibliography to the field. WHAT’S IN THE BIBLIOGRAPHY The next chapter contains the entire bibliography for the project, including (1) all 245 items identified through the literature search and processed using the inclusion criteria, (2) additional studies that were either published after the search or added by the authors, and (3) references that are cited in this publication for background, but that do not provide research evidence regarding the influence of the NSES. Annotations are included for the research studies that authors discuss in their review papers and that ground their arguments and conclusions. In cases where a series of studies are included, the most recent one is annotated and earlier ones are mentioned in that annotation. While all annotations have been written using the same guidelines (as noted above), they vary in style and length due to the fact that many different people wrote them. The authors’ rationale explaining how studies were singled out for inclusion in their reviews is contained within each author’s paper and is not part of the bibliography.