2
Committee Procedures and Characteristics of Studies Reviewed

As explained in Chapter 1, our charge as committee members was to evaluate the quality of the evaluations of the 13 National Science Foundation-(NSF-) supported and 6 commercially generated mathematics curriculum materials.

We were not charged with and therefore did not:

  • Evaluate the curriculum materials directly

  • Rate or rank specific curricular programs

We recognize that both tasks could interest a broad constituency, but we believed that the field would profit from a careful, thorough review and summary of previous evaluations and research studies in relation to how previous work might inform and strengthen future efforts. We were aware that the mathematics education field lacks a clear consensus on what constitutes an effective curriculum and how to measure it to provide adequate, valid, and timely information to decision making bodies. It is appropriate to have a range of curricula from which to choose that represent a variety of preferences and values; when this is the case, decision making on curricular materials inevitably combines values and evidence-based reasoning. We did not intend to recommend the elimination of values in curricular decision making, but instead wished to contribute to efforts to increase the quality of evidence provided to the process.

Some readers may be disappointed by our not offering a “stamp of



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations 2 Committee Procedures and Characteristics of Studies Reviewed As explained in Chapter 1, our charge as committee members was to evaluate the quality of the evaluations of the 13 National Science Foundation-(NSF-) supported and 6 commercially generated mathematics curriculum materials. We were not charged with and therefore did not: Evaluate the curriculum materials directly Rate or rank specific curricular programs We recognize that both tasks could interest a broad constituency, but we believed that the field would profit from a careful, thorough review and summary of previous evaluations and research studies in relation to how previous work might inform and strengthen future efforts. We were aware that the mathematics education field lacks a clear consensus on what constitutes an effective curriculum and how to measure it to provide adequate, valid, and timely information to decision making bodies. It is appropriate to have a range of curricula from which to choose that represent a variety of preferences and values; when this is the case, decision making on curricular materials inevitably combines values and evidence-based reasoning. We did not intend to recommend the elimination of values in curricular decision making, but instead wished to contribute to efforts to increase the quality of evidence provided to the process. Some readers may be disappointed by our not offering a “stamp of

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations approval” on specific curricula or providing a report card, as others have done for state standards or tests (U.S. Department of Education, 1999; Achieve Inc., 2002). This decision was deliberate. As a committee of the National Research Council of The National Academies, our primary contribution was to clarify the phrases scientifically valid evaluation study and scientifically established as effective in the context of K-12 mathematics curricula. Such an analysis can elucidate the current knowledge of how these curricula were evaluated and help decision makers avoid judgment errors that are likely when the completeness or scientific rigor of evaluations of such materials is misunderstood. Recognizing the complexity of judging curricular effectiveness, this report is designed to assist future evaluators and policy makers in designing and conducting evaluations that provide accurate, comprehensive, and valid advice to decision makers and practitioners on the efficacy of curriculum materials. Our primary goal was to advise our audiences on what could be learned from these initial efforts and how lessons learned, strategic decisions, adaptations in method, errors and weaknesses, and tentative patterns of results could further future evaluation efforts and decision making on curricular policy. CURRICULA UNDER REVIEW The following 13 mathematics curricula programs1 (The K-12 Mathematics Curriculum Center, 2002) were supported by the NSF, and the evaluations of these materials were reviewed by our committee: Elementary School: Everyday Mathematics (EM), Grades K-6 (SRA/McGraw-Hill) Investigations in Number, Data and Space, Grades K-6 (Scott Foresman) Math Trailblazers, Grades K-6 (Kendall/Hunt Publishing Company) Middle School: Connected Mathematics Project (CMP), Grades 6-8 (Prentice Hall) 1   Each of the NSF-supported curricula is at least a three-year core curriculum (National Science Foundation, 1989, 1991). A condition of second-year funding for the NSF-supported curricula materials was a firm commitment by a publisher for national dissemination (National Science Foundation, 1989, 1991).

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations Mathematics in Context (MiC), Grades 5-8 (Holt, Rinehart and Winston) MathScape: Seeing and Thinking Mathematically, Grades 6-8 (Glencoe/McGraw-Hill) MathThematics (STEM), Grades 6-8 (McDougal Littell) Middle School Mathematics Through Applications Project (MMAP) Pathways to Algebra and Geometry, Grades 6-8 (currently unpublished) High School: Contemporary Mathematics in Context (Core-Plus), Grades 9-12 (Glencoe/McGraw-Hill) Interactive Mathematics Program (IMP), Grades 9-12 (Key Curriculum Press) MATH Connections: A Secondary Mathematics Core Curriculum, Grades 9-12 (IT’S ABOUT TIME, Inc.) Mathematics: Modeling Our World (MMOW/ARISE), Grades 9-12 (W.H. Freeman and Company) Systemic Initiative for Montana Mathematics and Science (SIMMS) Integrated Mathematics, Grades 9-12 (Kendall/Hunt Publishing Company) Given our expanded charge, we also included a few of the commercially published, non–NSF-funded curricula. We planned to select the curricula by market share; however, such data are highly proprietary and contested. An additional complicating factor was that most reports of market share are identified by publisher name rather than a particular product line. Publishers produce numerous overlapping and sometimes competing mathematics curriculum products, especially given recent acquisitions and mergers. Thus determining market share by program was problematic. We located two sources of market share data independent of the publishers (Education Market Research, 2001; Weiss et al., 2001). In addition, we received testimonial data from other suppliers of widely used curricular materials in mathematics, including Key Curriculum Press, Saxon Publishers,2 and Texas Instruments. Among the six curricula, we sought representation from each of the four major textbook publishers: 2   Saxon Publishers suggested Simba Information Inc.’s (2002, 2003) Print Publishing for the School Market 2001-2002 Yearly Report and Educational Marketer’s monthly newsletter as sources for market share data.

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations McGraw-Hill (including Direct Instruction, Frank Schaffler Publishing, Macmillan, Glencoe, SRA/Open Court, Everyday Mathematics, and the Wright Group) Reed Elsevier (including Harcourt, LexisNexis, Reinhard and Winston, Rigby, Steck-Vaughn, Reading Recovery, Heinemann, and Riverdeep) Vivendi (including Houghton Mifflin, McDougal Littell, Riverside Assessments, Sunburst Technology, and Great Source)3 Pearson (including Addison Wesley Longman, Scott Foresman, Silver Burdett Ginn, Simon and Schuster, Globe Fearon, Modern Curriculum Press, Celebration Press, Dale Seymour Publications, Prentice Hall School, Waterford Early Reading, Waterford Early Math and Science, Sing, Spell, Read and Write) We selected two publishers per grade band level (elementary, middle, and high school). Because our independent sources only identified publishers with the largest market share and not specific mathematics curriculum materials, we asked the publishers to select their curricula with the highest market share. The publishers then submitted the curricular materials and accompanying evaluation studies that they had conducted or were aware of for our review. We analyzed evaluations of the following six commercially generated programs: Elementary School: Math K-5, 2002 (Addison Wesley/Scott Foresman) Harcourt Math K-6 (Harcourt Brace) Middle School: Applications and Connections, Grades 6-8, 2001 (Glencoe/McGraw-Hill) An Incremental Development, Sixth Grade, Eighth Grade (2nd edition) and An Incremental Development, Seventh Grade, Algebra 1/2 and Algebra (3rd edition) (Saxon) 3   Houghton Mifflin Company was later sold by Vivendi (in December 2002). Houghton Mifflin Company sold Sunburst Technology in October 2002.

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations High School: Larson Series, Grades 9-12, 2002 (Houghton-Mifflin/McDougal Littell) University of Chicago School Mathematics Project Integrated Mathematics, Grades 9-12, 2002 (Prentice Hall) Prentice Hall4 was an exception and could not choose its curricular materials because we specifically asked for the secondary materials of the University of Chicago School Mathematics Project (UCSMP) to be part of our review. UCSMP was selected because its history and profile represented a “hybrid” between the two categories (NSF-supported and commercially generated curricular programs), and all of its development and research support for the first edition was provided by foundations.5 We chose UCSMP because, similar to the NSF curricula, its philosophy and program theory are aligned with the National Council of Teachers of Mathematics (NCTM) Standards (NCTM, 1989), although it preceded the NSF-supported curricula in its development period (Thompson et al., 2003). It also differs from the high school NSF-supported materials in that it preserves the traditional course sequence of algebra, geometry, algebra and trigonometry, and advanced mathematics, including newer topics such as statistics and discrete mathematics, whereas the other NSF-supported materials integrate across mathematical subfields at each grade level. UCSMP’s development was situated at a university, unlike any other commercially generated curricula. As a result, many published studies and doctoral dissertations were conducted on it. DATA GATHERING Information on evaluation studies of the 19 mathematics curricula projects was gathered in several ways. First, we found contacts for all 4   UCSMP, recently acquired by Prentice Hall, received broad NSF support and its secondary program was first headed by Zalman Usiskin and Sharon Senk. It eventually included five components, including an elementary component that produced Everyday Math (headed by Max Bell, with NSF support, and included in this study), a professional development component with NSF support, and an evaluation component headed by Larry Hedges and Susan Stodolsky. In this review, UCSMP refers to the secondary program only, and Everyday Mathematics is coded as EM. Following our charge, UCSMP is categorized as a secondary commercially generated project, whereas EM is categorized as NSF supported. Both had private foundation funding, and for grades 4 through 6 materials, EM received NSF funding. 5   Amoco Foundation, Carnegie Corporation of New York, and General Electric Foundation.

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations curricula under review and requested copies of curricular materials and evaluation studies. We received the requested curriculum materials from all publishers except Harcourt Brace. Seventeen of the 19 curricula submitted public evaluation materials to our committee (except Math K-5, 2002 [Addison Wesley/Scott Foresman] and Harcourt Math K-6 [Harcourt Brace]). We requested that principal investigators from the NSF-supported mathematics curriculum projects send reports they had submitted to the NSF, as well as their own evaluation studies of their materials or others of which they were aware. We also gathered evaluation studies from all four mathematics NSF Implementation Centers (http://www.ehr.nsf.gov/esie/resources/impsites.asp). We then checked citations and bibliography entries from these studies for possible additional evaluations and acquired copies of new studies. Finally, we conducted library and web searches, and emailed several mathematics and mathematics education listservs requesting evaluation studies. We then obtained copies of pertinent studies. A total of 698 evaluation studies were found or submitted for consideration. We held two evidence-gathering workshops in 2002. The two workshop panels addressed the following questions: How would you define or evaluate effectiveness of a K-5, 6-8, or 9-12 NSF-supported or commercially generated mathematics curriculum? What evidence would be needed? Be specific in terms of (1) primary and secondary variables, (2) methods of examining or measuring those variables, (3) research designs, and (4) other relationships under investigation. The first workshop consisted of panels addressing specific topics: Evaluation and cross-disciplinary frameworks on curriculum implementation in complex settings; Developer, researcher, and evaluator perspectives of curriculum effectiveness; The role of content analysis and research on learning in evaluating curricula effectiveness; Consideration of professional development needs in curricular implementation; and Curriculum decision making and evaluation in school settings. The second workshop on commercially generated materials asked the same general questions, with two additional requests for comments: How do you evaluate materials in relation to the quality and effectiveness of the materials themselves, including content analysis, theories of

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations learning, and teaching? Discuss the role of authors and developers in the process of evaluation. How does your company consider the issues of implementation in relation to effectiveness, such as professional development, high-stakes tests, standards, technology, equity, and the adoption of materials and marketing issues? Much of the debate around curriculum quality in the mathematics and mathematics education community resulted in part because content analysis is an ill-defined concept; therefore, we solicited statements on this topic. Sixteen prominent mathematicians and mathematics educators from a variety of perspectives on content analysis were identified. We sent a written request and received statements from eight: Richard Askey, University of Wisconsin, Madison; Eric Gutstein, University of Illinois, Chicago; Roger Howe, Yale University; William McCallum, University of Arizona; R. James Milgram, Stanford University; Luis Ortiz-Franco, Chapman University; Deborah Schifter, Education Development Center; and Hung Hsi Wu, University of California, Berkeley. We asked for their advice on content analysis by addressing the following questions: What should be included in a content analysis? How would you judge the quality of a content analysis? What is the definition of content analysis? Does your response represent the intended and enacted curriculum? What references are available in the field on this topic? THE STUDY MATRIX We included evaluation studies that focused on one or more of the 13 NSF-supported or 6 commercially generated mathematics curricula, and whose authorship and affiliation were identified. Evaluation studies also had to fall into one of the following categories: (1) Comparative Analysis, (2) Case Study, (3) Content Analysis, (4) Synthesis Study, (5) Background Information, Historical Documentation, or Report to the NSF; and (6) Informative Study (Chapters 4 through 6 provide category descriptions.) We did not wish to limit its initial review to published studies because the topic is relatively current and some papers may not yet have been published. Dissertations would have been excluded if only published studies had been chosen, and we believed these studies could contain useful information. Finally, we sought studies from the following classifications pertaining to curriculum implementation:

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations Studies with specific student outcomes Content analysis studies Studies of classroom implementation and school environment Studies of teacher knowledge, teacher characteristics, and professional development We decided to add to these classifications as we identified additional relevant categories. None were found. The decision tree (Figure 2-1) illustrates the process for categorizing the evaluation studies. We considered all 698 studies that were found or submitted. If the study met the criteria listed, it was added to the bibliography for review. If it did not meet these criteria, it was placed on a list along with the documented reasons for exclusion. A study whose inclusion was difficult to determine was submitted for committee review. The bibliography of studies that are included in our analysis appears in Appendix B. The 417 studies that met the inclusion criteria for categories 1 through 6 were entered into our database for review. STUDY CHARACTERISTICS Table 2-1 shows the distribution of studies by methodology and identifies them as NSF-supported, commercially generated, or UCSMP. We identified studies that fit the categories of content analysis (n=36), comparative analysis (n=95), case studies (n=45), and synthesis (n=16) as particularly salient. These 192 studies formed the core of the review because they provided direct information pertinent to reviewing the evaluation on materials’ effectiveness. Therefore, a large percentage of studies initially considered did not meet the criteria and were excluded from further review. The categories of background information, historical documentation, reports to the NSF, or informative studies were eliminated from further review, though they remain a valuable source of information about program theory and decision making that affected evaluation study designs. The number of studies in the commercial category was far smaller than the number of studies on the NSF-supported materials or UCSMP.6 Two factors seem to account for this disparity. First, many NSF- or foundation-supported curricula were required to provide evaluations. The NSF also funded some of these curriculum projects to conduct further evaluation studies. Second, the NSF and UCSMP materials were written primarily by 6   The committee separated UCSMP from the NSF-supported and commercially generated materials because of its hybrid nature.

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations FIGURE 2-1 Decision tree for categorizing evaluation studies.

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations TABLE 2-1 Distribution of Types of Studies Type of Study Number of Studies Percentage of Total 1. Comparative analysis 95 13.6 NSF-supported curricula 66 69.5 Commercially generated curricula 16 16.8 UCSMP 11 11.6 Not counted in above 2 2.1 2. Case 45 6.4 NSF-supported curricula 45 100.0 Commercially generated curricula 0 0.0 UCSMP 0 0.0 Not counted in above 0 0.0 3. Content analysis 36 5.2 NSF-supported curricula 17 47.2 Commercially generated curricula 1 2.8 UCSMP 12 33.3 Not counted in above 6 16.7 4. Synthesis 16 2.3 NSF-supported curricula 15 93.8 Commercially generated curricula 0 0.0 UCSMP 1 6.3 Not counted in above 0 0.0 5. and 6. Background information and Informative studies 225 32.2 7. Do not include 281 40.3 TOTAL 698 100.0 university faculty whose graduate students often conducted the studies as part of research toward their degrees. Finally, unlike the NSF-supported materials, commercial publishers often conducted market studies, which emphasize how potential purchasers will view the materials. Thus, many commercially generated studies were only marginally useful in evaluating curricular effectiveness. The evaluation studies were distributed unevenly across the curricula (Table 2-2). Three of the five curricula with the most evaluation studies under review received additional NSF funding to conduct revisions.7 The elementary, Everyday Mathematics, and secondary components of the UCSMP materials followed. A database was developed to summarize the studies. Each study was 7   Connected Mathematics, Mathematics In Context, and Core-Plus.

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations TABLE 2-2 Distribution of Curricula by Study Type   Type of Study* Number of Appearances in Any Study Curriculum Name 1. 2. 3. 4. 5. and 6. NSF-supported curricula Elementary Everyday Mathematics 17 9 7 2 16 51 Investigations in Number, Data and Space 5 1 2 2 9 19 Math Trailblazers 1 1 1 2 6 11 Middle school Connected Mathematics Project (CMP) 10 18 8 2 42 80 Mathematics in Context (MiC) 1 8 7 5 52 73 Math Thematics (STEM) 2 6 4 2 13 27 MathScape 0 2 1 1 5 9 Middle School Mathematics Through Applications Project (MMAP) 0 0 0 1 7 8 High school Contemporary Mathematics in Context (Core-Plus) 13 5 3 3 19 43 Interactive Mathematics Program (IMP) 17 2 4 2 12 37 Systemic Initiative for Montana Mathematics and Science (SIMMS) 5 1 2 2 10 20 Math Connections 2 0 2 2 6 12 Mathematics: Modeling Our World (MMOW/ARISE) 0 0 2 1 5 8 Commercially generated curricula Elementary Addison Wesley/Scott Foresman 0 0 2 0 1 3 Harcourt Brace 0 0 1 0 0 1 Middle school Saxon 13 0 6 0 21 40 Glencoe/McGraw-Hill 1 0 2 0 4 7 High school Prentice Hall/UCSMP 13 0 14 3 46 76 Houghton-Mifflin/ McDougal Littell 2 0 0 0 1 3 Number of evaluation studies 95 45 36 16 225 417 Number of times each curriculum is in each type of study 102 53 68 29 275 528 *Study types: (1.) Comparative Analysis, (2.) Case Study, (3.) Content Analysis, (4.) Synthesis, (5. and 6., respectively) Background Information and Informative Study.

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations read and analyzed by two or more committee members, the Mathematical Sciences Education Board staff, or graduate students trained to search for and record the study characteristics listed in Box 2-1. After initial review, we studied in depth the first four categories in Table 2-2 because these studies provided detailed evaluation data. STUDY CHARACTERISTICS FOR CATEGORIES 1 THROUGH 4 Table 2-3 shows distribution of the published studies by type of study. Of the studies reviewed, only 22 percent were published in journals or books. Approximately 28 percent of comparative studies and 31 percent of case studies were unpublished doctoral dissertations. Although dissertations are unpublished, these studies have been vigorously screened and provided valuable insight into current evaluation data on the curricula under review. TABLE 2-3 Distribution of Published Studies by Type of Study Type of Study Number of Studies Published Study Unpublished Study Unpublished Thesis or Dissertation 1. Comparative analysis 95 15 53 27 NSF-supported curricula 66 12 39 15 Commercially generated curricula 16 0 7 9 UCSMP 11 3 6 2 Not counted in above 2 0 1 1 2. Case 45 11 20 14 NSF-supported curricula 45 11 20 14 Commercially generated curricula 0 0 0 0 UCSMP 0 0 0 0 Not counted in above 0 0 0 0 3. Content analysis 36 3 33 0 NSF-supported curricula 17 3 14 0 Commercially generated curricula 1 0 1 0 UCSMP 12 0 12 0 Not counted in above 6 0 6 0 4. Synthesis 16 14 2 0 NSF-supported curricula 15 13 2 0 Commercially generated curricula 0 0 0 0 UCSMP 1 1 0 0 Not counted in above 0 0 0 0 TOTAL 192 43 108 41

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations BOX 2-1 Study Characteristics Author(s) Title and date of publication Sample program(s) of interest Comparison curriculum used program Design of experiment Author(s) background regarding study Version of material Published? Where? Unit of analysis Study of duration Research question Outcome measures: Student level Standardized tests Other measures (attitudes, absentee rates, or dropout rates) Outcome measures: Teacher level Content knowledge Attitude Student population: Sample and comparison Total number of students Gender Race/ethnicity Socioeconomic: Free or reduced lunch Other Teacher population: Sample and comparison Total number of teachers Hours of professional development received Use of supplemental materials Mathematics certified Average number of years of teaching experience School population: Sample and comparison Total number of schools Staff turnover rate School location (urban, suburban, rural) Enacted curriculum, measurement, and findings Author(s) findings/claims

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations TABLE 2-4 Distribution of Author’s Background by Type of Study   Author’s Background Regarding Studies Type of Study Number of Studies Internal External Graduate Student Unknown 1. Comparative analysis 95 35 27 29 4 NSF-supported curricula 66 20 24 19 3 Commercially generated curricula 16 9 1 6 0 UCSMP 11 5 2 4 0 Not counted in above 2 1 0 0 1 2. Case 45 10 14 14 7 NSF-supported curricula 45 10 14 14 7 Commercially generated curricula 0 0 0 0 0 UCSMP 0 0 0 0 0 Not counted in above 0 0 0 0 0 3. Content analysis 36 16 20 0 0 NSF-supported curricula 17 6 11 0 0 Commercially generated curricula 1 0 1 0 0 UCSMP 12 10 2 0 0 Not counted in above 6 0 6 0 0 4. Synthesis 16 16 0 0 0 NSF-supported curricula 15 15 0 0 0 Commercially generated curricula 0 0 0 0 0 UCSMP 1 1 0 0 0 Not counted in above 0 0 0 0 0 TOTAL 192 77 61 43 11 Table 2-4 shows the distribution of the author’s background by type of study. Across the four types of evaluation studies reviewed, 40 percent were done by authors internal to the curriculum project studied. If the study had more than one author, the authors were considered internal if one or more were related to the project (e.g., curriculum developer, curriculum project’s evaluator, and curriculum staff). Twenty-two percent of the studies were conducted by graduate students, who may be internal to the project because often they are the curriculum developer’s graduate students who perform the research studies. Because the relationship of the graduate student to the curriculum project is not always known, it can only be definitively stated that all authors were external to the project in 32 percent of the studies. The relationship of the author to the curriculum project is unknown in 6 percent of studies.

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations TABLE 2-5 Distribution of the Study Size by Type of Studies     Study Size (Students) Type of Study Number of Studies 0-299 300-999 >1,000 n/a 1. Comparative analysis 95 42 28 23 2 2. Case 45 16 3 3 23 Table 2-5 shows the number of students sampled in each of the comparative and case studies (n=140). Studies with fewer than 300 students made up the largest percentage in both comparative and case studies: 44 and 36 percent, respectively. Only 19 percent of the 140 studies had a student sample greater than 1,000 students. Table 2-6 shows the distribution of sample school locations by type of TABLE 2-6 Distribution of Sample School Location by Type of Study   Percentage of Studies That Reported Location* Type of Study Number of Studies Percent of Studies to Report Data Urban Suburban Rural Only the State/Region 1. Comparative analysis 95 88 36 46 31 37 NSF-supported curricula 66 89 29 38 24 35 Commercially generated curricula 16 81 31 44 19 38 UCSMP 11 91 40 60 40 20 Not counted in above 2 100 100 50 50 0 2. Case 45 71 27 55 30 30 NSF-supported curricula 45 71 27 55 30 30 Commercially generated curricula 0 0 0 0 0 0 UCSMP 0 0 0 0 0 0 Not counted in above 0 0 0 0 0 0 TOTAL 140 84 33 49 31 35 *Many studies report more than one of the following three types of locations: urban, suburban, or rural.

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations study. Approximately 88 percent of comparative studies and 71 percent of case studies reported data on school location (urban, suburban, rural, or state/region). Suburban students were the largest percentage in both study types. Rural students were the smallest sample in comparative studies, which implies less information is known about curricular effectiveness in these regions. Most studies did not break down the sample by each of the school location types; thus an exact percentage of school types could not be determined. The data that were reported showed wide variation in demographics, although compared with overall representation in the country, minority populations were undersampled (U.S. Department of Education, 2001). Content analysis studies are not included in Tables 2-5 and 2-6 because they do not report data on students or sample school locations. Synthesis studies are also excluded because they are summaries of multiple studies and typically did not report data on types of schools or students or include data from only some of the studies considered. Only 19 percent of comparative and case studies provided detailed information on teachers (e.g., certification, years of teaching, or measures of content knowledge) shown in Table 2-7. Generally, comparison groups were based on matching student and not teacher characteristics. Therefore, TABLE 2-7 Distribution of Studies That Reported Teacher Data by Type of Study   Percent of Reporting Studies Type of Study Number of Studies Reported Number of Teachers Reported Teacher Experience* 1. Comparative analysis 95 52 14 NSF-supported curricula 66 56 9 Commercially generated curricula 16 31 19 UCSMP 11 64 36 In 2 categories (not above) 2 0 0 2. Case 45 87 29 NSF-supported curricula 45 87 29 Commercially generated curricula 0 0 0 UCSMP 0 0 0 In 2 categories (not above) 0 0 0 TOTAL 140 63 19 *If a study reported on teacher certification or number of years of teaching, it was counted as “Reported Teacher Experience.”

OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations some bias may be present in the studies in terms of use of volunteer teachers. A substantial percentage of the studies included some measure of implementation by including teacher logs, classroom observations, interviews, and so forth. However, few included any type of measure of quality of instruction, although case studies were more likely to provide insight into these factors than were comparative studies.