National Academies Press: OpenBook

On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations (2004)

Chapter: 2 Committee Procedures and Characteristics of Studies Reviewed

« Previous: 1 The Challenge and the Charge
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

2
Committee Procedures and Characteristics of Studies Reviewed

As explained in Chapter 1, our charge as committee members was to evaluate the quality of the evaluations of the 13 National Science Foundation-(NSF-) supported and 6 commercially generated mathematics curriculum materials.

We were not charged with and therefore did not:

  • Evaluate the curriculum materials directly

  • Rate or rank specific curricular programs

We recognize that both tasks could interest a broad constituency, but we believed that the field would profit from a careful, thorough review and summary of previous evaluations and research studies in relation to how previous work might inform and strengthen future efforts. We were aware that the mathematics education field lacks a clear consensus on what constitutes an effective curriculum and how to measure it to provide adequate, valid, and timely information to decision making bodies. It is appropriate to have a range of curricula from which to choose that represent a variety of preferences and values; when this is the case, decision making on curricular materials inevitably combines values and evidence-based reasoning. We did not intend to recommend the elimination of values in curricular decision making, but instead wished to contribute to efforts to increase the quality of evidence provided to the process.

Some readers may be disappointed by our not offering a “stamp of

Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

approval” on specific curricula or providing a report card, as others have done for state standards or tests (U.S. Department of Education, 1999; Achieve Inc., 2002). This decision was deliberate. As a committee of the National Research Council of The National Academies, our primary contribution was to clarify the phrases scientifically valid evaluation study and scientifically established as effective in the context of K-12 mathematics curricula. Such an analysis can elucidate the current knowledge of how these curricula were evaluated and help decision makers avoid judgment errors that are likely when the completeness or scientific rigor of evaluations of such materials is misunderstood.

Recognizing the complexity of judging curricular effectiveness, this report is designed to assist future evaluators and policy makers in designing and conducting evaluations that provide accurate, comprehensive, and valid advice to decision makers and practitioners on the efficacy of curriculum materials. Our primary goal was to advise our audiences on what could be learned from these initial efforts and how lessons learned, strategic decisions, adaptations in method, errors and weaknesses, and tentative patterns of results could further future evaluation efforts and decision making on curricular policy.

CURRICULA UNDER REVIEW

The following 13 mathematics curricula programs1 (The K-12 Mathematics Curriculum Center, 2002) were supported by the NSF, and the evaluations of these materials were reviewed by our committee:

Elementary School:

  • Everyday Mathematics (EM), Grades K-6 (SRA/McGraw-Hill)

  • Investigations in Number, Data and Space, Grades K-6 (Scott Foresman)

  • Math Trailblazers, Grades K-6 (Kendall/Hunt Publishing Company) Middle School:

  • Connected Mathematics Project (CMP), Grades 6-8 (Prentice Hall)

1  

Each of the NSF-supported curricula is at least a three-year core curriculum (National Science Foundation, 1989, 1991). A condition of second-year funding for the NSF-supported curricula materials was a firm commitment by a publisher for national dissemination (National Science Foundation, 1989, 1991).

Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
  • Mathematics in Context (MiC), Grades 5-8 (Holt, Rinehart and Winston)

  • MathScape: Seeing and Thinking Mathematically, Grades 6-8 (Glencoe/McGraw-Hill)

  • MathThematics (STEM), Grades 6-8 (McDougal Littell)

  • Middle School Mathematics Through Applications Project (MMAP) Pathways to Algebra and Geometry, Grades 6-8 (currently unpublished)

High School:

  • Contemporary Mathematics in Context (Core-Plus), Grades 9-12 (Glencoe/McGraw-Hill)

  • Interactive Mathematics Program (IMP), Grades 9-12 (Key Curriculum Press)

  • MATH Connections: A Secondary Mathematics Core Curriculum, Grades 9-12 (IT’S ABOUT TIME, Inc.)

  • Mathematics: Modeling Our World (MMOW/ARISE), Grades 9-12 (W.H. Freeman and Company)

  • Systemic Initiative for Montana Mathematics and Science (SIMMS) Integrated Mathematics, Grades 9-12 (Kendall/Hunt Publishing Company)

Given our expanded charge, we also included a few of the commercially published, non–NSF-funded curricula. We planned to select the curricula by market share; however, such data are highly proprietary and contested. An additional complicating factor was that most reports of market share are identified by publisher name rather than a particular product line. Publishers produce numerous overlapping and sometimes competing mathematics curriculum products, especially given recent acquisitions and mergers. Thus determining market share by program was problematic.

We located two sources of market share data independent of the publishers (Education Market Research, 2001; Weiss et al., 2001). In addition, we received testimonial data from other suppliers of widely used curricular materials in mathematics, including Key Curriculum Press, Saxon Publishers,2 and Texas Instruments. Among the six curricula, we sought representation from each of the four major textbook publishers:

2  

Saxon Publishers suggested Simba Information Inc.’s (2002, 2003) Print Publishing for the School Market 2001-2002 Yearly Report and Educational Marketer’s monthly newsletter as sources for market share data.

Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
  1. McGraw-Hill (including Direct Instruction, Frank Schaffler Publishing, Macmillan, Glencoe, SRA/Open Court, Everyday Mathematics, and the Wright Group)

  2. Reed Elsevier (including Harcourt, LexisNexis, Reinhard and Winston, Rigby, Steck-Vaughn, Reading Recovery, Heinemann, and Riverdeep)

  3. Vivendi (including Houghton Mifflin, McDougal Littell, Riverside Assessments, Sunburst Technology, and Great Source)3

  4. Pearson (including Addison Wesley Longman, Scott Foresman, Silver Burdett Ginn, Simon and Schuster, Globe Fearon, Modern Curriculum Press, Celebration Press, Dale Seymour Publications, Prentice Hall School, Waterford Early Reading, Waterford Early Math and Science, Sing, Spell, Read and Write)

We selected two publishers per grade band level (elementary, middle, and high school). Because our independent sources only identified publishers with the largest market share and not specific mathematics curriculum materials, we asked the publishers to select their curricula with the highest market share. The publishers then submitted the curricular materials and accompanying evaluation studies that they had conducted or were aware of for our review.

We analyzed evaluations of the following six commercially generated programs:

Elementary School:

  • Math K-5, 2002 (Addison Wesley/Scott Foresman)

  • Harcourt Math K-6 (Harcourt Brace)

Middle School:

  • Applications and Connections, Grades 6-8, 2001 (Glencoe/McGraw-Hill)

  • An Incremental Development, Sixth Grade, Eighth Grade (2nd edition) and An Incremental Development, Seventh Grade, Algebra 1/2 and Algebra (3rd edition) (Saxon)

3  

Houghton Mifflin Company was later sold by Vivendi (in December 2002). Houghton Mifflin Company sold Sunburst Technology in October 2002.

Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

High School:

  • Larson Series, Grades 9-12, 2002 (Houghton-Mifflin/McDougal Littell)

  • University of Chicago School Mathematics Project Integrated Mathematics, Grades 9-12, 2002 (Prentice Hall)

Prentice Hall4 was an exception and could not choose its curricular materials because we specifically asked for the secondary materials of the University of Chicago School Mathematics Project (UCSMP) to be part of our review. UCSMP was selected because its history and profile represented a “hybrid” between the two categories (NSF-supported and commercially generated curricular programs), and all of its development and research support for the first edition was provided by foundations.5 We chose UCSMP because, similar to the NSF curricula, its philosophy and program theory are aligned with the National Council of Teachers of Mathematics (NCTM) Standards (NCTM, 1989), although it preceded the NSF-supported curricula in its development period (Thompson et al., 2003). It also differs from the high school NSF-supported materials in that it preserves the traditional course sequence of algebra, geometry, algebra and trigonometry, and advanced mathematics, including newer topics such as statistics and discrete mathematics, whereas the other NSF-supported materials integrate across mathematical subfields at each grade level. UCSMP’s development was situated at a university, unlike any other commercially generated curricula. As a result, many published studies and doctoral dissertations were conducted on it.

DATA GATHERING

Information on evaluation studies of the 19 mathematics curricula projects was gathered in several ways. First, we found contacts for all

4  

UCSMP, recently acquired by Prentice Hall, received broad NSF support and its secondary program was first headed by Zalman Usiskin and Sharon Senk. It eventually included five components, including an elementary component that produced Everyday Math (headed by Max Bell, with NSF support, and included in this study), a professional development component with NSF support, and an evaluation component headed by Larry Hedges and Susan Stodolsky. In this review, UCSMP refers to the secondary program only, and Everyday Mathematics is coded as EM. Following our charge, UCSMP is categorized as a secondary commercially generated project, whereas EM is categorized as NSF supported. Both had private foundation funding, and for grades 4 through 6 materials, EM received NSF funding.

5  

Amoco Foundation, Carnegie Corporation of New York, and General Electric Foundation.

Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

curricula under review and requested copies of curricular materials and evaluation studies. We received the requested curriculum materials from all publishers except Harcourt Brace. Seventeen of the 19 curricula submitted public evaluation materials to our committee (except Math K-5, 2002 [Addison Wesley/Scott Foresman] and Harcourt Math K-6 [Harcourt Brace]). We requested that principal investigators from the NSF-supported mathematics curriculum projects send reports they had submitted to the NSF, as well as their own evaluation studies of their materials or others of which they were aware. We also gathered evaluation studies from all four mathematics NSF Implementation Centers (http://www.ehr.nsf.gov/esie/resources/impsites.asp). We then checked citations and bibliography entries from these studies for possible additional evaluations and acquired copies of new studies. Finally, we conducted library and web searches, and emailed several mathematics and mathematics education listservs requesting evaluation studies. We then obtained copies of pertinent studies. A total of 698 evaluation studies were found or submitted for consideration.

We held two evidence-gathering workshops in 2002. The two workshop panels addressed the following questions:

How would you define or evaluate effectiveness of a K-5, 6-8, or 9-12 NSF-supported or commercially generated mathematics curriculum?

What evidence would be needed? Be specific in terms of (1) primary and secondary variables, (2) methods of examining or measuring those variables, (3) research designs, and (4) other relationships under investigation.

The first workshop consisted of panels addressing specific topics:

  • Evaluation and cross-disciplinary frameworks on curriculum implementation in complex settings;

  • Developer, researcher, and evaluator perspectives of curriculum effectiveness;

  • The role of content analysis and research on learning in evaluating curricula effectiveness;

  • Consideration of professional development needs in curricular implementation; and

  • Curriculum decision making and evaluation in school settings.

The second workshop on commercially generated materials asked the same general questions, with two additional requests for comments:

  • How do you evaluate materials in relation to the quality and effectiveness of the materials themselves, including content analysis, theories of

Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

learning, and teaching? Discuss the role of authors and developers in the process of evaluation.

  • How does your company consider the issues of implementation in relation to effectiveness, such as professional development, high-stakes tests, standards, technology, equity, and the adoption of materials and marketing issues?

Much of the debate around curriculum quality in the mathematics and mathematics education community resulted in part because content analysis is an ill-defined concept; therefore, we solicited statements on this topic. Sixteen prominent mathematicians and mathematics educators from a variety of perspectives on content analysis were identified. We sent a written request and received statements from eight: Richard Askey, University of Wisconsin, Madison; Eric Gutstein, University of Illinois, Chicago; Roger Howe, Yale University; William McCallum, University of Arizona; R. James Milgram, Stanford University; Luis Ortiz-Franco, Chapman University; Deborah Schifter, Education Development Center; and Hung Hsi Wu, University of California, Berkeley. We asked for their advice on content analysis by addressing the following questions:

  • What should be included in a content analysis?

  • How would you judge the quality of a content analysis?

  • What is the definition of content analysis?

  • Does your response represent the intended and enacted curriculum?

  • What references are available in the field on this topic?

THE STUDY MATRIX

We included evaluation studies that focused on one or more of the 13 NSF-supported or 6 commercially generated mathematics curricula, and whose authorship and affiliation were identified. Evaluation studies also had to fall into one of the following categories: (1) Comparative Analysis, (2) Case Study, (3) Content Analysis, (4) Synthesis Study, (5) Background Information, Historical Documentation, or Report to the NSF; and (6) Informative Study (Chapters 4 through 6 provide category descriptions.) We did not wish to limit its initial review to published studies because the topic is relatively current and some papers may not yet have been published. Dissertations would have been excluded if only published studies had been chosen, and we believed these studies could contain useful information. Finally, we sought studies from the following classifications pertaining to curriculum implementation:

Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
  • Studies with specific student outcomes

  • Content analysis studies

  • Studies of classroom implementation and school environment

  • Studies of teacher knowledge, teacher characteristics, and professional development

We decided to add to these classifications as we identified additional relevant categories. None were found. The decision tree (Figure 2-1) illustrates the process for categorizing the evaluation studies.

We considered all 698 studies that were found or submitted. If the study met the criteria listed, it was added to the bibliography for review. If it did not meet these criteria, it was placed on a list along with the documented reasons for exclusion. A study whose inclusion was difficult to determine was submitted for committee review. The bibliography of studies that are included in our analysis appears in Appendix B. The 417 studies that met the inclusion criteria for categories 1 through 6 were entered into our database for review.

STUDY CHARACTERISTICS

Table 2-1 shows the distribution of studies by methodology and identifies them as NSF-supported, commercially generated, or UCSMP.

We identified studies that fit the categories of content analysis (n=36), comparative analysis (n=95), case studies (n=45), and synthesis (n=16) as particularly salient. These 192 studies formed the core of the review because they provided direct information pertinent to reviewing the evaluation on materials’ effectiveness. Therefore, a large percentage of studies initially considered did not meet the criteria and were excluded from further review. The categories of background information, historical documentation, reports to the NSF, or informative studies were eliminated from further review, though they remain a valuable source of information about program theory and decision making that affected evaluation study designs.

The number of studies in the commercial category was far smaller than the number of studies on the NSF-supported materials or UCSMP.6 Two factors seem to account for this disparity. First, many NSF- or foundation-supported curricula were required to provide evaluations. The NSF also funded some of these curriculum projects to conduct further evaluation studies. Second, the NSF and UCSMP materials were written primarily by

6  

The committee separated UCSMP from the NSF-supported and commercially generated materials because of its hybrid nature.

Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

FIGURE 2-1 Decision tree for categorizing evaluation studies.

Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

TABLE 2-1 Distribution of Types of Studies

Type of Study

Number of Studies

Percentage of Total

1. Comparative analysis

95

13.6

NSF-supported curricula

66

69.5

Commercially generated curricula

16

16.8

UCSMP

11

11.6

Not counted in above

2

2.1

2. Case

45

6.4

NSF-supported curricula

45

100.0

Commercially generated curricula

0

0.0

UCSMP

0

0.0

Not counted in above

0

0.0

3. Content analysis

36

5.2

NSF-supported curricula

17

47.2

Commercially generated curricula

1

2.8

UCSMP

12

33.3

Not counted in above

6

16.7

4. Synthesis

16

2.3

NSF-supported curricula

15

93.8

Commercially generated curricula

0

0.0

UCSMP

1

6.3

Not counted in above

0

0.0

5. and 6. Background information and Informative studies

225

32.2

7. Do not include

281

40.3

TOTAL

698

100.0

university faculty whose graduate students often conducted the studies as part of research toward their degrees. Finally, unlike the NSF-supported materials, commercial publishers often conducted market studies, which emphasize how potential purchasers will view the materials. Thus, many commercially generated studies were only marginally useful in evaluating curricular effectiveness.

The evaluation studies were distributed unevenly across the curricula (Table 2-2). Three of the five curricula with the most evaluation studies under review received additional NSF funding to conduct revisions.7 The elementary, Everyday Mathematics, and secondary components of the UCSMP materials followed.

A database was developed to summarize the studies. Each study was

7  

Connected Mathematics, Mathematics In Context, and Core-Plus.

Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

TABLE 2-2 Distribution of Curricula by Study Type

 

Type of Study*

Number of Appearances in Any Study

Curriculum Name

1.

2.

3.

4.

5. and 6.

NSF-supported curricula

Elementary

Everyday Mathematics

17

9

7

2

16

51

Investigations in Number, Data and Space

5

1

2

2

9

19

Math Trailblazers

1

1

1

2

6

11

Middle school

Connected Mathematics Project (CMP)

10

18

8

2

42

80

Mathematics in Context (MiC)

1

8

7

5

52

73

Math Thematics (STEM)

2

6

4

2

13

27

MathScape

0

2

1

1

5

9

Middle School

Mathematics Through Applications Project (MMAP)

0

0

0

1

7

8

High school

Contemporary Mathematics in Context (Core-Plus)

13

5

3

3

19

43

Interactive Mathematics Program (IMP)

17

2

4

2

12

37

Systemic Initiative for Montana Mathematics and Science (SIMMS)

5

1

2

2

10

20

Math Connections

2

0

2

2

6

12

Mathematics: Modeling Our World (MMOW/ARISE)

0

0

2

1

5

8

Commercially generated curricula

Elementary

Addison Wesley/Scott Foresman

0

0

2

0

1

3

Harcourt Brace

0

0

1

0

0

1

Middle school

Saxon

13

0

6

0

21

40

Glencoe/McGraw-Hill

1

0

2

0

4

7

High school

Prentice Hall/UCSMP

13

0

14

3

46

76

Houghton-Mifflin/ McDougal Littell

2

0

0

0

1

3

Number of evaluation studies

95

45

36

16

225

417

Number of times each curriculum is in each type of study

102

53

68

29

275

528

*Study types: (1.) Comparative Analysis, (2.) Case Study, (3.) Content Analysis, (4.) Synthesis, (5. and 6., respectively) Background Information and Informative Study.

Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

read and analyzed by two or more committee members, the Mathematical Sciences Education Board staff, or graduate students trained to search for and record the study characteristics listed in Box 2-1.

After initial review, we studied in depth the first four categories in Table 2-2 because these studies provided detailed evaluation data.

STUDY CHARACTERISTICS FOR CATEGORIES 1 THROUGH 4

Table 2-3 shows distribution of the published studies by type of study. Of the studies reviewed, only 22 percent were published in journals or books. Approximately 28 percent of comparative studies and 31 percent of case studies were unpublished doctoral dissertations. Although dissertations are unpublished, these studies have been vigorously screened and provided valuable insight into current evaluation data on the curricula under review.

TABLE 2-3 Distribution of Published Studies by Type of Study

Type of Study

Number of Studies

Published Study

Unpublished Study

Unpublished Thesis or Dissertation

1. Comparative analysis

95

15

53

27

NSF-supported curricula

66

12

39

15

Commercially generated curricula

16

0

7

9

UCSMP

11

3

6

2

Not counted in above

2

0

1

1

2. Case

45

11

20

14

NSF-supported curricula

45

11

20

14

Commercially generated curricula

0

0

0

0

UCSMP

0

0

0

0

Not counted in above

0

0

0

0

3. Content analysis

36

3

33

0

NSF-supported curricula

17

3

14

0

Commercially generated curricula

1

0

1

0

UCSMP

12

0

12

0

Not counted in above

6

0

6

0

4. Synthesis

16

14

2

0

NSF-supported curricula

15

13

2

0

Commercially generated curricula

0

0

0

0

UCSMP

1

1

0

0

Not counted in above

0

0

0

0

TOTAL

192

43

108

41

Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

BOX 2-1
Study Characteristics

Author(s)

Title and date of publication

Sample program(s) of interest

Comparison curriculum used program

Design of experiment

Author(s) background regarding study

Version of material

Published? Where?

Unit of analysis

Study of duration

Research question

Outcome measures: Student level

Standardized tests

Other measures (attitudes, absentee rates, or dropout rates)

Outcome measures: Teacher level

Content knowledge

Attitude

Student population: Sample and comparison

Total number of students

Gender

Race/ethnicity

Socioeconomic: Free or reduced lunch

Other

Teacher population: Sample and comparison

Total number of teachers

Hours of professional development received

Use of supplemental materials

Mathematics certified

Average number of years of teaching experience

School population: Sample and comparison

Total number of schools

Staff turnover rate

School location (urban, suburban, rural)

Enacted curriculum, measurement, and findings

Author(s) findings/claims

Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

TABLE 2-4 Distribution of Author’s Background by Type of Study

 

Author’s Background Regarding Studies

Type of Study

Number of Studies

Internal

External

Graduate Student

Unknown

1. Comparative analysis

95

35

27

29

4

NSF-supported curricula

66

20

24

19

3

Commercially generated curricula

16

9

1

6

0

UCSMP

11

5

2

4

0

Not counted in above

2

1

0

0

1

2. Case

45

10

14

14

7

NSF-supported curricula

45

10

14

14

7

Commercially generated curricula

0

0

0

0

0

UCSMP

0

0

0

0

0

Not counted in above

0

0

0

0

0

3. Content analysis

36

16

20

0

0

NSF-supported curricula

17

6

11

0

0

Commercially generated curricula

1

0

1

0

0

UCSMP

12

10

2

0

0

Not counted in above

6

0

6

0

0

4. Synthesis

16

16

0

0

0

NSF-supported curricula

15

15

0

0

0

Commercially generated curricula

0

0

0

0

0

UCSMP

1

1

0

0

0

Not counted in above

0

0

0

0

0

TOTAL

192

77

61

43

11

Table 2-4 shows the distribution of the author’s background by type of study. Across the four types of evaluation studies reviewed, 40 percent were done by authors internal to the curriculum project studied. If the study had more than one author, the authors were considered internal if one or more were related to the project (e.g., curriculum developer, curriculum project’s evaluator, and curriculum staff). Twenty-two percent of the studies were conducted by graduate students, who may be internal to the project because often they are the curriculum developer’s graduate students who perform the research studies. Because the relationship of the graduate student to the curriculum project is not always known, it can only be definitively stated that all authors were external to the project in 32 percent of the studies. The relationship of the author to the curriculum project is unknown in 6 percent of studies.

Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

TABLE 2-5 Distribution of the Study Size by Type of Studies

 

 

Study Size (Students)

Type of Study

Number of Studies

0-299

300-999

>1,000

n/a

1. Comparative analysis

95

42

28

23

2

2. Case

45

16

3

3

23

Table 2-5 shows the number of students sampled in each of the comparative and case studies (n=140). Studies with fewer than 300 students made up the largest percentage in both comparative and case studies: 44 and 36 percent, respectively. Only 19 percent of the 140 studies had a student sample greater than 1,000 students.

Table 2-6 shows the distribution of sample school locations by type of

TABLE 2-6 Distribution of Sample School Location by Type of Study

 

Percentage of Studies That Reported Location*

Type of Study

Number of Studies

Percent of Studies to Report Data

Urban

Suburban

Rural

Only the State/Region

1. Comparative analysis

95

88

36

46

31

37

NSF-supported curricula

66

89

29

38

24

35

Commercially generated curricula

16

81

31

44

19

38

UCSMP

11

91

40

60

40

20

Not counted in above

2

100

100

50

50

0

2. Case

45

71

27

55

30

30

NSF-supported curricula

45

71

27

55

30

30

Commercially generated curricula

0

0

0

0

0

0

UCSMP

0

0

0

0

0

0

Not counted in above

0

0

0

0

0

0

TOTAL

140

84

33

49

31

35

*Many studies report more than one of the following three types of locations: urban, suburban, or rural.

Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

study. Approximately 88 percent of comparative studies and 71 percent of case studies reported data on school location (urban, suburban, rural, or state/region). Suburban students were the largest percentage in both study types. Rural students were the smallest sample in comparative studies, which implies less information is known about curricular effectiveness in these regions. Most studies did not break down the sample by each of the school location types; thus an exact percentage of school types could not be determined. The data that were reported showed wide variation in demographics, although compared with overall representation in the country, minority populations were undersampled (U.S. Department of Education, 2001).

Content analysis studies are not included in Tables 2-5 and 2-6 because they do not report data on students or sample school locations. Synthesis studies are also excluded because they are summaries of multiple studies and typically did not report data on types of schools or students or include data from only some of the studies considered.

Only 19 percent of comparative and case studies provided detailed information on teachers (e.g., certification, years of teaching, or measures of content knowledge) shown in Table 2-7. Generally, comparison groups were based on matching student and not teacher characteristics. Therefore,

TABLE 2-7 Distribution of Studies That Reported Teacher Data by Type of Study

 

Percent of Reporting Studies

Type of Study

Number of Studies

Reported Number of Teachers

Reported Teacher Experience*

1. Comparative analysis

95

52

14

NSF-supported curricula

66

56

9

Commercially generated curricula

16

31

19

UCSMP

11

64

36

In 2 categories (not above)

2

0

0

2. Case

45

87

29

NSF-supported curricula

45

87

29

Commercially generated curricula

0

0

0

UCSMP

0

0

0

In 2 categories (not above)

0

0

0

TOTAL

140

63

19

*If a study reported on teacher certification or number of years of teaching, it was counted as “Reported Teacher Experience.”

Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×

some bias may be present in the studies in terms of use of volunteer teachers. A substantial percentage of the studies included some measure of implementation by including teacher logs, classroom observations, interviews, and so forth. However, few included any type of measure of quality of instruction, although case studies were more likely to provide insight into these factors than were comparative studies.

Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 18
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 19
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 20
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 21
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 22
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 23
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 24
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 25
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 26
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 27
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 28
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 29
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 30
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 31
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 32
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 33
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 34
Suggested Citation:"2 Committee Procedures and Characteristics of Studies Reviewed." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025.
×
Page 35
Next: 3 Framework for Evaluating Curricular Effectiveness »
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations Get This Book
×
Buy Paperback | $59.00 Buy Ebook | $47.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This book reviews the evaluation research literature that has accumulated around 19 K-12 mathematics curricula and breaks new ground in framing an ambitious and rigorous approach to curriculum evaluation that has relevance beyond mathematics. The committee that produced this book consisted of mathematicians, mathematics educators, and methodologists who began with the following charge:

  • Evaluate the quality of the evaluations of the thirteen National Science Foundation (NSF)-supported and six commercially generated mathematics curriculum materials;
  • Determine whether the available data are sufficient for evaluating the efficacy of these materials, and if not;
  • Develop recommendations about the design of a project that could result in the generation of more reliable and valid data for evaluating such materials.

The committee collected, reviewed, and classified almost 700 studies, solicited expert testimony during two workshops, developed an evaluation framework, established dimensions/criteria for three methodologies (content analyses, comparative studies, and case studies), drew conclusions on the corpus of studies, and made recommendations for future research.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!