National Academies Press: OpenBook

Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments (2004)

Chapter: 3 Participation in NAEP and Other Large-Scale Assessments

« Previous: 2 Characteristics of the Students, the Assessments, and Commonly Used Accomadations
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

3
Participation in NAEP and Other Large-Scale Assessments

A variety of factors can influence the rates at which students with disabilities and English language learners participate in large-scale assessments. Legislative mandates have established requirements that, in general, all such students should be included in statewide accountability programs, but these requirements do allow for the exclusion of a small number of students from assessments. These laws require that accommodations be provided for students who need them but offer relatively scant guidance for determining how and when the accommodations should be provided. The National Assessment of Educational Progress (NAEP) assessments have not been subject to the same legal requirements regarding participation, and until 1996 they did not permit accommodations. NAEP’s participation rates for students with disabilities and English language learners have thus lagged behind those of state assessment programs. In this chapter, we present the information available on participation rates in NAEP and statewide assessments and discuss the implications of the available data.

PARTICIPATION RATES FOR NAEP

NAEP’s Research Study on Providing Accommodations

In the early 1990s, students with individualized education programs (IEPs) could be excluded from NAEP if they were placed in general education classrooms (mainstreamed) less than 50 percent of the time or judged to be incapable of meaningful participation in the assessment. Some English language learners—defined as students whose native language was not English, who had been enrolled

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

in an English speaking school for less than three years, or who were judged incapable of meaningful participation in the assessment—could also be excluded. Specifically, a NAEP publication describes the former procedures for these two groups in the following way (U.S. Department of Education National Center for Education Statistics, 1997).

Prior to 1990, administrations of NAEP … relied on the judgment of school administrators as to whether or not the student could take the assessment. Beginning with the 1990 NAEP, schools were given guidelines informing them that they may exclude a student with a disability if the student is mainstreamed less than 50% of the time and is judged incapable of participating meaningfully in the assessment, OR, the IEP team or equivalent group determined that the student is incapable of participating meaningfully in the assessment. Schools were instructed to include students with disabilities if school staff believed the students were capable of taking the assessment. Schools were also instructed that when there was doubt, students should be included. (pp. 14-15)

The NAEP procedures used prior to 1990 allowed schools to exclude sampled students if they were LEP and if local school personnel judged the students incapable of participating meaningfully in the assessment Beginning in 1990, NAEP instructions to schools for excluding LEP students from the assessment required the following conditions to be met: the student is a native speaker of a language other than English AND the student has been enrolled in an English-speaking school for less than 2 years (not including bilingual education programs) AND school officials judged the student to be incapable of taking the assessment. The guidelines also stated that when in doubt, the student was to be included in the assessment. (p. 41)

NAEP’s sponsors took the first step in making the assessment more inclusive when they adopted a series of resolutions that established a plan for conducting research on the effects of including students with disabilities and English language learners in the assessment.

In these resolutions, NAEP’s sponsors articulated the dual priorities of including students who can meaningfully take part in the assessment while maintaining the integrity of the trend data that are considered a key component of NAEP. The resolution and research plan provided what NAEP officials have described as both “a bridge to the future,” because it would make NAEP more inclusive, and “a bridge to the past” (NRC, 2002a), because it would allow NAEP to continue to provide meaningful trend information. Protection of the capacity to report trend data was considered a necessary constraint on any changes in policies and procedures.

NAEP conducted a series of pilot studies in the early 1990s to examine the feasibility of allowing students to participate in the assessment with accommodations and then, in conjunction with the 1996 mathematics assessment, initiated a research plan. This plan called for data to be collected for three samples of tested students, referred to as S1, S2, and S3. For the S1 sample, administration proce-

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

dures were handled in the same way as in the early 1990s, and students with special needs could be excluded from the assessment. For the S2 sample, revisions aimed at increasing participation were made to the criteria given to schools for determining whether to include students with special needs, but no accommodations or adaptations were offered. For state NAEP, the schools were split between S1 and S2. For national NAEP, a third sample of schools, S3, was identified, in which the revised inclusion criteria were used, and accommodations were permitted for students with disabilities and English language learners. These students were allowed to participate with the accommodations that they routinely received in their state or district testing.

Analyses of the 1996 data revealed no differences in inclusion rates between the S1 and S2 samples, so the S1 criteria were discontinued, and further research was based on samples of schools that applied either the revised criteria. Comparison of the S2 and S3 samples provided the opportunity to examine the net effects of both types of changes, both more lenient criteria for inclusion and the use of accommodations.

The research continued with the 1998 national and state NAEP reading assessment and the 2000 assessments (math and science at the national level in grades 4, 8, and 12 and at the state level in grades 4 and 8; reading at the national level in grade 4). Analyses of the 1998 and 2000 data revealed that providing accommodations did increase the number of students with disabilities and English language learners included in NAEP in grades 4 and 8.

Table 3-1 presents information from the research study on the participation rates of students with disabilities and English language learners in NAEP’s reading and math assessments for fourth graders. Participation rates (column 6) were calculated by dividing the number of students assessed (column 5) by the number identified (column 4) and multiplying by 100. Thus, the table shows that the percentages of students with disabilities who participated in NAEP’s fourth grade reading assessment were 13.8 percent in 1992 and 34.1 percent in 1994. These were years in which students with disabilities and English language learners could participate but no accommodations were allowed. Participation rates for the S2 (accommodations not allowed) and S3 (accommodations allowed) study samples are displayed separately for each of the years the study was in place.

Comparisons of the participation rates in a given year demonstrate the impact of providing accommodations. For example, for the 1998 fourth grade reading assessment, the participation rate for students with disabilities was 6.3 percentage points higher for the S3 sample than for the S2 sample. The median differences over assessment years between the participation rates when accommodations were not allowed and were allowed were 17.7 percent for students with disabilities and 21.3 percent for English language learners.

Table 3-2 presents this same information from the research study on the participation rates of students with disabilities and English language learners in NAEP’s reading and math assessments for eighth graders. For eighth graders, the

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

TABLE 3-1 Results from NAEP’s Research: Participation of Students with Disabilities (SWD) and English Language Learners (ELL) in Fourth Grade NAEP Reading and Math Assessments

(1)

(2)

(3)

(4)

(5)

(6)

 

 

 

Students Identified

Students Assessed

Percent Participating

Assessment

Year

Accommodation Permitted

SWD

ELL

SWD

ELL

SWD

ELL

Reading

1992

No

1,149

945

159

110

13.8

11.6

 

1994

No

1,039

623

354

255

34.1

40.9

 

1998—S2a

No

490

527

243

204

49.6

38.7

 

1998—S3b

Yes

558

446

312

279

55.9

62.6

 

Differences in participation rates for two 1998 samples

 

6.3

23.9

 

2000—S2a

No

524

356

229

215

43.7

60.4

 

2000—S3b

Yes

510

446

317

287

62.2

64.4

 

Differences in participation rates for two 2000 samples

 

18.5

4.0

Math

1992

No

1,163

939

173

104

14.9

11.1

 

1996—S2a

No

359

142

206

75

57.4

52.8

 

1996—S3b

Yes

424

308

315

222

74.3

72.1

 

Differences in participation rates for two 1996 samples

 

16.9

19.3

 

2000—S2a

No

672

454

292

265

43.5

58.4

 

2000—S3b

Yes

706

472

526

385

74.5

81.6

 

Differences in participation rates for two 2000 samples

 

31.0

23.2

aResults from split sample study: Students taking this assessment were NOT allowed accommodations.

bResults from split sample study: Students taking this assessment WERE allowed accommodations.

SOURCE: Available: http://nces.ed.gov.

median differences over assessment years in the participation rates when accommodations were not allowed and were allowed were 21.0 percent for students with disabilities and 13.2 percent for English language learners.

NAEP’s Reporting of Participation Rates

The committee examined reports from several administrations of NAEP to gather information on participation rates for the most recent assessment. Table 3-3 presents the national data for students with disabilities and English language learners for the 1998-2002 reading assessment. Table 3-4 shows results by state for students with disabilities for the grade 4 reading assessment in 1998 and 2002, and Table 3-5 shows these results for English language learners. We note that the method for displaying the results presented in these tables has been improved from previous years. In the past, participation rates at the national level were reported separately for students with disabilities and English language learners,

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

TABLE 3-2 Results from NAEP’s Research: Participation of Students with Disabilities (SWD) and English Language Learners (ELL) in Eighth Grade NAEP Reading and Math Assessments

(1)

(2)

(3)

(4)

(5)

(6)

 

 

 

Students Identified

Students Assessed

Percent Participating

Assessment

Year

Accommodation Permitted

SWD

ELL

SWD

ELL

SWD

ELL

Reading

1992

No

1,522

836

199

86

13.1

10.3

 

1994

No

1,323

444

344

121

26.0

27.3

 

1998—S2a

No

975

449

451

315

46.3

70.2

 

1998—S3b

Yes

865

447

582

338

67.3

75.6

 

Differences in participation rates for two 1998 samples

 

21.0

5.4

Math

1992

No

1,538

838

215

88

14.0

10.5

 

1996a

No

310

106

161

68

51.9

64.2

 

1996b

Yes

557

226

374

175

67.1

77.4

 

Differences in participation rates for two 1996 samples

 

15.2

13.2

 

2000a

No

1,316

551

597

341

45.4

61.9

 

2000b

Yes

1,206

471

804

368

66.7

78.1

 

Differences in participation rates for two 2000 samples

 

21.3

16.2

aResults from split sample study: Students taking this assessment were NOT allowed accommodations.

bResults from split sample study: Students taking this assessment WERE allowed accommodations.

SOURCE: Available: http://nces.ed.gov.

but NAEP state-level reports combined participation data for the two groups. An example of the way these data were previously presented is shown in Table 3-6. With the state-level data for these two groups combined, it was impossible either to track participation rates separately for the two groups or to make comparisons between participation rates in state assessments and in NAEP (incomplete data from state assessments have also been an impediment to making these comparisons; that issue is discussed below).

The committee had intended to make a recommendation about the reporting of state-level participation rates, but before our deliberations were completed, NAEP for the first time presented the participation rates for students with disabilities and English language learners separately in both state and national reports. In the committee’s judgment, the revised version of the NAEP tables is a significant improvement and we encourage NAEP’s sponsors to continue to provide these data.

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

TABLE 3-3 Students with Disabilities (SWD) and/or Limited English Proficient (LEP) Students Identified, Excluded, and Assessed, When Accommodations Were Permitted, Grades 4, 8, and 12 Public and Nonpublic Schools: 1998–2002 for Reading

 

1998

2000

2002

 

Number of Students

Weighted Percentage of Students Sampled

Number of Students

Weighted Percentage of Students Sampled

Number of Students

Weighted Percentage of Students Sampled

Grade 4

 

 

 

 

 

 

SD and/or LEP students

 

 

 

 

 

 

Identified

973

16

906

18

28,073

19

Excluded

393

6

316

6

10,307

6

Assessed

580

10

590

12

17,766

13

Without accommodations

413

7

476

10

11,913

9

With accommodations

167

3

114

2

5,853

4

SD students

 

 

 

 

 

 

Identified

558

10

510

11

19,936

12

Excluded

246

4

193

4

8,042

5

Assessed

312

6

317

7

11,894

7

Without accommodations

179

3

209

5

6,631

4

With accommodations

133

3

108

2

5,263

3

LEP students

 

 

 

 

 

 

Identified

446

6

446

8

10,334

8

Excluded

167

2

159

3

3,410

2

Assessed

279

4

287

5

6,924

6

Without accommodations

238

3

273

5

6,020

6

With accommodations

41

1

14

#

904

1

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

Grade 8

 

 

 

 

 

 

SD and/or LEP students

 

 

 

 

 

 

Identified

1,252

12

20,137

17

Excluded

368

4

7,135

5

Assessed

884

9

13,002

11

Without accommodations

678

6

8,598

8

With accommodations

206

2

4,404

4

SD students

 

 

 

 

 

 

Identified

865

10

16,159

12

Excluded

283

3

5,939

4

Assessed

582

7

10,220

8

Without accommodations

404

5

6,074

5

With accommodations

178

2

4,146

3

LEP students

 

 

 

 

 

 

Identified

447

3

5,516

6

Excluded

109

1

1,907

2

Assessed

338

2

3,609

4

Without accommodations

307

2

3,113

4

With accommodations

31

#

496

#

Grade 12

 

 

 

 

 

 

SD and/or LEP students

 

 

 

 

 

 

Identified

975

7

1,556

12

Excluded

327

2

616

4

Assessed

648

5

940

8

Without accommodations

532

4

673

6

With accommodations

116

1

267

2

SD students

 

 

 

 

 

 

Identified

649

6

1,231

9

Excluded

285

2

535

3

Assessed

364

4

696

6

Without accommodations

266

3

446

4

With accommodations

98

1

250

2

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

 

1998

2000

2002

 

Number of Students

Weighted Percentage of Students Sampled

Number of Students

Weighted Percentage of Students Sampled

Number of Students

Weighted Percentage of Students Sampled

LEP students

 

 

 

 

 

 

Identified

353

2

419

3

Excluded

58

#

125

1

Assessed

295

2

294

3

Without accommodations

277

2

266

2

With accommodations

18

#

28

#

—Data were not collected at grades 8 and 12 in 2000.

# Percentage rounds to zero.

NOTE: Within each grade level, the combined SD/LEP portion of the table is not a sum of the separate SD and LEP portions becaus e some students were identified as both SD and LEP. Such students would be counted separately in the bottom portions but counted only once in the top portion.

Within each portion of the table, percentages may not add to totals, due to rounding.

The number of students at grades 4 and 8 are larger in 2002 than in previous years because the 2002 national sample was based on the combined sample of students in each participating state, plus an additional sample from nonparticipating states as well as a sample from private schools.

SOURCE: From U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress, 1998, 2000, and 2002 Reading Assessments.

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

TABLE 3-4 Percentage of Students with Disabilities (SD) Identified, Excluded, and Assessed, When Accommodations Were Permitted, Grade 4 Public Schools: By State, 1998 and 2002 for Reading

 

1998 SD Students

2002 SD Students

 

Identified

Excluded

Assessed

Assessed without Accommodations

Assessed with Accommodations

Identified

Excluded

Assessed

Assessed without Accommodations

Assessed with Accommodations

Nation (Public)

12

5

7

4

3

13

5

8

4

4

Alabama

13

8

4

3

1

13

2

11

8

2

Arizona

10

5

5

4

1

11

5

7

5

2

Arkansas

10

4

6

4

2

12

4

7

5

2

California

6

3

2

2

1

7

3

4

3

1

Connecticut

14

7

7

4

3

13

4

9

4

6

Delaware

14

1

12

9

4

15

7

8

3

5

Florida

14

5

9

5

4

17

5

13

6

7

Georgia

9

4

6

3

3

10

3

7

4

3

Hawaii

10

4

7

5

1

12

4

8

3

4

Idaho

13

4

9

7

2

Illinois

10

3

6

4

2

13

4

9

4

5

Indiana

12

4

8

6

2

Iowa

14

5

9

6

3

15

7

8

3

5

Kansas

9

3

6

3

3

14

4

10

4

5

Kentucky

12

7

5

3

2

11

8

4

2

1

Louisiana

14

7

7

2

5

19

10

8

3

5

Maine

15

7

7

4

3

16

6

10

5

6

Maryland

11

5

6

2

4

12

6

6

4

2

Massachusetts

16

4

12

7

5

16

4

12

3

9

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

 

1998 SD Students

2002 SD Students

 

Identified

Excluded

Assessed

Assessed without Accommodations

Assessed with Accommodations

Identified

Excluded

Assessed

Assessed without Accommodations

Assessed with Accommodations

Michigan

9

5

3

2

1

11

7

4

3

1

Minnesota

12

3

9

6

3

13

4

10

6

3

Mississippi

7

4

3

2

#

7

4

3

2

1

Missouri

14

6

7

3

4

15

8

7

4

3

Montana

10

2

7

5

2

13

5

8

4

4

Nebraska

18

4

13

7

6

Nevada

10

6

4

4

1

12

5

7

5

2

New Mexico

14

7

7

5

2

15

7

9

6

3

New York

9

4

5

1

4

14

6

8

2

5

North Carolina

14

6

8

2

6

17

10

6

3

4

North Dakota

16

5

11

8

3

Ohio

13

8

5

3

2

Oklahoma

13

9

5

3

1

17

5

13

8

5

Oregon

14

4

10

6

4

16

5

10

7

3

Pennsylvania

13

4

9

4

5

Rhode Island

14

5

10

6

3

19

3

15

6

10

South Carolina

15

7

8

5

3

16

4

11

8

3

Tennessee

12

3

9

7

2

11

3

8

6

1

Texas

14

7

8

5

2

14

8

6

5

2

Utah

10

4

6

4

1

12

4

7

5

3

Vermont

13

5

9

3

6

Virginia

14

6

8

4

4

14

8

6

3

3

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

Washington

11

4

8

5

3

13

4

9

6

4

West Virginia

12

8

4

2

1

15

10

5

3

2

Wisconsin

13

7

6

4

2

13

6

8

3

4

Wyoming

13

3

10

6

4

14

2

12

5

7

Other Jurisdictions

 

 

 

 

 

 

 

 

 

 

District of Columbia

10

6

4

2

2

14

7

7

3

4

DDESS1

7

3

4

2

2

10

3

7

3

4

DoDDS2

6

2

4

3

1

9

2

7

4

3

Guam

7

4

3

2

1

Virgin Islands

4

3

1

1

0

3

1

2

2

#

—Indicates that the jurisdiction did not participate.

# Percentage rounds to zero.

Indicates that the jurisdiction did not meet one or more of the guidelines for school participation in 2002.

1 Department of Defense Domestic Dependent Elementary and Secondary Schools.

2 Department of Defense Dependents Schools (Overseas).

NOTE: Percentages may not add to totals, due to rounding.

SOURCE: From U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational

Progress, 1998 and 2002 Reading Assessments.

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

TABLE 3-5 Percentage of Limited English Proficient (LEP) Students Identified, Excluded, and Assessed, When Accommodations Were Permitted, Grade 4 Public Schools: By State, 1998 and 2002 for Reading

 

1998 LEP Students

2002 LEP Students

 

Identified

Excluded

Assessed

Assessed without Accommodations

Assessed with Accommodations

Identified

Excluded

Assessed

Assessed without Accommodations

Assessed with Accommodations

Nation (Public)

5

2

3

3

1

9

2

7

6

1

Alabama

#

#

#

0

#

1

#

1

1

#

Arizona

14

6

7

6

1

21

5

16

15

1

Arkansas

1

1

1

1

0

3

1

3

3

#

California

26

12

14

13

1

29

3

26

26

#

Connecticut

5

4

1

1

#

4

2

2

2

#

Delaware

3

#

2

2

#

3

2

1

1

#

Florida

5

1

3

3

#

10

3

7

5

2

Georgia

2

1

#

#

#

4

1

2

2

#

Hawaii

6

2

4

4

0

8

2

6

4

1

Idaho

7

1

6

5

#

Illinois

5

3

2

2

#

9

4

5

4

1

Indiana

2

1

1

1

0

Iowa

1

1

1

1

0

2

1

1

1

#

Kansas

3

1

2

2

#

7

2

6

4

2

Kentucky

1

#

#

#

#

1

#

#

#

#

Louisiana

1

1

1

1

0

1

1

1

#

#

Maine

#

0

#

#

#

1

#

#

#

#

Maryland

2

1

2

1

#

3

2

1

1

#

Massachusetts

4

2

2

2

1

4

2

2

1

1

Michigan

2

1

1

1

#

3

1

2

2

#

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

Minnesota

4

1

3

3

1

7

2

5

4

1

Mississippi

#

0

#

#

0

#

#

#

#

0

Missouri

1

#

#

#

#

2

1

1

1

#

Montana

#

0

#

0

#

2

1

1

1

#

Nebraska

4

2

3

2

#

Nevada

10

6

4

4

#

18

7

11

10

1

New Mexico

16

4

12

11

1

27

6

21

19

2

New York

5

4

1

1

0

6

3

3

1

1

North Carolina

2

1

1

1

#

5

3

1

1

1

North Dakota

2

1

2

1

#

Ohio

1

1

1

1

0

Oklahoma

2

#

1

1

0

5

1

4

3

1

Oregon

7

2

5

4

1

12

4

8

6

2

Pennsylvania

2

1

1

1

#

Rhode Island

6

3

4

3

1

9

3

5

4

2

South Carolina

1

#

1

1

0

2

1

1

1

#

Tennessee

1

1

#

#

0

3

1

3

3

#

Texas

13

7

6

6

#

16

5

11

10

1

Utah

5

2

3

2

#

9

3

7

5

1

Vermont

2

#

1

1

#

Virginia

2

1

1

1

1

6

3

3

2

1

Washington

4

2

3

2

#

3

1

2

2

#

West Virginia

#

#

#

#

0

#

#

#

#

0

Wisconsin

3

1

2

1

#

6

3

3

2

1

Wyoming

1

1

#

#

#

5

1

4

3

1

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

 

1998 LEP Students

2002 LEP Students

 

Identified

Excluded

Assessed

Assessed without Accommodations

Assessed with Accommodations

Identified

Excluded

Assessed

Assessed without Accommodations

Assessed with Accommodations

Other Jurisdictions

 

 

 

 

 

 

 

 

 

 

District of Columbia

7

3

4

2

1

7

3

4

3

2

DDESS1

1

1

#

0

#

6

2

4

3

1

DoDDS2

2

1

1

1

#

8

1

7

6

1

Guam

36

5

31

25

6

Virgin Islands

4

2

2

1

1

5

2

3

3

1

—Indicates that the jurisdiction did not participate.

# Percentage rounds to zero.

Indicates that the jurisdiction did not meet one or more of the guidelines for school participation in 2002.

1 Department of Defense Domestic Dependent Elementary and Secondary Schools.

2 Department of Defense Dependents Schools (Overseas).

NOTE: Percentages may not add to totals, due to rounding.

SOURCE: From U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational

Progress, 1998 and 2002 Reading Assessments.

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

PARTICIPATION RATES FOR STATE ASSESSMENTS

The committee attempted to obtain information on the participation rates of the two student groups in statewide assessment programs in order to make comparisons with NAEP’s participation rates. However, we discovered that this information is not readily available. The most comprehensive source of data on state assessments is maintained by the Council of Chief State School Officers through their annual survey of state student assessment programs (Council of Chief State School Officers, 2002). With this survey, the council collects information about exemptions from statewide assessments; that is, the survey asks states if the number of special education or limited English proficient exemptions increased, decreased, or stayed the same over the past two to three years. However, the survey report does not provide data that could be used to calculate participation rates.

Two additional sources were identified. Some information on participation rates for students with disabilities in state assessments is available from the National Center on Education Outcomes. Based on their review of states’ biennial performance reports to the U.S. Department of Education, Thurlow et al. (2002) were able to obtain enough information for each state to calculate participation rates for all of the states. However, data were not provided for all of the tests states administer or in all of the grades assessed. Thus, it was not possible to examine these participation rates by grade level or by the subject matter of the test. Thurlow et al. reported a participation rate for every state, but in some states it is the only rate that could be calculated, while in others it is the highest of the rates calculated.

In the Thurlow report, state participation rates were calculated as follows: the numerator was the number of students with disabilities participating in the assessment; the denominator was the number of participating students with disabilities, plus the number taking the alternate assessment, plus the number not tested, which should equal the total number of students with disabilities enrolled in special education services in the state. Based on these calculations, the participation rates for students with disabilities were 90 percent or higher in 19 states, between 75 and 89 percent in 17 states, between 50 and 74 percent in 5 states, and between 25 and 49 percent in 1 state.

According to the authors of that report, the required analyses were quite complex, and determining the denominator for the participation rate was challenging. This was the first time analyses like this had been undertaken, primarily because in the past the Department of Education requirement to report results using a standard structure was not in place (M. Thurlow, personal communication, December 16, 2003).

In addition, Education Week (Education Week, 2004) recently reported the results of a survey of the states and the District of Columbia designed to gather data on the inclusion of students with disabilities in state testing and accountability

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

TABLE 3-6 Percentage of Students with Disabilities (SD) and/or Limited English Proficient (LEP) Students Identified, Excluded, and Assessed, When Accommodations Were Permitted, Grade 4 Public Schools: By State, 1998 and 2002 for Reading

 

1998 SD and/or LEP Students

 

Identified

Excluded

Assessed

Assessed without Accommodations

Assessed with Accommodations

All Students Assessed without Accommodations

Nation (Public)

18

7

11

7

3

90

Alabama

13

8

4

3

1

90

Arizona

22

10

12

10

1

88

Arkansas

11

5

6

4

2

93

California

31

14

16

15

1

84

Connecticut

18

10

8

5

3

87

Delaware

16

1

15

11

4

95

Florida

18

6

12

8

5

89

Georgia

11

5

6

3

3

93

Hawaii

15

5

10

9

1

94

Idaho

Illinois

14

6

8

6

2

92

Indiana

Iowa

15

5

10

7

3

92

Kansas

12

4

8

5

4

93

Kentucky

13

7

5

3

2

90

Louisiana

15

7

8

3

5

88

Maine

15

7

7

4

3

90

Maryland

13

6

8

4

4

90

Massachusetts

19

5

14

9

5

90

Michigan

10

6

4

3

1

93

Minnesota

15

3

12

9

3

94

Mississippi

7

4

3

2

#

95

Missouri

14

6

8

3

4

89

Montana

10

2

7

5

2

96

Nebraska

Nevada

20

11

9

8

1

88

New Mexico

28

9

18

16

2

88

New York

14

7

7

2

4

88

North Carolina

15

7

9

3

6

88

North Dakota

Ohio

Oklahoma

15

9

6

5

1

90

Oregon

20

6

14

10

4

90

Pennsylvania

Rhode Island

20

7

13

9

4

89

South Carolina

16

8

9

6

3

90

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

2002 SD and/or LEP Students

Identified

Excluded

Assessed

Assessed without Accommodations

Assessed with Accommodations

All Students Assessed without Accommodations

21

7

14

10

4

89

14

3

12

9

2

95

28

8

21

18

3

90

14

5

10

8

2

93

34

5

29

28

1

94

16

5

11

5

6

89

17

8

9

4

5

87

25

7

18

10

8

85

13

4

9

6

3

93

18

6

12

7

5

89

17

4

13

11

2

93

20

7

14

8

6

87

13

5

9

7

2

93

16

8

8

3

5

87

19

5

14

7

7

88

12

8

4

3

1

91

19

10

9

3

6

84

17

6

11

5

6

88

14

7

7

5

2

92

19

6

13

4

9

85

14

7

6

5

1

92

19

5

13

10

4

91

7

4

3

2

1

95

16

9

8

4

3

88

15

6

8

4

4

89

21

5

15

9

6

88

27

10

17

14

3

87

37

10

27

23

4

85

18

8

9

3

6

86

19

12

7

3

4

84

18

5

13

9

3

91

14

8

5

4

2

90

21

5

15

10

5

89

25

8

17

13

4

88

14

5

10

4

5

90

25

6

19

8

11

84

16

5

12

9

3

92

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

 

1998 SD and/or LEP Students

 

Identified

Excluded

Assessed

Assessed without Accommodations

Assessed with Accommodations

All Students Assessed without Accommodations

Tennessee

13

4

9

8

2

95

Texas

26

13

14

11

3

85

Utah

14

6

8

6

2

92

Vermont

Virginia

15

6

9

4

5

89

Washington

15

5

10

7

3

92

West Virginia

12

8

4

2

1

90

Wisconsin

16

8

8

5

3

89

Wyoming

14

3

10

6

4

93

Other Jurisdictions

 

 

 

 

 

 

District of Columbia

16

9

8

5

3

89

DDESS1

8

4

4

2

2

94

DoDDS2

7

3

4

3

1

96

Guam

Virgin Islands

8

5

3

2

1

94

—Indicates that the jurisdiction did not participate.

# Percentage rounds to zero.

Indicates that the jurisdiction did not meet one or more of the guidelines for school participation in 2002.

1 Department of Defense Domestic Dependent Elementary and Secondary Schools.

2 Department of Defense Dependents Schools (Overseas).

systems. Their publication, Quality Counts 2004, reports participation rates in state assessments in fourth, eighth, and tenth grade reading and mathematics assessments. Participation rates were calculated by dividing the number of students with disabilities who took the test in each grade level and subject area by the number of students with disabilities enrolled in each grade level and subject area. Table 3-7 summarizes their findings.

The authors note that 10 states and the District of Columbia were unable to provide the requested data for the 2002-2003 school year. In some states, this was because the data had not yet been reviewed and confirmed. In other cases the data could not be reported according to the specified grade levels. Some states could only compare the test-taking rates of special education students with those for all students (including those with disabilities), not just general education students,

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

2002 SD and/or LEP Students

Identified

Excluded

Assessed

Assessed without Accommodations

Assessed with Accommodations

All Students Assessed without Accommodations

14

3

10

9

1

95

27

11

16

14

2

87

19

6

13

9

4

91

15

5

10

4

6

89

18

10

8

5

3

87

15

5

11

7

4

92

16

10

5

3

2

87

19

8

10

5

5

87

17

3

15

7

7

90

19

8

11

5

5

86

14

4

10

6

4

92

16

3

13

9

4

93

39

7

32

26

6

87

7

3

4

4

1

97

NOTE: Percentages may not add to totals, due to rounding.

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress, 1998 and 2002 Reading Assessments.

TABLE 3-7 Participation Rates for Students with Disabilities in State Assessments for the 2002-2003 School Year

Participation Rate Range (%)

Fourth Grade Reading

Fourth Grade Math

Eighth Grade Reading

Eighth Grade Math

Tenth Grade Reading

Tenth Grade Math

95-100

29a

31

21

22

15

15

90-94

5

2

11

10

8

6

85-89

3

4

5

4

7

7

40-84

3

3

4

4

8

9

aNumber of states with participation rate in the specified range.

SOURCE: Education Week (2004, pp. 84-85).

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

and some states had “coding problems” (Quality Counts, p. 76). Like Thurlow, the authors note that compilation of these data was not straightforward, and they note that “differences in participation rates across states reflect, in part, the fact that states do not count students the same way when calculating such data” (p. 76). They cite the following as sources of differences:

  • If states did not have tests in place in the targeted grades, participation rates were based on tests for the next closest grade level.

  • If states did not have results for the 2002-2003 school year, participation rates were based on the most recent results available.

  • While all states count students with disabilities who take state tests without accommodations or with “standard” accommodations in their participation rates, only 26 states and the District of Columbia count those who take state tests with modifications.1

  • Fourteen of the states include students who took out-of-level tests in their participation rates.

  • While most states counted students who took alternate assessments in their participation rates, California and Indiana excluded them from their participation rates.

The participation rate data reported above all pertain to students with disabilities. We were unable to obtain data that would permit calculations of participation rates for English language learners.

SUMMARY

The provision of accommodations has clearly increased the overall participation of students with special needs in NAEP, but significant variations in accommodation policies, both among the states and between states and NAEP, remain an important issue to consider in evaluating the comparability of data about students with disabilities and English language learners. Nevertheless, state assessment programs vary in the constructs they are measuring, both from one another and from NAEP, and these differences account for some of the variation in policies. To the extent that the rates are significantly different, inferences made from comparisons of results from NAEP and state-level assessments for these two groups must be limited. While other differences between NAEP and state-level assessments limit the kinds of inferences that can be made from comparisons in any case (National Research Council, 1999b), it is nevertheless true that gross differences in performance on NAEP and a state assessment are often cited as reasons to further explore the state assessment results and possible reasons for the

1  

“Modifications” is used here as a synonym, for “accommodations.”

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×

discrepancy. It has been proposed that NAEP results could serve as an informal check on the results obtained through the assessments required under the No Child Left Behind Act (National Assessment Governing Board, 2002a). However, for even an informal comparison to be useful as an indication that NAEP and a statewide assessment are yielding results that do not contradict one another, the participation rates for the two groups on each assessment must also be compared.

There may be some legitimate reasons why the rates at which students with disabilities and English language learners participate in NAEP may never equal the participation rates for states. Both NAEP’s purpose and the specific constructs it measures are undoubtedly different in some ways from those of state assessments. NAEP is a low-stakes assessment to which tested students and their teachers attach relatively little importance because of its lack of immediate consequences for them. NAEP assessments are based on a sampling procedure (discussed in greater detail in Chapter 4), rather than the premise that it will provide individual results for every student. Moreover, some states offer alternate assessment options for some students with disabilities and English language learners that cannot be offered by NAEP. Nevertheless, important policy decisions made at the federal, state, and local levels are influenced by NAEP results. These decisions will affect all the students in the relevant jurisdiction and therefore should be based on complete information about all of the students in that jurisdiction. NAEP is designed to report results for the nation as a whole, and therefore it is the committee’s view that it should be guided by the same goal of maximizing participation rates that has been imposed on the states through legislation, so that information about all students can be obtained.

Currently, one reason that students with disabilities and English language learners are not able to participate in NAEP may be that the accommodations they need are not provided or not allowed by NAEP. It may be informative for NAEP to collect information on the extent to which students with disabilities and English language learners are not able to participate as a consequence of NAEP’s policies regarding accommodations, and specifically about the types of accommodations students require that NAEP does not allow or provide. This effort could lead to increased participation rates on NAEP, as well as to a better representation of the academic achievement of the nation’s student population.

Based on the information we have reviewed, the committee concludes that:

CONCLUSION 3-1: The increased use of accommodations with NAEP assessments has corresponded to increased participation rates for students with disabilities and English language learners.

Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 41
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 42
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 43
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 44
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 45
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 46
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 47
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 48
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 49
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 50
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 51
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 52
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 53
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 54
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 55
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 56
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 57
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 58
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 59
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 60
Suggested Citation:"3 Participation in NAEP and Other Large-Scale Assessments." National Research Council. 2004. Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Washington, DC: The National Academies Press. doi: 10.17226/11029.
×
Page 61
Next: 4 Factors That Affect the Accuracy of NAEP's Estimates of Achievement »
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments Get This Book
×
Buy Paperback | $49.00 Buy Ebook | $39.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

U.S. public schools are responsible for educating large numbers of English language learners and students with disabilities. This book considers policies for including students with disabilities and English language learners in assessment programs. It also examines the research findings on testing accommodations and their effect on test performance.

Keeping Score for All discusses the comparability of states’ policies with each other and with the National Assessment of Educational Progress (NAEP) policies and explores the impact of these differences on the interpretations of NAEP results. The book presents a critical review of the research literature and makes suggestions for future research to evaluate the validity of test scores obtained under accommodated conditions. The book concludes by proposing a new framework for conceptualizing accommodations. This framework would be useful both for policymakers, test designers, and practitioners in determining appropriate accommodations for specific assessments and for researchers in planning validity studies.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!