3
Participation in NAEP and Other Large-Scale Assessments

A variety of factors can influence the rates at which students with disabilities and English language learners participate in large-scale assessments. Legislative mandates have established requirements that, in general, all such students should be included in statewide accountability programs, but these requirements do allow for the exclusion of a small number of students from assessments. These laws require that accommodations be provided for students who need them but offer relatively scant guidance for determining how and when the accommodations should be provided. The National Assessment of Educational Progress (NAEP) assessments have not been subject to the same legal requirements regarding participation, and until 1996 they did not permit accommodations. NAEP’s participation rates for students with disabilities and English language learners have thus lagged behind those of state assessment programs. In this chapter, we present the information available on participation rates in NAEP and statewide assessments and discuss the implications of the available data.

PARTICIPATION RATES FOR NAEP

NAEP’s Research Study on Providing Accommodations

In the early 1990s, students with individualized education programs (IEPs) could be excluded from NAEP if they were placed in general education classrooms (mainstreamed) less than 50 percent of the time or judged to be incapable of meaningful participation in the assessment. Some English language learners—defined as students whose native language was not English, who had been enrolled



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments 3 Participation in NAEP and Other Large-Scale Assessments A variety of factors can influence the rates at which students with disabilities and English language learners participate in large-scale assessments. Legislative mandates have established requirements that, in general, all such students should be included in statewide accountability programs, but these requirements do allow for the exclusion of a small number of students from assessments. These laws require that accommodations be provided for students who need them but offer relatively scant guidance for determining how and when the accommodations should be provided. The National Assessment of Educational Progress (NAEP) assessments have not been subject to the same legal requirements regarding participation, and until 1996 they did not permit accommodations. NAEP’s participation rates for students with disabilities and English language learners have thus lagged behind those of state assessment programs. In this chapter, we present the information available on participation rates in NAEP and statewide assessments and discuss the implications of the available data. PARTICIPATION RATES FOR NAEP NAEP’s Research Study on Providing Accommodations In the early 1990s, students with individualized education programs (IEPs) could be excluded from NAEP if they were placed in general education classrooms (mainstreamed) less than 50 percent of the time or judged to be incapable of meaningful participation in the assessment. Some English language learners—defined as students whose native language was not English, who had been enrolled

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments in an English speaking school for less than three years, or who were judged incapable of meaningful participation in the assessment—could also be excluded. Specifically, a NAEP publication describes the former procedures for these two groups in the following way (U.S. Department of Education National Center for Education Statistics, 1997). Prior to 1990, administrations of NAEP … relied on the judgment of school administrators as to whether or not the student could take the assessment. Beginning with the 1990 NAEP, schools were given guidelines informing them that they may exclude a student with a disability if the student is mainstreamed less than 50% of the time and is judged incapable of participating meaningfully in the assessment, OR, the IEP team or equivalent group determined that the student is incapable of participating meaningfully in the assessment. Schools were instructed to include students with disabilities if school staff believed the students were capable of taking the assessment. Schools were also instructed that when there was doubt, students should be included. (pp. 14-15) The NAEP procedures used prior to 1990 allowed schools to exclude sampled students if they were LEP and if local school personnel judged the students incapable of participating meaningfully in the assessment Beginning in 1990, NAEP instructions to schools for excluding LEP students from the assessment required the following conditions to be met: the student is a native speaker of a language other than English AND the student has been enrolled in an English-speaking school for less than 2 years (not including bilingual education programs) AND school officials judged the student to be incapable of taking the assessment. The guidelines also stated that when in doubt, the student was to be included in the assessment. (p. 41) NAEP’s sponsors took the first step in making the assessment more inclusive when they adopted a series of resolutions that established a plan for conducting research on the effects of including students with disabilities and English language learners in the assessment. In these resolutions, NAEP’s sponsors articulated the dual priorities of including students who can meaningfully take part in the assessment while maintaining the integrity of the trend data that are considered a key component of NAEP. The resolution and research plan provided what NAEP officials have described as both “a bridge to the future,” because it would make NAEP more inclusive, and “a bridge to the past” (NRC, 2002a), because it would allow NAEP to continue to provide meaningful trend information. Protection of the capacity to report trend data was considered a necessary constraint on any changes in policies and procedures. NAEP conducted a series of pilot studies in the early 1990s to examine the feasibility of allowing students to participate in the assessment with accommodations and then, in conjunction with the 1996 mathematics assessment, initiated a research plan. This plan called for data to be collected for three samples of tested students, referred to as S1, S2, and S3. For the S1 sample, administration proce-

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments dures were handled in the same way as in the early 1990s, and students with special needs could be excluded from the assessment. For the S2 sample, revisions aimed at increasing participation were made to the criteria given to schools for determining whether to include students with special needs, but no accommodations or adaptations were offered. For state NAEP, the schools were split between S1 and S2. For national NAEP, a third sample of schools, S3, was identified, in which the revised inclusion criteria were used, and accommodations were permitted for students with disabilities and English language learners. These students were allowed to participate with the accommodations that they routinely received in their state or district testing. Analyses of the 1996 data revealed no differences in inclusion rates between the S1 and S2 samples, so the S1 criteria were discontinued, and further research was based on samples of schools that applied either the revised criteria. Comparison of the S2 and S3 samples provided the opportunity to examine the net effects of both types of changes, both more lenient criteria for inclusion and the use of accommodations. The research continued with the 1998 national and state NAEP reading assessment and the 2000 assessments (math and science at the national level in grades 4, 8, and 12 and at the state level in grades 4 and 8; reading at the national level in grade 4). Analyses of the 1998 and 2000 data revealed that providing accommodations did increase the number of students with disabilities and English language learners included in NAEP in grades 4 and 8. Table 3-1 presents information from the research study on the participation rates of students with disabilities and English language learners in NAEP’s reading and math assessments for fourth graders. Participation rates (column 6) were calculated by dividing the number of students assessed (column 5) by the number identified (column 4) and multiplying by 100. Thus, the table shows that the percentages of students with disabilities who participated in NAEP’s fourth grade reading assessment were 13.8 percent in 1992 and 34.1 percent in 1994. These were years in which students with disabilities and English language learners could participate but no accommodations were allowed. Participation rates for the S2 (accommodations not allowed) and S3 (accommodations allowed) study samples are displayed separately for each of the years the study was in place. Comparisons of the participation rates in a given year demonstrate the impact of providing accommodations. For example, for the 1998 fourth grade reading assessment, the participation rate for students with disabilities was 6.3 percentage points higher for the S3 sample than for the S2 sample. The median differences over assessment years between the participation rates when accommodations were not allowed and were allowed were 17.7 percent for students with disabilities and 21.3 percent for English language learners. Table 3-2 presents this same information from the research study on the participation rates of students with disabilities and English language learners in NAEP’s reading and math assessments for eighth graders. For eighth graders, the

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments TABLE 3-1 Results from NAEP’s Research: Participation of Students with Disabilities (SWD) and English Language Learners (ELL) in Fourth Grade NAEP Reading and Math Assessments (1) (2) (3) (4) (5) (6)       Students Identified Students Assessed Percent Participating Assessment Year Accommodation Permitted SWD ELL SWD ELL SWD ELL Reading 1992 No 1,149 945 159 110 13.8 11.6   1994 No 1,039 623 354 255 34.1 40.9   1998—S2a No 490 527 243 204 49.6 38.7   1998—S3b Yes 558 446 312 279 55.9 62.6   Differences in participation rates for two 1998 samples   6.3 23.9   2000—S2a No 524 356 229 215 43.7 60.4   2000—S3b Yes 510 446 317 287 62.2 64.4   Differences in participation rates for two 2000 samples   18.5 4.0 Math 1992 No 1,163 939 173 104 14.9 11.1   1996—S2a No 359 142 206 75 57.4 52.8   1996—S3b Yes 424 308 315 222 74.3 72.1   Differences in participation rates for two 1996 samples   16.9 19.3   2000—S2a No 672 454 292 265 43.5 58.4   2000—S3b Yes 706 472 526 385 74.5 81.6   Differences in participation rates for two 2000 samples   31.0 23.2 aResults from split sample study: Students taking this assessment were NOT allowed accommodations. bResults from split sample study: Students taking this assessment WERE allowed accommodations. SOURCE: Available: http://nces.ed.gov. median differences over assessment years in the participation rates when accommodations were not allowed and were allowed were 21.0 percent for students with disabilities and 13.2 percent for English language learners. NAEP’s Reporting of Participation Rates The committee examined reports from several administrations of NAEP to gather information on participation rates for the most recent assessment. Table 3-3 presents the national data for students with disabilities and English language learners for the 1998-2002 reading assessment. Table 3-4 shows results by state for students with disabilities for the grade 4 reading assessment in 1998 and 2002, and Table 3-5 shows these results for English language learners. We note that the method for displaying the results presented in these tables has been improved from previous years. In the past, participation rates at the national level were reported separately for students with disabilities and English language learners,

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments TABLE 3-2 Results from NAEP’s Research: Participation of Students with Disabilities (SWD) and English Language Learners (ELL) in Eighth Grade NAEP Reading and Math Assessments (1) (2) (3) (4) (5) (6)       Students Identified Students Assessed Percent Participating Assessment Year Accommodation Permitted SWD ELL SWD ELL SWD ELL Reading 1992 No 1,522 836 199 86 13.1 10.3   1994 No 1,323 444 344 121 26.0 27.3   1998—S2a No 975 449 451 315 46.3 70.2   1998—S3b Yes 865 447 582 338 67.3 75.6   Differences in participation rates for two 1998 samples   21.0 5.4 Math 1992 No 1,538 838 215 88 14.0 10.5   1996a No 310 106 161 68 51.9 64.2   1996b Yes 557 226 374 175 67.1 77.4   Differences in participation rates for two 1996 samples   15.2 13.2   2000a No 1,316 551 597 341 45.4 61.9   2000b Yes 1,206 471 804 368 66.7 78.1   Differences in participation rates for two 2000 samples   21.3 16.2 aResults from split sample study: Students taking this assessment were NOT allowed accommodations. bResults from split sample study: Students taking this assessment WERE allowed accommodations. SOURCE: Available: http://nces.ed.gov. but NAEP state-level reports combined participation data for the two groups. An example of the way these data were previously presented is shown in Table 3-6. With the state-level data for these two groups combined, it was impossible either to track participation rates separately for the two groups or to make comparisons between participation rates in state assessments and in NAEP (incomplete data from state assessments have also been an impediment to making these comparisons; that issue is discussed below). The committee had intended to make a recommendation about the reporting of state-level participation rates, but before our deliberations were completed, NAEP for the first time presented the participation rates for students with disabilities and English language learners separately in both state and national reports. In the committee’s judgment, the revised version of the NAEP tables is a significant improvement and we encourage NAEP’s sponsors to continue to provide these data.

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments TABLE 3-3 Students with Disabilities (SWD) and/or Limited English Proficient (LEP) Students Identified, Excluded, and Assessed, When Accommodations Were Permitted, Grades 4, 8, and 12 Public and Nonpublic Schools: 1998–2002 for Reading   1998 2000 2002   Number of Students Weighted Percentage of Students Sampled Number of Students Weighted Percentage of Students Sampled Number of Students Weighted Percentage of Students Sampled Grade 4             SD and/or LEP students             Identified 973 16 906 18 28,073 19 Excluded 393 6 316 6 10,307 6 Assessed 580 10 590 12 17,766 13 Without accommodations 413 7 476 10 11,913 9 With accommodations 167 3 114 2 5,853 4 SD students             Identified 558 10 510 11 19,936 12 Excluded 246 4 193 4 8,042 5 Assessed 312 6 317 7 11,894 7 Without accommodations 179 3 209 5 6,631 4 With accommodations 133 3 108 2 5,263 3 LEP students             Identified 446 6 446 8 10,334 8 Excluded 167 2 159 3 3,410 2 Assessed 279 4 287 5 6,924 6 Without accommodations 238 3 273 5 6,020 6 With accommodations 41 1 14 # 904 1

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments Grade 8             SD and/or LEP students             Identified 1,252 12 — — 20,137 17 Excluded 368 4 — — 7,135 5 Assessed 884 9 — — 13,002 11 Without accommodations 678 6 — — 8,598 8 With accommodations 206 2 — — 4,404 4 SD students             Identified 865 10 — — 16,159 12 Excluded 283 3 — — 5,939 4 Assessed 582 7 — — 10,220 8 Without accommodations 404 5 — — 6,074 5 With accommodations 178 2 — — 4,146 3 LEP students             Identified 447 3 — — 5,516 6 Excluded 109 1 — — 1,907 2 Assessed 338 2 — — 3,609 4 Without accommodations 307 2 — — 3,113 4 With accommodations 31 # — — 496 # Grade 12             SD and/or LEP students             Identified 975 7 — — 1,556 12 Excluded 327 2 — — 616 4 Assessed 648 5 — — 940 8 Without accommodations 532 4 — — 673 6 With accommodations 116 1 — — 267 2 SD students             Identified 649 6 — — 1,231 9 Excluded 285 2 — — 535 3 Assessed 364 4 — — 696 6 Without accommodations 266 3 — — 446 4 With accommodations 98 1 — — 250 2

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments   1998 2000 2002   Number of Students Weighted Percentage of Students Sampled Number of Students Weighted Percentage of Students Sampled Number of Students Weighted Percentage of Students Sampled LEP students             Identified 353 2 — — 419 3 Excluded 58 # — — 125 1 Assessed 295 2 — — 294 3 Without accommodations 277 2 — — 266 2 With accommodations 18 # — — 28 # —Data were not collected at grades 8 and 12 in 2000. # Percentage rounds to zero. NOTE: Within each grade level, the combined SD/LEP portion of the table is not a sum of the separate SD and LEP portions becaus e some students were identified as both SD and LEP. Such students would be counted separately in the bottom portions but counted only once in the top portion. Within each portion of the table, percentages may not add to totals, due to rounding. The number of students at grades 4 and 8 are larger in 2002 than in previous years because the 2002 national sample was based on the combined sample of students in each participating state, plus an additional sample from nonparticipating states as well as a sample from private schools. SOURCE: From U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress, 1998, 2000, and 2002 Reading Assessments.

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments TABLE 3-4 Percentage of Students with Disabilities (SD) Identified, Excluded, and Assessed, When Accommodations Were Permitted, Grade 4 Public Schools: By State, 1998 and 2002 for Reading   1998 SD Students 2002 SD Students   Identified Excluded Assessed Assessed without Accommodations Assessed with Accommodations Identified Excluded Assessed Assessed without Accommodations Assessed with Accommodations Nation (Public) 12 5 7 4 3 13 5 8 4 4 Alabama 13 8 4 3 1 13 2 11 8 2 Arizona 10 5 5 4 1 11 5 7 5 2 Arkansas 10 4 6 4 2 12 4 7 5 2 California‡ 6 3 2 2 1 7 3 4 3 1 Connecticut 14 7 7 4 3 13 4 9 4 6 Delaware 14 1 12 9 4 15 7 8 3 5 Florida 14 5 9 5 4 17 5 13 6 7 Georgia 9 4 6 3 3 10 3 7 4 3 Hawaii 10 4 7 5 1 12 4 8 3 4 Idaho — — — — — 13 4 9 7 2 Illinois‡ 10 3 6 4 2 13 4 9 4 5 Indiana — — — — — 12 4 8 6 2 Iowa‡ 14 5 9 6 3 15 7 8 3 5 Kansas‡ 9 3 6 3 3 14 4 10 4 5 Kentucky 12 7 5 3 2 11 8 4 2 1 Louisiana 14 7 7 2 5 19 10 8 3 5 Maine 15 7 7 4 3 16 6 10 5 6 Maryland 11 5 6 2 4 12 6 6 4 2 Massachusetts 16 4 12 7 5 16 4 12 3 9

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments   1998 SD Students 2002 SD Students   Identified Excluded Assessed Assessed without Accommodations Assessed with Accommodations Identified Excluded Assessed Assessed without Accommodations Assessed with Accommodations Michigan 9 5 3 2 1 11 7 4 3 1 Minnesota‡ 12 3 9 6 3 13 4 10 6 3 Mississippi 7 4 3 2 # 7 4 3 2 1 Missouri 14 6 7 3 4 15 8 7 4 3 Montana‡ 10 2 7 5 2 13 5 8 4 4 Nebraska — — — — — 18 4 13 7 6 Nevada 10 6 4 4 1 12 5 7 5 2 New Mexico 14 7 7 5 2 15 7 9 6 3 New York‡ 9 4 5 1 4 14 6 8 2 5 North Carolina 14 6 8 2 6 17 10 6 3 4 North Dakota‡ — — — — — 16 5 11 8 3 Ohio — — — — — 13 8 5 3 2 Oklahoma 13 9 5 3 1 17 5 13 8 5 Oregon 14 4 10 6 4 16 5 10 7 3 Pennsylvania — — — — — 13 4 9 4 5 Rhode Island 14 5 10 6 3 19 3 15 6 10 South Carolina 15 7 8 5 3 16 4 11 8 3 Tennessee‡ 12 3 9 7 2 11 3 8 6 1 Texas 14 7 8 5 2 14 8 6 5 2 Utah 10 4 6 4 1 12 4 7 5 3 Vermont — — — — — 13 5 9 3 6 Virginia 14 6 8 4 4 14 8 6 3 3

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments Washington‡ 11 4 8 5 3 13 4 9 6 4 West Virginia 12 8 4 2 1 15 10 5 3 2 Wisconsin‡ 13 7 6 4 2 13 6 8 3 4 Wyoming 13 3 10 6 4 14 2 12 5 7 Other Jurisdictions                     District of Columbia 10 6 4 2 2 14 7 7 3 4 DDESS1 7 3 4 2 2 10 3 7 3 4 DoDDS2 6 2 4 3 1 9 2 7 4 3 Guam — — — — — 7 4 3 2 1 Virgin Islands 4 3 1 1 0 3 1 2 2 # —Indicates that the jurisdiction did not participate. # Percentage rounds to zero. ‡ Indicates that the jurisdiction did not meet one or more of the guidelines for school participation in 2002. 1 Department of Defense Domestic Dependent Elementary and Secondary Schools. 2 Department of Defense Dependents Schools (Overseas). NOTE: Percentages may not add to totals, due to rounding. SOURCE: From U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress, 1998 and 2002 Reading Assessments.

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments TABLE 3-5 Percentage of Limited English Proficient (LEP) Students Identified, Excluded, and Assessed, When Accommodations Were Permitted, Grade 4 Public Schools: By State, 1998 and 2002 for Reading   1998 LEP Students 2002 LEP Students   Identified Excluded Assessed Assessed without Accommodations Assessed with Accommodations Identified Excluded Assessed Assessed without Accommodations Assessed with Accommodations Nation (Public) 5 2 3 3 1 9 2 7 6 1 Alabama # # # 0 # 1 # 1 1 # Arizona 14 6 7 6 1 21 5 16 15 1 Arkansas 1 1 1 1 0 3 1 3 3 # California‡ 26 12 14 13 1 29 3 26 26 # Connecticut 5 4 1 1 # 4 2 2 2 # Delaware 3 # 2 2 # 3 2 1 1 # Florida 5 1 3 3 # 10 3 7 5 2 Georgia 2 1 # # # 4 1 2 2 # Hawaii 6 2 4 4 0 8 2 6 4 1 Idaho — — — — — 7 1 6 5 # Illinois‡ 5 3 2 2 # 9 4 5 4 1 Indiana — — — — — 2 1 1 1 0 Iowa‡ 1 1 1 1 0 2 1 1 1 # Kansas‡ 3 1 2 2 # 7 2 6 4 2 Kentucky 1 # # # # 1 # # # # Louisiana 1 1 1 1 0 1 1 1 # # Maine # 0 # # # 1 # # # # Maryland 2 1 2 1 # 3 2 1 1 # Massachusetts 4 2 2 2 1 4 2 2 1 1 Michigan 2 1 1 1 # 3 1 2 2 #

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments Minnesota‡ 4 1 3 3 1 7 2 5 4 1 Mississippi # 0 # # 0 # # # # 0 Missouri 1 # # # # 2 1 1 1 # Montana‡ # 0 # 0 # 2 1 1 1 # Nebraska — — — — — 4 2 3 2 # Nevada 10 6 4 4 # 18 7 11 10 1 New Mexico 16 4 12 11 1 27 6 21 19 2 New York‡ 5 4 1 1 0 6 3 3 1 1 North Carolina 2 1 1 1 # 5 3 1 1 1 North Dakota‡ — — — — — 2 1 2 1 # Ohio — — — — — 1 1 1 1 0 Oklahoma 2 # 1 1 0 5 1 4 3 1 Oregon 7 2 5 4 1 12 4 8 6 2 Pennsylvania — — — — — 2 1 1 1 # Rhode Island 6 3 4 3 1 9 3 5 4 2 South Carolina 1 # 1 1 0 2 1 1 1 # Tennessee‡ 1 1 # # 0 3 1 3 3 # Texas 13 7 6 6 # 16 5 11 10 1 Utah 5 2 3 2 # 9 3 7 5 1 Vermont — — — — — 2 # 1 1 # Virginia 2 1 1 1 1 6 3 3 2 1 Washington‡ 4 2 3 2 # 3 1 2 2 # West Virginia # # # # 0 # # # # 0 Wisconsin‡ 3 1 2 1 # 6 3 3 2 1 Wyoming 1 1 # # # 5 1 4 3 1

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments   1998 LEP Students 2002 LEP Students   Identified Excluded Assessed Assessed without Accommodations Assessed with Accommodations Identified Excluded Assessed Assessed without Accommodations Assessed with Accommodations Other Jurisdictions                     District of Columbia 7 3 4 2 1 7 3 4 3 2 DDESS1 1 1 # 0 # 6 2 4 3 1 DoDDS2 2 1 1 1 # 8 1 7 6 1 Guam — — — — — 36 5 31 25 6 Virgin Islands 4 2 2 1 1 5 2 3 3 1 —Indicates that the jurisdiction did not participate. # Percentage rounds to zero. ‡ Indicates that the jurisdiction did not meet one or more of the guidelines for school participation in 2002. 1 Department of Defense Domestic Dependent Elementary and Secondary Schools. 2 Department of Defense Dependents Schools (Overseas). NOTE: Percentages may not add to totals, due to rounding. SOURCE: From U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress, 1998 and 2002 Reading Assessments.

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments PARTICIPATION RATES FOR STATE ASSESSMENTS The committee attempted to obtain information on the participation rates of the two student groups in statewide assessment programs in order to make comparisons with NAEP’s participation rates. However, we discovered that this information is not readily available. The most comprehensive source of data on state assessments is maintained by the Council of Chief State School Officers through their annual survey of state student assessment programs (Council of Chief State School Officers, 2002). With this survey, the council collects information about exemptions from statewide assessments; that is, the survey asks states if the number of special education or limited English proficient exemptions increased, decreased, or stayed the same over the past two to three years. However, the survey report does not provide data that could be used to calculate participation rates. Two additional sources were identified. Some information on participation rates for students with disabilities in state assessments is available from the National Center on Education Outcomes. Based on their review of states’ biennial performance reports to the U.S. Department of Education, Thurlow et al. (2002) were able to obtain enough information for each state to calculate participation rates for all of the states. However, data were not provided for all of the tests states administer or in all of the grades assessed. Thus, it was not possible to examine these participation rates by grade level or by the subject matter of the test. Thurlow et al. reported a participation rate for every state, but in some states it is the only rate that could be calculated, while in others it is the highest of the rates calculated. In the Thurlow report, state participation rates were calculated as follows: the numerator was the number of students with disabilities participating in the assessment; the denominator was the number of participating students with disabilities, plus the number taking the alternate assessment, plus the number not tested, which should equal the total number of students with disabilities enrolled in special education services in the state. Based on these calculations, the participation rates for students with disabilities were 90 percent or higher in 19 states, between 75 and 89 percent in 17 states, between 50 and 74 percent in 5 states, and between 25 and 49 percent in 1 state. According to the authors of that report, the required analyses were quite complex, and determining the denominator for the participation rate was challenging. This was the first time analyses like this had been undertaken, primarily because in the past the Department of Education requirement to report results using a standard structure was not in place (M. Thurlow, personal communication, December 16, 2003). In addition, Education Week (Education Week, 2004) recently reported the results of a survey of the states and the District of Columbia designed to gather data on the inclusion of students with disabilities in state testing and accountability

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments TABLE 3-6 Percentage of Students with Disabilities (SD) and/or Limited English Proficient (LEP) Students Identified, Excluded, and Assessed, When Accommodations Were Permitted, Grade 4 Public Schools: By State, 1998 and 2002 for Reading   1998 SD and/or LEP Students   Identified Excluded Assessed Assessed without Accommodations Assessed with Accommodations All Students Assessed without Accommodations Nation (Public) 18 7 11 7 3 90 Alabama 13 8 4 3 1 90 Arizona 22 10 12 10 1 88 Arkansas 11 5 6 4 2 93 California‡ 31 14 16 15 1 84 Connecticut 18 10 8 5 3 87 Delaware 16 1 15 11 4 95 Florida 18 6 12 8 5 89 Georgia 11 5 6 3 3 93 Hawaii 15 5 10 9 1 94 Idaho — — — — — — Illinois‡ 14 6 8 6 2 92 Indiana — — — — — — Iowa‡ 15 5 10 7 3 92 Kansas‡ 12 4 8 5 4 93 Kentucky 13 7 5 3 2 90 Louisiana 15 7 8 3 5 88 Maine 15 7 7 4 3 90 Maryland 13 6 8 4 4 90 Massachusetts 19 5 14 9 5 90 Michigan 10 6 4 3 1 93 Minnesota‡ 15 3 12 9 3 94 Mississippi 7 4 3 2 # 95 Missouri 14 6 8 3 4 89 Montana‡ 10 2 7 5 2 96 Nebraska — — — — — — Nevada 20 11 9 8 1 88 New Mexico 28 9 18 16 2 88 New York‡ 14 7 7 2 4 88 North Carolina 15 7 9 3 6 88 North Dakota‡ — — — — — — Ohio — — — — — — Oklahoma 15 9 6 5 1 90 Oregon 20 6 14 10 4 90 Pennsylvania — — — — — — Rhode Island 20 7 13 9 4 89 South Carolina 16 8 9 6 3 90

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments 2002 SD and/or LEP Students Identified Excluded Assessed Assessed without Accommodations Assessed with Accommodations All Students Assessed without Accommodations 21 7 14 10 4 89 14 3 12 9 2 95 28 8 21 18 3 90 14 5 10 8 2 93 34 5 29 28 1 94 16 5 11 5 6 89 17 8 9 4 5 87 25 7 18 10 8 85 13 4 9 6 3 93 18 6 12 7 5 89 17 4 13 11 2 93 20 7 14 8 6 87 13 5 9 7 2 93 16 8 8 3 5 87 19 5 14 7 7 88 12 8 4 3 1 91 19 10 9 3 6 84 17 6 11 5 6 88 14 7 7 5 2 92 19 6 13 4 9 85 14 7 6 5 1 92 19 5 13 10 4 91 7 4 3 2 1 95 16 9 8 4 3 88 15 6 8 4 4 89 21 5 15 9 6 88 27 10 17 14 3 87 37 10 27 23 4 85 18 8 9 3 6 86 19 12 7 3 4 84 18 5 13 9 3 91 14 8 5 4 2 90 21 5 15 10 5 89 25 8 17 13 4 88 14 5 10 4 5 90 25 6 19 8 11 84 16 5 12 9 3 92

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments   1998 SD and/or LEP Students   Identified Excluded Assessed Assessed without Accommodations Assessed with Accommodations All Students Assessed without Accommodations Tennessee‡ 13 4 9 8 2 95 Texas 26 13 14 11 3 85 Utah 14 6 8 6 2 92 Vermont — — — — — — Virginia 15 6 9 4 5 89 Washington‡ 15 5 10 7 3 92 West Virginia 12 8 4 2 1 90 Wisconsin‡ 16 8 8 5 3 89 Wyoming 14 3 10 6 4 93 Other Jurisdictions             District of Columbia 16 9 8 5 3 89 DDESS1 8 4 4 2 2 94 DoDDS2 7 3 4 3 1 96 Guam — — — — — — Virgin Islands 8 5 3 2 1 94 —Indicates that the jurisdiction did not participate. # Percentage rounds to zero. ‡ Indicates that the jurisdiction did not meet one or more of the guidelines for school participation in 2002. 1 Department of Defense Domestic Dependent Elementary and Secondary Schools. 2 Department of Defense Dependents Schools (Overseas). systems. Their publication, Quality Counts 2004, reports participation rates in state assessments in fourth, eighth, and tenth grade reading and mathematics assessments. Participation rates were calculated by dividing the number of students with disabilities who took the test in each grade level and subject area by the number of students with disabilities enrolled in each grade level and subject area. Table 3-7 summarizes their findings. The authors note that 10 states and the District of Columbia were unable to provide the requested data for the 2002-2003 school year. In some states, this was because the data had not yet been reviewed and confirmed. In other cases the data could not be reported according to the specified grade levels. Some states could only compare the test-taking rates of special education students with those for all students (including those with disabilities), not just general education students,

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments 2002 SD and/or LEP Students Identified Excluded Assessed Assessed without Accommodations Assessed with Accommodations All Students Assessed without Accommodations 14 3 10 9 1 95 27 11 16 14 2 87 19 6 13 9 4 91 15 5 10 4 6 89 18 10 8 5 3 87 15 5 11 7 4 92 16 10 5 3 2 87 19 8 10 5 5 87 17 3 15 7 7 90 19 8 11 5 5 86 14 4 10 6 4 92 16 3 13 9 4 93 39 7 32 26 6 87 7 3 4 4 1 97 NOTE: Percentages may not add to totals, due to rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress, 1998 and 2002 Reading Assessments. TABLE 3-7 Participation Rates for Students with Disabilities in State Assessments for the 2002-2003 School Year Participation Rate Range (%) Fourth Grade Reading Fourth Grade Math Eighth Grade Reading Eighth Grade Math Tenth Grade Reading Tenth Grade Math 95-100 29a 31 21 22 15 15 90-94 5 2 11 10 8 6 85-89 3 4 5 4 7 7 40-84 3 3 4 4 8 9 aNumber of states with participation rate in the specified range. SOURCE: Education Week (2004, pp. 84-85).

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments and some states had “coding problems” (Quality Counts, p. 76). Like Thurlow, the authors note that compilation of these data was not straightforward, and they note that “differences in participation rates across states reflect, in part, the fact that states do not count students the same way when calculating such data” (p. 76). They cite the following as sources of differences: If states did not have tests in place in the targeted grades, participation rates were based on tests for the next closest grade level. If states did not have results for the 2002-2003 school year, participation rates were based on the most recent results available. While all states count students with disabilities who take state tests without accommodations or with “standard” accommodations in their participation rates, only 26 states and the District of Columbia count those who take state tests with modifications.1 Fourteen of the states include students who took out-of-level tests in their participation rates. While most states counted students who took alternate assessments in their participation rates, California and Indiana excluded them from their participation rates. The participation rate data reported above all pertain to students with disabilities. We were unable to obtain data that would permit calculations of participation rates for English language learners. SUMMARY The provision of accommodations has clearly increased the overall participation of students with special needs in NAEP, but significant variations in accommodation policies, both among the states and between states and NAEP, remain an important issue to consider in evaluating the comparability of data about students with disabilities and English language learners. Nevertheless, state assessment programs vary in the constructs they are measuring, both from one another and from NAEP, and these differences account for some of the variation in policies. To the extent that the rates are significantly different, inferences made from comparisons of results from NAEP and state-level assessments for these two groups must be limited. While other differences between NAEP and state-level assessments limit the kinds of inferences that can be made from comparisons in any case (National Research Council, 1999b), it is nevertheless true that gross differences in performance on NAEP and a state assessment are often cited as reasons to further explore the state assessment results and possible reasons for the 1   “Modifications” is used here as a synonym, for “accommodations.”

OCR for page 41
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments discrepancy. It has been proposed that NAEP results could serve as an informal check on the results obtained through the assessments required under the No Child Left Behind Act (National Assessment Governing Board, 2002a). However, for even an informal comparison to be useful as an indication that NAEP and a statewide assessment are yielding results that do not contradict one another, the participation rates for the two groups on each assessment must also be compared. There may be some legitimate reasons why the rates at which students with disabilities and English language learners participate in NAEP may never equal the participation rates for states. Both NAEP’s purpose and the specific constructs it measures are undoubtedly different in some ways from those of state assessments. NAEP is a low-stakes assessment to which tested students and their teachers attach relatively little importance because of its lack of immediate consequences for them. NAEP assessments are based on a sampling procedure (discussed in greater detail in Chapter 4), rather than the premise that it will provide individual results for every student. Moreover, some states offer alternate assessment options for some students with disabilities and English language learners that cannot be offered by NAEP. Nevertheless, important policy decisions made at the federal, state, and local levels are influenced by NAEP results. These decisions will affect all the students in the relevant jurisdiction and therefore should be based on complete information about all of the students in that jurisdiction. NAEP is designed to report results for the nation as a whole, and therefore it is the committee’s view that it should be guided by the same goal of maximizing participation rates that has been imposed on the states through legislation, so that information about all students can be obtained. Currently, one reason that students with disabilities and English language learners are not able to participate in NAEP may be that the accommodations they need are not provided or not allowed by NAEP. It may be informative for NAEP to collect information on the extent to which students with disabilities and English language learners are not able to participate as a consequence of NAEP’s policies regarding accommodations, and specifically about the types of accommodations students require that NAEP does not allow or provide. This effort could lead to increased participation rates on NAEP, as well as to a better representation of the academic achievement of the nation’s student population. Based on the information we have reviewed, the committee concludes that: CONCLUSION 3-1: The increased use of accommodations with NAEP assessments has corresponded to increased participation rates for students with disabilities and English language learners.