5
Inclusion and Accommodation
In the November 1997 legislation that established the National Assessment Governing Board's (NAGB) responsibility for the development of the Voluntary National Tests (VNT), Congress required NAGB to make four determinations. The third of these is “whether the test development process and test items take into account the needs of disadvantaged, limited English proficient and disabled students” (P.L. 105-78: Sec. 307 (b) (3)). The same legislation called on the National Research Council (NRC) “to evaluate whether the test items address the needs of disadvantaged, limited English proficient, and disabled students.”
There are two key challenges to testing students with disabilities or limited English proficiency. The first challenge is to establish effective procedures for identifying and screening such students, so they can appropriately be included in assessment programs. Federal law and state and local policy increasingly demand participation of these special populations in all education activities, both as a means of establishing the educational needs and progress of individual students and for purposes of system accountability. The second challenge is to identify and provide necessary accommodations (e.g., large-print type, extended time) to students with special needs while maintaining comparable validity of test performance with that in the general population (see National Research Council, 1997, 1999b). That is, any accommodation should alter only the conditions of assessment without otherwise affecting the measurement of performance. This issue is growing in importance, along with the number of students with disabilities or with limited English proficiency. Students with disabilities now comprise 12.3 percent of all students in elementary and secondary school, and students with limited English proficiency are 5.5 percent of all students.
Findings
While NAGB has just approved a set of “principles” for the inclusion and accommodation of students with disabilities and limited English proficiency in the pilot test of the VNT (National Assessment Governing Board, 1998c), there has as yet been little developmental work on the project to
address the special needs of these populations. The 430 participants in cognitive laboratory sessions included students with disabilities (32 in reading and 19 in math) and limited English proficiency (11 in reading and 12 in math), but their numbers were too small to provide substantial or reliable information about the participation of such students in the VNT.
In addition to the statement of principles, the American Institutes for Research (AIR) has two planning documents: “Revised Inclusion and Accommodations Work Plan” (May 8, 1998), which was approved by NAGB at its May meeting, and “Background Paper Reviewing Laws and Regulations, Current Practice, and Research Relevant to Inclusion and Accommodations for Students with Disabilities” (July 23, 1998). AIR has also developed lists of organizations with particular interest in the educational and testing needs of students with disabilities or limited English proficiency for NAGB's use in holding public hearings about inclusion and accommodation. We understand that a parallel background paper on inclusion and accommodation of students with limited English proficiency will be presented to NAGB at its November 1998 meeting. As noted above, we know of no VNT development activities or plans specifically aimed at the needs of disadvantaged students or those with limited English proficiency, but such students did participate in one development activity—the cognitive labs.
Because of the compressed schedule in the early phases of VNT development, along with the desire to achieve close correspondence between the VNT and NAEP, the NAGB plans and the AIR background paper on students with disabilities both focus on recent NAEP practices for inclusion and accommodation, rather than taking a broader, more proactive stance. We believe the federal government has an important leadership role to play in subsidizing and demonstrating valid efforts to include these populations.
The procedures discussed in the draft documents are intended to increase participation and provide valid assessments for all students, but they essentially involve retrofitting established assessment instruments and procedures to special populations of students; another approach would be to design and develop assessments from the beginning that are accessible to and valid for all students.
From its beginning around 1970 and through the middle 1990s, NAEP assessments were carried out without accommodation of any kind. Procedures for “inclusion” usually focused on the exclusion of some students from the assessments, rather than on universal participation. Since the mid-1990s, as the growth of special student populations and the importance of their participation in large-scale assessments have increasingly been recognized, NAEP has experimented with new, more inclusive participation and accommodation policies, which are recapitulated with reference to the VNT in the NAGB principles and AIR planning documents. For example, in NAGB's draft principles, and depending on the test and population in question, accommodations for the pilot test may include large-print booklets, extended time, small-group administration, one-on-one administration, a scribe or computer to record answers, reading a test aloud by an examiner, other format or equipment modifications, or a bilingual dictionary if it is normally allowed by the school. However, the success of these policies in increasing participation is not yet established, nor have their effects on test performance and score comparability been validated.
Unless extensive development work is done with students with disabilities and with limited English proficiency, it would be unreasonable to expect that the VNT will be valid for use with these student populations. Both of these populations are heterogeneous, e.g., in primary language, level of proficiency in English, and specific type of disability. Moreover, they differ from the majority of students, not only in ways that affect test-taking directly—e.g., those that can be accommodated through additional time or assistive devices—but also in styles of learning and in levels of motivation or anxiety. Such differences are very likely to reduce the validity and comparability of test performance.
The Committee on Appropriate Test Use has identified two important ways in which inclusion
and accommodation can be improved (National Research Council, 1999b). First, the focus should be on inclusion and accommodation issues throughout item and test development, so a test is designed from the ground up to be accessible and comparable among special populations. For example, the NRC report recommends oversampling of students with disabilities and with limited English proficiency in the course of pilot testing so there will be sufficient numbers of cases in major subgroups of these students to permit DIF analyses. Second, test developers should explore the use of new technologies, such as computer-based, adaptive testing for students who need extra time, which show promise of substantially reducing or eliminating irrelevant performance differentials between many students who require accommodation and other students. The NRC Committee on Appropriate Test Use recognized, however, that development work of this kind is just beginning, and there are presently few successful exemplars of it.
Conclusion
The statement of principles and the AIR planning documents provide a limited basis for evaluation of provisions for inclusion and accommodation in the VNT—and no specific basis to address the quality of item development relative to the needs of those students.
NAGB's desire to maintain correspondence between NAEP and the VNT has not precluded departure from current NAEP practices in other areas—for example, the use of intertextual items and gridded responses. In our judgment, a major opportunity for improved large-scale assessment is being lost in NAGB's conservative approach to inclusion and accommodation in the VNT.
Recommendation
5-1. NAGB should accelerate its plans and schedule for inclusion and accommodation of students with disabilities and limited English proficiency in order to increase the participation of both those student populations and to increase the comparability of VNT performance among student populations.
This recommendation requires prompt action because so much of the development work in the first round of the VNT has already been completed. We have already noted the modest attention to students with special needs in the cognitive laboratory sessions. In the pilot test, NAGB plans to identify students with disabilities and with limited English proficiency and with the types of accommodations that have been provided. However, there are no provisions in the design to ensure that there will be sufficient numbers of these students—such as students requiring specific types of accommodation—to support reliable DIF analyses. We think that it would be feasible to include larger numbers of such students in the pilot and field tests, for example, by increasing sampling fractions of such students within schools. Moreover, there appears to be no plan to translate the 8th-grade mathematics test into Spanish (or any other language), a decision that is likely to affect participation in the VNT by major school districts. There has been some discussion of a Spanish translation after the field test, but this would be too late for the item analyses needed to construct comparable English and Spanish forms.