NAEP Reporting Practices Investigating District-Level and Market-Basket Reporting (2001) / Chapter Skim
Currently Skimming:
Changed NAEP: Use of a short-Form Version of NAEP
Changed NAEP: Use of a short-Form Version of NAEP
Pages 71-85
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 71... ...
NAG B has active working groups (National Assessment Governing Board, 1999c; National Assessment Governing Board, 2000a) looking into alternate delivery and reporting models for NAEP, and the short form and market-basket concepts originated from the activities of those groups.
|
From page 72... ...
. WHAT ARE NAEP SHORT FORMS AND HOW MIGHT THEY BE USED' Policy for the NAEP Short Form In the most recent redesign policy, the short form is cited as a mechanism for simplifying NAEP design, specifically (National Assessment Governing Board, 1999a:71: Plans for the short-form of the National Assessment, using a single test booklet, are being implemented.
|
From page 73... ...
The short form could provide additional data collection opportunities that are not part of the standard NAEP schedule, such as testing in off years or in other subjects not assessed at the state level. TruLy described how some people envision using a short form: If short forms were developed and kept secure, they could provide flexibility to states and any jurisdiction below the state level that were interested in using NAEP for surveying student achievement in subjects, grades, and times that were not part of the regular state-NAEP schedule.
|
From page 74... ...
Participants also liked the idea of having a NAEP test to administer in "off-years" from regular NAEP administrations. Because NAEP does not currently administer every subject to every grade every year, workshop participants believed the short form could help fill the "gaps." The short form could be given every year thereby enabling the compilation of yearly trend data.
|
From page 75... ...
These concerns about the short form are discussed in detail in the workshop summary (National Research Council, 20001. Review of the Pilot Short Forms As explained in Chapter 4, ETS prepared two fourth grade mathematics short forms as part of the year 2000 pilot study.
|
From page 76... ...
While it does not seem to be the intent to represent the current NAEP item pool, or to create scales, the short form needs to be capable of providing estimates of the true score distribution that is the target for full NAEP. That distribution is needed to support policy Principle 5, "Since it is desirable to report the results of the short form using the achievement levels, the content achievement level descriptions should be considered dur
|
From page 77... ...
5. It should be possible to compare results from alternate forms of short forms for a curriculum area, which implies that the short forms are to be put on a common score scale, perhaps through an equating process.
|
From page 78... ...
Thus for practical reasons, the short form's representation of the framework must be incomplete, and the short form would represent the framework less well than the full NAEP pool represented the framework. An even looser interpretation of"represent" could be that the items on a short form provide selected examples of the kinds of items developed
|
From page 79... ...
Approaches to Constructing Short Forms The StandLard~s for Educational and Psychological Tests (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999) present guidelines to be followed in constructing a short form version of a longer test.
|
From page 80... ...
The short form would simply be a test containing fewer items than the long form. Under a domain sampling approach, a short form of NAEP could be developed by selecting a smaller sample of items than for the long form.
|
From page 81... ...
One clear difference between the short form and the long form is that scores from the short form will have a different reliability and standard error structure] than those from the full NAEP pool even though the short form and full NAEP provide samples from the same clomain of content (National Research Council, 20001.
|
From page 82... ...
Requirements 3 and 4 suggest that the short form allow estimation of means and percentages of distributions on the NAEP scale. This implies that the short form would produce scores on the same composite of skills as the full NAEP pool.
|
From page 83... ...
It will not be possible for a short form design to support all uses described by workshop participants. The most positive result that can be expected from attempts at short form construction is that the short form is shown to measure the same composite of skills and knowledge as the full NAEP pool and that the distribution of statistical item characteristics is such that the shape of the estimated score distribution will be similar, though not identical, to that for current NAEP.
|
From page 84... ...
Use of different score scales would preclude making direct comparisons, but the short form may still have value as a more frequent monitor of student capabilities. However, it is worth restating here that, to many workshop participants, being able to make comparisons with main NAEP was one ofthe more appealing features of the short form.
|
From page 85... ...
CHANGED NAEP: USE OFA SHORT-FORM VERSION OFNAEP that will provide estimates of proportions above achievement levels and means for subgroups of the examinee population that are of similar accuracy to those from current NAEP. RECOMMENDATION 5-3: If the decision is made to proceed with the short form, methods should be developed for reporting performance on the short form in a way that is meanin~ and not misleading given the differences in quality of estimates for current NAEP and the short form.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.