NAEP Reporting Practices Investigating District-Level and Market-Basket Reporting (2001) / Chapter Skim
Currently Skimming:
Market-Basket Reporting
Market-Basket Reporting
Pages 50-70
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 50... ...
These documents referred to market-basket reporting as "domainscore reporting" where a "goodly number of test questions are developed that encompass the subject, and student results are reported as a percentage of the 'domain' that students 'know and can do"' (National Assessment Governing Board, 1996:131. According to these documents, the general idea of a NAEP market basket draws on an image similar to the Consumer Price Index (CPI)
|
From page 51... ...
. They consider percent correct scores as the metric for summarizing performance on the collection of items, a metric NAEP's sponsors believe is widely understood (National Assessment Governing Board, 19971.
|
From page 52... ...
The final section of the chapter presents details on the methodology for constructing and reporting results for the CPI market basket. STUDY APPROACH During the course of the study, the committee reviewed the literature and the policy guidelines pertaining to market-basket reporting, including the following documents: Design and Feasibility Team's report to NAGB (Forsyth et al., 1996~; ETS's proposal (Educational Testing Service, 1998~; various studies on domain score reporting (Bock, 1997; Bock, Thissen, & Zimowski, 1997; Pommerich & Nicowander, 1998~; and policy guidelines included in the 1996 NAEP Redesign (National Assessment Governing Board, 1996)
|
From page 53... ...
During the workshop, Robert Mislevy laid the conceptual groundwork for the technical and measurement issues involved in market-basket reporting (Mislevy, 20001; Andrew Kolstad traced the history of NAEP reporting practices (Kolstad, 20001; and John Mazzeo described features of the pilot study currently under way on the market basket. In the section that follows, we draw from the ideas presented by Mislevy, Kolstad, and Mazzeo and from other sources to delineate the psychometric issues that must be addressed in designing a NAEP market basket.
|
From page 54... ...
54 o so o :Fill H E Q ~ U' ~ A o ~ O Z ~ ~ ~ '`a a ~ ~ In ~ It .
|
From page 55... ...
In constructing the market basket, the alignment of the item pool to the framework, as well as the framework's representation of the broad domain, have a substantial impact on the potential validity of inferences based on market-basket scores. Given that the pilot study defines the domain as the pool of existing and newly constructed NAEP items (Mazzeo, 2000)
|
From page 56... ...
A third alternative could be a collection of synthetic short forms, which together would provide a more detailed representation of the NAEP framework than would a single short form. This would overcome the limitations in coverage of a single short form.
|
From page 57... ...
Constructing Multiple Market-Basket Forms As stated above, market-basket reporting could be based on a single short form. Given the breadth of the NAEP frameworks, however, short forms necessarily will be limited in the way that they represent the NAEP frameworks.
|
From page 58... ...
Percent correct scores based on different forms can, thus, be equated, and adjusted percent correct scores reported. 1 Classically parallel forms must, theoretically, yield score distributions with identical means and variances for both observed scores and true scores.
|
From page 59... ...
estimated latent trait proficiency scores. Observed Scores The observed score metric is based on a tally of the number of right answers or the number of points received.
|
From page 60... ...
Observed scores can be quickly converted to a percent correct or percent of maximum score by adding the number correct on the multiplechoice items and the points received on the constructed response items and then dividing the sum by the total number of possible points. Observed scores have the problem of being tied to the composition and difficulty of the collection of items on the test form.
|
From page 61... ...
Reporting on an estimated true score or estimated observed score metric means working with predictive distributions of these scores which requires statistical procedures that are more complex than those for reporting observed number correct or percent correct scores. Estimated(Domain Score As defined by Bock (1997)
|
From page 62... ...
NAEP's current system for developing and field testing items was set up to support the construction of a system of arbitrary test forms in an efficient manner and does not yet have guidelines for constructing market baskets or intact tests. A NAEP test development committee handled construction of the short forms.
|
From page 63... ...
One oftheir planned studies involves conducting separate analyses of the year-2000 data using methods appropriate for arbitrary forms, methods appropriate for congeneric forms, and methods appropriate for parallel forms. Each of these sets of analyses will produce results in an observed score metric as well as a true score metric.
|
From page 64... ...
Even though the market basket set of items would be representative of NAEP, some state testing programs cover content similar to that assessed by NAEP. Therefore, NAEP's release of items could increase understanding of state and local assessments.
|
From page 65... ...
Assessment directors and measurement experts commented that percent correct scores were not as simple as they might seem. For instance, would percent correct be based on the number of correct answers or the number of possible points?
|
From page 66... ...
Because the NAEP market basket has been linked with the CPI from its inception, the committee thought it would be useful to learn more about how the CPI was constructed and how it might be applied in an educational setting. During the committee's workshop on market-basket reporting, Kenneth Stewart from the Bureau of Labor Statistics (BLS)
|
From page 67... ...
In the diary component of the Consumer Expenditure survey, consumers are asked to maintain a complete record of expenses for two consecutive one-week periods. The Consumer Expenditure diary survey was designed to obtain detailed data on frequently purchased small items, including food and beverages (both at home and in eating places)
|
From page 68... ...
Once selected, the CPI geographic sample is fixed for 10 years until new census data become available. Using the information supplied by families in the Consumer Expenditure surveys, the BLS constructs the CPI market basket by partitioning the set of all consumer goods and services into a hierarchy of increasingly detailed categories, referred to as the CPI item structure.
|
From page 69... ...
In considering the proposals to develop and report a summary measure from the existing NAEP frameworks, the committee realized that the proposals for the NAEP market basket differ fundamentally from purpose and construction of the CPI market basket. Although the NAEP frameworks are developed by committees of experts familiar with school-level curricula,
|
From page 70... ...
RECOMMENDATION 4-2: In describing the various proposals for reporting a summary measure from the existing NAEP frameworks, NAEP's sponsors should refrain from using the term "market basket" because of inaccuracies in the implied analogy with the CPI. RECOMMENDATION 4-3: If, given the issues raised about market-basket reporting, NAEP's sponsors wish to pursue the development of this concept, they should consider developing an educational index that possesses characteristics analogous to those of the Consumer Price Index: (1)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.