NAEP Reporting Practices Investigating District-Level and Market-Basket Reporting (2001) / Chapter Skim
Currently Skimming:
Reporting District-Level NAEP Results
Reporting District-Level NAEP Results
Pages 30-49
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 30... ...
For a variety of reasons, neither attempt revealed much interest on the part of school districts. The lack of interest was attributable, in part, to financial considerations and to unclear policy about whether the state or the district had the ultimate authority to make participation decisions.
|
From page 31... ...
Individuals representing state and district assessment offices participated and commented on their interest in and potential uses for district-level results. Representatives from national organizations (Council of Chief State School Officers and Council of Great City Schools)
|
From page 32... ...
But reporting results at the state level could increase pressure on states to change their instructional practices, which could threaten the validity of NAEP scores (Koretz 1991:21)
|
From page 33... ...
Furthermore, correlations of student achievement on NAEP with data about instructional practices obtained from the background information do not imply causal relationships. For example, the 1994 NAEP reading results showed that fourth-grade students who received more than 90 minutes of reading instruction a day actually performed worse than students receiving less instruction.
|
From page 34... ...
Although these changes could not be directly attributed to the implementation of the TSA, they reflected priorities also set for the NAEP reading assessment. In addition, many state assessment measures were expanded to include more open-ended response items, with an increased emphasis on the use of authentic texts and passages, like those found on NAEP (Hartka & Stancavage, 19941.
|
From page 35... ...
He finds that NAEP's ability to collect high quality data comparably over time and across sites lends it to powerful uses for tracking both student achievement and background information. According to Selden, questions that might be addressed by trend data include: are instructional practices changing in the desired directions; are the characteristics of the teacher workforce getting better; and are home reading practices improving.
|
From page 36... ...
According to Roeber (1994:42) , such overt action would be needed "to assure that reporting does not distort instruction nor negatively impact the validity of the NAEP results now reported at the state and national levels." EXPERIENCES WITH DISTRICT-LEVEL REPORTING NAG B and NCES supported the initiative to provide district-level results, hoping that school districts would choose to use NAEP data to inform a variety of education reform initiatives at the local level (National Assessment Governing Board, 1995a; National Assessment Governing Board, 1995b)
|
From page 37... ...
To gauge interest in the plan, NCES and ETS sponsored a meeting during the 1995 annual meeting of the American Educational Research Association, inviting representatives from several of the larger districts in the country. Based on this meeting and further interaction with district representatives, NCES identified approximately 10 school systems interested in obtaining their NAEP results.
|
From page 38... ...
The reports summarizing performance by teaching practices and by background variables and institutional practices were more useful and interesting. Milwaukee officials found that the NAEP results generally supported the types of instructional practices they had been encouraging.
|
From page 39... ...
NAGB discussed the issue at its August 1999 meeting and decided that no further offers of district results should be made until it was clear who should be the deciding entity (National Assessment Governing Board, 199941. TECHNICAL AND POLICY CONSIDERATIONS FOR DISTRICT-LEVEL REPORTING As part of the workshop on district-level reporting, the committee asked representatives from NAGB, NCES, ETS, and Westat to discuss the technical issues related to sampling and scoring methodologies and the policy issues related to participation and reporting decisions.
|
From page 40... ...
Districts that desired to report mean proficiencies by background characteristics such as race, ethnicity, type of courses taken, home-related variables, instructional variables, and teacher variables would need sample sizes approximately one-half of their corresponding state sample sizes, or approximately 1,500 students from a minimum of 50 schools. For reporting, the "rule of 62" would apply, meaning that disaggregated results would be provided only for groups with at least 62 students (National Assessment Governing Board, 1995b: Guideline 31.
|
From page 41... ...
She described procedures used to generate the likely ability distributions for individuals, based on their background characteristics and responses to NAEP items (the conditioning procedures) , and to randomly draw five ability estimates (plausible values)
|
From page 42... ...
Because of this ambiguity about decision-making procedures, NAGB passed the following resolution (National Assessment Governing Board, 199941: Since the policy on release of district-level results did not envision a disagreement between state and district officials, the Governing Board hereby suspends implementation of this policy, pending legislation which would provide that the release of district-level NAEP results must be approved by both the district and state involved. The committee asked workshop participants to discuss their opinions about the entity (states or districts)
|
From page 43... ...
District officials believed that if performance could be compared among districts with similar characteristics, state officials might be more likely to set more reasonable and achievable expectations. Further, they noted that this practice might allow them to identify districts performing better than expected, given their demographics, and attention could focus on determining instructional practices that work well.
|
From page 44... ...
Attempts to use NAEP as a means of external validation for the state assessment is problematic when the state assessment is aligned with instruction and NAEP is not, particularly if results from the different assessments suggest different findings about students' achievement. In addition, confusion arises when NAEP results are released at the same time as state or local assessment results.
|
From page 45... ...
Those that looked forward to receiving data summarized by background characteristics would likely be disappointed given the sample sizes required to obtain such information. Other state and district officials commented that their reactions to the propositions set forth by NAEP's sponsors would depend upon the details.
|
From page 46... ...
In smaller districts, all schools might be included in the sample, thereby eliminating completely the portion of sampling error associated with between-school differences. Technical experts and others at the workshop encouraged NCES and Westat to pursue sampling specifications and focus on the estimated overall accuracy of results rather than on specifying an arbitrary minimum number of schools based on current procedures for State or National NAEP.
|
From page 47... ...
District officials, state officials, and other NAEP users the potential users of the new product had a difficult time responding to questions about the product's desirability because a clear conception of its characteristics was not available. The most important issues requiring resolution are described below.
|
From page 48... ...
The message from several workshop speakers (particularly district representatives) was that district-level reports would raise the stakes associated with NAEP and change the way NAEP results are used.
|
From page 49... ...
The reactions of potential users and the responses from product designers tend to produce a series of interactions like "Tell me what the new product is and I will tell you if I like it," versus "Tell me what you would like the product to be and I will make sure it will have those characteristics." During the committee's workshop, state and district representatives were put in the position of responding to the latter question. Here, the developer is asking the user to do some of the design work.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.