National Academies Press: OpenBook

Reporting District-Level NAEP Data: Summary of a Workshop (2000)

Chapter: 6 Summing Up: Issues to Consider and Resolve

« Previous: 5 Factors That Influence Interest in District-Level NAEP
Suggested Citation:"6 Summing Up: Issues to Consider and Resolve." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×

6

Summing Up: Issues to Consider and Resolve

One of the intended goals of the workshop was to highlight factors that would affect states’ and districts’ interest in having district-level results. While no attempt was made to establish consensus about any of the suggestions that were made, several themes emerged from the interactions among workshop participants. The following discussion points out issues that need to be considered and resolved before implementation decisions can be made.

CLARIFY THE GOALS AND OBJECTIVES

The goals of district-level reporting for NAEP were not apparent to workshop participants. Some spoke of using tests for accountability purposes, pondering whether NAEP could be used in this way or not. They discussed the amount of testing currently done in their schools and stressed that new testing would need to be accompanied by new (and better) information. However, some had trouble identifying what new and better information might come from district-level NAEP data. Their comments might have been different, and perhaps more useful, if they had a clear idea of the purposes and objectives for district-level reporting. It would be helpful to have an explicit statement of the goals and objectives for district-level reporting combined with a logical argument for how the program is expected to achieve the desired outcomes.

Suggested Citation:"6 Summing Up: Issues to Consider and Resolve." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×

DEVELOP SPECIFICATIONS

There were varying understandings among the participants about the nature and characteristics of district-level reporting. Participants talked about the possibility of receiving information that is currently part of national NAEP, commenting that the breadth of content areas and grades tested were attractive features. Some discussed receiving school-level or individual-level scores. Others generated their own assumptions in order to respond to the questions posed to them. One speaker stated his self-formulated assumptions then noted that if any of the assumptions proved to be inaccurate, “my testimony would change accordingly.”

Many speakers highlighted background data as being very useful and were intrigued by the prospect of seeing performance data broken down by background characteristics. Speakers were not aware that given the sample sizes required, such information is not likely to be provided. Their comments suggest the need for clear statements on the specifications for and constraints on district-level reporting.

EVALUATE COSTS AND BENEFITS

What would districts and states receive? When would they receive the information? How much would it cost? What benefits would be realized from the information? Workshop participants responded to questions about their interests in the program without having answers to these questions. Nonetheless, many said that their interest would depend on the answers. They need information on the types of reports to be prepared—a prototype report would be very useful for this—and the associated costs. Would there be varying levels of expense depending on the nature of the report?

Very important to the participants would be the time lag in receiving reports. Would they receive information in time to use it in their decision and policy making? Or would the time delays be such as to render the information useless? Are there options for reports that would require less preparation time?

In addition to monetary concerns, the costs in terms of time and effort on the part of teachers and students must also be considered. School systems already extensively test fourth and eighth graders. If more time is to be taken away from instruction for the purpose of additional testing, the benefits of the testing need to be laid out. Will additional testing amplify the information already provided? Or will the information be redundant

Suggested Citation:"6 Summing Up: Issues to Consider and Resolve." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×

to that provided from current tests? Will the redundancy make it useful for external validation? Such information is important if NAEP’s stewards want to assess actual levels of interest.

EVALUATE PARTICIPATION LEVELS

Many speakers talked of the value of being able to make interdistrict comparisons based on districts with like characteristics. However, this use of the results rides on the assumption that sufficient numbers of districts will participate. Previous experiences with district-level reporting resulted in between 10 and 12 interested districts in 1996 and virtually no interested districts in 1998.

Meaningful comparisons, as defined by demographic, political, and other contextual variables that districts believe are important, depend on there being a wide variety of other districts with district-level reports. Having only a handful of districts that meet the sampling criteria may limit one of the most fundamental functions for district-level reporting, that is, having a carefully selected group of other districts against which to compare results. Thus, if making comparisons is the primary objective for receiving district-level reports, the targeted districts must feel secure in knowing that there are sister districts also completing the necessary procedures for receiving district-level results. The extent of participation will limit the ability to make the desired comparisons.

CONSIDER WAYS TO SERVE SMALL DISTRICTS

According to the sampling criteria, participation would be limited because many districts would not qualify to receive reports. However, if having district-level data turns out to be associated with educational improvements, should small districts be denied access to such an important program? The Third International Mathematics and Science Survey (TIMSS) has permitted a district consortium for the most recent assessment. Would such consortia be allowed for NAEP reporting, and would the reports be meaningful to the participants? If credible results are to be provided for all individual districts, would this necessitate implementing district NAEP as a census rather than a sample? These are issues that NAEP’s stewards need to address as they consider the goals, objectives, specifications, and components for district-level reporting.

Suggested Citation:"6 Summing Up: Issues to Consider and Resolve." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×

CONSIDER THE IMPACT OF RAISING THE STAKES

A concern expressed when state NAEP was first implemented related to the increased stakes that would be associated with reporting data for smaller units. The message from several speakers (particularly district representatives) was that district-level reports would raise the stakes associated with NAEP. An evaluation of the effects of higher stakes, particularly as they relate to the types of inferences that may be made, would be important.

Suggested Citation:"6 Summing Up: Issues to Consider and Resolve." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×
Page 38
Suggested Citation:"6 Summing Up: Issues to Consider and Resolve." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×
Page 39
Suggested Citation:"6 Summing Up: Issues to Consider and Resolve." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×
Page 40
Suggested Citation:"6 Summing Up: Issues to Consider and Resolve." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×
Page 41
Next: References »
Reporting District-Level NAEP Data: Summary of a Workshop Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National Assessment of Education Progress (NAEP) has earned a reputation as one of the nation's best measures of student achievement in key subject areas. Since its inception in 1969, NAEP has summarized academic performance for the nation as a whole and, beginning in 1990, for the individual states. Increasingly, NAEP results get the attention of the press, the public, and policy makers. With this increasing prominence have come calls for reporting NAEP results below the national and state levels. Some education leaders argue that NAEP can provide important and useful information to local educators and policy makers. They want NAEP to serve as a district-level indicator of educational progress and call for NAEP results to be summarized at the school district level.

Reporting District-Level NAEP Data explores with various stakeholders their interest in and perceptions regarding the likely impacts of district level reporting.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!