National Academies Press: OpenBook
« Previous: Front Matter
Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×

Summary

In 2007, the District of Columbia made a bold change in the way it governs public education with the goal of shaking up the system and bringing new energy to efforts to improve outcomes for students. The Public Education Reform Amendment Act (PERAA) shifted control of the city’s public schools from an elected school board to the mayor, created a new state department of education, created the position of chancellor, and made other significant management changes. PERAA also mandated an independent, comprehensive, 5-year evaluation to determine “whether sufficient progress in public education has been achieved to warrant continuation of the provisions and requirements of this act or whether a new law, and a new system of education, should be enacted by the District government.…”

To plan that evaluation, the Committee on the Independent Evaluation of DC Schools was convened by the National Research Council in response to a request from the City Council of the District of Columbia. The committee was asked not to conduct the evaluation, but to provide initial guidance on the focus and structure of the required evaluation. The work included identifying available data and assessing its quality and utility; developing a preliminary set of indicators; engaging with various stakeholder groups, including civic leaders, parents, researchers, and national and local reform experts; and exploring the desirability, feasibility, and scope of the optional next phases of the evaluation.

This report documents the committee’s plan for the evaluation. It lays out a plan for a comprehensive, long-term program of evaluation that is designed not only to examine short-term effects of the changes made under PERAA, but also to provide the District with a structure for continuous,

Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×

independent monitoring of important features of its school system. The plan is based on the committee’s review of preliminary data and on its conclusion that first impressions of the implementation of PERAA and its effects, though informative, are not sufficient as a basis for decisions about PERAA or continued improvement of the city’s education system.

The committee agreed on several basic assumptions and goals that have guided our work. First, although many U.S. cities have undertaken significant reforms to change their schools and researchers have examined what they have done, there is no established model for evaluating a district involved in reform—or, for that matter, any district. Second, school districts are judged primarily on the academic achievement of their students, but achievement depends on how effectively a school district accomplishes its many responsibilities and pursues many valued educational outcomes. Third, we interpreted PERAA’s requirement for an evaluation broadly: to establish for the residents and leaders of DC a sustainable ongoing program of evaluation that provides reliable information they can use to improve the school system continuously, regardless of future political or personnel changes. Last, the committee approached the most challenging part of its charge—to explore the effects of the reform legislation itself—by distinguishing among the intent of the reform, as articulated in the law; its implementation, that is, the actions taken by the DC Public Schools (DCPS) and other responsible city agencies; and its effects on student learning and other valued outcomes.

CONTEXT

PERAA is the latest in a long line of changes in the way the city’s public schools are governed. Since 1804, there have been 17 different governance and administrative structures, and PERAA was the second new approach since 2000. Many of these changes were responses to concerns about students’ academic performance, the quality of the schools and the teachers, and an ineffective central bureaucracy, as well as the perception that many DC residents were indifferent to the persistent problems.

The city’s education problems have been intensified by a history of segregation, and the city continues to struggle with many challenges related to race, poverty, and geography. Those challenges include inequitable distribution of resources and supports to schools in the lowest-income sections of the city, which are largely black, tensions over demographic shifts that change the character of neighborhoods, and a strong charter school movement. They have made reform efforts more urgent while complicating the city’s response to them.

Another factor in DC has been the city’s distinctive political status as a small geographic area under the jurisdiction of the federal government.

Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×

Because DC is not part of any state and elected its first mayor and city council only in 1973, it does not have a long tradition of self-governance. The U.S. Congress retains considerable authority over its affairs and budget.

PERAA was a response in part to these historical circumstances, but it was also spurred by impressions of the effectiveness of reforms in other urban districts facing at least somewhat similar economic, social, and historical challenges. Districts in Boston, Chicago, Cincinnati, Minneapolis, and New York City (among others) have focused on the alignment of content and performance standards with curricula, instruction, and other aspects of the school system. They have used data to guide their decisions, emphasizing such goals as improved professional development for teachers and principals; more frequent formative assessments; and the development of a culture of learning and collaboration among teachers. These approaches are widely used and are supported by some promising evidence, but the research literature is not yet settled enough to provide firm guidance on best practices for district reform or evaluation.

Some districts have also focused on the governance of schools, and a few (e.g., Boston, Chicago, Cleveland, and New York City) have given their mayors control over the public schools. Such reforms are designed to “jolt” the system by changing dysfunctional institutional relationships and giving leaders new lines of authority and accountability. Evaluation of these governance reforms is critical to knowing what really works and what does not, but few cities have made this a priority, so there are neither clear exemplars nor substantial evidence to guide the District as it implements PERAA.

IMPLEMENTATION OF PERAA

The District of Columbia has made many changes called for in PERAA. Thoroughly documenting the city’s efforts will be a critical component of the comprehensive evaluation the law requires, and until this is done, no firm conclusions should be drawn about how well the city has implemented PERAA and fulfilled its intentions. As a first step, however, we offer an outline of the city’s response to PERAA.

The new structures mandated in PERAA have largely been put into place. The mayor now has responsibility for most key aspects of the school system, including appointment of a chancellor who establishes educational priorities, adopts curricula and assessments, and ensures that the schools are appropriately staffed and managed. Also in place are the Department of Education, the Deputy Mayor for Education, the Office of the State Superintendent of Education (OSSE), the State Board of Education, and the Public Charter School Board. Not currently in place are the Office of Ombudsman for Public Education (a position that had been filled but was later eliminated) and the comprehensive data system.

Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×

DCPS has also adopted strategies to meet the goals of PERAA. Among them are efforts intended to improve the quality of teachers, principals, and administrators, including a new system for evaluating teacher performance; a new teaching and learning framework, which describes the specific instructional practices the district has identified as most likely to promote student learning; and improvements to school facilities.

FIRST IMPRESSIONS OF THE DC SCHOOL SYSTEM UNDER PERAA

Student Achievement

Public attention frequently is focused on fluctuations in student achievement scores, in DC as in the rest of the nation. First impressions offer a mixed picture: in general, scores on the District of Columbia Comprehensive Assessment System (DC CAS) have continued on an upward trajectory (which began before PERAA was enacted), and they have flattened slightly during the most recent 2 school years. However, definitive conclusions about PERAA’s effects cannot be drawn from these preliminary results for three major reasons:

  1. The DC CAS is designed to measure students’ mastery of specific academic skills, but determining whether and how the changes in district policies or strategies have contributed to those skills requires additional empirical evidence: the scores themselves do not provide evidence about what accounts for them.
  2. The available scores are averaged across the entire student population, and do not provide information on the status or progress of specific groups: some may be making sharp gains while others are not.
  3. Because DC is a highly mobile district and the student population changes every year, score fluctuations may be the result of changes in the characteristics of the students taking the test, rather than improvements or declines in students’ knowledge and skills.

Thus, in order to draw any conclusions about the effect of PERAA on student achievement as measured by DC CAS, further study of patterns for types of schools, individual schools, grade levels, neighborhoods, wards, and population subgroups is needed, and this should include longitudinal studies of cohorts of students within the District.

Scores from the National Assessment of Educational Progress (NAEP) provide independent information about achievement trends in all 50 states and DC, and these results also suggest that, in general, DC students’ performance has been improving. However, as with the DC CAS scores, more

Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×

study is needed to understand the reasons for trends. Like DC CAS, NAEP does not account for changes in the demographics of the population, so it is not possible to tell from these scores alone whether the improvements are the result of demographic shifts rather than changes in educational policies, programs, or practices.

School Quality and Operations

Like any district, DC is responsible for setting high expectations for all students and providing them with the instruction and resources necessary to meet them. Test scores only provide evidence, partial at that, about one aspect of the system. A school system’s responsibilities are more complicated, and can be categorized in five broad areas:

  1. quality of personnel (teachers, principals, and others),
  2. quality of classroom teaching and learning,
  3. capacity to serve vulnerable children and youth,
  4. promotion of family and community engagement, and
  5. quality and equity of operations, management, and facilities.

The District seems to have made changes in these areas, but a comprehensive evaluation would be needed to determine whether, how, and where conditions are improving. The District is already collecting data on many of these functions, and a first step in the evaluation will be to systematically assess these measures, determine which will be useful for the evaluation program, and identify priorities for new data collection—a task that was beyond the resources of this committee.

FROM IMPRESSIONS TO EVIDENCE: AN EVALUATION PLAN

RECOMMENDATION 1 We recommend that the District of Columbia establish an evaluation program that includes long-term monitoring and public reporting of key indicators as well as a portfolio of in-depth studies of high-priority issues. The indicator system should provide long-term trend data to track how well the programs and structure of the city’s public schools are working, the quality and implementation of key strategies undertaken to improve education, the conditions for student learning, and the capacity of the system to attain valued outcomes. The in-depth studies should build on indicator data. Both types of analysis should answer specific questions about each of the primary aspects of public education for which the District is responsible: personnel (teachers, principals, and others); classroom teaching and learning; vulnerable children and youth; fam-

Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×

ily and community engagement; and operations, management, and facilities.

Figure S-1 depicts our proposed evaluation framework. It begins with the goals the district has set for itself, as shown in the horizontal box that appears at the top of the figure. The logic of this framework reflects the point that passing a law does not automatically result in increased student learning, reduced achievement gaps, increased graduation rates, or other valued outcomes. To achieve these outcomes, the new structures and relationships that PERAA mandated have to be established and working as intended; school system leaders have to have implemented strategies that are likely to be effective; those strategies have to be well implemented; and the conditions for student learning—such as the quality of school staff and instruction—have to have improved.

In addition to the elements of reform, the evaluation has to cover the broad areas of the school district’s responsibility: see the shaded horizontal stripes that cut across the elements of reform in Figure S-2. In this elaboration of Figure S-1, the elements of reform and the broad evaluation questions pertaining to them are depicted in the vertical boxes, and the substantive areas of responsibility are depicted with shaded horizontal bands. This framework is designed to guide the evaluation so that it is comprehensive: even if resources limit the specific analyses that can be undertaken at a given time, use of the framework will ensure that the most important aspects of the system are examined.

The framework is a depiction of the primary components of reform and of the district’s responsibilities. The basic questions to be asked under each of the four elements and across the five areas of responsibility will need to be answered using many different study designs, data collection methods, and types of analysis. Thus, the evaluation framework provides a guide to the kinds of information that are needed to fully inform policy makers and the public. The indicators—which should be developed in conjunction with OSSE and DCPS and members of the community—should provide long-term disaggregated trend data to track how well district roles and structures are working, the quality and implementation of key strategies undertaken to improve education, the conditions for student learning, and valued outcomes. The in-depth studies should draw from the indicators, as well as other data, to provide detailed answers to specific questions about key aspects of public education in the District.

It will be critical to establish stable indicators as soon as possible, supplementing and refining those the District is already collecting as needed; however, the program of focused evaluation studies will evolve over time as changes in the city’s policies, challenges, and circumstances require. Many empirical questions are subsumed in the elements of reform and

Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×

image

FIGURE S-1 A framework for evaluating DC public education under the Public Education Reform Amendment Act.

Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×

image

FIGURE S-2 Evaluation priorities in key areas of distinct responsibility.

Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×

the broad categories of district responsibility. The evaluators will look to District leaders and other members of the community to establish the priorities and available resources that will guide the choice of specific indicators and studies, and the long-term indicator system will build on data collection efforts already in place in the District. The evaluation needs to engage the perspectives, concerns, and needs of all who are part of and care about the system: students (and youth who are disconnected from school), families, educators, administrators, and the community. The key evaluation questions, the data used to answer them, and how these answers are shared and used need to be designed with the concerns and goals of the community in mind.

RECOMMENDATION 2 The Office of the Mayor of the District of Columbia should produce an annual report to the city on the status of the public schools, drawing on information produced by the District of Columbia Public Schools and other education agencies and by the independent evaluation program that includes

  • summary and analysis of trends in regularly collected indicators,
  • summary of key points from in-depth studies of target issues, and
  • an appendix with complete data and analysis.

Building and maintaining a high-quality indicator system, designing studies that address pressing issues, and presenting and disseminating findings so that all stakeholders can act on them will require deliberate and skillful management. An independent evaluation program that is an ongoing source of objective information and analysis will be an invaluable resource for the city under changing political circumstances. To make such a program work, the District of Columbia will need to engage potential research partners and funders in planning and developing an infrastructure for ongoing independent evaluation of the city’s public schools.

Urban districts face some of the most difficult challenges in U.S. public education, and many have pursued ambitious reforms. Valuable lessons have begun to emerge from their experiences; systematically evaluating these efforts and their effects is a critical part of education reform. Objective evidence derived from multiple sources of data is a tool for monitoring progress and guiding continuous improvement in a city’s schools—and also for ensuring that their benefits can be sustained and replicated in other districts. It is our hope that this model will be of use to districts around the country.

Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×

This page intentionally left blank.

Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×
Page 1
Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×
Page 2
Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×
Page 3
Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×
Page 4
Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×
Page 5
Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×
Page 6
Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×
Page 7
Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×
Page 8
Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×
Page 9
Suggested Citation:"Summary." National Research Council. 2011. A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence. Washington, DC: The National Academies Press. doi: 10.17226/13114.
×
Page 10
Next: 1 Introduction »
A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence Get This Book
×
Buy Paperback | $46.00 Buy Ebook | $36.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The District of Columbia (DC) has struggled for decades to improve its public education system. In 2007 the DC government made a bold change in the way it governs public education with the goal of shaking up the system and bringing new energy to efforts to improve outcomes for students. The Public Education Reform Amendment Act (PERAA) shifted control of the city's public schools from an elected school board to the mayor, developed a new state department of education, created the position of chancellor, and made other significant management changes.

A Plan for Evaluating the District of Columbia's Public Schools offers a framework for evaluating the effects of PERAA on DC's public schools. The book recommends an evaluation program that includes a systematic yearly public reporting of key data as well as in-depth studies of high-priority issues including: quality of teachers, principals, and other personnel; quality of classroom teaching and learning; capacity to serve vulnerable children and youth; promotion of family and community engagement; and quality and equity of operations, management, and facilities. As part of the evaluation program, the Mayor's Office should produce an annual report to the city on the status of the public schools, including an analysis of trends and all the underlying data.

A Plan for Evaluating the District of Columbia's Public Schools suggests that D.C. engage local universities, philanthropic organizations, and other institutions to develop and sustain an infrastructure for ongoing research and evaluation of its public schools. Any effective evaluation program must be independent of school and city leaders and responsive to the needs of all stakeholders. Additionally, its research should meet the highest standards for technical quality.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!