|
||||||||||||||||
|
||||||||||||||||
The following HTML text is provided to enhance online readability. Many aspects of typography translate only awkwardly to HTML. Please use the page image as the authoritative form to ensure accuracy. 1
|
||||||||||||||||
 |
 |
Page 11 |
 |
 |
 |
|
| ||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 11
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
1
Introduction: The Government Performance and Results Act, the Program Assessment Rating Tool, and the Environmental Protection Agency
Federal administrations have long attempted to improve alignment of the spending decisions of the U.S. federal government with the expected results of the decisions (OMB 2004). In the 1990s, Congress and the executive branch devised a statutory and management framework to strengthen the performance and accountability of all federal agencies; the Government Performance and Results Act (GPRA) of 1993 was its centerpiece.
GPRA focused agency and oversight attention on the performance and results of government activities by requiring federal agencies to measure and report annually on the results of their activities. For the first time, each of about 1,000 federal programs was required to explicitly identify metrics and goals for judging its performance and to collect information each year to determine whether it was meeting the goals (OMB 2004). GPRA required agencies to develop a strategic plan that set goals and objectives for at least a 5-year period, an annual performance plan that translated the goals into annual targets, and an annual performance report that demonstrated whether the targets were met (NRC 1999). A key objective of GPRA was to create closer and clearer links between the process of allocating limited resources and the expected results to be achieved with them (Posner 2004).
INHERENT DIFFICULTIES IN EVALUATING RESEARCH
As agencies developed strategies to comply with GPRA, it became clear that the evaluation of science and technology research programs, especially those involving basic research, created challenges for both the agencies and
OCR for page 12
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
oversight bodies. That, of course, is true for many other fields and practices. In the particular case of science, especially basic research, a fundamental challenge is that the course of research cannot be planned or known in advance; research entails continual feedback from observation and experimentation, which leads to new directions.
As Donald Stokes has written, “research proceeds by making choices. Although the activities by which scientific research develops new information or knowledge are exceedingly varied, they always entail a sequence of decisions or choices.” They include the choice of a problem, construction of theories or models, development of instruments or metrics, and design of experiments or observations (Stokes 1997). Stokes wrote that the defining quality of basic research is that it seeks to widen the understanding of phenomena in a scientific field. In any search for new understanding, the researcher cannot know in advance what that understanding will be and therefore cannot know how long it will take, how much it will cost, and what instrumentation will be required; so the ability to evaluate progress against benchmarks is slight. Applied research is similar to basic research in that it has the same underlying process of inquiry, but it is often distinct from basic research in emphasizing the extension of fundamental understanding to “some individual or group or societal need or use” (Stokes 1997). The intended outcomes of applied research, which include methods development and monitoring, are usually well known in advance.
The committee believes that the terms basic research, applied research, and development describe overlapping and complementary activities. The process of research might be visualized as the development and flow of knowledge within and across categorical boundaries through collaboration, feedback loops, and fortuitous insights. Agencies support many levels of research to sustain a needed flow of knowledge, respond quickly to current demands, and prepare for future challenges.
Attempts to evaluate research in terms of efficiency may founder because of the very nature of research. Thus, a negative or unexpected result of a scientific test can have value even if the time and other resources consumed by the test might be judged “inefficient” by some metrics. In addition, much of the work of researchers involves building on, integrating, and replicating previous results and this might also appear “inefficient.”
RESEARCH TERMS AT THE ENVIRONMENTAL PROTECTION AGENCY
The Environmental Protection Agency (EPA) uses a particular nomenclature to describe its research, including the terms core research and problem-driven research. Those terms were coined by a National Research Council committee that recommended “that EPA’s research program maintain a balance between problem-driven research, targeted at understanding and solving particular identified environmental problems and reducing the uncertainties associated
OCR for page 13
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
with them, and core research, which aims to provide broader, more generic information to help improve understanding relevant to environmental problems for the present and the future” (NRC 1997). The report added that the distinction at EPA between core and problem-driven research is not always clear.1
EVALUATING RESEARCH UNDER THE GOVERNMENT PERFORMANCE AND RESULTS ACT
In the late 1990s, the National Academies was asked for advice on how to evaluate the research programs of federal agencies, and the Committee on Science, Engineering, and Public Policy (COSEPUP) undertook a study (NRC 1999) that began with a review of how federal research agencies were addressing GPRA. That committee determined that
The useful outcomes of basic research cannot be measured directly on an annual basis because their timing and nature are inherently unpredictable.
Meaningful criteria do exist by which the performance of basic research can be evaluated while the research is in progress: quality, relevance, and, when appropriate, leadership, as measured in the context of international standards.
Such evaluations are best performed by “expert-review panels,” which, in addition to experts in the field under review, include experts in related fields who may be drawn from academe, industry, government, and other appropriate sectors.
Measurements based on those criteria can be reported regularly to assure the nation a good return on its investments in basic research.
Two years later, when more information about agencies’ efforts to comply with GPRA was available, a panel appointed by COSEPUP reiterated and expanded on the original recommendations.2 The panel focused on the five agencies that provide the majority of federal funding for research3 and found that all had made good-faith efforts to comply with the requirements of GPRA. It also determined that some oversight bodies and agencies needed clearer procedures to validate agency evaluations and that compliance techniques, communication
1
The report also pointed out that “the terms were not the same as basic vs applied research, fundamental vs directed research, or short-term vs long-term research, which are typically used by other federal agencies and researchers.”
2
In addition, COSEPUP has extended experience since the time of GPRA in helping OMB to interpret the application of government-wide criteria to agency research programs. Workshops were organized by COSEPUP for the OMB in 2001-2002 on the R&D Investment Criteria, and in 2004 on the PART.
3
The five agencies are the Department of Defense, the Department of Energy, the National Aeronautics and Space Administration, the Department of Health and Human Services (specifically, the National Institutes of Health), and the National Science Foundation.
OCR for page 14
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
with oversight bodies, and timing requirements of agencies varied widely (NRC 2001).
THE RATIONALE AND FUNCTION OF THE PROGRAM ASSESSMENT RATING TOOL
Although GPRA increased the information on the results and performance of federal agencies, the Office of Management and Budget (OMB) wanted to improve the usefulness of the process and therefore developed the Program Assessment Rating Tool (PART) in 2002. PART was designed for use throughout the federal government in the larger context of “performance budgeting” and “performance measurement.” Performance budgeting seeks to design budgeting procedures that optimize efficiency and effectiveness, including the most effective mechanisms for allocating available resources and holding program managers accountable for results. The first draft of PART was released for comment in May 2002 (OMB 2004), and OMB announced its intention to review each federal program every 5 years and to complete the first cycle in 2007 (OMB 2004).
PART itself is a questionnaire of at least 25 questions, whose number varies slightly with the type of program being evaluated.4 The questions are arranged in four categories by which programs are assessed: purpose and design, strategic planning, management, and results and accountability. The answers to the questions result in a cumulative numeric score of 0-100 (100 is the best). Depending on that score, the program’s rating is “effective,” “moderately effective,” “adequate,” or “ineffective.” Programs that have not developed acceptable performance metrics or sufficient performance data generally receive a rating of “Results not demonstrated.” Box 1-1 shows the distribution of PART scores for the 1,016 programs rated.
PART constitutes an important step in program assessment. As one General Accountability Official told Congress, “PART may mark a new chapter in performance-based budgeting by more successfully stimulating demand for this information—that is, using the performance information generated through GPRA’s planning and reporting processes to more directly feed into executive branch budgetary decisions” (Posner 2004).
The process of PART implementation is still new, and agencies are still developing their compliance methods. Some 19% of federal programs are rated “Results not demonstrated” (OMB 2007a).
THE APPLICATION OF THE PROGRAM ASSESSMENT RATING TOOL TO RESEARCH
For evaluating research programs, the PART system adopted the criteria of quality and relevance suggested by National Academies reports. Those crite-
4
See Appendix C for the full list of PART questions.
OCR for page 15
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
ria are addressed in multiple PART questions, especially the first two sections, which explore program purpose and design and strategic planning. In lieu of the “leadership” criterion, OMB developed a new criterion of “performance.”5 This is described in terms of both effectiveness (the ability to achieve useful results) and efficiency6 (the ability to achieve results with little waste). OMB states in its most recent PART guidance document that “because a program’s performance goals represent its definition of success, the quality of the performance goals and actual performance [in achieving] those goals are the primary determinants of an overall PART rating” (OMB 2007b, p. 7). An annual retrospective analysis of results is also required.7 Box 1-2 indicates how PART questions are scored. Reports and scores for EPA ORD programs can be found in OMB (2007a).
It appears, from the outset of its planning for PART, OMB recognized that research programs “would pose particular challenges for performance assessments and evaluations. For instance, in both applied and basic research, projects take several years to complete and require more time before their meaning for the field can be adequately understood and captured in performance reporting systems” (Posner 2004).
BOX 1-1
Distribution of PART Scores
Rating
Percent and Number
Effective
19% (186)
Moderately effective
31% (319)
Adequate
29% (289)
Ineffective
3% (27)
Results not demonstrated
19% (195)
According to OMB, “[PART] assumes that a program that cannot demonstrate positive results is no more entitled to funding, let alone an increase, than a program that is clearly failing.” The consequences of a failing PART grade therefore would be nontrivial for a public agency, especially such a regulatory agency as EPA, whose actions take place in a political context.
Source: OMB 2007a.
5
This distinction drew from the Army Research Laboratory’s evaluation criteria of quality, relevance, and productivity.
6
Efficiency is the ratio of the outcome or output to the input of any program (OMB 2006).
7
According to OMB (2007b, p. 85), “programs must document performance against previously defined output and outcome measures, including progress toward objectives, decisions, and termination points or other transitions.”
OCR for page 16
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
BOX 1-2
How PART Questions Are Scored
The four assessment categories are weighted according to the following scheme:
Program purpose and design: 20% (5 questions).
Strategic planning: 10% (8 questions).
Program management: 20% (11 questions for R&D programs).
Results and accountability: 50% (5 questions).
As a default, individual questions within a category are assigned equal weighting that total 100% for each section. However, weighting may be altered to emphasize key factors of the program.
In its 2004 budget statement, OMB wrote that the difficulty was most relevant to its preferred evaluation approach, which was to measure the outcomes8 of research (OMB 2004): “It is preferable to have outcome measures, but such measures are often not very practical to collect or use on an annual basis. The fact is there are no ‘right’ measures for some programs. Developing good measures is critical for making sure the program is getting results and making an impact.”
To assist agencies with significant research programs, additional instructions were added to the PART guidance and titled the “Research and Development Program Investment Criteria.” The R&D Investment Criteria are found in Appendix C of the PART instructions (see Appendix G). The main body of the PART instructions applies to all federal agencies and programs, including those that perform R&D; the R&D Investment Criteria attempt to clarify OMB’s expectations for R&D programs. However, a shortcoming of the Investment Criteria is that the section on Performance does not use the word efficiency, so that agencies have had to extrapolate from other sections in the guidance in evaluating that criterion.
THE ORGANIZATION AND PERFORMANCE OF RESEARCH AND DEVELOPMENT AT THE ENVIRONMENTAL PROTECTION AGENCY
Although PART applies to all federal research programs, the present report is concerned specifically with EPA’s experience under PART. That agency has experienced difficulties in achieving PART compliance for its R&D activi-
8
Outcomes may be defined as the results of research that have been integrated, assessed, and given regulatory or otherwise practical shape through a variety of actions. For example, an outcome of research on particulate matter may be improved air quality.
OCR for page 17
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
ties. This section briefly reviews the purpose and organization of those activities.
EPA is primarily a regulatory agency charged with developing regulations that broadly affect human health and the environment, but its regulatory actions are intended to be based on the best possible scientific knowledge as developed both within and outside the agency. Like any other agency, EPA cannot generate all the research it needs, but several previous National Research Council reports have underscored the importance of maintaining an active and credible program of internal research (NRC 2000b, 2003). A 1992 EPA report also states that
science is one of the soundest investments the nation can make for the future. Strong science provides the foundation for credible environmental decision making. With a better understanding of environmental risks to people and ecosystems, EPA can target the hazards that pose the greatest risks, anticipate environmental problems before they reach a critical level, and develop strategies that use the nation’s, and the world’s, environmental protection dollars wisely (EPA 1992).
Research at federal agencies, like other activities, is organized to support an agency mission. EPA’s process is described in its strategic plan. EPA drew up its first strategic plan in 1996 in response to GPRA. That plan, which has been renewed every 3 years, stated that “the mission of the U.S. Environmental Protection Agency is to protect human health and to safeguard the natural environment—air, water, and land—upon which life depends” (EPA 1997a). The current strategic plan (2006-2011) [EPA 2006] has five principal goals, all of which have scientific underpinnings:
Clean air and addressing global climate change.
Clean and safe water.
Land preservation and restoration.
Healthy communities and ecosystems.
Compliance and environmental stewardship.
The plan also lists three “cross-goal strategies” that describe values meant to guide planning for all five goals: results and accountability, innovation and collaboration, and best available science (see Appendix D).9
Research-related activities at EPA, both internal and external, are the responsibility of the Office of Research and Development (ORD) as well as EPA’s program offices and regional laboratories. The committee chose to focus its review on ORD’s research program as this is where the controversy regarding the
9
See Appendix D for more information about EPA’s strategic planning and multi-year planning process.
OCR for page 18
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
development of its efficiency measures arose.10 The ORD conducts research in its in-house laboratories, develops risk-assessment methods and regulatory criteria, and provides technical services in support of the agency’s mission and its program and regional offices. It is organized in three national laboratories, four national centers, and two offices in 14 facilities around the country and its headquarters in Washington, DC. ORD also has an extramural budget for grants, cooperative and interagency agreements, contracts, and fellowships that has accounted for 40-50% of its total budget in recent years.11 Its first strategic plan set forth a straightforward vision (EPA 1996, 1997b): “ORD will provide the scientific foundation to support EPA’s mission.”
In 1999, ORD began to organize its scientific activities in multi-year plans to improve continuity and strategic integration. The multi-year plans, typically covering 5 years, are developed by research teams in ORD laboratories and centers and are peer-reviewed. They cover 16 subjects (such as drinking water, safe food, and ecologic research) and are updated annually (EPA 2008).
USES OF RESULTS OF THE ENVIRONMENTAL PROTECTION AGENCY RESEARCH
The results of EPA research are used both by the agency itself and by various others outside the agency. The explicit purpose of both ORD research and extramural research is to provide scientific bases of EPA actions. The research may lead to end outcomes when results are integrated, assessed, and given regulatory or otherwise practical shape through actions in or outside EPA.
Another important goal of EPA’s work, however, is to provide knowledge outputs for diverse organizations that have environmental interests and responsibilities, such as state and local governments, nongovernment organizations, international organizations, and community groups. Such entities interpret and use ORD outputs for their own planning and regulatory purposes.
SUMMARY
In the last decade and a half, two important efforts to evaluate the work of government agencies have been developed: GPRA, passed into law by Congress in 1993, and PART, designed in 2002 by OMB and first applied in 2003. Both efforts apply to all federal programs, including R&D programs. Evaluating the efficiency of R&D, required under PART, has proved to be challenging for all
10
For this reason, the committee did not address economic analysis, as this work is not conducted by ORD. A report summarizing the history of economics research at EPA through the 1990’s can be found at: http://yosemite.epa.gov/ee/epa/eed.nsf/webpages/EconomicsResearchAtEPA.html.
11
For the FY 2007 enacted budget, 43% ($239,168,600 of $555,383,000) was budgeted to be spent extramurally. [EPA, ORD, personal communication, 2007]
OCR for page 19
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
research-intensive agencies.12 One of those is EPA, which has sought the assistance of the National Academies in its effort to comply with the efficiency questions of PART. The next chapter examines in some detail the complex process of PART compliance and how various agencies have responded.
REFERENCES
EPA (U.S. Environmental Protection Agency). 1992. Safeguarding the Future: Credible Science, Credible Decisions. The Report of the Expert Panel on the Role of Science at EPA. EPA/600/9-91/050. U.S. Environmental Protection Agency, Washington, DC.
EPA (U.S. Environmental Protection Agency). 1996. Strategic Plan for the Office of Research and Development. EPA/600/R-96/059. Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC.
EPA (U.S. Environmental Protection Agency). 1997a. EPA Strategic Plan. EPA/190-R-97-002. Office of the Chief Financial Officer, U.S. Environmental Protection Agency, Washington, DC.
EPA (U.S. Environmental Protection Agency). 1997b. Update to ORD’s Strategic Plan. EPA/600/R-97/015. Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC.
EPA (U.S. Environmental Protection Agency). 2006. 2006-2011 EPA Strategic Plan: Charting Our Course. Office of the Chief Financial Officer, U.S. Environmental Protection Agency. September 2006 [online]. Available: http://www.epa.gov/cfo/plan/2006/entire_report.pdf [accessed Feb. 7, 2008].
EPA (U.S. Environmental Protection Agency). 2008. Research Directions: Multi-Year Plans. Office of Research and Development, U.S. Environmental Protection Agency [online]. Available: http://www.epa.gov/ord/htm/aboutord.htm [accessed Feb. 7, 2008].
NRC (National Research Council). 1997. Building a Foundation for Sound Environmental Decisions. Washington, DC: National Academy Press.
NRC (National Research Council). 1999. Evaluating Federal Research Programs: Research and the Government Performance and Results Act. Washington, DC: The National Academies Press.
NRC (National Research Council). 2000a. Experiments in International Benchmarking of U.S. Research Fields. Washington, DC: National Academy Press.
NRC (National Research Council). 2000b. Strengthening Science at the U.S. Environmental Protection Agency. Washington, DC: National Academy Press.
NRC (National Research Council). 2001. Pp. 2-3 in Implementing the Government Performance and Results Act for Research Programs: A Status Report. Washington, DC: The National Academies Press.
NRC (National Research Council). 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency’s Science to Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press.
12
In this report, research-intensive is used to describe agencies of whose mission’s research is an essential, though not necessarily dominant, aspect. For example, research is important at EPA but not the primary function of the agency, as it is for the National Institutes of Health or the National Science Foundation.
OCR for page 20
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
OMB (Office of Management and Budget). 2004. Pp. 49-52 in Rating the Performance of Federal Programs. The Budget for Fiscal Year 2004. Office of Management and Budget [online]. Available: http://www.gpoaccess.gov/usbudget/fy04/pdf/budget/performance.pdf [accessed Nov. 7, 2007].
OMB (Office of Management and Budget). 2007a. ExpectMore.gov. Office of Management and Budget [online]. Available: http://www.whitehouse.gov/omb/expectmore/ [accessed Nov. 7, 2007].
OMB (Office of Management and Budget). 2007b. Guide to the Program Assessment Rating Tool (PART). Office of Management and Budget. January 2007 [online]. Available: http://stinet.dtic.mil/cgi-bin/GetTRDoc?AD=ADA471562&Location=U2&doc=GetTRDoc.pdf [accessed Nov. 7, 2007].
Posner, P.L. 2004. Performance Budgeting: OMB’s Performance Rating Tool Presents Opportunities and Challenges for Evaluating Program Performance: Testimony before the Subcommittee on Environment, Technology, and Standards, Committee on Science, House of Representatives, March 11, 2004. GAO-04-550T. Washington, DC: U.S. General Accounting Office [online]. Available: http://www.gao.gov/new.items/d04550t.pdf [accessed Nov. 7, 2007].
Stokes, D.E. 1997. Pp. 6-8 in Pasteur’s Quadrant: Basic Science and Technological Innovation. Washington, DC: Brookings Institution.