Evaluating Research Efficiency in the U.S. Environmental Protection Agency (2008) / Chapter Skim
Currently Skimming:
2 Efficiency Metrics Used by the Environmental Protection Agency and Other Federal Research and Development Programs
2 Efficiency Metrics Used by the Environmental Protection Agency and Other Federal Research and Development Programs
Pages 21-37
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 21... ...
The committee reviewed documents, interviewed agency personnel, and heard presentations during a workshop in April 2007 that was attended by most of the research-intensive agencies and several large corporations that emphasize research.1 This chapter summarizes some of the challenges of evaluating research efficiency and ways in which agencies have approached those challenges. 1 The workshop was held on April 24, 2007, at the National Academies, 2101 Constitution Avenue, Washington, DC 20418.
|
From page 22... ...
. THE PROGRAM ASSESSMENT RATING TOOL AND EFFICIENCY Efficiency is a common enough concept, as illustrated by familiar dictionary definitions: "effective operation as measured by a comparison of production with cost (as in energy, time, and money)
|
From page 23... ...
THE PROGRAM ASSESSMENT RATING TOOL GRADING SYSTEM The overall structure of PART was introduced in Chapter 1; additional aspects are described briefly here. Although this report focuses on the criterion of efficiency, the PART questions, taken in their entirety, attempt to address a broad range of issues with the goal of a thorough evaluation of federal programs.
|
From page 24... ...
That and other difficulties are addressed further in Chapter 3. THE USE OF "EXPERT REVIEW" AT THE ENVIRONMENTAL PROTECTION AGENCY EPA uses multiple mechanisms to evaluate its R&D activities, including internal processes of strategic plans, multi-year plans, and annual performance goals (mentioned in Chapter 1)
|
From page 25... ...
. Expert review was also recommended by the Committee on Science, Engineering, and Public Policy panel cited in Chapter 1 for evaluating research to comply with the Government Performance and Results Act.
|
From page 26... ...
"5 In spring 2006, a BOSC panel added the charge questions discussed above to its evaluation criteria; it used them for the first time early in 2007. Although there is much overlap between the BOSC investigations and the PART questions regarding results of a program, OMB had not by the time of this committee's investigation determined whether the use of BOSC's revised charge met the requirements of PART.6 During the 2005 PART review for EPA's drinking-water program, EPA experimented with a quantitative version of the BOSC evaluation.
|
From page 27... ...
According to EPA staff, there is increasing pressure from OMB to focus on outcome-based efficiency metrics, but the agency has been unable to establish such metrics for research. EMERGING ISSUES One important role of expert review is to complement the ability of program managers and agency leaders to anticipate important emerging issues.
|
From page 28... ...
It is used by agencies and other organizations in many management settings, such as construction projects and facilities operations, where the outcome (such as a new 7 The members of the Human Health Subcommittee, according to the BOSC Web site, had "considerable expertise in the area of human health research, including formal education, training, and research experience in biology, chemistry, biochemistry, environmental carcinogenesis, pharmacology, molecular biology and molecular mechanisms of carcinogenicity and toxicity, toxicology, physiologically based pharmacokinetic (PBPK) modeling, exposure modeling, risk assessment, epidemiology, biomarkers and biological monitoring, and public health, with additional expertise in the areas of children's health, community-based human exposure studies, and clinical experience" (EPA 2005, p.
|
From page 29... ...
Other programs that have not passed the efficiency questions include the National Ambient Air Quality Standards Program and the Ground Water and Drinking Water Program, for similar reasons. In examining OMB documents and the ExpectMore.gov Web site, the committee found that for most research-intensive agencies other than EPA, OMB accepted efficiency measures similar to those proposed by EPA, such as scheduled regulatory decision-making activities.
|
From page 30... ...
. 11 Most recently, the ERP received a positive rating, again as reported by InsideEPA.com: "The rating was part of OMB's Program Assessment Rating Tool (PART)
|
From page 31... ...
efficiency. The committee compiled and reviewed an "efficiency measures table" of efficiency metrics used for research programs by 11 federal agencies, including EPA, in following the PART guidance13 (see Appendix E for Table E-1)
|
From page 32... ...
Chapter 3 asks whether those metrics are "sufficient" for evaluating research efficiency. Department of Energy The Department of Energy (DOE)
|
From page 33... ...
The representative notes that because the OMB policy-makers generally did not have research backgrounds, NASA spends considerable time in educating them about the relevant differences in R&D programs, as these differences are not well considered in PART guidance. The representative suggests more flexibility in the actions of the reviewers -- for example, in recognizing that short-term decreases in efficiency might lead to long-term efficiency gains and in seeing the need to balance efficiency with effectiveness.14 National Institutes of Health National Institutes of Health (NIH)
|
From page 34... ...
NIOSH uses independent expert review to evaluate its 30 research programs, which exist in a "matrix" with substantial overlaps. A model for evaluating a research program was provided by a framework developed for NIOSH by the National Academies with a blend of quantitative and qualitative elements.
|
From page 35... ...
on research activities, but none used EVM, none evaluated research in terms of efficiency, and none described direct links to outcomes (sales) .15 Chapter 3 offers additional perspective on evaluating research efficiency in industry.
|
From page 36... ...
2006a. Guide to the Program Assessment Rating Tool (PART)
|
From page 37... ...
2007a. Guide to the Program Assessment Rating Tool (PART)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.