Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page R1
Evaluating Research Efficiency in the U.S. Environmental Protection Agency Evaluating Research Efficiency in the U.S. Environmental Protection Agency Committee on Evaluating the Efficiency of Research and Development Programs at the U.S. Environmental Protection Agency Committee on Science, Engineering, and Public Policy Policy and Global Affairs Board on Environmental Studies and Toxicology Division on Earth and Life Studies NATIONAL RESEARCH COUNCIL OF THE NATIONAL ACADEMIES THE NATIONAL ACADEMIES PRESS Washington, D.C. www.nap.edu
OCR for page R2
Evaluating Research Efficiency in the U.S. Environmental Protection Agency THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This project was supported by Contract 68-C-03-081 between the National Academy of Sciences and the U.S. Environmental Protection Agency. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the authors and do not necessarily reflect the view of the organizations or agencies that provided support for this project. International Standard Book Number-13 978-0-309-11684-8 International Standard Book Number-10 0-309-11684-8 Additional copies of this report are available from The National Academies Press 500 Fifth Street, NW Box 285 Washington, DC 20055 800-624-6242 202-334-3313 (in the Washington metropolitan area) http://www.nap.edu Copyright 2008 by the National Academy of Sciences. All rights reserved. Printed in the United States of America.
OCR for page R3
Evaluating Research Efficiency in the U.S. Environmental Protection Agency THE NATIONAL ACADEMIES Advisers to the Nation on Science, Engineering, and Medicine The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council. www.national-academies.org
OCR for page R4
Evaluating Research Efficiency in the U.S. Environmental Protection Agency This page intentionally left blank.
OCR for page R5
Evaluating Research Efficiency in the U.S. Environmental Protection Agency COMMITTEE ON EVALUATING THE EFFICIENCY OF RESEARCH AND DEVELOPMENT PROGRAMS AT THE U.S. ENVIRONMENTAL PROTECTION AGENCY Members GILBERT S. OMENN (Chair), University of Michigan, Ann Arbor GEORGE V. ALEXEEFF, California Environmental Protection Agency, Oakland RADFORD BYERLY, JR., University of Colorado, Boulder EDWIN H. CLARK II, Earth Policy Institute, Washington, DC SUSAN E. COZZENS, Georgia Institute of Technology, Atlanta LINDA J. FISHER, E. I. du Pont de Nemours and Company, Wilmington, DE J. PAUL GILMAN, Oak Ridge Center for Advanced Studies, Oak Ridge, TN T.J. GLAUTHIER, TJG Energy Associates, LLC, Moss Beach, CA CAROL J. HENRY, Independent Consultant, Bethesda, MD ROBERT J. HUGGETT, College of William and Mary, Seaford, VA SALLY KATZEN, George Mason University School of Law, Fairfax, VA TERRY F. YOUNG, Environmental Defense, Oakland, CA Staff RICHARD BISSELL, Executive Director, Committee on Science, Engineering, and Public Policy DEBORAH STINE, Associate Director, Committee on Science, Engineering, and Public Policy (up to August 2007) EILEEN ABT, Senior Program Officer ALAN ANDERSON, Consultant Writer NORMAN GROSSBLATT, Senior Editor JENNIFER SAUNDERS, Associate Program Officer RAE BENEDICT, Mirzayan Science & Technology Policy Fellow MIRSADA KARALIC-LONCAREVIC, Manager, Toxicology Information Center NEERAJ P. GORKHALY, Senior Program Assistant MORGAN R. MOTTO, Senior Program Assistant Sponsor U.S. ENVIRONMENTAL PROTECTION AGENCY
OCR for page R6
Evaluating Research Efficiency in the U.S. Environmental Protection Agency COMMITTEE ON SCIENCE, ENGINEERING, AND PUBLIC POLICY Members GEORGE WHITESIDES (Chair), Woodford L. and Ann A. Flowers University Professor, Harvard University, Boston, MA CLAUDE R. CANIZARES, Vice President for Research, Massachusetts Institute of Technology, Cambridge RALPH J. CICERONE (Ex officio), President, National Academy of Sciences, Washington, DC EDWARD F. CRAWLEY, Executive Director, CMI, and Professor, Massachusetts Institute of Technology, Cambridge RUTH A. DAVID, President and Chief Executive Officer, Analytic Services, Inc., Arlington, VA HAILE T. DEBAS, Executive Director, UCSF Global Health Sciences, Maurice Galante Distinguished Professor of Surgery, San Francisco, CA HARVEY FINEBERG (Ex officio), President, Institute of Medicine, Washington, DC JACQUES S. GANSLER, Vice President for Research, University of Maryland, College Park ELSA M. GARMIRE, Professor, Dartmouth College, Hanover, NH M. R. C. GREENWOOD (Ex officio), Professor of Nutrition and Internal Medicine, University of California, Davis W. CARL LINEBERGER, Professor of Chemistry, University of Colorado, Boulder C. DAN MOTE, JR. (Ex officio), President and Glenn Martin Institute Professor of Engineering, University of Maryland, College Park ROBERT M. NEREM, Parker H. Petit Professor and Director, Institute for Bioengineering and Bioscience, Georgia Institute of Technology, Atlanta LAWRENCE T. PAPAY, Retired, Sector Vice President for Integrated Solutions, Science Applications International Corporation, La Jolla, CA ANNE C. PETERSEN, Professor of Psychology, Stanford University, Stanford, CA SUSAN C. SCRIMSHAW, President, Simmons College, Boston, MA WILLIAM J. SPENCER, Chairman Emeritus, SEMATECH, Austin, TX LYDIA THOMAS (Ex officio), Retired, Mitretek Systems, Inc., Falls Church, VA CHARLES M. VEST (Ex officio), President, National Academy of Engineering, Washington, DC NANCY S. WEXLER, Higgins Professor of Neuropsychology, Columbia University, New York, NY MARY LOU ZOBACK, Vice President Earthquake Risk Applications, Risk Management Solutions, Inc., Newark, CA
OCR for page R7
Evaluating Research Efficiency in the U.S. Environmental Protection Agency Staff RICHARD BISSELL, Executive Director DEBORAH STINE, Associate Director (up to August 2007) MARION RAMSEY, Administrative Coordinator NEERAJ P. GORKHALY, Senior Program Assistant
OCR for page R8
Evaluating Research Efficiency in the U.S. Environmental Protection Agency BOARD ON ENVIRONMENTAL STUDIES AND TOXICOLOGY Members JONATHAN M. SAMET (Chair), Johns Hopkins University, Baltimore, MD RAMÓN ALVAREZ, Environmental Defense Fund, Austin, TX JOHN M. BALBUS, Environmental Defense Fund, Washington, DC DALLAS BURTRAW, Resources for the Future, Washington, DC JAMES S. BUS, Dow Chemical Company, Midland, MI RUTH DEFRIES, University of Maryland, College Park COSTEL D. DENSON, University of Delaware, Newark E. DONALD ELLIOTT, Willkie Farr & Gallagher LLP, Washington, DC MARY R. ENGLISH, University of Tennessee, Knoxville J. PAUL GILMAN, Oak Ridge Center for Advanced Studies, Oak Ridge, TN SHERRI W. GOODMAN, Center for Naval Analyses, Alexandria, VA JUDITH A. GRAHAM (Retired), Pittsboro, NC WILLIAM P. HORN, Birch, Horton, Bittner and Cherot, Washington, DC WILLIAM M. LEWIS, JR., University of Colorado, Boulder JUDITH L. MEYER, University of Georgia, Athens DENNIS D. MURPHY, University of Nevada, Reno PATRICK Y. O’BRIEN, ChevronTexaco Energy Technology Company, Richmond, CA DOROTHY E. PATTON, (Retired) U.S. Environmental Protection Agency, Chicago, IL DANNY D. REIBLE, University of Texas, Austin JOSEPH V. RODRICKS, ENVIRON International Corporation, Arlington, VA ARMISTEAD G. RUSSELL, Georgia Institute of Technology, Atlanta ROBERT F. SAWYER, University of California, Berkeley KIMBERLY M. THOMPSON, Massachusetts Institute of Technology, Cambridge MONICA G. TURNER, University of Wisconsin, Madison MARK J. UTELL, University of Rochester Medical Center, Rochester, NY CHRIS G. WHIPPLE, ENVIRON International Corporation, Emeryville, CA LAUREN ZEISE, California Environmental Protection Agency, Oakland Senior Staff JAMES J. REISA, Director DAVID J. POLICANSKY, Scholar RAYMOND A. WASSEL, Senior Program Officer for Environmental Studies EILEEN N. ABT, Senior Program Officer for Risk Analysis SUSAN N.J. MARTEL, Senior Program Officer for Toxicology KULBIR BAKSHI, Senior Program Officer ELLEN K. MANTUS, Senior Program Officer RUTH E. CROSSGROVE, Senior Editor
OCR for page R9
Evaluating Research Efficiency in the U.S. Environmental Protection Agency OTHER REPORTS OF THE COMMITTEE ON SCIENCE, ENGINEERING, AND PUBLIC POLICY Advanced Research Instrumentation and Facilities (2006) Beyond Bias and Barriers: Fulfilling the Potential of Women in Academic Science and Engineering (2006) Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future (2005) Policy Implications of International Graduate Students and Postdoctoral Scholars in the United States (2005) Setting Priorities for NSF-sponsored Large Research Facility Projects (2004) Facilitating Interdisciplinary Research (2004) Science and Technology in the National Interest: Ensuring the Best Presidential and Federal Advisory Committee and Technology Appointments (2004) Electronic Scientific, Technical and Medical Journal Publishing and ITS Implications (2004) Observations on the President’s Fiscal Year 2003 Federal Science and Technology Budget (2003) Implementing the Government Performance and Results Act for Research: A Status Report (2003) Scientific and Medical Aspects of Human Reproductive Cloning (2002) Observations on the President’s Fiscal Year 2002 Federal Science and Technology Budget (2002) Experiments in International Benchmarking of U.S. Research Fields (2000) Observations on the President’s Fiscal Year 2001 Federal Science and Technology Budget (2000) Enhancing the Postdoctoral Experience for Scientists and Engineers: A Guide for Postdoctoral Scholars, Advisors, Institutions, Funding Organizations, and Disciplinary Societies (2000) Science and Technology in the National Interest: The Presidential Appointment Process (2000) Evaluating Federal Research Programs: Research and the Government Performance and Results Act (1999) Capitalizing on Investments in Science and Technology (1999) Observations on the President’s Fiscal Year 2000 Federal Science and Technology Budget (1999) Observations on the President’s Fiscal Year 1999 Federal Science and Technology Budget (1998) Adviser, Teacher, Role Model, Friend: On Being a Mentor to Students in Science and Engineering (1997) Proceedings of the National Convocation on Science and Engineering Doctoral Education (1996) Careers in Science and Engineering: A Student Planning Guide to Grad School and Beyond (1996)
OCR for page R10
Evaluating Research Efficiency in the U.S. Environmental Protection Agency An Assessment of the National Science Foundation’s Science and Technology Centers Program (1996) Allocating Federal Funds for Science and Technology (1995) Reshaping the Graduate Education of Scientists and Engineers (1995) On Being a Scientist: Responsible Conduct in Research (1995) Major Award Decisionmaking at the National Science Foundation (1994) Science, Technology, and the Federal Government: National Goals for a New Era (1993) Responsible Science Volume 2: Background Papers and Resource Documents (1993) Responsible Science Volume 1: Ensuring the Integrity of the Research Process (1992) Policy Implications of Greenhouse Warming: Mitigation, Adaptation, and the Science Base (1991) Copies of these reports may be ordered from the National Academies Press (800) 624-6242 or (202) 334-3313 www.nap.edu
OCR for page R11
Evaluating Research Efficiency in the U.S. Environmental Protection Agency OTHER REPORTS OF THE BOARD ON ENVIRONMENTAL STUDIES AND TOXICOLOGY Respiratory Diseases Research at NIOSH (2008) Hydrology, Ecology, and Fishes of the Klamath River Basin (2008) Estimating Mortality Risk Reduction and Economic Benefits from Controlling Ozone Air Pollution (2008) Applications of Toxicogenomic Technologies to Predictive Toxicology and Risk Assessment (2007) Models in Environmental Regulatory Decision Making (2007) Toxicity Testing in the Twenty-first Century: A Vision and a Strategy (2007) Sediment Dredging at Superfund Megasites: Assessing the Effectiveness (2007) Environmental Impacts of Wind-Energy Projects (2007) Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget (2007) Assessing the Human Health Risks of Trichloroethylene: Key Scientific Issues (2006) New Source Review for Stationary Sources of Air Pollution (2006) Human Biomonitoring for Environmental Chemicals (2006) Health Risks from Dioxin and Related Compounds: Evaluation of the EPA Reassessment (2006) Fluoride in Drinking Water: A Scientific Review of EPA’s Standards (2006) State and Federal Standards for Mobile-Source Emissions (2006) Superfund and Mining Megasites—Lessons from the Coeur d’Alene River Basin (2005) Health Implications of Perchlorate Ingestion (2005) Air Quality Management in the United States (2004) Endangered and Threatened Species of the Platte River (2004) Atlantic Salmon in Maine (2004) Endangered and Threatened Fishes in the Klamath River Basin (2004) Cumulative Environmental Effects of Alaska North Slope Oil and Gas Development (2003) Estimating the Public Health Benefits of Proposed Air Pollution Regulations (2002) Biosolids Applied to Land: Advancing Standards and Practices (2002) The Airliner Cabin Environment and Health of Passengers and Crew (2002) Arsenic in Drinking Water: 2001 Update (2001) Evaluating Vehicle Emissions Inspection and Maintenance Programs (2001) Compensating for Wetland Losses Under the Clean Water Act (2001) A Risk-Management Strategy for PCB-Contaminated Sediments (2001) Acute Exposure Guideline Levels for Selected Airborne Chemicals (six volumes, 2000-2008) Toxicological Effects of Methylmercury (2000) Strengthening Science at the U.S. Environmental Protection Agency (2000) Scientific Frontiers in Developmental Toxicology and Risk Assessment (2000)
OCR for page R12
Evaluating Research Efficiency in the U.S. Environmental Protection Agency Ecological Indicators for the Nation (2000) Waste Incineration and Public Health (2000) Hormonally Active Agents in the Environment (1999) Research Priorities for Airborne Particulate Matter (four volumes, 1998-2004) The National Research Council’s Committee on Toxicology: The First 50 Years (1997) Carcinogens and Anticarcinogens in the Human Diet (1996) Upstream: Salmon and Society in the Pacific Northwest (1996) Science and the Endangered Species Act (1995) Wetlands: Characteristics and Boundaries (1995) Biologic Markers (five volumes, 1989-1995) Science and Judgment in Risk Assessment (1994) Pesticides in the Diets of Infants and Children (1993) Dolphins and the Tuna Industry (1992) Science and the National Parks (1992) Human Exposure Assessment for Airborne Pollutants (1991) Rethinking the Ozone Problem in Urban and Regional Air Pollution (1991) Decline of the Sea Turtles (1990) Copies of these reports may be ordered from the National Academies Press (800) 624-6242 or (202) 334-3313 www.nap.edu
OCR for page R13
Evaluating Research Efficiency in the U.S. Environmental Protection Agency Preface In an effort to ensure the wise use of taxpayers’ money, the federal government has undertaken major initiatives to evaluate the performance and results of federally funded programs, including research and development (R&D) programs. The initiatives include the Government Performance and Results Act of 1993 and the Office of Management and Budget (OMB) Program Assessment Rating Tool (PART), developed in 2002. The latter was designed in the context of “performance budgeting” and “performance measurement” and focused on evaluating the efficiency of programs. Evaluation of R&D programs has proved challenging for federal agencies, including the Environmental Protection Agency (EPA), a mission-oriented agency with a substantial research component. All agencies have experienced difficulties in complying with the PART requirements to measure the efficiency of their research, to use outcome-based metrics in doing so, and to achieve and quantitate annual efficiency improvements. In 2006, EPA asked the National Academies for assistance in developing better assessment tools to comply with PART, with emphasis on efficiency. The Academies’ Committee on Science, Engineering, and Public Policy (COSEPUP) and the National Research Council Board on Environmental Studies and Toxicology (BEST) convened the Committee on Evaluating the Efficiency of Research and Development Programs at the U.S. Environmental Protection Agency. In this report, the committee addresses the efficiency measures now used for federally funded R&D programs and evaluates whether they are sufficient and are based on outcomes, not only inputs and outputs. The committee provides principles that it hopes will guide the development of efficiency measures for federally funded R&D programs and makes recommendations about efficiency measures for EPA’s basic and applied R&D programs and about OMB’s review process. The committee gratefully acknowledges the following for making presentations to the committee: Diana Espinosa, Daren Wong, Brian Kleinman, and Kevin Neyland, of OMB; Marcus Peacock, Hugh Tilson, Phillip Juengst, Lori Kowalski, of EPA; Bill Valdez, Darrell Beschen, and Brian Card, of the Department of Energy; Pat Tsuchitani, of the National Science Foundation; Deb-
OCR for page R14
Evaluating Research Efficiency in the U.S. Environmental Protection Agency orah Duran, of the National Institutes of Health; Julie Pollitt, of the National Aeronautics and Space Administration; Raymond Sinclair, of the National Institute for Occupational Safety and Health; George Daston, of the Procter & Gamble Company; Joseph Kenney and Bernice Rogowitz, of IBM Global Business Services; James Bus, of Dow Chemical Company; and Patrick Atkins, retired from Alcoa. The committee is grateful for the assistance of the National Research Council staff in preparing this report. Staff members who contributed to the effort are Richard Bissell, project director from August 2007 and director of COSEPUP; Deborah Stine, project director (up to August 2007); Eileen Abt, senior program officer; James Reisa, director of BEST; Alan Anderson, consultant science writer; Jennifer Saunders, associate program officer; Rae Benedict, Mirzayan Science & Technology Policy Fellow; Norman Grossblatt, senior editor; Mirsada Karalic-Loncarevic, manager of the Toxicology Information Center; and Neeraj Gorkhaly and Morgan Motto, senior project assistants. I especially thank my colleagues on the committee for their engagements in the contentious issues underlying our charge and for what we believe are constructive principles and feasible recommendations that have emerged from our deliberations and the iterative development of this report. Gilbert S. Omenn, Chair Committee on Evaluating the Efficiency of Research and Development Programs at the U.S. Environmental Protection Agency
OCR for page R15
Evaluating Research Efficiency in the U.S. Environmental Protection Agency Acknowledgment of Review Participants This report has been reviewed in draft form by persons chosen for their diverse perspectives and technical expertise in accordance with procedures approved by the National Research Council’s Report Review Committee. The purpose of the independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards of objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We thank the following for their review of the report: Irwin Feller, Pennsylvania State University; Haren S. Gandhi, Ford Motor Company; Bernard D. Goldstein, University of Pittsburgh; Victoria F. Haynes, Research Triangle Institute; Martha A. Krebs, California Energy Commission; James H. Johnson, Jr., Howard University; Genevieve M. Matanoski, Johns Hopkins University; Shelley H. Metzenbaum, University of Maryland School of Public Policy; and David Trinkle, University of California, Berkeley. Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of the report was overseen by John F. Ahearne, Sigma Xi, The Scientific Research Society, and Alan Schriesheim, Argonne National Laboratory. Appointed by the National Research Council, they were responsible for making certain that an independent examination of the report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the author committee and the institution.
OCR for page R16
Evaluating Research Efficiency in the U.S. Environmental Protection Agency This page intentionally left blank.
OCR for page R17
Evaluating Research Efficiency in the U.S. Environmental Protection Agency Contents SUMMARY 3 1 INTRODUCTION: THE GOVERNMENT PERFORMANCE AND RESULTS ACT, THE PROGRAM ASSESSMENT RATING TOOL, AND THE ENVIRONMENTAL PROTECTION AGENCY 11 Inherent Difficulties in Evaluating Research, 11 Research Terms at the Environmental Protection Agency, 12 Evaluating Research under the Government Performance and Results Act, 13 The Rationale and Function of the Program Assessment Rating Tool, 14 The Application of the Program Assessment Rating Tool to Research, 14 The Organization and Performance of Research and Development at the Environmental Protection Agency, 16 Uses of Results of Environmental Protection Agency Research, 18 Summary, 18 References, 19 2 EFFICIENCY METRICS USED BY THE ENVIRONMENTAL PROTECTION AGENCY AND OTHER FEDERAL RESEARCH AND DEVELOPMENT PROGRAMS 21 Evaluating Research and Development, 22 The Program Assessment Rating Tool and Efficiency, 22 Applying Efficiency to Inputs, Outputs, and Outcomes, 22 The Program Assessment Rating Tool Grading System, 23 The Use of “Expert Review” at the Environmental Protection Agency, 24 Emerging Issues, 27 Metrics Proposed by the Environmental Protection Agency, 27 Metrics That Did Not Pass the Program Assessment Rating Tool Process, 29 The Consequences of a “No” Answer to a Program Assessment Rating Tool Question, 30 Evaluation Mechanisms Used by Other Agencies, 31 Methods Used by Industry, 35 Summary, 35 References, 36
OCR for page R18
Evaluating Research Efficiency in the U.S. Environmental Protection Agency 3 ARE THE EFFICIENCY METRICS USED BY FEDERAL RESEARCH AND DEVELOPMENT PROGRAMS SUFFICIENT AND OUTCOME-BASED? 38 Attempting to Evaluate Efficiency in Terms of Ultimate Outcomes, 38 Placing “Research Efficiency” in Perspective, 40 Process Efficiency and Investment Efficiency, 41 What Are “Sufficient” Metrics of Process Efficiency? 42 A Critique of the Efficiency Metrics Used by Federal Research Programs, 44 Factors that Reduce the Efficiency of Research, 47 Evaluating Research Efficiency in Industry, 48 The Shortcomings of Retrospective Review, 49 Summary and Recommendations, 49 References, 50 4 A MODEL FOR EVALUATING RESEARCH AND DEVELOPMENT PROGRAMS 52 Summary, 56 References, 56 5 FINDINGS, PRINCIPLES, AND RECOMMENDATIONS 58 Findings, 59 Principles, 60 Recommendations, 65 Additional Recommendation for the Office of Management and Budget, 67 References, 68 APPENDIX A: BIOGRAPHIC INFORMATION ON THE COMMITTEE ON EVALUATING THE EFFICIENCY OF RESEARCH AND DEVELOPMENT PROGRAMS AT THE U.S. ENVIRONMENTAL PROTECTION AGENCY 69 APPENDIX B: EVALUATING THE EFFICIENCY OF RESEARCH AND DEVELOPMENT PROGRAMS AT THE U.S. ENVIRONMENTAL PROTECTION AGENCY: WORKSHOP SUMMARY 75 APPENDIX C: PROGRAM ASSESSMENT RATING TOOL (PART) QUESTIONS 86 APPENDIX D: THE ENVIRONEMENTAL PROTECTION AGENCY’S STRATEGIC AND MULTI-YEAR PLANNING PROCESS 90
OCR for page R19
Evaluating Research Efficiency in the U.S. Environmental Protection Agency APPENDIX E: AGENCY AND INDUSTRY EFFICIENCY MEASURES 95 APPENDIX F: DRAFT BOARD OF SCIENTIFIC COUNSELORS HANDBOOK FOR SUBCOMMITTEE CHAIRS: DRAFT PROPOSED CHARGE QUESTIONS FOR BOSC REVIEWS 109 APPENDIX G: OMB’S RESEARCH AND DEVELOPMENT PROGRAM INVESTMENT CRITERIA 113 APPENDIX H: CHARGE TO THE BOSC SUBCOMMITTEE ON SAFE PESTICIDES/SAFE PRODUCTS RESEARCH 119 APPENDIX I: PART GUIDANCE ON EFFICIENCY MEASURES 128 BOXES, FIGURES, AND TABLES BOXES 1-1 Distribution of PART Scores, 15 1-2 How PART Questions Are Scored, 16 2-1 BOSC: An Example of Expert Review, 28 I-1 Measures and PARTWeb, 129 I-2 Question Linkages, 131 FIGURES 4-1 EPA research presented as a logical model, 54 TABLES E-1 Agency and Industry Efficiency Measures, 96
OCR for page R20
Evaluating Research Efficiency in the U.S. Environmental Protection Agency This page intentionally left blank.