The Measure of STAR

Review of the U.S. Environmental Protection Agency’s Science To Achieve Results (STAR) Research Grants Program

Committee to Review EPA’s Research Grants Program

Board on Environmental Studies and Toxicology

Division on Earth and Life Studies

NATIONAL RESEARCH COUNCIL
OF THE NATIONAL ACADEMIES

THE NATIONAL ACADEMIES PRESS
Washington, D.C.
www.nap.edu



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
The Measure of STAR Review of the U.S. Environmental Protection Agency’s Science To Achieve Results (STAR) Research Grants Program Committee to Review EPA’s Research Grants Program Board on Environmental Studies and Toxicology Division on Earth and Life Studies NATIONAL RESEARCH COUNCIL OF THE NATIONAL ACADEMIES THE NATIONAL ACADEMIES PRESS Washington, D.C. www.nap.edu

OCR for page R1
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This project was supported by Contract 68-C-01-119 between the National Academy of Sciences and the U.S. Environmental Protection Agency. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the authors and do not necessarily reflect the view of the organizations or agencies that provided support for this project. International Standard Book Number 0-309-08938-7 (Book) International Standard Book Number 0-309-51674-9 (PDF) Additional copies of this report are available from: The National Academies Press 500 Fifth Street, NW Box 285 Washington, DC 20055 800-624-6242 202-334-3313 (in the Washington metropolitan area) http://www.nap.edu Copyright 2003 by the National Academy of Sciences. All rights reserved. Printed in the United States of America

OCR for page R1
THE NATIONAL ACADEMIES Advisers to the Nation on Science, Engineering, and Medicine The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council www.national-academies.org

OCR for page R1
This page in the original is blank.

OCR for page R1
COMMITTEE TO REVIEW EPA’S RESEARCH GRANTS PROGRAM Members HAROLD A. MOONEY (Chair), Stanford University, Stanford, CA RAYMOND C. LOEHR (Vice Chair), University of Texas, Austin ANDERS W. ANDREN, University of Wisconsin, Madison EDWIN H. CLARK, II, Clean Sites, Inc., Washington, DC COSTEL D. DENSON, University of Delaware, Newark JOHN C. ELSTON, Spring Lake Heights, NJ, retired from New Jersey State Department of Environmental Protection CAROL J. HENRY, American Chemistry Council, Arlington, VA MARTHA A. KREBS, Science Strategies, Los Angeles, CA RICHARD F. LEE, Skidaway Institute of Oceanography, Savannah, GA GERALD VAN BELLE, University of Washington, Seattle TERRY F. YOUNG, Environmental Defense, Oakland, CA LAUREN A. ZEISE, California Environmental Protection Agency, Oakland Staff EILEEN N. ABT, Project Director ROBERTA M. WEDGE, Program Director for Risk Analysis NORMAN GROSSBLATT, Senior Editor KELLY CLARK, Assistant Editor JENNIFER E. SAUNDERS, Research Assistant MIRSADA KARALIC-LONCAREVIC, Research Assistant LUCY V. FUSCO, Senior Project Assistant Sponsor U.S. ENVIRONMENTAL PROTECTION AGENCY

OCR for page R1
BOARD ON ENVIRONMENTAL STUDIES AND TOXICOLOGY1 Members GORDON ORIANS (Chair), University of Washington, Seattle JOHN DOULL (Vice Chair), University of Kansas Medical Center, Kansas City DAVID ALLEN, University of Texas, Austin THOMAS BURKE, Johns Hopkins University, Baltimore, MD JUDITH C. CHOW, Desert Research Institute, Reno, NV CHRISTOPHER B. FIELD, Carnegie Institute of Washington, Stanford, CA WILLIAM H. GLAZE, Oregon Health and Science University, Beaverton SHERRI W. GOODMAN, Center for Naval Analyses, Alexandria, VA DANIEL S. GREENBAUM, Health Effects Institute, Cambridge, MA ROGENE HENDERSON, Lovelace Respiratory Research Institute, Albuquerque, NM CAROL HENRY, American Chemistry Council, Arlington, VA ROBERT HUGGETT, Michigan State University, East Lansing BARRY L. JOHNSON Emory University, Atlanta, GA JAMES H. JOHNSON, Howard University, Washington, DC JAMES A. MACMAHON, Utah State University, Logan PATRICK V. O’BRIEN, Chevron Research and Technology, Richmond, CA DOROTHY E. PATTON, International Life Sciences Institute, Washington, DC ANN POWERS, Pace University School of Law, White Plains, NY LOUISE M. RYAN, Harvard University, Boston, MA JONATHAN M. SAMET, Johns Hopkins University, Baltimore, MD KIRK SMITH, University of California, Berkeley LISA SPEER, Natural Resources Defense Council, New York, NY G. DAVID TILMAN, University of Minnesota, St. Paul CHRIS G. WHIPPLE, Environ Incorporated, Emeryville, CA LAUREN A. ZEISE, California Environmental Protection Agency, Oakland Senior Staff JAMES J. REISA, Director DAVID J. POLICANSKY, Associate Director RAYMOND A. WASSEL, Senior Program Director for Environmental Sciences and Engineering KULBIR BAKSHI, Program Director for the Committee on Toxicology ROBERTA M. WEDGE, Program Director for Risk Analysis K. JOHN HOLMES, Senior Staff Officer SUSAN N.J. MARTEL, Senior Staff Officer SUZANNE VAN DRUNICK, Senior Staff Officer EILEEN N. ABT, Senior Staff Officer ELLEN K. MANTUS, Senior Staff Officer RUTH E. CROSSGROVE, Managing Editor 1   This study was planned, overseen, and supported by the Board on Environmental Studies and Toxicology.

OCR for page R1
OTHER REPORTS OF THE BOARD ON ENVIRONMENTAL STUDIES AND TOXICOLOGY Cumulative Environmental Effects of Alaska North Slope Oil and Gas Development (2003) Estimating the Public Health Benefits of Proposed Air Pollution Regulations (2002) Biosolids Applied to Land: Advancing Standards and Practices (2002) Ecological Dynamics on Yellowstone’s Northern Range (2002) The Airliner Cabin Environment and Health of Passengers and Crew (2002) Arsenic in Drinking Water: 2001 Update (2001) Evaluating Vehicle Emissions Inspection and Maintenance Programs (2001) Compensating for Wetland Losses Under the Clean Water Act (2001) A Risk-Management Strategy for PCB-Contaminated Sediments (2001) Acute Exposure Guideline Levels for Selected Airborne Chemicals (3 volumes, 2000-2003) Toxicological Effects of Methylmercury (2000) Strengthening Science at the U.S. Environmental Protection Agency (2000) Scientific Frontiers in Developmental Toxicology and Risk Assessment (2000) Ecological Indicators for the Nation (2000) Modeling Mobile-Source Emissions (2000) Waste Incineration and Public Health (1999) Hormonally Active Agents in the Environment (1999) Research Priorities for Airborne Particulate Matter (4 volumes, 1998-2003) Ozone-Forming Potential of Reformulated Gasoline (1999) Arsenic in Drinking Water (1999) The National Research Council’s Committee on Toxicology: The First 50 Years (1997) Carcinogens and Anticarcinogens in the Human Diet (1996) Upstream: Salmon and Society in the Pacific Northwest (1996) Science and the Endangered Species Act (1995) Wetlands: Characteristics and Boundaries (1995) Biologic Markers (5 volumes, 1989-1995) Review of EPA's Environmental Monitoring and Assessment Program (3 volumes, 1994-1995) Science and Judgment in Risk Assessment (1994) Pesticides in the Diets of Infants and Children (1993) Dolphins and the Tuna Industry (1992) Science and the National Parks (1992) Human Exposure Assessment for Airborne Pollutants (1991) Rethinking the Ozone Problem in Urban and Regional Air Pollution (1991) Decline of the Sea Turtles (1990) Copies of these reports may be ordered from The National Academies Press (800) 624-6242 or (202) 334-3313 www.nap.edu

OCR for page R1
This page in the original is blank.

OCR for page R1
Preface The Environmental Protection Agency (EPA) is a mission agency established in 1970 to protect human health and to safeguard the natural environment. EPA’s regulatory and decision-making role requires that the agency have access to the best available science that is relevant to its mission. In an effort to improve the scientific foundation of its decision-making process, the agency established the Science To Achieve Results (STAR) research grants program in 1995. The STAR program is a competitive, peer-reviewed, extramural research grants program created to encourage interagency collaboration and to increase EPA’s access to the nation’s best scientists and engineers in academic and nonprofit research institutions. The program supports research in a variety of fields relevant to EPA’s mission, ranging from human health protection to environmental preservation. It is designed to maximize the independence of the researchers supported and to provide an equal opportunity for all researchers to qualify for support. In 2000, EPA asked the National Research Council to conduct an independent assessment of the STAR program. In response, the Research Council established the Committee to Review EPA’s Research Grants Program. In this report, the committee analyzes information provided by EPA, STAR grant recipients and fellows, and other sources to assess the program’s scientific merit, effect on the agency’s policies and decisions, and overall relevance to EPA’s mission. In addition, the committee compares some of the procedural aspects of the STAR program with those of basic and applied research grant programs of other agencies. Finally, the committee recom

OCR for page R1
mends ways to enhance the program and improve data collection for future program evaluations. This report has been reviewed in draft form by persons chosen for their diverse perspectives and technical expertise in accordance with procedures approved by the Research Council’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards of objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following people for their review of this report: William Glaze, University of North Carolina, Chapel Hill, North Carolina; Bernard D. Goldstein, University of Pittsburgh, Pittsburgh, Pennsylvania; Mark A. Harwell, Florida A&M University, Tallahassee, Florida; George Lucier (retired), National Institute of Environmental Health Sciences, Pittsboro, North Carolina; Perry L. McCarty, Stanford University, Stanford, California; Paul G. Risser, The Oklahoma State System of Higher Education, Oklahoma City, Oklahoma; Joan B. Rose, Michigan State University, East Lansing, Michigan; Jane Warren, Health Effects Institute, Boston, Massachusetts. Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by Bailus Walker, Jr., Howard University Medical Center, Washington, D.C. Appointed by the Research Council, he was responsible for making certain that an independent examination of the report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of the report rests entirely with the committee and the institution. The committee gratefully acknowledges the following people for making presentations to it: Peter Preuss, Paul Gilman, Timothy Oppelt, John Bachmann, Patricia Bradley, Elizabeth Bryan, William Farland, and James Hanlon, EPA; Robert Huggett, Michigan State University; John Wanska, General Accounting Office; Jerry Elwood, Department of Energy; Peter Johnson, U.S. Department of Agriculture; Claudia Thompson, National Institute of Environmental Health Sciences; Susan Cozzens, Ann Bostrom, and Alan Porter, Georgia Institute of Technology; Penny Firth, National Science Foundation; and Nils Newman, IISCO. The committee also wishes to thank the following, who provided further background information:

OCR for page R1
Theodore Just, Jack Puzak, Jeffrey Harris, Manju Gupta, Terry Simpson, Matthew Clark, Gina Perovich, Shirley Hamilton, and James Gentry, EPA; Claudia Magdalena Abendroth, Office of Management and Budget; Jeanne Powell, National Institute of Standards and Technology; Julia Melkers, University of Chicago; Christopher Allen, University of Vermont; Robert Selden, Air Force Scientific Advisory Board; Leslie Peasant, Air Force Office of Scientific Research; James Coleman, Nevada National Science Foundation; and Deborah Stine and Scott Weidman, National Research Council. In addition, the committee gives special thanks to the EPA project officers in three research programs—Endocrine Disruptors, Elaine Francis and David Reese; Particulate Matter, Gail Robarge and Stacey Katz; and Ecological Indicators, Barbara Levinson—who were available to discuss their programs in detail. We are also grateful to the many National Center for Environmental Research (NCER) staff who invested extensive time and effort in responding to all the committee’s requests for information. Finally, we appreciate the contributions of the numerous EPA STAR grantees and fellowship recipients who provided input on the program. The committee is grateful for the assistance of the Research Council staff in preparing this report: Eileen Abt, project director; James Reisa, director of the Board on Environmental Studies and Toxicology; Roberta Wedge, program director for risk analysis; Jennifer Saunders and Mirsada Karalic-Loncarevic, research assistants; Ruth E. Crossgrove, managing editor; Norman Grossblatt, senior editor; Kelly Clark, assistant editor; Lucy Fusco and Bryan Shipley, senior project assistants; and Robert Policelli and Tamara Dawson, project assistants. Finally, I thank the members of the committee for their dedicated efforts throughout the development of this report. Harold Mooney Chair, Committee to Review EPA’s Research Grants Program

OCR for page R1
This page in the original is blank.

OCR for page R1
Contents     Summary   1 1   Introduction   14     EPA’s Research Program,   15     The STAR Program,   16     The National Research Council Committee,   17     Organization of the Report,   18     References,   19 2   Overview of the STAR Program   20     Evolution of the Program,   21     Components of the Program,   27     Research Fields,   29     Operation of the Program,   44     Fellowships,   55     Previous Evaluations of the Program,   57     Conclusions and Recommendations,   59     References,   60 3   Competitive Grant Programs in Other Federal Agencies   64     The National Science Foundation,   65     National Institute of Environmental Health Sciences in NIH,   74

OCR for page R1
    Department of Energy, Office of Biological and Environmental Research,   78     U.S. Department of Agriculture National Research Initiative Competitive Grants Program,   80     National Aeronautics and Space Administration,   83     National Oceanic and Atmospheric Administration,   86     Conclusions,   89     References,   90 4   Measure for Measure   91     What are Metrics?   92     Use of Metrics in Evaluation,   94     Bibliometric Analysis,   98     Government Performance and Results Act,   101     Conclusions,   105     References,   106 5   Taking the Measure of STAR   108     Research Program,   109     Fellowship Program,   137     Conclusions,   139     Recommendations,   142     References,   143 Appendix A.   Biographic Information on the Committee to Review EPA’s Research Grants Program   149 Appendix B.   Previous Reviews of the STAR Program   153 Appendix C.   Performance Measures Used by Other Agencies and Organizations   165

OCR for page R1
The Measure of STAR Review of the U.S. Environmental Protection Agency’s Science to Achieve Results (STAR) Research Grants Program

OCR for page R1
This page in the original is blank.