SETTING PRIORITIES for LARGE RESEARCH FACILITY PROJECTS supported by the NATIONAL SCIENCE FOUNDATION
THE NATIONAL ACADEMIES PRESS
Washington, D.C. www.nap.edu
THE NATIONAL ACADEMIES PRESS
500 Fifth Street, N.W. Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
Support for this project was provided by the National Science Foundation (under grant OIA-0304912). Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
International Standard Book Number 0-309-09084-9 (Book)
International Standard Book Number 0-309-52775-9 (PDF)
Available from the Committee on Science, Engineering, and Public Policy; and the Board on Physics and Astronomy, 500 Fifth Street, NW, Washington, DC 20001; 202-334-2807; Internet, http://www.nationalacademies.org/cosepup and http://www.nationalacademies.org/bpa.
Additional copies of this report are available from the
National Academies Press,
500 Fifth Street, NW, Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu.
Copyright 2004 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council.
COMMITTEE ON SETTING PRIORITIES FOR NSF-SPONSORED LARGE RESEARCH FACILITY PROJECTS
WILLIAM BRINKMAN (Chair), retired,
Department of Physics, Princeton University, Princeton, New Jersey
DAVID AUSTON, President,
Kavli Foundation, Oxnard, California
PERSIS DRELL, Professor and Director of Research,
Stanford Linear Accelerator Center, Menlo Park, California
ALAN DRESSLER, Astronomer, Member of Scientific Staff,
Observatories of the Carnegie Institution, Pasadena, California
WILLIAM FRIEND, Executive Vice President (retired),
Bechtel Group, Inc., and
Chairman,
University of California’s President’s Council-National Laboratories, Washington, D.C.
BRUCE HEVLY, Associate Professor,
Department of History, University of Washington, Seattle, Washington
WESLEY HUNTRESS, JR., Director,
Geophysical Laboratory, Carnegie Institution of Washington, Washington, D.C.
SIR CHRIS LLEWELLYN-SMITH, Director,
The United Kingdom Atomic Energy Authority Culham Division, Abingdon, United Kingdom
LINDA (LEE) MAGID, Professor of Chemistry,
University of Tennessee, Knoxville, and
Acting Director,
Joint Institute for Neutron Sciences
MARC PELAEZ, Rear Admiral United States Navy (retired), former Chief of Naval Research, and Vice President (retired),
Newport News Shipbuilding, Newport News, Virginia
ROBERT RUTFORD, Professor,
Geosciences Department, University of Texas at Dallas, Dallas, Texas
JOSEPH TAYLOR, Dean of Faculty and James S. McDonnell Distinguished University Professor of Physics,
Princeton University, Princeton, New Jersey
MICHAEL TELSON,
University of California, Office of Federal Government Relations, Washington, D.C.
G. DAVID TILMAN, Regents Professor, McKnight Presidential University Chair,
Department of Ecology, Evolution, and Behavior, and
Director,
Cedar Creek Natural History Area Ecology, University of Minnesota, Saint Paul, Minnesota
Principal Project Staff
DEBORAH STINE, Study Director
DONALD SHAPERO, Collaborating Board Director
TIMOTHY MEYER, Program Officer
STEVE OLSON, Consultant Science Writer
KEVIN ROWAN, Project Associate
LAURA HOLLIDAY, Program Associate
JONATHAN TUCKER, Christine Mirzayan Science and Technology Policy Intern
BLAKE PURNELL, Christine Mirzayan Science and Technology Policy Intern
ARTI GARG, Christine Mirzayan Science and Technology Policy Intern
NORMAN GROSSBLATT, Senior Editor
COMMITTEE ON SCIENCE, ENGINEERING, AND PUBLIC POLICY
MAXINE F. SINGER (Chair), President Emerita,
Carnegie Institution of Washington, Washington, D.C.
BRUCE ALBERTS (ex officio), President,
The National Academies, Washington, D.C.
R. JAMES COOK, R. James Cook Endowed Chair in Wheat Research,
Washington State University, Pullman, Washington
HAILE DEBAS, Dean, School of Medicine, and Vice Chancellor,
Medical Affairs, University of California, San Francisco, California
GERALD DINNEEN (ex officio), Vice President (retired),
Science and Technology, Honeywell, Inc., Edina, Minnesota
HARVEY FINEBERG (ex officio), President,
Institute of Medicine, Washington, D.C.
MARYE ANNE FOX (ex officio), Office of the Chancellor,
North Carolina State University, Raleigh, North Carolina
ELSA GARMIRE, Sydney E. Junkins Professor of Engineering,
Dartmouth College, Hanover, New Hampshire
WILLIAM JOYCE (ex officio), President and CEO,
Union Carbide Corporation, Wilmington, Delaware
MARY-CLAIRE KING, American Cancer Society Professor of Medicine and Genetics,
University of Washington, Seattle, Washington
W. CARL LINEBERGER, Professor of Chemistry,
Joint Institute for Laboratory Astrophysics, University of Colorado, Boulder, Colorado
GERALD RUBIN, Vice President for Biomedical Research,
Howard Hughes Medical Institute, Chevy Chase, Maryland
EDWARD SHORTLIFFE, Professor and Chair,
Department of Biomedical Informatics, Columbia University, New York, New York
HUGO SONNENSCHEIN, Charles L. Hutchinson Distinguished Service Professor,
Department of Economics, The University of Chicago, Chicago, Illinois
IRVING WEISSMAN, Karel and Avice Beekhuis Professor of Cancer Biology,
Stanford University, Palo Alto, California
SHEILA WIDNALL, Abbey Rockefeller Mauze Professor of Aeronautics,
Massachusetts Institute of Technology, Cambridge, Massachusetts
WM. A. WULF (ex officio), President,
National Academy of Engineering, Washington, D.C.
Staff
RICHARD BISSELL, Executive Director
DEBORAH STINE, Associate Director
LAUREL HAAK, Program Officer
MARION RAMSEY, Administrative Associate
JAMES MCKINNEY, Financial Associate
BOARD ON PHYSICS AND ASTRONOMY
BURTON RICHTER (Chair), Paul Pigott Professor in the Physical Sciences,
Stanford University and
Director Emeritus,
Stanford Linear Accelerator Center, Stanford University, Menlo Park, California
ANNEILA SARGENT (Vice Chair), Director,
Owens Valley Radio Observatory, and
Senior Research Associate,
California Institute of Technology, Pasadena, California
ELIHU ABRAHAMS, Professor of Physics,
Emeritus, Rutgers University, Piscataway, New Jersey
JONATHAN BAGGER, Professor of Physics and Astronomy,
Johns Hopkins University, Baltimore, Maryland
GORDON BAYM, George and Ann Fisher Distinguished Professor of Engineering,
University of Illinois at Urbana-Champaign, Urbana, Illinois
WILLIAM EATON, Chief of the Laboratory of Chemical Physics,
National Institutes of Health, Bethesda, Maryland
RAYMOND FONCK, Professor in the Department of Engineering Physics,
University of Wisconsin at Madison, Madison, Wisconsin
WENDY FREEDMAN, Director,
Carnegie Observatories, Pasadena, California
LAURA GREENE, Swan Lund Chair and Professor of Physics,
University of Illinois at Urbana-Champaign, Urbana, Illinois
FRANCES HELLMAN, Professor of Physics,
University of California, San Diego, California
ERICH IPPEN, Professor of Engineering,
Massachusetts Institute of Technology, Cambridge, Massachusetts
LINDA (LEE) MAGID, Professor of Chemistry,
University of Tennessee, Knoxville, and
Acting Director,
Joint Institute for Neutron Sciences, Knoxville, Tennessee
THOMAS O’NEIL, Professor and Chair of the Department of Physics,
University of California, San Diego, California
JULIA PHILLIPS,
Physical and Chemical Sciences Center, Sandia National Laboratories, Albuquerque, New Mexico
JOSEPH TAYLOR, JR., Professor of Physics,
Princeton University, Princeton, New Jersey
THOMAS THEIS, Director of Physical Sciences,
IBM Research Division, Yorktown Heights, New York
C. MEGAN URRY,
Center for Astronomy and Astrophysics, Department of Physics, Yale University, New Haven, Connecticut
CARL WIEMAN, Professor of Physics and Fellow of JILA,
University of Colorado/JILA, Boulder, Colorado
Staff
DONALD SHAPERO, Director
ROBERT RIEMER, Senior Program Officer
BRIAN DEWHURST, Research Associate
TIMOTHY MEYER, Program Officer
PAMELA LEWIS, Project Associate
NELSON QUIÑONES, Project Assistant
VAN AN, Financial Associate
Preface
For many years, policy makers and the scientific community have focused attention on the support provided by the National Science Foundation (NSF) for large facilities used in scientific and engineering research. Previous reports have addressed the complex issues that arise in choosing among facility proposals and in balancing support for facilities and other tools with support for research conducted by individual investigators.1 As large facilities have become an increasingly prominent part of the nation’s research and development portfolio and as NSF has entered a period of budgetary expansion, concerns once again have intensified.
In a letter to the president of the National Academies dated June 12, 2002, Senators Barbara Mikulski, Christopher Bond, Ernest Hollings, John
1 |
Appendix G provides the executive summary of the Committee on Science, Engineering, and Public Policy’s previous report from the Panel on NSF Decisionmaking for Major Awards, chaired by Robert Rutford, titled Major Award Decisionmaking at the National Science Foundation (Washington, D.C.: National Academy Press, 1994). Other reports of interest include reports from the National Science Board, Criteria for the Selection of Research Projects by the National Science Foundation (Washington, D.C.: National Science Foundation, 1974); Office of Technology Assessment, US Congress, Federally Funded Research: Decisions for a Decade (Washington, D.C.: US Government Printing Office, 1991); President’s Council of Advisors on Science and Technology, Megaprojects in the Sciences (Washington, D.C.: Office of Science and Technology Policy, Executive Office of the President, 1992); and National Science Board, Science and Engineering Infrastructure for the 21st Century: The Role of the National Science Foundation (Arlington, Virginia: National Science Foundation, 2003). |
McCain, Edward Kennedy, and Judd Gregg stated that “questions remain as to whether the NSF has a satisfactory process for prioritizing multiple competing large-scale research facility projects.” The letter said that NSF funding of requests for large facility projects appears to be “ad hoc and subjective.” It also pointed out that the NSF inspector general had recently found “significant deficiencies in the Foundation’s management and oversight of its large facility projects resulting in significant cost overruns not contemplated in their original budgets.” To address those concerns—which also have been expressed by members of the House Committee on Science and by the members and staffs of other congressional committees and subcommittees—the letter requested that the National Academy of Sciences “review the current prioritization process and report to us on how it can be improved.”
In response to the request, the National Academies appointed the Committee on Setting Priorities for NSF-Sponsored Large Research Facility Projects2 to address the following charge:
-
Review NSF’s current prioritization process as well as processes and procedures used by other relevant organizations.
-
Develop the criteria that should be considered in developing priorities among competing large research facility proposals.
-
Provide recommendations for optimizing and strengthening the process used by the NSF to set priorities among large research facility project proposals and to manage their incorporation into the President’s budget.
-
Provide recommendations for improving the construction and operation of NSF-funded large research facility projects.
-
Provide recommendations regarding the role of the current and future availability of international and interagency research facility projects in the decision-making process for NSF funding of large research facility projects.3
This report focuses on a portion of NSF’s activities that is small (less than 4 perccent) compared with the foundation’s overall budget but is nevertheless central to its mission. It examines the policies and procedures governing awards made through the Major Research Equipment and Facilities Construction (MREFC) account.4 NSF uses the MREFC
2 |
Appendix A provides biographical information on the committee members. |
3 |
Appendix B provides a copy of the charge, the Senate letter, and related congressional documents. |
4 |
Appendix C provides histories of all current MREFC projects and those approved but not funded. |
account to support the “acquisition, construction, commissioning, and upgrading of major research equipment, facilities, and other such capital assets” that cost more than several tens of millions of dollars. The report looks at how plans and proposals for large research facilities originate, how NSF chooses which facilities to support, and how it oversees their construction. These “large research facility projects” represent major investments in the future of a given field of research. Funding the construction of a large facility affects the direction of research for many years and implies continued support for the operations and maintenance of the facility. Large research facilities also can have a substantial effect on regional economies, public perceptions of science, workforce training, and international cooperation in research. NSF’s support of large facility projects is a critical element of US science and technology policy and warrants sustained attention from policy makers and the research community.
In responding to its charge, the committee examined numerous NSF documents,5 National Science Board (NSB) minutes and presentations, congressional testimony, and news articles, web sites, and reports that discuss the facilities. The committee also compared NSF’s current process with that used by the Department of Energy (DOE) Office of Science, the National Aeronautics and Space Administration (NASA) Office of Space Science, the United Kingdom, and Germany.6 In addition, it compiled examples of criteria that have been used or proposed for use to set research priorities by various organizations and several countries.7 This study also builds on the Committee on Science, Engineering, and Public Policy’s (COSEPUP’s) 1994 report Major Award Decisionmaking at NSF, which addressed some of the same issues that are of concern here.8 Finally, the committee had useful discussions with the staff of the Senate Appropriations Committee, the House Science Committee, NSF, DOE, NASA, and disciplinary societies and researchers. Those people are listed in the acknowledgments section.
Given the ever-changing and draft nature of NSF’s process for setting priorities among its proposals for large research facilities, the committee decided that it would not be fair to NSF to conduct an investigation of each decision it had made since 1995 (when the MREFC account was created) or even earlier (when some of the current projects began construction). The committee chose instead to examine the process as it exists today as outlined by NSF and to focus on how that process can be improved from the time of project conception to operation.
5 |
See key excerpts in Appendix F. |
6 |
See Appendix D. |
7 |
See Appendix E. |
8 |
See Appendix G. |
In doing so, the committee concluded that although NSF has improved its process for setting priorities among large facility projects, further strengthening is needed, if NSF is to meet the demands that will be made of it in the future. This report lays out specific recommendations that describe how large facility projects should be ranked within and among disciplines. In addition, it discusses how NSF can enhance preapproval planning and budgeting of projects and oversight of construction and operation once projects are approved to ensure that the nation’s investment is ultimately successful. As research opportunities and agency initiatives change, the recommendations in this report should remain at the core of the procedures used to identify, develop, set priorities among, and manage large facility projects. By implementing the report’s recommendations, NSF, in partnership with the research community, can develop a system of short-term and long-term planning that is sufficiently robust to direct funding to the most meritorious research projects. In that way, NSF can increase its already substantial contributions to the nation’s science and engineering enterprise.
William F. Brinkman, Chair
Committee on Setting Priorities for NSF-Sponsored Large Research Facility Projects
Acknowledgments
This report is the product of many people. First, we thank all those who provided information at our committee meetings in 2003. They were (in alphabetical order)
MARC ALLEN, Director, Strategic and International Planning, National Aeronautics and Space Administration
BARRY BARISH, Maxine and Ronald Linde Professor of Physics and Director of Laser Interferometer Gravitational Wave Observatory Laboratory, California Institute of Technology
JOSEPH BORDOGNA, Deputy Director, National Science Foundation
RITA COLWELL, Director, National Science Foundation
PATRICIA DEHMER, Associate Director, Office of Basic Energy Sciences, Office of Science, Department of Energy
ADRIENNE FROELICH, Director of Public Policy, American Institute of Biological Sciences
SCOTT GILES, Deputy Chief of Staff, Majority Staff, House Committee on Science
DAVID GOLDSTON, Chief of Staff, Majority Staff, House Committee on Science
MARTHA HAYNES, Goldwin Smith Professor of Astronomy and Director of Undergraduate Studies, Cornell University
ANITA JONES, Chair, Committee on Programs and Plans, National Science Board
CHEH KIM, Professional Staff Member, Majority Staff, Senate Appropriations Committee
MIKE LUBELL, Director of Public Affairs, American Physical Society, and Chairman, Physics Department, City College of New York
JOHN MARBURGER III, Director, White House Office of Science and Technology Policy
KEVIN MARVEL, Deputy Executive Officer, American Astronomical Society
TED MOORE, Professor of Geological Sciences, University of Michigan
ERIC NAGY, President, Organization of the Biological Field Stations, and Associate Director, Mountain Lake Biological Station, and Assistant Professor, Department of Biology, University of Virginia
RAY ORBACH, Director, Office of Science, Department of Energy
DAVID RADZANOWSKI, Branch Chief, Science and Space Programs Branch, Office of Management and Budget
PETER ROONEY, Staff Director, Majority Staff, Research Subcommittee, House Committee on Science
JAMES TIEDJE, University Distinguished Professor of Microbiology and Director of National Science Foundation Center for Microbial Ecology, Michigan State University
DAVID TRINKLE, Program Examiner, Science and Space Programs Branch, Office of Management and Budget
JIM WILSON, Professional Staff Member, Minority Staff, Research Subcommittee, House Committee on Science
Without the input of each of those speakers, this report would not have been possible.
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC’s Report Review Committee. The purpose of the independent review is to provide candid and critical comments that will assist the institution in making the published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.
We wish to thank the following for their participation in the review of this report: John Armstrong (Retired), IBM Corp; Arthur Bienenstock, Stanford University; Bernard Burke, Massachusetts Institute of Technology; Claude Canizares, Massachusetts Institute of Technology; James Clark, Duke University; John Evans (Retired), Comcast Corporation; Paul Gaffney, Monmouth University; Norine Noonan, College of Charleston; Yves Petroff, European Synchrotron Radiation Facility; J. Michael Rowe,
National Institute of Standards and Technology; and Patrick Windham, Windham Consulting.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by John Ahearne, Sigma Xi (The Scientific Research Society), and France Cordova, University of California, Riverside. Appointed by the National Research Council, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the author committee and the institution.
In addition, we thank the guidance group that oversaw this project:
MAXINE SINGER (Chair), President Emerita, Carnegie Institution of Washington
SHIRLEY CHIANG, Professor of Physics, Department of Physics, University of California, Davis
JAMES DUDERSTADT, President Emeritus and University Professor of Science and Engineering, University of Michigan
ELSA GARMIRE, Sydney E. Junkins Professor of Engineering, Dartmouth College
JOHN HUCHRA, Professor, Harvard-Smithsonian Center for Astrophysics
BURTON RICHTER, Paul Pigott Professor in the Physical Sciences, Stanford University, and Director Emeritus, Stanford Linear Accelerator Center
HUGO SONNENSCHEIN, Charles L. Hutchinson Distinguished Service Professor, Department of Economics, University of Chicago
Finally, we thank the staff of this project, including Deborah Stine, associate director of the Committee on Science, Engineering, and Public Policy and study director, who managed the project; Timothy Meyer, program officer of the Board on Physics and Astronomy; Steve Olson, the science writer for this report; Kevin Rowan and Laura Holliday, who provided project support; National Academies Science and Technology Policy Interns Jonathan Tucker, who conducted the initial background research for the committee, and Blake Purnell and Arti Garg, who developed the historical analysis in Appendix C; Donald Shapero, director of the Board on Physics and Astronomy; and Richard Bissell, executive director of the Committee on Science, Engineering, and Public Policy and of the Policy and Global Affairs Division.