THINKING STRATEGICALLY
THE APPROPRIATE USE OF METRICS FOR THE CLIMATE CHANGE SCIENCE PROGRAM
THE NATIONAL ACADEMIES PRESS
Washington, D.C. www.nap.edu
THE NATIONAL ACADEMIES PRESS
500 Fifth Street, N.W. Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study was supported by the federal agencies of the U.S. Climate Change Science Program through the National Aeronautics and Space Administration under Contract No. NASW-01008. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
International Standard Book Number (ISBN) 0-309-09659-6 (Book)
Library of Congress Control Number 2005929740
Additional copies of this report are available from the
National Academies Press,
500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313; Internet http://www.nap.edu
Cover design by Van Nguyen, the National Academies Press.
Copyright 2005 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council.
COMMITTEE ON METRICS FOR GLOBAL CHANGE RESEARCH
ERIC J. BARRON, Chair,
Pennsylvania State University, University Park
ROGER C. BALES,
University of California, Merced
JOHN B. CARBERRY,
E.I. duPont de Nemours & Company, Newark, Delaware
DAVID J.C. CONSTABLE,
GlaxoSmithKline, King of Prussia, Pennsylvania
PAUL V. DESANKER,
Pennsylvania State University, University Park
MARVIN A. GELLER,
State University of New York, Stonybrook
EILEEN E. HOFMANN,
Old Dominion University, Norfolk, Virginia
HENRY D. JACOBY,
Massachusetts Institute of Technology, Cambridge
JOYCE E. PENNER,
University of Michigan, Ann Arbor
EUGENE A. ROSA,
Washington State University, Pullman
SUSAN E. TRUMBORE,
University of California, Irvine
KARL K. TUREKIAN,
Yale University, New Haven, Connecticut
CARL WUNSCH,
Massachusetts Institute of Technology, Cambridge
NRC Staff
ANNE LINN, Study Director
CLIMATE RESEARCH COMMITTEE
ANTONIO J. BUSALACCHI, JR., Chair,
University of Maryland, College Park
LEE E. BRANSCOME,
Climatological Consulting Cooperation, Palm Beach Gardens, Florida
JAMES A. COAKLEY, JR.,
Oregon State University, Corvallis
JULIA E. COLE,
University of Arizona, Tucson
JUDITH A. CURRY,
Georgia Institute of Technology, Atlanta
CLARA DESER,
National Center for Atmospheric Research, Boulder, Colorado
DAVID KAROLY,
University of Oklahoma, Norman
ROBERT J. LEMPERT,
RAND Corporation, Santa Monica, California
LINDA O. MEARNS,
National Center for Atmospheric Research, Boulder, Colorado
GERALD A. MEEHL,
National Center for Atmospheric Research, Boulder, Colorado
PETER B. RHINES,
University of Washington, Seattle
W. JAMES SHUTTLEWORTH,
University of Arizona, Tucson
LYNNE D. TALLEY,
Scripps Institution of Oceanography, La Jolla, California
HASSAN VIRJI,
International START Secretariat, Washington, D.C.
YUK YUNG,
California Institute of Technology, Pasadena
NRC Staff
AMANDA STAUDT, Senior Program Officer
ROB GREENWAY, Senior Program Assistant
BOARD ON ATMOSPHERIC SCIENCES AND CLIMATE
ROBERT J. SERAFIN, Chair,
National Center for Atmospheric Research, Boulder, Colorado
M. JOAN ALEXANDER,
Colorado Research Associates, Boulder
FREDERICK R. ANDERSON,
McKenna Long & Aldridge LLP, Washington, D.C.
MICHAEL L. BENDER,
Princeton University, New Jersey
ROSINA M. BIERBAUM,
University of Michigan, Ann Arbor
MARY ANNE CARROLL,
University of Michigan, Ann Arbor
CAROL ANNE CLAYSON,
Florida State University, Tallahassee
WALTER F. DABBERDT,
Vaisala Inc., Boulder, Colorado
KERRY A. EMANUEL,
Massachusetts Institute of Technology, Cambridge
DENNIS L. HARTMANN,
University of Washington, Seattle
PETER R. LEAVITT,
Weather Information, Inc., Newton, Massachusetts
JENNIFER A. LOGAN,
Harvard University, Cambridge, Massachusetts
VERNON R. MORRIS,
Howard University, Washington, D.C.
WILLIAM J. RANDEL,
National Center for Atmospheric Research, Boulder, Colorado
THOMAS H. VONDER HAAR,
Colorado State University/CIRA, Fort Collins
ROGER M. WAKIMOTO,
University of California, Los Angeles Ex Officio Members
ANTONIO J. BUSALACCHI, JR.,
University of Maryland, College Park
ERIC F. WOOD,
Princeton University, New Jersey
NRC Staff
CHRIS ELFRING, Director
PAUL CUTLER, Senior Program Officer
AMANDA STAUDT, Senior Program Officer
MARIA UHLE, Program Officer
ELIZABETH A. GALINIS, Senior Program Assistant
ROB GREENWAY, Senior Program Assistant
DIANE GUSTAFSON, Administrative Coordinator
ANDREAS SOHRE, Financial Associate
Acknowledgments
This report has been reviewed by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report:
Jack Azar, XEROX Corporation, Webster, New York
Rosina Bierbaum, University of Michigan, Ann Arbor
Susan Cozzens, Georgia Institute of Technology, Atlanta
Jack Fellows, University Corporation for Atmospheric Research, Boulder, Colorado
Debra Knopman, RAND Corporation, Arlington, Virginia
Roger Lukas, University of Hawaii, Honolulu
Michael Mann, University of Virginia, Charlottsville
Philip Marcus, University of California, Berkeley
Linda Mearns, National Center for Atmospheric Research, Boulder, Colorado
Elinor Ostrom, Indiana University, Bloomington
David Skole, Michigan State University, East Lansing
Although the individuals listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions and recommendations nor did they see the final draft of the report before its release. The review of this report was overseen by Robert Frosch, Harvard University, and Thomas Graedel, Yale University. Appointed by the National Research Council, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.
Preface
Federal agencies are increasingly being asked to document progress and measure performance to improve their accountability to Congress and the public and to provide information useful for making budget decisions. This task can be difficult to accomplish, especially in a program as complex as the Climate Change Science Program (CCSP), which spans all of the environmental and related social science disciplines and includes activities ranging from basic research to decision making in 13 federal agencies. Current approaches to evaluate progress (e.g., peer review of basic research, reduction of uncertainty) focus on particular aspects of the CCSP and/or have other limitations. For example, gaining improved understanding of the climate system can lead to increased uncertainties about some aspects of the system, yet progress has clearly been made. So, the question remains: How can progress in climate science be demonstrated after nearly 15 years of sponsored research and observations?
At the request of Dr. James Mahoney, director of the U.S. Climate Change Science Program and chair of the Subcommittee on Global Change Research, the National Research Council established a committee to develop quantitative metrics and performance measures for documenting progress and evaluating future performance for selected areas of global change and climate change research. Committee membership included researchers drawn from a wide range of global change disciplines and experts from industry and government with practical experience in developing and using metrics.
The Committee on Metrics for Global Change Research held three meetings from December 2003 to June 2004 to discuss the issues and to gather input in three major areas:
-
the different types of metrics (e.g., input, outcome) and the different scales of programs that can be evaluated usefully by such measures;
-
the experience of industry, academia, and federal government agencies in measuring performance; and
-
lessons learned from retrospective analysis of science programs.
A fourth meeting (September-October 2004) was devoted to writing this report. In preparing its report the committee strove to provide practical advice on the applicability of performance measures across the full range of CCSP goals and approaches—from discovery science, to modeling and assessment, to communicating results and managing risk.
The committee thanks the following individuals for making presentations or providing other input: David Bader, Susan Cozzens, James Hack, Richard Hallgren, Jack Kaye, Charles Kennel, Mike MacCracken, James Mahoney, Richard Moss, Franklin Nutter, Cheryl Oros, John Parascandola, Craig Robinson, Jason Rothenberg, Sherwood Rowland, and Spencer Weart. Thanks also go to members of the Climate Research Committee and Board on Atmospheric Sciences and Climate—particularly Anthony Busalacchi, James Coakley, David Karoly, Robert Serafin, and Lynne Talley—for their input and guidance throughout the study. Finally, the committee extends its appreciation to the NRC staff, particularly study director Anne Linn, for their highly professional contributions to this report.
Eric Barron
Chair