The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program
identify three to five areas of climate change and global change research that can and should be evaluated through quantitative performance measures;
for these areas, recommend specific metrics for documenting progress, measuring future performance (such as skill scores, correspondence across models, correspondence with observations), and communicating levels of performance; and
discuss possible limitations of quantitative performance measures for other areas of climate change and global change research.
The committee approached its task first by examining the experience of industry, federal agencies, and academia with implementing metrics, and then by formulating possible metrics for a wide range of CCSP objectives. It began its deliberations with some skepticism as to whether metrics would apply to many of the elements of the program. However, analysis showed that it is possible to develop meaningful and useful measures for all parts of the CCSP. The difficulty arises in selecting a few areas of global change and climate change for which metrics should be developed (charge 2). The committee found that it was not possible to make this selection without a clearer sense of program priorities. The CCSP strategic plan does not contain measures of success, and program objectives are written too broadly for them to be inferred. However, even if such guidance were available, the committee found that a broader range of quantitative and qualitative metrics would be a more valuable tool for managing the program. The key to promoting successful outcomes is to consider the program from end to end, starting with program processes (e.g., planning and peer review) and inputs (e.g., resources) and extending to outputs (e.g., assessments, forecasts), outcomes (e.g., results for science and society), and long-term impacts. Principles and a framework for creating and implementing metrics for the entire CCSP are described below.
PRINCIPLES FOR DEVELOPING METRICS
Industry, federal agencies, and academia have different objectives in developing metrics. Industry has long used metrics to gauge progress in meeting business objectives and to identify where adjustments should be made to optimize performance and increase profits. Federal agencies are increasingly relying on metrics, either to manage programs or to increase their accountability to Congress and the public. The latter motivation was strengthened by the Government Performance and Results Act of 1993, which required federal agencies to set strategic goals and to measure program performance against those goals. Finally, academia uses metrics to supplement peer evaluation in decisions to hire or promote faculty members,