5
Process of Developing Metrics

The committee was asked to identify performance measures and metrics for documenting progress, measuring future performance, and communicating levels of performance in three to five areas of climate and global change research. This task presented a considerable challenge because of the enormous breadth of the Climate Change Science Program (CCSP) and the lack of models for developing metrics for multiagency programs. This chapter describes the process by which the committee developed metrics for the CCSP.

FRAMEWORK FOR MEASURING PROGRESS TOWARD CCSP GOALS

Metrics are intended to assess progress toward stated goals. The Office of Management and Budget (OMB) notes that many strategic goals are difficult to measure and encourages agencies to develop “specific, operational performance goals that align with strategic goals.”1 Performance measures are then created for these operational performance goals.

1  

Office of Management and Budget, 2005, Guidance for Completing the Program Assessment Rating Tool (PART), p. 9, <http://www.whitehouse.gov/omb/part/fy2005/2005_guidance.doc>.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 63
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program 5 Process of Developing Metrics The committee was asked to identify performance measures and metrics for documenting progress, measuring future performance, and communicating levels of performance in three to five areas of climate and global change research. This task presented a considerable challenge because of the enormous breadth of the Climate Change Science Program (CCSP) and the lack of models for developing metrics for multiagency programs. This chapter describes the process by which the committee developed metrics for the CCSP. FRAMEWORK FOR MEASURING PROGRESS TOWARD CCSP GOALS Metrics are intended to assess progress toward stated goals. The Office of Management and Budget (OMB) notes that many strategic goals are difficult to measure and encourages agencies to develop “specific, operational performance goals that align with strategic goals.”1 Performance measures are then created for these operational performance goals. 1   Office of Management and Budget, 2005, Guidance for Completing the Program Assessment Rating Tool (PART), p. 9, <http://www.whitehouse.gov/omb/part/fy2005/2005_guidance.doc>.

OCR for page 63
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program OMB Approach CCSP Committee Approach Strategic goals Strategic goals CCSP strategic goals Performance goals Questions     Milestones, products, payoffs Eight themes Performance measures   Metrics The CCSP strategic plan does not contain operational performance goals, and the committee found that the five CCSP goals are indeed stated in terms that are too broad to serve as a framework for developing meaningful metrics. Consequently, the committee considered the 224 milestones, products, and payoffs identified in the CCSP strategic plan, which provide greater specificity about what the program is trying to achieve. The committee found that the milestones, products, and payoffs could be grouped into eight themes for which metrics could be developed. These themes are improve data sets in space and time (e.g., create maps, databases, and data products; densify data networks); improve estimates of physical quantities (e.g., through improvement of a measurement); improve understanding of processes; improve representation of processes (e.g., through modeling); improve assessment of uncertainty, predictability, or predictive capabilities; improve synthesis and assessment to inform; improve the assessment and management of risk; and improve decision support for adaptive management and policy making. The phrasing of these eight themes either matches or is closely allied with the phrasing of nearly all of the program’s milestones, products, and payoffs. In addition, the themes represent a sequence in scientific investigation, starting from the development of new or better observations, to an improved understanding of processes, to an improved capability to predict or forecast future climate changes, and finally to improved use of information to better serve society. As such, they offer an organizing framework for developing metrics for assessing the full range of CCSP activities. The committee proceeded under the assumption that metrics would be very different for each of these themes and that developing quantifiable measures for many elements of the CCSP would be difficult. For example, metrics to assess improvements in CO2 observing systems seemed likely to differ from metrics to evaluate new knowledge about processes that control

OCR for page 63
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program carbon sources and sinks. Moreover, it seemed likely that metrics might be specific to a particular program element, such as understanding climate change feedbacks.2 To test these assumptions, the committee chose one or two case studies for each of the eight themes, based on the areas of expertise of the members. Each case study followed the same format: (1) an introduction to the issue being assessed; (2) a description of the relevant milestone, product, or payoff being addressed as stated in the CCSP strategic plan; and (3) example metrics divided into the categories of process, input, output, outcome, and impact metrics. In addition, the difficulty of assigning the metrics to the case study objectives and generalizing them to other parts of the CCSP was assessed. Two case studies illustrating different kinds of CCSP objectives are presented in the next section, and a selection of others, in the preliminary form that guided the committee, is summarized in Appendix B. The case studies were also mapped onto the CCSP overarching goals and themes as a second check on the breadth of analysis provided by this approach (Table 5.1). EXAMPLE CASE STUDIES Two examples illustrate the differences and similarities of metrics developed for very different parts of the program. The first—the effect of carbon dioxide on land carbon balance—is science oriented and illustrates the CCSP theme of improving understanding of processes (theme 3). The second—adaptive management of water resources—is oriented toward decision support (theme 8). In developing the respective metrics, Tables 5.2 and 5.3, the committee considered relevant performance measures from agency strategic plans and Program Assessment Rating Tool (PART) submissions (e.g., Tables 2.4 and 2.5), generic research and development (R&D) metrics developed elsewhere (Appendix C), the literature, and the committee’s experience with the CCSP research question. 2   Metrics for the latter were proposed in National Research Council, 2003, Understanding Climate Change Feedbacks, The National Academies Press, Washington, D.C., 152 pp. They include (1) comparison of observed and simulated response of clouds, water vapor, and lapse rate to every well-observed forcing mechanism and time scale, including the diurnal and seasonal response, the response to ENSO (El Niño-Southern Oscillation), and the response to volcanic eruptions; (2) the accuracy with which Earth system models can reproduce observed diurnal and seasonal variations of the hydrological cycle over land; (3) total water column heat content along decadally monitored transoceanic cross sections; and (4) tropical Pacific sea surface temperature and pycnocline depth to evaluate model performance and to diagnose and monitor decadal and longer-term changes in ENSO statistics.

OCR for page 63
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program TABLE 5.1 Relationship Between the CCSP Question That the Case Study (a-j) Is Trying to Address, CCSP Overarching Goals, and Committee-Identified Themes Themes CCSP Overarching Goals Improve Knowledge Improve Quantification Reduce Uncertainty Understand Adaptability Manage Risk Improve data sets a, b a, b b     Estimate physical quantities c c c     Understand processes d d       Represent processes e e e e   Assess uncertainty, predictability f f f f   Synthesize and assess to inform g g g g g Assess and manage risk h h h h h Adaptive management, policy making i   i, j i, j i, j KEY: a = Case study on solar forcing of climate: To what extent are climate changes as observed in instrumental and paleoclimate records related to volcanic and solar variability, and what mechanisms are involved in producing climate responses to these natural forcings? b = Case study on aerosols and their role in climate forcing: What are the climate-relevant chemical, microphysical, and optical properties, and the spatial and temporal distributions, of human-caused and naturally occurring aerosols? c = Case study on sea-level rise: What are the projected contributions from different components of the climate system to future sea-level changes, what are the uncertainties in the projections, and how can they be reduced? d = Case study on the effect of carbon dioxide on land carbon balance: What are the potential consequences of global change for ecological systems? e =Case study on climate-vegetation feedbacks: What are the most important feedbacks between ecological systems and global change (especially climate), and what are their quantitative relationships?

OCR for page 63
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program f = Case study on paleoclimate time series as benchmarks of climate variability and change: To what extent are climate changes as observed in instrumental and paleoclimate records related to volcanic and solar variability, and what mechanisms are involved in producing climate responses to these natural forcings? g = Case study on human health and climate: What are the potential human health effects of global environmental change, and what climate, socioeconomic, and environmental information is needed to assess the cumulative risk to health from these effects? h = Case study on assessing, preventing, and managing public health threats of infectious diseases: How can the methods and capabilities for societal decision making under conditions of complexity and uncertainty about global environmental variability and change be enhanced? What are the potential human health effects of global environmental change, and what climate, socieoeconomic, and environmental information is needed to assess the cumulative risk to health from these effects? i = Case study on adaptive management of water resources: How can information on climate variability and change be most efficiently developed, integrated with nonclimatic knowledge, and communicated in order to best serve societal needs? j = Case study on policy making based on scenarios of greenhouse emissions and climate response: What are the current and potential future impacts of global environmental variability and change on human welfare, what factors influence the capacity of human societies to respond to change, and how can resilience be increased and vulnerability decreased? Effect of Carbon Dioxide on Land Carbon Balance Related CCSP Questions, Milestones, and Products. Question 8.2: “What are the potential consequences of global change for ecological systems?”3 The related milestones, products, and payoffs include improved understanding of processes about (1) how elevated CO2 concentrations, warming, and altered hydrology will influence the productivity of land plants and the net carbon balance of terrestrial ecosystem; and (2) how this response will evolve over time in response to other factors that influence carbon storage in ecosystems over the next century. Rationale. The capacity for CO2 “fertilization” in land ecosystems may be responsible for some of the apparent land carbon sink observed in the 1990s.4 However, the means by which the products of increased photosynthesis are allocated and the fate (and therefore residence time) of plant 3   Climate Change Science Program and Subcommittee on Global Change Research, 2003, Strategic Plan for the U.S. Climate Change Science Program, Washington, D.C., pp. 87–89. 4   Schimel, D.S., Terrestrial ecosystems and the carbon cycle, Global Change Biology, 1, 77–92, 1995; Intergovernmental Panel on Climate Change, Working Group I, 2001, Climate Change 2001: The Scientific Basis, Cambridge University Press, Cambridge, U.K., pp. 195–196; Schimel, D.S., and 29 coauthors, 2001, Recent patterns and mechanisms of carbon exchange by terrestrial ecosystems, Nature, 414, 169–172.

OCR for page 63
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program TABLE 5.2 Effect of CO2 on Land Carbon Balance Type Example Metrics Process • Has the leadership of this overall effort, which spans several agencies, been identified? • Does a structure exist that will involve the scientific community in planning the sites and conditions chosen for manipulation or gradient studies? • Is there a 5-10-year plan for implementation of the manipulation experiments, to be revisited and updated in accord with new discoveries? • Is there a plan to incorporate longer-term aspects of the problem that extend beyond the 5-10-year horizon (i.e., multiple generations of plants exposed to altered atmospheric conditions)? • Do a mechanism and timetable exist for periodic review of experimental implementations, including testing of model predictions outside experimental areas? • Do a mechanism and timetable exist to disseminate results to potential stakeholders (particularly the agricultural community) and involve them in planning discussions? Input • Is there sufficient theoretical basis for the design and interpretation of experiments? • Is the technology available to perform experiments assuming multiple, long-term (decadal) manipulations of plots of sufficient size to test hypotheses? • Are sufficient resources (people, dollars) available to implement and support a measurement network, modeling, and interpretive activities for the appropriate period of time (decades)? • Is there an identified stakeholder community to take advantage of scientific advances? Output • Peer-reviewed, published results generated for each site and synthesis activities across sites that identify the most important mechanisms at work • Production of a facility that (1) can be put into the field for years at a time and (2) can maintain atmospheric CO2 levels at a specific set point (e.g., 50 ppm [parts per million] above ambient levels), with a precision (averaged over 1 hour) of 5 ppm. For a subset of these systems, additional control over either atmospheric ozone levels, temperature (i.e., increase by 5°C compared to the control plot), soil moisture, or species diversity is required • Development of a suite of new measurement techniques that can detect carbon allocation patterns on time scales of (1) hours, (2) days to weeks, and (3) a growing season in response to external variables and photosynthetic rates of plants in control versus experimentally manipulated systems • Incorporation of relationships between photosynthetic rates, carbon allocation, and external and internal variables into process-based models that simulate patterns of photosynthetic response and allocation (on appropriate time scales for each process) and that can be tested against other observations as well as in other kinds of manipulated systems • Technology developed for rapid control of trace gas concentrations at high precision

OCR for page 63
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program Type Example Metrics Outcome • Peer-reviewed and published knowledge of the processes by which increasing atmospheric CO2 can influence the carbon balance at (1) the whole plant level and (2) the ecosystem level. Determination of the sign and magnitude (to 30%) of the feedback between CO2 levels and the amount of carbon stored over the first year of the manipulation (and subsequent years as they become available) • Models of suitable spatial scale that incorporate process-level understanding are used to predict the response of ecosystems to multiple stressors, such as increased CO2 and temperature or CO2 and ozone • Policy makers are informed about —The potential for different kinds of ecosystems to store or release carbon under conditions of a 50 ppm increase in atmospheric CO2 —The magnitude of release or uptake of CO2 and how this understanding will be modified by the presence of more investigators in the field • Peer-reviewed assessments that quantify the potential effects of changing atmospheric composition on the yield of different crops • Improved prediction of future trends in atmospheric CO2 levels, given a scenario of fossil fuel emissions and deforestation Impact • Crop productivity is improved because of use of forecasts that take into account changes in CO2, ozone, and climate • Conservation reserves are more resilient because of use of knowledge of how changes in CO2 affect plant competition and ecosystem structure carbon stores are still matters of debate and uncertainty.5 On longer time scales, when factors such as disturbance frequency must be included in assessments of land carbon balance, even the sign of land carbon response to elevated CO2 is uncertain.6 Higher-CO2 conditions may favor one kind of plant over another—changing the structure of ecological communities, their functions (including carbon storage), and their vulnerability to disturbance such as fire. Further complications arise when increased CO2 is correlated with other factors that affect plant productivity, such as changes in climate, deposition of excess nitrogen, presence of high O3 levels, or 5   Bazazz, F.A., 1990, The response of natural ecosystems to rising global CO2 levels, Annual Review of Ecology and Systematics, 21, 167–196; Woodwell, G.M., F.T. Mackenzie, R.A. Houghton, M. Apps, E. Gorham, and E. Davidson, 1998, Biotic feedbacks in the warming of the Earth, Climatic Change, 40, 495–518. 6   Korner C., 2004, Through enhanced tree dynamics carbon dioxide enrichment may cause tropical forests to lose carbon, Philosophical Transactions of the Royal Society of London, Series B, 359, 493–498; Chambers, J.Q., and W.L. Silver, 2004, Some aspects of ecophysiological and biogeochemical responses of tropical forests to atmospheric change, Philosophical Transactions of the Royal Society of London, Series B, 359, 463–476.

OCR for page 63
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program invasion of nonnative plants.7 The lack of understanding of fundamental biogeochemical and ecological processes limits our ability to predict the ultimate consequences of elevated CO2 on land carbon balance. Background. Investigation of the effects of elevated CO2 on land carbon balance has relied on manipulative experiments and natural gradient studies to isolate the physiological responses of plants on a variety of time scales. The CCSP strategic plan calls for augmentation of these manipulative studies, including addition of factors such as nitrogen or ozone, to improve the understanding of ecosystem response to climate change. Additional studies must be conducted in a variety of ecosystem types and include participation by a diverse range of scientists to study the physiological processes that mediate plant response to elevated CO2, the mechanisms (and time scale) by which those changes in plant carbon balance are translated into ecosystem carbon storage, and the spatial variability of edaphic factors (e.g., climate, nutrient availability) that regulate the magnitude of response of individual plants and ecosystems. Improved understanding of these processes will ultimately be incorporated into models that estimate the magnitude of the global carbon land balance. Adaptive Management of Water Resources Related CCSP Questions, Milestones, and Products. Question 4.5: “How can information on climate variability and change be most efficiently developed, integrated with non-climatic knowledge, and communicated in order to best serve societal needs?”8 Relevant milestones and products can be grouped into three themes: (1) develop experimental hydrologic forecasting and decision support systems that take advantage of emerging CCSP data and information; (2) pilot those systems in specific operational settings, using them in parallel with current forecasting and decision support systems; and (3) pilot use of new information in existing decision support systems.9 7   Isebrands, J.G., E.P. McDonald, E. Kruger, G. Hendrey, K. Percy, K. Pregitzer, J. Sober, and D.F. Karnosky, 2001, Growth responses of Populus tremuloides clones to interacting elevated carbon dioxide and tropospheric ozone, Environmental Pollution, 115, 359–371; Krupa, S., 2003, Atmosphere and agriculture in the new millennium, Environmental Pollution, 126, 293–300. 8   Climate Change Science Program and the Subcommittee on Global Change Research, 2003, Strategic Plan for the U.S. Climate Change Science Program, Washington, D.C., p. 50. 9   Climate Change Science Program and the Subcommittee on Global Change Research, 2003, Strategic Plan for the U.S. Climate Change Science Program, Washington, D.C., pp. 59–62.

OCR for page 63
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program Rationale. Two goals in Chapter 11 (“Decision Support Resources Development”) of the CCSP strategic plan are support for adaptive management (largely at the regional level) and support for operational decisions on climate variability and change. Developing information resources is central to each goal. Recent reports highlight the need for new information to support water resources management and other water-related decisions.10 Explosive population growth and changing climate have combined to create imbalances between water supply and demand. Background. Solutions to water shortages are limited by both inadequate institutional structures and inadequate physical facilities. As water becomes an increasingly valuable commodity, more accurate information will be needed to support estimates of the volume of natural water reservoirs (e.g., snow pack, groundwater), to understand fluxes (e.g., evapotranspiration, groundwater recharge), to perform hydrologic modeling (e.g., stream flow forecasting), and to support decision making. A program to collect such information could begin with the development of measurement networks, data management systems, and integrative tools (e.g., models) (1) to support water resources research and (2) to identify specific process studies and regions in which new information can both enhance existing decision support methods and encourage the use of emerging, experimental, decision support tools. In the long term, advances in research measurement networks, data systems, and integrative tools could bring new knowledge and technology into routine use in a variety of applications and operations. Research and interactions with decision makers are needed to address the efficacy of well-established practices and forecast methods that are based on historical data and performance. Such interaction is also necessary to determine information needs for future climate conditions that may lie outside the range of past system behavior. Thus, a sustained research effort that allows concurrent use of existing and emerging technology will be essential to demonstrate that the new tools will improve decision making. CONCLUSIONS Analysis of the case studies above and in Appendix B has revealed a number of challenges in applying and generalizing metrics. 10   U.S. Bureau of Reclamation, 2003, Water 2025: Preventing Crises and Conflict in the West, <http://www.doi.gov/water2025.pdf>.

OCR for page 63
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program TABLE 5.3 Adaptive Management of Water Resources Type Example Metrics Process • Does the CCSP have an effective planning structure, involving both agency managers and the scientific community, that is used to set priorities and implement water resource programs? • Does an adequate structure exist for peer review of both CCSP water resource programs and the research supported by those programs? • Does the CCSP support programs that effectively sustain research-applications partnerships, carry out a continuing assessment process, and provide test-beds for emerging water resource information and decision support tools? • Is the science in these planned programs responsive to the needs of regional stakeholders? • Does the CCSP water resources plan provide for the measurements, modeling, and decision support needed to link water cycle research and operational needs? Input • Annual R&D expenditures are sufficient to implement and sustain the following: —Principal investigator (PI) and/or “centers” projects directed toward achieving the objectives —Investigation of ○ Competing ideas and interpretations of causes ○ Competing interpretations of data ○ Innovative approaches for gathering or interpreting water resources data • Funds are available for the development and maintenance of a sustainable water resources scientific community of sufficient depth and diversity Output • Established (accepted, peer-reviewed, published) baselines for hydrologic forecasting improved as a result of CCSP-supported research • Consistent and reliable estimates and forecasts of water resources quantities (e.g., volume of natural water reservoirs, fluxes) to support adaptive management • Water resource planning scenarios that take into account contingencies such as substantial decreases in mountain snowpack expected as a result of further climate warming or multiyear droughts that stress water resources systems well beyond their design capacity • Accurate regional and national measures of the hydrologic effects likely associated with climate change • Quantitative information on components of the regional, national, and global water cycle that are important for water resources management, such as precipitation patterns and trends, streamflow trends, snowpack, and groundwater changes • Establishment of the degree to which these components are changing because of factors other than natural variability, such as moisture fluxes and precipitation • Sustainable information systems that make water resource data and information readily available to research and applications users

OCR for page 63
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program Type Example Metrics Outcome • Effective pilot research-applications partnerships result in experimental use of more accurate hydrologic forecasting tools and improved decision making • A regional demand exists among stakeholders for emerging CCSP data and information to support decision making • Decision support systems have been adapted to use emerging CCSP data and information • Improved information and technology have resulted in improved operational management of water resources, such as water allocations and reservoir operations • New infrastructure (e.g., groundwater backup systems for surface reservoirs) provides a more stable supply of water • More effective water resources planning structures, such as state drought task forces and agency capital investment plans, have been initiated that explicitly consider climate change Impact • Increased resilience of the water supply has decreased the vulnerability of populations to hydrologic aspects of climate variability and change Challenges in the Application of Metrics 1. Several case studies revealed that a number of metrics are not amendable to numerical scores and require qualitative assessments. In general, the higher-order themes (e.g., theme 6—improving the assessment and management of risk) and higher-order measures (e.g., outcome and impact) appear to be more amenable to qualitative assessments based on peer review and stakeholder analysis. However, numerical scores may also be difficult to apply to lower-order themes such as data collection and analysis. For example, the design of a sampling scheme (necessary accuracy, precision, sampling scale) for climatic forcing factors and analysis of the results requires expert judgment (a) to avoid aliasing (the inevitable tendency of high-frequency components to appear to the observer as erroneous lower-frequency components or even space-time mean values if sampling criteria are not met) or (b) to recognize valuable uses of data, even when data accuracy proves to be less than the a priori measurement requirements. The case study of paleoclimate time series showed that no simple quantitative metric, such as the number of cores examined, would establish a milestone in understanding. Sometimes one core is sufficient to transform our views of ancient climate history, and additional observations may prove useful only for establishing generality. The goal should not be to increase the number of records but to improve their quality and usefulness for interpreting past climate. Achieving this goal requires expert judgment.

OCR for page 63
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program 2. The CCSP plan is characterized by a large number of milestones and products, many of which are dependent on reaching other milestones. For example, metrics associated with the assessment and management of risk related to public health threats (theme 7) depend on scientific advances in a number of areas, including climate prediction, the linkages between environment and health, and the fundamental ecology of infectious diseases. Reaching these interdependent milestones could (a) involve different (and multiple) leaders, perhaps serving agencies with different missions and success criteria; (b) reflect different capabilities in observing and modeling; and (c) involve different assessments of the level of understanding, depending on whether the milestone involves social, biological, or atmospheric sciences. The difficulty of integrating all of these dependencies presents a significant challenge to producing and implementing useful metrics. 3. The case studies revealed concerns about whether a weak score on one element could dominate the evaluation of an otherwise strong program. Low scores generally indicate where work is needed, and high scores indicate substantial progress or successes. However, a high score does not mean that improvements cannot be made. Similarly, not every low score indicates problems. Low scores on outcome and impact measures may reveal that the science is still in the discovery phase, not that the program element is a failure. Moreover, a low score on lower-order metrics may not preclude a high score on higher-order metrics. For example, an output metric in theme 2 (improved accuracy of measuring sea level) might receive a low score, but the observations might inform a more important outcome metric (e.g., measurements of sufficient accuracy to inform assessments and policy) or lead to unanticipated outcomes. In many areas of interest, the success of a program can be evaluated only in the context of the sometimes myriad uses to which it can be put. 4. The evolution of knowledge can have a cascading effect on metrics. For example, theme 3 (understanding processes) measures frequently include an element of testing predictions against measurable quantities and periodic assessment of forecast ability. These, in turn, require involvement of the scientific community in activities designed to ensure that measurements made at many different locations and times are suitable for testing understanding of process models. Increased understanding may lead to new requirements that exceed the original data collection requirements. Additional progress may therefore depend on additional coordination of measurement networks and cooperation by the scientific community. 5. Because of the importance of “reducing uncertainties” within the CCSP strategic plan, numerous case studies examined potential metrics associated with complexity and uncertainty. Because complexity will remain or increase even as knowledge advances, scores for output and outcome metrics may not improve significantly with time. Similarly, the number of

OCR for page 63
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program uncertainties associated with climate change will likely be reduced through research and expanding knowledge. However, many of these will be replaced with new uncertainties, preventing metric scores from improving. Consequently, it is necessary to develop decision structures that can assess evolving uncertainties. 6. Impact metrics may prove difficult to quantify on a routine basis. First, the time scales associated with assessing impact often exceed the time frame for policy decisions. National policy making about a climate response also necessarily takes place in a political context. Thus, the salience of particular issues and the timeliness of different assessment methods and case analyses may depend on year-to-year national and international events. This circumstance places a premium on the development of a wide portfolio of analysis tools and applications. However, it is difficult to imagine an impact metric for performance of such a portfolio. 7. The importance of peer review, expert opinion, and stakeholder judgments was noted in the majority of case studies. 8. It may be a significant challenge to evaluate priorities between different measurement areas that involve the needs of different disciplines. Ability to Generalize Process and input metrics appear to be easily generalized among case study topics. Higher-order measures (output, outcome, and impact) appear to be more specific both to the tasks required to achieve the overall objective and to the unique features of the specific domains. As such, they do not appear to be directly transferable. However, even these measures can be generalized if the overall objectives are taken into consideration, such as improved knowledge of processes, improved forecasting capability, improved understanding of uncertainties and limitations, and improved management. The key to developing generalized metrics is the level of aggregation. For example, modeling needs for the assessment of broad-scale, long-term global environmental agreements are very different from those required for policies on specific technologies, such as pumping CO2 into depleted oil fields. It may be possible, however, to develop process metrics to assess, at a higher level of aggregation, goals concerned with the establishment of a capacity to carry out policy-relevant integrated studies, the development of a portfolio of different capabilities, or the degree to which various studies are useful for policy making, without addressing a specific policy.

OCR for page 63
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program This page intentionally left blank.