Click for next page ( 18


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 17
2 Process for Evaluating Progress: Task 1a T he first part of the committee’s charge (task 1a) was to develop a pro- cess for evaluating progress in the Climate Change Science Program (CCSP). The objective was to design an evaluation that (1) would encompass the major components of the program at a sufficient level of detail to enable program managers to make any necessary adjustments, and (2) would be practical for the CCSP to implement. A number of approaches have been proposed, ranging from the comprehensive evaluation frame- work laid out in Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program (NRC, 2005) to simply tracking completion of CCSP products. This chapter examines the strengths and limitations of these approaches and recommends a methodology to evalu- ate the progress of the CCSP. The committee’s preliminary assessment (task 1b), based on this methodology, is summarized in Chapter 3 and presented in more detail in Part II. WHAT CAN BE EVALUATED A key step in designing any evaluation is to divide the program into meaningful pieces against which progress can be measured (NRC, 2005). Too coarse a division of the program will capture many disparate elements and will both be difficult to evaluate and yield ambiguous results. Too fine a division will be costly and time consuming to evaluate, and the evaluation results may not be useful to program managers. Because progress has to be assessed in the context of applied resources, the pieces being analyzed 1

OCR for page 17
1 EVALUATING PROGRESS OF THE U.S. CCSP should have associated budgets. Some natural divisions in the CCSP and the information available for the evaluation are described below. Organizing the Evaluation The CCSP has three major components: overarching goals, research elements, and cross-cutting issues (Figure 2.1). Any of these could serve as themes for organizing the evaluation, and all have advantages and disad- vantages. The overarching goals (e.g., reduction of uncertainty) represent what the program is trying to achieve, but they are broad, some of them overlap (especially goals 1, 2, and 3), and progress in one goal usually de- pends on progress in another goal. The research elements (e.g., water cycle, ecosystems) represent a research agenda agreed on by multiple agencies with a time line for the delivery of specified milestones and products (Table 1.1). As such, they represent the strongest connection to the programs of partici- pating agencies, where climate research activities are funded and managed. However, they offer only limited insight on the more applied aspects of the program. Finally, the cross-cutting issues (e.g., modeling, observations and monitoring) cover the types of activities the program supports, but their breadth makes them difficult to evaluate. For example, observation goal 1 is “design, develop, deploy, integrate, and sustain observation components into a comprehensive system” (CCSP, 2003), which applies to some extent to the entire program. Overarching Goals (5) Major Components Research Elements (7) Cross-cutting Issues (6) Research Focus Areas Subcomponents Goals (22) Questions (21) (33) FIGURE 2.1 Hierarchical structure of CCSP components. The focus areas of the overarching goals are linked with both the research questions and the cross-cutting issues. The CCSP does not report budgets for boxes shaded in gray. fig 2-1

OCR for page 17
1 PROCESS FOR EVALUATING PROGRESS: TASK 1A Each of the major components of the program has a subcomponent: the overarching goals are divided into 21 focus areas, the research elements are divided into 33 research questions, and the cross-cutting issues have 22 goals (Figure 2.1). Although the program can be analyzed at both hierar- chical levels, the committee generally found it most fruitful to evaluate the program at the subcomponent level. Information for the Evaluation The information required to carry out a credible evaluation includes a list of program activities and results and the amount of funding devoted to these activities. None of these are available for the CCSP in useful forms. The activities included in the program are designated by the participating agencies and vary from year to year (GAO, 2006). Results are reported in many places, including the CCSP web site and publications and the scien- tific literature, but they are not linked directly to the major components or subcomponents of the program. The most unambiguous source of informa- tion on program progress is found in the CCSP’s annual report to Congress, Our Changing Planet, which provides selected examples of progress and plans each year. Our Changing Planet also tallies agency budgets into some CCSP categories (overarching goals, focus areas, and research elements; see Figure 2.1), but this is primarily an accounting exercise, rather than a true allocation of funding to achieve CCSP objectives. Indeed, in response to a questionnaire prepared by the committee, agency managers had difficulty matching their programs to the CCSP overarching goals.1 Uncertainties about what activities are included in the program and how much they cost makes an in-depth evaluation of program progress difficult, if not impossible, for an external review committee. However, the available information is sufficient in most cases to test different approaches to evaluating progress and to draw some high-level conclusions about CCSP progress to date. The one case in which additional information was needed to even test evaluation approaches concerns the human contributions and responses re- search element. Although a research program on human contributions and responses is outlined in the CCSP strategic plan, the CCSP now manages it with the decision support resources cross-cutting issue (see CCSP, 2005). Consequently, it is no longer clear what research activities on human contri- 1 Presentation to the committee on June 15, 2006, by Don Anderson (NASA, goal 1), Jay Fein (NSF, goal 1), Phil DeCola (NASA, goal 2), Roger Dahlman (DOE, goal 2), Bill Hohen- stein (USDA, goal 5), and Jerry Elwood (DOE, goal 5). Written responses to a committee questionnaire were also provided by Anjuli Bamzai (DOE, goal 3), Chet Koblinski (NOAA, goal 3), and DeWayne Cecil (NASA, goal 4).

OCR for page 17
20 EVALUATING PROGRESS OF THE U.S. CCSP butions and responses are supported by the CCSP. In response to a commit- tee request, the CCSP Interagency Working Group on Human Contributions and Responses/Decision Support Resources sent out a questionnaire to eight participating agencies. The resulting list of programs and budgets is given in Appendix B and was used in the committee’s evaluation. This inquiry showed that few agency programs are aimed explicitly at human contributions and responses research, so detailed estimates of expenditures could not be generated. Relevant research may or may not be counted as CCSP, and some research that is clearly peripheral to research element objectives is included in the program accounts. For example, the National Institutes of Health (NIH) program on health effects of strato- spheric ozone constitutes more than two-thirds of the reported human contributions and responses budget, yet it is only tangentially concerned with climate change or social science research. Another large fraction of the funding goes to decision support activities, most of which lack a human dimensions research component (see Chapter 5). Including such programs paints a distorted picture of CCSP human contributions and responses re- search. Funding for human dimensions research is likely on the order of $25 million to $30 million per year, excluding NIH research on the health ef- fects of ozone and National Aeronautics and Space Administration (NASA) decision support activities (Appendix B). APPLICATION OF THE NRC (2005) EVALUATION FRAMEWORK The National Research Council (NRC, 2005) report Thinking Strategi- cally: The Appropriate Use of Metrics for the Climate Change Science Pro- gram lays out a comprehensive framework for evaluating the progress of the CCSP. The report identified five categories of metrics that could be used to measure progress and guide strategic thinking across the entire CCSP: 1. Process metrics: measure a course of action taken to achieve a goal 2. Input metrics: measure tangible quantities put into a process to achieve a goal 3. Output metrics: measure the products and services delivered 4. Outcome metrics: measure results that stem from use of the outputs and influence stakeholders outside the program 5. Impact metrics: measure the long-term societal, economic, or envi- ronmental consequences of an outcome Specific metrics within these categories are listed in Table 2.1, and key conclusions from the report are summarized in Box 2.1. Both CCSP manag- ers and the committee have tried to apply the metrics to a major component

OCR for page 17
21 PROCESS FOR EVALUATING PROGRESS: TASK 1A BOX 2.1 Key Conclusions from Thinking Strategically Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Sci- ence Program (NRC, 2005) found that progress can be assessed for most aspects of the CCSP, from enhancement of data networks to improved public awareness of climate change issues. The key to promoting progress is to consider the program from end to end, starting with program processes (e.g., planning and peer review) and inputs (e.g., resources) and extending to outputs (e.g., assessments, fore- casts), outcomes (e.g., near-term results for science and society), and long-term impacts. Metrics for evaluating all of these stages are given in Table 2.1. Of these, the most appropriate will be the subset that enables managers to identify and monitor program strengths and weaknesses. These measures will become appar- ent from even rough scores or yes-no answers to the metrics. Detailed analysis and tracking can then be focused on the parts of the program for which better results are desired. The process and input metrics provide clues about why scores on program results (outputs, outcomes, and impacts) might be low. For example, a project may not have succeeded because it lacked a leader with authority to direct sufficient resources to the effort. As the agencies gain experience, this subset of metrics will be refined until only the most useful remain. of the program (overarching goals and research elements, respectively), as described below. Overarching Goals In 2006, a few CCSP managers tried applying the NRC (2005) metrics to the five CCSP overarching goals. Their rough evaluation found that nearly all of the metrics were relevant to the CCSP and that quantitative scores could be assigned with a reasonably high level of confidence for most metrics.2 Scoring was most difficult for metrics dealing with impacts, com- munication of results, and use of results by stakeholders. CCSP managers have neither pursued this process nor made other ef- forts to use the NRC metrics. Resources are scarce, and the agencies are seeking a cost-effective, practical approach to assessing progress (see task 1a, Box 1.1). Consequently, they are weighing the extent to which they should adopt the type of broad, strategic evaluation framework recom- mended by the NRC (2005) against developing CCSP-wide and/or agency- specific metrics related to the CCSP (see “CCSP Approaches” below). 2 Presentation to the committee by Jack Kaye, NASA, on April 28, 2006.

OCR for page 17
22 EVALUATING PROGRESS OF THE U.S. CCSP Research Elements The committee tested the NRC (2005) evaluation framework on the research elements, the component of the program that it knows best. Two contrasting research elements were chosen for the trial: one well established (carbon cycle) and one emerging (human contributions and responses). The evaluation was conducted only at the top hierarchical level (e.g., carbon cycle) because the committee lacked the budget and management informa- tion to score the process and input metrics at subcomponent level (i.e., research questions). As recommended in NRC (2005), scores to the metrics were assigned using only the committee’s knowledge of the research elements, and pro- grammatic and budget information provided in CCSP reports and by the relevant interagency working groups (IWGs). This exercise enabled the committee to gain familiarity with the concepts presented in NRC (2005) and to see how the evaluation framework might be implemented. The re- sults, which are presented for illustration purposes only, appear in Table 2.1. Definitive conclusions about progress in these research elements will require additional programmatic information and peer review. The committee found that this type of preliminary analysis is useful for identifying strengths and weaknesses within a particular research ele- ment and for comparing research elements to one another. For example, the carbon cycle research element is comparatively well funded and has a long history of strategic planning and science community involvement. Its primary outcomes are scientific advances. Other stakeholders have had little involvement to date, and nonscientific types of societal benefits (e.g., carbon management) are only beginning to be realized. In contrast, the hu- man contributions and responses research element has had relatively little funding, multiagency coordination, or science community participation. However, nonscientist stakeholders are more engaged in selected agency initiatives, and as a result the program has had some successes in informing resource management and decision making, as well as in advancing science. Analyses of the differences among research elements would enable more strategic decisions to be made about where additional investments might best accelerate progress. Overall, the committee found the evaluation framework laid out in NRC (2005) to be a viable method for assessing progress and making stra- tegic decisions about the CCSP. However, available information on program planning and resource allocation was insufficient for a rigorous evaluation of the process and input metrics. Consequently, the committee sought an alternative method that would be based on readily available information for its preliminary assessment of CCSP progress.

OCR for page 17
2 PROCESS FOR EVALUATING PROGRESS: TASK 1A EVALUATING PROGRESS BASED ON PROGRAM RESULTS Evaluations of progress commonly focus on program results, such as publications and services delivered. The CCSP is considering using annual evaluations of agency-specific performance metrics, which focus on out- comes, and tracking completion of CCSP products to assess progress. The committee developed a “matrix” evaluation approach that focuses only on program results. Program results are relatively well known to external reviewers, so a peer review evaluation, which is important for assessing quality (NRC, 2005), is possible. The agency and committee approaches are described below. CCSP Approaches Agency-Specific Metrics The 1993 Government Performance and Results Act requires federal government agencies to set strategic goals and to measure performance against them on an annual basis.3 Every federal agency has metrics to mea- sure performance, and CCSP program managers are trying to determine whether the metrics of participating agencies can also be used to assess CCSP progress as a whole.4 However, the aggregate of agency metrics reflects neither the level of agency involvement in CCSP programs nor the breadth of the CCSP, a conclusion also reached in NRC (2005). For exam- ple, five agencies (Department of Energy [DOE], NASA, National Oceanic and Atmospheric Administration [NOAA], National Science Foundation, and U.S. Geological Survey) have research projects and initiatives to ad- dress CCSP overarching goal 3: reduce uncertainty in projections of how the Earth’s climate and related systems may change in the future (CCSP, 2006a). Yet, only three of these agencies have metrics related to this over- arching goal (Table 2.2), and these miss key aspects of climate predictions (e.g., droughts) and interagency contributions (e.g., an integrated Earth system analysis capability). Some of NOAA’s goals are aligned with CCSP goals, but the other agencies are committed to several different objectives that may or may not have a specific climate component. The mismatch between CCSP and agency goals suggests that it may be difficult to make more than limited progress in the program overall. 3 PublicLaw 103-62. 4 Presentation to the committee by Jack Kaye, NASA, on April 28, 2006, and by Mary Glackin, NOAA, on September 15, 2006.

OCR for page 17
TABLE 2.1 Application of NRC (2005) Metrics to Two CCSP Research Elements 2 Metric Carbon Cycle Human Contributions and Responses Process Metrics 1. Leader with sufficient authority to Leadership exists in IWG members, who The IWG has been expanded to cover decision allocate resources, direct research effort, have agency budget authority support tools, which makes it difficult to assess the and facilitate progress effectiveness of leadership on human contributions and responses 2. A multiyear plan that includes goals, CCSP strategic plan as well as North CCSP strategic plan contains good research focused statement of task, implementation, American Carbon Program and Ocean questions, but little information on discovery, applications, and integration Carbon and Climate Change plans have implementation. A science community plan with scientific community input and review common research goals also exists under the International Human Dimensions Programme on Global Environmental Change, but its connection with CCSP is unclear 3. A functioning peer review process Program peer review through the IWG’s Peer review takes place within agency programs, in place involving all appropriate science steering group but there is no peer review process for the research stakeholders, with (a) underlying processes element and timetables, (b) assessment of progress toward achieving program goals, and (c) an ability to revisit the plan in light of new advances 4. A strategy for setting priorities and Science steering group is one input for Much of the supported research appears to be allocating resources among different priority setting motivated by natural science research needs, rather elements of the program (including than by a prioritization of what questions need to those that cross agencies) and advancing be addressed to solve problems promising avenues of research and applications

OCR for page 17
5. Procedures in place that enable or Minimal development of procedures to Little effort to communicate results beyond a facilitate the use or understanding of the facilitate communication or partnerships; few individual projects (e.g., Knowledge Systems, results by others (e.g., scientists in other workshop on the state of carbon cycle Decision Making Under Uncertainty [DMUU], disciplines, operational users, decision research attempted to identify and engage Regional Integrated Sciences and Assessments makers) and promote partnerships stakeholders on carbon management [RISAs]) Input Metrics 1. Sufficient intellectual and technologic Science questions are mature; a research Intellectual, technological, and financial support foundation to support the research pool actively pursuing these questions and are insufficient to build the research community technological tools exist; progress is not and carry out research on CCSP goals primarily limited by technology 2. Sufficient commitment of resources (i.e., Implementing the complete carbon science Resources are inadequate to carry out the stated people, infrastructure, financial) directed plan requires more resources than currently research program specifically to allow the planned program to available be carried out 3. Sufficient resources to implement and Opportunities for basic research cut across With the exception of a few focused programs, sustain each of the following: (a) research agencies; new approaches have been research is ad hoc and scattered across enabling unanticipated scientific discovery, identified within the current structure the agencies, making it difficult to build (b) investigation of competing ideas and comprehensive approaches interpretations, and (c) development of innovative and comprehensive approaches Continued 25

OCR for page 17
TABLE 2.1 Continued 2 Metric Carbon Cycle Human Contributions and Responses 4. Sufficient resources to promote the Funds largely used for basic science, No apparent coordination of resources to build development and maintenance of each observations, and modeling rather than for capacity in the human dimensions or to facilitate of the following: (a) human capital; (b) developing stakeholder linkages the transfer of knowledge to stakeholders measurement systems, predictive models, and synthesis and interpretive activities; (c) transition to operational activities where warranted; and (d) services that enable the use of data and information by relevant stakeholders 5. The program takes advantage of existing Syntheses of existing and historical data Little evidence that existing social and economic resources (e.g., U.S. and foreign historical sets have been used to reconstruct forest databases have been used by the CCSP, or data records, infrastructure) inventory and land cover changes. Strong that the CCSP is emphasizing the creation of links exist to international carbon cycle new databases necessary to advance human research efforts contributions and responses research Output Metrics 1. The program produces peer-reviewed Large number of peer-reviewed publications A number of successful initiatives have been and broadly accessible results, such as (a) and data products for all except (d) undertaken, but they appear to be insufficient to data and information, (b) quantification of address the research element important phenomena or processes, (c) new and applicable measurement techniques, (d) scenarios and decision support tools, and (e) well-described and demonstrated relationships aimed at improving understanding of processes or enabling forecasting and prediction

OCR for page 17
2. An adequate community and/or Research community exists; infrastructure The research community is small and may be infrastructure to support the program has includes observation networks and insufficient to meet the requirements of the been developed modeling. However, planned expansion of research element networks has not materialized 3. Appropriate stakeholders judge these Work benefits and is benefited by research Little documented interaction with stakeholders, results to be sufficient to address scientific in other science fields (e.g., land use with possible exceptions such as the RISAs and the questions and/or to inform management and land cover, ocean acidification and International Research Institute for Climate and and policy decisions ecosystems). Links to carbon management Society (IRI) communities are weaker—best links are with U.S. Department of Agriculture’s Agricultural Research Service and Forest Service inventory, but less interaction with carbon emissions scenario building (carbon markets) 4. Synthesis and assessment products CCSP synthesis and assessment product 2.2 Synthesis and assessment products are focused are created that incorporate these new is under review (more or less on schedule; mostly on decision support developments see Appendix A) 5. Research results are communicated to an Some uncoordinated efforts exist to link A few programs (e.g., RISAs, DMUU centers) have appropriate range of stakeholders land management to carbon and net climate created structures for communication with and effects transfer of knowledge to stakeholders Outcome Metrics 1. The research has engendered significant New science questions (e.g., role of fire or Program initiatives have pointed toward new new avenues of discovery disturbance, ocean acidification) have been avenues of discovery, such as characterizing engendered by research irreducible uncertainties about climate change and its impacts, assessing vulnerabilities, and understanding the role of institutions in climate- related decision making Continued 2

OCR for page 17
2 TABLE 2.1 Continued Metric Carbon Cycle Human Contributions and Responses 2. The program has led to the identification Scientists still cannot balance the global Program initiatives (e.g., DMUU centers) are of uncertainties, increased understanding of carbon budget, although uncertainties in beginning to lead to increased understanding uncertainties, or reduced uncertainties that some areas have narrowed. Feedbacks of uncertainties and development of decision support decision making or facilitate the remain important areas of research strategies and tools that incorporate uncertainties, advance of other areas of science (coupled modeling advances) which could inform decision making at several scales and sectors 3. The program has yielded improved Coupled climate-carbon models have been Investment in scenarios and economic modeling understanding, such as (a) more developed, although they are in the early has yielded limited knowledge useful for informing consistent and reliable predictions or stages policy making forecasts, (b) increased confidence in our ability to simulate and predict climate change and variability, and (c) broadly accepted conclusions about key issues or relationships 4. Research results have been transitioned Carbon management is in the early stages Some valuable outputs have been transferred to to operational use of operations users (e.g., by RISAs and IRI) in areas such as drought response, fire management, and fisheries management 5. Institutions and human capacity have Links to carbon markets are weak; no Little human capacity has been created been created that can better address a range institutions have been established of related problems and issues

OCR for page 17
6. The measurements, analysis, and results Inadequate progress beyond basic science Activities to achieve these outcomes have taken are being used (a) to answer the high- place on a very limited scale priority climate questions that motivated them, (b) to address objectives outside the program plan, or (c) to support beneficial applications and decision making, such as forecasting, cost-benefit analysis, or improved assessment and management of risk Impact Metrics 1. The results of the program have Science advances are informing assessments Some programs (e.g., RISAs) appear to have informed policy and improved decision of the Intergovernmental Panel on Climate informed policy making, although not to the making Change extent envisioned in the CCSP strategic plan 2. The program has benefited society in Carbon management is being addressed at Positive societal benefit, especially if measured terms of enhancing economic vitality, the state or regional level, if at all against the level of investment promoting environmental stewardship, protecting life and property, and reducing vulnerability to the impacts of climate change 3. Public understanding of climate issues Improved public understanding of the links Insufficient data to measure the effect of initiatives has increased to increase public understanding of climate between CO2 and warming 2

OCR for page 17
0 EVALUATING PROGRESS OF THE U.S. CCSP TABLE 2.2 Metrics of Participating Agencies Related to CCSP Overarching Goal 3 Agency Multiyear Performance Goals and Annual Performance Measures NOAA Understand climate variability and change to enhance society’s ability to plan and respond. Annual performance measures: • U.S. temperature forecasts (cumulative skill score computed over the regions where predictions are made) • Reduce uncertainty in the magnitude of the North American carbon uptake • Reduce uncertainty in model simulations of the influence of aerosols on climate • Determine the national explained variance (%) for temperature and precipitation for the contiguous United States using U.S. Climate Reference Network stations DOE Deliver improved climate data and models for policy makers to determine safe levels of greenhouse gases for the Earth’s system. By 2013, substantially reduce differences between observed temperature and model simulations at subcontinental scales using several decades of recent data. Annual performance measure: • Improve climate models: Produce a new continuous time series of retrieved cloud properties at each Atmospheric Radiation Measurement site and evaluate the extent of agreement between climate model simulations of water vapor concentration and cloud properties and measurements of these quantities on time scales of 1 to 4 days NASA Progress in understanding and improving predictive capability for changes in the ozone layer, climate forcing, and air quality associated with changes in atmospheric composition. Annual performance measure: • Demonstrate that NASA-developed data sets, technologies, and models enhance understanding of the Earth system, leading to improved predictive capability in each of the six science focus area road maps Progress in quantifying global land cover change and terrestrial and marine productivity, and in improving carbon cycle and ecosystem models. Annual performance measure: • Demonstrate that NASA-developed data sets, technologies, and models enhance understanding of the Earth system, leading to improved predictive capability in each of the six science focus area road maps Progress in quantifying the key reservoirs and fluxes in the global water cycle and in improving models of water cycle change and freshwater availability. Annual performance measures: • Demonstrate that NASA-developed data sets, technologies, and models enhance understanding of the Earth system, leading to improved predictive capability in each of the six science focus area road maps • Complete Global Precipitation Measurement confirmation review

OCR for page 17
1 PROCESS FOR EVALUATING PROGRESS: TASK 1A TABLE 2.2 Continued Agency Multiyear Performance Goals and Annual Performance Measures NASA Progress in understanding the role of oceans, atmosphere, and ice in (continued) the climate system and in improving predictive capability for its future evolution. Annual performance measures: • Demonstrate that NASA-developed data sets, technologies, and models enhance understanding of the Earth system, leading to improved predictive capability in each of the six science focus area road maps • Complete Operational Readiness Review for the NPOESS Preparatory Project SOURCE: DOC, 2006; DOE, 2006; NASA, 2006; . Product Status Reports Tracking the status of products is the simplest way to monitor one kind of program result. The CCSP is tracking the 21 synthesis and assessment products associated with the overarching goals, and participating agencies are tracking the 208 milestones, products, and payoffs associated with the research elements. Reporting whether each milestone, product, or payoff is completed; on or behind schedule; or discontinued responds directly to Office of Management and Budget directions in the Fiscal Year 2006 federal budget plans (OMB, 2005). The status of synthesis and assessment products is also monitored and the products are evaluated for quality, which makes the metric more use- ful. All synthesis and assessment products undergo three stages of develop- ment: (1) development of a prospectus, (2) preparation and revision of the product, and (3) final approval by the National Science and Technology Council and publication. Product quality is evaluated via peer review and public comment during development of the prospectus, during drafting of the product, and at the request of lead authors, just before final approval. Changes in the status of the products are announced through community e-mails, the Federal Register, and web site updates. Whether or not a planned product was created on time is a useful met- ric. However, use of this measure alone would suggest that progress of the CCSP is inadequate, given that only two synthesis and assessment product have been completed in the scheduled time (Appendix A). Committee’s Matrix Approach The committee developed a new evaluation approach, which uses a matrix to evaluate the 33 research questions in the research elements (rows

OCR for page 17
2 EVALUATING PROGRESS OF THE U.S. CCSP of the matrix) against outputs and outcomes (columns of the matrix). The columns of the matrix were derived from five categories of CCSP prod- ucts and results, themselves generalized from categories identified in NRC (2005): Category A: improve data sets in space and time, and improve estimates of physical quantities Category B: improve understanding and representation of processes Category C: improve predictability, predictive capabilities, or assess- ment of uncertainty Category D: improve synthesis and assessment to inform Category E: improve assessment and management of risk, and improve decision support for management and policy making These product categories provide a measure of the maturity of the program, starting from data collection, to predictions of future climate changes, to improved use of information to better serve society. The com- plete matrix is given in Appendix C. The research questions were chosen for evaluation because nearly all of the CCSP’s milestones, products, and payoffs are associated with them. Moreover, they are linked to the focus areas of the overarching goals (Fig- ure 2.1). The product categories overlap with the cross-cutting issues. In particular, category A includes observations and monitoring, category C includes modeling, category D includes communication, and category E includes decision support. Consequently, the matrix enables a reasonably broad assessment of CCSP progress. The committee used the matrix to make a preliminary assessment of CCSP progress (task 1b). Scores for the rows of the matrix provide infor- mation on progress in the research questions, research elements, and over- arching goals (see Chapters 3 and 4). Scores for the columns of the matrix provide a measure of the maturity of the research element as well as an indication of progress in the cross-cutting issues (see Chapters 3 and 5). CONCLUSIONS Recommendation. CCSP progress should be evaluated in two stages: (1) a broad overview of the entire program based on the knowledge of the reviewers, and (2) a more in-depth analysis of areas in which progress has been inadequate, using the process and input metrics from NRC (2005). Agency approaches to monitoring progress (i.e., compiling agency met- rics, tracking milestones and products) are useful, but open only a narrow

OCR for page 17
 PROCESS FOR EVALUATING PROGRESS: TASK 1A window onto the program. More information about the program as a whole can be obtained from the matrix approach developed by the com- mittee. Overall, the committee found that the matrix can be used to assess progress toward the research elements and overarching goals, although it is unwieldy to evaluate the goals in this manner. The matrix also provides information on progress in the cross-cutting issues, to the extent that these issues overlap with the product categories (i.e., the columns of the matrix). However, this method provides little insight on why good progress has or has not been made. Diagnosing the causes of strengths and weaknesses requires knowledge of planning, leadership, and resources in addition to program results. The process and input metrics from the NRC (2005) framework offer a means to evaluate these issues. An evaluation of CCSP progress using either the committee’s matrix or the NRC (2005) evaluation framework can be daunting. The matrix contains 165 cells, but more than 165 scores are necessary since many of the research questions cover a complex set of issues. A similar number of scores would be required to evaluate the major components of the program (five overarching goals, seven research elements, and six cross-cutting is- sues) using the 24 metrics identified in NRC (2005). Evaluating the research questions by this method would require nearly 800 scores. The evaluation process can be made more practical by breaking it into stages, with the initial evaluation aimed at identifying successes and finding weaknesses. Preliminary scores can be assigned using only the knowledge of the reviewers and information on programs and results in readily avail- able publications. A community workshop, such as the one organized by the committee, is one way to obtain the breadth of knowledge required to carry out the first stage. A stage 1 evaluation of the entire program might be necessary only when there is a major change in the program, such as a new strategic plan. The second stage of evaluation can focus exclusively on areas identified in the first stage as not meeting program objectives and expectations. These areas would be evaluated with the process and input metrics from NRC (2005), which provide tools for diagnosing the reasons for the weakness. Detailed programmatic and budgetary information would be required to carry out this stage of evaluation, which should continue until outcomes improve.

OCR for page 17