PAPERBACK
$38.25
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 17
2
Process for Evaluating Progress: Task 1a
T
he first part of the committee’s charge (task 1a) was to develop a pro-
cess for evaluating progress in the Climate Change Science Program
(CCSP). The objective was to design an evaluation that (1) would
encompass the major components of the program at a sufficient level of
detail to enable program managers to make any necessary adjustments, and
(2) would be practical for the CCSP to implement. A number of approaches
have been proposed, ranging from the comprehensive evaluation frame-
work laid out in Thinking Strategically: The Appropriate Use of Metrics
for the Climate Change Science Program (NRC, 2005) to simply tracking
completion of CCSP products. This chapter examines the strengths and
limitations of these approaches and recommends a methodology to evalu-
ate the progress of the CCSP. The committee’s preliminary assessment (task
1b), based on this methodology, is summarized in Chapter 3 and presented
in more detail in Part II.
WHAT CAN BE EVALUATED
A key step in designing any evaluation is to divide the program into
meaningful pieces against which progress can be measured (NRC, 2005).
Too coarse a division of the program will capture many disparate elements
and will both be difficult to evaluate and yield ambiguous results. Too fine
a division will be costly and time consuming to evaluate, and the evaluation
results may not be useful to program managers. Because progress has to
be assessed in the context of applied resources, the pieces being analyzed
1
OCR for page 17
1 EVALUATING PROGRESS OF THE U.S. CCSP
should have associated budgets. Some natural divisions in the CCSP and the
information available for the evaluation are described below.
Organizing the Evaluation
The CCSP has three major components: overarching goals, research
elements, and cross-cutting issues (Figure 2.1). Any of these could serve as
themes for organizing the evaluation, and all have advantages and disad-
vantages. The overarching goals (e.g., reduction of uncertainty) represent
what the program is trying to achieve, but they are broad, some of them
overlap (especially goals 1, 2, and 3), and progress in one goal usually de-
pends on progress in another goal. The research elements (e.g., water cycle,
ecosystems) represent a research agenda agreed on by multiple agencies with
a time line for the delivery of specified milestones and products (Table 1.1).
As such, they represent the strongest connection to the programs of partici-
pating agencies, where climate research activities are funded and managed.
However, they offer only limited insight on the more applied aspects of the
program. Finally, the cross-cutting issues (e.g., modeling, observations and
monitoring) cover the types of activities the program supports, but their
breadth makes them difficult to evaluate. For example, observation goal 1
is “design, develop, deploy, integrate, and sustain observation components
into a comprehensive system” (CCSP, 2003), which applies to some extent
to the entire program.
Overarching Goals (5)
Major
Components
Research Elements (7) Cross-cutting Issues (6)
Research Focus Areas Subcomponents
Goals (22)
Questions (21)
(33)
FIGURE 2.1 Hierarchical structure of CCSP components. The focus areas of the
overarching goals are linked with both the research questions and the cross-cutting
issues. The CCSP does not report budgets for boxes shaded in gray.
fig 2-1
OCR for page 17
1
PROCESS FOR EVALUATING PROGRESS: TASK 1A
Each of the major components of the program has a subcomponent:
the overarching goals are divided into 21 focus areas, the research elements
are divided into 33 research questions, and the cross-cutting issues have 22
goals (Figure 2.1). Although the program can be analyzed at both hierar-
chical levels, the committee generally found it most fruitful to evaluate the
program at the subcomponent level.
Information for the Evaluation
The information required to carry out a credible evaluation includes a
list of program activities and results and the amount of funding devoted to
these activities. None of these are available for the CCSP in useful forms.
The activities included in the program are designated by the participating
agencies and vary from year to year (GAO, 2006). Results are reported in
many places, including the CCSP web site and publications and the scien-
tific literature, but they are not linked directly to the major components or
subcomponents of the program. The most unambiguous source of informa-
tion on program progress is found in the CCSP’s annual report to Congress,
Our Changing Planet, which provides selected examples of progress and
plans each year. Our Changing Planet also tallies agency budgets into some
CCSP categories (overarching goals, focus areas, and research elements; see
Figure 2.1), but this is primarily an accounting exercise, rather than a true
allocation of funding to achieve CCSP objectives. Indeed, in response to a
questionnaire prepared by the committee, agency managers had difficulty
matching their programs to the CCSP overarching goals.1
Uncertainties about what activities are included in the program and
how much they cost makes an in-depth evaluation of program progress
difficult, if not impossible, for an external review committee. However, the
available information is sufficient in most cases to test different approaches
to evaluating progress and to draw some high-level conclusions about CCSP
progress to date.
The one case in which additional information was needed to even test
evaluation approaches concerns the human contributions and responses re-
search element. Although a research program on human contributions and
responses is outlined in the CCSP strategic plan, the CCSP now manages
it with the decision support resources cross-cutting issue (see CCSP, 2005).
Consequently, it is no longer clear what research activities on human contri-
1 Presentation to the committee on June 15, 2006, by Don Anderson (NASA, goal 1), Jay
Fein (NSF, goal 1), Phil DeCola (NASA, goal 2), Roger Dahlman (DOE, goal 2), Bill Hohen-
stein (USDA, goal 5), and Jerry Elwood (DOE, goal 5). Written responses to a committee
questionnaire were also provided by Anjuli Bamzai (DOE, goal 3), Chet Koblinski (NOAA,
goal 3), and DeWayne Cecil (NASA, goal 4).
OCR for page 17
20 EVALUATING PROGRESS OF THE U.S. CCSP
butions and responses are supported by the CCSP. In response to a commit-
tee request, the CCSP Interagency Working Group on Human Contributions
and Responses/Decision Support Resources sent out a questionnaire to eight
participating agencies. The resulting list of programs and budgets is given
in Appendix B and was used in the committee’s evaluation.
This inquiry showed that few agency programs are aimed explicitly
at human contributions and responses research, so detailed estimates of
expenditures could not be generated. Relevant research may or may not be
counted as CCSP, and some research that is clearly peripheral to research
element objectives is included in the program accounts. For example, the
National Institutes of Health (NIH) program on health effects of strato-
spheric ozone constitutes more than two-thirds of the reported human
contributions and responses budget, yet it is only tangentially concerned
with climate change or social science research. Another large fraction of
the funding goes to decision support activities, most of which lack a human
dimensions research component (see Chapter 5). Including such programs
paints a distorted picture of CCSP human contributions and responses re-
search. Funding for human dimensions research is likely on the order of $25
million to $30 million per year, excluding NIH research on the health ef-
fects of ozone and National Aeronautics and Space Administration (NASA)
decision support activities (Appendix B).
APPLICATION OF THE NRC (2005) EVALUATION FRAMEWORK
The National Research Council (NRC, 2005) report Thinking Strategi-
cally: The Appropriate Use of Metrics for the Climate Change Science Pro-
gram lays out a comprehensive framework for evaluating the progress of
the CCSP. The report identified five categories of metrics that could be used
to measure progress and guide strategic thinking across the entire CCSP:
1. Process metrics: measure a course of action taken to achieve a
goal
2. Input metrics: measure tangible quantities put into a process to
achieve a goal
3. Output metrics: measure the products and services delivered
4. Outcome metrics: measure results that stem from use of the outputs
and influence stakeholders outside the program
5. Impact metrics: measure the long-term societal, economic, or envi-
ronmental consequences of an outcome
Specific metrics within these categories are listed in Table 2.1, and key
conclusions from the report are summarized in Box 2.1. Both CCSP manag-
ers and the committee have tried to apply the metrics to a major component
OCR for page 17
21
PROCESS FOR EVALUATING PROGRESS: TASK 1A
BOX 2.1 Key Conclusions from Thinking Strategically
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Sci-
ence Program (NRC, 2005) found that progress can be assessed for most aspects
of the CCSP, from enhancement of data networks to improved public awareness of
climate change issues. The key to promoting progress is to consider the program
from end to end, starting with program processes (e.g., planning and peer review)
and inputs (e.g., resources) and extending to outputs (e.g., assessments, fore-
casts), outcomes (e.g., near-term results for science and society), and long-term
impacts. Metrics for evaluating all of these stages are given in Table 2.1. Of these,
the most appropriate will be the subset that enables managers to identify and
monitor program strengths and weaknesses. These measures will become appar-
ent from even rough scores or yes-no answers to the metrics. Detailed analysis
and tracking can then be focused on the parts of the program for which better
results are desired. The process and input metrics provide clues about why scores
on program results (outputs, outcomes, and impacts) might be low. For example, a
project may not have succeeded because it lacked a leader with authority to direct
sufficient resources to the effort. As the agencies gain experience, this subset of
metrics will be refined until only the most useful remain.
of the program (overarching goals and research elements, respectively), as
described below.
Overarching Goals
In 2006, a few CCSP managers tried applying the NRC (2005) metrics
to the five CCSP overarching goals. Their rough evaluation found that
nearly all of the metrics were relevant to the CCSP and that quantitative
scores could be assigned with a reasonably high level of confidence for most
metrics.2 Scoring was most difficult for metrics dealing with impacts, com-
munication of results, and use of results by stakeholders.
CCSP managers have neither pursued this process nor made other ef-
forts to use the NRC metrics. Resources are scarce, and the agencies are
seeking a cost-effective, practical approach to assessing progress (see task
1a, Box 1.1). Consequently, they are weighing the extent to which they
should adopt the type of broad, strategic evaluation framework recom-
mended by the NRC (2005) against developing CCSP-wide and/or agency-
specific metrics related to the CCSP (see “CCSP Approaches” below).
2 Presentation to the committee by Jack Kaye, NASA, on April 28, 2006.
OCR for page 17
22 EVALUATING PROGRESS OF THE U.S. CCSP
Research Elements
The committee tested the NRC (2005) evaluation framework on the
research elements, the component of the program that it knows best. Two
contrasting research elements were chosen for the trial: one well established
(carbon cycle) and one emerging (human contributions and responses). The
evaluation was conducted only at the top hierarchical level (e.g., carbon
cycle) because the committee lacked the budget and management informa-
tion to score the process and input metrics at subcomponent level (i.e.,
research questions).
As recommended in NRC (2005), scores to the metrics were assigned
using only the committee’s knowledge of the research elements, and pro-
grammatic and budget information provided in CCSP reports and by the
relevant interagency working groups (IWGs). This exercise enabled the
committee to gain familiarity with the concepts presented in NRC (2005)
and to see how the evaluation framework might be implemented. The re-
sults, which are presented for illustration purposes only, appear in Table
2.1. Definitive conclusions about progress in these research elements will
require additional programmatic information and peer review.
The committee found that this type of preliminary analysis is useful
for identifying strengths and weaknesses within a particular research ele-
ment and for comparing research elements to one another. For example,
the carbon cycle research element is comparatively well funded and has
a long history of strategic planning and science community involvement.
Its primary outcomes are scientific advances. Other stakeholders have had
little involvement to date, and nonscientific types of societal benefits (e.g.,
carbon management) are only beginning to be realized. In contrast, the hu-
man contributions and responses research element has had relatively little
funding, multiagency coordination, or science community participation.
However, nonscientist stakeholders are more engaged in selected agency
initiatives, and as a result the program has had some successes in informing
resource management and decision making, as well as in advancing science.
Analyses of the differences among research elements would enable more
strategic decisions to be made about where additional investments might
best accelerate progress.
Overall, the committee found the evaluation framework laid out in
NRC (2005) to be a viable method for assessing progress and making stra-
tegic decisions about the CCSP. However, available information on program
planning and resource allocation was insufficient for a rigorous evaluation
of the process and input metrics. Consequently, the committee sought an
alternative method that would be based on readily available information
for its preliminary assessment of CCSP progress.
OCR for page 17
2
PROCESS FOR EVALUATING PROGRESS: TASK 1A
EVALUATING PROGRESS BASED ON PROGRAM RESULTS
Evaluations of progress commonly focus on program results, such as
publications and services delivered. The CCSP is considering using annual
evaluations of agency-specific performance metrics, which focus on out-
comes, and tracking completion of CCSP products to assess progress. The
committee developed a “matrix” evaluation approach that focuses only
on program results. Program results are relatively well known to external
reviewers, so a peer review evaluation, which is important for assessing
quality (NRC, 2005), is possible. The agency and committee approaches
are described below.
CCSP Approaches
Agency-Specific Metrics
The 1993 Government Performance and Results Act requires federal
government agencies to set strategic goals and to measure performance
against them on an annual basis.3 Every federal agency has metrics to mea-
sure performance, and CCSP program managers are trying to determine
whether the metrics of participating agencies can also be used to assess
CCSP progress as a whole.4 However, the aggregate of agency metrics
reflects neither the level of agency involvement in CCSP programs nor the
breadth of the CCSP, a conclusion also reached in NRC (2005). For exam-
ple, five agencies (Department of Energy [DOE], NASA, National Oceanic
and Atmospheric Administration [NOAA], National Science Foundation,
and U.S. Geological Survey) have research projects and initiatives to ad-
dress CCSP overarching goal 3: reduce uncertainty in projections of how
the Earth’s climate and related systems may change in the future (CCSP,
2006a). Yet, only three of these agencies have metrics related to this over-
arching goal (Table 2.2), and these miss key aspects of climate predictions
(e.g., droughts) and interagency contributions (e.g., an integrated Earth
system analysis capability).
Some of NOAA’s goals are aligned with CCSP goals, but the other
agencies are committed to several different objectives that may or may
not have a specific climate component. The mismatch between CCSP and
agency goals suggests that it may be difficult to make more than limited
progress in the program overall.
3 PublicLaw 103-62.
4 Presentation
to the committee by Jack Kaye, NASA, on April 28, 2006, and by Mary
Glackin, NOAA, on September 15, 2006.
OCR for page 17
TABLE 2.1 Application of NRC (2005) Metrics to Two CCSP Research Elements
2
Metric Carbon Cycle Human Contributions and Responses
Process Metrics
1. Leader with sufficient authority to Leadership exists in IWG members, who The IWG has been expanded to cover decision
allocate resources, direct research effort, have agency budget authority support tools, which makes it difficult to assess the
and facilitate progress effectiveness of leadership on human contributions
and responses
2. A multiyear plan that includes goals, CCSP strategic plan as well as North CCSP strategic plan contains good research
focused statement of task, implementation, American Carbon Program and Ocean questions, but little information on
discovery, applications, and integration Carbon and Climate Change plans have implementation. A science community plan with
scientific community input and review common research goals also exists under the
International Human Dimensions Programme on
Global Environmental Change, but its connection
with CCSP is unclear
3. A functioning peer review process Program peer review through the IWG’s Peer review takes place within agency programs,
in place involving all appropriate science steering group but there is no peer review process for the research
stakeholders, with (a) underlying processes element
and timetables, (b) assessment of progress
toward achieving program goals, and (c)
an ability to revisit the plan in light of new
advances
4. A strategy for setting priorities and Science steering group is one input for Much of the supported research appears to be
allocating resources among different priority setting motivated by natural science research needs, rather
elements of the program (including than by a prioritization of what questions need to
those that cross agencies) and advancing be addressed to solve problems
promising avenues of research and
applications
OCR for page 17
5. Procedures in place that enable or Minimal development of procedures to Little effort to communicate results beyond a
facilitate the use or understanding of the facilitate communication or partnerships; few individual projects (e.g., Knowledge Systems,
results by others (e.g., scientists in other workshop on the state of carbon cycle Decision Making Under Uncertainty [DMUU],
disciplines, operational users, decision research attempted to identify and engage Regional Integrated Sciences and Assessments
makers) and promote partnerships stakeholders on carbon management [RISAs])
Input Metrics
1. Sufficient intellectual and technologic Science questions are mature; a research Intellectual, technological, and financial support
foundation to support the research pool actively pursuing these questions and are insufficient to build the research community
technological tools exist; progress is not and carry out research on CCSP goals
primarily limited by technology
2. Sufficient commitment of resources (i.e., Implementing the complete carbon science Resources are inadequate to carry out the stated
people, infrastructure, financial) directed plan requires more resources than currently research program
specifically to allow the planned program to available
be carried out
3. Sufficient resources to implement and Opportunities for basic research cut across With the exception of a few focused programs,
sustain each of the following: (a) research agencies; new approaches have been research is ad hoc and scattered across
enabling unanticipated scientific discovery, identified within the current structure the agencies, making it difficult to build
(b) investigation of competing ideas and comprehensive approaches
interpretations, and (c) development of
innovative and comprehensive approaches
Continued
25
OCR for page 17
TABLE 2.1 Continued
2
Metric Carbon Cycle Human Contributions and Responses
4. Sufficient resources to promote the Funds largely used for basic science, No apparent coordination of resources to build
development and maintenance of each observations, and modeling rather than for capacity in the human dimensions or to facilitate
of the following: (a) human capital; (b) developing stakeholder linkages the transfer of knowledge to stakeholders
measurement systems, predictive models,
and synthesis and interpretive activities; (c)
transition to operational activities where
warranted; and (d) services that enable the
use of data and information by relevant
stakeholders
5. The program takes advantage of existing Syntheses of existing and historical data Little evidence that existing social and economic
resources (e.g., U.S. and foreign historical sets have been used to reconstruct forest databases have been used by the CCSP, or
data records, infrastructure) inventory and land cover changes. Strong that the CCSP is emphasizing the creation of
links exist to international carbon cycle new databases necessary to advance human
research efforts contributions and responses research
Output Metrics
1. The program produces peer-reviewed Large number of peer-reviewed publications A number of successful initiatives have been
and broadly accessible results, such as (a) and data products for all except (d) undertaken, but they appear to be insufficient to
data and information, (b) quantification of address the research element
important phenomena or processes, (c) new
and applicable measurement techniques,
(d) scenarios and decision support tools,
and (e) well-described and demonstrated
relationships aimed at improving
understanding of processes or enabling
forecasting and prediction
OCR for page 17
2. An adequate community and/or Research community exists; infrastructure The research community is small and may be
infrastructure to support the program has includes observation networks and insufficient to meet the requirements of the
been developed modeling. However, planned expansion of research element
networks has not materialized
3. Appropriate stakeholders judge these Work benefits and is benefited by research Little documented interaction with stakeholders,
results to be sufficient to address scientific in other science fields (e.g., land use with possible exceptions such as the RISAs and the
questions and/or to inform management and land cover, ocean acidification and International Research Institute for Climate and
and policy decisions ecosystems). Links to carbon management Society (IRI)
communities are weaker—best links are
with U.S. Department of Agriculture’s
Agricultural Research Service and Forest
Service inventory, but less interaction with
carbon emissions scenario building (carbon
markets)
4. Synthesis and assessment products CCSP synthesis and assessment product 2.2 Synthesis and assessment products are focused
are created that incorporate these new is under review (more or less on schedule; mostly on decision support
developments see Appendix A)
5. Research results are communicated to an Some uncoordinated efforts exist to link A few programs (e.g., RISAs, DMUU centers) have
appropriate range of stakeholders land management to carbon and net climate created structures for communication with and
effects transfer of knowledge to stakeholders
Outcome Metrics
1. The research has engendered significant New science questions (e.g., role of fire or Program initiatives have pointed toward new
new avenues of discovery disturbance, ocean acidification) have been avenues of discovery, such as characterizing
engendered by research irreducible uncertainties about climate change
and its impacts, assessing vulnerabilities, and
understanding the role of institutions in climate-
related decision making
Continued
2
OCR for page 17
2
TABLE 2.1 Continued
Metric Carbon Cycle Human Contributions and Responses
2. The program has led to the identification Scientists still cannot balance the global Program initiatives (e.g., DMUU centers) are
of uncertainties, increased understanding of carbon budget, although uncertainties in beginning to lead to increased understanding
uncertainties, or reduced uncertainties that some areas have narrowed. Feedbacks of uncertainties and development of decision
support decision making or facilitate the remain important areas of research strategies and tools that incorporate uncertainties,
advance of other areas of science (coupled modeling advances) which could inform decision making at several
scales and sectors
3. The program has yielded improved Coupled climate-carbon models have been Investment in scenarios and economic modeling
understanding, such as (a) more developed, although they are in the early has yielded limited knowledge useful for informing
consistent and reliable predictions or stages policy making
forecasts, (b) increased confidence in our
ability to simulate and predict climate
change and variability, and (c) broadly
accepted conclusions about key issues or
relationships
4. Research results have been transitioned Carbon management is in the early stages Some valuable outputs have been transferred to
to operational use of operations users (e.g., by RISAs and IRI) in areas such as
drought response, fire management, and fisheries
management
5. Institutions and human capacity have Links to carbon markets are weak; no Little human capacity has been created
been created that can better address a range institutions have been established
of related problems and issues
OCR for page 17
6. The measurements, analysis, and results Inadequate progress beyond basic science Activities to achieve these outcomes have taken
are being used (a) to answer the high- place on a very limited scale
priority climate questions that motivated
them, (b) to address objectives outside the
program plan, or (c) to support beneficial
applications and decision making, such
as forecasting, cost-benefit analysis, or
improved assessment and management of
risk
Impact Metrics
1. The results of the program have Science advances are informing assessments Some programs (e.g., RISAs) appear to have
informed policy and improved decision of the Intergovernmental Panel on Climate informed policy making, although not to the
making Change extent envisioned in the CCSP strategic plan
2. The program has benefited society in Carbon management is being addressed at Positive societal benefit, especially if measured
terms of enhancing economic vitality, the state or regional level, if at all against the level of investment
promoting environmental stewardship,
protecting life and property, and reducing
vulnerability to the impacts of climate
change
3. Public understanding of climate issues Improved public understanding of the links Insufficient data to measure the effect of initiatives
has increased to increase public understanding of climate
between CO2 and warming
2
OCR for page 17
0 EVALUATING PROGRESS OF THE U.S. CCSP
TABLE 2.2 Metrics of Participating Agencies Related to CCSP
Overarching Goal 3
Agency Multiyear Performance Goals and Annual Performance Measures
NOAA Understand climate variability and change to enhance society’s ability
to plan and respond. Annual performance measures:
• U.S. temperature forecasts (cumulative skill score computed
over the regions where predictions are made)
• Reduce uncertainty in the magnitude of the North American
carbon uptake
• Reduce uncertainty in model simulations of the influence of
aerosols on climate
• Determine the national explained variance (%) for temperature
and precipitation for the contiguous United States using U.S. Climate
Reference Network stations
DOE Deliver improved climate data and models for policy makers to
determine safe levels of greenhouse gases for the Earth’s system. By
2013, substantially reduce differences between observed temperature
and model simulations at subcontinental scales using several decades
of recent data. Annual performance measure:
• Improve climate models: Produce a new continuous time
series of retrieved cloud properties at each Atmospheric Radiation
Measurement site and evaluate the extent of agreement between
climate model simulations of water vapor concentration and cloud
properties and measurements of these quantities on time scales of 1 to
4 days
NASA Progress in understanding and improving predictive capability for
changes in the ozone layer, climate forcing, and air quality associated
with changes in atmospheric composition. Annual performance
measure:
• Demonstrate that NASA-developed data sets, technologies,
and models enhance understanding of the Earth system, leading to
improved predictive capability in each of the six science focus area
road maps
Progress in quantifying global land cover change and terrestrial and
marine productivity, and in improving carbon cycle and ecosystem
models. Annual performance measure:
• Demonstrate that NASA-developed data sets, technologies,
and models enhance understanding of the Earth system, leading to
improved predictive capability in each of the six science focus area
road maps
Progress in quantifying the key reservoirs and fluxes in the global
water cycle and in improving models of water cycle change and
freshwater availability. Annual performance measures:
• Demonstrate that NASA-developed data sets, technologies,
and models enhance understanding of the Earth system, leading to
improved predictive capability in each of the six science focus area
road maps
• Complete Global Precipitation Measurement confirmation
review
OCR for page 17
OCR for page 17
OCR for page 17
OCR for page 17
1
PROCESS FOR EVALUATING PROGRESS: TASK 1A
TABLE 2.2 Continued
Agency Multiyear Performance Goals and Annual Performance Measures
NASA Progress in understanding the role of oceans, atmosphere, and ice in
(continued) the climate system and in improving predictive capability for its future
evolution. Annual performance measures:
• Demonstrate that NASA-developed data sets, technologies,
and models enhance understanding of the Earth system, leading to
improved predictive capability in each of the six science focus area
road maps
• Complete Operational Readiness Review for the NPOESS
Preparatory Project
SOURCE: DOC, 2006; DOE, 2006; NASA, 2006;
2 EVALUATING PROGRESS OF THE U.S. CCSP
of the matrix) against outputs and outcomes (columns of the matrix). The
columns of the matrix were derived from five categories of CCSP prod-
ucts and results, themselves generalized from categories identified in NRC
(2005):
Category A: improve data sets in space and time, and improve estimates
of physical quantities
Category B: improve understanding and representation of processes
Category C: improve predictability, predictive capabilities, or assess-
ment of uncertainty
Category D: improve synthesis and assessment to inform
Category E: improve assessment and management of risk, and improve
decision support for management and policy making
These product categories provide a measure of the maturity of the
program, starting from data collection, to predictions of future climate
changes, to improved use of information to better serve society. The com-
plete matrix is given in Appendix C.
The research questions were chosen for evaluation because nearly all
of the CCSP’s milestones, products, and payoffs are associated with them.
Moreover, they are linked to the focus areas of the overarching goals (Fig-
ure 2.1). The product categories overlap with the cross-cutting issues. In
particular, category A includes observations and monitoring, category C
includes modeling, category D includes communication, and category E
includes decision support. Consequently, the matrix enables a reasonably
broad assessment of CCSP progress.
The committee used the matrix to make a preliminary assessment of
CCSP progress (task 1b). Scores for the rows of the matrix provide infor-
mation on progress in the research questions, research elements, and over-
arching goals (see Chapters 3 and 4). Scores for the columns of the matrix
provide a measure of the maturity of the research element as well as an
indication of progress in the cross-cutting issues (see Chapters 3 and 5).
CONCLUSIONS
Recommendation. CCSP progress should be evaluated in two stages:
(1) a broad overview of the entire program based on the knowledge
of the reviewers, and (2) a more in-depth analysis of areas in which
progress has been inadequate, using the process and input metrics from
NRC (2005).
Agency approaches to monitoring progress (i.e., compiling agency met-
rics, tracking milestones and products) are useful, but open only a narrow
PROCESS FOR EVALUATING PROGRESS: TASK 1A
window onto the program. More information about the program as a
whole can be obtained from the matrix approach developed by the com-
mittee. Overall, the committee found that the matrix can be used to assess
progress toward the research elements and overarching goals, although it
is unwieldy to evaluate the goals in this manner. The matrix also provides
information on progress in the cross-cutting issues, to the extent that these
issues overlap with the product categories (i.e., the columns of the matrix).
However, this method provides little insight on why good progress has or
has not been made. Diagnosing the causes of strengths and weaknesses
requires knowledge of planning, leadership, and resources in addition to
program results. The process and input metrics from the NRC (2005)
framework offer a means to evaluate these issues.
An evaluation of CCSP progress using either the committee’s matrix
or the NRC (2005) evaluation framework can be daunting. The matrix
contains 165 cells, but more than 165 scores are necessary since many of
the research questions cover a complex set of issues. A similar number of
scores would be required to evaluate the major components of the program
(five overarching goals, seven research elements, and six cross-cutting is-
sues) using the 24 metrics identified in NRC (2005). Evaluating the research
questions by this method would require nearly 800 scores.
The evaluation process can be made more practical by breaking it into
stages, with the initial evaluation aimed at identifying successes and finding
weaknesses. Preliminary scores can be assigned using only the knowledge
of the reviewers and information on programs and results in readily avail-
able publications. A community workshop, such as the one organized by
the committee, is one way to obtain the breadth of knowledge required to
carry out the first stage. A stage 1 evaluation of the entire program might
be necessary only when there is a major change in the program, such as a
new strategic plan.
The second stage of evaluation can focus exclusively on areas identified
in the first stage as not meeting program objectives and expectations. These
areas would be evaluated with the process and input metrics from NRC
(2005), which provide tools for diagnosing the reasons for the weakness.
Detailed programmatic and budgetary information would be required to
carry out this stage of evaluation, which should continue until outcomes
improve.
Share
Research
Rights & Permissions
