5
Advice for Effective Assessments

Certain strengths and weaknesses, common to several assessments analyzed by the committee, illuminate critical features of effective assessments. For example, a well-defined mandate and consistent support from those requesting the assessment contributed importantly to the effectiveness of the Arctic Climate Impact Assessment (ACIA) and the Stratospheric Ozone Assessments, while the process outcome of the Global Biodiversity Assessment (GBA) was impaired by lack of a clear mandate from the target audience. Several assessments benefited significantly from well-articulated, multifaceted, and extensive communication strategies. The Ozone Assessments were especially effective in providing relevant information for decision-making processes, whereas the ACIA was particularly outstanding in the scope of its communications outreach. Other components of effective assessments included superior leadership, extensive and well-designed stakeholder engagement, and a transparent and effective science-policy interface. Perhaps the most common weakness of past assessments has been a discrepancy between the scope of the mandate and the funding provided for the assessment effort.

Drawing both on the analysis in this and the preceding chapter and on the relevant literature, the committee identified 11 essential elements of effective assessments:

  1. A clear strategic framing of the assessment process, including a well-articulated mandate, realistic goals consistent with the needs of decision makers, and a detailed implementation plan.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 99
Analysis of Global Change Assessments Lessons Learned 5 Advice for Effective Assessments Certain strengths and weaknesses, common to several assessments analyzed by the committee, illuminate critical features of effective assessments. For example, a well-defined mandate and consistent support from those requesting the assessment contributed importantly to the effectiveness of the Arctic Climate Impact Assessment (ACIA) and the Stratospheric Ozone Assessments, while the process outcome of the Global Biodiversity Assessment (GBA) was impaired by lack of a clear mandate from the target audience. Several assessments benefited significantly from well-articulated, multifaceted, and extensive communication strategies. The Ozone Assessments were especially effective in providing relevant information for decision-making processes, whereas the ACIA was particularly outstanding in the scope of its communications outreach. Other components of effective assessments included superior leadership, extensive and well-designed stakeholder engagement, and a transparent and effective science-policy interface. Perhaps the most common weakness of past assessments has been a discrepancy between the scope of the mandate and the funding provided for the assessment effort. Drawing both on the analysis in this and the preceding chapter and on the relevant literature, the committee identified 11 essential elements of effective assessments: A clear strategic framing of the assessment process, including a well-articulated mandate, realistic goals consistent with the needs of decision makers, and a detailed implementation plan.

OCR for page 99
Analysis of Global Change Assessments Lessons Learned Adequate funding that is both commensurate with the mandate and effectively managed to ensure an efficient assessment process. A balance between the benefits of a particular assessment and the opportunity costs (e.g., commitments of time and effort) to the scientific community. A timeline consistent with assessment objectives, the state of the underlying knowledge base, the resources available, and the needs of decision makers. Engagement and commitment of interested and affected parties, with a transparent science-policy interface and effective communication throughout the process. Strong leadership and an organizational structure in which responsibilities are well articulated. Careful design of interdisciplinary efforts to ensure integration, with specific reference to the assessment’s purpose, users needs, and available resources. Realistic and credible treatment of uncertainties. An independent review process monitored by a balanced panel of review editors. Maximizing the benefits of the assessment by developing tools to support use of assessment results in decision making at differing geographic scales and decision levels. Use of a nested assessment approach, when appropriate, using analysis of large-scale trends and identification of priority issues as the context for focused, smaller-scale impacts and response assessments at the regional or local level. The committee concludes that attention to these elements, many of which have been identified in the previous literature, increase the probability that an assessment will be credible, legitimate, and salient, and therefore will effectively inform both decision makers and other target audiences. In the following findings and recommendations, the committee provides general guidance for incorporating these elements into future assessments. FRAMING THE ASSESSMENT Establishing a Clear Mandate Whether domestic or international in scope, an assessment will benefit by an authorizing environment that ensures it has a clear mandate and the resources necessary to respond to the task. As described in the case of the National Assessment of Climate Change Impacts (NACCI), the process was greatly facilitated by the fact that it was mandated by the U.S. Global

OCR for page 99
Analysis of Global Change Assessments Lessons Learned Change Research Act (GCRA) of 1990. This requirement helped develop support for the assessment from representatives of the agencies participating in the U.S. Global Climate Change Research Program (GCRP). Similarly, a clear mandate and a strong authorizing environment were provided when a formal proposal was adopted by the foreign ministers of the eight Arctic countries to establish the ACIA. In this case, the mandate by the foreign ministers of the Arctic countries guaranteed that the assessment had a target audience that was not only receptive but also interested in the outcome (Corell 2006). Lack of a mandate can be considered the most significant pitfall of the GBA (Watson 2006). Although the GBA was established by the United Nations Environment Programme to provide a scientific basis for the Convention on Biological Diversity (CBD), it was never formally authorized by the CBD and as a result encountered a number of barriers that might have otherwise been avoided. These examples illustrate how an assessment with a clear mandate from decision makers is more likely to be viewed as legitimate and salient, particularly if the mandate specifies the degree to which the assessment should identify and evaluate policy or other response options. In fact, maintaining legitimacy requires an assessment to respect the boundary between science and policy, which means that it is to provide policy-relevant information and not to exceed its mandate by providing policy-prescriptive recommendations unless clearly asked to do so. Specifying Goals and Objectives The state of the relevant scientific knowledge and the decision-making context that an assessment seeks to inform affects the kinds of decisions to which it is relevant, which in turn should help define the appropriate goals and objectives of an assessment. For example, the goal may be to establish the state of knowledge, to indicate the latest understanding of impacts, or to provide response options. For the last, the goal might be to provide information on the effects of alternative response strategies on relevant impact categories. Different goals will have profound implications for the decisions undertaken during the course of an assessment. Confusion on the part of participants regarding the goals of the assessment can severely limit its effectiveness. Assessment participants must come to a mutual understanding of what they are being asked to do in response to the charge. As part of the mandate, goals and objectives need to be well articulated, including the kinds of decisions that the assessment should inform, how the assessment will be implemented, and how progress toward goals will be measured (NRC 2005). In particular, in the scoping phase of the assessment, organizers need to identify the target audience and the decisions it is intended to inform as well as the types of information needed to make

OCR for page 99
Analysis of Global Change Assessments Lessons Learned those decisions. The ACIA provides an excellent example of an assessment with clear goals and objectives that were set by the sponsors. These goals appeared to remain intact throughout the assessment process. Since external conditions may evolve, goals and objectives may have to be adjusted during an assessment to remain salient to the target audience. The ozone assessments provide an exemplary case of an adaptive assessment process, including explicit changes in goals and objectives made by the authorizing body in response to changes in the state of the knowledge. When changes in approach are required, they should involve a deliberate and transparent decision on the part of the authorizers and leadership of the assessment to ensure legitimacy. To ensure credibility and legitimacy, the rationale for such changes has to be well documented. An Appropriate Framework Because design choices made during the initiation of the assessment tend to be structural and difficult to change once the process is under way, they have to be considered carefully. An appropriate framework, which may vary depending on the type of assessment, is required to answer the mandate effectively. Ideally, the framework is specified in advance to guide the process. Problems often arise when a discrepancy exists between the questions to be addressed and the framework and approach used. The ACIA took great care in preparing such a framework by conducting meetings during a two-year planning phase prior to initiating the assessment process. It drafted a proposal specifying the approach, which was vetted by the Arctic council and the participants. It provided clear guidelines and guaranteed ownership by the relevant target audience and stakeholders. The ozone assessments and the Intergovernmental Panel on Climate Change (IPCC) provide good examples of choosing an appropriate framework. Both of these studies include effective response assessments in addition to process and impact assessments. The focus on identifying specific technologies to reduce environmental impacts in particular applications or industrial processes seems to have benefited especially the ozone assessments, given the impact the Technology and Economic Assessment Panel (TEAP) had on technology choices, and hence decision making, in the private sector. The attempt to engage individuals from the private sector with the necessary technical expertise to distinguish technical feasibility from economic feasibility has been particularly noteworthy. Therefore, the committee considers the TEAP a particularly effective model for a technology assessment, a special case of response assessment. Although the NACCI was designed primarily to be an impact assessment, it required the development of scenarios linking atmospheric concentrations of greenhouse gases to their impact on key climate variables.

OCR for page 99
Analysis of Global Change Assessments Lessons Learned Outputs from general circulation models were developed based on scenarios of human and natural emissions to provide the inputs for sectoral and regional analyses. In fact, most assessments of the impacts of global change nevertheless require incorporation of basic science (process assessments). Since the NACCI was not limited to incorporating references to preexisting literature in assessing impacts, it was able to use the latest model run outputs. However, because of a late start, it was not able to incorporate a phased approach in the selection of specific climate model scenarios. As a result, those responsible for the impact assessment were lacking some of the necessary scientific information to guide their deliberations. In addition to timing issues, the NACCI was also criticized because the models were not generated by U.S. scientists. This example illustrates how the choice of information to be included may affect the credibility of the process, but more importantly, it stresses the importance of project management and timeliness. The NACCI also illustrates how important the choice of scope and geographic scale is for many other design issues of the assessment. The scope and scale of the assessment should be chosen to match the scope and scale of the decisions it is intended to inform. In the case of climate change, there are a multitude of decisions at various scales spanning numerous public-and private-sector decision makers that could potentially benefit from an assessment. The scope and scale of the information produced (among other considerations) will determine whether or not the assessment outcomes are salient. The choice might also require a phased approach—for example, starting with a more global or national assessment, followed by regional-scale assessments. In fact, the NACCI process was intended as an iterative process, where the national assessment would inform the regional-scale assessment and vice versa. Clearly, such decisions have to be addressed during the inception of the assessment as part of the general framework. The framework is best specified in a guidance document. This guidance document needs to clearly articulate the mandate and specify the decision the assessment should inform, which will guide the type of assessment (process, impact, response) required. Other issues that the framework should specify include the degree of integration necessary, scope, timing, target audience, leadership, communication strategy, funding, and measures of success. In addition, the respective roles of those requesting and those funding the assessment in scoping the assessment should be clarified in the original guidance document in order to avoid major discrepancies between the assessment’s mandate, expected results, and available funding. Establishing these guidelines in advance and putting them through a vetting process will greatly enhance salience, credibility, and legitimacy. It is also important to specify how information is judged as credible enough to be included in the assessment. Standards must be established

OCR for page 99
Analysis of Global Change Assessments Lessons Learned for the inclusion of information. In the case of process assessments, this may simply require a decision on whether only peer-reviewed information should be included or whether original research, not yet peer-reviewed, can be considered. In the case of impact assessments, the development of standards become much more difficult, because of the need to include stakeholder information, such as indigenous knowledge, and the need to make some degree of value judgments. Further, integrated assessments will result in new combinations of multiple types of preexisting knowledge. Although alternative interpretations of the knowledge base are inevitable and must be reflected in the report, standards must be developed to determine what belongs in the knowledge base and how to reflect uncertainty. To ensure the greatest credibility, the knowledge base must be evidence based and not value based, particularly in impact assessments. Recommendation: The leadership of and those requesting assessments should develop a guidance document that provides a clear strategic framework, including a well-articulated mandate and a detailed implementation plan realistically linked to budgetary requirements. The guidance document should specify the decisions the assessment intends to inform; the assessment’s scope, timing, priorities, target audiences, leadership, communication strategy, funding, and the degree of interdisciplinary integration needed; and measures of success. Strategic Plan for CCSP’s Assessment Activities Although the U.S. Climate Change Science Program (CCSP) has a mandate under GCRA to conduct assessments, the program lacks a long-term strategic framework for meeting this mandate. Prior to undertaking future assessments, CCSP will need to clearly express long-term goals for the program as well as specific goals for each assessment, specifying decisions the program intends to inform. A strategic plan comprising overall goals, mandate, and implementation strategy for CCSP assessment activities would enhance the salience, credibility, and legitimacy of future assessments—especially if the plan is accepted at high levels of government as well as within the science agencies and the scientific community. Such an overarching long-term strategic plan for CCSP assessment activities would foster programmatic and funding continuity that could adapt to evolving circumstances and to changes in political administration. Recommendation: The CCSP should develop a broad strategic plan for its assessment activities that focuses not only on a specific short-term objective, such as preparing the next report or assessment product, but also on longer-term objectives that are in the national interest.

OCR for page 99
Analysis of Global Change Assessments Lessons Learned ADEQUATE FUNDING A common problem of several past assessments has been the discrepancy between the mandate and the funding. For example, the NACCI was funded within preexisting agency budgets, and funds were not available for all regions and sectors in a timely way. This resulted in limited support for some teams and delayed funding for others. Overall, it led to unevenness in the team reports and exacerbated difficulties in creating a coherent and consistently high-quality synthesis of all regions and sectors. The budget for the GBA was much lower than for other global assessments of its scope. This had a deleterious effect on the number of meetings that could be held, the number of reports that could be prepared and published, and the size of the support staff. Because funding is required for a broad and representative participation of experts and stakeholders; for administrative support to facilitate the compilation, writing, and review of the product; and for an extensive communication, dissemination, and outreach efforts, insufficient funds can jeopardize critical aspects of an assessment. This discrepancy between mandate and funding can stem from the fact that those who mandate the study do not actually fund it. As an example, ACIA had a very clear mandate from the Arctic Council and the International Arctic Science Committee; however, the funding came from government agencies in the eight Arctic nations. This became a problem from the beginning because the agencies tasked with providing funds were not all fully supportive of the activity. When broad expert participation is required, the legitimacy of the assessment might be questioned if insufficient funds are available to support individuals from stakeholder communities who would not otherwise be able to participate. Lack of sufficient funding can have a particularly negative effect on the legitimacy of a global assessment if it limits or prevents developing-country involvement. A well-designed communication strategy requires significant administrative support throughout the process, particularly at the end for dissemination. Therefore, funds need to be reserved for the final phase when intensive outreach and dissemination efforts are required. Because inadequately funded assessments may have to shortchange critically important process steps that lead to the loss of credibility, legitimacy, and/or salience, organizers must be especially strategic about proceeding with the assessment if funding sources are not secured from the outset. Therefore, organizers need to address the following questions before initiating the process: Does it make sense to begin with insufficient funding in the hope that additional resources will become available? If so, what are the implications from a process perspective relative to the goals and objectives? Will it compromise the credibility of the assessment? Should the mandate and scope be adjusted to the available funds?

OCR for page 99
Analysis of Global Change Assessments Lessons Learned Although the presence of a clear mandate can provide the impetus for fund raising, few assessments had as clear a mandate at their inception as the NACCI, yet a funding shortfall turned out to be a significant problem. An additional consideration relative to funding is the need for programmatic continuity, especially in light of the potential for major changes in direction that result from changes in administration. Since global change research necessarily requires a long-term perspective, abrupt changes in focus and scope can lead to losses in salience, credibility, and legitimacy. When preparing a framework to be vetted and approved by the sponsors and participants, it is desirable to have the framework contain sufficient detail to link key activities to resource requirements. Large budget items should be anticipated and agreed upon by the key players, with realistic estimates of costs. Failure to anticipate the full range of funding needs can lead to underfunding and an uncertain outcome. It is difficult to raise funds during an ongoing assessment. Recommendation: Resources made available to conduct an assessment should be commensurate with the mandate. Therefore, the guidance document for the assessment should clarify the role in scoping the assessment mandate of those who are requesting and funding it. The budgeting of resources should focus on ensuring the success of the highest-priority components of the assessment, including aspects that have been shortchanged in the past, such as supporting broad stakeholder participation, communication activities, and dissemination. ASSESSMENT BENEFITS, OPPORTUNITY COSTS, AND EFFICIENCY CONSIDERATIONS Assessments as well as the activities tailored to support assessments have mobilized a large number of scientists over the last decades. This effort has affected the national and international research agenda and has engaged research institutions and universities in new types of activities. In certain cases, the need to participate in assessments has facilitated the development of new research disciplines or has brought together different scientific communities that had never cooperated in the past. The Climatic Impact Assessment Program (CIAP) organized in the early 1970s by the U.S. Department of Transportation provides a striking example that illustrates how the need to address a specific and urgent environmental question has contributed to the development of a new research field. CIAP brought together specialists in dynamic meteorology, radiative transfer, and atmospheric chemistry and led to the formation of a new research community that specializes in questions of the middle atmosphere. This community played a decisive role a few years later when the ques-

OCR for page 99
Analysis of Global Change Assessments Lessons Learned tion of ozone depletion by industrially manufactured chlorofluorocarbons became an important issue. In many other cases, assessments have brought together experts from different disciplines, teaching them to work together and resulting in subsequent interdisciplinary research projects. In addi-In addi-In addition, assessments can articulate the progress, limitations, and opportunities associated with climate research and promote new directions of research by better defining critical scientific questions and research needs. Although striving for credibility, legitimacy, and salience is likely to result in an effective assessment, the process might pose a daunting workload and might seem burdensome and inefficient. This issue is exacerbated by the growing number of assessments in which U.S. scientists are involved and the fact that many ongoing assessments are growing in magnitude (Mitchell et al. 2006). The number of experts who participate and the volume of the output are sometimes used as implicit indicators of the credibility and seriousness of the assessment activity (Reid 2006). However, if these indicators are taken to their limit, the assessment process may have diminishing returns and may no longer be efficient. It is important to balance the human and financial costs associated with any assessment against the value and impact of the assessment outcome. More importantly, one has to consider the opportunity cost associated with the human resources being devoted to assessments instead of advancing the scientific understanding of the issue itself. The assessments examined by the committee have provided significant and tangible benefit, but they also provide valuable perspective on the human resources invested by the scientific community that is not focused on research. To illustrate the growing demand on human resources by assessments, consider, for example, the following observations about U.S. assessment activities: The number of global change assessment activities is increasing, including some 21 activities planned or under way by the CCSP, which are combined with U.S. leadership or participation in many other activities such as the IPCC. The scale of assessments is growing significantly. The IPCC is a good case in point. Consider just the metric of the size of the reports. In 1990, 1995, and 2005, the scientific assessment portions of the IPCC were 364, 570, and 881 pages, respectively. The impacts and vulnerabilities sections in 1990, 1995, and 2000 were 242, 447, and 1032 pages, respectively. The response strategies topics in 1990, 1995, and 2000 were 270, 447, and 750 pages, respectively. An argument could be made that these increases in volume describe the wealth of investment and accomplishment associated with new research, but it is also evident that the reports increasingly tend to be comprehensive documents rather than statements of progress or

OCR for page 99
Analysis of Global Change Assessments Lessons Learned identification of issues. In 2000, the synthesis report for the IPCC alone was 396 pages long. This increase in volume often does not stem from an increase in mandate but from the difficulty assessment participants have limiting the scope and the amount of information to include. The IPCC is not the only assessment generating considerable volumes of information. For example, the first Millennium Ecosystem Assessment (MA) produced 81 chapters totaling more than 3,000 pages. Many major assessments include a schedule for repeating the process at regular intervals. For example, the act of Congress that created the GCRP also requires national assessments on four-year intervals; the first comprehensive national assessment on climate change was completed over a four-year span, and the current U.S. national assessment effort is now producing 21 synthesis and assessment reports by the CCSP. The IPCC is scheduled at five- to six-year intervals. However, the next IPCC assessment starts immediately upon completion of the last, and it has become essentially a continuous process. There is a perception that the rate of scientific progress is slower than the rate at which the assessments are being conducted. The Ozone Assessments under the Montreal Protocol requires assessments of science, impacts, technology, and economics every four years but is notably an exception, in that it was designed to include updates on the science only and appears to be producing shorter reports with time. Assessments involve hundreds of climate and climate-related scientists. The tasks of lead authors for the IPCC chapters of various working groups require a significant time commitment. Similarly, the MA involved 1,360 experts from 95 countries. The investment in time includes not just the number of authors, but also the number of reviewers. Consider, for example, that the MA processed 20,745 review comments from 2,516 experts. There are more than 200 international treaties, most of which require periodic assessments (Mitchell et al. 2006). In 2003 alone, more than 12 such assessments were under way, each engaging hundreds to thousands of scientists. The considerable time investment also raises questions about the potential for the following unintended and undesirable consequences: The level of review and the willingness of reviewers to evaluate assessment products may decline with volume and repetition. As a case in point, in the NACCI, the 5-page description of the climate basis included in the overview chapter received 18 pages of expert review (federal agencies and solicitations from expert scientists), while the 60-page foundation chapter, which served as its basis, received only 3 pages of expert review. During the public comment period, the overview received 10 times the vol-

OCR for page 99
Analysis of Global Change Assessments Lessons Learned ume of comments as the foundation section on climate. Two conclusions are possible from this analysis. First, the community perceives that the synthesis or overview report element will have more impact and therefore is more important to analyze and comment upon. Second, reviewers are taxed by the volume and, therefore, few are willing to review an entire assessment report. Unfortunately, the level and/or quality of review may decline if the task is too onerous. The growing magnitude of the assessment process may begin to change the participation by scientists from different communities. The magnitude of the effort can influence whether the best expertise can be engaged in the process, either because the “best” experts are often already quite busy or because they may have experienced some burnout in earlier activities and not be willing to serve on additional assessments. Participation may be particularly challenging for young university scientists because of the combined teaching and research mission of these individuals. Since participation in comprehensive assessments cannot be budgeted as sponsored research or teaching, it becomes an unfunded mandate for university researchers. Consequently, the job descriptions (nongovernmental organization [NGO], university, government) and career level of the scientists who participate may change with the growing magnitude and repetition of assessments. The motivation for participation in any assessment process changes if the process becomes too time consuming. If the assessment is perceived to be of considerable political importance or contentious, then the process may motivate participation from the tails of distribution of scientific opinion or perspective to either help ensure or help prevent a particular outcome. The impact of assessment products will also change with the volume of reports simply because stakeholder comprehension and willingness to read lengthy reports will decline. The most important outcomes of an assessment may also be blurred by the sheer volume of the discussion. One conclusion from this analysis is that the human and financial resources required to create a credible assessment should be examined at the start of the process and actions incorporated to ensure that the use of the resources is effective and efficient. Given the important contribution of assessments to policy making and to society in general and growing number of international treaties and national mandates, considerations of efficiency become increasingly important to minimize the opportunity cost to the research community. It may also be time to consider alternative modes of participation, changes to the assessment process, or both, particularly for assessments that are scheduled to reoccur at a given interval. Possibilities include the following:

OCR for page 99
Analysis of Global Change Assessments Lessons Learned building on prior experiences. A systematic effort is required to improve assessments in the future by drawing lessons from past experience and by developing assessment methodologies and tools. Capacity building should therefore include research support for improving assessment methodologies as well as ensuring that the assessment leadership and participants are familiar with the most recent assessment methodologies and tools. Investments in capacity building can have payoffs in multiple areas, including (1) expanding the informed audience for the assessment, (2) contributing to future assessment effectiveness, (3) expanding the ability of decision makers to act on scientific information, (4) equipping participants with new knowledge in assessment methodology and tools, and (5) building a scientific community that is more sensitive to needs and concerns of the broader society. In some cases the value of the assessment process, which may involve considerable time commitments on the part of participants, might not be immediately apparent. Thus, additional effort may be required to communicate the benefits and to structure the questions and process such that they are relevant to the participants the assessment aims to engage. Private-sector participation has been noted as a serious deficiency in multiple U.S. and international assessments. Because engaging business interests has historically been very challenging, special considerations are required to successfully engage private-sector participants. The success of the TEAP has clearly demonstrated the great benefit from designing a process that engages the private sector. Developing a strategy to encourage its participation requires consideration of its decision-making context and business needs. Face-to-face meetings are expensive in terms of time and money, and the connection to either short- or long-term benefits needs to be clear. The global change community needs to be strategic about constructive and creative ways of engagement with the private sector. This might be accomplished by conducting workshops for particular sectors, focused on their concerns, such as identifying economic risks and opportunities and “news you can use,” and through web-based communication techniques. Recommendation: Capacity-building efforts for diverse stakeholders and assessment participants from various disciplines should be undertaken by CCSP in order to develop a common language and a mutual understanding of the science and the decision-making context. This capacity building may be required to ensure that the most salient questions are being addressed and to meaningfully engage diverse stakeholders in assessment activities.

OCR for page 99
Analysis of Global Change Assessments Lessons Learned Communication Strategy A communication strategy is fundamental to the success of an assessment: without effective communication, the scholarly effort is diminished. Ideally, communication is a two-way process of education. Only if an assessment’s scientific findings are effectively communicated, understood, and accepted by targeted audiences can they optimally inform policies and decisions to address the environmental challenges analyzed in the assessment. Furthermore, the target audience must be able to communicate its information needs to the experts conducting the assessment to guarantee that the relevant questions are being addressed. Communication must, therefore, be regarded as a process, not merely an appendage to a report-writing exercise. The Enquete Kommission provides an example in which the two-way dialogue is guaranteed by having scientists and politicians involved in the process continuously, thereby also increasing the likelihood of timely delivery of the information. However, the Enquete Kommission’s direct involvement of politicians in the process is unlike all other assessments evaluated by the committee. The ACIA exemplifies a more typical approach, where the politicians providing the mandate are not directly engaged. The ACIA is universally recognized for having a well-articulated communication strategy to support the policy-making process. Two important factors contributed to the success of the ACIA’s communication strategy: (1) its communication strategy was planned from the initiation throughout the process and carried out beyond the report production phase including dissemination activities targeting a broad range of audiences; and (2) the intended target audience was identified in advance, and tailored communications were produced. The IPCC and the MA are making very extensive use of the Internet. Reports are readily available and easily downloaded, including excellent color graphics. In addition, these reports are available in multiple languages. The characteristic complexity of the science and the range of scientific uncertainties add to the communication challenge. There is often an inherent conflict between scientists’ penchant for exactitude and the effective presentation of an environmental assessment to a nontechnical audience (Johnson and Slovic 1995; Johnson 2003). This challenge must be addressed through conscious efforts to simplify language, tables, and scenarios to make them more understandable and to illustrate difficult concepts creatively, particularly when designing dissemination products aimed at the general public. The report should avoid academic jargon and be crafted for easy accessibility. Extreme care must be exercised, however, that any simplified text prepared for differing audiences does not conflict with the more scientific presentations designed for the assessment’s original sponsor. The ozone assessments have grown increasingly sophisticated over time in their communication approaches and in simplifying their message. From

OCR for page 99
Analysis of Global Change Assessments Lessons Learned the 1985 assessment, which did not even include an executive summary, subsequent assessments evolved to produce reports with carefully prepared summaries, viewgraphs, talking points, and nontechnical publications (e.g., Common Questions About Ozone) that summarize current knowledge in commonsense terms and (implicitly) address any current attempts to mislead or obscure the consensus. Executive summaries are one of the most crucial elements for the successful impact of the assessment exercise on policy and decision making. It should be concise, value-free, and clear about assumptions and uncertainties, and should be crafted and reviewed with attention to clarity, substance, relevance, absence of jargon, and the differing needs of policy and decision makers—recognizing that they are generally not specialists. Recommendation: Assessments should have a comprehensive, multifaceted communication strategy from the start, encompassing an analysis of the potential audiences, ranging from those requesting the assessment to the general public; use multiple modes of engaging them; focus on the decisions the assessment intends to inform (e.g., policy decisions, legislation, technological innovation, standards, international treaties); and include appropriate dissemination activities. LEADERSHIP AND ORGANIZATIONAL STRUCTURE The management structure tends to vary from assessment to assessment. The NACCI established the National Assessment Synthesis Team (NAST) made up of experts from industry, academia, government laboratories, and nongovernmental organizations. The NAST, with its three co-chairmen, had substantial authority to guide the process, and their guidance was critical to the successes of the NACCI. The ACIA established a management structure consisting of various steering committees and local secretariats. The ACIA also benefited from having leadership with substantial understanding of the ability to incorporate lessons learned from previous assessment processes. Major decisions of the IPCC are taken by the plenary of government representatives, which elects 30 chairs and vice-chairs who make up the IPCC bureau. Each working group is supported by a technical support unit, and the overall Bureau is supported by a secretariat. The leadership in the ozone assessment was particularly effective due to both familiarity with the scientific issue and the political awareness necessary to communicate the scientific findings effectively (NRC 2005). The need for a strong leadership structure is self-evident and was a common thread running through the committee’s discussions with those responsible for conducting assessments. A general principle is that the decision-making structure within the assessment needs to be well articulated

OCR for page 99
Analysis of Global Change Assessments Lessons Learned from the outset. However, identifying the appropriate leadership is challenging, and the committee concurs with the findings of the National Research Council (NRC 2005) on the characteristics of good leadership: [Good leaders] are committed to progress and are capable of articulating a vision, entraining strong participants, promoting partnerships, recognizing and enabling progress, and creating institutional and programmatic flexibility. Good leaders facilitate and encourage the success of others. They are vested with authority by their peers and institutions, through title, an ability to control resources, or other recognized mechanisms. Without leadership, programmatic resources and research efforts cannot be directed and then redirected to take advantage of new scientific, technological, or political opportunities. (p. 48) The choice of leadership structures and individuals may not be straightforward, but it is crucial to the success of the endeavor, with significant implications for how effectively the assessment is conducted and how well it is received by the target audience and other stakeholders. Effective assessment leaders respond easily to a changing political environment, provide transparent and legitimate rationale for such a response, and provide consistent messages to participants. In the best of circumstances, individuals with appropriate scientific credentials will naturally emerge, who enjoy the confidence of both the political and the scientific communities, have experience in conducting successful assessments, and are willing to undertake the present one. Since the leaders commonly function as spokespeople for the process, decisions regarding leadership must consider implications for perceptions of objectivity, credibility, and legitimacy. Recommendation: The leadership and organizational structure of the assessment should be made clear, and the responsibilities of individuals and organizations well articulated. INTEGRATED ASSESSMENTS Degree and Nature of Integration Although multiple definitions of integrated assessments are being used by the community, the committee considers such assessments to result from a process that integrates social, biological, and physical sciences and engineering and allows interdisciplinary synthesis and analysis. Some integrated assessments are integrated after the fact, like the IPCC, and some are actually interdisciplinary and integrated from the beginning, such as the MA. Others, following a more restrictive use of the definition of integrated assessment, comprise a model that explicitly links the dynamics of

OCR for page 99
Analysis of Global Change Assessments Lessons Learned social, biological, and physical systems. All types allow understanding of complex interlinked phenomena and their implications, as well as building a stronger fabric for decision making. However, a well-integrated assessment is difficult to undertake, and managing them is far more difficult than is generally recognized. The IPCC is one of the few examples of an attempt at integrated assessment; however, it still develops core science findings in separate teams from the impact and response assessments. The requirement that the IPCC draw only on the peer-reviewed literature has increased its scientific credibility, but at the price of making the integration effort slower and less flexible. In synthesizing information from the literature it is difficult to tell whether the individual pieces are based on a set of common assumptions. To some extent the IPCC can address this problem by reviewing the results of integrated assessments conducted by individual research teams. In addition, difficulties in integrating across the three working groups also stem from the distinct paradigms each is working with. Another issue with integrated assessments is the difficulty of developing a common language between different disciplines, particularly between social and natural scientists. As mentioned before, such difficulties can be overcome if resources are devoted to capacity building and development of a common language between the various disciplines. These problems raise concerns about whether integrated assessments of global change can be conducted effectively at the international level. Because a fully integrated assessment is much more complex and difficult to achieve than assessment of a single issue, there has to be a clear reason why this approach is undertaken. The effectiveness of the early stratospheric ozone assessments in informing decisions demonstrates that not all assessments need to be fully integrated, although there are many benefits to working toward an integrated approach, including greatly enhanced potential to be policy relevant. An alternative approach is “nesting” specific assessments in a broader matrix to provide for a clear focus and illustrative examples at the smaller scale while allowing for generalized lessons in the larger frame. For example, an integrated assessment model may be much more easily developed around a specific decision-making process at a regional scale, such as water resource management. Consider the challenge of providing a credible assessment of the impact of climate change on a specific watershed in which the objective is to assess the availability of future water resources. The nature of climate projections, including the factors that control the spatial and temporal character of precipitation and evaporation are essential, but insufficient to define future water availability. For example, land-use and land-cover change will also have substantial impact on runoff and evaporation rates, and changes in the character of human waste streams (both air- and waterborne) will substantially influence water quality. Our ability to examine future water resource availability

OCR for page 99
Analysis of Global Change Assessments Lessons Learned requires an integrated assessment because the impacts and decisions that influence water are place-based, but the drivers of these impacts are also drawn from a much larger scale (e.g., climate change). Such an integrated assessment can best be completed with a “matrix” or “nested” approach, in which the large-scale drivers (e.g., scenarios or projections for future climate change) become one element of the assessment process that can serve as a foundation for a series of other regional or sector assessments. At a regional scale, the vast amount of place-based information, including the additional drivers (e.g., land-use change), can be incorporated into the analysis to provide a more comprehensive treatment of potential changes in water quality and quantity. Such an approach might include the use of regionally based mesoscale models that better address the spatial character of the watershed, detailed watershed models, regional observing and information systems, and projections of population growth and the evolution of human systems. Such a model can be used to illustrate both impacts and response options, and lessons from that model may be applied to other scales or decision-making processes. The degree and nature of the integration of the assessment represents a design decision that should be made with specific reference to the user’s needs and the purpose of the assessment. This approach is one way to ensure that broad-scale assessments can continue to be developed, while at the same time enhancing the relevance in individual applications where many resource decisions are made. Recommendation: The degree and nature of interdisciplinary integration of assessments should be chosen with specific reference to the users and purpose of the assessment and the resources needed to do integrated assessments well. Because fully integrated assessments are more readily done at a specific local decision-making scale, attempts should be made to nest them within a global assessment, which may not need to be fully integrated. Importance of Integrated Assessments for CCSP The assessment activities mandated by the 1990 GCRA necessitate some degree of integration in that the act requires reporting on the state of scientific understanding of global change, the effects of global change on a range of sectors, and future trends. It is not clear whether the 21 Synthesis and Assessment Products currently being conducted by CCSP to address this mandate will meet the needs of policy makers in the same way that a more integrated approach might. In particular, integrated assessments have the potential to better link understanding of global change phenomena with their impacts, thereby providing better information for decision making. At the same time, the committee recognizes that integrated assessments are

OCR for page 99
Analysis of Global Change Assessments Lessons Learned challenging. Incorporating a more integrated approach into some of CCSP’s assessment activities could provide an important opportunity to learn more about how to conduct more effective integrated assessments, while also producing integrated, societally relevant outcomes. Recommendation: The CCSP should invest in experimental applications of integrated assessments, with a specific focus on advising future applications of truly integrated, ongoing, interdisciplinary assessments in the United States. TREATMENT OF UNCERTAINTY One of the most difficult tasks in an assessment is the expression of uncertainty. In the case of climate change, uncertainties remain as part of the evolving understanding of various aspects of the greenhouse effect, its likely consequences, and the efficacy of various countermeasures. Some of these uncertainties will not be resolved for decades, if then. An effective characterization of uncertainty in assessments requires determining what kinds of uncertainty information would be useful for decision makers as well as developing quantitative or qualitative measures of uncertainty. While there is evidence that decision makers have an aversion to ambiguity, uncertainty is unavoidable in many decision-making contexts. Once decision makers understand that they are operating in an uncertain environment, they typically prefer that the conclusions of an assessment be accompanied by a description of the level and source of relevant uncertainties. The manner in which uncertainties are acknowledged and characterized will affect both the salience and the credibility of the assessment. Ways of addressing uncertainty include clearly identifying the uncertainties; characterizing and identifying the source and magnitude of the uncertainties; expert judgments of the level of confidence; and testing this sensitivity through the development of plausible future scenarios. For example, the IPCC has attempted to deal with uncertainty by using words to indicate judgment estimates of confidence (e.g., “virtually certain” denotes a greater than 99 percent chance that a result is true, “very likely” denotes a 90-99 percent chance, etc.). Alternatively, there are formal methods for eliciting expert judgments (Morgan and Henrion 1990), but these can be quite time consuming and, therefore, can only be applied selectively. Recommendation: Uncertainties should be well articulated in global change assessments to the extent they are understood, and the sources of the uncertainty should be described. There should be a deliberate effort to clarify the importance of alternative assumptions and to illustrate the impacts of uncertainties.

OCR for page 99
Analysis of Global Change Assessments Lessons Learned A CREDIBLE AND INDEPENDENT REVIEW PROCESS Assessments differ from more standard scientific publications, and therefore the typical science peer-review process needs modification. Assessments build on prior knowledge; identify recent advances and research needs; attempt to reach consensus on scientific debates; and in some cases, provide response options, including policy options. Because assessments may include policy-relevant information and even some value judgments in the case of impact assessments, assessment reviews need to be conducted to achieve salience, legitimacy, and credibility. In contrast, typical science peer review focuses solely on scientific credibility. Therefore, the review process should be consistent with the goals of the assessment and the type of assessment. For process assessments that focus only on the scientific understanding of the process, an expert peer review may suffice. However, assessments providing decision support or policy options, such as impact and response assessments, may require a broader review, involving stakeholders and decision makers, in particular. Many assessments that the committee examined have well-established review mechanisms. For the IPCC, the review process includes a peer review, followed by an expert and government review, and finally a review by the governments who are party to the United Nations Framework Convention on Climate Change. The report and the various summaries are then subject to acceptance by the governments, which meet in plenary for this approval process. This government review process has raised some issues regarding credibility due to the potential for government interference with the scientific conclusions. In contrast, the ACIA provides a model of how a government review can be conducted successfully without the resulting perception that governments influenced the scientific consensus. The government review process for the ACIA involved scientists, who had the ultimate authority over the scientific conclusions, and government representatives, who were given the editorial authority over policy options. In the example of the TEAP, no external review process was undertaken because all the key players were already involved and the assessment contained proprietary information. Because a well-designed review process has the potential to greatly enhance broad stakeholder ownership and the quality of the outcome, it is essential to the credibility, salience, and legitimacy of the assessment. As previously mentioned, an assessment review is distinct from a peer review in that it cannot be undertaken solely from the perspective of scientific credibility, but must also focus on issues of salience and legitimacy. The committee found that an effective approach is a staged review, such as employed in the ACIA, beginning with the scientific community, with subsequent involvement of governments and other relevant stakeholders. To ensure that legitimacy and credibility can be enhanced simultaneously, an approach should

OCR for page 99
Analysis of Global Change Assessments Lessons Learned be designed that addresses criticisms regarding government reviews that attempt to alter the report’s scientific conclusions inappropriately. This can be addressed by providing clear and transparent guidelines giving experts the ultimate editorial authority over the scientific conclusions in response to government reviews and comments. In addition, neutral review editors from a broad range of disciplines could function as referees to ensure that comments are responded to appropriately and that well-defined guidelines are followed to avoid the perception of government reviews altering scientific conclusions. Recommendation: An assessment review process should enhance salience and legitimacy in addition to credibility, by engaging interested and affected parties in the review process in addition to the expert community. The design of the review process should be adapted depending on whether it is a process, impact, or response assessment. The use of a well-balanced panel of review editors from a broad range of backgrounds should be considered to ensure that the review comments are responded to appropriately. In addition, a transparent mechanism for a legitimate and credible government review needs to be designed. DEVELOPING DECISION-SUPPORT APPLICATIONS Decision-support tools include a wide range of tools and models that link analyses, environmental and social data, and information about decisions and outcomes. They help decision makers understand the sensitivity of relevant systems, assess vulnerability, identify management alternatives, characterize uncertainties, and plan for implementation (Chen et al. 2004; Pyke and Pulwarty 2006). For example, regional tools were developed during the development of the NACCI that allow web-based access to assessment data to assist in making agricultural crop decisions. In its strategic plan (CCSP 2003), the CCSP identified the need for increased efforts to develop decision-support applications, a new emphasis that was lauded in the NRC review of the plan (NRC 2004). Adaptation to global change in general, and climate change in particular, requires that the institutional context of decisions be recognized in the development of decision-support tools as well as adaptation and mitigation activities. Assessments should be designed to be policy relevant but not policy prescriptive. For example, a response assessment may provide policy options and analysis describing possible policy outcomes but it should not prescribe which response to choose. There are many ways to ensure that decision-support efforts are properly focused and effective, but it will not be possible to support every type of decision at every scale. When selecting specific case studies to be nested within the broader assessment activity,

OCR for page 99
Analysis of Global Change Assessments Lessons Learned CCSP needs to be strategic about the kinds of decisions to support and the scale at which such support is most urgently needed. It is also important that sufficient resources be dedicated to supporting the development of decision-support tools, which is a relatively new area of emphasis for CCSP (NRC 2004). The critical issue in decision support is to provide useful, policy-neutral information targeted for use in particular sectors and for specific applications. For assessments intended to inform national- and international-level decisions about how to effectively manage the climate change risk, the information is being applied to issues that are apt to be highly politically charged. Thus, CCSP needs to be thoughtful about how it supports development of decision-support tools so that the information resulting from such tools is credible. The area of cost-benefit analysis is particularly challenging; for example, the IPCC has struggled with whether to conduct such analyses as part of the assessment or instead to synthesize existing analyses conducted by others. Some options would be for CCSP to (1) support the development of the needed decision-support tools; (2) encourage the appropriate science or modeling community to focus their efforts on the needs of policy makers, and then synthesize the results in a manner that will be useful to the policy-making community; or (3) commission the development and application of the requisite decision-support tools as part of the assessment process itself. An example of the third option is CCSP Synthesis and Assessment Product 2.1, in which existing tools are being used to develop new emission scenarios, analyze their impact on the energy system, and assess the costs to the economy. Recommendation: CCSP should foster and support the development of knowledge systems that effectively build connections between those who generate scientific information and the decision makers who are most likely to benefit from access to the knowledge that is generated. One approach is to support the development of decision-support tools and applications at various scales of decision making that can be used in the context of assessments. In doing so the CCSP should identify decision-making processes of high priority or broad application that address key regional or sectoral vulnerabilities, and then evaluate the decision-support needs in those applications. New analytical and predictive tools can then be devised that have direct benefits in specific assessment applications. EMPLOYING A NESTED MATRIX APPROACH Adaptive approaches are needed to continually integrate advances in knowledge into the policy context. Although it would be ideal to address each sector and region at the local, regional, and national scales while

OCR for page 99
Analysis of Global Change Assessments Lessons Learned assessing impacts and responses, it is unlikely that sufficient resources will be available to do this comprehensively on an ongoing basis. One way to address the resource issues associated with assessment is to build a broad conceptual framework or matrix linked to smaller-scale illustrative examples. For example, an assessment could be conducted at a national level, accompanied by selected localized case studies of impacts on specific sectors or implications for specific local decision making. The work on the broad themes and trends can be an ongoing effort, while individual, integrated, local, or sectoral assessments can be nested strategically in the broader research agenda. This will help develop an ongoing assessment program that has more coherence over time. An example of the application of the nested matrix approach is using global climate models to identify likely future changes in temperature and precipitation at the national and regional level that may result from climate change. By connecting such outputs to hydrologic models, it is possible to identify a range of likely impacts on runoff for specific watersheds and evaluate potential vulnerabilities for regions and sectors. Based on that information, specific regions or sectors that are identified as areas of high vulnerability can be selected for a more focused integrated assessment that includes the demographic and institutional context as well as physical parameters. At a regional scale, the vast amount of place-based information, including additional drivers (e.g., land-use change), can be incorporated into the analysis to provide a more comprehensive treatment of potential changes in water quality and quantity. Recommendation: CCSP should consider implementing a nested matrix concept in developing subsequent assessments.