7
Conclusions and Recommendations

For the reasons presented in this report, the committee concludes that the bulletin proposed by the Office of Management and Budget (OMB 2006) is fundamentally flawed and recommends that it be withdrawn. Although the committee fully supports the goal of increasing the quality and objectivity of risk assessment in the federal government, it agrees unanimously that the OMB bulletin would not facilitate federal agencies in reaching this goal. The committee also agrees that OMB should encourage the federal agencies to describe, develop, and coordinate their own technical risk assessment guidance. Therefore, the committee recommends that after additional study of current agency practices and needs, a different type of risk assessment bulletin be issued by OMB. It should outline goals and general principles of risk assessment designed to enhance the quality, efficiency, and consistency of risk assessment in the federal government. It should direct the agencies to develop technical guidance that would implement the general principles, be consistent with each agency’s legislative mandates and missions, and draw on the expertise that exists in federal agencies and other organizations. The technical guidance developed or identified by the agencies should be peer-reviewed and contain procedures for ensuring agency compliance with the guidance. Although OMB should determine whether the technical guidance fully addresses the general principles, it should not be involved in the development or peer review of agency technical guidance. The committee strongly recommends that agencies addressing similar hazards or risks work together to develop common technical guidance for risk assessment. In that way, the appropriate consistency would be achieved in the federal government in risk assessment practices.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 105
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget 7 Conclusions and Recommendations For the reasons presented in this report, the committee concludes that the bulletin proposed by the Office of Management and Budget (OMB 2006) is fundamentally flawed and recommends that it be withdrawn. Although the committee fully supports the goal of increasing the quality and objectivity of risk assessment in the federal government, it agrees unanimously that the OMB bulletin would not facilitate federal agencies in reaching this goal. The committee also agrees that OMB should encourage the federal agencies to describe, develop, and coordinate their own technical risk assessment guidance. Therefore, the committee recommends that after additional study of current agency practices and needs, a different type of risk assessment bulletin be issued by OMB. It should outline goals and general principles of risk assessment designed to enhance the quality, efficiency, and consistency of risk assessment in the federal government. It should direct the agencies to develop technical guidance that would implement the general principles, be consistent with each agency’s legislative mandates and missions, and draw on the expertise that exists in federal agencies and other organizations. The technical guidance developed or identified by the agencies should be peer-reviewed and contain procedures for ensuring agency compliance with the guidance. Although OMB should determine whether the technical guidance fully addresses the general principles, it should not be involved in the development or peer review of agency technical guidance. The committee strongly recommends that agencies addressing similar hazards or risks work together to develop common technical guidance for risk assessment. In that way, the appropriate consistency would be achieved in the federal government in risk assessment practices.

OCR for page 105
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget The committee arrived at its position after extensive discussion and deliberate consideration of many factors, including primarily the great variations in risk assessments among and within federal agencies and the fact that the expertise in risk assessment in the federal government resides, for the most part, in the agencies or with those with whom the agencies work. Risk assessment is not a monolithic process or a single method. All risk assessments share some common principles, but their application varies widely among domains. Different technical issues arise in assessing the probability of exposure to a given dose of a chemical, of a malfunction of a nuclear power plant or air-traffic control system, or of the collapse of an ecosystem or a dam. And different technical issues arise in assessing the consequences of an accidental release from a nuclear power facility and an accidental release of a pesticide. Risk assessment is not a field peopled with all-purpose experts. There are some with expertise in toxicology, decision analysis, dose-response assessment, ecologic risk assessment, engineering, and exposure assessment. In industry, some firms that specialize in one domain would not take on work in another. Federal agencies have staff familiar with the issues that are relevant to their missions; agencies without resident expertise have contractors with whom they have been working or associations to which they can turn. One size does not fit all, nor can one set of technical guidance make sense for the heterogeneous risk assessments undertaken by federal agencies. Although the bulletin reflects that diversity and attempts to meet it with frequent references to “where appropriate” or “where feasible,” the committee concludes that this approach is not workable for the agencies. As stated above, the committee strongly recommends that technical guidance be produced by the agencies and that agencies dealing with the same or similar hazards work together to produce common guidance to ensure an appropriately consistent approach. As noted above, the committee agrees that there is room for improvement in risk assessment practices in the federal government and that additional guidance would help “to enhance the technical quality and objectivity of risk assessments prepared by federal agencies.” However, the bulletin conveys the impression that risk assessments can and should achieve total objectivity. Although any scientific work should be free of bias, scientifically accurate, and based on reliable evidence, risk assessments cannot be wholly objective, because some important assumptions and judgments are based on policy or statutes. The committee strongly

OCR for page 105
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget concludes that OMB should limit its efforts to stating goals and general principles of risk assessment and to directing the agencies to develop technical guidance consistent with the goals and principles. The committee has not provided suggestions for specific goals and principles in this report, because that was beyond the scope of its task. CONCLUSIONS Other conclusions that led to the committee’s position that the OMB bulletin should be withdrawn are provided below. Three overarching conclusions are especially important. In view of the diversity of risk assessment responsibilities and proficiencies in the federal government, it would be difficult, if not impossible, to produce a single detailed technical guidance document that would be applicable to all federal agencies. New guidance that departs from established risk assessment principles and practices and is not supported by the current state of the science is unlikely to achieve the goals stated in the bulletin. Without baseline assessments of current risk assessment practices, needs, and capacities for improvement in the federal agencies, neither OMB nor the committee can make informed judgments on the kinds of guidance needed to reach the goals set forth in the bulletin and the related resources required to achieve that end. Conclusions that are related to specific aspects of the proposed bulletin are provided below. In some general respects, the bulletin’s requirements for risk assessments (for example, the call for balanced presentations of data and for explicit justification of scientific conclusions) are consistent with previous reports, including those cited in the bulletin. However, other aspects of the bulletin are inconsistent with previous reports in important ways. For example, it adopts a new definition of risk assessment and ignores, without explaining, the important impact that risk assessment policies have on the process, such as the need for consistent defaults and for clear criteria for moving away from the defaults. Without explicit and clear direction on such matters, agency risk assessments are more susceptible to being manipulated to achieve a predetermined result. The bul-

OCR for page 105
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget letin’s call for formal analyses of uncertainties and for undefined “central or expected estimates” may, in the absence of adequate peer-reviewed technical guidance on the evaluation and expression of uncertainties, result in risk characterizations of reduced, rather than enhanced, quality. Those are serious concerns because any attempt to advance the practice of risk assessment that does not reflect the state of the art on these topics is likely to produce the opposite effect. The proposed definition of risk assessment in the OMB bulletin departs without justification from long-established concepts and practices, including those developed by National Research Council (NRC) and other expert committees and endorsed in existing peer-reviewed guidelines. In particular, the proposed definition broadens the definition of risk assessment to include components of risk assessment, such as hazard assessment and exposure assessment. Such a broadening, which treats different procedures under the same name, is needlessly confusing. More important, several of the standards proposed in the bulletin are not applicable to individual components of risk assessment. The committee also disagrees with defining risk assessment as a document; risk assessment is a process from which documents can result. The dominating theme of the bulletin and its supplementary information is improving the quality of risk assessments undertaken by federal agencies, but the stated goals do not all support this theme. The goals stated in the bulletin and the supplementary information emphasize efficiency in the conduct of risk assessment activities more than quality. The discussion of the range of risk estimates and central estimates in the proposed bulletin is incomplete and confusing. A central estimate and risk range might be misleading when sensitive populations are of primary concern. Those numerical quantities are meaningful only in the context of some distribution characterizing variable traits or uncertainties. The choice of summary statistics cannot be a blanket prescription but must reflect the specific context. The description of uncertainty and variability in the bulletin is simplistic. It does not recognize the complexities of different types of risk assessments or the need to tailor uncertainty analysis to an agency’s particular needs. There is no scientific consensus to support the bulletin’s universal prescriptions for how uncertainty should be evaluated. The bulletin’s treatment of adverse effects is simplistic and too restrictive. Effects chosen for risk assessment may be adverse effects, precursor effects, or nonadverse effects. The point of departure to be chosen in a risk assessment depends on a number of factors, such as the

OCR for page 105
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget questions being addressed, the scientific information available, and an understanding of the underlying mechanisms for the effect of interest. The bulletin is silent on several important aspects of the risk assessment process. Specifically, it gives little attention to risk assessments for which the end point is major failure of engineered systems, to sensitive populations, to the often decisive role of risk assessment policy in choices regarding default options, to the integral role of risk communication, and to risk assessment standards for stakeholder assessments submitted for use in the rule-making process. The bulletin also fails to explain the basis for exempting risk assessments associated with licensing and approval processes. Although risk assessment and risk management are closely related and it is desirable to build links between them, the committee agrees with accepted practice that they are distinct. The bulletin blurs the important distinction between them by setting risk assessment standards related to risk mitigation and comparative-risk activities usually regarded as risk management. Risk assessors should not be required to undertake what have been traditional risk management functions, such as identifying alternative mitigation strategies. The bulletin claims that it avoids addressing risk communication in any detail, but it includes quite specific guidance on this topic. The guidance provided is not well informed or consistent with previous expert panel reports. In general, the bulletin takes the outmoded view that risk communication is mainly a matter of disseminating key findings after a risk assessment has been completed and not the contemporary view that it is a continuing discussion among risk assessors, risk managers, and stakeholders from start to finish. The more objectionable risk communication guidance in the bulletin includes instructions to the agencies always to communicate ranges of plausible estimates and always to compare assessed risks with other familiar risks—guidance that is not consistent with relevant research literature. Although OMB has not constructed a baseline reflecting current agency risk assessment practices, the committee concludes on the basis of agency comments and its own knowledge of risk assessment practices that there are aspects of the bulletin that could be beneficial but that the cost—in staff resources, timeliness of completing risk assessments, and other factors—are likely to be substantial. Overall, the committee concludes that, while varied and uncertain to some extent, the potential for negative impacts on the practice of risk assessment in the federal gov-

OCR for page 105
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget ernment is very high if the currently proposed bulletin were to be implemented. RECOMMENDATIONS The committee offers OMB the following recommendations to consider in developing a new risk assessment bulletin. After withdrawing the current bulletin and before proceeding further, OMB should produce a description of current agency risk assessment practices and resources and the likely effects (both benefits and costs) of changing those practices. Before mandating substantial changes in agency risk assessment practices, OMB should ensure that sufficient funds and staffing are available on a continuing basis to support the agencies in their risk assessment responsibilities. Adequate staffing and funding are prerequisites to the kind of risk assessment envisioned in the bulletin. OMB should ensure that any government-wide risk assessment bulletin takes full account of and makes allowance for variations among agencies with respect to the types of risk assessments they engage in, the resources they have to devote to risk assessments, and their proficiency in risk assessment generally. Any guidance on risk assessment should provide a definition of risk assessment that is compatible with previous NRC documents and guidelines of other expert organizations; does not include information documents or individual components of risk assessment, such as hazard or exposure assessment; preserves the clear conventional distinctions between risk assessment and risk management; and refers to a process, not a document. OMB should develop goals for risk assessment that emphasize the central objective of enhanced scientific quality and the complementary objectives of efficiency and consistency among agencies evaluating the same or similar risks. The goals should support the production of risk assessments that provide clear, relevant, and scientifically sound information for policy-makers. OMB should develop general principles for risk assessment that are fully consistent with the recommendations provided by previous committees of NRC and those of other expert organizations. The committee recommends that the affected federal agencies develop their own

OCR for page 105
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget technical risk assessment guidelines that are consistent with the OMB general principles. The committee strongly recommends that discussion of uncertainty and variability, presentation of risk results, definition of adversity, and other similar topics be reserved for the technical guidance to be developed by the agencies. REFERENCES OMB (U.S. Office of Management and Budget). 2006. Proposed Risk Assessment Bulletin. Released January 9, 2006. Washington, DC: Office of Management and Budget, Executive Office of the President [online]. Available: http://www.whitehouse.gov/omb/inforeg/proposed_risk_assessment_bulletin_010906.pdf [accessed Oct. 11, 2006].

OCR for page 105
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget This page intentionally left blank.