2
A Risk-Characterization Framework

Difficult decisions are common for the Food and Drug Administration (FDA). Whether it is allocating scarce resources, deciding how to mitigate newly found risks, or deciding what investments in human capital, facilities, data, or analytic methods would be most useful, decision-makers in the FDA often need to integrate data of varied quality, recognize uncertainties, and make trade-offs to arrive at a decision. Public-health and public-safety concerns must be balanced with the economic realities of budgets and the political constraints of imposing new regulations. How a program is presented by the media and understood by the public is an important determinant of the acceptance and success of the program. Science and public preferences and perceptions must be considered if one is to understand the potential outcomes of different decision options. To inform the decision-making process, data of differing degrees of quality and robustness must be used and sometimes fed through an array of models of varied sophistication. Expert opinion must be used to interpret the relevance of available data and to solve problems on which available data are weak or nonexistent. Immovable deadlines can thwart uncompromising reliance on the most thorough analysis based on detailed quantitative data for a given decision.

To succeed in such an environment, FDA needs a framework within which alternatives can be defined and evaluated systematically. Although it is beyond the scope of the present study to provide a comprehensive decision-making procedure for FDA, the committee proposes a general framework for thinking about and characterizing the human-health dimensions of FDA decisions. Health consequences are only a subset of the large array of factors that must be considered for any given problem. However, they constitute a reasonable place to start the process of developing a decision framework inasmuch as such factors loom large in most FDA decisions and substantial work has already been done on methods for estimating the human-health consequences associated with various risks, hazards, and decisions.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 17
2 A Risk-Characterization Framework Difficult decisions are common for the Food and Drug Administration (FDA). Whether it is allocating scarce resources, deciding how to mitigate newly found risks, or deciding what investments in human capital, facilities, data, or analytic methods would be most useful, decision-makers in the FDA often need to integrate data of varied quality, recognize uncertainties, and make trade-offs to arrive at a decision. Public-health and public-safety concerns must be balanced with the economic realities of budgets and the political constraints of imposing new regulations. How a program is presented by the media and un- derstood by the public is an important determinant of the acceptance and success of the program. Science and public preferences and perceptions must be consid- ered if one is to understand the potential outcomes of different decision options. To inform the decision-making process, data of differing degrees of quality and robustness must be used and sometimes fed through an array of models of varied sophistication. Expert opinion must be used to interpret the relevance of avail- able data and to solve problems on which available data are weak or nonexistent. Immovable deadlines can thwart uncompromising reliance on the most thorough analysis based on detailed quantitative data for a given decision. To succeed in such an environment, FDA needs a framework within which alternatives can be defined and evaluated systematically. Although it is beyond the scope of the present study to provide a comprehensive decision-making pro- cedure for FDA, the committee proposes a general framework for thinking about and characterizing the human-health dimensions of FDA decisions. Health con- sequences are only a subset of the large array of factors that must be considered for any given problem. However, they constitute a reasonable place to start the process of developing a decision framework inasmuch as such factors loom large in most FDA decisions and substantial work has already been done on methods for estimating the human-health consequences associated with various risks, hazards, and decisions. 17

OCR for page 17
18 A Risk-Characterization Framework for Decision-Making at FDA The framework offered here provides a common language for describing potential public-health consequences of decisions, is designed to have wide ap- plicability among all FDA centers, and draws extensively on the well-vetted risk literature to define the relevant health dimensions for FDA decision-making. This chapter first provides a brief description of the proposed framework and the risk and decision contexts that influenced the committee’s approach. Next, the basis and definition of the risk attributes that characterize the framework are provided, and then some approaches for estimating the outcomes of decisions using the risk attributes are described. The chapter concludes with a discussion of how the output of the framework can be used to support decision-making. THE FRAMEWORK The risk-characterization framework is designed to be as general as possi- ble while providing consistent risk information in a way that can be used to sup- port the wide variety of decisions that FDA faces. It is intended to supplement and augment other risk-based and risk-informed approaches that are in use and under development by FDA, not to be a replacement or a one-size-fits-all pre- scription for conducting all risk-informed decision-making. Indeed, the commit- tee recognizes that the public-health-consequence factors highlighted in this framework will seldom be the only important considerations in the decision- making process, but they are almost always some of the key considerations. The U.S. Nuclear Regulatory Commission has also recognized that risks are not the only factors that must be taken into account in regulatory decision-making. It recently embraced the concept of risk-informed decision-making, which it de- fines as “the use of risk insights, along with other important information, to help in making decisions” (USNRC 2008, page 1-1). The committee’s framework focuses on risk information but also recognizes that other information will be relevant for most FDA decisions. The process is straightforward and involves three steps: Step 1. Identify and define the decision context: What decision options are being considered? What are the appropriate end points to evaluate and compare? Step 2. Estimate or characterize the public-health consequences of each option by using the risk attributes that are described below. The values of the risk attributes should be summarized in a table to facilitate comparison of the options. Step 3. Use the completed characterization as a way to compare decision options and to communicate their public-health consequences within the agency, to decision-makers, and to the public; use the comparison with other decision- relevant information to make informed decisions. Although the steps can be easily articulated, they involve thought and ef- fort to complete. The framework is not a cookbook and will require FDA to ex-

OCR for page 17
A Risk-Characterization Framework 19 ercise judgment in how it is used. As illustrated in the case studies that are pre- sented in the chapters that follow, successful implementation requires consis- tency (for example, use of a single set of attributes for every evaluation) and flexibility (for example, specific causes of death or illness that are considered important for a decision can be called out separately in the evaluation and sum- mary). Completing the attribute table (Step 2) may be relatively simple, or it may require substantial research and modeling or even additional data collection and analysis. It should be carried out with whatever level of rigor and effort is necessary and feasible for the decision being considered. Factors to be consid- ered in deciding how much effort to expend on the evaluation of public-health consequences include the timeframe in which the decision must be made and the relative importance of the public-health consequences compared with other key decision-making factors. If a decision must be made quickly and the public- health consequences are less important than other factors, it may be reasonable to complete the attribute table quickly on the basis of available information and judgment alone. However, if the decision options under consideration could lead to substantially different public-health consequences or if public-health conse- quences are highly uncertain or poorly understood and other factors are consid- ered less important, it may be worth substantial time and effort to develop more precise estimates for the attribute tables. The decision needs and the available resources should determine how much time and effort should be put into imple- menting the risk-characterization framework. CONTEXT PROVIDED BY RISK LITERATURE Science and Decisions: Advancing Risk Assessment (NRC 2009) lays out a “framework for risk-based decision-making” as part of its recommendations for improving risk assessment at the U.S. Environmental Protection Agency (EPA). Although that study was focused specifically on formal risk assessment, it sum- marized its approach into an overall decision framework that closely matches what the present committee concludes is appropriate for improving risk-based decision-making at FDA. Figure 2-1 illustrates the Science and Decisions framework. The three phases shown in Figure 2-1 correspond to the three steps in the framework listed above, but the emphasis of the present committee is somewhat different from that of the previous study. Specifically, the bulk of Science and Decisions focuses on the risk-assessment portion in phase II of the framework, as is highlighted in the figure. The framework proposed here focuses on the risk- characterization portion of the decision structure and adopts a more robust view of the importance of risk characterization in the overall process of making risk- informed decisions. Early descriptions of the risk-assessment process identify risk characteri- zation simply as a process of combining the exposure and dose-response ele- ments of a risk assessment to summarize and communicate results. In Figure

OCR for page 17
20 A Risk-Characterization Framework for Decision-Making at FDA 2-1, for example, risk characterization is placed in its traditional location as the last step in a risk assessment. Understanding Risk (NRC 1996, p. 16) laid out a challenge to that narrow interpretation and greatly expanded the scope and role that risk characterization should play in the overall process of making risk- informed decisions: We have concluded that the view of risk characterization as a summary is seriously deficient, and we propose a more robust construction. Risk char- acterization must be seen as an integral part of the entire process of risk decision making: what is needed for successful characterization of risk must be considered at the very beginning of the process and must to a great extent drive risk analysis. If risk characterization is to fulfill its pur- pose, it must (1) be decision-driven, (2) recognize all significant concerns, (3) reflect both analysis and deliberation, with appropriate input from the interested and affected parties, and (4) be appropriate to the decision. The present committee adopts the broader view of risk characterization and its relationship to risk-informed decision-making. PHASE I: PHASE II: PHASE III: PROBLEM FORMULATION PLANNING AND CONDUCT RISK MANAGEMENT AND SCOPING OF RISK ASSESSMENT Stage 1: Planning • For the given decision context, what are the attributes of assessments necessary to characterize risks of existing conditions and the effects on risk of proposed options? What level of uncertainty and variability analysis is appropriate? • What are the relative health or • What problems are associated environmental benefits of the with existing environmental proposed options? conditions? Stage 2: Risk Assessment • How are other decision- • If existing conditions appear making factors (technologies, to pose a threat to human or • Hazard Identification costs) affected by the proposed environmental health, what What adverse health or environmental effects options? options exist for altering those are associated with the agents of concern? conditions? • What is the decision, and its • Risk Characterization • Dose-Response Assessment justification, in light of benefits, • Under the given decision costs, and uncertainties in each context, what risk and other What is the nature and For each determining adverse effect, what is the option? technical assessments are magnitude of risk associated with relationship between dose and the probability of the necessary to evaluate the existing conditions? • How should the decision be occurrence of the adverse effect in the range of doses possible risk-management communicated? identified in the exposure assessment? options? What risk decreases (benefits) are associated with each of the • Is it necessary to evaluate the options? effectiveness of the decision? • Exposure Assessment Are any risks increased? What are • If so, how should this be done? the significant uncertainties? What exposures/doses are incurred by each population of interest under existing conditions? How does each option affect existing conditions and resulting exposures/doses? Stage 3: Confirmation of Utility NO YES • Does the assessment have the attributes called for in planning? • Does the assessment provide sufficient information to discriminate among risk-management options? • Has the assessment been satisfactorily peer reviewed? FORMAL PROVISIONS FOR INTERNAL AND EXTERNAL STAKEHOLDER INVOLVEMENT AT ALL STAGES • The involvement of decision-makers, technical specialists, and other stakeholders in all phases of the processes leading to decisions should in no way compromise the technical assessment of risk, which is carried out under its own standards and guidelines. FIGURE 2-1 A framework for risk-based decision-making that maximizes the utility of risk assessment. Source: NRC 2009, p. 243.

OCR for page 17
A Risk-Characterization Framework 21 The committee’s approach was also influenced by comparative risk analy- sis (CRA), which was first defined by EPA in Unfinished Business (EPA 1987). That report embraces the multidimensional nature of risk and ranks 31 identified environmental threats according to four attributes: cancer risk, noncancer risk, ecologic risk, and “welfare” effects. In a follow-up report, Reducing Risk (EPA SAB 1990), EPA’s Science Advisory Board endorsed the broad CRA approach and as a result spawned many applications of CRA at the office, region, state, and local levels (Minard 1996; Jones 1997). Those early CRA efforts led to questions about how best to facilitate comparisons and identify useful attributes for characterizing risks or risk-reduction opportunities. Progress on CRA method development continues, although its use remains relatively limited. In February 1994, a workshop organized by Resources for the Future for the president’s Office of Science and Technology Policy brought to- gether researchers in CRA with the goal of developing a systematic process for comparing risks among different federal agencies (Davies 1996). As part of that work, researchers at Carnegie Mellon University developed a framework for ranking risks that included both quantitative and qualitative measures of relevant programmatic attributes (Fischhoff 1995; Morgan et al. 1996). They included health-effect measures (such as morbidity and mortality) and psychometric measures that research shows play an important role in the evaluation of risks (such as fairness, scientific understanding, and uncertainty). That work spawned a series of research projects and papers that refined and applied the framework (see, for example, Morgan et al. 1999, 2001; Long and Fischhoff 2000; Morgan et al. 2000; DeKay et al. 2001; Florig et al. 2001; Willis et al. 2004, 2005; Fischhoff 2006; Gutiérrez et al. 2006; Bronfman et al. 2007, 2008a,b), including a discussion directly related to food safety (DeKay et al. 2005). The framework defined here builds on that work while embracing the no- tion that risk characterization should be decision-focused rather than restricted to ranking risks. The framework focuses on describing the potential effects of al- ternative decisions on health rather than on comparing different health and envi- ronmental hazards. It also focuses on the identification and use of a clear and consistent set of risk attributes relevant to public health to describe the effects of alternative decisions. Although the health outcomes of alternative decisions are not the only fac- tors influencing regulatory decision-making, such information is often highly useful in weighing the merits of alternative decisions. For example, in deciding whether to withdraw approval for a product for which a new adverse health ef- fect has been identified, knowledge of the extent, likelihood, and severity of the newly identified adverse health effect is important, but so is knowledge of the product’s benefits and of the effects that are likely to occur if the product is no longer available. An important question for the present committee was how to define a set of attributes that would be robust enough such that risk information relevant to the broad array of decisions that FDA faces could be adequately cap- tured and used consistently for risk-informed decision-making.

OCR for page 17
22 A Risk-Characterization Framework for Decision-Making at FDA DECISION CONTEXT As noted in Chapter 1, FDA provided the committee with 16 decision sce- narios (see Appendix C) and used them to characterize the types of decisions that it faces. FDA asked the committee to consider three types of decisions: mitigation-selection decisions, targeting decisions, and strategic-investment de- cisions (Bertoni 2010). For purposes of developing a decision-focused risk- characterization framework for FDA, the committee adopted FDA’s categoriza- tion of decisions. Other ways of categorizing decisions are possible, and some decisions that are within FDA’s authority are difficult to fit within the three categories defined here. For example, decisions about setting or modifying regulatory standards or establishing certification standards have the potential to affect numerous FDA- regulated products simultaneously, and the effects of such standards depend not only on the standards but on the response of the regulated industry to the stan- dards and on the effectiveness of enforcement actions. The committee has not explicitly addressed the standard-setting decisions in the present report, although the concepts presented here could be extended to address such decisions. Others have addressed the use of a comprehensive risk perspective in setting standards (Fischhoff 1984), including consideration of when standard-setting is preferable to case-by-case decision-making. Mitigation-Selection Decisions Mitigation-selection decisions require FDA to choose among two or more options that are available to reduce or mitigate identified risks. The first step in applying the framework to such decisions is to identify and specify the mitiga- tion decision and the decision options to be evaluated and compared. For exam- ple, in the case study described in Chapter 3, the decision context is a hypotheti- cal situation in which a vaccine side effect is believed to be occurring at a rate higher than expected. Two decision options are considered—remove or do not remove the vaccine from the market. In this case study, the committee deter- mined that it was important to consider the health consequences of both the vac- cine and the underlying disease that the vaccine is intended to prevent. The decision scenarios provided to the committee by FDA included sev- eral additional examples of mitigation-selection decisions, some of which may require evaluating and comparing a larger number or greater variety of options beyond simple binary choices of recall or not. For example, for product-recall decisions, there might be different levels or types of recall that could be exe- cuted, each of which could lead to different health consequences; there might be mitigation options that combine recall of unused product with other risk- reduction options for product already in use. For risk-mitigation decisions with several options, each of the viable risk-mitigation approaches needs to be speci- fied, and the consequences of each must be estimated by using the risk attributes

OCR for page 17
A Risk-Characterization Framework 23 defined below. More generally, almost any decision that results in a change in product availability or the extent of use of a particular product could be charac- terized as a mitigation-selection decision; the outcomes will differ in the degree of use of the product, and the resulting health consequences can be characterized and compared by using the framework proposed here. Targeting Decisions Targeting decisions as described by FDA are essentially priority-setting or resource-allocation decisions. They appear to be made primarily within a pro- gram and focus on how a particular resource, such as inspection capability, should be allocated among a broad set of products or product categories. In this type of decision, the options or alternatives theoretically available to FDA are vast: virtually any amount of a resource could be allocated to any of a subset of the identified products or product categories, and the only constraint would be total resource availability. The large array of options will need to be narrowed judiciously before the effects of the resulting decisions can be evaluated in de- tail. For each product, product category, or other item for which resources might be targeted, FDA will need to identify the resources that it is considering allocating. In some cases, the decision may be an “all or none” decision, such as a decision to inspect a facility or not to inspect it. In others, it may be a particular level of effort and resources, such as a specific rigor of inspection of a facility. Generally, each potential target of a resource allocation would require evaluation of at least two decisions: allocating “x” level of resources vs allocating “y” level of resources. The difference between the values of the risk attributes of an allocation of “x” vs “y” defines the public-health benefit of allocation of resources at those levels to each product or product category. As noted by FDA (Bertoni 2010), targeting decisions can be seen as simi- lar to the risk-ranking questions that are historically the main focus of CRA studies and that remain of interest in some FDA centers. For example, the Cen- ter for Food Safety and Applied Nutrition (CFSAN) has developed a “Fresh Produce Risk Ranking Tool,” which can be used to rank produce-pathogen pairs in order of risk (FoodRisk.org 2011). Similarly, a detailed risk assessment of Listeria monocytogenes conducted in 2003 led to a ranking of ready-to-eat foods by the likelihood and frequency of illnesses (FDA/USDA 2003). It is unclear, however, how risk ranking can or should be used to make risk-informed decisions. When different allocations of resources are expected to be equally effective in reducing the risks associated with the product or product category in question, a ranking of potential targets according to unreduced risk may provide sufficient decision support. For example, in making decisions about what facilities to inspect in a specific year, CFSAN may be able to use a risk ranking of facilities in their current conditions and operations with the assump- tion that additional resources will be equally effective in reducing the risks at

OCR for page 17
24 A Risk-Characterization Framework for Decision-Making at FDA any facility. Given that assumption, resources should be focused on the highest- risk facilities. To create such a risk ranking, FDA would need to estimate the public-health consequences of the food products from each facility that is a can- didate for inspection—or, more generally, for each potential target of the re- source allocation—on the basis of its best assessment of the current conditions and operations of the facilities. The resulting ranked list could then be used as the basis of resource-allocation decisions. However, the assumption of equal effectiveness is not often appropriate in real decision problems. If the situation is more complicated—that is, if the risks associated with some products or product categories being evaluated can be re- duced much more effectively or at much lower cost than the risks associated with others—a narrowly focused risk ranking will not be an appropriate tool for making the decisions. In such situations, FDA should define the decision options more explicitly, as discussed above. For example, the risk associated with a food from a particular facility could be determined for two inspection regimes. The basis of the status quo inspection procedures could be extracted from an existing risk-ranking exercise. Then FDA would also need to evaluate the public-health consequences associated with the food from the facility if a decision were made to allocate a different level of inspection resources to the facility; this evaluation would include changes in any risks affected by the different inspection. The same exercise would then need to be performed for any other food-facility com- bination that is being considered in the resource-allocation decision. That ap- proach would lead to a “risk-change” ranking rather than a risk ranking and would enable FDA to take into account the possibility that some inspections are more effective than others in reducing risks. If resources are to be reallocated from existing inspection programs to new programs, the increases in risk due to the reduction in funding in one must be balanced against the decreases in risk due to increased funding in the other. In Chapter 4, the committee illustrates the use of the framework to evalu- ate three food categories for their current health effects. The results of the evaluation could be used to rank the food categories by risk, and the ranking could be used, subject to the limitations described above, to support a targeting decision for a hypothetical example of food-safety inspections. In Chapter 6, the committee illustrates a targeting decision that involves a more explicit compari- son of the effects of different allocations of resources. Strategic-Investment Decisions Strategic-investment decisions are longer-term internal decisions about where FDA should invest its resources to enable better risk-informed decision- making. For example, additional research can be conducted, more data collected and analyzed, and new tools developed to increase understanding of and reduce uncertainties about a wide array of potential risks. Investments in such activities typically enable better evaluations and more informed decisions, so they provide

OCR for page 17
A Risk-Characterization Framework 25 value to FDA and the American public. However, they are tied only indirectly to the more common metrics of public health, and it can be difficult to demonstrate their benefits and to decide which strategic investments are most worth while. The decision-analytic concept of value of information can be used to evaluate the potential benefits of strategic-investment decisions about data or information collection. This concept is that new information has decision- relevant value only if it could lead to actions different from the actions that would be taken without the information. That is, information and strategic in- vestments to collect it have value only if they have the potential to change deci- sions and thus potentially improve outcomes. For example, if a physician orders a diagnostic test but would recommend the same treatment regardless of the test results, the information has no decision-relevant value. In contrast, if the treat- ment recommendation depends on the diagnosis and the test results will be used to differentiate among diagnoses, the information potentially has decision- relevant value. The risk-characterization framework can provide some of the key elements necessary for a formal value-of-information analysis: definition of a decision context (Step 1) and characterization of decision outcomes (Step 2). Some extensions to the framework would be necessary to implement a value-of- information analysis fully, and those are explored in several of the case studies that follow. In the case study in Chapter 5, the committee uses its approach to evaluate the public-health consequences of a strategic-investment decision to enhance postmarket surveillance of two medical devices. Application of the committee’s proposed approach to this decision category is the most challenging both be- cause identifying the strategic-investment decision options to be evaluated is complicated and because evaluating the effects of long-term strategic invest- ments on public health is difficult. The case study allows some comparisons to be made but probably represents a narrower scope of strategic-investment op- tions than what FDA would consider. CHARACTERIZING THE HEALTH CONSEQUENCES OF DECISION OPTIONS In the risk-assessment paradigm, risks have traditionally been character- ized by a single attribute, such as the number of deaths that could occur as a result of the hazard being evaluated or the probability that an exposed individual will experience an identified adverse effect. Several complications arise with that simple characterization of risks. Multiple outcomes, such as illnesses and deaths, are often of interest. Uncertainty in the outcomes makes simple charac- terization and reporting problematic: reporting only expected fatalities will ob- scure information about the other possible outcomes that are important to deci- sion-makers and policy-makers. Providing only single values for health outcomes may convey far greater certainty than is appropriate. Characterizing the public-health consequences requires recognizing and accommodating the

OCR for page 17
26 A Risk-Characterization Framework for Decision-Making at FDA multidimensional nature of risk and the uncertainties involved in estimating the outcomes. The Many Dimensions of Risk The comparison of alternative decisions may seem easier when a single metric is used and when the harm and the benefit are measured with the same metric. For example, in deciding whether the association of clozapine with the potential fatal side effect of agranulocytosis is sufficient to warrant keeping clozapine off the market, the number of deaths caused by clozapine-induced agranulocytosis can be directly compared with the number of suicides that clozapine would be expected to prevent. However, those estimates may differ greatly in their uncertainty, and other benefits of clozapine, such as improved ability to function, would also have to be considered in such a decision. When the harm and the benefit of alternative decisions cannot be meas- ured with the same metric, decision-making is less straightforward, and the im- portance of informing decision-makers of the various harms and benefits be- comes more important. For example, infliximab is highly effective in reducing pain and improving function in people who have rheumatoid arthritis, but its immunosuppressive properties may permit serious and possibly fatal infections to emerge. Deciding whether the risk of a serious adverse effect outweighs the benefits in increased mobility and quality of life is not a technical or scientific question but rather a question of personal and societal values. An integral part of the committee’s proposed approach is that it characterizes various effects explic- itly so that decision-makers can make informed decisions that account for them rather than combining them into a single metric based on implicit weightings (see Box 2-1). The risk-assessment paradigm gives rise to one set of attributes for charac- terizing the health risks associated with FDA-regulated products or FDA deci- sions: factors that are used to determine the number, type, and rate of occurrence of adverse health effects (including deaths) that could result from implementa- tion of a particular decision option. Those attributes are exposed population, mortality, and morbidity, each described in more detail below. Studies of risk perception and public attitudes about risks have consis- tently shown that although the mortality and morbidity components in risk esti- mation are important, they are not the only things that people care about when they think about risks and about risk acceptability (Slovic 1992). Numbers of deaths and illnesses or injuries matter, but so do other factors, such as whether a risk is voluntary, how much control a person has over risks, and whether the hazard being considered has the potential to lead to a large number of simulta- neous deaths. The list of factors hypothesized to be important in understanding risk is long (see Appendix E). Relatively few risk attributes have been studied in

OCR for page 17
A Risk-Characterization Framework 27 BOX 2-1 Summary Measures vs Detailed Characterization of Public-Health Consequences In the public health literature, it has become common to measure and communicate the burden of disease by using a summary measure that com- bines mortality and morbidity into a single value. Some measures, such as disability adjusted life years (DALYs), have been used to evaluate the global burden of disease (Lopez et al. 2006). Others, such as quality adjusted life years (QALYs), have been promoted as tools for comparing the cost effec- tiveness of various medical interventions (Broome 1993) or risk reducing regulatory strategies (IOM 2006). Each summary measure necessarily em- beds a set of value judgments about different levels of health impairment, and each provides a narrow, rather than a robust, characterization of the risk. The committee finds that a richer characterization of the public-health consequences of decision options is needed for routine comparison of deci- sion options. As described in Improving Risk Communication, “reducing different kinds of hazard to a common metric (such as number of fatalities per year) and presenting comparisons only on that metric have great poten- tial to produce misunderstanding and conflict and to engender mistrust of expertise” (NRC 1989, p. 52). detail, but the available studies show that the various factors are highly corre- lated on a relatively small number of dimensions, and it has become common to refer to these dimensions as reflecting key “factors” that characterize risk per- ceptions (Slovic 1992; NRC 1996). The first dimension captures the quantitative aspects, such as the number and type of adverse health outcomes. The second dimension is roughly characterized by the degree of knowledge about the haz- ards or risks and how well they are understood. The third dimension is charac- terized by variables that are less easily summarized by a single category: whether a risk is voluntary, how much control an exposed person has over it, the ease with which it can be reduced, and whether it is catastrophic (that is, can lead to multiple simultaneous deaths or injuries). Understanding Risk urged that risk characterization explicitly include con- sideration of those additional factors and described a number of ways in which they could be taken into account in decision processes (NRC 1996, p. 65). That approach has also been embraced as a key part of CRA and in the risk-ranking studies mentioned above. The risk-ranking studies include both quantitative in- formation about the number of deaths and injuries and more qualitative informa- tion reflecting some of the variables identified in the risk-perception literature. Little general guidance, however, has been offered about what risk attributes might be widely appropriate. Investigators in the studies cited above typically acknowledge that attribute selection is complicated and then choose attributes

OCR for page 17
A Risk-Characterization Framework 35 tainties to decision-makers and policy-makers without introducing biases, such as overconfidence and false precision. Using the Best Available Information to Develop Estimates The data required to support the evaluation of the public-health conse- quences of different decisions can vary widely in quality and availability. For some problems, large volumes of epidemiologic data may be available to de- scribe the mortality risk of some medical product. For others, there may be an array of detailed computer models that attempt to characterize the morbidity- reduction effectiveness of various inspection procedures. For still others, only scant data or models may be available, and it may be necessary to rely solely on the judgment of knowledgeable professionals to develop such estimates. Often, a hybrid approach will be necessary: using experts to identify relevant information from studies or models and relying on the experts to interpret the data or model results in the context of the decisions being evaluated. If data are available, their relevance to the required estimates must be considered carefully. For example, the available data may indicate deaths that were temporally associated with a product or device but were not caused by the product or device; judgment will be required to interpret such data appropriately and to estimate the mortality associated with the different decision options being evaluated. Critical decisions often must be made when uncertainty about some rele- vant factors remains; in those cases, judgment is required. Lack of easily acces- sible data does not mitigate the need to develop the best possible estimates of public-health consequences of products being evaluated; in some cases, care- fully assessed expert judgment may be the only option available for ensuring that decision-makers have all the relevant information that is available to make decisions. Delaying a critical decision until additional information can be col- lected may be an option in some circumstances, but it should not be considered the “default” decision when uncertainty exists. Instead, all relevant options, in- cluding the option to delay a decision until later or to allocate scarce resources to additional data collection or model development, should be evaluated on the basis of the best available information, including expert judgment, about the public-health consequences of the decision. Many risk-assessment and evaluation tools are available and are described in detail in other studies, including National Research Council and FDA studies described above. Box 2-3 summarizes current perspectives at EPA about how to quantify uncertain outcomes, including a hierarchy of approaches and the role of expert judgment. It is not within the present committee’s scope to evaluate, compare, or recommend specific approaches or models for risk quantification. The committee simply urges FDA to bring the best available data and expertise to bear on the evaluation that would be consistent with the importance of the decision being evaluated and the time and resources available to complete the assessment.

OCR for page 17
36 A Risk-Characterization Framework for Decision-Making at FDA BOX 2-3 Methods for Estimating Uncertain Quantities A white paper written for EPA outlines a general hierarchy of methods that can be used to estimate quantities when there is substantial uncertainty about their “true” values (Frey et al. 2003). Four general categories of methods are described without any implied preferences or priorities: Statistical methods based on empirical data, which use classical sta- tistics to draw inferences from “hard” data alone. Statistical methods based on judgment, in which expert judgments and Bayesian approaches to statistical analysis are included, often in com- bination with “hard” data. Other quantitative methods that involve approaches not based on probability theory, such as interval methods, fuzzy methods, and meta- analytic methods. Qualitative methods that can be used when key aspects of uncer- tainty cannot be captured by quantitative methods (for example, uncertainty caused by problem formulation or the existence of competing models). Of those approaches, the first is widely used and accepted in risk as- sessment, the second is widely used outside the risk-assessment commu- nity and is expanding in use and acceptance in the risk-assessment com- munity, and the third and fourth approaches are not generally used. EPA discussed the use of expert elicitation (or expert judgment) and made the following comment: Expert elicitation is recognized as a powerful and legitimate quantitative method for characterization of uncertainty and for providing probabilistic distributions to fill data gaps where additional research is not feasible. The academic and research community, as well as numerous review bodies, have recognized the limitation of empirical data for characteri- zation of uncertainty and have acknowledged the potential for using [expert elicitation] for this purpose. In Science and Judgment in Risk Assessment [NRC 1994] the NAS [National Academy of Sciences] rec- ognized that for “parameter uncertainty, enough objective probability data are available in some cases to permit estimation of the probability distribution. In other cases, subjective probabilities might be needed.” In this “Blue Book” report, the NAS further recognized the “difficulties of using subjective probabilities in regulation” and identified perceived bias as one major impediment; but, noted that “in most problems real or perceived bias pervades EPA’s current point-estimate approach.” In addition, the NAS stated that “there can be no rule that objective prob- ability estimates are always preferred to subjective estimates, or vice versa” (EPA 2009, p. 5). Thus, EPA recognized the importance of using statistical methods based on judgment to derive probability estimates and emphasized that objective methods are not always preferable to subjective methods.

OCR for page 17
A Risk-Characterization Framework 37 Characterizing Uncertainty Although substantial effort has been devoted to motivating and guiding the inclusion of uncertainty in policy analysis throughout the federal government (see Box 2-4), including it is difficult, and resistance remains high. Individual researchers and policy analysts are reluctant to make quantitative estimates with- out what they feel to be “sufficient” data, and they are even more reluctant to quantify the uncertainties inherent in their domain. It is always possible to col- lect more data and do more analyses to try to develop “better” estimates, but there will always be uncertainty, and decisions often must be made on the basis of existing information. Quantifying what is known and what is not known (the uncertainty) is an important way to ensure that decisions are as well informed as possible. The difficulty of developing best estimates and uncertainties when data are sparse is exacerbated by the norm of using probability distributions for quantifi- cation. Probability is not a natural way of thinking about uncertainty for every- one, and often efforts are made to sidestep the problem by using qualitative or categorical measures of uncertainty, such as “likely,” “very unlikely,” and “pos- sible.” That approach may make the risk-characterization task more palatable, but it also makes it considerably less useful. The very ambiguity that provides comfort makes the task of communicating and comparing uncertainties difficult. Flexible definitions that vary from domain to domain make cross-cutting analy- sis impossible. Is the probability value associated with a "high likelihood" of death equal (either numerically or cognitively) to the probability value associ- ated with a "high likelihood" of rain? The effort needed to define the uncertainty categories unambiguously and in sufficient detail would certainly approach if not exceed the effort needed to follow one of the standardized approaches of assessing a distribution directly. In addition, for some problem domains, assess- ing the complete distribution is not necessary; a few key points selected from the underlying distribution will describe the underlying uncertainty sufficiently to permit the necessary comparisons and analyses. That approach is demonstrated in the case studies that follow this chapter. Options for succinctly describing a probability distribution vary; however, the shorthand notation should be able to provide an indication of a distribution’s first three moments—central tendency, spread, and skew—and should be easily assessable from data, model output, or expert judgment. In the case studies that follow, that task is accomplished by using three values: the distribution’s 5th, 50th (median), and 95th percentiles. The 5th percentile is the numerical value of the quantity being estimated (such as the number of deaths) that bounds the lower 5% tail of the distribution; for example, the probability that the actual value will be lower than the 5th percentile is 0.05. The 95th percentile is the numerical value that will be exceeded with a probability of 0.05; that is, it is the boundary of the upper 5% tail of the distribution. The median value is a measure of the distribution’s central tendency and generally represents the

OCR for page 17
38 A Risk-Characterization Framework for Decision-Making at FDA BOX 2-4 Importance of Characterizing Uncertainty The assessment, quantification, and communication of uncertainty are essential, as has been recognized in the risk-assessment literature since at least 1983. Numerous high-level government advisory panels have dis- cussed the importance of capturing and presenting uncertain values. Risk Assessment in the Federal Government: Managing the Process (NRC 1983) formalized the integration of uncertainty and risk into policy analysis. The guidance provided in that early report has been expanded on in a se- ries of reports (NRC 1993, 1994, 1996) and recently reinforced in Science and Decisions: Advancing Risk Assessment (NRC 2009, p. 93), which states at the beginning of a chapter on uncertainty and variability that “char- acterizing uncertainty and variability is key to the human health risk- assessment process, which must engage the best available science in the presence of uncertainties and difficult-to-characterize variability to inform risk-management decisions.” In recent guidelines on probabilistic risk as- sessment, the U.S. Nuclear Regulatory Commission states that risk- informed decision-making requires the “appropriate consideration of uncer- tainty…in the analyses used to support the decision and in the interpretation of the findings of those analyses” (USNRC 2009, page iii). Over the last 25 years, EPA has developed a series of guidelines that detail how uncertainty should be integrated into regulatory policy-making and evaluation (EPA 1984, 1995, 2000, 2004). Estimating the Public Health Benefits of Proposed Air Pollution Regulations (NRC 2002) offers advice to EPA on the importance of characterizing uncertainty even when data are sparse or lacking: EPA should move the assessment of uncertainty from its ancillary analyses into its primary analyses to provide a more realistic depiction of the overall degree of uncertainty. This shift will entail the develop- ment of probabilistic, multiple-source uncertainty models based not only on available data but also on expert judgment … Uncertainty should be described as completely and as realistically as possible for all regula- tory options, recognizing that regulatory action might be necessary in the presence of substantial uncertainty. The regulatory decision proc- ess will be better informed by a fair assessment of the uncertainty and a realistic evaluation of the likely reductions in that uncertainty attain- able through further research (NRC 2002, p. 11). Explicit evaluation and presentation of uncertainty in risk assessments reduces the problem of false precision, makes risk characterization more informative, and increases the credibility of any ensuing risk communication. “best estimate” of the uncertain quantity. The probability that a value falls below the median is equal to the probability that it falls above the median. The spread of the distribution is indicated by the difference between the 5th and 95th per-

OCR for page 17
A Risk-Characterization Framework 39 centiles. The skew of the distribution is indicated by comparing the difference between the 5th percentile and the median and the difference between the me- dian and the 95th percentile. For a symmetric distribution, those differences will be equal. For a negative skew, the first difference will be larger; for a positive skew, the second difference will be larger. If the three percentiles are relied on to characterize a distribution, the as- sessment procedure for experts is relatively simple and well tested. With training and practice, an analyst can elicit an accurate representation of the three percen- tiles from experts by using easily described thought experiments, simple tools, and standard protocols. Numerous references describe a variety of approaches for assessing quantitative estimates of uncertain quantities, including examples of how expert assessments have been used to support a wide array of decisions (Merkhofer 1987; Cooke 1991; Hora 2007; EPA 2009; Jenni and van Luik 2010). It is important to note that a viable approach will require the considera- tion of the tails of the distribution. Assessed distributions are often too narrow, but considering rarer and more extreme events leads to assessed distributions that represent true uncertainties better. Although the three percentiles are precisely defined, their cognitive inter- pretation should not be lost. The goal is to have a risk characterization that in- cludes representation of the uncertainty in the size of the exposed population, in the number of deaths that may occur, and in the number of injuries or illnesses that may occur under various decision options. Rather than ask simply for a range or for low and high estimates of those quantities, the committee suggests using the concepts associated with the percentiles of a distribution as a way to ensure consistent assessment and interpretation of the low and high values. The 5th percentile represents a value below which the actual value is not likely to fall. Similarly, the 95th percentile represents a value above which the actual value is not likely to fall. The estimates are not upper and lower bounds, but values that indicate limits beyond which results would be surprising. Fixating on the precise values for the percentiles could cause unnecessary anxiety and com- municate false precision where none is intended or needed. Uncertainty in the health consequences of alternative decisions should be included in the characterization of risks in this framework. Specifically, a best estimate, a high estimate, and a low estimate—corresponding to the three per- centiles described above—should be determined for the size of the exposed population, mortality, and each of the three morbidity estimates. The other at- tributes could also be described by using similar distributional measures al- though the importance of doing so is less. When assessing the probabilities as- sociated with ability to detect, control, or mitigate, ranges may be appropriate instead of point values. For example, the probability that an informed institution will be able to reduce or mitigate adverse health effects associated with a par- ticular risk could be judged to be 0.80-0.90. The committee notes that there is likely to be a relationship between the quality of data available to develop the necessary estimates and the uncertainty

OCR for page 17
40 A Risk-Characterization Framework for Decision-Making at FDA in the estimates. Specifically, when relevant data are lacking, uncertainty about each of the outcomes will probably be high (that is, a large spread). USING THE FRAMEWORK TO SUPPORT DECISION-MAKING The framework proposed here is similar to the framework for risk-based decision-making described in Science and Decisions (see Figure 2-1) in its em- phasis on structuring and conducting risk assessments and risk characterization in the broader context of supporting agency decisions. Science and Decisions concludes by making recommendations for improving risk-based decision- making at EPA; although some of those recommendations may be applicable to FDA, the present committee was charged not with evaluating the risk-based decision-making processes at FDA but with developing a robust approach for characterizing public-health consequences. Those consequences are an impor- tant consideration in FDA decisions, and the attributes proposed in this frame- work provide such a characterization. Using this framework throughout the agency will ultimately lead to robust and consistent characterization of public- health consequences. How the information is used will ultimately be determined by FDA. Pub- lic-health consequences constitute only one of numerous factors—such as eco- nomic, social, and political factors—that FDA must consider when making deci- sions, and the committee neither expects nor recommends that the attributes in this report should form the sole basis of such decisions. However, careful and consistent evaluation of the public-health consequences of various options is an essential component of good decision-making, and there are several ways in which the risk characterizations described here could be used in decision- making. One approach was outlined in the committee’s letter report, which focused on developing a risk ranking (see Appendix A). The letter report describes an approach for involving stakeholders in an exercise to rank risks that used a con- sistent set of attributes. Specifically, it involved ranking risks on the basis of judgment and the described characteristics of those risks or ranking risks on the basis of a mathematical combination of the attribute scores and judgments about the importance of the different attributes, that is, using the logic of multiattribute utility theory (Keeney and Raiffa 1976). The approach described in the letter report is based on the risk-ranking literature described previously and could be adapted and applied in the decision-focused framework proposed here. The risk characterization of Step 2 of the present framework is also com- patible with several other approaches to decision-making—both more quantita- tive approaches and more inclusive processes. The analytic-deliberative process described in Understanding Risk (NRC 1996) includes scientists, public offi- cials, and other interested and affected parties in an iterative process in which all parties are involved in every step of the decision process. If FDA adopted that type of approach to risk-based decision-making, the risk attributes defined here

OCR for page 17
A Risk-Characterization Framework 41 could provide a starting point for the risk-characterization aspects of the analy- sis. Retaining the flexibility to refine the risk characterization throughout the process, however, is key to the analytic-deliberative process. At perhaps a different extreme, FDA may determine that for some deci- sions only a single public-health consequence is relevant. In that case, the other elements of the risk characterization could be ignored. Alternatively, the quanti- tative public-health consequences as a group may be considered relevant, and it may be considered useful to combine the attributes in the risk characterization into a single metric to compare options. Such integrative measures may be easier to develop, and especially valuable, when there is a logical set of mathematical relationships between the various attributes. For example, two rare causes of morbidity associated with a particular product might reasonably be considered additive in their effects; other attributes, such as the size of a population affected and the rate of adverse effects or the number of adverse effects and the likeli- hood that harm could be mitigated, would combine in a multiplicative fashion. Proven mathematical approaches for combining attributes into a single integra- tive measure (Keeney and Raiffa 1976; Fischhoff et al. 1984) could also be ap- plied here. However, the attributes defined here were chosen to provide a robust risk characterization rather than an easy mathematical combination, so develop- ing a single integrative measure would require careful consideration of the rela- tionship between the various attributes and their relative importance. As dis- cussed in Box 2-1, using integrative measures alone has several disadvantages, but relying only on the full list of attributes without integration can make consis- tency a challenge. Each case study that follows concludes with a discussion of how the pre- sent framework could be used to support a specific decision, but they are offered only for illustration. It remains for FDA to decide how to use the information, and whether and how to combine it with other decision-relevant factors. FLEXIBILITY AND EVOLUTION OF THE FRAMEWORK The risk-characterization framework proposed here is a flexible system. Although it is intended to capture elements of risk that are applicable across the broad array of risk-related decisions that FDA makes, it is not necessarily a comprehensive set of public-health-related factors relevant for every such deci- sion, and it clearly does not aim to include factors that are unrelated to public- health outcomes. The framework is not proposed as a “one-size-fits-all” method for all risk-related decisions at FDA for all time. Just as the framework and the risk attributes were refined through development of the case studies in this re- port, the committee expects that some elements of the framework will continue to evolve as FDA gains experience in using it. Additional risk characteristics may be identified for subsets of decisions and added to the list for those types of decisions. Furthermore, the existing definitions of attributes may be modified to

OCR for page 17
42 A Risk-Characterization Framework for Decision-Making at FDA yield more interesting insights, and some attributes may be dropped altogether if they are not found to be useful. REFERENCES AHRQ (Agency for Healthcare Research and Quality). 2005. U.S. Valuation of the Eu- roQol EQ-5D Health States. Agency for Healthcare Research and Quality, Rock- ville, MD. December 2005 [online]. Available: http://www.ahrq.gov/rice/EQ5D proj.htm [accessed Oct. 25, 2010]. Bertoni, M.J. 2010. Opening Remarks. Presentation at the 5th Meeting on Ranking FDA Product Categories Based on Health Consequences, Phase II, February 3, 2010, Washington, DC. Bronfman, N.C., L.A. Cifuentes, M.L. DeKay, and H.H. Willis. 2007. Accounting for variation in the explanatory power of the psychometric paradigm: The effects of aggregation and focus. J. Risk Res. 10(4):527-554. Bronfman, N.C., L.A. Cifuentes, and V.V. Gutierrez. 2008a. Participant-focused analysis: Explanatory power of the classic psychometric paradigm in risk perception. J. Risk Res. 11(6):735-753. Bronfman, N.C., E.L. Vazquez, V.V. Gutierrez, and LA. Cifuentes. 2008b. Trust, accep- tance and knowledge of technological and environmental hazards in Chile. J. Risk Res. 11(6):755-773. Broome, J. 1993. QALYs. J. Public Econ. 50(2):149-167. Cooke, R.M. 1991. Experts in Uncertainty: Opinion and Subjective Probability in Sci- ence. K. Shrader-Frechette, ed. New York: Oxford University Press. Davies, J.C. 1996. Comparing Environmental Risks: Tools for Setting Government Pri- orities. Washington, DC: Resources for the Future. DeKay, M.L., H.K. Florig, P.S. Fischbeck, M.G. Morgan, K.M. Morgan, B. Fischhoff, and K.E. Jenni. 2001. The use of public risk ranking in regulatory development. Pp. 208-230 in Improving Regulation: Cases in Environment, Health, and Safety, P.S. Fischbeck, and R.S. Farrow, eds. Washington, DC: Resources for the Future. DeKay, M.L., P.S. Fischbeck, H.K. Florig, M.G. Morgan, K.M. Morgan, B. Fischoff, and K.E. Jenni. 2005. Judgement-Based Risk Ranking for Food Safety Pp. 198-226 in Toward Safer Food: Perspectives on Risk and Priority Setting. S.A Hoffman and M.R. Taylor, eds. Washington, DC: RFF Press. EPA (U.S. Environmental Protection Agency). 1984. Risk Assessment and Management: Framework for Decision Making. EPA 600/9-85-002. U.S. Environmental Protec- tion Agency, Washington, DC. December 1984. EPA (U.S. Environmental Protection Agency). 1987. Unfinished Business: A Compara- tive Assessment of Environmental Problems. EPA/230/2-87/025. Office of Policy, Planning, and Evaluation, U.S. Environmental Protection Agency, Washington, DC. EPA (U.S. Environmental Protection Agency). 1995. Policy for Risk Characterization at the U.S. Environmental Protection Agency. U.S. Environmental Protection Agency, Washington, DC. March 1995 [Appendix A in EPA 2000]. EPA (U.S. Environmental Protection Agency). 2000. Risk Characterization Handbook. EPA 100-B-00-002. Office of Science Policy, Office of Research and Development, U.S. Environmental Protection Agency. December 2000 [online]. Available: http://itp-pfoa.ce.cmu.edu/docs/RChandbk_excerpts_re_exposure.pdf [accessed Oct. 25, 2010].

OCR for page 17
A Risk-Characterization Framework 43 EPA (U.S. Environmental Protection Agency). 2004. An Examination of EPA Risk As- sessment Principles and Practices, Staff Paper Prepared for the U.S. Environmental Protection Agency by Members of the Risk Assessment Task Force. EPA/100/B- 04/001. Office of the Science Advisor, U.S. Environmental Protection Agency, Washington, DC. March 2004 [online]. Available: http://www.epa.gov/osa/pdfs/ ratf-final.pdf [accessed Oct. 25, 2010]. EPA (U.S. Environmental Protection Agency). 2009. Expert Elicitation Task Force White Paper, External Review Draft. Science Policy Council, U.S. Environmental Protec- tion Agency. January 6, 2009 [online]. Available: http://www.epa.gov/osa/pdfs/ elicitation/Expert_Elicitation_White_Paper-January_06_2009.pdf [accessed Feb. 23, 2011]. EPA SAB (U.S. Environmental Protection Agency Science Advisory Board). 1990. Re- ducing Risk: Setting Priorities and Strategies for Environmental Protection. SAB- EC 90-021. Science Advisory Board, U.S. Environmental Protection Agency [online]. Available: http://yosemite.epa.gov/sab/sabproduct.nsf/28704D9C420FCB C1852573360053C692/$File/REDUCING+RISK++++++++++EC-90-021_90021 _5-11-1995_204.pdf [accessed Mar. 17, 2011]. EuroQoL Group. 1990. EuroQol – A new facility for the measurement of health-related quality of life. Health Policy 16(3):199-208. FDA/USDA (U.S. Food and Drug Administration and U.S. Department of Agriculture). 2003. Listeria monocytogenes Risk Assessment: Interpretive Summary. Center for Food and Applied Nutrition, U.S. Food and Drug Administration, and Food Safety and Inspection Service, U.S. Department of Agriculture. September 2003 [online]. Available: http://www.fda.gov/Food/ScienceResearch/ResearchAreas/RiskAssessmentSafetyAsse ssment/ucm185291.htm [accessed Feb. 23, 2010]. Fischhoff, B. 1984. Setting standards: A systematic approach to managing public health and safety risks. Manage. Sci. 30(7):823:843. Fischhoff, B., S.R. Watson, C. Hope. 1984. Defining risk. Policy Sciences. 17(2):123- 139. Fischhoff, B. 1995. Ranking risks. Risk Health Saf. Environ. 6:189-200. Fischhoff, B. 2006. Cognitive processes in stated preference methods. Pp. 937-968 in Handbook of Environmental Economics, Vol. 2. Valuing Environmental Changes, K.G. Mäler, and J.R. Vincent, eds. Amsterdam: Elsevier. Florig, H.K., M.G. Morgan, K.M. Morgan, K.E. Jenni, B. Fischoff, P.S. Fischbeck, and M.L. DeKay. 2001. A deliberative method for ranking risks (I): Overview and test bed development. Risk Analysis. 21(5):913. FoodRisk.org. 2011. FDA’s Fresh Produce Risk Ranking Tool. Joint Institute for Food Safety and Applied Nutrition, College Park, MD [online]. Available: http://www.foodrisk.org/exclusives/RRT/ [accessed Feb.11, 2011]. Frey, C., D. Crawford-Brown, Z. Junyu, and D. Loughlin. 2003. Hierarchy of Methods to Characterize Uncertainty: State of Science of Methods for Describing and Quanti- fying Uncertainty. EPA Contract No. 68-D-00-265. Prepared for Office of Air Quality Planning and Standards, U.S. Environmental Protection Agency, Research Triangle Park, NC. August 12, 2003 [online]. Available: http://www4.ncsu.edu/ ~frey/reports/Frey_CrawfordBrown_Zheng_Loughlin_2003.pdf [accessed Feb. 23, 2010]. Gutiérrez, V.V., L.A. Cifuentes, and N.C. Bronfman. 2006. The influence of information delivery on risk ranking by lay people. J. Risk Res. 9(6):641-655.

OCR for page 17
44 A Risk-Characterization Framework for Decision-Making at FDA Hora, S.C. 2007. Eliciting probabilities from experts. Pp. 129-153 in Advances in Deci- sion Analysis: From Foundations to Applications, W. Edwards, R.F. Miles, Jr., and D. von Winterfeldt, eds. New York: Cambridge University Press. IOM (Institute of Medicine). 2006. Valuing Health for Regulatory Cost-Effectiveness Analysis. Washington, DC: The National Academies Press. IOM (Institute of Medicine). 2009. Initial National Priorities for Comparative Effective- ness Research. Washington, DC: The National Academies Press. Jenni, K.E. 1997. Attributes for Risk Evaluation. Ph.D. Dissertation, Carnegie Mellon University, Pittsburgh, PA. Jenni, K.E., and A. van Luik. 2010. Assessment of expert judgments for safety analyses and performance assessment of geologic repository systems. Pp. 580-609 in Geo- logical Repository Systems for Safe Disposal of Spent Nuclear Fuels and Radioac- tive Waste, J. Ahn, and M.J. Apted, eds. Cambridge: Woodhead Publishing. Jones, K. 1997. A Retrospective on Ten Years of Comparative Risk. Prepared for Ameri- can Industrial Health Council, Washington, DC, by Green Mountain Institute for Environmental Democracy, Montpelier, VT. January 24, 1997 [online]. Available: http://www.heartland.org/custom/semod_policybot/pdf/4728.pdf [accessed March 17, 2011]. Keeney, R.L. and H. Raiffa. 1976. Decisions with Multiple Objectives: Preferences and Trade-Offs. New York: Wiley. Long, J. and B. Fischoff. 2000. Setting risk priorities: A formal model. Risk Analysis. 20(3):339-352. Lopez, A.D., C.D. Mathers, M. Ezzati, D.T. Jamison, and C.J.L. Murray, eds. 2006. Global Burden of Disease and Risk Factors. Washington, DC: World Bank [online]. Available: http://www.dcp2.org/pubs/GBD [accessed Feb. 14, 2011]. Merkhofer, M.W. 1987. Quantifying judgmental uncertainty: Methodology, experiences, and insights. IEEE T. Syst. Man Cyb. 17(5):741-752. Minard Jr., R.A. 1996. Can States Make a Market for Environmental Goals? Pp. 243-280 in Linking Science and Technology to Society's Environmental Goals. National Research Council. Washington, DC: The National Academies Press. Morgan, M.G., B. Fischhoff, L. Lave, and P. Fischbeck. 1996. A proposal for ranking risks within federal agencies. Pp. 111-148 in Comparing Environmental Risks: Tools for Setting Government Priorities, J.C. Davies, ed. Washington, DC: Re- sources for the Future. Morgan, K.M., M.L. DeKay, and P.S. Fischbeck. 1999. A multi-attribute approach to risk prioritization. Risk Policy Report 6(6):38-40. Morgan, M.G., H.K. Florig, M.L. DeKay, and P. Fischbeck. 2000. Categorizing risks for risk ranking. Risk Analysis. 20(1):49-58. Morgan, K.M., M.L. DeKay, P.S. Fischbeck, M.G. Morgan, B. Fischhoff, and H.K. Florig. 2001. A deliberative method for ranking risks (II): Evaluation of validity and agreement among managers. Risk Anal. 21(5):923-937. NRC (National Research Council). 1983. Risk Assessment in the Federal Government: Managing the Process. Washington, DC: National Academy Press. NRC (National Research Council). 1989. Improving Risk Communication. Washington, DC: National Academy Press. NRC (National Research Council). 1993. Pesticides in the Diets of Infants and Children. Washington, DC: National Academy Press. NRC (National Research Council). 1994. Science and Judgment in Risk Assessment. Washington, DC: National Academy Press.

OCR for page 17
A Risk-Characterization Framework 45 NRC (National Research Council). 1996. Understanding Risk: Informing Decisions in a Democratic Society. Washington, DC: National Academy Press. NRC (National Research Council). 2002. Estimating the Public Health Benefits of Pro- posed Air Pollution Regulations. Washington, DC: The National Academies Press. NRC (National Research Council). 2009. Science and Decisions: Advancing Risk As- sessment. Washington, DC: National Academy Press. Slovic, P. 1987. Perception of risk. Science 236(4799):280-285. Slovic, P. 1992. Perception of risk: Reflections on the psychometric paradigm. Pp. 117- 152 in Social Theories of Risk, S. Krimsky, and D. Golding, eds. New York: Prae- ger. USNRC (U.S. Nuclear Regulatory Commission). 2008. Risk-Informed Decisionmaking for Nuclear Material and Waste Applications: Revision 1. U.S. Nuclear Regulatory Commission, Washington, DC [online]. Available: http://www.hss.energy.gov/nu clearsafety/ns/rawg/docs/RIDM_2008_2.pdf [accessed Apr. 13, 2011]. USNRC (U.S. Nuclear Regulatory Commission). 2009. Guidance on the Treatment of Uncertainties Associated with PRAs in Risk-Informed Decision Making Main Re- port, Volume 1. NUREG-1855. U.S. Nuclear Regulatory Commission, Washing- ton, DC [online]. Available: http://www.nrc.gov/reading-rm/doc-collections/nure gs/staff/sr1855/v1/sr1855v1.pdf [accessed Apr. 13, 2011]. Willis, H.H., M.L. DeKay, M.G. Morgan, H.K. Florig, and P.S. Fischbeck. 2004. Eco- logical risk ranking: Development and evaluation of a method for improving pub- lic participation in environmental decision making. Risk Analysis. 24(2):363-378. Willis, H.H., M.L. DeKay, B. Fischoff, and M.G. Morgan. 2005. Aggregate, disaggre- gate, and hybrid analyses of ecological risk perceptions. Risk Analysis. 25(2):405- 428.