|
||||||||||||||||
|
||||||||||||||||
The following HTML text is provided to enhance online readability. Many aspects of typography translate only awkwardly to HTML. Please use the page image as the authoritative form to ensure accuracy. 2
|
||||||||||||||||
 |
 |
Page 17 |
 |
 |
 |
|
| ||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 17
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
2
A Risk-Characterization Framework
Difficult decisions are common for the Food and Drug Administration (FDA). Whether it is allocating scarce resources, deciding how to mitigate newly found risks, or deciding what investments in human capital, facilities, data, or analytic methods would be most useful, decision-makers in the FDA often need to integrate data of varied quality, recognize uncertainties, and make trade-offs to arrive at a decision. Public-health and public-safety concerns must be balanced with the economic realities of budgets and the political constraints of imposing new regulations. How a program is presented by the media and understood by the public is an important determinant of the acceptance and success of the program. Science and public preferences and perceptions must be considered if one is to understand the potential outcomes of different decision options. To inform the decision-making process, data of differing degrees of quality and robustness must be used and sometimes fed through an array of models of varied sophistication. Expert opinion must be used to interpret the relevance of available data and to solve problems on which available data are weak or nonexistent. Immovable deadlines can thwart uncompromising reliance on the most thorough analysis based on detailed quantitative data for a given decision.
To succeed in such an environment, FDA needs a framework within which alternatives can be defined and evaluated systematically. Although it is beyond the scope of the present study to provide a comprehensive decision-making procedure for FDA, the committee proposes a general framework for thinking about and characterizing the human-health dimensions of FDA decisions. Health consequences are only a subset of the large array of factors that must be considered for any given problem. However, they constitute a reasonable place to start the process of developing a decision framework inasmuch as such factors loom large in most FDA decisions and substantial work has already been done on methods for estimating the human-health consequences associated with various risks, hazards, and decisions.
OCR for page 18
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
The framework offered here provides a common language for describing potential public-health consequences of decisions, is designed to have wide applicability among all FDA centers, and draws extensively on the well-vetted risk literature to define the relevant health dimensions for FDA decision-making. This chapter first provides a brief description of the proposed framework and the risk and decision contexts that influenced the committee’s approach. Next, the basis and definition of the risk attributes that characterize the framework are provided, and then some approaches for estimating the outcomes of decisions using the risk attributes are described. The chapter concludes with a discussion of how the output of the framework can be used to support decision-making.
THE FRAMEWORK
The risk-characterization framework is designed to be as general as possible while providing consistent risk information in a way that can be used to support the wide variety of decisions that FDA faces. It is intended to supplement and augment other risk-based and risk-informed approaches that are in use and under development by FDA, not to be a replacement or a one-size-fits-all prescription for conducting all risk-informed decision-making. Indeed, the committee recognizes that the public-health-consequence factors highlighted in this framework will seldom be the only important considerations in the decision-making process, but they are almost always some of the key considerations. The U.S. Nuclear Regulatory Commission has also recognized that risks are not the only factors that must be taken into account in regulatory decision-making. It recently embraced the concept of risk-informed decision-making, which it defines as “the use of risk insights, along with other important information, to help in making decisions” (USNRC 2008, page 1-1). The committee’s framework focuses on risk information but also recognizes that other information will be relevant for most FDA decisions.
The process is straightforward and involves three steps:
Step 1. Identify and define the decision context: What decision options are being considered? What are the appropriate end points to evaluate and compare?
Step 2. Estimate or characterize the public-health consequences of each option by using the risk attributes that are described below. The values of the risk attributes should be summarized in a table to facilitate comparison of the options.
Step 3. Use the completed characterization as a way to compare decision options and to communicate their public-health consequences within the agency, to decision-makers, and to the public; use the comparison with other decision-relevant information to make informed decisions.
Although the steps can be easily articulated, they involve thought and effort to complete. The framework is not a cookbook and will require FDA to ex-
OCR for page 19
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
ercise judgment in how it is used. As illustrated in the case studies that are presented in the chapters that follow, successful implementation requires consistency (for example, use of a single set of attributes for every evaluation) and flexibility (for example, specific causes of death or illness that are considered important for a decision can be called out separately in the evaluation and summary). Completing the attribute table (Step 2) may be relatively simple, or it may require substantial research and modeling or even additional data collection and analysis. It should be carried out with whatever level of rigor and effort is necessary and feasible for the decision being considered. Factors to be considered in deciding how much effort to expend on the evaluation of public-health consequences include the timeframe in which the decision must be made and the relative importance of the public-health consequences compared with other key decision-making factors. If a decision must be made quickly and the public-health consequences are less important than other factors, it may be reasonable to complete the attribute table quickly on the basis of available information and judgment alone. However, if the decision options under consideration could lead to substantially different public-health consequences or if public-health consequences are highly uncertain or poorly understood and other factors are considered less important, it may be worth substantial time and effort to develop more precise estimates for the attribute tables. The decision needs and the available resources should determine how much time and effort should be put into implementing the risk-characterization framework.
CONTEXT PROVIDED BY RISK LITERATURE
Science and Decisions: Advancing Risk Assessment (NRC 2009) lays out a “framework for risk-based decision-making” as part of its recommendations for improving risk assessment at the U.S. Environmental Protection Agency (EPA). Although that study was focused specifically on formal risk assessment, it summarized its approach into an overall decision framework that closely matches what the present committee concludes is appropriate for improving risk-based decision-making at FDA. Figure 2-1 illustrates the Science and Decisions framework.
The three phases shown in Figure 2-1 correspond to the three steps in the framework listed above, but the emphasis of the present committee is somewhat different from that of the previous study. Specifically, the bulk of Science and Decisions focuses on the risk-assessment portion in phase II of the framework, as is highlighted in the figure. The framework proposed here focuses on the risk-characterization portion of the decision structure and adopts a more robust view of the importance of risk characterization in the overall process of making risk-informed decisions.
Early descriptions of the risk-assessment process identify risk characterization simply as a process of combining the exposure and dose-response elements of a risk assessment to summarize and communicate results. In Figure 2-1,
OCR for page 20
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
for example, risk characterization is placed in its traditional location as the last step in a risk assessment. Understanding Risk (NRC 1996, p. 16) laid out a challenge to that narrow interpretation and greatly expanded the scope and role that risk characterization should play in the overall process of making risk-informed decisions:
We have concluded that the view of risk characterization as a summary is seriously deficient, and we propose a more robust construction. Risk characterization must be seen as an integral part of the entire process of risk decision making: what is needed for successful characterization of risk must be considered at the very beginning of the process and must to a great extent drive risk analysis. If risk characterization is to fulfill its purpose, it must (1) be decision-driven, (2) recognize all significant concerns, (3) reflect both analysis and deliberation, with appropriate input from the interested and affected parties, and (4) be appropriate to the decision.
The present committee adopts the broader view of risk characterization and its relationship to risk-informed decision-making.
FIGURE 2-1 A framework for risk-based decision-making that maximizes the utility of risk assessment. Source: NRC 2009, p. 243.
OCR for page 21
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
The committee’s approach was also influenced by comparative risk analysis (CRA), which was first defined by EPA in Unfinished Business (EPA 1987). That report embraces the multidimensional nature of risk and ranks 31 identified environmental threats according to four attributes: cancer risk, noncancer risk, ecologic risk, and “welfare” effects. In a follow-up report, Reducing Risk (EPA SAB 1990), EPA’s Science Advisory Board endorsed the broad CRA approach and as a result spawned many applications of CRA at the office, region, state, and local levels (Minard 1996; Jones 1997). Those early CRA efforts led to questions about how best to facilitate comparisons and identify useful attributes for characterizing risks or risk-reduction opportunities.
Progress on CRA method development continues, although its use remains relatively limited. In February 1994, a workshop organized by Resources for the Future for the president’s Office of Science and Technology Policy brought together researchers in CRA with the goal of developing a systematic process for comparing risks among different federal agencies (Davies 1996). As part of that work, researchers at Carnegie Mellon University developed a framework for ranking risks that included both quantitative and qualitative measures of relevant programmatic attributes (Fischhoff 1995; Morgan et al. 1996). They included health-effect measures (such as morbidity and mortality) and psychometric measures that research shows play an important role in the evaluation of risks (such as fairness, scientific understanding, and uncertainty). That work spawned a series of research projects and papers that refined and applied the framework (see, for example, Morgan et al. 1999, 2001; Long and Fischhoff 2000; Morgan et al. 2000; DeKay et al. 2001; Florig et al. 2001; Willis et al. 2004, 2005; Fischhoff 2006; Gutiérrez et al. 2006; Bronfman et al. 2007, 2008a,b), including a discussion directly related to food safety (DeKay et al. 2005).
The framework defined here builds on that work while embracing the notion that risk characterization should be decision-focused rather than restricted to ranking risks. The framework focuses on describing the potential effects of alternative decisions on health rather than on comparing different health and environmental hazards. It also focuses on the identification and use of a clear and consistent set of risk attributes relevant to public health to describe the effects of alternative decisions.
Although the health outcomes of alternative decisions are not the only factors influencing regulatory decision-making, such information is often highly useful in weighing the merits of alternative decisions. For example, in deciding whether to withdraw approval for a product for which a new adverse health effect has been identified, knowledge of the extent, likelihood, and severity of the newly identified adverse health effect is important, but so is knowledge of the product’s benefits and of the effects that are likely to occur if the product is no longer available. An important question for the present committee was how to define a set of attributes that would be robust enough such that risk information relevant to the broad array of decisions that FDA faces could be adequately captured and used consistently for risk-informed decision-making.
OCR for page 22
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
DECISION CONTEXT
As noted in Chapter 1, FDA provided the committee with 16 decision scenarios (see Appendix C) and used them to characterize the types of decisions that it faces. FDA asked the committee to consider three types of decisions: mitigation-selection decisions, targeting decisions, and strategic-investment decisions (Bertoni 2010). For purposes of developing a decision-focused risk-characterization framework for FDA, the committee adopted FDA’s categorization of decisions.
Other ways of categorizing decisions are possible, and some decisions that are within FDA’s authority are difficult to fit within the three categories defined here. For example, decisions about setting or modifying regulatory standards or establishing certification standards have the potential to affect numerous FDA-regulated products simultaneously, and the effects of such standards depend not only on the standards but on the response of the regulated industry to the standards and on the effectiveness of enforcement actions. The committee has not explicitly addressed the standard-setting decisions in the present report, although the concepts presented here could be extended to address such decisions. Others have addressed the use of a comprehensive risk perspective in setting standards (Fischhoff 1984), including consideration of when standard-setting is preferable to case-by-case decision-making.
Mitigation-Selection Decisions
Mitigation-selection decisions require FDA to choose among two or more options that are available to reduce or mitigate identified risks. The first step in applying the framework to such decisions is to identify and specify the mitigation decision and the decision options to be evaluated and compared. For example, in the case study described in Chapter 3, the decision context is a hypothetical situation in which a vaccine side effect is believed to be occurring at a rate higher than expected. Two decision options are considered—remove or do not remove the vaccine from the market. In this case study, the committee determined that it was important to consider the health consequences of both the vaccine and the underlying disease that the vaccine is intended to prevent.
The decision scenarios provided to the committee by FDA included several additional examples of mitigation-selection decisions, some of which may require evaluating and comparing a larger number or greater variety of options beyond simple binary choices of recall or not. For example, for product-recall decisions, there might be different levels or types of recall that could be executed, each of which could lead to different health consequences; there might be mitigation options that combine recall of unused product with other risk-reduction options for product already in use. For risk-mitigation decisions with several options, each of the viable risk-mitigation approaches needs to be specified, and the consequences of each must be estimated by using the risk attributes
OCR for page 23
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
defined below. More generally, almost any decision that results in a change in product availability or the extent of use of a particular product could be characterized as a mitigation-selection decision; the outcomes will differ in the degree of use of the product, and the resulting health consequences can be characterized and compared by using the framework proposed here.
Targeting Decisions
Targeting decisions as described by FDA are essentially priority-setting or resource-allocation decisions. They appear to be made primarily within a program and focus on how a particular resource, such as inspection capability, should be allocated among a broad set of products or product categories. In this type of decision, the options or alternatives theoretically available to FDA are vast: virtually any amount of a resource could be allocated to any of a subset of the identified products or product categories, and the only constraint would be total resource availability. The large array of options will need to be narrowed judiciously before the effects of the resulting decisions can be evaluated in detail.
For each product, product category, or other item for which resources might be targeted, FDA will need to identify the resources that it is considering allocating. In some cases, the decision may be an “all or none” decision, such as a decision to inspect a facility or not to inspect it. In others, it may be a particular level of effort and resources, such as a specific rigor of inspection of a facility. Generally, each potential target of a resource allocation would require evaluation of at least two decisions: allocating “x” level of resources vs allocating “y” level of resources. The difference between the values of the risk attributes of an allocation of “x” vs “y” defines the public-health benefit of allocation of resources at those levels to each product or product category.
As noted by FDA (Bertoni 2010), targeting decisions can be seen as similar to the risk-ranking questions that are historically the main focus of CRA studies and that remain of interest in some FDA centers. For example, the Center for Food Safety and Applied Nutrition (CFSAN) has developed a “Fresh Produce Risk Ranking Tool,” which can be used to rank produce-pathogen pairs in order of risk (FoodRisk.org 2011). Similarly, a detailed risk assessment of Listeria monocytogenes conducted in 2003 led to a ranking of ready-to-eat foods by the likelihood and frequency of illnesses (FDA/USDA 2003).
It is unclear, however, how risk ranking can or should be used to make risk-informed decisions. When different allocations of resources are expected to be equally effective in reducing the risks associated with the product or product category in question, a ranking of potential targets according to unreduced risk may provide sufficient decision support. For example, in making decisions about what facilities to inspect in a specific year, CFSAN may be able to use a risk ranking of facilities in their current conditions and operations with the assumption that additional resources will be equally effective in reducing the risks at
OCR for page 24
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
any facility. Given that assumption, resources should be focused on the highest-risk facilities. To create such a risk ranking, FDA would need to estimate the public-health consequences of the food products from each facility that is a candidate for inspection—or, more generally, for each potential target of the resource allocation—on the basis of its best assessment of the current conditions and operations of the facilities. The resulting ranked list could then be used as the basis of resource-allocation decisions.
However, the assumption of equal effectiveness is not often appropriate in real decision problems. If the situation is more complicated—that is, if the risks associated with some products or product categories being evaluated can be reduced much more effectively or at much lower cost than the risks associated with others—a narrowly focused risk ranking will not be an appropriate tool for making the decisions. In such situations, FDA should define the decision options more explicitly, as discussed above. For example, the risk associated with a food from a particular facility could be determined for two inspection regimes. The basis of the status quo inspection procedures could be extracted from an existing risk-ranking exercise. Then FDA would also need to evaluate the public-health consequences associated with the food from the facility if a decision were made to allocate a different level of inspection resources to the facility; this evaluation would include changes in any risks affected by the different inspection. The same exercise would then need to be performed for any other food-facility combination that is being considered in the resource-allocation decision. That approach would lead to a “risk-change” ranking rather than a risk ranking and would enable FDA to take into account the possibility that some inspections are more effective than others in reducing risks. If resources are to be reallocated from existing inspection programs to new programs, the increases in risk due to the reduction in funding in one must be balanced against the decreases in risk due to increased funding in the other.
In Chapter 4, the committee illustrates the use of the framework to evaluate three food categories for their current health effects. The results of the evaluation could be used to rank the food categories by risk, and the ranking could be used, subject to the limitations described above, to support a targeting decision for a hypothetical example of food-safety inspections. In Chapter 6, the committee illustrates a targeting decision that involves a more explicit comparison of the effects of different allocations of resources.
Strategic-Investment Decisions
Strategic-investment decisions are longer-term internal decisions about where FDA should invest its resources to enable better risk-informed decision-making. For example, additional research can be conducted, more data collected and analyzed, and new tools developed to increase understanding of and reduce uncertainties about a wide array of potential risks. Investments in such activities typically enable better evaluations and more informed decisions, so they provide
OCR for page 25
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
value to FDA and the American public. However, they are tied only indirectly to the more common metrics of public health, and it can be difficult to demonstrate their benefits and to decide which strategic investments are most worth while.
The decision-analytic concept of value of information can be used to evaluate the potential benefits of strategic-investment decisions about data or information collection. This concept is that new information has decision-relevant value only if it could lead to actions different from the actions that would be taken without the information. That is, information and strategic investments to collect it have value only if they have the potential to change decisions and thus potentially improve outcomes. For example, if a physician orders a diagnostic test but would recommend the same treatment regardless of the test results, the information has no decision-relevant value. In contrast, if the treatment recommendation depends on the diagnosis and the test results will be used to differentiate among diagnoses, the information potentially has decision-relevant value. The risk-characterization framework can provide some of the key elements necessary for a formal value-of-information analysis: definition of a decision context (Step 1) and characterization of decision outcomes (Step 2). Some extensions to the framework would be necessary to implement a value-of-information analysis fully, and those are explored in several of the case studies that follow.
In the case study in Chapter 5, the committee uses its approach to evaluate the public-health consequences of a strategic-investment decision to enhance postmarket surveillance of two medical devices. Application of the committee’s proposed approach to this decision category is the most challenging both because identifying the strategic-investment decision options to be evaluated is complicated and because evaluating the effects of long-term strategic investments on public health is difficult. The case study allows some comparisons to be made but probably represents a narrower scope of strategic-investment options than what FDA would consider.
CHARACTERIZING THE HEALTH CONSEQUENCES OF DECISION OPTIONS
In the risk-assessment paradigm, risks have traditionally been characterized by a single attribute, such as the number of deaths that could occur as a result of the hazard being evaluated or the probability that an exposed individual will experience an identified adverse effect. Several complications arise with that simple characterization of risks. Multiple outcomes, such as illnesses and deaths, are often of interest. Uncertainty in the outcomes makes simple characterization and reporting problematic: reporting only expected fatalities will obscure information about the other possible outcomes that are important to decision-makers and policy-makers. Providing only single values for health outcomes may convey far greater certainty than is appropriate. Characterizing the public-health consequences requires recognizing and accommodating the
OCR for page 26
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
multidimensional nature of risk and the uncertainties involved in estimating the outcomes.
The Many Dimensions of Risk
The comparison of alternative decisions may seem easier when a single metric is used and when the harm and the benefit are measured with the same metric. For example, in deciding whether the association of clozapine with the potential fatal side effect of agranulocytosis is sufficient to warrant keeping clozapine off the market, the number of deaths caused by clozapine-induced agranulocytosis can be directly compared with the number of suicides that clozapine would be expected to prevent. However, those estimates may differ greatly in their uncertainty, and other benefits of clozapine, such as improved ability to function, would also have to be considered in such a decision.
When the harm and the benefit of alternative decisions cannot be measured with the same metric, decision-making is less straightforward, and the importance of informing decision-makers of the various harms and benefits becomes more important. For example, infliximab is highly effective in reducing pain and improving function in people who have rheumatoid arthritis, but its immunosuppressive properties may permit serious and possibly fatal infections to emerge. Deciding whether the risk of a serious adverse effect outweighs the benefits in increased mobility and quality of life is not a technical or scientific question but rather a question of personal and societal values. An integral part of the committee’s proposed approach is that it characterizes various effects explicitly so that decision-makers can make informed decisions that account for them rather than combining them into a single metric based on implicit weightings (see Box 2-1).
The risk-assessment paradigm gives rise to one set of attributes for characterizing the health risks associated with FDA-regulated products or FDA decisions: factors that are used to determine the number, type, and rate of occurrence of adverse health effects (including deaths) that could result from implementation of a particular decision option. Those attributes are exposed population, mortality, and morbidity, each described in more detail below.
Studies of risk perception and public attitudes about risks have consistently shown that although the mortality and morbidity components in risk estimation are important, they are not the only things that people care about when they think about risks and about risk acceptability (Slovic 1992). Numbers of deaths and illnesses or injuries matter, but so do other factors, such as whether a risk is voluntary, how much control a person has over risks, and whether the hazard being considered has the potential to lead to a large number of simultaneous deaths. The list of factors hypothesized to be important in understanding risk is long (see Appendix E). Relatively few risk attributes have been studied in
OCR for page 27
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
BOX 2-1
Summary Measures vs Detailed Characterization of Public-Health Consequences
In the public health literature, it has become common to measure and communicate the burden of disease by using a summary measure that combines mortality and morbidity into a single value. Some measures, such as disability adjusted life years (DALYs), have been used to evaluate the global burden of disease (Lopez et al. 2006). Others, such as quality adjusted life years (QALYs), have been promoted as tools for comparing the cost effectiveness of various medical interventions (Broome 1993) or risk reducing regulatory strategies (IOM 2006). Each summary measure necessarily embeds a set of value judgments about different levels of health impairment, and each provides a narrow, rather than a robust, characterization of the risk. The committee finds that a richer characterization of the public-health consequences of decision options is needed for routine comparison of decision options. As described in Improving Risk Communication, “reducing different kinds of hazard to a common metric (such as number of fatalities per year) and presenting comparisons only on that metric have great potential to produce misunderstanding and conflict and to engender mistrust of expertise” (NRC 1989, p. 52).
detail, but the available studies show that the various factors are highly correlated on a relatively small number of dimensions, and it has become common to refer to these dimensions as reflecting key “factors” that characterize risk perceptions (Slovic 1992; NRC 1996). The first dimension captures the quantitative aspects, such as the number and type of adverse health outcomes. The second dimension is roughly characterized by the degree of knowledge about the hazards or risks and how well they are understood. The third dimension is characterized by variables that are less easily summarized by a single category: whether a risk is voluntary, how much control an exposed person has over it, the ease with which it can be reduced, and whether it is catastrophic (that is, can lead to multiple simultaneous deaths or injuries).
Understanding Risk urged that risk characterization explicitly include consideration of those additional factors and described a number of ways in which they could be taken into account in decision processes (NRC 1996, p. 65). That approach has also been embraced as a key part of CRA and in the risk-ranking studies mentioned above. The risk-ranking studies include both quantitative information about the number of deaths and injuries and more qualitative information reflecting some of the variables identified in the risk-perception literature. Little general guidance, however, has been offered about what risk attributes might be widely appropriate. Investigators in the studies cited above typically acknowledge that attribute selection is complicated and then choose attributes
OCR for page 35
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
tainties to decision-makers and policy-makers without introducing biases, such as overconfidence and false precision.
Using the Best Available Information to Develop Estimates
The data required to support the evaluation of the public-health consequences of different decisions can vary widely in quality and availability. For some problems, large volumes of epidemiologic data may be available to describe the mortality risk of some medical product. For others, there may be an array of detailed computer models that attempt to characterize the morbidity-reduction effectiveness of various inspection procedures. For still others, only scant data or models may be available, and it may be necessary to rely solely on the judgment of knowledgeable professionals to develop such estimates. Often, a hybrid approach will be necessary: using experts to identify relevant information from studies or models and relying on the experts to interpret the data or model results in the context of the decisions being evaluated. If data are available, their relevance to the required estimates must be considered carefully. For example, the available data may indicate deaths that were temporally associated with a product or device but were not caused by the product or device; judgment will be required to interpret such data appropriately and to estimate the mortality associated with the different decision options being evaluated.
Critical decisions often must be made when uncertainty about some relevant factors remains; in those cases, judgment is required. Lack of easily accessible data does not mitigate the need to develop the best possible estimates of public-health consequences of products being evaluated; in some cases, carefully assessed expert judgment may be the only option available for ensuring that decision-makers have all the relevant information that is available to make decisions. Delaying a critical decision until additional information can be collected may be an option in some circumstances, but it should not be considered the “default” decision when uncertainty exists. Instead, all relevant options, including the option to delay a decision until later or to allocate scarce resources to additional data collection or model development, should be evaluated on the basis of the best available information, including expert judgment, about the public-health consequences of the decision.
Many risk-assessment and evaluation tools are available and are described in detail in other studies, including National Research Council and FDA studies described above. Box 2-3 summarizes current perspectives at EPA about how to quantify uncertain outcomes, including a hierarchy of approaches and the role of expert judgment. It is not within the present committee’s scope to evaluate, compare, or recommend specific approaches or models for risk quantification. The committee simply urges FDA to bring the best available data and expertise to bear on the evaluation that would be consistent with the importance of the decision being evaluated and the time and resources available to complete the assessment.
OCR for page 36
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
BOX 2-3
Methods for Estimating Uncertain Quantities
A white paper written for EPA outlines a general hierarchy of methods that can be used to estimate quantities when there is substantial uncertainty about their “true” values (Frey et al. 2003). Four general categories of methods are described without any implied preferences or priorities:
Statistical methods based on empirical data, which use classical statistics to draw inferences from “hard” data alone.
Statistical methods based on judgment, in which expert judgments and Bayesian approaches to statistical analysis are included, often in combination with “hard” data.
Other quantitative methods that involve approaches not based on probability theory, such as interval methods, fuzzy methods, and metaanalytic methods.
Qualitative methods that can be used when key aspects of uncertainty cannot be captured by quantitative methods (for example, uncertainty caused by problem formulation or the existence of competing models).
Of those approaches, the first is widely used and accepted in risk assessment, the second is widely used outside the risk-assessment community and is expanding in use and acceptance in the risk-assessment community, and the third and fourth approaches are not generally used.
EPA discussed the use of expert elicitation (or expert judgment) and made the following comment:
Expert elicitation is recognized as a powerful and legitimate quantitative method for characterization of uncertainty and for providing probabilistic distributions to fill data gaps where additional research is not feasible. The academic and research community, as well as numerous review bodies, have recognized the limitation of empirical data for characterization of uncertainty and have acknowledged the potential for using [expert elicitation] for this purpose. In Science and Judgment in Risk Assessment [NRC 1994] the NAS [National Academy of Sciences] recognized that for “parameter uncertainty, enough objective probability data are available in some cases to permit estimation of the probability distribution. In other cases, subjective probabilities might be needed.” In this “Blue Book” report, the NAS further recognized the “difficulties of using subjective probabilities in regulation” and identified perceived bias as one major impediment; but, noted that “in most problems real or perceived bias pervades EPA’s current point-estimate approach.” In addition, the NAS stated that “there can be no rule that objective probability estimates are always preferred to subjective estimates, or vice versa” (EPA 2009, p. 5).
Thus, EPA recognized the importance of using statistical methods based on judgment to derive probability estimates and emphasized that objective methods are not always preferable to subjective methods.
OCR for page 37
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
Characterizing Uncertainty
Although substantial effort has been devoted to motivating and guiding the inclusion of uncertainty in policy analysis throughout the federal government (see Box 2-4), including it is difficult, and resistance remains high. Individual researchers and policy analysts are reluctant to make quantitative estimates without what they feel to be “sufficient” data, and they are even more reluctant to quantify the uncertainties inherent in their domain. It is always possible to collect more data and do more analyses to try to develop “better” estimates, but there will always be uncertainty, and decisions often must be made on the basis of existing information. Quantifying what is known and what is not known (the uncertainty) is an important way to ensure that decisions are as well informed as possible.
The difficulty of developing best estimates and uncertainties when data are sparse is exacerbated by the norm of using probability distributions for quantification. Probability is not a natural way of thinking about uncertainty for everyone, and often efforts are made to sidestep the problem by using qualitative or categorical measures of uncertainty, such as “likely,” “very unlikely,” and “possible.” That approach may make the risk-characterization task more palatable, but it also makes it considerably less useful. The very ambiguity that provides comfort makes the task of communicating and comparing uncertainties difficult. Flexible definitions that vary from domain to domain make cross-cutting analysis impossible. Is the probability value associated with a "high likelihood" of death equal (either numerically or cognitively) to the probability value associated with a "high likelihood" of rain? The effort needed to define the uncertainty categories unambiguously and in sufficient detail would certainly approach if not exceed the effort needed to follow one of the standardized approaches of assessing a distribution directly. In addition, for some problem domains, assessing the complete distribution is not necessary; a few key points selected from the underlying distribution will describe the underlying uncertainty sufficiently to permit the necessary comparisons and analyses. That approach is demonstrated in the case studies that follow this chapter.
Options for succinctly describing a probability distribution vary; however, the shorthand notation should be able to provide an indication of a distribution’s first three moments—central tendency, spread, and skew—and should be easily assessable from data, model output, or expert judgment. In the case studies that follow, that task is accomplished by using three values: the distribution’s 5th, 50th (median), and 95th percentiles. The 5th percentile is the numerical value of the quantity being estimated (such as the number of deaths) that bounds the lower 5% tail of the distribution; for example, the probability that the actual value will be lower than the 5th percentile is 0.05. The 95th percentile is the numerical value that will be exceeded with a probability of 0.05; that is, it is the boundary of the upper 5% tail of the distribution. The median value is a measure of the distribution’s central tendency and generally represents the
OCR for page 38
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
BOX 2-4
Importance of Characterizing Uncertainty
The assessment, quantification, and communication of uncertainty are essential, as has been recognized in the risk-assessment literature since at least 1983. Numerous high-level government advisory panels have discussed the importance of capturing and presenting uncertain values. Risk Assessment in the Federal Government: Managing the Process (NRC 1983) formalized the integration of uncertainty and risk into policy analysis. The guidance provided in that early report has been expanded on in a series of reports (NRC 1993, 1994, 1996) and recently reinforced in Science and Decisions: Advancing Risk Assessment (NRC 2009, p. 93), which states at the beginning of a chapter on uncertainty and variability that “characterizing uncertainty and variability is key to the human health risk-assessment process, which must engage the best available science in the presence of uncertainties and difficult-to-characterize variability to inform risk-management decisions.” In recent guidelines on probabilistic risk assessment, the U.S. Nuclear Regulatory Commission states that risk-informed decision-making requires the “appropriate consideration of uncertainty…in the analyses used to support the decision and in the interpretation of the findings of those analyses” (USNRC 2009, page iii).
Over the last 25 years, EPA has developed a series of guidelines that detail how uncertainty should be integrated into regulatory policy-making and evaluation (EPA 1984, 1995, 2000, 2004). Estimating the Public Health Benefits of Proposed Air Pollution Regulations (NRC 2002) offers advice to EPA on the importance of characterizing uncertainty even when data are sparse or lacking:
EPA should move the assessment of uncertainty from its ancillary analyses into its primary analyses to provide a more realistic depiction of the overall degree of uncertainty. This shift will entail the development of probabilistic, multiple-source uncertainty models based not only on available data but also on expert judgment … Uncertainty should be described as completely and as realistically as possible for all regulatory options, recognizing that regulatory action might be necessary in the presence of substantial uncertainty. The regulatory decision process will be better informed by a fair assessment of the uncertainty and a realistic evaluation of the likely reductions in that uncertainty attainable through further research (NRC 2002, p. 11).
Explicit evaluation and presentation of uncertainty in risk assessments reduces the problem of false precision, makes risk characterization more informative, and increases the credibility of any ensuing risk communication.
“best estimate” of the uncertain quantity. The probability that a value falls below the median is equal to the probability that it falls above the median. The spread of the distribution is indicated by the difference between the 5th and 95th per-
OCR for page 39
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
centiles. The skew of the distribution is indicated by comparing the difference between the 5th percentile and the median and the difference between the median and the 95th percentile. For a symmetric distribution, those differences will be equal. For a negative skew, the first difference will be larger; for a positive skew, the second difference will be larger.
If the three percentiles are relied on to characterize a distribution, the assessment procedure for experts is relatively simple and well tested. With training and practice, an analyst can elicit an accurate representation of the three percentiles from experts by using easily described thought experiments, simple tools, and standard protocols. Numerous references describe a variety of approaches for assessing quantitative estimates of uncertain quantities, including examples of how expert assessments have been used to support a wide array of decisions (Merkhofer 1987; Cooke 1991; Hora 2007; EPA 2009; Jenni and van Luik 2010). It is important to note that a viable approach will require the consideration of the tails of the distribution. Assessed distributions are often too narrow, but considering rarer and more extreme events leads to assessed distributions that represent true uncertainties better.
Although the three percentiles are precisely defined, their cognitive interpretation should not be lost. The goal is to have a risk characterization that includes representation of the uncertainty in the size of the exposed population, in the number of deaths that may occur, and in the number of injuries or illnesses that may occur under various decision options. Rather than ask simply for a range or for low and high estimates of those quantities, the committee suggests using the concepts associated with the percentiles of a distribution as a way to ensure consistent assessment and interpretation of the low and high values. The 5th percentile represents a value below which the actual value is not likely to fall. Similarly, the 95th percentile represents a value above which the actual value is not likely to fall. The estimates are not upper and lower bounds, but values that indicate limits beyond which results would be surprising. Fixating on the precise values for the percentiles could cause unnecessary anxiety and communicate false precision where none is intended or needed.
Uncertainty in the health consequences of alternative decisions should be included in the characterization of risks in this framework. Specifically, a best estimate, a high estimate, and a low estimate—corresponding to the three percentiles described above—should be determined for the size of the exposed population, mortality, and each of the three morbidity estimates. The other attributes could also be described by using similar distributional measures although the importance of doing so is less. When assessing the probabilities associated with ability to detect, control, or mitigate, ranges may be appropriate instead of point values. For example, the probability that an informed institution will be able to reduce or mitigate adverse health effects associated with a particular risk could be judged to be 0.80-0.90.
The committee notes that there is likely to be a relationship between the quality of data available to develop the necessary estimates and the uncertainty
OCR for page 40
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
in the estimates. Specifically, when relevant data are lacking, uncertainty about each of the outcomes will probably be high (that is, a large spread).
USING THE FRAMEWORK TO SUPPORT DECISION-MAKING
The framework proposed here is similar to the framework for risk-based decision-making described in Science and Decisions (see Figure 2-1) in its emphasis on structuring and conducting risk assessments and risk characterization in the broader context of supporting agency decisions. Science and Decisions concludes by making recommendations for improving risk-based decision-making at EPA; although some of those recommendations may be applicable to FDA, the present committee was charged not with evaluating the risk-based decision-making processes at FDA but with developing a robust approach for characterizing public-health consequences. Those consequences are an important consideration in FDA decisions, and the attributes proposed in this framework provide such a characterization. Using this framework throughout the agency will ultimately lead to robust and consistent characterization of public-health consequences.
How the information is used will ultimately be determined by FDA. Public-health consequences constitute only one of numerous factors—such as economic, social, and political factors—that FDA must consider when making decisions, and the committee neither expects nor recommends that the attributes in this report should form the sole basis of such decisions. However, careful and consistent evaluation of the public-health consequences of various options is an essential component of good decision-making, and there are several ways in which the risk characterizations described here could be used in decision-making.
One approach was outlined in the committee’s letter report, which focused on developing a risk ranking (see Appendix A). The letter report describes an approach for involving stakeholders in an exercise to rank risks that used a consistent set of attributes. Specifically, it involved ranking risks on the basis of judgment and the described characteristics of those risks or ranking risks on the basis of a mathematical combination of the attribute scores and judgments about the importance of the different attributes, that is, using the logic of multiattribute utility theory (Keeney and Raiffa 1976). The approach described in the letter report is based on the risk-ranking literature described previously and could be adapted and applied in the decision-focused framework proposed here.
The risk characterization of Step 2 of the present framework is also compatible with several other approaches to decision-making—both more quantitative approaches and more inclusive processes. The analytic-deliberative process described in Understanding Risk (NRC 1996) includes scientists, public officials, and other interested and affected parties in an iterative process in which all parties are involved in every step of the decision process. If FDA adopted that type of approach to risk-based decision-making, the risk attributes defined here
OCR for page 41
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
could provide a starting point for the risk-characterization aspects of the analysis. Retaining the flexibility to refine the risk characterization throughout the process, however, is key to the analytic-deliberative process.
At perhaps a different extreme, FDA may determine that for some decisions only a single public-health consequence is relevant. In that case, the other elements of the risk characterization could be ignored. Alternatively, the quantitative public-health consequences as a group may be considered relevant, and it may be considered useful to combine the attributes in the risk characterization into a single metric to compare options. Such integrative measures may be easier to develop, and especially valuable, when there is a logical set of mathematical relationships between the various attributes. For example, two rare causes of morbidity associated with a particular product might reasonably be considered additive in their effects; other attributes, such as the size of a population affected and the rate of adverse effects or the number of adverse effects and the likelihood that harm could be mitigated, would combine in a multiplicative fashion. Proven mathematical approaches for combining attributes into a single integrative measure (Keeney and Raiffa 1976; Fischhoff et al. 1984) could also be applied here. However, the attributes defined here were chosen to provide a robust risk characterization rather than an easy mathematical combination, so developing a single integrative measure would require careful consideration of the relationship between the various attributes and their relative importance. As discussed in Box 2-1, using integrative measures alone has several disadvantages, but relying only on the full list of attributes without integration can make consistency a challenge.
Each case study that follows concludes with a discussion of how the present framework could be used to support a specific decision, but they are offered only for illustration. It remains for FDA to decide how to use the information, and whether and how to combine it with other decision-relevant factors.
FLEXIBILITY AND EVOLUTION OF THE FRAMEWORK
The risk-characterization framework proposed here is a flexible system. Although it is intended to capture elements of risk that are applicable across the broad array of risk-related decisions that FDA makes, it is not necessarily a comprehensive set of public-health-related factors relevant for every such decision, and it clearly does not aim to include factors that are unrelated to public-health outcomes. The framework is not proposed as a “one-size-fits-all” method for all risk-related decisions at FDA for all time. Just as the framework and the risk attributes were refined through development of the case studies in this report, the committee expects that some elements of the framework will continue to evolve as FDA gains experience in using it. Additional risk characteristics may be identified for subsets of decisions and added to the list for those types of decisions. Furthermore, the existing definitions of attributes may be modified to
OCR for page 42
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
yield more interesting insights, and some attributes may be dropped altogether if they are not found to be useful.
REFERENCES
AHRQ (Agency for Healthcare Research and Quality). 2005. U.S. Valuation of the EuroQol EQ-5D Health States. Agency for Healthcare Research and Quality, Rockville, MD. December 2005 [online]. Available: http://www.ahrq.gov/rice/EQ5Dproj.htm [accessed Oct. 25, 2010].
Bertoni, M.J. 2010. Opening Remarks. Presentation at the 5th Meeting on Ranking FDA Product Categories Based on Health Consequences, Phase II, February 3, 2010, Washington, DC.
Bronfman, N.C., L.A. Cifuentes, M.L. DeKay, and H.H. Willis. 2007. Accounting for variation in the explanatory power of the psychometric paradigm: The effects of aggregation and focus. J. Risk Res. 10(4):527-554.
Bronfman, N.C., L.A. Cifuentes, and V.V. Gutierrez. 2008a. Participant-focused analysis: Explanatory power of the classic psychometric paradigm in risk perception. J. Risk Res. 11(6):735-753.
Bronfman, N.C., E.L. Vazquez, V.V. Gutierrez, and LA. Cifuentes. 2008b. Trust, acceptance and knowledge of technological and environmental hazards in Chile. J. Risk Res. 11(6):755-773.
Broome, J. 1993. QALYs. J. Public Econ. 50(2):149-167.
Cooke, R.M. 1991. Experts in Uncertainty: Opinion and Subjective Probability in Science. K. Shrader-Frechette, ed. New York: Oxford University Press.
Davies, J.C. 1996. Comparing Environmental Risks: Tools for Setting Government Priorities. Washington, DC: Resources for the Future.
DeKay, M.L., H.K. Florig, P.S. Fischbeck, M.G. Morgan, K.M. Morgan, B. Fischhoff, and K.E. Jenni. 2001. The use of public risk ranking in regulatory development. Pp. 208-230 in Improving Regulation: Cases in Environment, Health, and Safety, P.S. Fischbeck, and R.S. Farrow, eds. Washington, DC: Resources for the Future.
DeKay, M.L., P.S. Fischbeck, H.K. Florig, M.G. Morgan, K.M. Morgan, B. Fischoff, and K.E. Jenni. 2005. Judgement-Based Risk Ranking for Food Safety Pp. 198-226 in Toward Safer Food: Perspectives on Risk and Priority Setting. S.A Hoffman and M.R. Taylor, eds. Washington, DC: RFF Press.
EPA (U.S. Environmental Protection Agency). 1984. Risk Assessment and Management: Framework for Decision Making. EPA 600/9-85-002. U.S. Environmental Protection Agency, Washington, DC. December 1984.
EPA (U.S. Environmental Protection Agency). 1987. Unfinished Business: A Comparative Assessment of Environmental Problems. EPA/230/2-87/025. Office of Policy, Planning, and Evaluation, U.S. Environmental Protection Agency, Washington, DC.
EPA (U.S. Environmental Protection Agency). 1995. Policy for Risk Characterization at the U.S. Environmental Protection Agency. U.S. Environmental Protection Agency, Washington, DC. March 1995 [Appendix A in EPA 2000].
EPA (U.S. Environmental Protection Agency). 2000. Risk Characterization Handbook. EPA 100-B-00-002. Office of Science Policy, Office of Research and Development, U.S. Environmental Protection Agency. December 2000 [online]. Available: http://itp-pfoa.ce.cmu.edu/docs/RChandbk_excerpts_re_exposure.pdf [accessed Oct. 25, 2010].
OCR for page 43
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
EPA (U.S. Environmental Protection Agency). 2004. An Examination of EPA Risk Assessment Principles and Practices, Staff Paper Prepared for the U.S. Environmental Protection Agency by Members of the Risk Assessment Task Force. EPA/100/B-04/001. Office of the Science Advisor, U.S. Environmental Protection Agency, Washington, DC. March 2004 [online]. Available: http://www.epa.gov/osa/pdfs/ratf-final.pdf [accessed Oct. 25, 2010].
EPA (U.S. Environmental Protection Agency). 2009. Expert Elicitation Task Force White Paper, External Review Draft. Science Policy Council, U.S. Environmental Protection Agency. January 6, 2009 [online]. Available: http://www.epa.gov/osa/pdfs/elicitation/Expert_Elicitation_White_Paper-January_06_2009.pdf [accessed Feb. 23, 2011].
EPA SAB (U.S. Environmental Protection Agency Science Advisory Board). 1990. Reducing Risk: Setting Priorities and Strategies for Environmental Protection. SAB-EC 90-021. Science Advisory Board, U.S. Environmental Protection Agency [online]. Available: http://yosemite.epa.gov/sab/sabproduct.nsf/28704D9C420FCBC1852573360053C692/$File/REDUCING+RISK++++++++++EC-90-021_90021_5-11-1995_204.pdf [accessed Mar. 17, 2011].
EuroQoL Group. 1990. EuroQol – A new facility for the measurement of health-related quality of life. Health Policy 16(3):199-208.
FDA/USDA (U.S. Food and Drug Administration and U.S. Department of Agriculture). 2003. Listeria monocytogenes Risk Assessment: Interpretive Summary. Center for Food and Applied Nutrition, U.S. Food and Drug Administration, and Food Safety and Inspection Service, U.S. Department of Agriculture. September 2003 [online]. Available: http://www.fda.gov/Food/ScienceResearch/ResearchAreas/RiskAssessmentSafetyAssessment/ucm185291.htm [accessed Feb. 23, 2010].
Fischhoff, B. 1984. Setting standards: A systematic approach to managing public health and safety risks. Manage. Sci. 30(7):823:843.
Fischhoff, B., S.R. Watson, C. Hope. 1984. Defining risk. Policy Sciences. 17(2):123-139.
Fischhoff, B. 1995. Ranking risks. Risk Health Saf. Environ. 6:189-200.
Fischhoff, B. 2006. Cognitive processes in stated preference methods. Pp. 937-968 in Handbook of Environmental Economics, Vol. 2. Valuing Environmental Changes, K.G. Mäler, and J.R. Vincent, eds. Amsterdam: Elsevier.
Florig, H.K., M.G. Morgan, K.M. Morgan, K.E. Jenni, B. Fischoff, P.S. Fischbeck, and M.L. DeKay. 2001. A deliberative method for ranking risks (I): Overview and test bed development. Risk Analysis. 21(5):913.
FoodRisk.org. 2011. FDA’s Fresh Produce Risk Ranking Tool. Joint Institute for Food Safety and Applied Nutrition, College Park, MD [online]. Available: http://www.foodrisk.org/exclusives/RRT/[accessed Feb.11, 2011].
Frey, C., D. Crawford-Brown, Z. Junyu, and D. Loughlin. 2003. Hierarchy of Methods to Characterize Uncertainty: State of Science of Methods for Describing and Quantifying Uncertainty. EPA Contract No. 68-D-00-265. Prepared for Office of Air Quality Planning and Standards, U.S. Environmental Protection Agency, Research Triangle Park, NC. August 12, 2003 [online]. Available: http://www4.ncsu.edu/~frey/reports/Frey_CrawfordBrown_Zheng_Loughlin_2003.pdf [accessed Feb. 23, 2010].
Gutiérrez, V.V., L.A. Cifuentes, and N.C. Bronfman. 2006. The influence of information delivery on risk ranking by lay people. J. Risk Res. 9(6):641-655.
OCR for page 44
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
Hora, S.C. 2007. Eliciting probabilities from experts. Pp. 129-153 in Advances in Decision Analysis: From Foundations to Applications, W. Edwards, R.F. Miles, Jr., and D. von Winterfeldt, eds. New York: Cambridge University Press.
IOM (Institute of Medicine). 2006. Valuing Health for Regulatory Cost-Effectiveness Analysis. Washington, DC: The National Academies Press.
IOM (Institute of Medicine). 2009. Initial National Priorities for Comparative Effectiveness Research. Washington, DC: The National Academies Press.
Jenni, K.E. 1997. Attributes for Risk Evaluation. Ph.D. Dissertation, Carnegie Mellon University, Pittsburgh, PA.
Jenni, K.E., and A. van Luik. 2010. Assessment of expert judgments for safety analyses and performance assessment of geologic repository systems. Pp. 580-609 in Geological Repository Systems for Safe Disposal of Spent Nuclear Fuels and Radioactive Waste, J. Ahn, and M.J. Apted, eds. Cambridge: Woodhead Publishing.
Jones, K. 1997. A Retrospective on Ten Years of Comparative Risk. Prepared for American Industrial Health Council, Washington, DC, by Green Mountain Institute for Environmental Democracy, Montpelier, VT. January 24, 1997 [online]. Available: http://www.heartland.org/custom/semod_policybot/pdf/4728.pdf [accessed March 17, 2011].
Keeney, R.L. and H. Raiffa. 1976. Decisions with Multiple Objectives: Preferences and Trade-Offs. New York: Wiley.
Long, J. and B. Fischoff. 2000. Setting risk priorities: A formal model. Risk Analysis. 20(3):339-352.
Lopez, A.D., C.D. Mathers, M. Ezzati, D.T. Jamison, and C.J.L. Murray, eds. 2006. Global Burden of Disease and Risk Factors. Washington, DC: World Bank [online]. Available: http://www.dcp2.org/pubs/GBD [accessed Feb. 14, 2011].
Merkhofer, M.W. 1987. Quantifying judgmental uncertainty: Methodology, experiences, and insights. IEEE T. Syst. Man Cyb. 17(5):741-752.
Minard Jr., R.A. 1996. Can States Make a Market for Environmental Goals? Pp. 243-280 in Linking Science and Technology to Society's Environmental Goals. National Research Council. Washington, DC: The National Academies Press.
Morgan, M.G., B. Fischhoff, L. Lave, and P. Fischbeck. 1996. A proposal for ranking risks within federal agencies. Pp. 111-148 in Comparing Environmental Risks: Tools for Setting Government Priorities, J.C. Davies, ed. Washington, DC: Resources for the Future.
Morgan, K.M., M.L. DeKay, and P.S. Fischbeck. 1999. A multi-attribute approach to risk prioritization. Risk Policy Report 6(6):38-40.
Morgan, M.G., H.K. Florig, M.L. DeKay, and P. Fischbeck. 2000. Categorizing risks for risk ranking. Risk Analysis. 20(1):49-58.
Morgan, K.M., M.L. DeKay, P.S. Fischbeck, M.G. Morgan, B. Fischhoff, and H.K. Florig. 2001. A deliberative method for ranking risks (II): Evaluation of validity and agreement among managers. Risk Anal. 21(5):923-937.
NRC (National Research Council). 1983. Risk Assessment in the Federal Government: Managing the Process. Washington, DC: National Academy Press.
NRC (National Research Council). 1989. Improving Risk Communication. Washington, DC: National Academy Press.
NRC (National Research Council). 1993. Pesticides in the Diets of Infants and Children. Washington, DC: National Academy Press.
NRC (National Research Council). 1994. Science and Judgment in Risk Assessment. Washington, DC: National Academy Press.
OCR for page 45
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration
NRC (National Research Council). 1996. Understanding Risk: Informing Decisions in a Democratic Society. Washington, DC: National Academy Press.
NRC (National Research Council). 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. NRC (National Research Council). 2009. Science and Decisions: Advancing Risk Assessment. Washington, DC: National Academy Press.
Slovic, P. 1987. Perception of risk. Science 236(4799):280-285.
Slovic, P. 1992. Perception of risk: Reflections on the psychometric paradigm. Pp. 117-152 in Social Theories of Risk, S. Krimsky, and D. Golding, eds. New York: Praeger.
USNRC (U.S. Nuclear Regulatory Commission). 2008. Risk-Informed Decisionmaking for Nuclear Material and Waste Applications: Revision 1. U.S. Nuclear Regulatory Commission, Washington, DC [online]. Available: http://www.hss.energy.gov/nu clearsafety/ns/rawg/docs/RIDM_2008_2.pdf [accessed Apr. 13, 2011].
USNRC (U.S. Nuclear Regulatory Commission). 2009. Guidance on the Treatment of Uncertainties Associated with PRAs in Risk-Informed Decision Making Main Report, Volume 1. NUREG-1855. U.S. Nuclear Regulatory Commission, Washington, DC [online]. Available: http://www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr1855/v1/sr1855v1.pdf [accessed Apr. 13, 2011].
Willis, H.H., M.L. DeKay, M.G. Morgan, H.K. Florig, and P.S. Fischbeck. 2004. Ecological risk ranking: Development and evaluation of a method for improving public participation in environmental decision making. Risk Analysis. 24(2):363-378.
Willis, H.H., M.L. DeKay, B. Fischoff, and M.G. Morgan. 2005. Aggregate, disaggregate, and hybrid analyses of ecological risk perceptions. Risk Analysis. 25(2):405-428.