Cover Image

PAPERBACK
$162.25



View/Hide Left Panel

Page 601

Appendix
N-1
The Case for "Plausible Conservatism" in Choosing and Altering Defaults

Adam M. Finkel

This Appendix was written by one member of our committee, who was asked to represent the viewpoint of those members of the committee who believe that EPA should choose and refine its default assumptions by continually evaluating them against two equally important standards: whether the assumption is scientifically plausible, and whether it is "conservative" and thus tends to safeguard public health in the face of scientific uncertainty. Indeed, these three themes of plausibility, uncertainty, and conservatism form most of the framework for the last six chapters of the CAPRA report, as reflected in the "cross-cutting" chapters on model evaluation, uncertainty and variability, and on implementing an iterative risk assessment/management strategy. The particular way these themes should come together in the selection and modification of default assumptions is controversial; hence, the remainder of this appendix is organized into five parts: (1) a general discussion of what "conservatism" does and does not entail; (2) an enumeration of reasons why conservatism is appropriately part of the rationale for choosing and departing from defaults; (3) the specific plan proposed for EPA's consideration;1 (4) a side-by-side analysis of this proposal against the competing principle of "meximum use of scientific information" (see Appendix N-2 following this paper); and (5) general conclusions.

1Although I will discuss and evaluate the general issue of conservatism in detail before I present our specific recommendations, I urge readers to consider whether the proposal detailed in this third section bears any resemblance to the kind of "conservatism for conservatism's sake" that critics decry.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 601
Page 601 Appendix N-1 The Case for "Plausible Conservatism" in Choosing and Altering Defaults Adam M. Finkel This Appendix was written by one member of our committee, who was asked to represent the viewpoint of those members of the committee who believe that EPA should choose and refine its default assumptions by continually evaluating them against two equally important standards: whether the assumption is scientifically plausible, and whether it is "conservative" and thus tends to safeguard public health in the face of scientific uncertainty. Indeed, these three themes of plausibility, uncertainty, and conservatism form most of the framework for the last six chapters of the CAPRA report, as reflected in the "cross-cutting" chapters on model evaluation, uncertainty and variability, and on implementing an iterative risk assessment/management strategy. The particular way these themes should come together in the selection and modification of default assumptions is controversial; hence, the remainder of this appendix is organized into five parts: (1) a general discussion of what "conservatism" does and does not entail; (2) an enumeration of reasons why conservatism is appropriately part of the rationale for choosing and departing from defaults; (3) the specific plan proposed for EPA's consideration;1 (4) a side-by-side analysis of this proposal against the competing principle of "meximum use of scientific information" (see Appendix N-2 following this paper); and (5) general conclusions. 1Although I will discuss and evaluate the general issue of conservatism in detail before I present our specific recommendations, I urge readers to consider whether the proposal detailed in this third section bears any resemblance to the kind of "conservatism for conservatism's sake" that critics decry.

OCR for page 601
Page 602 What Is "Conservatism"? The most controversial aspect of this proposal within the full committee was its emphasis on "conservatism" as one—not the only—organizing principle to judge—not to prejudge—the merits of defaults and their alternatives. Supporters of this proposal are well aware that there are strengths and weaknesses of the conservative orientation that make it one of the most hotly-contested topics in all of environmental policy analysis, but also believe that few topics have been surrounded by as much confusion and misinformation. Some observers of risk assessment appear to be convinced that EPA and other agencies have so overemphasized the principle of conservatism as to make most risk estimates alarmingly false and meaningless; others, including at least one member of this committee, have instead suggested that if anything, the claims of these critics tend to be more reflexive, undocumented by evidence, and exaggerated than are EPA's risk estimates themselves (Finkel, 1989). It is clear that partisans cannot agree on either the descriptive matter of whether risk assessment is too conservative or on the normative matter of how much conservatism (perhaps any at all) would constitute an excess thereof. However, at least some of the intensity marking this debate is due to a variety of misimpressions about what conservatism is and what its ramifications are. Before laying out the proposal, therefore, some of these definitional matters will be discussed. First, a useful definition of conservatism should help clarify it in the face of the disparate charges leveled against it. Conservatism is, foremost, one of several ways to generate risk estimates that allow risk management decisions to be made under conditions of uncertainty and variability. Simply put, a risk assessment policy that ignored or rejected conservatism would strive to always represent risks by their "true values" irrespective of uncertainty (or variability), whereas any attempt to consider adding (or removing) some measure of conservatism would lead the assessor to confront the uncertainty. Incorporating "conservatism" merely means that from out of the uncertainty and/or variability, the assessor would deliberately choose an estimate that he believes is more likely to overestimate than to underestimate the risk. Rationality in managing risks (as in any endeavor of private or social decision making) involves the attempt to maximize the benefit derived from choice under specific conditions in the world. If we do not know those conditions (uncertainty) or do not know to whom these conditions apply (human interindividual variability), we have to make the choice that would be optimal for a particular set of conditions and essentially hope for the best. If the true risk we are trying to manage is larger or smaller than we think it is (or if there are individuals for whom this is so) then our choice may be flawed, but we still have to choose. Unlike the search for scientific truth, where the "correct" action in the face of uncertainty is to reserve one's judgment, in managing risks decisions are inevitable, since reserving judgment is exactly equivalent to making the judg-

OCR for page 601
Page 603 ment that the status quo represents a desirable balance of economic costs expended (if any) and health risks remaining (if any). It is therefore vital that the risk assessment process handle uncertainties in a predictable way that is scientifically defensible, consistent with the Agency's statutory and public missions, and responsive to the needs of decision makers. Conservatism is a specific response to uncertainty that favors one type of error (overestimation) over its converse, but (especially if EPA follows the detailed prescriptions here) the fact that it admits that either type of error is possible is more important than the precise calculus it may use to balance those errors. It is also crucial to understand what this asymmetry in favor of overestimation does and does not mean. Conservatism is not about valuing human lives above the money spent to comply with risk management decisions. Instead, it acknowledges that if there was no uncertainty in risk, society could "optimally" decide to spend a dollar or a billion dollars to save each life involved—conservatism is silent about this judgment. Assuming that society decides how it wishes to balance lives and dollars, conservatism only affects the decision at the margin, by deliberately preferring, from among the inevitable errors that uncertainty creates, to favor those errors which lead to relatively more dollars spent for the lives saved than those which lead to relatively fewer lives saved for the dollars spent. Some would call this an orientation disposed to being "better safe than sorry" or a tendency towards "prudence," characterizations we do not dispute or shrink from. It is simply a matter of "good science" to admit that the true value of risk is surrounded by uncertainty, and that as a consequence, errors of overestimation or underestimation can still occur for whatever value of risk one chooses as the basis for risk management. Much detail about conservatism follows in this appendix of the report, but the essence of the disagreement between supporters of this proposal and supporters of the alternative position is simple; the former group believes that it is both prudent and scientifically justified to make reasonable attempts to favor errors of overestimation over those of underestimation. More importantly, it believes that not to do so would be both imprudent and scientifically questionable. This is no mere tautology, but encapsulates the disagreement with others who would argue that to eschew prudence is to advocate something "value-neutral" (and hence a morally superior position for scientists to espouse) and something more ''scientific." The controversies over conservatism are heightened by ambiguous definitions and uses of the term. The following section explains three dichotomies about the precise possible meanings of conservatism, in order to clarify some of the objections to it, and to foreshadow some of the features of this proposal for a principle of "plausible conservatism": (1) The distinction between prudence and misestimation. When a particular estimate of risk is criticized as being "too conservative," that criticism can

OCR for page 601
Page 604   mean one or both of two different things. The critic may actually mean that the assessor has chosen an estimate of risk which is designed to reduce the probability of errors of underestimation, but one which the critic deems overly zealous in that regard. In other words, one person's "prudence" may be another person's "overkill," although that distinction alone is purely one of differing personal values. On the other hand, the critic may instead mean that flaws in the estimation of a risk cause the estimate to be more skewed in the direction of prudence than the assessor himself intends, or than the risk manager comprehends. Such a criticism may not involve any personal value judgments. For example, the assessor may believe that a particular estimate falls at around the 95th percentile of the uncertainty distribution of the unknown risk; such an estimate would have a 5% probability of being an underestimate of the risk. If, in fact, the estimate given is so tilted towards minimizing underestimation that it falls at (say) the 99.9th percentile of the distribution, then the process would have built in more prudence than either party intended. It is possible that in many of the instances where EPA is under fire for allegedly being "too conservative," critics are espousing differing value judgments in addition to (or instead of) trying to point out disparities between the intended and actual level of conservatism. There is, as discussed below, little empirical evidence to suggest that EPA's potency, exposure, or risk estimates are markedly higher than estimates embodying a reasonable degree of prudence (i.e., the conventional benchmarks of the 95th or 99th percentiles that statisticians use). However, supporters of the proposal detailed in this appendix are clearly opposed to systematic misestimation, if and when it exists. We stress that our version of "plausible conservatism" in risk assessment does not allow EPA to adopt unreasonable assumptions or rely upon biased parameter values, and we believe that the entire committee's consensus recommendations in Chapters 9 and 10 will help combat this tendency, if it exists, and help shed light, rather than heat, on the question of whether EPA's risk estimates are more conservative than they are intended to be. (2) The distinction between conservatism as a response to uncertainty or as a response to variability. This important distinction bears upon the legitimacy of criticisms of conservatism. The two issues of uncertainty and variability involve different motivations and produce different results, even though the same terms and mathematical procedures are used to deal with each and though they may at times be hard to separate operationally. This appendix of the report deals primarily with the former, and then generally with the subcategory of conservatism regarding model uncertainty. In this discussion of uncertainty issues, because we are dealing with lack of knowledge as to the true value of risk, the science-policy balancing of errors of underestimation and overestimation does suggest the common aphorism of "better safe than sorry." The science-policy response to variability, on the other hand, involves coping with differences among people in their exposures or susceptibilities to adverse effects—that is,

OCR for page 601
Page 605   deciding with certainty (or with no additional uncertainty beyond that due to not knowing which models apply) who or what should be protected. In such cases, deciding how conservative to be in light of this variability is not about "being better safe than sorry," but involves a decision about who merits being safe and who may end up being sorry. Here, as elsewhere in the CAPRA report, the committee refrains from coming to policy judgments about how EPA should draw such lines in general or in particular. In this discussion, we simply stress that EPA should not let criticisms of its responses to uncertainty confuse it or necessarily cause it to rethink its responses to variability.2 (3) The distinction between the "level of conservatism" and the "amount of conservatism". Any estimate of an uncertain risk embodies conservatism, if any at all, in both relative and absolute senses. Here the new terms "level of conservatism" and "amount of conservatism" are coined to codify the difference, respectively, between the relative and absolute meanings of the term. The "level of conservatism'' is a relative indicator of how unlikely the assessor deems it that the estimate will fall below the true value of risk; thus, a 99th percentile estimate embodies a higher "level of conservatism" (four percentile points higher) than does a 95th percentile estimate. The "amount of conservatism," in contrast, is an absolute measure of the mathematical difference between the estimate itself and the central tendency of the unknown quantity. Thus, it is quite possible to have a high "level" and a small "amount" simultaneously, or vice versa. For example, scientists might know the speed of light to a high degree of precision, and report a 99th percentile upper confidence limit of 186,301 miles/sec. and a "best estimate" of 186,300 miles/sec. (here the absolute amount of conservatism would be 1 mile/sec.). On the other hand, when uncertainty is large, even a modest "level" (say the 75th percentile) may introduce a large amount of conservatism in absolute terms. These two concepts are related in a straightforward manner with important policy implications. As scientific knowledge increases and uncertainty decreases, the absolute difference between the central tendency and any particular upper percentile will also decrease. Therefore, agencies could try and maintain a fixed level of conservatism over time and yet expect that the absolute amount of conservatism, and thus the practical impact of attempts to shift the balance in favor of overestimation, will become progressively less and less important. When uncertainty is reduced to minimal levels, the conservative estimate and the central tendencies will become so similar that the distinction be- 2For two reasons, we believe it is logically consistent to espouse a principle of "plausible conservatism" with regard to model uncertainty and not explicitly recommend the same response to variability: (1) as a pragmatic matter, we believe scientists have more that they alone can contribute to a discussion of how to choose among competing scientific theories than they have to contribute to a discussion of what kind of individuals EPA should try to protect; and (2) we believe the public has more clearly expressed a preference for "erring on the side of safety" when the truth is unknown than it has regarding how much protection to extend to the extremes of variability distributions.

OCR for page 601
Page 606   comes practically irrelevant, although risk managers and public can remain assured that the probability of errors of underestimation remains constant and relatively small. Inherent Advantages of "Plausible Conservatism" It is perplexing to some members of this committee (and to many in the general populace) that the presumption that society should approach uncertain risks with a desire to be "better safe than sorry" has engendered so much skepticism. After all, perhaps it should instead be incumbent upon opponents of conservative defaults to defend their position that EPA ought to ignore or dilute plausible scientific theories that, if true, would mean that risks need to be addressed concertedly. That view, whatever its intellectual merits, seems at the outset not to give the public what it has consistently called for (explicitly in legislation and implicitly in the general conduct of professions ranging from structural engineering to medicine to diplomacy): namely, the attempt to guard against major errors that threaten health and safety. But the proposal for risk assessment based on "plausible conservatism" came about largely because of the wide variety of other factors supporting it, whether viewed through the lenses of logic, mathematics, procedure, or political economy. The following brief accounting of some of the virtues of a conservative orientation may seem somewhat superfluous, especially given the statements of earlier NRC committees on the topic.3 However, this committee's decision not to endorse "plausible conservatism" by consensus has prompted this more thorough enumeration of some factors some members had though were uncontroversial: A. "Plausible Conservatism" Reflects the Public's Preference Between Errors Resulting in Unnecessary Health Risks and those Resulting in Unnecessary Economic Expenditures. An examination of the two kinds of errors uncertainty in risk can cause supports the conclusion that society has not been indifferent between them. One type of error (caused by the overestimation of risk) leads to more resources invested than society would optimally invest if it knew the magnitude of the risk precisely. The other type (caused by underestimation of risk) leads to more lives lost (or more people subjected to unacceptably high individual risks) than society would tolerate if there was no uncertainty in risk. Whether the aversion to the latter type of error is due to the greater irreversibility of its consequences 3For example, consider this recent statement of the BEST Committee on Environmental Epidemiology (NRC, 1991): "public health policy requires that decisions be made despite incomplete evidence, with the aim of protecting public health in the future."

OCR for page 601
Page 607 compared to the former,4 the importance of regret (Bell, 1982) in most individual and social decision-making,5 or other factors is beyond our capacity to answer. What matters is, do Congress and the public view risk management as a social endeavor that should strive both for scientific truth and for the prudent avoidance of unnecessary public health risks, and therefore do not view risk assessment as purely an exercise in coming as close to the "right answer" as possible? If this is so, then the competing proposal offered in Appendix N-2 espouses an unscientific value judgment, and one that also is unresponsive to social realities. A counter-example may be illustrative here. In its recent indictment of conservatism in Superfund risk assessment, an industry coalition drew an extended analogy to link EPA's risk estimates with inflated predictions of the amount of time it would take someone to take a taxi ride to Dulles Airport (Hazardous Waste Cleanup Project, 1993). But this particular personal decision seems to be another prime example of where individuals and society would clearly prefer conservative estimates. As demonstrated below, any level of conservatism (positive, zero, or negative) corresponds to some underlying attitude towards errors of overestimation and underestimation. In this case, a conservative estimate of travel time simply means that the traveller regards each minute she arrives at the airport after the plane leaves as more costly to her than each minute of extra waiting time caused by arriving before the plane leaves. It is hardly surprising to conclude that a rational person would not be indifferent, but would rather be 10 minutes early than 10 minutes late to catch a plane. If, hypothetically, someone advising the traveller told her he wasn't sure whether the airline she chose would have a single ticket agent (and a 20-minute long line) or a dozen agents (and no line), it seems hard to believe that she would ask for a "best estimate" between zero and 20 minutes and allow only that much time (and even less likely she would assume that the long line simply couldn't happen). As long as the more "conservative" scenario was plausible, it would tend to dominate her thinking, simply because the decision problem is not about arriving at exactly the right moment, but about balancing the costs of a very early arrival against the qualitatively different costs of even a slightly late arrival. Again, reasonable people may differ widely about how large either asymmetry should be, but supporters of "plausible conservatism" are hard pressed to imagine not 4It is possible that profligacy in economic resources invested may also lead to adverse health consequences (MacRae, 1992). However, this "richer is safer" theory is based on controversial data (Graham et al., 1993), and at most offsets in an indirect way the more direct and irreversible consequences of underregulation in the eyes of the public. 5Anticipation of regret tends to make people choose courses of action that are less likely to leave them with the knowledge that they failed to take another available action that would have been much less damaging.

OCR for page 601
Page 608 admitting that some adjustment to make catching the plane more likely, or reducing the risk more probable, aligns with the expressed desires of the public. B. Conservative Defaults Help Increase the Chances that Risk Estimates Will Not be "Anti-Conservative." There are two different mathematical aspects of risk assessment under uncertainty that mitigate in favor of a conservative approach to selection of default options. Both factors tend to make risk estimates generated from conservative models less conservative than they might appear at first glance, and thus tip the balance further in favor of such models as minimally necessary to support prudent decisions. Let us assume at the outset that the assessor and decision-maker both desire that at the very least, risk estimates should not be "anti-conservative," that is, not underestimate the mean (arithmetic average) of the true but unknown risk. The mean, after all, is the minimum estimator that a so called "risk-neutral" decision-maker (e.g., a person who is not actually trying to catch a plane, but who stands to win a wager if she arrives at the airport either just before or just after the plane leaves) would need in order to balance errors of overestimation and underestimation. In this regard, there exists a basic mathematical property of uncertain quantities that introduces an asymmetry. For non-negative quantities (such as exposures, potencies, or risks), the uncertainties are generally distributed in such a way that larger uncertainty increases the arithmetic mean, due to the disproportionate influence of the right-hand tail. For example, if the median (50th percentile) of such an uncertainty distribution was X, but the assessor believed that the standard error of that estimate was a factor of 10 in either direction, then the 90th percentile (19X) and the arithmetic mean (14X) would be nearly identical; if the uncertainty was a factor of 25 in either direction, the mean and the 95th percentile would be virtually identical (see Table 9-4). Some of the most familiar examples of the need to impose a moderate "level of conservatism" in order not to underestimate the mean come from empirical data that exhibit variability. For example, it is unlikely, even in a state that includes areas of high radon concentration, that a randomly selected home would have a radon concentration exceeding approximately 10 picocuries/liter. Yet the mean concentration for all homes in that state might equal or even exceed 10 because of the influence on the mean of the small number of homes with much higher levels.6 6This mathematical truism that the more uncertainty, the greater the level of conservatism required not to underestimate the mean, seriously undermines one of the major claims made by those who accuse EPA of "cascading conservatism." If each of a series of uncertain quantities is distributed in such a way that a reasonably conservative estimator (say, the 95th percentile) approximates or even falls below the mean of that quantity, then the more steps in the cascade the less conservative the output becomes with respect to the correct risk-neutral estimator.

OCR for page 601
Page 609 The other basic mathematical advantage of introducing some conservatism into the scientific inferences that are made is the expectation that there may be other factors unknown to the assessor which would tend to increase uncertainty. This becomes a stronger argument for conservatism if one believes that more of these unknown influences would tend to increase than to decrease the true risk. Although it seems logical that factors science has not yet accounted for (such as unsuspected exposure pathways, additional mechanisms of toxicity, synergies among exposures, or variations in human susceptibility to carcinogenesis) would tend to add to the number or severity of pathways leading to exposure and/or greater risk, it is possible that "surprises" could also reveal humans to be more resistant to pollutants or less exposed than traditional analyses predict. C. "Plausible Conservatism" Fulfills the Statutory Mandate under which EPA Operates in the Air Toxics (and many other) Programs. The policy of preventive action in the face of scientific uncertainty has long been part of the Clean Air Act, as well as most of the other enabling legislation of EPA. Two key directives run through many of the sections of the Clean Air Act in this regard. First, various sections of the Act direct EPA to consider not merely substances that have been shown to cause harm, but those that "may reasonably be anticipated" to cause harm. As the D.C. Circuit court stated in its 1976 decision in Ethyl Corp. v. EPA, "commonly, reasonable medical concerns and theory long precede certainty. Yet the statutes and common sense demand regulatory action to prevent harm, even if the regulator is less than certain that harm is otherwise inevitable." Similarly, the Act has long required standards for air pollutants to provide "an ample margin of safety to protect public health." The leading case on the interpretation of Section 112, the 1987 case of Natural Resources Defense Council v. EPA, declared that In determining what is an "ample margin" the Administrator may, and perhaps must, take into account the inherent limitations of risk assessment and the limited scientific knowledge of the effects of exposure to carcinogens at various levels, and may therefore decide to set the level below that previously determined to be "safe."…[B]y its nature the finding of risk is uncertain and the Administrator must use his discretion to meet the statutory mandate. Again, support for the idea that "plausible conservatism" is the most rational approach for EPA to take is not necessarily based on a reading of the various statutes. After all, it is possible that the statutes may be changed in the near or far future. However, it seems central to EPA's mission that the Agency consider whether it is necessary to prevent or minimize adverse events, even events of low probability. Therefore, the Agency inevitably will find it necessary to use risk assessment techniques that are sensitive enough to reflect the risks of those events. At a minimum, its techniques must explore the nature of possible extreme outcomes, as a prelude to science-policy choices as to whether to factor those extremes into its risk characterizations. In essence, conservatism in the

OCR for page 601
Page 610 choice of default options is a way of making risk assessment a sensitive enough device to allow risk managers to decide to what extent they can fulfill the intent of the enabling legislation. For this reason, members of the committee advanced the proposition, which proved eventually to be controversial within the committee, that "plausible conservatism" gives decisionmakers some of the information they need to make precautionary risk management decisions. D. It Respects the Voice of Science, Not Only the Rights of Individual Scientists. By declaring that defaults would be chosen to be both scientifically supportable and health-protective, and that scientists would have to examine alternative models by these two criteria, EPA could help ensure that science will assume the leading role in defining evolving risk assessment methodology. Some have asserted that it shows disrespect for science to posit any standard for departure from defaults other than one that simply requires EPA to adopt "new and better science at the earliest possible time." But surely there is a generally inverse relationship between the amount of knowledgeable controversy over a new theory and the likely "staying power" and reliability of such "new science." At the extremes, EPA could either change its defaults over and over again with each new individual voice it hears complaining that a default is passé, or never change a default until absolute scientific unanimity had congealed and remained unshakable for some number of years. The "persuasive evidence" standard proposed here (see below) clearly falls between these two extremes. It reflects our belief that standards which rely more on scientific consensus than on the rights of individual scientists dissatisfied with the current situation are in fact more respectful of science as an institution. The only cost to a standard that values scientific consensus over "heed the loudest voice you hear" is that advocates of "new science" need to persuade the mainstream of their colleagues that new is indeed better. This standard is in fact a bargain for scientists, because it buys credibility in the public arena and some degree of immunity against being undercut by the next new theory that comes along. And, in addition to this give-and-take principle that elevates respect for scientific decisions by valuing the concord of scientists, advocates of "new science" must appreciate that the twin standards of plausibility and conservatism in fact remove a major source of arbitrariness in EPA's science-policy apparatus. If the Agency merely held up its defaults as unconnected "rules we live by" and required scientists to prove them "wrong,'' then the charge of bureaucracy-over-science would have merit. But this recommendation for EPA to reaffirm or rethink the set of defaults as "the most conservative of the plausible spectrum" sends a clear signal to the scientific community that each default only has merit insofar as it embodies those twin concepts, and gives scientists two clear bases for challenging and improving the set of inference assumptions.

OCR for page 601
Page 611 E. It Generates Routinely those Risk Estimates Essential to Various EPA Functions. The committee was also unable to reach agreement on the details of what roles "nonconservative" estimates should play in standard setting, priority setting, and risk communication, although the committee's recommendations in Chapter 9 reflect its belief that such estimates have utility in all of these arenas. However, no one has suggested that "nonconservative" estimates should drive out estimates produced via "plausible conservatism," but rather that they should supplement them. Indeed, the committee agrees that conservative estimates must be calculated for at least two important risk assessment purposes: (1) the foundation of the iterative system of risk assessment the committee has proposed is the screening-level analysis. Such analyses are solely intended to obviate the need for detailed assessment of risks that can to a high degree of confidence be deemed acceptable or de minimis. By definition, therefore, screening analyses must be conservative enough to eliminate the possibility that an exposure that indeed might pose some danger to health or welfare will fail to receive full scrutiny; and (2) even if EPA decided to use central-tendency risk estimates for standard-setting or other purposes, it would first have to explore the conservative end of the spectrum in order to have any clear idea where the expected value of the uncertain risk (as discussed above, the correct central-tendency estimate for a risk-neutral decision) actually falls. Because of the sensitivity of the expected value of a distribution to its right-hand tail, one cannot simply arrive at this midpoint in one step.7 For both reasons, risk assessment cannot proceed without the attempt to generate a conservative estimate, even if that estimate is only an input to a subsequent process. Therefore, the only argument among us is whether to modify or discard such estimates for some purposes other than screening or calculation of central tendencies, not whether they should be generated at all. Either way, a set of default assumptions embodying "plausible conservatism" must play some role. F. It Promotes an Orderly, Timely Process that Realistically Structures the Correct Incentives for Research. Many observers of risk assessment have pointed out that the scientific goal of "getting the right answer" for each risk assessment question conflicts directly with the regulatory and public policy goals of timeliness and striking a balance 7See Table 9-4 for various calculations showing how if the uncertainty is distributed continuously, the arithmetic mean can be very sensitive to the conservative percentiles. If instead, the uncertainty is dichotomous (say, the risk was either Y or zero depending on which of two models was correct), the expected value would depend completely on the value of Y and the subjective probability assigned to it. In either case, the upper bound must be estimated before the mean can be.

OCR for page 601
Page 618 any new information that might show the risk has been overstated; the emphasis here on scientific consensus does tend to slow the adoption of less conservative models at their early stages of development, but this should neither discourage thorough research nor discourage researchers from submitting quality data which EPA could readily incorporate into its existing model structure regardless of what effect it would have on the risk estimate. The fundamental concern about conservatism is that it has led to systematic exaggeration of all environmental health problems and has encouraged wasting of scarce resources on trivial risks. The latter part of this charge is a subjective matter of economic and social policy that falls outside this committee's purview. And while the former concern is an empirical one, it has sparked a vigorous debate that is far from resolved. On one side, those convinced that EPA's procedures yield estimates far above the true values of risk can cite numerous examples where individual assumptions seem to each contribute more and more conservatism (Nichols and Zeckhauser, 1988; OMB, 1990; Hazardous Waste Cleanup Project, 1993). Others believe the evidence shows that current procedures embody a mix of conservative, neutral, and anti-conservative assumptions, and that the limited observational "reality checks" available suggest that existing exposure, potency, and risk estimates are in fact not markedly conservative (Allen et al., 1988; Bailar et al., 1988; Goodman and Wilson, 1991; Finley and Paustenbach, in press; Cullen, in press). The practical and constructive question EPA must grapple with, however, is not whether "plausible conservatism" is ideal, but whether it is preferable to the alternative(s). The primary alternative to this proposal (Appendix N-2) directs EPA risk assessors to use defaults on the basis of the "best available scientific information," with the apparent goal of generating central-tendency estimates (CTEs) of risk. According to proponents of this approach, there is a clear boundary line between the "objective" activity of risk assessment and the value-laden activity of risk management, and the imposition of conservatism (if any) should occur in the latter phase, with managers adding "margins of safety" to make precautionary decisions out of the CTEs. In comparing this proposal with the alternative, it is important to consider the two fundations of the latter approach, the CTE (or "most scientific estimate'') and the margin of safety, and ask whether either concept is really as appealing as it may sound. The margin of safety idea is problematic, for one obvious reason: it is only through exploring the conservative models and parameter values that analysts or managers can have any idea what they are trying to be "safe" from. Perhaps it would be ideal for the manager rather than the assessor always to tailor the level of conservatism, but in reality, only the assessor can initially determine for the manager what a "conservative decision" would entail, because the assessor has the access to information on the spectrum of plausible values of risk. Applying any kind of generic safety factor to CTEs of risk would certainly result in a haphazard series of decisions, some (much) more conservative than a reasonable

OCR for page 601
Page 619 degree of prudence would call for, others (much) less so. Besides, taken as a whole the committee's report returns some discretion and responsibility to the risk manager that assessors have admittedly usurped in the past by presenting point estimates alone. The committee's emphasis on quantitative uncertainty and variability analysis gives risk managers the ability to tailor decisions so that the degree of protection (and the confidence with which it can be ensured) are only as stringent as they desire. But in the narrow area of model uncertainty, this proposal deems it unwise to encourage risk managers to guess at what a protective decision would be, by censoring information about models which, although conservative, are still deemed by experts to be plausibly true. The CTE also has potentially fatal problems associated with it. Even if the models used to construct CTEs are based on "good science," we have argued (above) that these estimates are not designed to predict the expected value of potency, exposure, or risk (for which the conservative end of the spectrum must be explored and folded in), but instead are surrogates for other central-tendency estimators such as the median or mode (maximum likelihood). These latter estimators generally do not even give neutral weight to errors of underestimation and overestimation, and hence must be regarded as "anti-conservative." But advocates of CTEs have also failed to consider the problems of the models from whence they come. The following are four examples, illustrating four archetypes of central-tendency estimation, which suggest that on a case-by-case basis, ''good science" may not be all its proponents advertise it to be: Case 1: "More science" merely means more data. Some of the alternative CTE estimates advocated by critics of conservatism are alleged to be more scientific because they make use of "all the data at hand." This distinction is hardly a cut-and-dried one, however. For example, consider the current EPA default of using the bioassay result from the most sensitive of the (usually no more than four) sex-species combination of rodent tested. Call this potency estimate "A," and the alternative that could be derived by pooling all (four) data sets as "ABCD." Assuming that we know very little about the relative susceptibilities of different varieties of rodents versus the average human (in general or for the particular substance at issue), we must logically admit that it is possible the true risk to the average human may be greater than that implied by A, less than that implied by ABCD, or somewhere in between. One could prefer ABCD to A on the basis of a different value judgment about the costs of overestimation and underestimation, but the only "scientific" difference is that ABCD makes use of more data. But "purchasing" an array of data is akin to buying cards in a blackjack game: "more is better" only holds true as long as all the individual elements are valuable rather than otherwise. Assuming rodent varieties A through D differ significantly (or we wouldn't be quarreling over the two estimators), then humans must either be most like variety A or most like one of the other three. If the former, then data points B, C, and D dilute and ruin what is already in fact

OCR for page 601
Page 620 the "best estimate"; if the latter, then more is indeed better (in the sense of moving us closer to the truth). Therefore, EPA's true dilemma is whether the additional data are more likely to hurt or to help, and this too is a policy judgment about balancing estimation errors, not a simple matter of "good science." However, as a matter of process and of implementation, there is a clear difference between a policy of choosing A and a policy of choosing ABCD. The former policy sets up incentives to actually advance the scientific foundation and get to the truth of which sex/species is the best predictor in specific or in general; when such information becomes available, "good science" will justifiably carry the day. On the other hand, the latter policy only encourages additional rote application of current bioassay designs to generate more data that assessors can pool. A related example in the exposure assessment arena is discussed in Chapter 10. A CTE of approximately 7 years of exposure to a typical stationary source of toxic air pollutants is indeed based on much more data (in this case, data on the variation in the number of years a person stays at one residence before moving) than is the standard 70-year assumption EPA has used. But as noted in Chapter 10, those data, although valid at face value, may speak to a different question than the one EPA must address. To ensure that individual lifetime risk is correctly calculated in a nation containing thousands of such sources, EPA would need to consider data not only on years at one residence, but also on the likelihood (that we consider substantial) that when someone moved away from proximity to a source, he or she would move to an area where there is still exposure to the same or similar carcinogens. In both examples, "a great deal more data" (on interspecies susceptibility or on autocorrelation of exposure rates as people move, respectively) would certainly be preferable to EPA's status quo assumption, but questions arise as to whether "a little more data" help or hurt the realism of the calculations. Case 2: "More science" means constructing chimeras out of incompatible theories. One brand of CTE that has gained some currency in recent years allegedly provides a means of incorporating all of the plausible scientific models, much as meta-analysis incorporates all of the available epidemiologic studies or bioassays on a particular compound. Unfortunately, there may be a world of difference between pooling related data sets and averaging incompatible theories. In Chapter 9, we discuss the obvious pitfalls of such hybrid CTEs, which arguably confuse rather than enrich the information base from which the risk manager can choose a course of action. For example, when faced with two conflicting theories about the potency of TCDD, EPA arguably should not have tried to change its potency estimate to "split the difference" between the two theories and make it appear that new science had motivated this change (Finkel, 1988). Rather, EPA could have achieved the same risk management objective by loosening regulatory standards on TCDD if it felt it could justify this on the grounds that there was a significant probability that the existing risk estimate

OCR for page 601
Page 621 was excessively conservative. The committee could not agree on what sort of advice to give decisionmakers when some risk is either zero (or nearly zero) or is at some unacceptably high level X, depending on which of two fundamentally incompatible biologic theories is in fact the correct one. The committee did agree, however, that analysis should certainly not report only a point estimate of risk equal to (1-p)X, where p is the subjective probability assigned to the chance that the risk is (near) zero. In the specific context of default options, this proposal remains that EPA should retain its "plausible conservative" default until scientific consensus emerges that the alternative model supplants the default at the conservative end of the plausible set of model choices. Case 3: "More science" means introducing more data-intensive models without considering uncertainty or variability in the parameters that drive them. This particular problem should be easy to rectify by incorporating the committee's recommendations in Chapters 9 and 10, but it is mentioned here because to date EPA has considered several departures from defaults (e.g., the case of methylene chloride, at least as interpreted by Portier and Kaplan, 1989) in which the level of conservatism may have changed abruptly because the parameters of the default model were assessed conservatively, but the parameters in the new model were either CTEs or point estimates of unknown conservatism. All of the burden should not fall upon purveyors of new models, however; EPA needs to level the playing field itself by systematically exploring the conservatism inherent in the parameters of its default models (for example, as we discuss in Chapter 11, is the surface area or 3/4 power correction a conservative estimate of interspecies scaling, or something else?). Case 4: "More science" is clearly an improvement but not airtight. It is noteworthy that the most detailed case-specific reassessment of a default assumption, the CIGA case discussed in Chapter 6, has recently been called into question on the grounds that the new science casts serious doubt upon EPA's default as applied to existing animal data, but does not itself provide unimpeachable support for an alternative risk estimate (Melnick, 1993). We do not presume to reach any conclusion about this dispute, or about its implications for the general process of departing from defaults. As a matter of process, the CIGA case would probably meet the "persuasive evidence" test recommended here, and therefore one should not necessarily characterize EPA's acceptance of this new science as a mistake in policy. However, for purposes of risk communication, EPA should understand and emphasize that scientific consensus in issues such as these does not necessarily imply scientific truth. Conclusions In summary, EPA's choice between competing principles for choosing and departing from defaults has important and provocative implications for four areas of environmental science and EPA programs.

OCR for page 601
Page 622 • Values. The choice between "plausible conservatism" and "best science" is inescapably one of science and of values. As this Appendix shows, both principles rely in part on science and decision theory, and both embody specific sets of value judgments. Unfortunately, some of the critics of "plausible conservatism" have shone a spotlight on the values inherent in that position while ignoring the value judgments inherent in the alternatives. We have argued that in many cases (and especially in the most practically important cases, where a dichotomy exists about whether a model predicting unacceptably high risk or a model predicting zero risk is correct), CTEs are difficult to derive and may not be meaningful. But even if CTEs were free of these pitfalls, one must recognize that choosing them over conservative estimates is a value-laden choice, and indeed that choosing among the three or more different brands of CTE, which may differ from each other by orders of magnitude in some cases, also requires value choices. The mode (maximum likelihood estimator) is a CTE with a particular purpose in decision theory; it maximizes the probability that the decision flowing from this estimate will be "right,'' without regard either to the direction or magnitude of deviations from this ideal. The median CTE seeks to balance the probability of the two types of error, again without regard to the magnitude of either; this, too, represents a particular value orientation. Finally, the mean attempts to balance the (unweighted) product of the probability and magnitude of errors of either type. The conservative estimate rounds out this set of possible choices, as it simply seeks to balance a weighted product of the probability and magnitude of error (see Figure N1-1, which gives examples of what purpose each estimator serves). Thus, the choice of risk assessment estimates can certainly be explicit, and can and should be unbiased in the sense of deriving from an open and honest process, but it cannot be wholly objective or value-neutral. Because leaving EPA with no principle for choosing among these estimators would itself be a value-laden decision, some members of the committee have advocated "plausible conservatism," cognizant that this is an alternative judgment. • Science. The tendency of critics of our position to hold up terms such as "best science" or "credible science" as the alternative should not confuse readers into inferring that any alternatives must espouse "bad science" or "incredible science." Defaults which are not credible have no place in either of the proposals advocated in these appendices. Supporters of "plausible conservatism" believe defaults based on this principle have additional merits beyond their credibility, and that "best science" ought to be more than data for data's sake or anti-conservatism for its own sake. Looking to the future, none of the members of the committee wishes to "freeze" risk assessment science in its current incarnation, or to suppress information about new scientific ideas that challenge existing ones. The intelligent question is not whether to improve the science, but how and when to include it in risk characterization for risk management. Again, in

OCR for page 601
Page 623   the vexing paradigm case where an alternative model incompatible with the default predicts vastly smaller or zero risk, the operational decision involves whether to leap to the new risk characterization, to develop a hybrid of the two theories, or to proceed cautiously until general scientific consensus supports the alternative. This section of the chapter has explored reasons why the last alternative is preferable, and if thoughtfully implemented, will fuel rather than freeze scientific research. • EPA Practice. As the previous chapter indicates, EPA has always had to walk the fine line between too much flexibility and too much rigidity. Even though the committee is concerned that EPA has not had an underlying rationale for its decisions to this point, on balance some of us feel that its individual decisions to depart from defaults have managed this tension admirably well. We are unaware of serious charges that EPA has been unreceptive to new scientific information; indeed, according to some of the references cited above, even in the methylene chloride and CIGA cases, EPA has arguably been too quick to adopt "new science." • Risk Management. This chapter has explored in some detail the intertwining, rather than the boundary, of risk assessment methodology and risk management practice. None of the suggestions made here violate the principle that risk management concerns should not unduly influence the conduct of risk assessment; they only reinforce the point that risk assessment exists to provide useful answers to the questions risk managers choose to ask. And whatever one thinks about the advisability of a clear attempt to separate risk assessment from risk management, this discussion has shown that a "plausible conservatism" orientation is no more violative of that boundary than a central tendency orientation would or could be. Furthermore, both "plausible conservatism" and the "best science" alternative leave vast room for risk managers to exercise their rightful discretion, particularly in the selection of decision alternatives and the integration of information external to the risk assessment (e.g., cost and efficiency estimates, public concerns) on which real decisions often hinge. Finally, we hope to have dispelled the false choice that others have posited between valuing science and the values held by scientists. Surely as scientists or otherwise, our values include respect for public health precaution, for predictably and order, and for striking a thoughtful and appropriate balance between the inevitable errors that uncertainty causes. We could decide that other values outweigh these, but we cannot rationalize such a choice by costuming it in the garb of "good science." "Plausible conservatism'' embraces the idea that assessors and managers need not abandon either their valuing of science or their values as scientists. Supporters of this principle hope that EPA will follow this path, even though it is presented here as a recommendation that the full committee could not agree to.

OCR for page 601
Page 624 Afterthoughts The alternative view which follows (Appendix N-2) was written after this Appendix was completed. Together, these two statements reflect reasoned disagreement which I hope will provide EPA with "grist for the mill" to help it resolve important questions about risk assessment principles and model uncertainty. However, there are a number of inconsistencies and misinterpretations in Appendix N-2 that I believe cloud this debate. Some of the ambiguity stems from the lack of responsiveness to important issues raised in this Appendix. For example, Appendix N-2 asserts that "risk managers should not be restricted by value judgments made during risk assessment," but nowhere does it explain how this vision could be realized, in light of the assertions herein that a vague call for ''full use of scientific information" must either impose a set of value judgments of its own or else restrict risk assessors to presenting every conceivable interpretation of every model, data set, and observation.14 Similarly, the statement that "risk characterizations must be as accurate as possible," and the implicit equating of accuracy with the amount of data amassed, responds neither to the assertion that accuracy may not be the most appropriate response to uncertainty nor to the four examples in Appendix N-1 showing that "more science" may lead to less accuracy as well as substitute risk-neutral or risk-prone value judgments for risk-averse ones. There are legitimate reasons for concern about a principle of "plausible conservatism," concerns that, if anything, might have been strengthened by more specificity in Appendix N-2 about the putative merits of an alternative. But in at least three respects, the material in N-2 misinterprets the stated intent of the "plausible conservatism" proposal, thus making a fair comparison impossible. (1) Proponents of the "plausible conservatism" approach assuredly do not believe that "the fundamental output of a risk assessment is [or should be] a single estimate of risk: one number." This "red herring" permeates Appendix N-2 despite the clear statements in Appendix N-1 that default options only provide a scaffolding upon which all of the uncertainties and variabilities contingent on the models selected must be assessed and communicated. In fact, Chapter 9 of the report states quite clearly the committee's view that risk assessors must abandon their reliance on single point estimates and instead routinely provide quantitative descriptions of uncertainty (preferably via probability distributions). Indeed, three of the four specific recommendations in Appendix N-1 for implementing the "plausible conservatism" proposal reinforce the purpose of Chapter 14In fact, the Appendix contradicts itself a few pages later when it states that "weighing the plausibility of alternatives is a highly judgmental evaluation that must be carried out by scientists." This is a clear call for scientists to play a role in science policy, which Appendix N-1 clearly endorses, but then the authors of N-2 return to the "hands off" view and re-contradict themselves with the admonition that "scientists should not attempt to resolve risk management disputes by influencing the choice of default options."

OCR for page 601
Page 625   9 by emphasizing "the uncertainty distributions that result" from proper risk assessments conducted according to guidelines containing default options.15 The only uncertainty not accounted for by estimating risk using default models unless there are specific reasons to replace them is the subjective probability that the other model(s) is/are correct. Even though this additional uncertainty may be substantial, the committee agreed in Chapter 9 that at present, both the methodology for coming up with the subjective weights and the theory for how to meaningfully "average" irreconcilable models are sufficiently rudimentary that a single risk characterization covering all plausible models would be a precarious basis for risk management and communication. Thus, however the proposal in Appendix N-1 and Appendix N-2 differ, they do not advocate different "fundamental outputs of risk assessment.'' (2) The authors of Appendix N-2 characterize their recommendation that "risk managers can and should override conservative default value judgments in the risk assessment process whenever they believe it is appropriate public policy to do so" as one that "contrasts sharply with the approach advocated by Dr. Finkel." For the record, nothing in Appendix N-1 advocates constraining the activities of risk managers in any such way. Indeed, the last paragraph of Appendix N-1 speaks to the "rightful discretion" risk managers should have to supplement, subordinate, or discard quantitative risk information when they deem this necessary to make sound decisions. If a risk characterization (again, a distribution, not just a number) emerging from the "plausibly conservative" models chosen suggests a significant risk, the manager can still let other concerns (economics, feasibility, equity, or even lack of confidence in the scientific underpinning of the risk assessment) justify not reducing the risk. What proponents of "plausible conservatism" object to, and what Appendix N-2 either leaves open or endorses (it is unclear), is for someone (a scientist? a manager?) to declare as a matter of science that a risk already is acceptable, simply because other models may exist that give more sanguine risk predictions than do the conservative defaults. (3) Despite all their criticism of how even examining "conservatism" intrudes into policy, the authors of Appendix N-2 admit they "do not object to ["plausible conservatism"] for selecting the default options," only to its use in deciding when to displace an option. What justification could make the same principle appropriate at the outset but objectionable from then on? Their stated objection is that it will "freeze risk characterizations at the level determined by 15 A substantial amount of uncertainty may be contributed by the parameters that drive risk models, even before interindividual variability is taken into account. For example, even if one specifies that the linearized multistage model must be used, the uncertainty in cancer potency due only to random sampling error in the typical bioassay can span five orders of magnitude at a 90 percent confidence level (Guess et al., 1977).

OCR for page 601
Page 626   the conservative default options." Appendix N-1, however, argues that consensus processes in science neither intend to nor result in the "freezing" of science, only in "freezing out" unpersuasive or poor-quality science until it improves. It seems, then, that the authors of Appendix N-2 really do object to reliance on prudence and conservatism for the initial selection of the defaults, and only tolerate the existing defaults because of their expectation that they could be abandoned speedily. In contrast to some of the issues raised above, where there really is less disagreement that Appendix N-2 indicates, here there is more controversy than Appendix N-2 admits to. Our lack of consensus on this most fundamental issue—how to choose and how to modify default options—is what caused the committee to decide not to recommend any principles for meeting these challenges. References Allen, B.C., K.S. Crump, and A.M. Shipp. 1988. Correlation between carcinogenic potency of chemicals in animals and humans. Risk Anal. 8:531-544. Bailar, J.C., III, E.A. Crouch, R. Shaikh, and D. Spiegelman. 1988. One-hit models of carcinogenesis: Conservative or not? Risk Anal. 8:485-497. Bell, D. 1982. Regret in decision-making under uncertainty. Operations Res. 30:961-981. Cullen, A. In press. Measures of compounding conservatism is probablistic risk assessment. Risk Anal. EPA (U.S. Environmental Protection Agency). 1986. Guidelines for carcinogen risk assessment. Fed. Regist. 51:33992-34003. Finkel, A. 1988. Dioxin: Are we safer now than before? Risk Anal. 8:161-165. Finkel, A. 1989. Is risk assessment really too "conservative?": Revising the revisionists. Columbia J. Environ. Law 14:427-467. Finley, B., and D. Paustenbach. In press. The benefits of probabilistic techniques in health risk assessment: Three case studies involving contaminated air, water, and soil. Paper presented at the National Academy of Sciences Symposium on Improving Exposure Assessment, Feb. 14-16, 1992, Washington, D.C. Risk Anal. Goodman, G., and R. Wilson. 1991. Quantitative prediction of human cancer risk from rodent carcinogenic potencies: A closer look at the epidemiological evidence for some chemicals not definitively carcinogenic in humans. Regul. Toxicol. Pharmacol. 14:118-146. Graham, J.D., B.-H. Chang, and J.S. Evans. 1992. Poorer is riskier. Risk Anal. 12:333-337. Hazardous Waste Cleanup Project. 1993. Exaggerating Risk: How EPA's Risk Assessments Distort the Facts at Superfund Sites Throughout the United States. Hazardous Waste Cleanup Project, Washington, D.C. MacRae, J.B., Jr. 1992. Statement of James B. MacRae, Jr., Acting Administrator, Office of Information and Regulatory Affairs, U.S. Office of Management and Budget. Hearing before the Committee on Government Affairs, U.S. Senate, March 19, Washington, D.C. Melnick, R.L. 1992. An alternative hypothesis on the role of chemically induced protein droplet (2-globulin) nephropathy in renal carcinogenesis. Regul. Toxicol. Pharmacol. 16:111-125. Nichols, A., and R. Zeckhauser. 1988. The perils of prudence: How conservative risk assessments distort regulation. Regul. Toxicol. Pharmacol. 8:61-75.

OCR for page 601
Page 627 NRC (National Research Council). 1991. Environmental Epidemiology. Public Health and Hazardous Wastes, Vol. 1. Washington, D.C.: National Academy Press. OMB (U.S. Office of Management and Budget). 1990. Regulatory Program of the U.S. Government, April 1, 1990-March 31, 1991. Washington, D.C.: U.S. Government Printing Office. Portier, C.J., and N.L. Kaplan. 1989. The variability of safe dose estimates when using complicated models of the carcinogenic process. A case study: Methylene chloride. Fundam. Appl. Toxicol. 13:533-544. Tollefson, L., R.J. Lorentzen, R.N. Brown, and J.A. Springer. 1990. Comparison of the cancer risk of methylene chloride predicted from animal bioassay data with the epidemiologic evidence. Risk Anal. 10:429-435.

OCR for page 601