Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

4 Department of Homeland Security Decision Requirements for Risk Management With finite resources for biodefense, the United States must decide how to invest optimally to best mitigate bioterrorism risk. Reducing uncertainty in risk analysis results has no direct impact on risk reduction; only the implementation of effective risk management strategies can reduce risk. —Department of Homeland Security, Bioterrorism Risk Assessment, 2006 RISK MANAGEMENT REQUIRES TIMELY, How much lead time (i.e., response time) is needed ACCURATE INFORMATION to implement effective interventions? How much lead time will surveillance provide? What assets should be Those who, for reasons enumerated in previous chapters, pre-positioned where and when to reduce and or man- have need of information from the Department of Homeland age the risk? Are transportation and communication Security (DHS) regarding the risk of terrorist acts need that resources sufficient to handle the surge in usage? information to be accurate, timely, and valid. In 2002, the • How many and what type of staff would be needed General Accounting Office (now Government Account- to prevent, respond to, contain, or manage the conse- ability Office) described the need to acquire and use the quences of a bioterrorist attack? How should clinics following information about the risk of terrorist acts in order be designed for optimal provision of services in a to achieve homeland security goals and objectives: (1) Who crisis (e.g., mass vaccination, drug dispensing, patient will do what and for what reason? (2) How (in what form), triage, flow, and care) to effectively minimize the con- where, and when will they do it? and (3) What will they use sequences of a biological terrorism attack? in order to do it? (GAO, 2002). Additionally, for bioter- rorism, stakeholders and decision makers need answers to These questions lead to further questions. For example, what several types of questions (Danzig, 2003; Fischhoff et al., specific interventions would be needed (i.e., vaccination for 2006; Whitworth, 2006): smallpox versus antibiotics for anthrax)? How fast should they be delivered? What staff and supplies will need pre- • What is the probability of a particular biological terror- positioning? (Cooper, 2006). ist threat and how imminent is an event? What would In reviews of unsuccessful past attempts to prevent, the consequences of the event be, characterized in prepare for, or respond efficiently to terrorist attacks, the terms of when, where, and in what populations? inability to put the right information into the hands of key • Can real-time detection be achieved to determine if an decision makers at the right time and in an understandable event has already occurred, if it is part of a larger plan, format has been identified as a major factor in those failures. or if it is a false alarm (perhaps intentionally generated Failure can result in increased suffering and fatalities through by terrorist actions)? How many ill and/or affected setting the wrong priorities and policies, in the underfunding people would be expected for attacks by different of important programs, and in poorly conceived plans (Aaby agents? How fast would different infectious agents in et al., 2006). Several barriers to the effective use of informa- a bioterrorist attack spread? How fast would people die tion in decision making for the prevention of, preparedness from different agents? How sick would survivors be? for, and response to bioterrorism have been identified (Ware • What are the most effective ways to manage that risk? et al., 2002): How effective and feasible are different strategies for the prevention, containment, and reduction of conse- • There are many stakeholders with varying degrees of quences of a bioterrorism event? Would public health authority for making and implementing decisions (e.g., measures be able to limit the spread of the infection the responsibilities for domestic bioterrorist attacks in to a small proportion of the total population? What the United States may involve more than 100 different impact would the availability and delivery of effec- government organizations). tive interventions have on potential consequences? 34 

DEPARTMENT OF HOMELAND SECURITY DECISION REQUIREMENTS FOR RISK MANAGEMENT 35 • Authorities, roles, responsibilities, tasks, and indicators The BTRA provides information to many stakeholders. of success frequently are unclear and poorly understood DHS identifies the primary customers for information from or coordinated. the BTRA as follows: • High volumes of different types of data and information (e.g., subjective judgments, objective observations, • White House Homeland Security Council: relative risks historical data, analytical data, probabilistic data, mod- and overall vulnerabilities; eling, simulation results) from disparate sources are • Department of Health and Human Services: medical presented in nonstandard and often poorly understood countermeasures needs; formats, flooding the system as crises are unfolding. • Department of Homeland Security/Infrastructure Pro- • Significant organizational friction frequently ex- tection: relative risks of different attack scenarios; ists among the producers, owners, stakeholders, and • Department of Homeland Security/Office of Intelli- consumers of information. Critical information is gence and Analysis: high-leverage intelligence needs; frequently owned or held by public and/or private • Department of Homeland Security/Science and Tech- organizations, accompanied by a general reluctance to nology: high-leverage scientific gaps; share information across these sectors. • Departments of Agriculture and of Health and Human • The producers and owners of important information Services: food security; often compartmentalize it in order to protect secu- • Environmental Protection Agency: water security; rity or to protect sources, at the expense of the timely and and integrated sharing of data and interpretation of • Department of Agriculture: agricultural agents and information. protection of the food supply. • Decision makers often have widely varying objec- tives and frequently little understanding of the medi- DHS stakeholders need risk analysis, including risk cal and scientific background needed to inform their management, for strategic planning, operations, and foren- decisions. sics. Further, Homeland Security Presidential Directive 10 (HSPD-10): Biodefense for the 21st Century (The White House, 2004) states that the “United States requires a con- THE BIOLOGICAL THREAT RISK ASSESSMENT tinuous, formal process for conducting routine capabilities SHOULD SUPPORT RISK MANAGEMENT assessments to guide prioritization of our on-going invest- The GAO (2002) report identified risk assessment as ments in biodefense-related research, development, plan- an important tool and source of information for strategic ning, and preparedness. These assessments will be tailored decision making for the prevention of bioterrorism, risk to meet the requirements in each of these areas. Second, reduction, preparedness, and response to bioterrorism. As the United States requires a periodic senior-level policy described in Chapter 3 of this report, the DHS Biological net assessment that evaluates progress in implementing Threat Risk Assessment (BTRA) of 2006 is one of the first this policy, identifies continuing gaps or vulnerabilities in terrorism risk assessment efforts to integrate information our biodefense posture, and makes recommendations for from a variety of sources to meet information needs. Further, re-balancing and refining investments among the pillars the BTRA of 2006 presents sensitivity analyses that permit of overall defense policy.” To the extent that the BTRA of an examination of the impact of different measures that could 2006 (with its subsequent improvements and revisions) is be taken to mitigate identified consequences of interest (i.e., used for risk analysis, the committee believes that it is most morbidity and mortality). However, as noted in Bioterror- applicable to supporting strategic decisions (those that ad- ism Risk Assessment, the DHS 2006 report that describes the dress the setting of priorities and policies, the acquisition BTRA methodology, with finite resources for biodefense, the and pre-positioning and/or allocation of resources, and the United States must decide how to invest optimally to best development of infrastructure), but that it is not designed to mitigate bioterrorism risk. As that report points out, risk as- support operations or forensics. sessment alone has no direct impact on risk reduction; “only In 1997, the Presidential/Congressional Commission on the implementation of effective risk management strategies Risk Assessment and Risk Management (1997 a,b; Omenn, can reduce risk” (DHS, 2006). 2003) agreed on a framework for environmental health risk management, which is applicable to managing risks involved Recommendation: Subsequent revision of the BTRA with bioterrorism. This framework has six stages: (1) formu- should increase emphasis on risk management. An in- late the problem in a broad public health context, (2) analyze creased focus on risk management will allow the BTRA to better support the risk-informed decisions that homeland   Rear Admiral Jay Cohen, Undersecretary of Science and Technology, security stakeholders are required to make. Department of Homeland Security. 2007. “DHS Science and Technology: Enabling Technology to Protect the Nation.” Briefing to the committee, February 9, 2007, Washington, D.C. 

36 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT the risks, (3) define the options to address the risks, (4) make that stakeholders have in the model’s methods, the validity sound risk reduction decisions, (5) implement those actions, of assumptions and data used to develop the model, and and (6) later evaluate the effectiveness of the actions taken. the level of understanding of the model’s outputs. Lack of The BTRA of 2006 focused on risk assessment, which confidence can be caused by an insufficient understanding addresses stages 1 and 2 above. However, DHS intends for its of or disagreement with relationships hypothesized among BTRA to be used by risk managers to “test and evaluate risk variables, the mathematical foundations of the model, and/ mitigation strategies and their impact on bioterrorism risk” or the validity of assumptions and values assigned to the (DHS, 2006, Ch. 1, p. 3). To broaden the focus to include model’s parameters. All of these problems can be mitigated risk management, stages 3 and 4 must be addressed. Stage by improved transparency. 3, defining the options to address the risks, includes identi- fying potential countermeasures and estimating the costs of RISK ASSESSMENT TRANSPARENCY deploying the countermeasures. So that the risk assessments IMPROVES CONFIDENCE provided by the BTRA can be effectively used, each poten- tial countermeasure must be mapped to a set of parameters In contemplating a complex problem involving uncer- within the model. Stage 4, making sound risk reduction tainty and risk, such as that involved with the threat of bioter- decisions, requires several related activities (Boardman et rorism, mental arithmetic can be riddled with error, while risk al., 2006): assessment models enable repeatable calculations, including sensitivity analyses, which explore the effects of uncertain • Estimation of risk reduction (in, for example, expected parameters on important consequences. However, risk as- lives, life-years, or quality-adjusted life-years) obtained sessment models can fail to include important knowledge by allocating countermeasures, as discussed in Chapter that is not readily quantified and/or understood, potentially 2 of this report; compromising the validity of model outputs (Fischhoff et • Optimization of the allocations, which identifies, for al., 2006). each given resource level or budget, the allocation of The accuracy of quantitative bioterrorism risk assess- countermeasures that maximizes total risk reduction ment models and the confidence placed in them depend on (or equivalently, for a given level of risk reduction, the validity of the assumptions and the availability of sound identifies the least-cost deployment of countermeasures data for each of the biological agents being analyzed. A for achieving a particular level of risk), as discussed in good model based on strong data, such as the one reported Chapter 7 of this report; in Whitworth (2006) that describes the difference between a • Optimization of risk-benefit, which, given the optimal response for an anthrax attack and that for a smallpox attack, allocation of resources for different budget levels and can inform judgment about the effectiveness of different in- a willingness-to-pay value for incremental risk reduc- terventions (i.e., antibiotics for anthrax versus vaccinations tion, identifies the best overall level of resources (and for smallpox) and the pre-positioning of staff and supplies corresponding best allocation); and to respond to an attack effectively. • Valuation of options, or consideration of how the re- Conversely, the lack of data and/or uncertainty in model sults from the previous stages are likely to change if parameters also can have important implications for the additional countermeasures are added, and using this degree of confidence placed in the results of risk assess- information, making decisions about which additional ment models (Elderd et al., 2006). Unfortunately, data for countermeasures are most worth developing. key parameters of many biothreat agents of concern are not available. If decision makers understand and trust the model, they will be more likely to use it with differ- TRANSPARENCY OF RISK ASSESSMENT IS ent assumptions and to test different response strategies NECESSARY FOR SUCCESSFUL RISK MANAGEMENT ( ­ Fischhoff et al., 2006). Transparency, as described by Oliver (2004), has been While transparency is a major factor in establishing defined in different dictionaries as “free from guile,” “candid confidence and trust in the methods of and outputs from or open,” or “forthright,” and has been applied to business risk assessment models, modeling is an important step to- and organizations as “allowing others to see the truth without ward transparency as it requires that assumptions be made trying to hide or shade the meaning or altering the facts to explicit. However, achieving transparency also requires the put things in a better light.” Oliver summarizes the current careful, explicit documentation of a model’s mathematical use of the word transparency as “letting the truth be available and structural foundations and of the sources of data used for others to see if they so choose, or perhaps think to look, in the analysis—a prerequisite for any scientific study—for or have the time, means, and skills to look,” and involving   Marc Lipsitch, Harvard School of Public Health. “Notes to the National “active disclosure.” Whether and how often risk assessment models are used Research Council Committee on Methodological Improvements to the DHS’s Biological Agent Risk Analysis.” Written communication to the in risk management will depend on the level of confidence committee, February 2007. 

DEPARTMENT OF HOMELAND SECURITY DECISION REQUIREMENTS FOR RISK MANAGEMENT 37 the purpose of facilitating external review. It is essential that THE DEPARTMENT OF HOMELAND SECURITY’S analysts document the following: (1) how they construct BTRA OF 2006 WAS NOT TRANSPARENT risk assessment models, (2) what assumptions are made to As described in Chapter 3 of this report, the model used characterize relationships among variables and parameters in the BTRA of 2006 is extremely complex, with 17 stages and the justifications for these, (3) the mathematical foun- and thousands of parameters for each of 28 biothreat agents dations of the analysis, (4) the source of values assigned to of concern (DHS, 2006). The considerable data that are parameters for which there are no available data, and (5) lacking for many of the parameters and probabilities in the anticipated impact of uncertainty for assumptions and the model may lead to questions about the validity of the parameters (Brisson and Edmunds, 2006). model’s output and to a lack of confidence and trust in the When working with classified or sensitive information, as results. Moreover, the results of simulations are presented in is the case with bioterrorism prevention, preparedness, and graphs, charts, and tables that are also complex and difficult response, there may be need to restrict the access to some to interpret and use. information to certain groups of users to protect overall The committee also finds the documentation for the model security. However, security or confidentiality concerns that used in the BTRA of 2006 to be incomplete, uneven, and can negatively affect the level of transparency reached in risk extremely difficult to understand. The BTRA of 2006 was assessment modeling include the following: done in a short time frame. However, deficiencies in docu- mentation, in addition to missing data for key parameters, • The compartmentalization of model development, would make reproducing the results of the model impossible algorithms, and execution for security concerns and to for independent scientific analysis. For example, although protect information and data sources; Latin Hypercube Sampling is mentioned in the description • Private-sector reluctance to share information, com- of the model many times as a key feature, no actual sample monly due to the protection of proprietary consider- design is specified. Although antithetic sampling methods ations, across sectors; and (e.g., matched samples or reused random number streams) • The need to balance civil liberties of citizens against the evidently are employed, insufficient details are provided need to keep important classified and sensitive informa- on how or where these numbers are generated, precluding tion out of the hands of terrorists. a third party, with suitable software and expertise, from reproducing the results—violating a basic principle of the Given the importance of establishing the confidence of de- scientific method. cision makers and stakeholders in risk assessment models, Finally, the current BTRA implementation must be run it is essential to strive for the highest level of transparency on a custom computer cluster in a DHS contractor facility possible while being sensitive to the need to restrict access taking many hours to compute; also the data are cumber- to those with a need to know. some to prepare. This makes answering “what-if” questions of the type that stakeholders are likely to ask an expensive THERE ARE SEVERAL OTHER WAYS and slow process. TO BUILD CONFIDENCE During the course of this study, in response to technical questions posed by three members of the committee, DHS The confidence of decision makers in the information provided the committee with a technical document (DHS, generated by a risk assessment model can be increased in 2007) that includes answers to the questions posed and which several ways, and increased confidence will heighten the became an essential piece of documentation missing from the likelihood that the information will be used. Decision makers original publications provided to the committee. In addition, can be engaged iteratively during model development; this the committee asked another expert to make an independent is critical in order to increase their understanding of how the review of the methodology employed for the model used in model is being constructed and to ensure that their informa- the BTRA of 2006 using the originally published material tion needs will be met. Complex systems can be simplified and the technical addendum. The independent review is to more-readily-understood scenarios and coupled with the reproduced in this report as Appendix I. The author of that capability of conducting real-time sensitivity analyses. In review encountered difficulties similar to those described addition, experts can conduct independent, periodic external here; one of three suggestions in Appendix I is “to report reviews; doing so is critical in order to assure stakeholders future results in a scientific fashion than can be reviewed by that the appropriate inputs and models are being used for risk scientists”—a suggestion that is echoed in the next recom- assessment. If the risk assessment model and/or its assump- mendation of this committee (see below). tions are not accurate or appropriate, the results of a model Some of the probabilities of important consequences in and accompanying sensitivity analyses can give a false sense the model used in the BTRA of 2006 were extremely close of security in the results, may lead to inappropriate policy decisions (Brisson and Edmunds, 2006), and ultimately will   Alan R. Washburn, Distinguished Professor Emeritus of Operations lead to a lack of confidence in the use of these models. Research, Naval Postgraduate School, Monterey, California. 

38 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT to zero. When risk is expressed in numerical form, whether THE BTRA SHOULD BECOME A or not decision makers are motivated to take action will DECISION SUPPORT SYSTEM frequently depend on how confident they are in the number. Decision support systems (DSSs) are interactive informa- Human beings are known to have difficulty in rationally tion technology platforms that facilitate the use of informa- processing numbers and probabilities (Paulos, 1988; Tversky tion in complex decision making. The goals of DSSs are to and Kahneman, 1973, 1974), and when probabilities are ex- improve the efficiency with which users make decisions and tremely small, decision makers will often give them greater to improve the effectiveness of their decisions (Shim et al., weight than is appropriate or possibly ignore them altogether, 2002; Pearson and Shim, 1995). DSSs are especially helpful at great peril when there are potentially significant and large in decision-making situations where there are multiple deci- consequences. Thus, extreme caution is needed to avoid an sion makers with different roles, functions, and responsibili- under- or overinterpretation of results that may cause errors ties, and different types and sources of data and databases. in decision making when probabilities of consequences are There are many different designs for DSSs, but in gen- estimated to be near zero, such as is the case with the model eral they include the following components: (1) database used in the BTRA of 2006. Systematic use of well-grounded management capabilities that provide access to relational models can guide decision makers to account for these prob- databases, information, and knowledge from a variety of abilities correctly. sources; (2) modeling and modeling management functions; and (3) a simple user-interface component that supports in- Recommendation: DHS should maintain a high level of teractive queries, reporting, and graphic functions (Marakas, transparency in risk assessment models, including a com- 1999; Druzdzel and Flynn, 2002; Shim et al., 2002; Ware prehensive, clear mathematical document and a complete et al., 2002). DSSs have been developed to support specific description of the sources of all input data. The documen- decisions involved with bioterrorism prevention, prepared- tation should be sufficient for scientific peer review. ness, and response (Bravata et al., 2002, 2004; Ware et al., 2002). Bravata et al. (2002) identified 217 information To carry out this recommendation, DHS should do the technology DSSs that were of potential use to clinicians and following: public health officials in the event of a bioterrorism event. One example of a DSS that facilitates data-based decision • Solicit input from multiple stakeholders involved with making for resource allocation problems and emergency the prevention of bioterrorism (whether directed at response planning is a stand-alone, large-scale DSS, called humans or at agriculture), preparedness, and response RealOpt. RealOpt pairs a flexible simulation component with throughout the development of the model in order to a set of analytical, decision-making algorithms. The system enhance their understanding of and therefore their con- comprises three integrating components: fidence in the model, its data inputs, and its results; • Clearly state the objectives and carefully define the • A simulation manager that runs simulations with input variables, sources of data, and associated conse- c ­ hanges in input parameters in order to investigate quence models; make assumptions explicit; and justify behavior and bottlenecks in the system for different sce- the values that are assigned to variables, parameters, narios, and calculates various outcome statistics (e.g., and probabilities; average wait time, queue length, and utilization rates); • Provide a guide to facilitate the interpretation of results, • An optimization manager that stores algorithms and especially in the context of important outcomes that are fast heuristics and iteratively calls on the simulation estimated to occur with probabilities approaching zero; manager to resolve and update resource-allocation and statistics (e.g., to maximize throughput, to minimize • Conduct a scientific, periodic external review of the staff usage to satisfy a specified throughput); and validity of the risk assessment model’s development • A user-interface manager and linker module that con- and analysis; carefully and completely document how nects the input of data to a display of results, including the model is developed and its mathematical founda- a graphics algorithm that allows users to design specific tions, using terms from a widely accepted, standard floorplans of different patient care and dispensing fa- technical lexicon in understandable language, such that cilities for vaccinations and different medications (Lee an independent, external panel of experts can duplicate et al., 2006). the results; have an independent blue team perform complete scenario dissection for selected paths through The characteristics of DSSs that are effective in giving de- the entire event tree; and take care to allow the widest cision makers access to the understandable information that possible review subject to security requirements. they need and can use for decision making are as follows: 

DEPARTMENT OF HOMELAND SECURITY DECISION REQUIREMENTS FOR RISK MANAGEMENT 39 • The DSS clearly states its objectives and desired out- with extremely small probabilities. Scenarios are especially comes (i.e., timely, quality decisions); useful when decision makers are inexperienced in systems • It addresses consequence and identifies key questions thinking. They may help inexperienced systems thinkers of stakeholders who were involved in framing the avoid using unrealistic assumptions, which can lead to the problems; development of incomplete, infeasible, or ineffective plans • It is user-friendly for a variety of stakeholders and does (Whitworth, 2006). not require sophisticated information technology skills Danzig (2003) described a number of situations in which for its operation; planning scenarios could be used. First, they can bring • The DSS is flexible, efficient, and includes an easy-to- awareness to sets of specific circumstances and hypothesized access help desk and documentation; chains of events, which, if understandable and conveyed • It is portable across different computer platforms and in a compelling manner such that the decision maker has personal digital assistants; confidence in the method, will have a greater likelihood for • It provides results that are well matched to decision resulting in action. Second, they can help in the development objectives. Decision makers can ask for and easily get of coordinated actions and plans among multiple stakehold- results of new simulations reflecting different assump- ers by keeping everyone focused on the same narrative or tions on how an event will present alternative responses alternative. Third, a planning scenario can serve as a refer- and interventions leading to different outcomes; ence case against which alternative strategies can be com- • The DSS requires minimal computation time for simu- pared and tested. Fourth, planning scenarios can be used to lation—seconds or minutes versus hours for individual establish resource and other requirements needed to prevent simulation runs; or respond to potential events. • It provides accurate results and information that can be Critics of scenarios are concerned that their use may make used to gauge how much confidence can be placed in assumptions unclear or inexplicit, complicating external model outputs. Systematic checks of data quality are review and assessment and making their validity difficult to built in to the analysis system and display of results; assess. When insufficient detail is provided, different stake- • It has displays that are simply designed with high- holders may arrive at different perceptions of a scenario and resolution data; it has relevant information presented end up coming to incompatible conclusions or developing and conveyed in an understandable way and accessed uncoordinated or incompatible plans. For this reason, scenar- easily; and its tables and graphs are well labeled. The ios must reflect the most complete, explicit, and transparent DSS’s displays should be accompanied by annotation, details available and allow for a ready comparison of per- details, other supplements, including limitations, aids ceptions among the various stakeholders. Finally, scenarios to interpretation of risk, confidence limits around risk, can become too rigid. They require continual updating as and confidence in solutions. new information becomes available. However, effectively planning for these possibilities can mitigate these organiza- Recommendation: Subsequent revision of the BTRA tion problems. The committee recognizes the difficulty in should enable a decision support system that can be run preparing and validating accurate and useful scenarios. For quickly to test the implications of new assumptions and that reason, it suggests that, as with other BTRA documenta- new data and provide insights to decision makers and tion, any such scenarios be peer reviewed. stakeholders to support risk-informed decision making. Sensitivity Analysis Is Important for Validation Use Scenarios Sensitivity analysis has been defined as the determination A successful DSS, as described above, would facilitate the of how “uncertainty in the output of a model (numerical or use of scenarios. Mathematical models (e.g., risk assessment otherwise) can be apportioned to different sources of uncer- models) often are so complex that their results are not easily tainty in the model input” (Saltelli and Tarantola, 2002). The understood, met with confidence, and used. Decision makers purposes of sensitivity analyses are to (1) give users of risk commonly deal with the uncertainty of future events by using assessment models information that they can use to identify “what-if” scenarios, which can bound uncertainty and bring key parameters and explore a range of impacts that can be multiple stakeholders together to consider a shared, selected expected with changes in input and parameter values, and set of hypothesized chains of events in narrative form, and to to evaluate the confidence they can place on model outputs; consider alternatives (Pomerol, 2001). Scenarios can make (2) identify sources of uncertainty in the model when as- abstract or nebulous threats more concrete, which can help sumptions and parameters vary across possible scenarios; decision makers avoid becoming lost in trying to assimilate (3) aid planners in comparing alternative strategies and test large numbers of variables, relationships, and parameters how a given plan would work should assumptions be wrong; 

40 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT (4) help decision makers make the best possible decisions strategies for tools that are easier for users to employ are in the presence of uncertainty; and (5) set priorities for the presented in the next subsection. collection of additional information (Meltzer, 2001; Whit- The purpose of the sensitivity analysis will affect the worth, 2006). method to use for the analysis. Borgonovo (2006) described Because considerable data are lacking for many of the three families of analytic techniques (i.e., variance-based, parameters and probabilities in the model used for the BTRA input-output correlation, and moment-independent analy- of 2006, there may be an accompanying lack of confidence in ses), the choice of which would depend on the stated purpose the results. Thus, being able to conduct a sensitivity analysis of the sensitivity analysis. is an essential feature of the BTRA model if it is to be used Precautions must be taken when drawing conclusions for decision making. Although the BTRA was apparently from sensitivity analyses, as the accuracy of the informa- developed so that the impact of different parameters on con- tion is conditional on the validity of the underlying model sequences of importance could be assessed through sensitiv- structure and the methods used to exercise it. If the structure ity analysis, the process for running sensitivity analyses cur- of the risk assessment model and/or the assumptions used in rently is not interactive and appears quite “user-unfriendly” the model are not accurate or appropriate, then the results of and cumbersome. To see results of the model for different a sensitivity analysis can give a false sense of security in the scenarios (changing values for parameters and in different results and may lead to inappropriate policy decisions (Bris- branches of the tree for different agents), changes in values son and Edmunds, 2006). Extreme caution also is needed to must be submitted to analysts who rerun the model. Results avoid a misinterpretation or overinterpretation of results and are then evidently available hours to days later, making the the making of errors in decision making when valid data for sensitivity analysis process difficult for decision makers parameters or probabilities are lacking. to use immediately. Finally, the results of simulations are presented in graphs, charts, and tables that are complex and Create a Context for Use difficult to interpret and use. For these reasons, the committee questions whether the In addition to the approaches discussed so far, strengthen- results of the 2006 Biological Threat Risk Assessment model ing the overall environment for data-based decision making are answering the highest-priority questions of different deci- is critical. A comprehensive and continually updated set of sion makers; whether they are being conveyed in the most guidelines, protocols, and checklists that provide essential understandable, useful, and compelling manner possible; and details on clear courses of actions that decision makers would whether the current sensitivity analysis feature is meeting make, conditional on the information made available to them information needs. User-friendly sensitivity analysis could from risk assessment models analyzing a set of structured also be a part of any DSS. scenarios, must be developed, tested, and in place. These The uncertainty or lack of data and/or errors in measure- materials should be prepared for different stakeholders and ment for many key variables and parameters in risk assess- should include a range of possible decisions and actions, by ment models for potential bioterrorism events can affect scenario, for different authorities, roles and responsibilities, the confidence that decision makers place on the output of desired outcomes, and benchmarks for tracking progress for the model. Accepted good modeling practices require that different scenarios. models be continually tested and validated by evaluating the Different strategies and protocols should be practiced as effect of uncertainties with regard to values of parameters tabletop and TOPOFF exercises to identify areas where ad- and probabilities of the model. Sensitivity analysis has be- ditional attention, planning, resource acquisition and alloca- come an accepted and important approach to the testing and tion, and practice are required. A trained workforce within validation of risk assessment models of complex systems and across multiple sectors, agencies, and institutions in the (Borgonovo, 2006). public and private sectors is essential. In the future, it will be important to move the sensitivity Relevant, accurate, timely information that is available in analysis from questions about risk assessment (for example, understandable formats and terms, with guides to the inter- How does uncertainty about the infectious dose for this agent pretation of outcomes, is critical. Environments that support change my expected consequences?) to questions about risk the use of data in decision making are those in which the right management (for example, If I had improved knowledge resources (e.g., staff, drugs, vaccines, respirators, other) are about the infectious dose for this agent, would I adopt dif- pre-positioned strategically and available to the right staff ferent countermeasure strategies?). Currently, simulation at the right time. runs take an extended period of time to run owing to the complexity and size of the model and its input data; outputs from the model are not presented in easily used, interactive, understandable and compelling formats. Strategies for reduc-   In Chapter 7, the committee discusses the benefits of red teaming in ing the complexity of the model are presented in Chapter 7; TOPOFF (Top Officials) exercises. 

DEPARTMENT OF HOMELAND SECURITY DECISION REQUIREMENTS FOR RISK MANAGEMENT 41 REFERENCES Lee, E.K., S. Maheshwary, J. Mason, and W. Glisson. 2006. “Large-Scale Dispensing for Emergency Response to Bioterrorism and Infectious- Aaby, K., J.W. Herrmann, C.S. Jordan, M. Treadwell, and K. Wood. 2006. Disease Outbreak.” Interfaces 36(6):591-607. “Montgomery County’s Public Health Service Uses Operations Re- Marakas, G.M. 1999. Decision Support Systems in the Twenty-First Century. search to Plan Emergency Mass Dispensing and Vaccination Clinics.” Upper Saddle River, N.J.: Prentice Hall. Interfaces 36(6):569-579. Meltzer, D. 2001. “Addressing Uncertainty in Medical Cost-effectiveness Boardman, A.E., D.H. Greenberg, A.R. Vining, and D.L.Weimer. 2006. Analysis. Implications of Expected Utility Maximization for Methods Cost-Benefit Analysis: Concepts and Practice, 3rd Edition. Upper to Perform Sensitivity Analyses and Use of Cost-effectiveness Analysis Saddle River, N.J.: Pearson Prentice Hall. to Set Priorities for Medical Research.” Journal of Health Economics Borgonovo, E. 2006. “Measuring Uncertainty Importance: Investigation 20(1):109-129. and Comparison of Alternative Approaches.” Risk Analysis 20(5):1349- Oliver, R.W. 2004. What Is Transparency? New York: McGraw-Hill. 1361. Omenn, G.S. 2003. “On the Significance of ‘the Red Book’ in the Evolution Bravata, D.M., K. McDonald, and D.K. Owens. 2002. Bioterrorism Pre- of Risk Assessment and Risk Management.” Human and Ecological paredness and Response: Use of Information Technologies and Decision Risk Assessment 9(5):1155-1167. Support Systems. University of California, San Francisco—Stanford Paulos, J.A. 1988. Innumeracy: Mathematical Illiteracy and Its Conse- Evidence-based Practice Center. AHRQ Publication No. 02-E028. quences. New York: Hill and Wang. Available at www.ahrq.gov/clinic/tp/bioittp.htm. Accessed January Pearson, J.M., and J.P. Shim. 1995. “An Empirical Investigation into DSS 30, 2007. Structures and Environments.” Decision Support Systems 13(2):141- Bravata, D.M., V. Sundaram, K.M. McDonald, W.M. Smith, H. Szeto, M.D. 158. Schleinitz, and D.K. Owens. 2004. “Evaluating Detection and Diagnos- Pomerol, J.C. 2001. “Scenario Development and Practical Decision Making tic Decision Support Systems for Bioterrorism Response.” Emerging Under Uncertainty.” Decision Support Systems 31(2):197-204. Infectious Diseases 10(1):100-108. Presidential/Congressional Commission on Risk Assessment and Risk Brisson, M., and W.J. Edmunds. 2006. “Impact of Model, Methodologi- Manage­ment. 1997a. Framework for Environmental Health Risk cal, Parameter Uncertainty in the Economic Analysis of Vaccination ­ anagement. Final Report Vol. 1. Washington D.C.: U.S. Government M Programs.” Medical Decision Making 26(5):434-446. Printing Office. Available at www.riskworld.com/Nreports/1997/ Cooper, B. 2006. “Poxy Models and Rash Decisions.” Proceedings of risk-rpt/pdf/EPAJAN.pdf. Accessed January 30, 2008. the National Academy of Sciences of the United States of America Presidential/Congressional Commission on Risk Assessment and Risk Man- 103(33):12221-12222. agement. 1997b. Risk Assessment and Risk Management in Regulatory Danzig, R. 2003. Catastrophic Bioterrorism—What Is to Be Done. Center Decision Making. Final Report Vol. 2. Washington D.C.: U.S. Govern- for Technology and National Security Policy. Washington D.C.: National ment Printing Office. Available at www.riskworld.com/Nreports/1997/ Defense University. risk-rpt/volume2/pdf/v2epa.pdf. Accessed January 30, 2008. DHS (Department of Homeland Security). 2006. Bioterrorism Risk Assess- Saltelli, A., and S. Tarantola. 2002. “On the Relative Importance of Input ment. Biological Threat Characterization Center of the National Biode- Factors in Mathematical Models: Safety Assessment for Nuclear Waste fense Analysis and Countermeasures Center. Fort Detrick, Md. Disposal.” Journal of the American Statistical Association 97(459):702- DHS. 2007. “A Lexicon of Risk Terminology and Methodological De- 709. scription of the DHS Bioterrorism Risk Assessment.” Written com- Shim, J.P., M. Warkentin, J.F. Courtney, D.J. Power, R. Sharda, and C. munication to the Committee on Methodological Improvements to the Carlson. 2002. “Past, Present, and Future of Decision Support Technol- Department of Homeland Security’s Biological Agent Risk Analysis. ogy.” Decision Support Systems 33(2): 111-126. April 14, 2007. Tversky, A., and D. Kahneman. 1973. “Availability: A Heuristic for Judging Druzdzel, M., and R.R. Flynn. 2002. “Decision Support Systems.” Ency- Frequency and Probability.” Cognitive Psychology 5(2):207-232. clopedia of Library and Information Science, 2nd Edition. Allen Kent Tversky, A., and D. Kahneman. 1974. “Judgment Under Uncertainty: Heu- (ed.), New York: Marcel Dekker. Available at www.pitt.edu/~druzdzel/ ristics and Biases.” Science 185(4157):1124-1131. psfiles/dss.pdf. Accessed January 30, 2008. Ware, B.S., A. Beverina, L. Gong, and B. Colder. 2002. A Risk-Based Elderd, B.D., V.M. Dukic, and G. Dwyer. 2006. “Uncertainty in Predictions Decision Support System for Antiterrorism. Available at www.dsbox. of Disease Spread and Public Health Responses to Bioterrorism and com/Documents/MSS_A_Risk-Based_Decision_Support_System_for_ Emerging Diseases.” Proceedings of the National Academy of Sciences Antiterrorism.pdf. Accessed January 30, 2007. of the United States of America 103(42):15693-15697. The White House. 2004. Homeland Security Presidential Directive 10 Fischhoff, B., W.B. Bruin, U. Güvenç, D. Caruso, and L. Brilliant. 2006. [HSPD-10]: Biodefense for the 21st Century. Available at www.fas. “Analyzing Disaster Risks and Plans: An Avian Flu Example.” Journal org/irp/offdocs/nspd/hspd-10.html. Accessed January 16, 2008. of Risk and Uncertainty 33(1):131-149. Whitworth, M.H. 2006. “Designing the Response to an Anthrax Attack.” GAO (General Accounting Office). 2002. National Preparedness: Integrat- Interfaces 36(6):562-568. ing New and Existing Technology and Information Sharing into an ­ ffective Homeland Security Strategy. GAO-02-811T. Available at www. E gao.gov/new.items/d02811t.pdf. Accessed January 30, 2008.