Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Summary Armed with a single vial of a biological agent small groups of fanatics, or failing states, could gain the power to threaten great nations, threaten the world peace. America, and the entire civilized world, will face this threat for decades to come. We must confront the danger with open eyes and unbending purpose. —President George W. Bush, February 11, 2004 DEPARTMENT OF HOMELAND SECURITY’S some of the basic assumptions underlying HSPD-10 BIOLOGICAL THREAT RISK ASSESSMENT to chemical, biological, radiological, and nuclear (CBRN) threats, calling for an integrated CBRN risk The Committee on Methodological Improvements to the assessment. Department of Homeland Security’s Biological Agent Risk Analysis was established by the National Research Council DHS produced its report Bioterrorism Risk Assessment in and convened in August 2006 to review the Department of 2006 (DHS, 2006). The BTRA of 2006 and the DHS (2006) Homeland Security’s (DHS’s) Biological Threat Risk As- report, which documents the analysis, respond directly to the sessment (BTRA) of 2006. The BTRA is a computer-based requirements of HSPD-10 and of the National Strategy for tool that has been applied by DHS to assess the risk associ- Homeland Security (Office of Homeland Security, 2002) for ated with the intentional release of each of 28 biological DHS to assess the biological weapons threat. threat agents categorized by the Centers for Disease Control This committee has been called to provide an indepen- and Prevention. dent, scientific peer review of the methodology that led to The threat posed by biological agents employed in a the BTRA of 2006 and that will be the foundation for future terrorist attack on the United States is arguably the most biennial updates. At this writing, DHS is preparing a revision important homeland security challenge of our era. Whether of its bioterrorism risk analysis responding to HSPD-18; this natural pathogens are cultured or new variants are bioen- analysis will presumably appear, as directed, in 2008. The gineered, the consequence of a terrorist-induced pandemic committee did not have the draft of the DHS report docu- could be millions of casualties—far more than we would menting the analysis of the BTRA of 2008, but it was briefed expect from nuclear terrorism, chemical attacks, or conven- on some of the enhancements and changed procedures that tional attacks on the infrastructure of the United States such will influence the BTRA of 2008 and considered all informa- as the attacks of September 11, 2001. Even if there were tion provided in the course of its review. fewer casualties, additional second-order consequences (in- The committee has identified a number of fundamental cluding psychological, social, and economic effects) would concerns with the BTRA of 2006, ranging from mathemati- dramatically compound the effects. Bioengineering is no cal and statistical mistakes that have corrupted results, to un- longer the exclusive purview of state sponsors of terrorism; necessarily complicated probability models and models with this technology is now available to small terrorist groups and fidelity far exceeding existing data, to more basic questions even to deranged individuals. about how terrorist behavior should be modeled. All of these The executive branch recognizes this grave threat, as issues are covered in the body of this report. witnessed by the following: Rather than merely criticizing what was done in the BTRA of 2006, the committee sought outside experts and • Homeland Security Presidential Directive 10 (HSPD- collected a number of proposed alternatives that it believes 10): Biodefense for the 21st Century (The White House, would improve DHS’s ability to assess potential terrorist 2004) calls for DHS to conduct biennial assessments of behavior as a key element of risk-informed decision making, biological threats, and and it explains these alternatives in the specific context of • Homeland Security Presidential Directive 18 (HSPD- the BTRA and the bioterrorism threat. 18): Medical Countermeasures Against Weapons of The committee set for itself the following gauge of suc- Mass Destruction (The White House, 2007) applies cess for its various deliberations and its final report: If DHS  

 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT follows the committee’s recommendations (drawn from the DHS intended that the BTRA of 2006 be an “end-to-end individual chapters of this report and presented as a com- risk assessment of the bioterrorism threat” with potential plete set in the next section), the resulting product will more catastrophic consequences to human health and the national reliably assess the possible acts of terrorists, will be better economy and that it “assist and guide biodefense strategic documented and understood by its clients, and will be more planning” (DHS, 2006, Ch. 1, p. 1) in response to the HSPD- responsive and able not only to assess risk, but to effectively 10 directive to “conduct biennial assessments of biological inform strategic investments in risk management. threats.” Guided by DHS’s customers for information from HSPD-10 states: the assessment, the BTRA of 2006 was designed to produce assessments in the form of risk-prioritized groups of biologi- Another critical element of our biodefense policy is the de- velopment of periodic assessments of the evolving biological cal threat agents. These prioritized lists could then be used weapons threat. First, the United States requires a continu- to identify gaps or vulnerabilities in the U.S. biodefense ous, formal process for conducting routine capabilities as- posture and make recommendations for rebalancing and sessments to guide prioritization of our on-going investments refining investments in the overall U.S. biodefense policy. in biodefense-related research, development, planning, and DHS has assembled a confederation of researchers and preparedness (The White House, 2004). subject-matter experts and is collaborating with national laboratories that can contribute to expanding the knowledge In accord with HSPD-10, the fundamental concerns of the base of bioterrorism. committee are not only modeling or mathematical details, but the provision to homeland security policy makers of bet- ter tools to use when deciding how to invest huge sums of RECOMMENDATIONS money to protect this nation against a grave threat. Overall Assessment THE CHARGE TO THE COMMITTEE The committee met on August 28-29, 2006, with repre- sentatives of DHS in response to a DHS request for guidance The charge to the committee for this final report is as on its near-term BTRA development efforts. In November follows: 2006, in response to that request and based on the informa- tion it had received at the 2-day meeting with DHS, the • Recommend how the methodology can incorporate committee electronically issued its Interim Report (repro- changing probability distributions that reflect how duced as Appendix J in this final report). Subsequently the various actors (e.g., terrorists, first responders, public committee received the full DHS (2006) report documenting health community) adjust their choices over time or in the analysis in the BTRA of 2006. While DHS agreed with different contexts; the recommendations of the Interim Report and planned to • Recommend further improvements to the consequence address them, the committee did not learn of any progress analysis component of the methodology, including its up to the conclusion of its deliberations in May 2007 that models of economic effects; would obviate those recommendations, which require sus- • Identify any emerging methods for handling large de- tained work. grees of uncertainty (e.g., fuzzy logic, possibility analy- However, the content of the DHS (2006) report and sis) that merit consideration for future incorporation; information gained at additional meetings with DHS and • Recommend further improvements to the transparency national experts have significantly changed the committee’s and usability of the methodology; overall assessment of the BTRA of 2006. The committee • Discuss in more detail beyond the first report [the identified errors in mathematics, risk assessment modeling, committee’s Interim Report] how the methodology computing, presentation, and other weaknesses in the BTRA could be extended to risks associated with classes of of 2006. It recommends against using this current BTRA agents, including enhanced or engineered agents that for bioterrorism risk assessment as presented in the BTRA have yet to be developed; and of 2006 or proposed for 2008. Instead, the committee offers • Discuss in more detail beyond the first report the feasi- improvements that can significantly simplify and improve bility of extending the methodology to also serve as a future risk assessments. The improved BTRA should be used framework for risk analysis of chemical or radioactive for risk management as well as risk assessment, as intended threats. by HSPD-10. The committee discusses the elements of risk analyses, In order to attend to this charge, this committee reviewed including risk management, and identifies the crucial differ- all of the detail in the BTRA of 2006, interviewed its imple- ences between the use of risk analysis to assess and manage menters, and called on outside experts. It also received brief- the risks of natural disasters and its use to assess and manage ings from DHS on planned improvements to the BTRA of risks from terrorist attacks. Representing terrorist decision 2008. During this process, the committee recorded deficien- making exclusively as random variables, as is appropriate cies and recommended improvements in the assessment. 

SUMMARY  in the case of natural disasters, is a fundamental problem experts, but these assessments must be conditioned on all of with the BTRA. these inputs. This is a daunting task for any subject-matter expert. Appendix G of this report contains material on alter- nate methods that can be used to quantify uncertainty. This Risk Analysis Lexicon report explains in detail that probability theory is suited to the The DHS (2006) report and DHS presentations of its con- task and that no alternative is needed. However, this report tent use inconsistent, imprecise technical language and do discusses at length weaknesses in DHS’s use of probability not define many key terms. Clear and consistent risk analysis in theory, conception, and computation in the BTRA. definitions are essential for precise technical work and clear Instead of directly assessing conditional probabilities for communication with diverse stakeholders. The committee outcomes, DHS subject-matter experts are asked to assess prepared a risk analysis lexicon for its own use (included as conditional probability distributions over the probabilities Appendix A in this final report) with definitions and their of outcomes. This complication is shown to be unnecessary; sources. It is intended to be an example of a lexicon to be the analysis would be unchanged if only the expected value used in future DHS reports and presentations. of these distributions was used. This simplification would significantly reduce data re- Recommendation: The Department of Homeland Se- quirements and accelerate computation. The BTRA software curity should use an explicit risk analysis lexicon for implementation seems to the committee to be cumbersome defining each technical term appearing in its reports and and slow and requires tending by its creators to produce risk presentations. assessments. The committee advises simplification so that the BTRA can be used for responsive risk assessment and risk management. Approach to Determining the Probabilities of Terrorist Decisions Recommendation: The event-tree probability elicitation DHS has made an important contribution by structuring should be simplified by assessing probabilities instead a nominal bioterrorist attack and identifying the bioagents of probability distributions for the outcomes of each that should be assessed. The committee closely examined event. the assumptions and the mathematical details of the BTRA of 2006 and found that there are weaknesses in the model’s Regarding Normalization of Risk Assessment Results conception, errors in some of the underlying mathematics and statistics, and unnecessary complexity. DHS has chosen to represent “normalized” relative risk, The BTRA represents adversarial decisions by means without specifying the normalization constant. This decision of probabilities assessed by subject-matter experts. How- has obscured the results of the analysis and made it impos- ever, when dealing with an intelligent, goal-oriented, and sible to understand the results, to reproduce any particular resourceful adversary (the terrorist), the exclusive use of BTRA result, or to use independent means to assess the subjectively assessed probabilities for terrorist decisions is veracity of any result. Moreover, normalization provides inappropriate. For decision problems as complex as those insufficient information for risk assessment and risk manage- dealt with in the BTRA, the probability that an adversary will ment. Homeland security decision makers and stakeholders choose a course of action should be an output of analysis, need to see the calculated probabilities and consequences not an input. Accordingly: to make risk-informed decisions. This is not to say that the committee believes that precise absolute levels of probabili- Recommendation: To assess the probabilities of terror- ties and consequences can be predicted or are needed. But ist decisions, DHS should use elicitation techniques and risk managers and decision makers need some sense of the decision-oriented models that explicitly recognize terror- magnitude of the probabilities and consequences, and that is ists as intelligent adversaries who observe U.S. defensive not available after normalization. preparations and seek to maximize the achievement of their own objectives. Recommendation: Normalization of BTRA risk assess- ment results obscures information that is essential for risk-informed decision making. BTRA results should not Simplifying the Assessment of Outcome Probabilities be normalized. Decisions, by both terrorist attacker and U.S. defender, should be outputs of a decision support model. The de- Simplification of the BTRA Event Tree termination of data sources and their reliability is outside the scope of this report. However, data concerning threats, The committee finds Stage 1, Frequency of Initiation [of resource levels, technological facts, and so forth are inputs. an attack] by Terrorist Group, of the BTRA fixed-hierarchy Adversarial decisions can be assessed by subject-matter event-tree sequence to be a distracting embellishment. Also, 

 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT the representation of potential multiple (sequential) terrorist description of the sources of all input data. The documen- attacks in the BTRA of 2006 is incorrect, both technically tation should be sufficient for scientific peer review. and philosophically, and adds an unnecessary layer of com- plexity to the analysis. The computation of the expected Recommendation: Subsequent revision of the BTRA number of attacks is shown to be mathematically incorrect, should enable a decision support system that can be run and the (random) distribution of consequences of such re- quickly to test the implications of new assumptions and peated attacks is shown to be represented incorrectly. How- new data and provide insights to decision makers and ever, even if the mathematics were correct, the committee stakeholders to support risk-informed decision making. believes that, after the first terrorist attack, all assumptions and parameter values in the BTRA would change, so that Rapid Assessment Strategy for New Information the previous risk analysis would no longer apply. Eliminat- ing the BTRA multiple-attack feature would significantly The committee has highlighted the dynamic nature of simplify the model. the biological threat and was asked to show how the BTRA The committee also finds that some of the stages in the might be applied to enhanced or engineered biological BTRA characterization of the steps leading to a terrorist at- agents. The committee suggests a rapid assessment tool and tack might be aggregated to the minimum number of stages proposes a template that suggests how to quickly estimate necessary to calculate probabilities and consequences, mak- the threat from emerging or suspected agents to determine ing data acquisition simpler without sacrificing fidelity. whether a more detailed exigent study is necessary. It agrees that this is an important goal and makes the following Recommendation: Two significant simplifications should recommendation. be made to the BTRA of 2006 event tree: • DHS should eliminate Stage 1, Frequency of Initia- Recommendation: The BTRA should be broad enough to tion [of an attack] by Terrorist Group, and Stage 16, encompass a variety of bioterrorism threats while allow- Potential for Multiple Attacks; and ing for changing situations and new information. DHS • DHS should seek opportunities to aggregate some should develop a strategy for the rapid assessment of stages of the tree to only those essential to calcu- newly recognized and poorly characterized threats. late probabilities and consequences with realistic fidelity. Existing Knowledge and the Detail in Consequence Models Need for Transparent, User-Friendly The committee examined the consequence analysis of Decision Support System the BTRA. It finds that the susceptible, exposed, infected, Risk assessment, such as the BTRA, has no direct impact and recovered (SEIR) model used to analyze the health on risk reduction. Only effective risk management strate- consequences of a bioterrorist attack requires, with regard gies can reduce risk, and there are several barriers to the to pathogens, data that do not exist. There is scant empirical effective use of information from the BTRA in decision basis for pathogens that have only recently been discovered making. These include numerous stakeholders with different in nature and with which there is little experience. Extremely responsibilities, authority, and indicators of success; dispa- limited clinical and epidemiologic data exist about many of rate data and data sources; and organizational friction and the pathogens in the BTRA of 2006. The granularity of detail compartmentalization within and among stakeholders. To in the SEIR models is not supported by existing data on any support risk-informed decision making and mitigate some pathogen on the BTRA list. of these problems, DHS needs transparent and user-friendly decision support models. Accordingly, the committee makes Recommendation: The susceptible, exposed, infected, the following three recommendations. and recovered (SEIR) model adopted by DHS is more complex than can be supported by existing data or Recommendation: Subsequent revision of the BTRA knowledge. DHS should make its SEIR model as simple should increase emphasis on risk management. An in- as possible consistent with existing knowledge. creased focus on risk management will allow the BTRA to better support the risk-informed decisions that homeland Consequences Besides Mortality and security stakeholders are required to make. Morbidity That Need to Be Modeled Recommendation: DHS should maintain a high level of DHS is planning to include second-order economic transparency in risk assessment models, including a com- effects in the BTRA of 2008. The committee highlights prehensive, clear mathematical document and a complete those effects, including important agricultural effects, and 

SUMMARY  discusses the use of cost-benefit analysis to provide a com- be compared on a common consequence scale to suggest mon measure. and evaluate risk management strategies that encompass all terrorist threats. Recommendation: While human mortality and the magnitude and duration of morbidity should remain the Regarding the Use of the BTRA in Its Present Form primary focus of DHS bioterrorism risk analysis, DHS should incorporate other measures of societal loss, in- For the reasons noted in this report’s recommendations cluding the magnitude and duration of first- and second- and their justifying text, the committee believes that the order economic loss and environmental and agricultural BTRA in its present form should not be used to assess the effects. risk of bioterrorism threats. For the same reasons, the com- mittee does not recommend trying to extend the BTRA to the qualitatively different chemical and radioactive threats. Methods for Improved Modeling of Intelligent Adversaries The committee attaches great importance to the realistic Recommendation: The BTRA should not be used as a ba- representation of the behavior of an intelligent adversary. sis for decision making until the deficiencies noted in this BTRA probabilities are conditioned on past events and are report have been addressed and corrected. DHS should retrospective, whereas the terrorist is prospective, constantly engage an independent, senior technical advisory panel adjusting tactics to exploit any evident weakness in U.S. to oversee this task. In its current form, the BTRA should defenses. not be used to assess the risk of biological, chemical, or To offer some concrete examples of how to credibly rep- radioactive threats. resent the behavior of an intelligent adversary, the committee presents three ways to represent adversarial decisions: (1) a The committee takes very seriously the bioterrorism “bioterrorism decision model” using off-the-shelf software; threats and potential consequences that it has had to consider (2) a tri-level decision support model to allocate defensive in this study. It is fully aware of the potential impact of its investments (visible to the attacker) that represents an attack- recommendations on the BTRA of 2008 and the stakehold- er’s reasonable response to observing these preparations, and ers who await it. However, it believes that the failure to reactions to any attack with the resources made available by properly model intelligent adversaries and a continuation the defensive investments; and (3) a game-theoretic model on the path of unnecessary complexity in computer model- of the adversaries that randomizes expected consequences to ing and simulations will not help the United States defend capture the variability of outcomes. These are not mere theo- against the bioterrorist threats in the 21st century and will retical tools, but rather substantive suggestions drawn from not meet the intent of HSPD-10. Therefore, the committee extensive research and experience in the military and in the unanimously believes that an improved BTRA is needed to private sector. These suggestions can significantly improve provide a much more credible foundation for risk-informed the credibility and usefulness of the BTRA. decision making. Recommendation: In addition to using event trees, DHS REFERENCES should explore alternative models of terrorists as intel- ligent adversaries who seek to maximize the achievement DHS (Department of Homeland Security). 2006. Bioterrorism Risk Assess- ment. Biological Threat Characterization Center of the National Biode- of their objectives. fense Analysis and Countermeasures Center. Fort Detrick, Md. Office of Homeland Security. 2002. National Strategy for Homeland Security. Available at www.dhs.gov/xlibrary/assets/nat_strat_hls.pdf. Use of Intelligent-Adversary Risk Analysis Accessed November 1, 2006. Techniques for Other Threat Areas The White House. 2004. Homeland Security Presidential Directive 10 [HSPD-10]: Biodefense for the 21st Century. Available at www.fas. The committee believes that each of its suggested exten- org/irp/offdocs/nspd/hspd-10.html. Accessed January 16, 2008. sions to realistically represent adversarial behavior is ap- The White House. 2007. Homeland Security Presidential Directive 18 plicable to biological, chemical, and/or radioactive threats. [HSPD-18]: Medical Countermeasures Against Weapons of Mass Although distinct models may need to be developed for the Destruction. Available at www.fas.org/irp/offdocs/nspd/hspd-18.html. analysis of each of these threats, the resulting analyses can Accessed January 16, 2008.