Cover Image

PAPERBACK
$39.75



View/Hide Left Panel

Summary

Armed with a single vial of a biological agent small groups of fanatics, or failing states, could gain the power to threaten great nations, threaten the world peace. America, and the entire civilized world, will face this threat for decades to come. We must confront the danger with open eyes and unbending purpose.

—President George W. Bush, February 11, 2004

DEPARTMENT OF HOMELAND SECURITY’S BIOLOGICAL THREAT RISK ASSESSMENT

The Committee on Methodological Improvements to the Department of Homeland Security’s Biological Agent Risk Analysis was established by the National Research Council and convened in August 2006 to review the Department of Homeland Security’s (DHS’s) Biological Threat Risk Assessment (BTRA) of 2006. The BTRA is a computer-based tool that has been applied by DHS to assess the risk associated with the intentional release of each of 28 biological threat agents categorized by the Centers for Disease Control and Prevention.

The threat posed by biological agents employed in a terrorist attack on the United States is arguably the most important homeland security challenge of our era. Whether natural pathogens are cultured or new variants are bioengineered, the consequence of a terrorist-induced pandemic could be millions of casualties—far more than we would expect from nuclear terrorism, chemical attacks, or conventional attacks on the infrastructure of the United States such as the attacks of September 11, 2001. Even if there were fewer casualties, additional second-order consequences (including psychological, social, and economic effects) would dramatically compound the effects. Bioengineering is no longer the exclusive purview of state sponsors of terrorism; this technology is now available to small terrorist groups and even to deranged individuals.

The executive branch recognizes this grave threat, as witnessed by the following:

  • Homeland Security Presidential Directive 10 (HSPD-10): Biodefense for the 21st Century (The White House, 2004) calls for DHS to conduct biennial assessments of biological threats, and

  • Homeland Security Presidential Directive 18 (HSPD-18): Medical Countermeasures Against Weapons of Mass Destruction (The White House, 2007) applies some of the basic assumptions underlying HSPD-10 to chemical, biological, radiological, and nuclear (CBRN) threats, calling for an integrated CBRN risk assessment.

DHS produced its report Bioterrorism Risk Assessment in 2006 (DHS, 2006). The BTRA of 2006 and the DHS (2006) report, which documents the analysis, respond directly to the requirements of HSPD-10 and of the National Strategy for Homeland Security (Office of Homeland Security, 2002) for DHS to assess the biological weapons threat.

This committee has been called to provide an independent, scientific peer review of the methodology that led to the BTRA of 2006 and that will be the foundation for future biennial updates. At this writing, DHS is preparing a revision of its bioterrorism risk analysis responding to HSPD-18; this analysis will presumably appear, as directed, in 2008. The committee did not have the draft of the DHS report documenting the analysis of the BTRA of 2008, but it was briefed on some of the enhancements and changed procedures that will influence the BTRA of 2008 and considered all information provided in the course of its review.

The committee has identified a number of fundamental concerns with the BTRA of 2006, ranging from mathematical and statistical mistakes that have corrupted results, to unnecessarily complicated probability models and models with fidelity far exceeding existing data, to more basic questions about how terrorist behavior should be modeled. All of these issues are covered in the body of this report.

Rather than merely criticizing what was done in the BTRA of 2006, the committee sought outside experts and collected a number of proposed alternatives that it believes would improve DHS’s ability to assess potential terrorist behavior as a key element of risk-informed decision making, and it explains these alternatives in the specific context of the BTRA and the bioterrorism threat.

The committee set for itself the following gauge of success for its various deliberations and its final report: If DHS



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Summary Armed with a single vial of a biological agent small groups of fanatics, or failing states, could gain the power to threaten great nations, threaten the world peace. America, and the entire civilized world, will face this threat for decades to come. We must confront the danger with open eyes and unbending purpose. —President George W. Bush, February 11, 2004 DEPARTMENT OF HOMELAND SECURITY’S some of the basic assumptions underlying HSPD-10 BIOLOGICAL THREAT RISK ASSESSMENT to chemical, biological, radiological, and nuclear (CBRN) threats, calling for an integrated CBRN risk The Committee on Methodological Improvements to the assessment. Department of Homeland Security’s Biological Agent Risk Analysis was established by the National Research Council DHS produced its report Bioterrorism Risk Assessment in and convened in August 2006 to review the Department of 2006 (DHS, 2006). The BTRA of 2006 and the DHS (2006) Homeland Security’s (DHS’s) Biological Threat Risk As- report, which documents the analysis, respond directly to the sessment (BTRA) of 2006. The BTRA is a computer-based requirements of HSPD-10 and of the National Strategy for tool that has been applied by DHS to assess the risk associ- Homeland Security (Office of Homeland Security, 2002) for ated with the intentional release of each of 28 biological DHS to assess the biological weapons threat. threat agents categorized by the Centers for Disease Control This committee has been called to provide an indepen- and Prevention. dent, scientific peer review of the methodology that led to The threat posed by biological agents employed in a the BTRA of 2006 and that will be the foundation for future terrorist attack on the United States is arguably the most biennial updates. At this writing, DHS is preparing a revision important homeland security challenge of our era. Whether of its bioterrorism risk analysis responding to HSPD-18; this natural pathogens are cultured or new variants are bioen- analysis will presumably appear, as directed, in 2008. The gineered, the consequence of a terrorist-induced pandemic committee did not have the draft of the DHS report docu- could be millions of casualties—far more than we would menting the analysis of the BTRA of 2008, but it was briefed expect from nuclear terrorism, chemical attacks, or conven- on some of the enhancements and changed procedures that tional attacks on the infrastructure of the United States such will influence the BTRA of 2008 and considered all informa- as the attacks of September 11, 2001. Even if there were tion provided in the course of its review. fewer casualties, additional second-order consequences (in- The committee has identified a number of fundamental cluding psychological, social, and economic effects) would concerns with the BTRA of 2006, ranging from mathemati- dramatically compound the effects. Bioengineering is no cal and statistical mistakes that have corrupted results, to un- longer the exclusive purview of state sponsors of terrorism; necessarily complicated probability models and models with this technology is now available to small terrorist groups and fidelity far exceeding existing data, to more basic questions even to deranged individuals. about how terrorist behavior should be modeled. All of these The executive branch recognizes this grave threat, as issues are covered in the body of this report. witnessed by the following: Rather than merely criticizing what was done in the BTRA of 2006, the committee sought outside experts and • Homeland Security Presidential Directive 10 (HSPD- collected a number of proposed alternatives that it believes 10): Biodefense for the 2st Century (The White House, would improve DHS’s ability to assess potential terrorist 2004) calls for DHS to conduct biennial assessments of behavior as a key element of risk-informed decision making, biological threats, and and it explains these alternatives in the specific context of • Homeland Security Presidential Directive 18 (HSPD- the BTRA and the bioterrorism threat. 18): Medical Countermeasures Against Weapons of The committee set for itself the following gauge of suc- Mass Destruction (The White House, 2007) applies cess for its various deliberations and its final report: If DHS 

OCR for page 1
2 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT follows the committee’s recommendations (drawn from the DHS intended that the BTRA of 2006 be an “end-to-end individual chapters of this report and presented as a com- risk assessment of the bioterrorism threat” with potential plete set in the next section), the resulting product will more catastrophic consequences to human health and the national reliably assess the possible acts of terrorists, will be better economy and that it “assist and guide biodefense strategic documented and understood by its clients, and will be more planning” (DHS, 2006, Ch. 1, p. 1) in response to the HSPD- responsive and able not only to assess risk, but to effectively 10 directive to “conduct biennial assessments of biological inform strategic investments in risk management. threats.” Guided by DHS’s customers for information from HSPD-10 states: the assessment, the BTRA of 2006 was designed to produce assessments in the form of risk-prioritized groups of biologi- Another critical element of our biodefense policy is the de- cal threat agents. These prioritized lists could then be used velopment of periodic assessments of the evolving biological to identify gaps or vulnerabilities in the U.S. biodefense weapons threat. First, the United States requires a continu- posture and make recommendations for rebalancing and ous, formal process for conducting routine capabilities as- refining investments in the overall U.S. biodefense policy. sessments to guide prioritization of our on-going investments DHS has assembled a confederation of researchers and in biodefense-related research, development, planning, and preparedness (The White House, 2004). subject-matter experts and is collaborating with national laboratories that can contribute to expanding the knowledge In accord with HSPD-0, the fundamental concerns of the base of bioterrorism. committee are not only modeling or mathematical details, but the proision to homeland security policy makers of bet- RECOMMENDATIONS ter tools to use when deciding how to inest huge sums of money to protect this nation against a grae threat. Overall Assessment THE CHARGE TO THE COMMITTEE The committee met on August 28-29, 2006, with repre- sentatives of DHS in response to a DHS request for guidance The charge to the committee for this final report is as on its near-term BTRA development efforts. In November follows: 2006, in response to that request and based on the informa- tion it had received at the 2-day meeting with DHS, the • Recommend how the methodology can incorporate committee electronically issued its Interim Report (repro- changing probability distributions that reflect how duced as Appendix J in this final report). Subsequently the various actors (e.g., terrorists, first responders, public committee received the full DHS (2006) report documenting health community) adjust their choices over time or in the analysis in the BTRA of 2006. While DHS agreed with different contexts; the recommendations of the Interim Report and planned to • Recommend further improvements to the consequence address them, the committee did not learn of any progress analysis component of the methodology, including its up to the conclusion of its deliberations in May 2007 that models of economic effects; would obviate those recommendations, which require sus- • Identify any emerging methods for handling large de- tained work. grees of uncertainty (e.g., fuzzy logic, possibility analy- However, the content of the DHS (2006) report and sis) that merit consideration for future incorporation; information gained at additional meetings with DHS and • Recommend further improvements to the transparency national experts have significantly changed the committee’s and usability of the methodology; overall assessment of the BTRA of 2006. The committee • Discuss in more detail beyond the first report [the identified errors in mathematics, risk assessment modeling, committee’s Interim Report] how the methodology computing, presentation, and other weaknesses in the BTRA could be extended to risks associated with classes of of 2006. It recommends against using this current BTRA agents, including enhanced or engineered agents that for bioterrorism risk assessment as presented in the BTRA have yet to be developed; and of 2006 or proposed for 2008. Instead, the committee offers • Discuss in more detail beyond the first report the feasi- improvements that can significantly simplify and improve bility of extending the methodology to also serve as a future risk assessments. The improved BTRA should be used framework for risk analysis of chemical or radioactive for risk management as well as risk assessment, as intended threats. by HSPD-10. The committee discusses the elements of risk analyses, In order to attend to this charge, this committee reviewed including risk management, and identifies the crucial differ- all of the detail in the BTRA of 2006, interviewed its imple- ences between the use of risk analysis to assess and manage menters, and called on outside experts. It also received brief- the risks of natural disasters and its use to assess and manage ings from DHS on planned improvements to the BTRA of risks from terrorist attacks. Representing terrorist decision 2008. During this process, the committee recorded deficien- making exclusively as random variables, as is appropriate cies and recommended improvements in the assessment.

OCR for page 1
 SUMMARY in the case of natural disasters, is a fundamental problem experts, but these assessments must be conditioned on all of with the BTRA. these inputs. This is a daunting task for any subject-matter expert. Appendix G of this report contains material on alter- nate methods that can be used to quantify uncertainty. This Risk Analysis Lexicon report explains in detail that probability theory is suited to the The DHS (2006) report and DHS presentations of its con- task and that no alternative is needed. However, this report tent use inconsistent, imprecise technical language and do discusses at length weaknesses in DHS’s use of probability not define many key terms. Clear and consistent risk analysis in theory, conception, and computation in the BTRA. definitions are essential for precise technical work and clear Instead of directly assessing conditional probabilities for communication with diverse stakeholders. The committee outcomes, DHS subject-matter experts are asked to assess prepared a risk analysis lexicon for its own use (included as conditional probability distributions over the probabilities Appendix A in this final report) with definitions and their of outcomes. This complication is shown to be unnecessary; sources. It is intended to be an example of a lexicon to be the analysis would be unchanged if only the expected value used in future DHS reports and presentations. of these distributions was used. This simplification would significantly reduce data re- Recommendation: The Department of Homeland Se- quirements and accelerate computation. The BTRA software curity should use an explicit risk analysis lexicon for implementation seems to the committee to be cumbersome defining each technical term appearing in its reports and and slow and requires tending by its creators to produce risk presentations. assessments. The committee advises simplification so that the BTRA can be used for responsive risk assessment and risk management. Approach to Determining the Probabilities of Terrorist Decisions Recommendation: The event-tree probability elicitation should be simplified by assessing probabilities instead DHS has made an important contribution by structuring of probability distributions for the outcomes of each a nominal bioterrorist attack and identifying the bioagents event. that should be assessed. The committee closely examined the assumptions and the mathematical details of the BTRA of 2006 and found that there are weaknesses in the model’s Regarding Normalization of Risk Assessment Results conception, errors in some of the underlying mathematics and statistics, and unnecessary complexity. DHS has chosen to represent “normalized” relative risk, The BTRA represents adversarial decisions by means without specifying the normalization constant. This decision of probabilities assessed by subject-matter experts. How- has obscured the results of the analysis and made it impos- ever, when dealing with an intelligent, goal-oriented, and sible to understand the results, to reproduce any particular resourceful adversary (the terrorist), the exclusive use of BTRA result, or to use independent means to assess the subjectively assessed probabilities for terrorist decisions is veracity of any result. Moreover, normalization provides inappropriate. For decision problems as complex as those insufficient information for risk assessment and risk manage- dealt with in the BTRA, the probability that an adversary will ment. Homeland security decision makers and stakeholders choose a course of action should be an output of analysis, need to see the calculated probabilities and consequences not an input. Accordingly: to make risk-informed decisions. This is not to say that the committee believes that precise absolute levels of probabili- Recommendation: To assess the probabilities of terror- ties and consequences can be predicted or are needed. But ist decisions, DHS should use elicitation techniques and risk managers and decision makers need some sense of the decision-oriented models that explicitly recognize terror- magnitude of the probabilities and consequences, and that is ists as intelligent adversaries who observe U.S. defensive not available after normalization. preparations and seek to maximize the achievement of their own objectives. Recommendation: Normalization of BTRA risk assess- ment results obscures information that is essential for risk-informed decision making. BTRA results should not Simplifying the Assessment of Outcome Probabilities be normalized. Decisions, by both terrorist attacker and U.S. defender, should be outputs of a decision support model. The de- Simplification of the BTRA Event Tree termination of data sources and their reliability is outside the scope of this report. However, data concerning threats, The committee finds Stage 1, Frequency of Initiation [of resource levels, technological facts, and so forth are inputs. an attack] by Terrorist Group, of the BTRA fixed-hierarchy Adversarial decisions can be assessed by subject-matter event-tree sequence to be a distracting embellishment. Also,

OCR for page 1
 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT description of the sources of all input data. The documen- the representation of potential multiple (sequential) terrorist tation should be sufficient for scientific peer review. attacks in the BTRA of 2006 is incorrect, both technically and philosophically, and adds an unnecessary layer of com- Recommendation: Subsequent revision of the BTRA plexity to the analysis. The computation of the expected should enable a decision support system that can be run number of attacks is shown to be mathematically incorrect, quickly to test the implications of new assumptions and and the (random) distribution of consequences of such re- new data and provide insights to decision makers and peated attacks is shown to be represented incorrectly. How- stakeholders to support risk-informed decision making. ever, even if the mathematics were correct, the committee believes that, after the first terrorist attack, all assumptions and parameter values in the BTRA would change, so that Rapid Assessment Strategy for New Information the previous risk analysis would no longer apply. Eliminat- ing the BTRA multiple-attack feature would significantly The committee has highlighted the dynamic nature of simplify the model. the biological threat and was asked to show how the BTRA The committee also finds that some of the stages in the might be applied to enhanced or engineered biological BTRA characterization of the steps leading to a terrorist at- agents. The committee suggests a rapid assessment tool and tack might be aggregated to the minimum number of stages proposes a template that suggests how to quickly estimate necessary to calculate probabilities and consequences, mak- the threat from emerging or suspected agents to determine ing data acquisition simpler without sacrificing fidelity. whether a more detailed exigent study is necessary. It agrees that this is an important goal and makes the following Recommendation: Two significant simplifications should recommendation. be made to the BTRA of 2006 event tree: • DHS should eliminate Stage 1, Frequency of Initia- Recommendation: The BTRA should be broad enough to tion [of an attack] by Terrorist Group, and Stage 16, encompass a variety of bioterrorism threats while allow- Potential for Multiple Attacks; and ing for changing situations and new information. DHS • DHS should seek opportunities to aggregate some should develop a strategy for the rapid assessment of stages of the tree to only those essential to calcu- newly recognized and poorly characterized threats. late probabilities and consequences with realistic fidelity. Existing Knowledge and the Detail in Consequence Models Need for Transparent, User-Friendly The committee examined the consequence analysis of Decision Support System the BTRA. It finds that the susceptible, exposed, infected, Risk assessment, such as the BTRA, has no direct impact and recovered (SEIR) model used to analyze the health on risk reduction. Only effective risk management strate- consequences of a bioterrorist attack requires, with regard gies can reduce risk, and there are several barriers to the to pathogens, data that do not exist. There is scant empirical effective use of information from the BTRA in decision basis for pathogens that have only recently been discovered making. These include numerous stakeholders with different in nature and with which there is little experience. Extremely responsibilities, authority, and indicators of success; dispa- limited clinical and epidemiologic data exist about many of rate data and data sources; and organizational friction and the pathogens in the BTRA of 2006. The granularity of detail compartmentalization within and among stakeholders. To in the SEIR models is not supported by existing data on any support risk-informed decision making and mitigate some pathogen on the BTRA list. of these problems, DHS needs transparent and user-friendly Recommendation: The susceptible, exposed, infected, decision support models. Accordingly, the committee makes and recovered (SEIR) model adopted by DHS is more the following three recommendations. complex than can be supported by existing data or Recommendation: Subsequent revision of the BTRA knowledge. DHS should make its SEIR model as simple should increase emphasis on risk management. An in- as possible consistent with existing knowledge. creased focus on risk management will allow the BTRA to better support the risk-informed decisions that homeland Consequences Besides Mortality and security stakeholders are required to make. Morbidity That Need to Be Modeled Recommendation: DHS should maintain a high level of DHS is planning to include second-order economic transparency in risk assessment models, including a com- effects in the BTRA of 2008. The committee highlights prehensive, clear mathematical document and a complete those effects, including important agricultural effects, and

OCR for page 1
 SUMMARY discusses the use of cost-benefit analysis to provide a com- be compared on a common consequence scale to suggest mon measure. and evaluate risk management strategies that encompass all terrorist threats. Recommendation: While human mortality and the magnitude and duration of morbidity should remain the Regarding the Use of the BTRA in Its Present Form primary focus of DHS bioterrorism risk analysis, DHS should incorporate other measures of societal loss, in- For the reasons noted in this report’s recommendations cluding the magnitude and duration of first- and second- and their justifying text, the committee believes that the order economic loss and environmental and agricultural BTRA in its present form should not be used to assess the effects. risk of bioterrorism threats. For the same reasons, the com- mittee does not recommend trying to extend the BTRA to the qualitatively different chemical and radioactive threats. Methods for Improved Modeling of Intelligent Adversaries Recommendation: The BTRA should not be used as a ba- The committee attaches great importance to the realistic sis for decision making until the deficiencies noted in this representation of the behavior of an intelligent adversary. report have been addressed and corrected. DHS should BTRA probabilities are conditioned on past events and are engage an independent, senior technical advisory panel retrospective, whereas the terrorist is prospective, constantly to oversee this task. In its current form, the BTRA should adjusting tactics to exploit any evident weakness in U.S. not be used to assess the risk of biological, chemical, or defenses. radioactive threats. To offer some concrete examples of how to credibly rep- resent the behavior of an intelligent adversary, the committee presents three ways to represent adversarial decisions: (1) a The committee takes very seriously the bioterrorism “bioterrorism decision model” using off-the-shelf software; threats and potential consequences that it has had to consider (2) a tri-level decision support model to allocate defensive in this study. It is fully aware of the potential impact of its investments (visible to the attacker) that represents an attack- recommendations on the BTRA of 2008 and the stakehold- er’s reasonable response to observing these preparations, and ers who await it. However, it believes that the failure to reactions to any attack with the resources made available by properly model intelligent adversaries and a continuation the defensive investments; and (3) a game-theoretic model on the path of unnecessary complexity in computer model- of the adversaries that randomizes expected consequences to ing and simulations will not help the United States defend capture the variability of outcomes. These are not mere theo- against the bioterrorist threats in the 21st century and will retical tools, but rather substantive suggestions drawn from not meet the intent of HSPD-10. Therefore, the committee extensive research and experience in the military and in the unanimously believes that an improved BTRA is needed to private sector. These suggestions can significantly improve provide a much more credible foundation for risk-informed the credibility and usefulness of the BTRA. decision making. Recommendation: In addition to using event trees, DHS REFERENCES should explore alternative models of terrorists as intel- ligent adversaries who seek to maximize the achievement DHS (Department of Homeland Security). 2006. Bioterrorism Risk Assess- ment. Biological Threat Characterization Center of the National Biode- of their objectives. fense Analysis and Countermeasures Center. Fort Detrick, Md. Office of Homeland Security. 2002. National Strategy for Homeland Security. Available at www.dhs.gov/xlibrary/assets/nat_strat_hls.pdf. Use of Intelligent-Adversary Risk Analysis Accessed November 1, 2006. Techniques for Other Threat Areas The White House. 2004. Homeland Security Presidential Directive 10 [HSPD-10]: Biodefense for the 2st Century. Available at www.fas. The committee believes that each of its suggested exten- org/irp/offdocs/nspd/hspd-10.html. Accessed January 16, 2008. sions to realistically represent adversarial behavior is ap- The White House. 2007. Homeland Security Presidential Directive 18 plicable to biological, chemical, and/or radioactive threats. [HSPD-18]: Medical Countermeasures Against Weapons of Mass Although distinct models may need to be developed for the Destruction. Available at www.fas.org/irp/offdocs/nspd/hspd-18.html. Accessed January 16, 2008. analysis of each of these threats, the resulting analyses can