The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change
Summary
Armed with a single vial of a biological agent small groups of fanatics, or failing states, could gain the power to threaten great nations, threaten the world peace. America, and the entire civilized world, will face this threat for decades to come. We must confront the danger with open eyes and unbending purpose.
—President George W. Bush, February 11, 2004
DEPARTMENT OF HOMELAND SECURITY’SBIOLOGICAL THREAT RISK ASSESSMENT
The Committee on Methodological Improvements to the Department of Homeland Security’s Biological Agent Risk Analysis was established by the National Research Council and convened in August 2006 to review the Department of Homeland Security’s (DHS’s) Biological Threat Risk Assessment (BTRA) of 2006. The BTRA is a computer-based tool that has been applied by DHS to assess the risk associated with the intentional release of each of 28 biological threat agents categorized by the Centers for Disease Control and Prevention.
The threat posed by biological agents employed in a terrorist attack on the United States is arguably the most important homeland security challenge of our era. Whether natural pathogens are cultured or new variants are bioengineered, the consequence of a terrorist-induced pandemic could be millions of casualties—far more than we would expect from nuclear terrorism, chemical attacks, or conventional attacks on the infrastructure of the United States such as the attacks of September 11, 2001. Even if there were fewer casualties, additional second-order consequences (including psychological, social, and economic effects) would dramatically compound the effects. Bioengineering is no longer the exclusive purview of state sponsors of terrorism; this technology is now available to small terrorist groups and even to deranged individuals.
The executive branch recognizes this grave threat, as witnessed by the following:
Homeland Security Presidential Directive 10 (HSPD-10): Biodefense for the 21st Century (The White House, 2004) calls for DHS to conduct biennial assessments of biological threats, and
Homeland Security Presidential Directive 18 (HSPD-18): Medical Countermeasures Against Weapons ofMass Destruction (The White House, 2007) applies some of the basic assumptions underlying HSPD-10 to chemical, biological, radiological, and nuclear (CBRN) threats, calling for an integrated CBRN risk assessment.
DHS produced its report Bioterrorism Risk Assessment in 2006 (DHS, 2006). The BTRA of 2006 and the DHS (2006) report, which documents the analysis, respond directly to the requirements of HSPD-10 and of the National Strategy forHomeland Security (Office of Homeland Security, 2002) for DHS to assess the biological weapons threat.
This committee has been called to provide an independent, scientific peer review of the methodology that led to the BTRA of 2006 and that will be the foundation for future biennial updates. At this writing, DHS is preparing a revision of its bioterrorism risk analysis responding to HSPD-18; this analysis will presumably appear, as directed, in 2008. The committee did not have the draft of the DHS report documenting the analysis of the BTRA of 2008, but it was briefed on some of the enhancements and changed procedures that will influence the BTRA of 2008 and considered all information provided in the course of its review.
The committee has identified a number of fundamental concerns with the BTRA of 2006, ranging from mathematical and statistical mistakes that have corrupted results, to unnecessarily complicated probability models and models with fidelity far exceeding existing data, to more basic questions about how terrorist behavior should be modeled. All of these issues are covered in the body of this report.
Rather than merely criticizing what was done in the BTRA of 2006, the committee sought outside experts and collected a number of proposed alternatives that it believes would improve DHS’s ability to assess potential terrorist behavior as a key element of risk-informed decision making, and it explains these alternatives in the specific context of the BTRA and the bioterrorism threat.
The committee set for itself the following gauge of success for its various deliberations and its final report: If DHS
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole. Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change
Summary
Armed with a single vial of a biological agent small groups of fanatics, or failing states, could gain the power to threaten great nations, threaten the world peace. America, and the entire civilized world, will face this threat for decades to come. We must confront the danger with open eyes and unbending purpose.
—President George W. Bush, February 11, 2004
DEPARTMENT OF HOMELAND SECURITY’S BIOLOGICAL THREAT RISK ASSESSMENT
The Committee on Methodological Improvements to the Department of Homeland Security’s Biological Agent Risk Analysis was established by the National Research Council and convened in August 2006 to review the Department of Homeland Security’s (DHS’s) Biological Threat Risk Assessment (BTRA) of 2006. The BTRA is a computer-based tool that has been applied by DHS to assess the risk associated with the intentional release of each of 28 biological threat agents categorized by the Centers for Disease Control and Prevention.
The threat posed by biological agents employed in a terrorist attack on the United States is arguably the most important homeland security challenge of our era. Whether natural pathogens are cultured or new variants are bioengineered, the consequence of a terrorist-induced pandemic could be millions of casualties—far more than we would expect from nuclear terrorism, chemical attacks, or conventional attacks on the infrastructure of the United States such as the attacks of September 11, 2001. Even if there were fewer casualties, additional second-order consequences (including psychological, social, and economic effects) would dramatically compound the effects. Bioengineering is no longer the exclusive purview of state sponsors of terrorism; this technology is now available to small terrorist groups and even to deranged individuals.
The executive branch recognizes this grave threat, as witnessed by the following:
Homeland Security Presidential Directive 10 (HSPD-10): Biodefense for the 21st Century (The White House, 2004) calls for DHS to conduct biennial assessments of biological threats, and
Homeland Security Presidential Directive 18 (HSPD-18): Medical Countermeasures Against Weapons of Mass Destruction (The White House, 2007) applies some of the basic assumptions underlying HSPD-10 to chemical, biological, radiological, and nuclear (CBRN) threats, calling for an integrated CBRN risk assessment.
DHS produced its report Bioterrorism Risk Assessment in 2006 (DHS, 2006). The BTRA of 2006 and the DHS (2006) report, which documents the analysis, respond directly to the requirements of HSPD-10 and of the National Strategy for Homeland Security (Office of Homeland Security, 2002) for DHS to assess the biological weapons threat.
This committee has been called to provide an independent, scientific peer review of the methodology that led to the BTRA of 2006 and that will be the foundation for future biennial updates. At this writing, DHS is preparing a revision of its bioterrorism risk analysis responding to HSPD-18; this analysis will presumably appear, as directed, in 2008. The committee did not have the draft of the DHS report documenting the analysis of the BTRA of 2008, but it was briefed on some of the enhancements and changed procedures that will influence the BTRA of 2008 and considered all information provided in the course of its review.
The committee has identified a number of fundamental concerns with the BTRA of 2006, ranging from mathematical and statistical mistakes that have corrupted results, to unnecessarily complicated probability models and models with fidelity far exceeding existing data, to more basic questions about how terrorist behavior should be modeled. All of these issues are covered in the body of this report.
Rather than merely criticizing what was done in the BTRA of 2006, the committee sought outside experts and collected a number of proposed alternatives that it believes would improve DHS’s ability to assess potential terrorist behavior as a key element of risk-informed decision making, and it explains these alternatives in the specific context of the BTRA and the bioterrorism threat.
The committee set for itself the following gauge of success for its various deliberations and its final report: If DHS
OCR for page 2
Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change
follows the committee’s recommendations (drawn from the individual chapters of this report and presented as a complete set in the next section), the resulting product will more reliably assess the possible acts of terrorists, will be better documented and understood by its clients, and will be more responsive and able not only to assess risk, but to effectively inform strategic investments in risk management.
HSPD-10 states:
Another critical element of our biodefense policy is the development of periodic assessments of the evolving biological weapons threat. First, the United States requires a continuous, formal process for conducting routine capabilities assessments to guide prioritization of our on-going investments in biodefense-related research, development, planning, and preparedness (The White House, 2004).
In accord with HSPD-10, the fundamental concerns of the committee are not only modeling or mathematical details, but the provision to homeland security policy makers of better tools to use when deciding how to invest huge sums of money to protect this nation against a grave threat.
THE CHARGE TO THE COMMITTEE
The charge to the committee for this final report is as follows:
Recommend how the methodology can incorporate changing probability distributions that reflect how various actors (e.g., terrorists, first responders, public health community) adjust their choices over time or in different contexts;
Recommend further improvements to the consequence analysis component of the methodology, including its models of economic effects;
Identify any emerging methods for handling large degrees of uncertainty (e.g., fuzzy logic, possibility analysis) that merit consideration for future incorporation;
Recommend further improvements to the transparency and usability of the methodology;
Discuss in more detail beyond the first report [the committee’s Interim Report] how the methodology could be extended to risks associated with classes of agents, including enhanced or engineered agents that have yet to be developed; and
Discuss in more detail beyond the first report the feasibility of extending the methodology to also serve as a framework for risk analysis of chemical or radioactive threats.
In order to attend to this charge, this committee reviewed all of the detail in the BTRA of 2006, interviewed its implementers, and called on outside experts. It also received briefings from DHS on planned improvements to the BTRA of 2008. During this process, the committee recorded deficiencies and recommended improvements in the assessment.
DHS intended that the BTRA of 2006 be an “end-to-end risk assessment of the bioterrorism threat” with potential catastrophic consequences to human health and the national economy and that it “assist and guide biodefense strategic planning” (DHS, 2006, Ch. 1, p. 1) in response to the HSPD-10 directive to “conduct biennial assessments of biological threats.” Guided by DHS’s customers for information from the assessment, the BTRA of 2006 was designed to produce assessments in the form of risk-prioritized groups of biological threat agents. These prioritized lists could then be used to identify gaps or vulnerabilities in the U.S. biodefense posture and make recommendations for rebalancing and refining investments in the overall U.S. biodefense policy. DHS has assembled a confederation of researchers and subject-matter experts and is collaborating with national laboratories that can contribute to expanding the knowledge base of bioterrorism.
RECOMMENDATIONS
Overall Assessment
The committee met on August 28-29, 2006, with representatives of DHS in response to a DHS request for guidance on its near-term BTRA development efforts. In November 2006, in response to that request and based on the information it had received at the 2-day meeting with DHS, the committee electronically issued its Interim Report (reproduced as Appendix J in this final report). Subsequently the committee received the full DHS (2006) report documenting the analysis in the BTRA of 2006. While DHS agreed with the recommendations of the Interim Report and planned to address them, the committee did not learn of any progress up to the conclusion of its deliberations in May 2007 that would obviate those recommendations, which require sustained work.
However, the content of the DHS (2006) report and information gained at additional meetings with DHS and national experts have significantly changed the committee’s overall assessment of the BTRA of 2006. The committee identified errors in mathematics, risk assessment modeling, computing, presentation, and other weaknesses in the BTRA of 2006. It recommends against using this current BTRA for bioterrorism risk assessment as presented in the BTRA of 2006 or proposed for 2008. Instead, the committee offers improvements that can significantly simplify and improve future risk assessments. The improved BTRA should be used for risk management as well as risk assessment, as intended by HSPD-10.
The committee discusses the elements of risk analyses, including risk management, and identifies the crucial differences between the use of risk analysis to assess and manage the risks of natural disasters and its use to assess and manage risks from terrorist attacks. Representing terrorist decision making exclusively as random variables, as is appropriate
OCR for page 3
Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change
in the case of natural disasters, is a fundamental problem with the BTRA.
Risk Analysis Lexicon
The DHS (2006) report and DHS presentations of its content use inconsistent, imprecise technical language and do not define many key terms. Clear and consistent risk analysis definitions are essential for precise technical work and clear communication with diverse stakeholders. The committee prepared a risk analysis lexicon for its own use (included as Appendix A in this final report) with definitions and their sources. It is intended to be an example of a lexicon to be used in future DHS reports and presentations.
Recommendation: The Department of Homeland Security should use an explicit risk analysis lexicon for defining each technical term appearing in its reports and presentations.
Approach to Determining the Probabilities of Terrorist Decisions
DHS has made an important contribution by structuring a nominal bioterrorist attack and identifying the bioagents that should be assessed. The committee closely examined the assumptions and the mathematical details of the BTRA of 2006 and found that there are weaknesses in the model’s conception, errors in some of the underlying mathematics and statistics, and unnecessary complexity.
The BTRA represents adversarial decisions by means of probabilities assessed by subject-matter experts. However, when dealing with an intelligent, goal-oriented, and resourceful adversary (the terrorist), the exclusive use of subjectively assessed probabilities for terrorist decisions is inappropriate. For decision problems as complex as those dealt with in the BTRA, the probability that an adversary will choose a course of action should be an output of analysis, not an input. Accordingly:
Recommendation: To assess the probabilities of terrorist decisions, DHS should use elicitation techniques and decision-oriented models that explicitly recognize terrorists as intelligent adversaries who observe U.S. defensive preparations and seek to maximize the achievement of their own objectives.
Simplifying the Assessment of Outcome Probabilities
Decisions, by both terrorist attacker and U.S. defender, should be outputs of a decision support model. The determination of data sources and their reliability is outside the scope of this report. However, data concerning threats, resource levels, technological facts, and so forth are inputs. Adversarial decisions can be assessed by subject-matter experts, but these assessments must be conditioned on all of these inputs. This is a daunting task for any subject-matter expert. Appendix G of this report contains material on alternate methods that can be used to quantify uncertainty. This report explains in detail that probability theory is suited to the task and that no alternative is needed. However, this report discusses at length weaknesses in DHS’s use of probability in theory, conception, and computation in the BTRA.
Instead of directly assessing conditional probabilities for outcomes, DHS subject-matter experts are asked to assess conditional probability distributions over the probabilities of outcomes. This complication is shown to be unnecessary; the analysis would be unchanged if only the expected value of these distributions was used.
This simplification would significantly reduce data requirements and accelerate computation. The BTRA software implementation seems to the committee to be cumbersome and slow and requires tending by its creators to produce risk assessments. The committee advises simplification so that the BTRA can be used for responsive risk assessment and risk management.
Recommendation: The event-tree probability elicitation should be simplified by assessing probabilities instead of probability distributions for the outcomes of each event.
Regarding Normalization of Risk Assessment Results
DHS has chosen to represent “normalized” relative risk, without specifying the normalization constant. This decision has obscured the results of the analysis and made it impossible to understand the results, to reproduce any particular BTRA result, or to use independent means to assess the veracity of any result. Moreover, normalization provides insufficient information for risk assessment and risk management. Homeland security decision makers and stakeholders need to see the calculated probabilities and consequences to make risk-informed decisions. This is not to say that the committee believes that precise absolute levels of probabilities and consequences can be predicted or are needed. But risk managers and decision makers need some sense of the magnitude of the probabilities and consequences, and that is not available after normalization.
Recommendation: Normalization of BTRA risk assessment results obscures information that is essential for risk-informed decision making. BTRA results should not be normalized.
Simplification of the BTRA Event Tree
The committee finds Stage 1, Frequency of Initiation [of an attack] by Terrorist Group, of the BTRA fixed-hierarchy event-tree sequence to be a distracting embellishment. Also,
OCR for page 4
Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change
the representation of potential multiple (sequential) terrorist attacks in the BTRA of 2006 is incorrect, both technically and philosophically, and adds an unnecessary layer of complexity to the analysis. The computation of the expected number of attacks is shown to be mathematically incorrect, and the (random) distribution of consequences of such repeated attacks is shown to be represented incorrectly. However, even if the mathematics were correct, the committee believes that, after the first terrorist attack, all assumptions and parameter values in the BTRA would change, so that the previous risk analysis would no longer apply. Eliminating the BTRA multiple-attack feature would significantly simplify the model.
The committee also finds that some of the stages in the BTRA characterization of the steps leading to a terrorist attack might be aggregated to the minimum number of stages necessary to calculate probabilities and consequences, making data acquisition simpler without sacrificing fidelity.
Recommendation: Two significant simplifications should be made to the BTRA of 2006 event tree:
DHS should eliminate Stage 1, Frequency of Initiation [of an attack] by Terrorist Group, and Stage 16, Potential for Multiple Attacks; and
DHS should seek opportunities to aggregate some stages of the tree to only those essential to calculate probabilities and consequences with realistic fidelity.
Need for Transparent, User-Friendly Decision Support System
Risk assessment, such as the BTRA, has no direct impact on risk reduction. Only effective risk management strategies can reduce risk, and there are several barriers to the effective use of information from the BTRA in decision making. These include numerous stakeholders with different responsibilities, authority, and indicators of success; disparate data and data sources; and organizational friction and compartmentalization within and among stakeholders. To support risk-informed decision making and mitigate some of these problems, DHS needs transparent and user-friendly decision support models. Accordingly, the committee makes the following three recommendations.
Recommendation: Subsequent revision of the BTRA should increase emphasis on risk management. An increased focus on risk management will allow the BTRA to better support the risk-informed decisions that homeland security stakeholders are required to make.
Recommendation: DHS should maintain a high level of transparency in risk assessment models, including a comprehensive, clear mathematical document and a complete description of the sources of all input data. The documentation should be sufficient for scientific peer review.
Recommendation: Subsequent revision of the BTRA should enable a decision support system that can be run quickly to test the implications of new assumptions and new data and provide insights to decision makers and stakeholders to support risk-informed decision making.
Rapid Assessment Strategy for New Information
The committee has highlighted the dynamic nature of the biological threat and was asked to show how the BTRA might be applied to enhanced or engineered biological agents. The committee suggests a rapid assessment tool and proposes a template that suggests how to quickly estimate the threat from emerging or suspected agents to determine whether a more detailed exigent study is necessary. It agrees that this is an important goal and makes the following recommendation.
Recommendation: The BTRA should be broad enough to encompass a variety of bioterrorism threats while allowing for changing situations and new information. DHS should develop a strategy for the rapid assessment of newly recognized and poorly characterized threats.
Existing Knowledge and the Detail in Consequence Models
The committee examined the consequence analysis of the BTRA. It finds that the susceptible, exposed, infected, and recovered (SEIR) model used to analyze the health consequences of a bioterrorist attack requires, with regard to pathogens, data that do not exist. There is scant empirical basis for pathogens that have only recently been discovered in nature and with which there is little experience. Extremely limited clinical and epidemiologic data exist about many of the pathogens in the BTRA of 2006. The granularity of detail in the SEIR models is not supported by existing data on any pathogen on the BTRA list.
Recommendation: The susceptible, exposed, infected, and recovered (SEIR) model adopted by DHS is more complex than can be supported by existing data or knowledge. DHS should make its SEIR model as simple as possible consistent with existing knowledge.
Consequences Besides Mortality and Morbidity That Need to Be Modeled
DHS is planning to include second-order economic effects in the BTRA of 2008. The committee highlights those effects, including important agricultural effects, and
OCR for page 5
Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change
discusses the use of cost-benefit analysis to provide a common measure.
Recommendation: While human mortality and the magnitude and duration of morbidity should remain the primary focus of DHS bioterrorism risk analysis, DHS should incorporate other measures of societal loss, including the magnitude and duration of first- and second-order economic loss and environmental and agricultural effects.
Methods for Improved Modeling of Intelligent Adversaries
The committee attaches great importance to the realistic representation of the behavior of an intelligent adversary. BTRA probabilities are conditioned on past events and are retrospective, whereas the terrorist is prospective, constantly adjusting tactics to exploit any evident weakness in U.S. defenses.
To offer some concrete examples of how to credibly represent the behavior of an intelligent adversary, the committee presents three ways to represent adversarial decisions: (1) a “bioterrorism decision model” using off-the-shelf software; (2) a tri-level decision support model to allocate defensive investments (visible to the attacker) that represents an attacker’s reasonable response to observing these preparations, and reactions to any attack with the resources made available by the defensive investments; and (3) a game-theoretic model of the adversaries that randomizes expected consequences to capture the variability of outcomes. These are not mere theoretical tools, but rather substantive suggestions drawn from extensive research and experience in the military and in the private sector. These suggestions can significantly improve the credibility and usefulness of the BTRA.
Recommendation: In addition to using event trees, DHS should explore alternative models of terrorists as intelligent adversaries who seek to maximize the achievement of their objectives.
Use of Intelligent-Adversary Risk Analysis Techniques for Other Threat Areas
The committee believes that each of its suggested extensions to realistically represent adversarial behavior is applicable to biological, chemical, and/or radioactive threats. Although distinct models may need to be developed for the analysis of each of these threats, the resulting analyses can be compared on a common consequence scale to suggest and evaluate risk management strategies that encompass all terrorist threats.
Regarding the Use of the BTRA in Its Present Form
For the reasons noted in this report’s recommendations and their justifying text, the committee believes that the BTRA in its present form should not be used to assess the risk of bioterrorism threats. For the same reasons, the committee does not recommend trying to extend the BTRA to the qualitatively different chemical and radioactive threats.
Recommendation: The BTRA should not be used as a basis for decision making until the deficiencies noted in this report have been addressed and corrected. DHS should engage an independent, senior technical advisory panel to oversee this task. In its current form, the BTRA should not be used to assess the risk of biological, chemical, or radioactive threats.
The committee takes very seriously the bioterrorism threats and potential consequences that it has had to consider in this study. It is fully aware of the potential impact of its recommendations on the BTRA of 2008 and the stakeholders who await it. However, it believes that the failure to properly model intelligent adversaries and a continuation on the path of unnecessary complexity in computer modeling and simulations will not help the United States defend against the bioterrorist threats in the 21st century and will not meet the intent of HSPD-10. Therefore, the committee unanimously believes that an improved BTRA is needed to provide a much more credible foundation for risk-informed decision making.
REFERENCES
DHS (Department of Homeland Security). 2006. Bioterrorism Risk Assessment. Biological Threat Characterization Center of the National Biodefense Analysis and Countermeasures Center. Fort Detrick, Md.
Office of Homeland Security. 2002. National Strategy for Homeland Security. Available at www.dhs.gov/xlibrary/assets/nat_strat_hls.pdf. Accessed November 1, 2006.
The White House. 2004. Homeland Security Presidential Directive 10 [HSPD-10]: Biodefense for the 21st Century. Available at www.fas.org/irp/offdocs/nspd/hspd-10.html. Accessed January 16, 2008.
The White House. 2007. Homeland Security Presidential Directive 18 [HSPD-18]: Medical Countermeasures Against Weapons of Mass Destruction. Available at www.fas.org/irp/offdocs/nspd/hspd-18.html. Accessed January 16, 2008.
