|
Items in cart [4] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 44
4
Cost-Benefit Analysis
A well-constructed cost-benefit analysis (CBA) is designed to provide insight about the
effects of alternative decisions, whether the benefits of a given program exceed its costs, and
which choices are most cost-effective. This provides a structure for analyzing whether a
proposed action or program is reasonable and justified.
As part of the study task, the committee was asked to evaluate the Domestic Nuclear
Detection Office’s (DNDO’s) CBA of the advanced spectroscopic portal (ASP) technology. As
of February 2009, the committee had not seen DNDO’s completed CBA, but was provided with
briefings on the status of the CBA, most recently in October of 2008. At that point, much of the
information in DNDO’s CBA was described to the committee as still in draft form. There is no
single definitive approach to doing analysis of such complex cost-risk tradeoffs. The committee
has chosen to provide suggestions to guide DNDO in completing its CBA, with the
understanding that DNDO’s analysis will not be complete until after testing and technical
evaluation are completed, in the hopes that they will help DNDO make the best case it can for
each option, but recognizing that the result will still be subject to criticism.
In addition, the committee hopes that carrying out these analyses will lead DHS to
reexamine assumptions, practices, and objectives to create a firm foundation for continuing to
improve. For example, DHS ought to consider whether the version of the DOE guidance on
threat quantities and configurations that DNDO is using is operationally realistic and relevant to
the threat and the nature of commerce.
ASSESSING COSTS AND BENEFITS OF THE ASP PROGRAM
There could be two bases for saying that a decision to procure ASPs might meet cost-
benefit criteria: (1) It might lead to a reduction in net dollar costs of procurement, deployment,
and operation,37 or (2) it might increase significantly the likelihood of detecting threat materials
and increase the deterrent value of the systems at a reasonable cost. The preliminary analysis
presented to the committee by DNDO suggests that the former (criterion 1) likely is not the case
for ASP deployments.
FINDING
Because DNDO’s preliminary estimates indicate that the cost increases from
replacing the PVT/RIID combination with ASPs outweigh the cost reductions from
operational efficiencies, it is important to consider carefully the conditions under which the
benefits of deploying ASPs justify the program costs.
37
In standard accounting practices, costs that are assigned dollar values are associated with a relative time of
expenditure for comparison and discounted appropriately to give net present value, or some other, similarly
consistent discounting method is applied. It is the committee’s understanding that DNDO is following standard
practice for discounting.
44
Prepublication Copy
OCR for page 45
CHAPTER 4: COST-BENEFIT ANALYSIS 45
When, where, and how terrorists might attempt to use a nuclear weapon is unknown.
This uncertainty makes it difficult to decide whether or not to invest in a system for detecting
nuclear weapons. The consequences of a successful nuclear detonation by a terrorist group could
be catastrophic. The potential magnitude of the consequences is a major factor in justifying
investments to reduce the risks from nuclear scenarios. The risk of such scenarios depends also
on how likely it is that they might occur. However, because the likelihood of such an attack
cannot be precisely specified it is difficult to estimate the risk from nuclear weapons and the
extent to which this risk is reduced through defensive countermeasures. Despite this uncertainty,
a structured CBA can help to guide decision-making in such a situation.
It is important to consider the standards used to measure ASP performance and whether
meeting these standards is sufficient to warrant the program expenditure. A well-constructed
cost-benefit analysis can aid in evaluating whether the criteria in DHS’s definition of a
“significant increase in operational effectiveness” (shown in Sidebar 3.1) add benefits sufficient
to justify the cost of the ASP program.
FINDING
DHS’ definition of “significant increase in operational effectiveness” is a modest set
of goals: The increases in operational efficiency do not by themselves appear to outweigh
the cost increases from replacing the PVT/RIID combination with ASPs, based on DNDO’s
preliminary estimates, and the criteria do not require significantly improved ability to
detect special nuclear material in primary screening (see Sidebar 3.1).
If the ASPs meet the defined criteria and are able to detect the minimum quantities of nuclear
threat material that DOE recommends (DOE guidance), DHS still will not know whether the
benefits of the ASPs outweigh the additional costs associated with them, or whether the funds are
more effectively spent on other elements of the Global Architecture.
In particular, to determine whether the benefits outweigh the costs, the following issues
need to be addressed:
• The relative effect of the ASP system, relative to the existing PVT/RIID system, on
reducing the probability that an adversary would try to smuggle nuclear material into the
United States (deterrence);
• The relative effect of the ASP system, relative to the existing PVT/RIID systems, on the
probability that an adversary would succeed in smuggling nuclear material into the
United States.
• Whether any benefits identified in the above effects assessments and the improvements
required by the SIOE criteria merit the cost of the improved technology deployment;
RECOMMENDATION
The CBA should provide a convincing narrative involving all relevant costs and
benefits in order to justify spending funds on the ASP program.
Many of the costs and particularly the benefits involved in threat detection systems such
as the ASPs are not easily quantifiable. However, the CBA should provide a convincing
Prepublication Copy
OCR for page 46
46 EVALUATING TESTING, COSTS, & BENEFITS OF ASPs: INTERIM REPORT
narrative involving the relevant costs and benefits in order to justify spending funds on the ASP
program. This narrative should provide qualitative justifications and explanations where
quantitative justifications are not available. It may be difficult to justify a threat detection
program based only on easily quantifiable benefits such as such as the benefits of reduced man-
hours spent scanning cargo. However, a major benefit of threat detection programs is an
increase in security, and careful consideration needs to be given to addressing these benefits in a
thorough way. The committee provides some guidance to DNDO on addressing security benefits
later in this chapter.
In a structured CBA, several key elements need to be thoroughly addressed,
quantitatively where possible, and qualitatively where necessary, as discussed above. These key
elements have been defined by the United States Office of Management and Budget (OMB,
2003) as:
• A clear statement of the objectives of the program, which would address what the ASP
technology is meant to accomplish relative to the polyvinyl toluene (PVT) technology
and how it fits into the rest of the Global Architecture (defined in Chapter 1).
• An assessment of meaningful alternatives, which would address a full range of reasonable
options and the benefits of the ASP program relative to these options, including a good
baseline (typically a no-action alternative).
• A comprehensive, credible and transparent analysis of benefits and costs as appropriate,
which would address a full range of qualitative and quantitative benefits, including
security benefits, as thoroughly as possible.
In the following sections, the committee offers guidance to DNDO in these three areas.
STATEMENT OF THE PROGRAM OBJECTIVES
RECOMMENDATION
A cost-benefit analysis should clearly define the ASP program objectives, including:
• Describing the new and unique capabilities of the ASPs in the context of their role in
the Global Architecture; and
• Defining a realistic baseline alternative against which to compare the ASP
deployment.
A structured CBA begins with a clear description of the objectives of the program and the
specific needs it is designed to meet. For the ASP program, the committee judges that the major
issues to be addressed when stating the program objectives are:
• Describing the unique and new capabilities of the ASPs that will enable them to meet
the program objectives and clarifying their role in the Global Architecture; and
• Defining a realistic baseline scenario, to give the program full credit for benefits and
costs.
The following sections discuss some specific recommendations from the committee for
DNDO’s future work that concern a clear statement of the program objectives.
Prepublication Copy
OCR for page 47
CHAPTER 4: COST-BENEFIT ANALYSIS 47
Describing new and unique capabilities and the role in the Global Architecture
RECOMMENDATION
The larger context of global security should be considered in the ASP program’s
cost-benefit analysis. In particular, DHS should consider tradeoffs and interactions among
different elements of the Global Architecture. Alternative approaches can be used to
prevent the smuggling of nuclear materials into the United States. Some are alternative
approaches to the cargo screening problem, while others are outside the program scope
(such as prevention of smuggling via rail, aircraft, or small boats).
It is important to show clearly the new and unique capabilities that deploying the ASPs
will provide, relevant to stated goals and operational outcomes. In particular, this will involve
considering how the ASP’s capabilities contribute to the larger context of the threat detection
system intended to prevent nuclear and radiological threat material from entering the United
States, known as the “Global Architecture.” Radiation portal monitors (RPMs) detect threat
materials entering the United States in cargo containers on trucks via land border crossings and
seaports, and constitute one piece of this system. The Global Architecture also encompasses
screening for nuclear threat material brought across U.S. borders by plane, by personal
watercraft, by rail, or that is transferred promptly to on-dock rail cars at seaports. In October
2008, DNDO presented the scope of its CBA as limited to the then-current deployment plans,
which included ASP-C38 (land and sea cargo) and ASP-D (the wide-load variant) but excluded
rail deployments.39 The committee had not seen any more recent information regarding the
scope of DNDO’s CBA as of the writing of this report (February 2009).
There are tradeoffs that need to be considered among the many programs that make up
the Global Architecture. Given the limited resources available, investment used to strengthen the
Global Architecture can be applied towards: (1) using different (newer) technologies to fill the
same gap; or (2) filling different gaps, for example, different threats, different geographies, and
different modes of transport. Furthermore, the preferred modes and routes of shipping and
transportation are not static. Nor are threats. A more comprehensive evaluation of security
benefits would factor in such trends For example, the enhanced capabilities provided by the
ASP-C system are relevant to cargo containers entering the United States by truck, but not by
rail. In the future, it is probable that less cargo will be brought directly into major U.S. seaports.
Some of the fastest-growing ports in North America are in Canada and Mexico,40 and it is
expected that these ports will handle increasing amounts of cargo destined for the United States.
Much of this cargo will be unloaded onto on-dock rail and will cross U.S. borders on rail.
The tradeoffs among different spending options need to be considered in the ASP CBA or
in a higher level analysis about allocation of efforts and funds. Indeed, it may be more
appropriate for such an analysis to be carried out at a higher level so that it can provide guidance
and support for multiple programs in a coordinated fashion. The committee has, however, seen
no evidence that the higher level tradeoff analysis has been done, and a recent report by the GAO
38
Anything that is legal to drive on a road can pass through the ASP-C variant.
39
The ASP-D is considered a minor variant; however, DNDO informed the committee in October, 2008 that
because the modifications needed to accommodate on-dock rail were significant, they would not be including on-
dock rail as part of the program scope.
40
Some of the fastest-growing ports in North America include Manzanillo, Lazaro Cardenas, and Vancouver. For
further information, see http://aapa.files.cms-plus.com/PDFs/North_American_Container_Traffic.pdf.
Prepublication Copy
OCR for page 48
48 EVALUATING TESTING, COSTS, & BENEFITS OF ASPs: INTERIM REPORT
(2009) tends to confirm this view. Furthermore, an ASP CBA should at the very least consider
alternatives arising from work in the same office on the same mission (e.g., improved RIIDs).
Defining a good baseline
The benefits and costs of a program are defined in comparison with a clearly stated
baseline, typically a “no-action” alternative. This is the type of baseline DNDO indicated to the
committee that they had chosen as of October 2008: In this baseline, no ASPs would be
deployed, and PVT deployment would be expanded. PVTs would continue to be used in both
primary and secondary screening, and the handheld RIIDs would be used in secondary screening
according to Custom and Border Protection’s (CBP) current Concept of Operations (CONOPS).
It is important that the baseline option reflect the key features of the actual systems as
they are deployed today, to ensure that costs and benefits are being accurately assigned to the
ASP program. In considering DNDO’s baseline scenario, one must take into account several
assumptions about the current operations of the PVTs to give the ASPs appropriate benefit for
providing increased security. In this case, a good baseline might include:
• Using the actual sensitivity settings of PVTs deployed in various ports; and
• Correctly accounting for the range of densities of typical materials in containers.
As of DNDO’s October 2008 presentation to the committee, the baseline alternative in
the CBA presumed that the PVTs at all ports were set to the same detection level and operated
identically. As discussed earlier in this report, the PVTs signal a “detection” when the observed
radiation exceeds a given level. When this occurs, the conveyance is pulled aside for a
secondary screening using a second PVT (the truck passes through the detector at a slower rate)
and a hand-held RIID operated by a CBP officer. However, cargo containing NORM (for
example, granite countertops or porcelain toilets) can set off these detectors, which are unable to
distinguish between NORM cargo and threat materials. If the PVT is set more sensitively (the
system alarms when it detects lower levels of radiation) then more NORM cargo is diverted to
secondary screening than if the PVT is set less sensitively.
If the alarm threshold is set too low (the system is too sensitive), the flow of commerce
can be affected by the amount of NORM cargo sent to secondary screening. If the alarm
threshold is set too high (the system is not sensitive enough) then material of concern could pass
through without an alarm. More background regarding thresholds and sensitivity can be found in
Chapter 2. To the extent that the ASPs allow detectors to be operated with a greater level of
sensitivity, they would be expected to detect threat materials more reliably.
In addition, the baseline needs to recognize and account for (to the extent possible) the
range of material densities in typical containers that is brought into a given port. Threat material
could be shielded by cargo brought into the United States in ordinary commerce. An
operationally realistic range of material densities can then be used to define the range for
comparative evaluation between the ASPs and the PVT/RIID systems. At present, the DOE
threat guidance uses a single value for cargo density, and that value does not represent the upper
limit of the range of typical cargo densities. Lower density cargo provides less shielding and
therefore a less challenging detection problem.
Prepublication Copy
OCR for page 49
CHAPTER 4: COST-BENEFIT ANALYSIS 49
ASSESSMENT OF A RANGE OF MEANINGFUL ALTERNATIVES
RECOMMENDATION
A cost-benefit analysis for the ASP program should demonstrate that a full set of
meaningful alternatives has been assessed, including alternative deployments, operations,
and technologies.
A well-performed CBA demonstrates that a full set of meaningful alternative approaches
to the proposed program has been assessed relative to the baseline. According to OMB Circular
A-4 (OMB, 2003), a well-performed analysis “describe[s] the alternatives available and the
reasons for choosing one alternative over another … [i]t is not adequate simply to report a
comparison of the agency’s preferred option to the chosen baseline.”
DNDO informed the committee in October of 2008 that three potential deployment plans
for the ASPs are currently being considered, in addition to a baseline plan, discussed above.
These three plans include:
1. Deployment in secondary: expand the deployment of PVTs in primary screening, and
deploy ASPs in secondary screening;
2. Deployment in primary and secondary: deploy ASPs in primary screening and
secondary screening; and
3. Hybrid deployment: For primary screening, deploy some primarily in high traffic
ports, and retain PVTs in other, lower-traffic ports. For secondary screening, install
ASPs at all ports.
The committee was not shown assessments of other alternatives (apart from the baseline
alternative) as of February 2009, although DNDO may have analyzed others. Increased security
might be achieved without the deployment of ASPs, and a good CBA would clearly demonstrate
that all reasonable possibilities have been assessed, or give reasons why these possibilities were
not assessed.41
Several alternative approaches are possible. The comparative costs and benefits of
changes to CBP’s CONOPS could be assessed. For example, detection equipment could be
deployed at every exit and CBP could select random nuisance alarms to be examined, potentially
resulting in heightened deterrence effects; or secondary scanning times could be increased,
providing time for high purity germanium (HPGe) detectors to perform Identifications. Another
alternative could be to maintain current CONOPS, but deploy alternative technologies, such as
improved RIIDs. The use of newly available RIIDs and associated software could improve the
performance of secondary screening: newer, more sensitive models of RIIDs are available.
Alternatively, software on the existing RIIDs could be improved. 42 This list is meant to be
illustrative and not comprehensive. DNDO and CBP may have insights into alternatives of which
the committee is not aware. At the same time, for practical reasons, the set has to be finite and
relatively small, so the cases should be chosen carefully to represent the most promising
alternatives.
41
In addition, note that CBP officers may prefer the use of handheld detectors, and some deployment of RIIDS may
be needed even in the case that the ASPs are deployed in secondary.
42
For example, RIIDs using high purity germanium detectors are available. Improvements to RIIDs and to the
associated software are discussed in more detail in Chapter 3 of this report.
Prepublication Copy
OCR for page 50
50 EVALUATING TESTING, COSTS, & BENEFITS OF ASPs: INTERIM REPORT
A COMPREHENSIVE, CREDIBLE, AND TRANSPARENT ANALYSIS
OF BENEFITS AND COSTS
RECOMMENDATION
A cost-benefit analysis for the ASP program should include a comprehensive,
credible and transparent analysis of benefits and costs, although the committee
acknowledges that DNDO will not be able to perform a full, quantitative cost-benefit
analysis.
DNDO’s CBA is intended to provide guidance about whether the additional project costs
associated with deploying the ASP systems are outweighed by the improvements in detection
and other benefits. In October of 2008, DNDO presented a preliminary set of total life cycle cost
estimates (LCCE) for the three ASP deployment scenarios outlined previously as well for a
baseline scenario. However, the committee did not see a breakdown of these costs into
categories such as design and development; procurement; deployment; maintenance; operations;
or decommissioning, so it is unable to assess the validity of the projected costs. The approach
that DNDO described to the committee for the ASP LCCE appeared to use a reasonable
methodology. The committee did not see the details of this assessment; however, they suggest
that it is essential to a valid CBA to supply uncertainties associated with the projected costs. The
cost assessment should cover all phases of the acquisition life cycle in a manner that is
independent of contractor or program office biases and assess the risk of cost escalation
associated with the estimate. It is also possible that adoption of new technologies will lead to
cost reductions, although this is less common in such procurements.
As of the writing of this report (February 2009), DNDO had not yet presented an
assessment of the security benefits of the ASP program. DNDO had considered some benefits,
such as the ability for CBP officers to be reassigned to other missions, time saved in secondary
screening, and a reduced number of conveyances referred to secondary screening. However,
according to DNDO (2008b) these benefits alone are unlikely to justify the costs of the ASP
program, and other, more difficult to quantify benefits of will need to be taken into account,
including security benefits. This is a point on which DNDO has asked for specific advice from
the committee.
The committee recognizes the likely inability of the DNDO (or anyone else) to perform a
full, quantitative cost-benefit analysis for the ASP program. Despite these difficulties, at
minimum, a logical connection of the program effort to its goals needs to be presented.
A well-performed CBA that helps in procurement decisions for ASPs is not going to be a
simple analysis following standard formulae commonly used in other kinds of procurements.
DNDO expressed its difficulty in assessing two of the cost-risk elements in the cost-benefit
analysis with respect to equipment performance: assessing the probabilities of failure to detect
threat material, and factoring in potential consequences of such a failure. There are four
probabilities involved in analyzing security benefits: Pdetect, or the probability of detecting the
threat material; Pidentification, or the probability of correctly identifying the threat material; Pinterdict,
or the probability of interdicting the threat material; and Pencounter, the probability of an adversary
attempting to smuggle threat material into the United States in the first place. The last probability
is likely to be highly complex to evaluate, and indeed impossible to determine definitively.
Although it is difficult to estimate such probabilities with high confidence, analysts need to
Prepublication Copy
OCR for page 51
CHAPTER 4: COST-BENEFIT ANALYSIS 51
understand what they can about these probabilities based on analytical tools and input from the
intelligence community and other sources.43
The consequences are likewise uncertain, although in this case because of indeterminacy.
The consequences of a successful nuclear or radiological attack could be assessed within
reasonable uncertainty bounds if the nature of the weapon, its location, and all of the
environmental conditions, including where the people are (e.g., is it rush hour?), were specified.
But in a site-generic, time-independent assessment, the variability in the possible consequences
is very broad. All of these factors make it quite difficult to consider avoided events in the cost-
benefit analysis. However, that difficulty does not make it less important to consider these
factors.
Security benefits can result from changes in any of the above probabilities. The benefits
can take the form of higher detection, identification, and interdiction probabilities. They can also
result from deflection (e.g. to overseas targets) or deterrence (effectively reducing the probability
of encounter), so the security benefits associated with these factors also need to be considered.
(Note, however, that deflection can push adversaries to use different avenues to the same target,
as explained below.) The existence of some radiation monitoring at seaports and land border
crossings may provide sufficient discouragement to potential adversaries from smuggling via this
route. However, it may also increase the probability that the terrorist will focus on other gaps in
the nation’s security that are identified as easier targets. For these reasons, benefits from
increased detection probabilities may be modest as long as there are significant gaps in the
Global Architecture. Improved detection can be expected to become more beneficial as those
gaps are filled.
There are several analytical approaches that may be useful to DNDO in performing an
analysis of the security benefits of the different alternatives proposed for the ASP program.
Below are three examples. Each of these options suffers a common shortcoming – they do not
answer the question of whether the benefits of implementing ASP exceed the program’s costs.
However, each in a different way can provide insights that could help the Secretary weigh the
merits of acquiring and deploying ASPs or alternative nuclear detection technologies.
A capability-based planning approach would provide a structured assessment of how
alternative detection technologies or deployment strategies reduce the risk of a nuclear
detonation in the United States. This approach has been applied in defense applications to
compare and contrast a set of options for approaching a given operational challenge across a
wide range of circumstances (Davis 2002). In the case of ASP, the operational challenge is to
prevent a nuclear or radiological attack by detecting, identifying and interdicting materials or a
weapon smuggled into the United States. The set of options could include alternative
deployments of ASPs and PVTs, deployment of alternative RIIDs, or (depending on the scope of
analysis) shifting emphasis between port-of-entry (POE) and non-POE detection. The
circumstances considered would include relevant dimensions of the adversary capability and
tactics, the operating environment, and the technologies themselves. For example, circumstances
could include different types of nuclear materials, use of different shielding or masking methods,
technology performance now or in the future, different operating environments, different
numbers of weapons available to the adversary, or failure of primary and secondary inspection
43
Other parts of DHS have conducted expert elicitations of threat probabilities with members of the intelligence
community. A recent report by the National Research Council cautioned about limitations of this approach for the
bioterrorism risk assessment. The committee, however, sees value in factoring in what the intelligence community
knows and suspects, accounting at the same time for the confidence that can be placed in that knowledge.
Prepublication Copy
OCR for page 52
52 EVALUATING TESTING, COSTS, & BENEFITS OF ASPs: INTERIM REPORT
due to a common source of failure. Using a model that reflects the detection systems
performance, the alternatives are then compared across multiple metrics appropriate for these
circumstances (e.g., probability of detection of a representative threat object. The strength of a
capabilities-based planning approach is that it can provide a rich comparison of the security
benefits emphasizing the circumstances under which each option might be preferred. The
weakness of this approach is that exploring the circumstances that affect the systems capability
can quickly lead to large and complex analysis and judgment of the analysts is required to
balance this complexity against and ability to draw salient insights about a systems capabilities.
Game theory could provide insight into the benefits from deterrence associated with the
PVTs and ASPs. Studies in other areas have found that the simple presence of security can
significantly deter individuals from choosing the action that the security is meant to protect
against. For example, Ayres and Leavitt (1998) used game theory to predict that the ability of a
criminal to observe security measures affects the deterrent ability of that security measure and
validated this prediction with observations from vehicle theft statistics: An increase in the
percentage of lojack-equipped44 vehicles in a given area is associated with a substantial decline
in auto theft.45 In contrast, observable security devices against car theft tended to merely shift
the risk of theft to other vehicles, but not lower overall rates of theft. In another example of
game theory being used for security policy, Kunreuther (2005) demonstrated the public policy
opportunity that exists in commercial aviation because of tipping effects that would lead to mass
adoption of baggage screening technologies under the right policy incentives. Researchers who
have used game theory to assess protection of critical infrastructure from terrorism are just
beginning to explore the utility of these approaches to decisions about radiation portal monitors
(Bier and Azaiez 2009; Dighe et al. 2009), as noted earlier. In the context of ASP procurement,
one would have to look at the incremental benefit of installing new detectors. The general
weakness of game theory is that if analysts are unable to estimate parameters of their model, they
are only able to draw broad and conditional conclusions about adversary behavior. Examples of
parameters from security applications that are difficult to estimate include the value to the
adversary of different outcomes (i.e., what might constitute successes and what are the costs of
being caught), the probability of attack, and the costs of the defender falsely suspecting an attack
is occurring. In the absence of being able to establish values for model parameters, broad or
conditional conclusions might or might not provide actionable advice to policymakers.46
Finally, cost-effectiveness analysis and break-even analysis are related approaches that
have been used to assess costs and benefits when performing a complete cost-benefit analysis is
difficult or impossible. Cost-effectiveness analysis is used when valuation of benefits is
contentious and it is not possible to quantify benefits in monetary terms. Because the security
goals of the ASP program may be difficult to value monetarily, comparing program alternatives
using cost-effectiveness measures such as dollars per life saved or dollars per attack avoided
could provide insights into their relative merits. In contrast, break-even analysis can be used
44
Lojack is a hidden radio-transmitter device used for retrieving stolen vehicles.
45
This occurs without a drop in other types of crimes. (Ayres and Leavitt, 1998)
46
Consider the case of deterrence. Partial screening (screening a fraction of the total number of containers entering
the United States) may provide for effective deterrence (or deflection), if detection probabilities are sufficiently high
and if smugglers cannot predict which containers will not be screened. This benefit quickly evaporates if adversaries
are able to stage several smuggling attempts simultaneously because the chance of at least one attempt succeeding
grows rapidly with the number of attempts (Bier and Haphuriwat, 2009). Like other game theoretic analyses, this
enters a psychological realm, ascribing logical values to the adversary (e.g., that the threat material is a scarce and
valuable asset and that the risk of discovery at a port of entry is not desired).
Prepublication Copy
OCR for page 53
CHAPTER 4: COST-BENEFIT ANALYSIS 53
when it is difficult to even assess the magnitude of benefits in any units. Break-even analysis
determines conditions that must be met for benefits to exceed costs. In security applications,
these conditions could be a required reduction in overall risk (see Willis and LaTourrette 2008
analysis of the Western Hemisphere Travel Initiative) or a baseline estimate of a threat of attack
that exists (see Martonosi et al. 2006 analysis of 100% container inspection). In some instances,
framing the problem in this manner has proven useful because it also can provide a simple, yet
fully parameterized model of the system being evaluated which a policymaker can explore to
understand conditions that must be met for benefits to exceed costs (see von Winterfeldt and
O’Sullivan analysis of acquisition of MANPADs defenses on commercial aviation). In cases
where break-even analysis identifies meaningful bounds on decisions, that is threshold
conditions that can easily be judged to exist or not exist, this approach can simplify
decisionmaking. The downfall of break-even analysis is that these conditions do not always
exist.
The committee reiterates that methods for evaluating security benefits, examples of
which are provided above, can provide different insights based on their approach, and none is
likely to provide fully quantitative and definitive results. But most policy decisions are made
without fully quantitative and definitive results, so DNDO should incorporate these benefits to
provide the most informative CBA it can.
The committee recommends that DHS not proceed with further procurement until it has
addressed the findings and recommendations in this report and the ASP is shown to be a favored
option in the cost-benefit analysis.
Prepublication Copy
