|
Items in cart [4] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 3
Summary
Containerized cargo entering the United States at sea ports and land-border crossings for
trucks is currently screened for radiation using detectors, called radiation portal monitors (RPMs)
made from a plastic scintillator, called PVT,1 in conjunction with handheld radioisotope
identifiers (RIIDs). The Department of Homeland Security (DHS) is seeking to deploy new
radiation detectors, called advanced spectroscopic portals (ASPs), to replace the PVT and RIID
combination, which has known deficiencies. Title IV of division E of the Consolidated
Appropriations Act, 2008 (Public Law 110-161) requires the Secretary of Homeland Security to
submit to Congress a report certifying that a “significant increase in operational effectiveness”
over continued use of the existing screening devices will be achieved with the ASP before “funds
appropriated under this heading shall be obligated for full-scale procurement of Advanced
Spectroscopic Portal Monitors.” DHS is testing and evaluating the ASPs to inform the
Secretary’s certification decision. If the Secretary certifies the ASPs, DHS may purchase more
than one billion dollars worth of ASPs. The net lifecycle cost of these ASPs could be more than
twice that figure.
The U.S. Congress directed DHS to request that the National Research Council of the
National Academies conduct a study prior to certification to: (1) evaluate the adequacy of the
past testing and analyses of the ASP systems performed by DHS’s Domestic Nuclear Detection
Office (DNDO); (2) evaluate the scientific rigor and robustness of DNDO's current testing and
analysis approach; and (3) evaluate DNDO's cost-benefit analysis of ASP technology. Due to
delays in the test and evaluation program, the Academies and DHS agreed that the study
committee would issue an interim report that provides the committee’s evaluation of testing
plans and execution it has seen, and advice on how DNDO can complete and make more
rigorous its ASP evaluation for the Secretary and the nation.
This interim report is based on testing done before 2008, plans for and preliminary results
from tests done in 2008, and the agency’s draft cost-benefit analysis as of October 2008. The
committee received briefings on the performance test results and analysis and on the cost-benefit
analysis, but the committee did not receive written reports on those topics by February 2009,
when the interim report entered the Academy peer review process. The committee addresses
each element of the study task below.
PAST PERFORMANCE TESTING
Performance tests prior to 2008 had serious flaws that were identified by the Government
Accountability Office and the Secretary’s ASP Independent Review Team. All truck-conveyed
containers at ports and border crossings pass through a PVT portal which constitutes primary
screening, and those trucks that trigger an alarm are sent to secondary screening, which is
conducted with a PVT portal and RIID. The tests prior to 2008 did not adequately assess the
capabilities of the ASP systems in primary and secondary screening compared with the currently
deployed PVT and RIID screening systems, nor whether the ASP systems met criteria for
procurement. DNDO utilized the same sources in performance testing that were used to set up
and calibrate this testing. The number of sources available was small, but this is not sufficient
1
PVT stands for polyvinyl toluene.
3
Prepublication Copy
OCR for page 4
4 EVALUATING TESTING, COSTS, & BENEFITS OF ASPs: INTERIM REPORT
reason to use the same sources for both set up and testing. Device setup and any calibration must
use separate sources from those used for testing. A component of the standard operating
procedures for the RIIDs in secondary screening was not followed in the performance tests,
which disadvantaged the RIID in comparisons with ASPs.
2008 PERFORMANCE TESTING
In describing and discussing the tests with the committee, DNDO staff acknowledged
several pre-2008 deficiencies. According to the 2008 test plan and briefings to the committee in
Washington, D.C., and at the Nevada Test Site, these deficiencies were corrected. This is
consistent with the committee’s observations of tests and questioning of test personnel.
Because they have large detectors and because of their configuration, ASPs would be
expected to improve isotope identification, and provide greater consistency in screening each
container, greater coverage of each container, and increased speed of screening over that of the
PVT/RIID combination when used in secondary screening. Consequently, tests of ASPs in
secondary screening focused on confirming and quantifying that advantage for several threat
objects, cargos, and configurations.
When used for primary screening, an ASP system must be compared to the existing
combined primary and secondary screening system (both PVT and RIID) because of differences
in standard operating procedures for primary screening (ASPs in primary have an identification
function). DNDO’s preliminary analysis did account for this difference.
The 2008 performance tests were an improvement over previous tests. DNDO physically
tested some of the limits of the systems. However, the following shortcomings remain. (1)
Without modeling to complement the physical experiments, the selected test configurations are
too limited; (2) the sample sizes are small and limit the confidence that can be placed in
comparisons among the results; and (3) in its analysis, some of the performance metrics are not
the correct ones for comparing operational performance of screening systems. These
shortcomings are described in greater detail within the report. For these reasons, DHS cannot
conclude definitively whether ASPs will consistently outperform the current PVT-RIID systems
in routine practice until the shortcomings are addressed. Better measurement and characterization
are a necessary first step but may not be sufficient to enable DHS to conclude that the ASPs meet
the criteria DHS has defined for achieving a “significant increase in operational effectiveness.”
The committee recommends modifications to the current DHS approach to the evaluation
procedure. These modifications would influence subsequent procurement steps.
RECOMMENDED APPROACH FOR TESTING AND EVALUATION
To make the testing and evaluation more scientifically rigorous, the committee
recommends an iterative approach with modeling and physical testing complementing each
other. The threat space—that is, the set of possible threat objects, configurations, surrounding
cargoes, and conditions of transport—is so large and multidimensional that DNDO needs an
analytical basis for understanding the capabilities of detectors for screening cargo. DNDO’s
current approach is to physically test small portions of the threat space and to use other
experimental data to interpolate and extrapolate throughout the threat space to test the
identification algorithms in the detector systems.
Prepublication Copy
OCR for page 5
SUMMARY 5
For a more rigorous approach, DNDO should use theory and models of threat objects,
radiation transport, and detector response to simulate performance and predict outcomes. Then
DNDO can use physical experiments to validate the predictions and allow a critique of the
models’ fidelity to reality. This would enable developers to refine the models iteratively. With
validated models, DNDO can evaluate the performance of the ASP systems over a larger, more
meaningful range of cases and threat space than is feasible with physical tests alone.
This kind of interaction between computer models and physical tests is standard for the
development of some high-technology equipment and is essential for building scientific
confidence. The performance tests conducted in 2008, and even prior to 2008, can be used to
help refine and validate models.
RECOMMENDED APPROACH FOR THE PROCUREMENT PROCESS
The idea of an iterative approach extends to deployment, too. The committee noted that
DHS’s testing philosophy is oriented toward a one-time certification decision in the near future.
However, the mandate for passive radiation screening of cargo at ports of entry is expected to
continue indefinitely. Rather than focusing on the single decision about the deployment of ASPs,
the current testing should be viewed as a first step in a continuous process of improvement and
adaptation of the systems. The threat environment, the composition of container cargo,
technological and analytical capabilities, and the nature of commerce at the ports of entry have
changed significantly over the last decade and can be expected to evolve in both predictable and
unpredictable ways in the coming years. DHS should develop a process for incremental
deployment and continuous improvement, with experience leading to refinements in both
technologies and operations over time, rather than a single product purchase to replace current
screening technology. The process should be developed to address and exploit changes. This
would result in a system that can be adapted and updated continuously so that it would not be
outdated by the time all of the ASPs are deployed.
As the first step in this process DHS should deploy its currently unused low-rate initial
production ASPs for primary and secondary inspection at various sites as extended operational
testing. Such deployment, even on this limited scale, would provide additional data concerning
their operation, reliability, and performance, and allow DHS to better assess their capabilities in
multiple environments without investing in a much larger acquisition at the outset.
The development of the hardware for radiation detection and the software for analyzing
the signals from the detectors is separable. It has been useful to have a competitive approach for
the combined systems and to see the results. However, as DHS moves forward, it should match
the best hardware to the best software (particularly the algorithms), drawing on tools developed
for the competition and elsewhere, such as the national laboratories.
ASPs will not eliminate the need for handheld detectors with spectroscopic capabilities.
Because some of the improvement in isotope identification offered by the ASPs over the RIIDs
is a result of software improvements, the best software package also should be incorporated into
improved handheld detectors. Newer RIIDs with better software might significantly improve
their performance and expand the range of deployment options available to CBP for cargo
screening.
By separating these elements and engaging the broader science and engineering
community, DHS would have increased confidence in its procurement of the best product
available with current technology, and simultaneously could advance the state of the art.
Prepublication Copy
OCR for page 6
6 EVALUATING TESTING, COSTS, & BENEFITS OF ASPs: INTERIM REPORT
RECOMMENDED APPROACH FOR COST-BENEFIT ANALYSIS
The preliminary analysis presented to the committee suggests that benefits of deploying
the ASPs may not be clearly and undeniably greater than the costs. Because DNDO’s
preliminary estimates indicate that the cost increases from replacing the PVT/RIID combination
with ASPs outweigh the cost reductions from operational efficiencies, it is important to consider
carefully the conditions under which the benefits of deploying ASPs justify the program costs. A
cost-benefit analysis (CBA) can provide a structure for evaluating whether a proposed program
(such as the ASP program) is reasonable and justified.
The Secretary’s decision on ASP certification is to rely, at least in part, on whether the
ASPs meet the objectives in DHS’ definition of “significant increase in operational
effectiveness” (SIOE); however, other factors relating to the costs and benefits of the proposed
ASP program will also need to be taken into account. DHS’ definition of a SIOE is a modest set
of goals: As noted above, the increases in operational efficiency do not by themselves appear to
outweigh the cost increases from replacing the PVT/RIID combination with ASPs, based on
DNDO’s preliminary estimates, and the criteria do not require significantly improved ability to
detect SNM in primary screening (see Sidebar 3.1). If the ASPs meet the defined criteria and are
able to detect the minimum quantities of nuclear threat material that DOE recommends (the
“DOE guidance”), DHS still will not know whether the benefits of the ASPs outweigh the
additional costs associated with them, or whether the funds are more effectively spent on other
elements of the Global Architecture.
A CBA can provide insight about the effects of alternative decisions, whether the benefits
of a given program exceed its costs, and which choices are most cost-effective. To do this, the
cost-benefit analysis needs to include three key elements: (1) a clear statement of the objectives
of the screening program; (2) an assessment of meaningful alternatives to deploying ASPs; and
(3) a comprehensive, credible and transparent analysis of in-scope benefits and costs.
The CBA should begin by stating clearly what operational problem the ASPs are
intended to address. This statement will define the role that the system plays in providing a layer
in the defense against the importation of a nuclear or radiological device. It should include a
narrative that clarifies how the task of improving detection for containers at ports of entry to the
United States fits into a larger effort to implement or improve detection capabilities, in
recognition of the many ways that materials could be brought into the United States through
ports of entry that are not already screened, or across uncontrolled stretches of border.
Furthermore, to be useful in a procurement decision, a CBA will need to address whether funds
are better spent to replace the currently deployed equipment rather than to expand coverage to
other pathways that currently have no radiation screening. This is needed in the ASP CBA
because it is not evident that it has been done elsewhere.
The CBA needs to account for meaningful alternatives (including non-ASP programs) to
reveal the scale of the benefits of ASPs for radiation screening and determine whether these
benefits outweigh the additional costs. The complexity of the container screening task provides
opportunities for many different options worthy of consideration. These options include
variations on deployment configuration and operational processes, and application of
technologies beyond the PVT/RIID and ASP detectors such as improved versions of existing
handheld passive detectors (deploying handhelds with state-of-the-art software) and advanced
Prepublication Copy
OCR for page 7
SUMMARY 7
methods for detecting nuclear materials. Considerations should include active interrogation,
improved imaging systems, and integration of existing technologies.
These alternatives need to be compared to a baseline that reflects as realistically as
possible the screening capability that DHS currently has in place. Thus, the baseline should
reflect the number and placement of PVT and RIID detectors, sensitivity of the sensors based on
how they are operated at each port, and performance of existing handheld detectors in the
manner they are used in the field. Such an analysis would indicate what capability an investment
in ASPs will provide beyond the existing systems as they are currently deployed and operated or
beyond alternative technologies that could be developed and deployed for radiation detection.
In comparing these alternatives, it is important that the cost-benefit analysis treat benefits
and costs in a comprehensive, credible, and transparent manner. The benefit assessment should
show how this program contributes to improving security with respect to prevention of the
detonation of a nuclear device or radiological weapon in the United States. Because this is the
primary objective of the ASP program, a cost-benefit analysis that is silent on this subject would
be incomplete. Such an assessment is difficult and no assessment of such benefits will be
definitive or unassailable, however it remains important to consider these factors. The cost
assessment should cover all phases of the acquisition life cycle in a manner that is independent of
contractor or program office biases and assess the risk of cost escalation associated with the
estimate.
The committee recommends that DHS not proceed with further procurement until it has
addressed the findings and recommendations in this report and the ASP is shown to be a favored
option in the cost-benefit analysis.
Prepublication Copy
