Page 1
Executive Summary
In 1997, the Federal Aviation Administration (FAA) was directed by President Clinton and authorized by Congress (PL 104-264, PL 104-208) to deploy 54 FAA-certified explosives-detection systems1 (EDSs) and more than 400 trace explosives-detection devices (TEDDs) at airports around the country. The purpose of these deployments was to prevent attacks against civil aviation, such as the recent attempt described in Box ES-1. This report, which assesses the FAA's progress in deploying and utilizing equipment and procedures to enhance aviation security, was produced by the National Research Council (NRC) in response to a congressional directive to the FAA (PL 104-264 § 303). This is the first of four reports assessing the deployment of technologies (i.e., equipment and procedures) by the FAA. In this report the 1997–1998 deployment of technologies by the FAA to improve aviation security is assessed. This panel was convened under the auspices of the NRC Committee on Commercial Aviation Security. Although appropriations are authorized for this assessment through fiscal year 2001, the Committee on Commercial Aviation Security will conclude its work in 1999. Therefore, with the agreement of the FAA, the assessment will be continued by a new committee that will be convened by the NRC in 1999. The form of this report reflects the panel's understanding of this study as part of an ongoing assessment (based on the enabling congressional language). For this reason, the panel carefully distinguished issues and topical areas that could be completed in the first year from those that would require further study.
This report assesses the operational performance of explosives-detection equipment and hardened unit-loading devices (HULDs) in airports and compares their operational performance to their laboratory performance, with a focus on improving aviation security. As requested by Congress, this report addresses (in part) the following issues:
1. Assess the weapons and explosive-detection technologies available at the time of the study that are capable of being effectively deployed in commercial aviation.
2. Determine how the technologies referred to in paragraph (1) could be used more effectively to promote and improve security at airport and aviation facilities and other secured areas.
3. Assess the cost and advisability of requiring hardened cargo containers to enhance aviation security and reduce the required sensitivity of bomb-detection equipment.
4. On the basis of the assessments and determinations made under paragraphs (1), (2), and (3), identify the most promising technologies for improving the efficiency and cost effectiveness of weapons and explosives detection.
This panel considers aviation security as a total system architecture and measures the effectiveness of deployment on that basis.
Deployed Technologies
The congressionally mandated deployment of bulk explosives-detection equipment began in January 1997 and continued throughout 1998. The FAA formed the Security Equipment Integrated Product Team (SEIPT) to carry out this deployment. The SEIPT assessed the availability of explosives-detection equipment capable of being effectively deployed in commercial aviation and formulated a plan to deploy this equipment in airports throughout the United States. In a separate program, the FAA has tested HULDs designed to contain a discrete explosive blast. Ten HULDs
1 The following terminology is used throughout this report. An explosives-detection system is a self-contained unit composed of one or more integrated devices that has passed the FAA's certification test. An explosives-detection device is an instrument that incorporates a single detection method to detect one or more explosive material categories. Explosives-detection equipment is any equipment, certified or otherwise, that can be used to detect explosives.
Page 2
|
have been deployed to three air carriers for operational testing.
The FAA's aviation security equipment and procedures include bulk2 explosives-detection equipment, TEDDs, HULDs, computer-aided passenger screening (CAPS), and positive passenger-bag matching (PPBM). These equipment and procedures are described in Table ES-1.
Findings
It is well documented (e.g., GAO 1998; DOT, 1998) that the FAA/SEIPT is behind schedule in the deployment of aviation security equipment. In 1997, Congress provided $144.2 million for the purchase of commercially available screening equipment, and the FAA/SEIPT planned to deploy 54 certified EDSs and 489 TEDDs by December 1997 (GAO, 1998). In addition, the FAA planned to implement CAPS fully by December 1997. Once it became apparent that these goals could not be met, the FAA set a new goal of deploying 54 certified EDSs, 22 noncertified bulk explosives-detection devices, and 489 TEDDs by December 31, 1998. The FAA also planned to implement CAPS fully by December 31, 1998. As of January 1, 1999, 71 certified EDSs, six noncertified bulk explosives-detection devices, and 366 TEDDs had been installed in airports, and CAPS and PPBM had been adopted by six airlines. In addition, 10 HULDs have been deployed to three airlines for operational testing.
The panel concluded that the combined efforts of the government, the airlines, and the airports to date have been effective in deploying aviation security technologies (improving aviation security to a level that will be quantified when additional data are collected during future studies), although, because of the urgent need for immediate action against incipient terrorism (White House Commission on Aviation Safety and Security, 1997), equipment and procedures were implemented rapidly without regard for how they would contribute to a total architecture for aviation security (TAAS). The panel believes that definition of such an architecture is essential to the success of this program; hence, it suggests formality in defining and using a TAAS. That is, although the capacity of individual pieces of equipment to discretely improve security at the point of deployment is known to some degree, the integrated effect of the total deployment of equipment and the implementation of procedures on the whole of aviation security is not. After much deliberation, the panel concluded that the performance of the TAAS could be measured by a single factor, the security enhancement factor (SEF), which will enable a quantitative evaluation of the performance of diverse deployment scenarios and show the importance of specific elements (e.g., explosives-detection equipment) to the performance of the TAAS.
Response to Congress
Protecting civil aviation against terrorist threats is a complex problem. Given the short response time and the complexity of the terrorist threat, the panel concluded that the research, development, and deployment by the FAA and others have been successful in qualitative terms. The urgent need for security equipment and procedures, expressed by the White House Commission on Aviation Security and Safety and by Congress in 1997, did not leave time for extensive system analyses. Therefore, the FAA proceeded with the deployment of hardware as it became available. Hence, the security system has evolved as the hardware has become available. It is not surprising, therefore, that data describing the efficacy of the deployed equipment are inadequate. The lack of performance data and the incomplete integration of the equipment into a complete security architecture are issues that any large system developer would be likely to encounter at this stage of development. The absence of a system architecture is the basis for the major recommendations of the panel. Nevertheless, the FAA will have to address these issues in the future.
Explosives-detection equipment and HULDs are part of a total system architecture and should be evaluated in the context of a TAAS. Although the FAA, its contractors, the airlines, and the airports have adopted some elements of the
2In this report, bulk explosives include all forms and configurations of an explosive at threat level (e.g., shaped explosives, sheet explosives, etc.).
Page 3
| ||||||||||||||||
total systems approach, in the panel's opinion they have not gone far enough. This study, and future aviation security studies conducted by the NRC, will be most useful to the FAA if they adopt the recommended comprehensive TAAS approach. Furthermore, adopting the TAAS approach will enable the FAA (and others) to characterize improvements in aviation security quantitatively using the SEF.
The panel has addressed (in part) the four points raised by Congress below. For clarity these points are listed again, followed by the relevant conclusions and recommendations.
1. Assess the weapons and explosives-detection technologies available at the time of the study that are capable of being effectively deployed in commercial aviation.
This study focused on explosives-detection technologies. While it is conceivable that some of these technologies could also be used for weapons detection, this topic was not addressed in this report.
Bulk Explosives-Detection Equipment
The vast majority of bulk explosives-detection equipment deployed is the FAA-certified In Vision CTX-series EDS (explosives-detection system). Most of the performance data on this equipment was generated during laboratory testing—largely certification testing—at the FAA Technical Center. Certification tests, however, only reflect the ability of the equipment to detect a bag that contains an explosive, and the detection rates are based on bag-alarm rates. That is, an explosive is considered to be detected if the alarm is set off for the bag containing the explosive, even if the alarm is triggered by a nonexplosive object in the bag. Certification testing does not measure alarm resolution and does not include testing in the operational environment of an airport, making it difficult to assess explosives-detection technologies for deployment. In the panel's opinion, some of the unanticipated problems encountered with the CTX-5000 SP in the field can be reasonably related to the limitations of certification testing. Under current certification guidelines, equipment certified in the future may encounter similar problems.
Recommendation
During certification testing, the FAA should, whenever possible, measure both true detection rates (i.e., correctly identifying where an explosive is when an alarm occurs), and false-detection rates (i.e., an alarm triggered by something other than an explosive in a bag that contains an explosive). The FAA should also include the ability of explosives-detection equipment to assist operators in resolving alarms (including in an airport) as part of
Page 4
certification testing. Alarm resolution should be included in the measurement of throughput rate, detection rate, and false-alarm rate.
Trace Explosives-Detection Devices
TEDDs are widely used in airports, but no comprehensive methodology has been developed to evaluate their effectiveness, such as standard test articles or instrument and operator requirements. Because no standard test articles for TEDDs have been demonstrated—and because of the resultant inability to separate instrument and operator performance—it is not possible to measure the performance of TEDDs.
Recommendation
The FAA should develop and implement a program to evaluate the effectiveness of deployed trace explosives-detection devices. This evaluation should include measurements of instrument and operator performance, including measurements in the deployed (i.e., airport) environment.
Computer-Assisted Passenger Screening and Positive Passenger-Bag Matching
CAPS appears to be an effective way to screen passengers to identify selectees who require further security measures, such as bag matching or bag screening. The panel anticipates that PPBM combined with CAPS will be an effective tool for improving aviation security. Despite the positive attributes of CAPS, the panel is concerned that the FAA has not demonstrated a measure for characterizing quantitatively the effectiveness of CAPS. A CAPS selectee could bypass PPBM by checking a bag at the gate or the door of the aircraft (as opposed to the ticket counter). Furthermore, PPBM has not been demonstrated to be effective when a selectee changes planes at a connecting airport. That is, passengers identified as selectees at originating airports (who are then subject to PPBM) are not subject to PPBM on subsequent connections of that flight. Another shortfall of PPBM is when a passenger checks a bag (or bags) at the gate.
Recommendation
Computer-assisted passenger screening (CAPS) should continue to be used as a means of identifying selectee passengers whose bags will be subject to positive passenger-bag matching (PPBM), screening by explosives-detection equipment, or both. PPBM combined with CAPS should be part of the five-year plan recommended below. Passengers designated as selectees at the origination of their flights should remain selectees on all connecting legs of their flights. Within six months, the FAA should develop and implement a method of testing the effectiveness of CAPS.
2. Determine how the technologies referred to in paragraph (1) could be used more effectively to promote and improve security at airport and aviation facilities and other secured areas.
Progress in the Deployment of Aviation Security Equipment
The panel concluded that the FAA/SEIPT, the airlines, airports, and associated contractors have gained significant experience from the initial deployment of security equipment and procedures, and the current implementation of security equipment does not appear to have interfered unreasonably with airline operations. Most importantly, in the collective opinion of the panel, the deployment of security equipment has improved aviation security. The panel believes that continued emphasis on, funding of, and deployment of security equipment will further enhance aviation security. Future deployments should be more efficient if they are based on the experience from the initial deployment.
Recommendation
The U.S. Congress should continue to fund and mandate the deployment of commercially available explosives-detection equipment through the FAA/SEIPT. Continued deployments will increase the coverage of domestic airports and eventually provide state-of-the-art security equipment systemwide. Further deployments can improve aviation security in the short term and provide the infrastructure for mitigating potential threats in the long term.
Operator Performance
Human operators are integral to the performance of all deployed explosives-detection equipment. Because fully automated explosives-detection equipment will not be developed in the foreseeable future, particularly with respect to alarm resolution, human operators will continue to be immensely important to realizing the full potential of deployed security hardware. The TAAS analysis presented in this report quantifies the impact of the operator on the SEF. Certification testing of explosives-detection equipment, however, does not include testing of human operators. Current testing only defines the operational capability (or performance) of the equipment.
Recommendation
The FAA should institute a program to qualify security-equipment operators to ensure that the human operator/explosives-detection system (EDS) combination meets the performance requirements of a certified EDS. This program should include the definition of operator performance standards and a means of monitoring operator
Page 5
performance. The FAA should implement this program within six months of receipt of this report.
Measuring Operational Performance
Because of the paucity of operational data for deployed explosives-detection equipment, the panel found it impracticable to characterize the deployment status of security equipment and processes quantitatively. The data are insufficient both for the equipment and for operator performance, and no quantitative measures of the effectiveness of the total security system (e.g., TAAS) were provided to the panel. The majority of data focused on subsystems, such as bulk explosives-detection systems. A thorough assessment of equipment and system performance requires well defined performance metrics and the collection of data. The panel concluded that the FAA has not defined adequate performance metrics for security subsystems (e.g., TEDDs) or for the TAAS.
Recommendation
The FAA should make a concerted effort to define operational performance metrics for security subsystems and for the total architecture for aviation security (TAAS). The FAA should also create an action team in the next six months to systematically collect operational data, which should be used to optimize the TAAS, as well as to identify and correct substandard performance of equipment and operators. The data collected would also provide insights into the deployment and use of equipment in the future.
Measuring Security Enhancement
Besides the dearth of operational data and total-system performance metrics, the FAA has not defined an overall measure of security enhancement. The primary performance measure for the TAAS is, of course, protection against the threat of explosives. Consequently, the panel believes the critical factor in assessing the performance of the TAAS is the measure of false negatives (i.e., unidentified bags that contain explosives). The panel defined improved performance (i.e., the SEF) as the ratio of the number of simulated bombs that defeat the baseline security system to the number of simulated bombs that defeat the newly deployed system.
Recommendation
The FAA should formulate a security enhancement factor (SEF) for the integrated total architecture for aviation security systems. The SEF should be calculated from data collected during operational testing. Nonclassified SEF measures should be published and used as a project-control and management-control tool. The SEF would provide the FAA with a quantitative measure of the impact of security equipment and procedures.
Five-Year Deployment Plan
Decisions based on systems of systems analysis (e.g., TAAS) involve both management and cost factors, which are airport and airline specific. Stakeholder3 involvement, therefore, will be crucial for the development of an effective deployment strategy. Furthermore, airline and airport buy-in will be critical to the successful implementation of the deployment strategy. The FAA did not provide the panel with a long-range (five-year) TAAS deployment plan developed jointly and agreed to by the FAA and other stakeholders. Thus, the panel concluded that the FAA has not obtained comprehensive airline buy-in for a long-term deployment plan that addresses all of the relevant issues, such as operator training, the optimal location of detection equipment, and the operational deployment of HULDs.
Recommendation
Within one year, in cooperation with the other stakeholders, the FAA should develop a five-year joint-deployment plan that includes cost, stakeholder responsibilities, quality measures, and other important factors. This plan should be a living document that is formally updated annually. Buy-in from all stakeholders will be necessary for the plan to be effective.
3. Assess the cost and advisability of requiring hardened cargo containers to enhance aviation security and reduce the required sensitivity of bomb-detection equipment.
Two HULDs (both LD-3 size) that conform to NAS-3610-2K2C airworthiness criterion have passed the FAA blast and shockholing4 tests. The LD-3 container is used only on wide-body aircraft, however. Thus, no HULD concept for narrow-body aircraft has passed the FAA test, although 75 percent of the aircraft in service (as of 1994) are narrow-body aircraft, and more than 70 percent of bombing attempts have been against narrow-body aircraft.
The panel's greatest concern is that research on HULDs has not been conducted on a system-of-systems (SOS) basis and has not involved all of the stakeholders, mainly the airlines. So far, HULDs have largely been developed and designed as single stand-alone entities. Limited research has
3 In this report the term stakeholder includes the FAA, the airlines, and the airports. Although there are certainly other stakeholders in aviation security, these three will have the most influence on the deployment strategy for aviation security equipment.
4 A shockholing (or fragmentation) test measures the ability of a HULD to prevent perforation of its walls by a metal fragment traveling at a relatively high velocity.
Page 6
been done on their role as part of a TAAS. Coordination with the airlines, airports, and aircraft manufacturers has been focused mainly on specific designs and utility requirements rather than on the interactions, boundary conditions, and trade-offs (including cost and operational considerations) of using HULDs along with other security measures, such as passenger profiling and baggage screening. The panel believes that alternative HULD designs may be more practical than existing designs in the TAAS context.
Recommendation
The FAA should continue to support research and development on hardened unit-loading devices (HULDs), including ongoing operational testing. If the FAA recommends, mandates, or regulates the use of HULDs, explosion-containment strategies for narrow-body aircraft, including the development of narrow-body HULDs and cargo-hold hardening concepts, should be investigated. However, the FAA should not deploy HULDs unless they are part of the TAAS joint five-year deployment plan.
4. On the basis of the assessments and determinations made under paragraphs (1), (2), and (3), identify the most promising technologies for improving the efficiency and cost effectiveness of weapons and explosives detection.
The data were not sufficient for a comprehensive assessment of available technologies for improving aviation security. Therefore, at this time the panel is not able to identify or recommend the most promising technologies for improving the efficiency and cost effectiveness of weapons and explosives detection. If the recommendations in this report are followed, these data will become available for subsequent assessments.
References
DOT (U.S. Department of Transportation). 1998. Aviation Security: Federal Aviation Administration. Washington, D.C.: U.S. Department of Transportation, Office of the Inspector General. Also available on line: http://www.dot.gov/oig/audits/av1998134.html
FBI (Federal Bureau of Investigation). 1995. Terrorism in the United States in 1995. Available on line at: http://www.fbi.gov/publish/terror/terrorin.htm
GAO (General Accounting Office). 1998. Aviation Security: Implementation of Recommendations Is Under Way, But Completion Will Take Several Years. GAO/RCED-98-102. Washington, D.C.: General Accounting Office. Also available on line at: http://www.gao.gov/AlndexFY98/abstracts/rc98102.htm
White House Commission on Aviation Safety and Security. 1997. Final Report to the President. Also available on line at: http://www.aviationcommission.dot.gov/
Zuckerman, M.B. 1996. Are order and liberty at odds? U.S. News and World Report 121(5): 64.
