National Academies Press: OpenBook

Configuration Management and Performance Verification of Explosives-Detection Systems (1998)

Chapter: Appendix F: Test Protocol for Bulk Explosives-Detection Equipment

« Previous: Appendix E: Sample ISO 9000 Effort for a 50-Person Company
Suggested Citation:"Appendix F: Test Protocol for Bulk Explosives-Detection Equipment." National Research Council. 1998. Configuration Management and Performance Verification of Explosives-Detection Systems. Washington, DC: The National Academies Press. doi: 10.17226/6245.
×

Appendix F Test Protocol for Bulk Explosives-Detection Equipment

Daily Performance Verification

Equipment of the sophistication of bulk explosives-detection equipment will likely have diagnostic procedures designed by the manufacturer that are resident in the instrument. Typically these diagnostic procedures determine if certain parameters (voltage, signal, frequency, etc.) are within certain boundaries. In conjunction with test articles, these diagnostic procedures can test functional capabilities of the equipment. If the test articles and parameter values are defined and supplied by the vendor and are tested during certification and if performance-verification testing are shown to correlate with performance, they could be used as a daily calibration and check of explosives-detection equipment.

An example test is shown in Figure F-1, which is composed of secondary standard materials embedded in a matrix that simulates clothes, books, plastic, etc. The panel is not proposing that the explosives-detection equipment manufacturers or the FAA fabricate or use the test article exactly as shown in Figure F-1. Rather, the test article shown in Figure F-1 is a conceptual example that more closely represents the threat than does a test article that merely determines certain performance metrics of explosives-detection equipment. The main point of this conceptual example is that the development of a standard test article—one that is independent of the equipment being tested within a specific technology area (e.g., x-ray computed tomography [CT])—representative of the threat to aviation security is of critical importance. A collaborative effort between the FAA and explosives-detection equipment manufacturers is needed to develop a standard test article that includes materials representative of explosives and common nonexplosive materials (i.e., items typically found in passenger baggage). Any test article proposed as a standard (for a particular technology area) for performance verification of explosives-detection systems (EDSs) should be validated against the threat materials it is intended to represent using EDSs based on the same technology (e.g., x-ray CT) produced by at least two different manufacturers.

For the case of equipment based on CT (e.g., CTX-5000-SP) one CT slice would be taken at every level (as shown in the side view in Figure F-1) such that the following system performance parameters could be measured while testing the test article: spatial resolution, contrast sensitivity, noise level, or perhaps the probability of detection and the probability of false alarm. Daily testing of bulk explosives-detection equipment, as recommended by the panel, will involve using a test article similar to that described in Figure F-1 at each personnel shift change. Given the condition that baseline parameter values will be known, subsequent test results can be referenced to these values as a measure of system performance. This daily test could be automated such that the operator could simply put the test article in the machine, push a button, and be presented with a "go" or "no-go" message.

Comprehensive Performance Verification

The process recommended by the panel for comprehensive performance verification in the field requires that the FAA establish a testing approach (a protocol) and baseline performance measures that are used to verify the performance of each device or system at the manufacturing site or at an airport. To date only one manufacturer has produced an FAA-certified EDS (i.e., InVision). The EDS is a bulk explosives-detection system (as opposed to a trace-type system). For this reason, the panel has focused on approaches for performance verification of FAA-certified bulk EDS. Much of what follows, however, could be applied to performance verification of noncertified bulk explosives-detection equipment.

For comprehensive performance verification of deployed bulk equipment the panel recommends using a secondary standard bag set to obtain estimates of the probability of detection

Suggested Citation:"Appendix F: Test Protocol for Bulk Explosives-Detection Equipment." National Research Council. 1998. Configuration Management and Performance Verification of Explosives-Detection Systems. Washington, DC: The National Academies Press. doi: 10.17226/6245.
×

Figure F-1

Example standard test article for daily performance  verification of bulk explosives-detection equipment. 

Note: In this test article, N different simulated  explosives are surrounded by different background  materials that simulate common materials found in  luggage (e.g., food, clothes, plastic, aluminum, etc.).

(PD) and the probability of false alarm (PFA) in addition to the PD and PFA values obtained using the primary standard bag set used in the certification process. This secondary-standard bag set is then used to test and obtain estimates of the probabilities (PD, PFA) for the deployed explosives-detection equipment. If statistically significant agreement exists between the test with the secondary-standard bag set at the FAA's Technical Center (at the time of certification) and the one at the airport, then the performance of the explosives-detection equipment is said to be verified.

Secondary Standard Bag Set

A secondary standard bag set should be developed by the FAA to consist of a number of representative international passenger bags (that do not contain explosives or simulated explosives) and a number of representative international passenger bags that do contain simulated 1 explosives. These simulated explosives should be validated by the FAA (see Annex II, NRC, 1993) and mimic the real explosives used in the primary standard bag set. This secondary standard bag set should be developed and controlled by the FAA personnel responsible for conducting the certification test and utilized in the conduct of all comprehensive verification testing. All data generated by the use of this bag set should be collected, analyzed, reported, and maintained by the FAA personnel.

As the final phase of certification testing, the FAA should test explosives-detection equipment against the secondary standard bag set to obtain estimates of PD and PFA. The bag set and the data collected would be retained by the FAA as the baseline performance-verification data base. The FAA should make this secondary standard bag set large enough to yield statistically meaningful data, yet small enough to be manageable in terms of transporting the bags from the FAA Technical Center to the various sites. Furthermore, it is desirable to be able to conduct the test in a reasonable amount of time so as not to interfere with routine airport and airline operations.

For example, a secondary standard bag set that is 20 percent of the size of the primary standard bag set (which totals some 2,150 bags) would result in a secondary standard bag set consisting of 430 bags. This would yield estimates of PD and PFA with a standard error 51/2 (2.236) times larger than the error in estimated PD and PFA obtained using the primary standard bag set. In general, if the secondary standard bag set is 1/N of the size of the primary standard bag set, the standard error of the estimate will be N1/2 times larger.

1  

The panel believes that the FAA, in cooperation with the EDS manufacturers, should develop such simulants for the various technologies being considered or used for bulk EDSs. These simulants should be made available to the developers of the EDS so that they can be used in the process of early determination of the detection capabilities of the technology.

Suggested Citation:"Appendix F: Test Protocol for Bulk Explosives-Detection Equipment." National Research Council. 1998. Configuration Management and Performance Verification of Explosives-Detection Systems. Washington, DC: The National Academies Press. doi: 10.17226/6245.
×
Statistical Analysis

After the physical testing has been completed (i.e., the secondary standard bag set has been tested by the explosives-detection equipment), the following statistical tests should be performed:

  • Test that the performance probabilities for the to-be-deployed explosives-detection equipment are statistically equal to or better than the baseline values of the equipment.
  • Test that a "significant" number of the bags in the secondary standard bag set are correctly classified by both the explosives-detection equipment that underwent certification testing and the explosives-detection equipment being tested in the field.

The former test will be used in the following example. Testing the hypothesis that the false-alarm rate of the explosives-detection equipment to be deployed is less than or equal to that of the equipment certified by the FAA (with M bags in the secondary standard bag set used for estimating PFA- pFA) with an error of the first kind of 5 percent, the value of the Y statistic is given by the following equation:

where PFA is the false-alarm rate observed at certification and pFA is the false-alarm rate observed on the to-be-deployed equipment. If PFA- pFA is greater than or equal to Y, then one would have no reason to reject the hypothesis that the "population" value of the false-alarm rate of the to-be-deployed equipment is less than or equal to the "population" value of the false-alarm rate of the equipment at certification. This statement is made at the 95 percent level of confidence. As a guide to determining the size of the secondary standard bag set, M must be sufficiently large to allow one to assume normality in the distribution of PFA- pFA.

Conducting a similar test using the simulated explosive threat bags, if it can be determined that there is no reason to reject the hypothesis that the population value of the detection rate of the to-be-deployed EDS is greater than or equal to the population value of the detection rate of the EDS at certification (again at the 95 percent level of confidence), then the performance of the to-be-deployed EDS is verified.

Although the above refers to the to-be-deployed equipment, the same approach can be used to verify that the performance of deployed equipment is still at the level (or better) as when the equipment was certified at the FAA.

Reference

NRC (National Research Council). 1993. Detection of Explosives for Commercial Aviation Security. National Materials Advisory Board. Washington, D.C.: National Academy Press.

Suggested Citation:"Appendix F: Test Protocol for Bulk Explosives-Detection Equipment." National Research Council. 1998. Configuration Management and Performance Verification of Explosives-Detection Systems. Washington, DC: The National Academies Press. doi: 10.17226/6245.
×
Page 64
Suggested Citation:"Appendix F: Test Protocol for Bulk Explosives-Detection Equipment." National Research Council. 1998. Configuration Management and Performance Verification of Explosives-Detection Systems. Washington, DC: The National Academies Press. doi: 10.17226/6245.
×
Page 65
Suggested Citation:"Appendix F: Test Protocol for Bulk Explosives-Detection Equipment." National Research Council. 1998. Configuration Management and Performance Verification of Explosives-Detection Systems. Washington, DC: The National Academies Press. doi: 10.17226/6245.
×
Page 66
Next: Appendix G: Biographical Sketches of Panel Members »
Configuration Management and Performance Verification of Explosives-Detection Systems Get This Book
×
Buy Paperback | $47.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This report assesses the configuration-management and performance-verification options for the development and regulation of commercially available Explosive Detection Systems (EDS) and other systems designed for detection of explosives. In particular, the panel authoring this report (1) assessed the advantages and disadvantages of methods used for configuration management and performance verification relative to the FAA's needs for explosives-detection equipment regulation, (2) outlined a "quality management program" that the FAA can follow that includes configuration management and performance verification and that will encourage commercial development and improvement of explosives-detection equipment while ensuring that such systems are manufactured to meet FAA certification requirements, and (3) outlined a performance-verification strategy that the FAA can follow to ensure that EDSs continue to perform at certification specifications in the airport environment.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!