Detection of Explosives for Commercial Aviation Security (1993) / Chapter Skim
Currently Skimming:
A: A GENERAL TESTING PROTOCOL FOR BULK EXPLOSIVE DETECTION SYSTEMS
A: A GENERAL TESTING PROTOCOL FOR BULK EXPLOSIVE DETECTION SYSTEMS
Pages 37-80
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 37... ...
APPENDIX A A GENERAL TESTING PROTOCOL FOR BULK EXPLOSIVE DETECTION SYSTEMS Developed in Consultation With The Committee on Commercial Aviation Security January 1993 37
|
From page 41... ...
This version of the protocol has been developed in consultation with the National Research Council's Committee on Commercial Aviation Security. The committee has differentiated between those testing aspects that pertain to certification and those that pertain to verification testing; added to the discussion regarding the use and composition of a standard set of baggage for certification testing; clearly recommends a FAA dedicated test site for certification testing; requires that the rationale for deviations from the protocol be documented; and provides for revisions to the protocol as additional testing experience is gained, added some discussion on testing with countermeasures, required analysis of false alarm data for verification testing, and required the documentation of deviations from the protocol.
|
From page 42... ...
This protocol addresses two types of testing: Pass/Fail testing, required for certification testing; and parametric testing, used to obtain statistically-valid verification performance data. Table Al summarizes the primary differences between these two types of tests.
|
From page 43... ...
Chapter AS deals with issues related to the conduct of the test, while Chapter A6 discusses the data analyses and evaluations of the test data. Finally, Annex I discusses how the FAA could establish a standard set of bags and explosives to conduct operational tests or certification tests at a dedicated FAA test site.
|
From page 45... ...
However, since this is a generic protocol for a variety of explosive detection devices or systems (using different technologies) , there may be some additional factors that may need to be considered, and the discussion that follows should not preclude additional factors from being included in the final test plan if those factors are considered to be relevant by the test team.
|
From page 46... ...
Bag selection is a crucial topic in designing a test plan that is fair and effective. For certification testing of an EDS, the testing should be performed on a standard set of bags to provide a fair and consistent comparison against the EDS Standard.
|
From page 47... ...
Similar considerations apply to EDD verification testing at the FAA test site, although a bag population representative of a specific airport may be used. For testing conducted at airports, available passenger bag populations could be used.
|
From page 48... ...
G A Standard Set Of Bags And Threats At the present time, the FAA has not developed a standard set of bags, standard suite of explosives, and a dedicated test site.
|
From page 49... ...
The following three sections discuss specific areas that must be addressed before the test team can develop the detailed test plan: identification of the set of distinct bag populations that must be used in the testing of the equipment; selection of the threat package to be used; and, specification of the procedures used to measure baggage processing rates. In addition, a short discussion of the issues involved with pre-testing is presented.
|
From page 50... ...
For each of these locations, the equipment manufacturer should be informed by the FAA what maximum fraction of false alarms will be acceptable, so that the equipment can be properly calibrated and the detection threshold established. For testing at an airport, the test team will determine who will process passenger bags from these populations to provide the observed measurements of the explosive characteristics for each processed passenger bag.
|
From page 51... ...
This requires that the test team know the bag characteristics, as measured by the equipment, for each of the populations. To this extent, some uncertainty will be introduced into the testing process, since the true bag populations are defined by the actual passenger bag populations and the test team is generating artificial populations.
|
From page 52... ...
The first opportunity to collect operational data is when the test team is obtaining data to select the final set of bag populations. During these tests, procedures should be established to ensure that meaningful temporal data is collected.
|
From page 53... ...
which might add to the processing times. To avoid these problems, the processing rate data should be generated with that testing associated with the false alarm rate estimation since almost all of the operational processing will take place with passenger bags (not containing explosives)
|
From page 55... ...
The FAA test director will determine if the plan is complete and suitable. - As appropriate, the acceptable minimum detection probability and maximum false alarm probability required to certify an EDS should be clearly identified to the test team.
|
From page 56... ...
Determining this relationship will require the testing of various bag populations at different detection thresholds. The means to adjust the detection threshold of the equipment must be available to the operators, and all such adjustments recorded.
|
From page 57... ...
- Select Representative Bags When Non-Passenger Bags Are Required. For each of the different populations, designated in Chapter As, a set of test bags must be available that reflects the multi-variate frequency distribution of the characteristic measurements of bags from that population.
|
From page 58... ...
This approach will provide estimates of the false alarm probability for the time period being observed, using actual passenger bags, and estimates of the detection probability either using the modified passenger bags or fabricated bags.
|
From page 59... ...
, the hypothesis testing method allows for the control of both Type I and Type II errors. For example, assume that the null hypothesis is: "the probability of detection, PD, of the tested device is equal to or greater than 90 percent." The al ternative hypothesis can be formulated as: 'I the probability of detection, PD' of the tested device is less than 90 percent." For a PD of 0.90, setting the Type I error at 5 percent, setting the Type II error at 5 percent and evaluating it at PD of 85 percent, the sample size required to discriminate between the two hypotheses is estimated to be 470 bags containing explosives.
|
From page 61... ...
o Selected the set of bags to be used in the testing and understand how these bags represent the population of bags being considered. o Determined the threat samples and countermeasures to be used during The following areas should be considered by the test team in the execution of the testing: o Independence from the Manufacturer Although the test team may be required to rely heavily on factory personnel for support in conducting the tests, procedures must be established to minimize the possibility that the contractor could influence the test results.
|
From page 62... ...
- At the end of each test day, print outs of all test data should be collected by the test team; a backup package should be available for the FAA, and this data retained for possible future use/confirmation - All available data should be collected, even if it will not be used immediately. At a minimum, threshold values and alarm/no-alarm readings on each item tested should be recorded.
|
From page 63... ...
63 - The equipment being tested will be used as set up by the manufacturer; i.e., there will be no non-routine changes or adjustments made during the evaluation testing. Annex III contains a model test scenario that could be used to guide actual testing
|
From page 65... ...
o The fraction of false alarms observed pita) for different bag populations.
|
From page 66... ...
. If any one of the above assumptions is not true, then these fractions must be computed for appropriate subsets of the test data and these fractions weighted to account for any differences (for example, if the bag populations or threats are not equally likely)
|
From page 67... ...
O Certification testing should be conducted at an FAA facility, using the above Standard Bag Set and the Standard Threat Package. These items, or a duplicate, should be made available to the vendor prior to the actual testing so that the contractor can determine using parametric tests, the thresholds of the system being offered for test.
|
From page 68... ...
If not, then threat packages should be developed which can be placed on the outside of the bags which are being tested for threat detection. The threat packages should also be barcoded and well marked.
|
From page 69... ...
Parametric testing should give the FAA significant insight as to whether or not the system has a reasonable possibility of passing certification testing. If at the end of the testing, the perceived likelihood of passing the certification test is low, the contractor should be told that no certification testing will be conducted until the FAA determines there is a reasonable possibility of success.
|
From page 71... ...
This should be used to reconfirm the bag selection process for validation testing and to help guide the test teem in the bag selection process for testing at an airport terminal. The test team must monitor all aspects of the validation testing, record data for each bag processed, verify the sequencing of bags through the detection equipment, place all explosives and simulants in the appropriate bags, and observe the verification tests of each of the samples of explosives used for comparison against the simulants.
|
From page 72... ...
The validated simulants should be suitably marked and immediately placed in the custody of the test team who will then deliver them to the test site at the appropriate time. If the simulant characteristics can change over time, the simulants should be revalidated after appropriate time periods.
|
From page 73... ...
Three statistical tests can be used to determine if these samples came from the same distribution, so as to justify the validation of the m-th simulant of the j-th type explosive.
|
From page 74... ...
An appropriate null hypothesis to test would be
|
From page 75... ...
as defined above, can be considered in an Analysis of Variance for each explosive type j. The Analysis of Variance approach provides a statistical test of the composite null hypothesis: H(09: E~D(i,j against the alternative, ,m)
|
From page 76... ...
One approach to this test is the use of the l-test. In using this test if the composite null hypothesis is not discarded all simulants of the j-th type explosive would be declared to be validated.
|
From page 77... ...
If in its initial use, none of the simulants are validated individual l-tests should be run and the simulant which differed most significantly from its null hypothesis would be declared to be invalidated and dropped from the group of simulants being tested. The Analysis of Variance approach would then be repeated using the reduced group.
|
From page 79... ...
If passenger bags cannot be used, then it is suggested that for each of the bag populations, six groups of twenty bags be selected as representative of that population. If it is not possible to "reproduce" the distribution because an adequate number of bags is not available, a statistical weighting process could be used to "match" the distribution.
|
From page 80... ...
The total number of observations on empty bags is 1080 while the number of observations on bags with threats is 1080, with each threat being tested in 6 different bags, and in 9 locations in each bag. Before running the six groups of twenty bags through the equipment, two control bags should be processed 10 times, one bag without a threat and the second bag contained one of the threats.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.