Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 89
Ballistic Imaging PART II Current Ballistic Imaging and Databases
OCR for page 90
Ballistic Imaging This page intentionally left blank.
OCR for page 91
Ballistic Imaging 4 Current Ballistic Imaging Technology “It has been common practice for firearms examiners to maintain an ‘open-case file’ of physical evidence from unsolved crimes, sorted by caliber,” Thompson et al. (2002:8) note of the traditional approach to generating investigative leads through firearms identification. “When faced with a crime on which little evidence was available, the examiner would then go to the storage area for evidence from unsolved cases and choose some potentially similar cases for examination of the originals.” This process can be extremely time consuming—not only the direct examination of evidence, but also the steps of retrieval, filing, and reporting. “Because of the time required for the manual comparison of evidence, the effectiveness of this method can be severely limited by the staffing and workload of an agency’s examiners (which determines how much time examiners have to search the open-case file).” In this chapter, we briefly review the background of imaging technology in firearms identification (Section 4–A), the basic structure of Integrated Ballistics Identification System (IBIS) equipment (4–B), and the manner in which the IBIS equipment is used to acquire images (4–C). Section 4–D discusses what is publicly known about IBIS procedures for scoring, ranking, and analysis, crucial to assessing the technical capability of this technical platform to “scale up” to meet the demands of a much larger database. Section 4–E reviews the major studies that have been conducted to date on IBIS performance, particularly with large-scale databases or datasets consisting of test fires from new weapons. Section 4–F presents basic assessments of the current technology (specific recommendations related to IBIS usage are in Chapter 6). An appendix to the chapter, Section 4–G, summarizes and
OCR for page 92
Ballistic Imaging elaborates on technical evaluation tests performed on IBIS by the state of California. Because it is important to consider IBIS and the National Integrated Ballistic Information Network (NIBIN) together, a summary and our conclusions on the evidence in this chapter are in Chapter 6, together with those from Chapter 5. 4–A BACKGROUND Contemporary ballistic imaging technology is the latest step in a gradual move over several decades to use technology to make it easier to maintain and search open case files of ballistics evidence, including cases distant in time. During the 1970s, calls were made to develop automated index systems to assist examiners in search, as well as to explore new directions for the imaging of ballistics evidence. Biasotti (1970:12) made an early call for a computer-based open case file that would permit examiners to describe observed class characteristics “for all rifled weapons [and] unidentified bullets and cartridge cases” in a central repository. However, in this early vision, imaging was not considered; instead, characteristics were to be expressed using an alphanumeric string (e.g., FW105-100-1357-20-0102-001-001), coding such factors as the measured caliber and land widths of bullet evidence. When new evidence arrived, a query on the database could then determine whether cases with similar class characteristics or modi operandi were on file. On the technical side, other researchers suggested the utility of more high-powered microscopy techniques for the comparison of ballistics evidence, including several papers arguing for the use of scanning electron microscopy (Gardner, 1979; Goebel et al., 1980; Grove et al., 1972).1 Grove et al. (1972:20) considered scanning electron microscopy “ideally suited for firing pin impression examination because of its ability to reveal topographical features at the base of the impression.” The researchers examined “series of up to 50 rounds” from “numerous .32 caliber semi-automatic pistols,” analyzing the first, second, tenth, “and in some instances the fiftieth firing pin impression.” “In all the firing pin impressions examined, a match could be made using a criteria of 4 or more points of identification” whereas “no points of identification” could be found for firings from different guns; moreover, they concluded that “the first and fiftieth impressions can be matched.” 1 As summarized by Grove et al. (1972:20), scanning electron microscopy “consists basically of a finely focused beam of electrons which sweeps over the sample surface. This primary electron beam causes the formation of low energy electrons (secondary electrons) due to interaction with the sample surface. These secondary electrons are then collected and displayed on a cathode ray oscilloscope producing an image that gives extremely good topographical information with great depth of field.”
OCR for page 93
Ballistic Imaging In the 1960s, the Los Angeles Police Department (LAPD) developed a “Balliscan” camera designed specifically to photograph the exterior surface of a bullet, using a rotated slit to expose the film as a drum turned the bullet at the same speed.2 Blackwell and Framan (1980) suggested an Automated Firearms Identification System for the analysis of bullet evidence, based on the consecutive matching striations methodology of Biasotti (1959) and utilizing scanned versions of Balliscan images as the image data. Though they sketched a schematic diagram for such a system and did some preliminary analysis of bullets used in the Biasotti (1959) study, no apparent further action on developing the system was taken. In 1989 the Federal Bureau of Investigation (FBI) announced a program called DRUGFIRE, which used a system for acquiring images from cartridge evidence. A few years later, the Bureau of Alcohol, Tobacco, Firearms, and Explosives (ATF) adopted the BULLETPROOF system for imaging bullet evidence, marketed by what is now Forensic Technology WAI, Inc. (FTI), of Montréal, Canada, as the basis for its CEASEFIRE network. As described in more detail in Chapter 5, the two databases operated in parallel for several years until CEASEFIRE evolved into the NIBIN program, using as its platform the IBIS formed by combining BULLETPROOF with a BRASSCATCHER apparatus for imaging cartridge casings. IBIS was made the technical base for the new NIBIN database, and the major ballistic image databases in operation today (including NIBIN and the state reference ballistic image databases in Maryland and New York) use IBIS. IBIS is also in use by law enforcement agencies in several foreign countries; through IBIS, FTI is essentially the only provider of ballistic imaging technology. At root, the IBIS platform combines a microscope with a camera that acquires two-dimensional greyscale images of bullet and cartridge case evidence; features of the traditional comparison microscope can then be emulated using the images, and the images can be compared with each other to assess similarity. Box 4-1 makes an important note about current usage of the term “IBIS.” 4–B IBIS EQUIPMENT Formally, IBIS represents the integration of two separate systems. The BULLETPROOF microscope and comparison apparatus for acquiring images from bullets was developed first, beginning in 1991. It was aug- 2 A subsidiary of McDonnell-Douglas, Corp., later produced the Balliscan camera based on the LAPD design (Blackwell and Framan, 1980). Balliscan images became prominent in later years because images made following the assassination of Robert F. Kennedy were reexamined by firearms examiners in the mid-1970s, in support of the work of the U.S. House Select Committee on Assassinations.
OCR for page 94
Ballistic Imaging BOX 4-1 “IBIS” Terminology As of January 2007, Forensic Technology WAI, Inc. (FTI), repositioned its line of products to emphasize its existing BulletTRAX-3D platform and developing BrassTRAX-3D platform for the acquisition of three-dimensional measurements from bullets and cartridge cases, respectively. Both of these are said to constitute the “IBIS-TRAX 3D” line, and as such has begun referring to these as IBIS (e.g., they formally refer to the product as “IBIS BulletTRAX-3D”). The IBIS described in this chapter—based on two-dimensional photography—has now been designated the “IBIS Heritage Series” on the firm’s Web site (http://www.fti-ibis.com), and FTI suggests that the two-dimensional product is no longer actively marketed. Though the name has now been linked with the new three-dimensional products, we use the term “IBIS” throughout this report to refer exclusively to the two-dimensional photography system, dating from the combination of the separate BRASSCATCHER and BULLETPROOF components and running through version 3.4 of the IBIS software. We do so because of the context of our study, which includes offering advice on the existing National Integrated Ballistic Information Network (NIBIN) and suggesting enhancements to it: the entire infrastructure of NIBIN is built on the two-dimensional photography IBIS. What is now dubbed the “IBIS Heritage Line” is in fact the current platform deployed to NIBIN partners; accordingly, it is the appropriate benchmark of comparison for our study. Likewise, the experimental research conducted by the National Institute of Standards and Technology (NIST) in support of the committee’s work—described in Chapter 8—compared the current IBIS two-dimensional to a prototype three-dimensional acquisition system. This is because we consider three-dimensional topographic measurement as a possible enhancement within the current NIBIN system, so that it is appropriate to get a sense of how well the three-dimensional measurements and scores compare with the IBIS two-dimensional currently used in NIBIN. mented in 1995 by BRASSCATCHER, which adapted the apparatus to work with cartridge case evidence (McLean, 1999). Most of the IBIS installations under the NIBIN program take the form of Remote Data Acquisition Stations (RDASs). One component of an RDAS is the Data Acquisition Station (DAS), a microscope with two built-in cameras mounted to it (one for bullets and one for cartridge cases). The RDAS also includes a computer so that demographic data3 associated 3 These auxiliary data might more accurately be described as metadata, but we retain “demographic data” as common usage in the field.
OCR for page 95
Ballistic Imaging with a case (e.g., gun caliber, date of crime, and firing pin shape) can be entered by an operator; the microscope cameras display their output on the computer monitor, so that the operator can determine how the image will be acquired, as described below. In an RDAS, the computer also serves as a Signature Analysis Station (SAS), where the results of comparisons with other images can be reviewed. However, the key component that an RDAS lacks is the “correlation server” that processes results from acquired images and compares them against other cases in a database. An RDAS alone must transmit its images to a correlation server for processing and await the results from the server. Standalone systems that include a correlation server along with the base RDAS equipment are referred to as hubs. As discussed in Chapter 5, the NIBIN program also makes use of three other related FTI products in addition to the base IBIS RDAS. As it is currently structured, all comparisons of images are routed through correlation servers (separate from an IBIS hub) located in ATF’s three national laboratories. To ease the task of reviewing results from image comparisons, FTI also markets Matchpoint systems—essentially, the computer hardware and software of a SAS, except that they are not built into the same physical cabinet as the DAS in an RDAS. Finally, several NIBIN sites make use of Rapid Brass Identification (RBI) units, portable suitcase-size microscope setups that allow technicians to acquire breech face and firing pin images in the field, including at crime scenes. RBI units are meant only for acquisition of images (and transmittal, through an RDAS, to a correlation server), and not the result of image comparisons. 4–C DATA ACQUISITION The obvious first step in working with IBIS (and NIBIN, using the IBIS platform) is to have bullet or cartridge case evidence to enter into the system. This evidence may be bullets or casings recovered at crime scenes, or it may be test firings from weapons obtained by the police in the course of investigations. In the first case, casings and (particularly) bullets present challenges because they may be damaged and may require cleaning prior to examination and entry.4 In the case of test firings, the ammunition used in the firings—typically done into a water tank, to facilitate capture of the undamaged bullet—is a critical choice. To the greatest extent possible, 4 Rector (2002) considers the effect of one cleaning process on IBIS performance for matching bullet evidence. An ultrasonic bath—in which high-frequency sound waves produce vapor bubbles in a liquid—can be used to dislodge some foreign materials that can prove stubborn to conventional means, including soil and drywall. However, Rector (2002) observed that immersion in an ultrasonic cleaner for longer periods of time (up to 30 minutes) generally reduced IBIS scores and that the surface etching done by the cleaner was directly visible on lead bullets.
OCR for page 96
Ballistic Imaging examiners prefer to match the test fire ammunition to the ammunition used in crimes involving a suspected gun; in order of suitability, from most to least, De Kinder (2002b:9) characterizes the typical preference hierarchy for test firing: ammunition from the same lot as the recovered bullets and casings; ammunition of the same brand and make as the recovered bullets and casings; ammunition from the same manufacturer as the recovered bullets and cartridge cases, having the same primer or bullet jacket composition but not necessarily being exactly the same type; and ammunition having the same primer or bullet jacket composition, but not necessarily being from the same manufacturer. Often, however, no such information is possible—and in the context of creating an reference ballistic image database (RBID), it can never be known what type or lot of ammunition will be used with a new firearm. To address ambiguous cases, ATF recommends certain “protocol ammunition” for particular calibers to its NIBIN agencies in the hopes of “[giving] the best chance overall for [test-fire] items to find matching evidence bullets and casings in a database.” The protocol ammunition is chosen to be “intermediate in recording toolmarks and impression hardness,” having bullet metal and primer surfaces that are neither too hard nor too soft for registration of marks (Thompson et al., 2002:15). 4–C.1 Mounting of Evidence and Demographic Data Entry To begin an IBIS entry, operators open a “case,” which can contain one or more constituent “exhibits,” bullets or cartridge casings. A case can also include information about a firearm, if it has been recovered. Links suggested by IBIS in comparing exhibits are made between exhibits and not cases as a whole. Although the case identification number is displayed in a column when comparison results are returned for analysis, the system does not make it readily apparent where individual exhibits from a particular case fall in the list of rankings reported by IBIS. IBIS training materials emphasize the importance of correct entry of auxiliary, context data about evidence and exhibits, the “demographic data.” For cartridge case markings, the training guide indicates that “automatic correlation requests use all of the following demographic information”—occurrence date, caliber, firing pin shape, and event type—“to select the test candidates from the database,” and all these pieces of information are described as “crucial for the correlation process” (Forensic Technology WAI, Inc., 2002a:2-10, 3-2). IBIS defines six basic event types, four for
OCR for page 97
Ballistic Imaging exhibits related to crime and two for test firings. The crime-related event codes are homicide (HOM), assaults with a deadly weapon (ADW), other crime (OTH), and unknown (UNK). The distinction between the two test fire events is whether the firearm is retained by police (and hence is out of circulation on the street) or whether it is returned to the owner after firing; these are coded as TF and TFR, respectively. (The basic manner in which the demographic data are used as filters is described in Section 4–D.1.) Bullet exhibits are linked to operator-entered information on certain general rifling characteristics that can be derived from the bullet and can narrow down the database search. These include caliber, twist (the orientation of the land and groove impressions, left or right, when looking from the base of the bullet), and the number of lands and grooves on the bullet. The composition and type (e.g., jacketed or hollow point) of the bullet may also be recorded. Although accurate demographic data entry is essential to the IBIS comparison process, the physical positioning of bullet or cartridge evidence under the microscope (and camera) is crucial to the acquisition of quality, comparable images. Indeed, Tontarski and Thompson (1998:644) observe that “the greatest initial concern using this technology was whether or not different examiners could enter projectile and cartridge casing images in a sufficiently consistent way for the database to be able to locate a match.” Though they go on to assert that “the equipment’s image capturing system and its robust algorithm have all but eliminated operator variability as a concern,” proper positioning of exhibits is still emphasized in IBIS training, and some studies (e.g., Chan, 2000) suggest that substantial misalignment can still cause problems in comparison. In its documentation, FTI suggests standardized protocols for orienting evidence that have also been adopted as standards by the NIBIN program. For instance, a cartridge bearing roughly horizontal breech face marks across the primer surface is supposed to be oriented so that the marks are as flat (not at an angle) as possible, rotated so that the ejector mark on the cartridge rim is in the southern hemisphere of the image. If the cartridge shows evidence of a firing pin drag mark, where the pin has scraped against the surface, the cartridge is supposed to be rotated so that the drag mark is at or around the 3 o’clock position. 4–C.2 Specification of Regions of Interest IBIS allows technicians to designate regions of interest on an image. Because these regions are circular for the markings left on cartridge casings, the regions of interest are also known as ring limits. For a breech face impression, the region of interest is indicated based on two circles. The computer derives an automatic, “default” placement of the rings, but
OCR for page 98
Ballistic Imaging they can be adjusted directly by the operator. The outer (blue) circle is to be set to the edge of the primer surface of the stamp and the inner (red) circle marks off the firing pin impact region; the image used in comparing breech face marks is based on the doughnut-shaped area between the two circles. Though marks on the cartridge case area may be irregularly shaped—the pit of the firing pin impression and areas where the primer metal has been pushed back out of the firing pin impression—the region of interest rings are strictly circular. Hence, the IBIS operator must make some judgment about exact placement of the circle, assessing the potential for “washout” areas (reflected light off of the jagged edge of the pit of the firing pin impression) to show up in the final image. Operators may also adjust procedures to accommodate specific firing pin types; for example, Glock firearms have a distinctive rectangular firing pin, and therefore technicians place the inner circle so that it circumscribes the four corners of the impression. Figure 4-1(a) shows an IBIS breech face image with the two circular delimiters superimposed. Once the regions of interest are set for acquiring a breech face, the image is taken using the IBIS standard ring lighting, intended to provide uniform illumination, and the system automatically suggests a lighting intensity “to provide optimum lighting for acquisition.” However, the IBIS training materials note (Forensic Technology WAI, Inc., 2002a:2-18): In numerous cases the suggested lighting may not appear optimal (for example, with smooth surfaces or uncommon metal primer compositions). In these cases, you will need to manually adjust the light setting with the light scroll bar in order to minimize washout. Eliminating the washed out (white halo) area surrounding the firing pin impression improves correlation accuracy as this area is sometimes a common feature between cartridge cases. This will increase score results on marks of lesser value. Always keep in mind that your goal is to find the lighting intensity that will provide the best contrast with the least washout. After acquiring the breech face image using the center light, the user has the option of taking a second picture using alternate lighting, a side light located at the 6 o’clock position relative to the mounted cartridge, while holding the cartridge fixed in the same orientation. Figure 4-1(b) illustrates a side light image of a cartridge breech face impression, side by side with the standard center light image, Figure 4-1(a). The side light image is better for seeing some impression of three-dimensional detail, though it necessarily also casts shadows on other parts of the image. If the side light image is acquired, it is filed with the case and remains available for viewing later on (including the “Multiviewer” interface for viewing multiple exhibits simultaneously, as when reviewing comparison scores). However, the side light
OCR for page 99
Ballistic Imaging FIGURE 4-1 IBIS breech face images. NOTES: The three images are (a) breech face image using the standard ring, center light; (b) breech face image using the side light; and (c) firing pin image using the standard ring, center light, acquired from the same cartridge casing. Although they are difficult to see in this reproduction, circular region-of-interest delimiters are indicated on images (a) and (c). The area between the outer circle and inner circle (a) defines the breech face impression, and the area inside the single circle (c) defines the firing pin impression.
OCR for page 122
Ballistic Imaging cleaned with acetone prior to firing to remove the lacquer. Two lacquer-coated and two lacquer-stripped casings fired from each of three different guns were entered into the New York City Police Department’s IBIS and scores were generated; at the time, the number of 9mm Luger, circular firing pin exhibits (the base set for this comparison) in the New York system was estimated at 5,700 images. Generally, guns known to produce clear characteristic breech face marks performed consistently regardless of the presence of lacquer, which is to say that pairs of lacquer-coated exhibits from the same gun were returned in the top ranks as were pairs of lacquer-stripped exhibits; guns known to produce fainter breech face marks produced lower-ranked matches, yet still generally in the top 10. However, matching lacquer-coated to lacquer-stripped exhibits from the same gun proved more problematic, apparently failing to clear the coarse correlation and 20 percent threshold steps for guns with weaker propensity to generate breech face marks (score reported as 0 and rank as “none”; Hayes et al., 2004:Table 1). The IBIS function for comparing bullet evidence plays a prominent role in a multi-part examination of criteria for identifying bullet matches, and in particular standards for the number of groups of consecutive matching striations that can be said to define a match (Miller and McLean, 1998; Miller, 2000, 2004; see also Miller, 2001). The committee’s own experimentation, conducted by NIST under a separate contract with the National Institute of Justice, involved reanalysis of some of the De Kinder et al. (2004) cartridge casings as well as construction of a new 144-exhibit set of test-fired casings, varying ammunition brand and gun manufacturer. These casings were processed using both IBIS and three-dimensional metrology techniques, and were also run through IBIS waiving the coarse comparison and 20 percent threshold steps. We also performed limited IBIS experimentation using the New York CoBIS RBID and the independent IBIS database of the New York Police Department. We discuss the full details in Chapter 8; in brief summary, our own investigation corroborated the major findings of the predecessor studies described in this chapter. 4–F ASSESSMENT The committee was charged to offer advice on the options of maintaining the current NIBIN program (limited to crime gun evidence) or enhancing it, and since NIBIN uses IBIS as its technical base, the evaluation of one requires evaluation of the other. Yet focusing too much on assessment of current IBIS is also somewhat unfair in light of the charge to our committee to evaluate the feasibility of a national RBID. As De Kinder et al. (2004:208) note, “currently, no technology has been perfected to deal spe-
OCR for page 123
Ballistic Imaging cifically with very large databases of images of marks made by firearms.” IBIS was developed to deal with smaller, regional “open case files” of images, and it is unreasonable to expect that the full system used to implement a national RBID would follow exactly the same lines as the current IBIS platform. However, an RBID system—perhaps streamlining the image acquisition process, allowing for mass entry of exhibits, and continuing to refine comparison procedures—would likely be based on IBIS, if only to maintain compatibility with NIBIN data. As noted above, a subgroup of our committee discussed the IBIS comparison algorithm in detail with FTI staff under a confidential agreement. It is our judgment that the algorithm is generally quite sound, novel, and appropriate to the task of comparing images of ballistics evidence. Based on the era in which it was developed, IBIS is a valuable system that is fundamentally a vast improvement over relying on either human memory or the posting of Polaroids on the forensic laboratory bulletin board for deriving matches to evidence in open case files. Properly used—as we describe in Chapter 6—we believe that IBIS provides an adequate investigative tool for local and regional searches of ballistics evidence images. However, as we explain in fuller detail in Chapter 8, the review of past studies of IBIS performance and our own experimental work suggest that IBIS does not operate at the precision needed for a national RBID. In its structure and implementation, the IBIS platform is a computerized version of the comparison microscope. This is beneficial in certain respects, in that it provides a familiar (albeit not exactly identical) interface for firearms examiners to review image data. Yet it is also, fundamentally, a limitation of the technology. Since its origins in the early 1990s, the progress in developing the existing IBIS platform for ballistic imaging has been evolutionary rather than revolutionary, in that it has remained anchored to the premise of emulating the functions of a comparison microscope. Direct pairwise comparisons of exhibits remain the heart of the process; IBIS was not designed to perform as a true image “search engine,” indexing and comparing across large sets of images, as would be desirable in a national RBID implementation. In its form and function, IBIS functions as a quick sorting and ranking mechanism: a tool for search, but not verification. There is great value in the sorting that is performed with relative ease and speed by IBIS. However, major problems arise when higher expectations are placed on the system than it was designed to accommodate. Users and policy makers bear a large part of the responsibility for “overselling” the system; it is unrealistic to expect “hits” on every database search, as effective use of the system depends as much or more on the timely entry of evidence into the system as on the ability of the system to detect a possible match. The system is also ill-served by the expectations of instantaneous and utterly definitive verification of
OCR for page 124
Ballistic Imaging evidence matches created by portrayals in popular media; Box 4-2 presents an example. Overly high expectations and inaccurate portrayals have the unfortunate consequence of fueling the perception of ballistic imaging technology as a test—a source of verification—rather than a search tool. Most recently, this perception arose in litigation in Illinois (People v. Pursley, 341 Ill. App. 3d 230; 2003 Ill. App. LEXIS 784, 2003). In 2000, in light of exonerations due to DNA evidence, Illinois code was amended to give convicts the right to make a “motion for fingerprint or forensic testing not available at trial regarding actual innocence”—that is, to permit appeals for DNA testing. Invoking this provision, a man convicted in 1993 of first degree murder (and sentenced to life), largely on the basis of firearms identification evidence, “filed a motion … seeking an order requiring that his handgun be tested under the Integrated Ballistics Identification System (IBIS).” The appeals court ruled against the convict’s motion for IBIS “testing,” holding that the relevant statute was intended only to apply to fingerprint and DNA testing. Nowhere in the ruling (or, presumably, the motion) is it indicated what a “test under IBIS” might entail, how a comparison score might be interpreted, or against what database images should be searched. Only once (summarizing the state’s motion to dismiss the convict’s appeal) is it noted that “IBIS is not a new test but a new system for cataloging for ballistics information” and that “application of the IBIS would not produce new, noncumulative evidence.”7 Following the Pursley decision, Carso (2007) argued that the Illinois statute should be amended to include “ballistics testing” using IBIS but also does not describe what such a test would involve. IBIS developers and proponents also bear responsibility for “over- 7 Judge Gertner’s ruling in United States v. Green (405 F.Supp. 2d 104; 2005 U.S. Dist. LEXIS 34273), described in Box 3-4, is also of interest because IBIS was used in the course of the investigation. It suggests that some basic concepts of IBIS scope and operation can be misconstrued. Section G of the ruling notes: [The sergeant/examiner] also used the Integratable [sic] Ballistic [sic] Identification System (IBIS) in his comparison, although the government represented that it would not offer IBIS results [as testimony]. A national computer database, IBIS allows examiners to identify the most likely matches for the evidence in a given case. IBIS uses a laser measuring device to evaluate shell casings and provides the examiner with a list of possible matches…. In fact, the IBIS system has been widely criticized. Its efficacy is limited by the detail with which police departments have scanned old shell casings into the computer and the accuracy of the mathematical algorithms used to compare casings. As with the individual examinations, no evidence was presented about the accuracy of the IBIS matches…. In any event, [the sergeant] acknowledged that even if the computer suggests numerous possible matches, he will not bother to check them all. That is, once he decides he has found a match, he will not eliminate all other alternatives by exhausting the IBIS-generated list of potential matches.
OCR for page 125
Ballistic Imaging BOX 4-2 CSI Ballistic Imaging Firearms identification concepts and the use of ballistic imaging have periodically been referenced on forensic science-themed television shows. One such example is episode 307 (“Fight Night”) of CSI: Crime Scene Investigation; this particular episode won an Emmy award for best writing. One scene finds a Las Vegas investigator talking with a firearms examiner who is peering intently into the microscope of what—externally—is a complete IBIS RDAS unit. The investigator asks, “Three guns found at the crime scene, none match the bullets recovered from the victim. What does that tell us?” “Shooter kept his weapon,” the examiner replies. “Means he likes his gun, and may have used it before,” says the investigator, as some part of the machinery makes a loud whirring noise. “Which is where the shell case and IBIS come in,” says the examiner cheerfully. “I’ll run it against the national database.” He wheels from the microscope to the keyboard and, off-camera, types a short sequence of characters. “Firing pin impressions and breech face marks—a closer look,” muses the investigator; instantaneously, the system makes a loud shuffling sound and several beeps. The camera now shows the “IBIS” screen, which prominently shows a single image of the entire base of a cartridge, headstamp and all; some text indicating “Halo On” and “Magnification 150X,” among other things, is superimposed over the corner of the image. Beside it is a four-column listing of “Case ID,” “Exhibit Number,” “Site Number,” and “Firing Pin;” the entries are obviously not sorted in descending order by the purported firing pin score (that is, three digit “scores” are interspersed with two digit scores). The middle entry (clearly not the highest legible score, albeit close) flashes blue several times as the system beeps; at no point does a second, comparison casing image appear. “Got us a hit,” the examiner intones, now reading off of a new window that has popped up on screen. “Los Angeles County Sheriff’s Department found … shell casings from the same gun … used in a gang murder two years ago.” The investigator interjects, “They get a conviction on the suspect?” “No. Guy beat the rap,” the examiner continues. “Timothy Fontaine, aka ‘Tiny Tim’ … member of the Snakebacks … current residence unknown.” The “Criminal Records” window that appears on the screen also includes entries for a vehicle license number and the name of an arresting officer; unfortunately, the space clearly reserved for a photo of the person is labeled “NP AVAILABLE.” The investigator says, “I bet I could find where he stays in Vegas,” and the scene ends. The total elapsed time of the scene is 44 seconds. promising” the system, in at least two crucial and related respects. The first is the pervasive mythology that has come to surround the “top 10” results in an IBIS search. The current IBIS provides as its default printed report a listing of the 10 highest scores by each type of marking, and IBIS training materials undercut guidance to consider gaps and features in the
OCR for page 126
Ballistic Imaging distribution of comparison scores by promoting the examination of the top 10 suggested matches. However, the implied physical or cognitive restriction to the top 10 results is not likely to be appropriate in all searches or all database sizes, and the focus on the top 10 results is inadequate for assessing the system’s performance and for understanding the variability of scores by demographic characteristics (e.g., gun make and model). We know of no substantiated rationale for the ad hoc cutoff at rank 10; the resulting assumption that nothing outside of the top 10 ranked is valuable puts unduly high expectations on the system. The second basic flaw is the use of the term “correlation” to describe the IBIS comparison process, which imputes to the system an unjustified air of technical exactness. The common, statistical use of the term implies a particular type of relationship and quantifies the strength of that relationship. In comparison, IBIS scores are described by the system’s own training materials as having no intrinsic value, severely limiting the ability to express the strength of similarity between two exhibits and to compare results across different runs of the system. As we suggest in Chapter 6, we believe that the usefulness of IBIS is compromised unless some meaning can be imputed to its “correlation” scores—to make them function more like true statistical correlations. 4–G APPENDIX: SUMMARY OF PERFORMANCE TESTS IN THE CALIFORNIA EVALUATION OF A REFERENCE BALLISTIC IMAGE DATABASE This appendix describes the tests performed by Tulleners (2001) in response to the California legislature’s directive that the state’s Department of Justice study the feasibility of a reference ballistic image database. We begin by profiling those tests that were actually completed; these summaries extract additional information from spreadsheet printouts that were included as an appendix to Tulleners (2001). We also describe those tests that were planned for the evaluation but were unable to be completed, and summarize the formal responses to and independent assessment of the California evaluation. 4–G.1 Completed Performance Tests The Tulleners (2001) technical evaluation was based on the completion of five performance tests.
OCR for page 127
Ballistic Imaging Test 1 —Basic System Correlation Two cartridges were fired from each of the 792 CHP pistols, one to be entered into a “test” database and the other retained as an “evidence” exhibit. All of these firings used the same Federal brand ammunition. The basic goals of this test were to assess the time required to enter specimens into a database and to test the accuracy of comparison as database size increases. The first component of this test considered the basic ability of the system to find exhibits for guns known to be in the database. A sample of 50 test cartridges (the same-gun pairs of “evidence” entries already in the database) was drawn, and queries were made against the full database. Twenty-four (48 percent) of these test casings matched to their sister evidence casing as the first-ranked entry in either breech face or firing pin mark. However, a surprisingly high 19 of the comparisons (38 percent) did not find the sister casing within the top 10 ranked items in either breech face or firing pin,8 of which 9 (18 percent) of these known-match comparisons failed to clear IBIS’ coarse comparison and 20 percent threshold.9 It does not appear that one mark was superior to the other in terms of generating possible matches: the 31 instances where the known sister was found in the top 10 by either mark are fairly evenly divided between cases where both marks were in the top 10 (10), only the breech face was in the top 10 (9), and only the firing pin was in the top 10 (12). A second component of the test selected five of the “evidence” casings used in the first test that had low ranks on one or both markings; these were reacquired by a second IBIS operator and matched against smaller subsets of the data to see if those changes affected the rankings. In terms of comparisons to the full database, the rankings changed using the image from the second operator but not grossly so; no very low-ranked exhibits were converted to high ranks, although two of the casings apparently failed to clear the 20 percent threshold in the reacquisition.10 The entries were compared against database subsets of size 100, 200, 300, 400, 500, 600, 700, and 792; generally, rankings degraded with the larger sample 8 The main text of Tulleners (2001) indicates these figures as being searches for matches in the top 15 ranked items, but it can be verified from the “raw data” spreadsheets in Appendix C of the technical evaluation that the statements hold for the stronger (and more conventional) top 10 filter. 9 Failure to clear the 20 percent threshold is assumed from the “Not in Selection” entry for both score types (breech face and firing pin) in the technical evaluation spreadsheets. 10 One of these, labeled E44 in the first test and E152A in the reacquisition, appears to have had a significant difference in the acquisition of the firing pin image. The exhibit was ranked 45 on breech face and 1 on firing pin in the first analysis, but apparently failed to clear the 20 percent threshold and was excluded from listing in the reanalysis (Tulleners, 2001: Appendix C).
OCR for page 128
Ballistic Imaging sizes, though very high-ranked exhibits tended to stay very high (e.g., a one-ranked exhibit on firing pin remained the number one rank for all the sizes, and a two-ranked exhibit for the 100-entry database slipped to rank six in the full 792 set).11 Test 2 —Cartridges Not in Database Ten cartridges were fired using the same Federal brand cartridges but using 10 pistols of the same make and model not from the new CHP order. The comparison scores for the best match on both marks were recorded and were judged to be consistent with the range of scores registered in Test 1, several with the high end of that range. However, the evaluation accepted FTI’s advice that “a score is only relevant within a particular correlation” and that “the score cannot be used to compare the ranking of two correlations.” The test was found to be inconclusive. Test 3 —Different Ammunition During the test firing of the CHP pistols, 22 of the pistols were also used to fire rounds using batches of five different ammunition brands: PMC-Eldorado (.40 S&W 180 grain), CORBON (.40 S&W 165 grain), ARMSCOR (.40 S&W 180 grain), Remington (.40 S&W 180 grain), and Winchester (.40 S&W 180 grain). Not all of the ammunition types were fired from each of the guns; 72 cartridges were acquired in total. Each of these casings was then compared with the 792-exhibit set to test the ability of the system to find the Federal-brand test fire from the same gun in the database. The test found poor results in finding matches to the images from Federal-brand ammunition using images from the other five brands. Sixteen of the 72 comparisons (22 percent) matched to the image from the same gun as the top-ranked result on either the breech face or firing pin impressions; in total, 21 of the comparisons (29 percent) had the known sister image occur in the top 10 ranks on either mark. Neither mark was better at generating matches; 13 top-10 matches were found on the breech face mark and 14 on the firing pin. No match was found in the top 10 ranks by either mark in 26 of the comparisons (36 percent), and 25 of the comparisons (35 percent) failed to clear the coarse comparison and 20 percent threshold. 11 A third part of the test timed the comparison times for three selected exhibits for database subsets of different sizes (100, 250, 500, 792). From the results, Tulleners (2001:8-5) concluded that “correlation times are not a significant issue for a large database” although he assumed a strict linear interpolation in processing times.
OCR for page 129
Ballistic Imaging Tabulations from Tulleners (2001:Appendix C) suggest that the ARMSCOR and Corbon ammunition proved particularly difficult to match. Out of 14 matches of ARMSCOR rounds to the Federal-brand images in the database by breech face, 6 failed to clear the coarse comparison step, and 8 ranked lower than 25; 13 Corbon comparisons were attempted, with 8 being rejected at the coarse comparison stage and only one ranking better than 25 (but out of the top 10). Results by firing pin were similar, with a few more rankings in the 11–25 range and one ARMSCOR round finding its Federal-brand sister as the top-ranked result. The Winchester rounds proved most amenable to matches in the 18 comparisons that were made: 7 found the Federal sister round in the top-ranked slot, with 1 ranking 11–25, 6 ranking below 25, and 4 missing the 20 percent threshold (see also De Kinder’s [2002b:11] analysis of the same spreadsheet). Test 4 —Altered Breech Face After firing the two Federal-brand cartridges for the “test” and “evidence” sets, the firing pin tip and breech face of one of the CHP pistols was subjected to “minimum file and sandpaper efforts” to attempt to change the firearm’s individualizing marks (Tulleners, 2001:B-5). “This filing alteration took about three minutes using a standard file” (Tulleners, 2001:7-3). A second set of two test fires with Federal ammunition was then performed, one for entry in the database and the other used as an “evidence” query. The two sets of exhibits, before and after alteration, matched to each other well: the pre-alteration casings matched to each other in the top-ranked position on both firing pin and breech face and the post-alteration casings matched as the top-ranked pairing on firing pin (however, the rank was 35 on breech face). However, no match was possible from the pre-alteration to the post-alteration exhibits; in both cases, the technical evaluation’s data appendix lists the matches as “not in selection list,” suggesting that the deliberate alteration prevented the exhibits from clearing the IBIS coarse comparison pass. Test 7 —Breech Face Longevity Study In a test intended to determine whether a breech face maintains individual marks over repeated firings, an independent laboratory was contracted to perform 600 test fires from each of two .40 caliber pistols; the make of one was described as a Glock type and the other as unknown (Tulleners, 2001:8-11). The Glock-type pistol was fired using CCI brand ammunition, and IMI ammunition was used in the unknown-make pistol. For each pistol, casings in the intervals 1–6, 101–106, 201–206, 301–306, 401–406, 501–506, and 595–600 were retained for analysis; one casing from each
OCR for page 130
Ballistic Imaging interval was used as the “test” database and the other as “evidence” entries. Ultimately, the Glock-type firings turned out to be unusable due to the lack of a larger database for comparison; none of the other CHP weapons have Glock type firing pins, so they could not be compared to the Glock firings due to IBIS’ demographic filtering. Tulleners (2001:8-11) concluded that there were signs of “definitive ranking degradation” as the firings from later intervals were tended to rank lower than those from the earlier firings among the IMI cartridges. However, the evaluation suggested that “further tests need to be conducted in this area.” 4–G.2 Incomplete Performance Tests Tulleners (2001) was unable to carry out tests 5, 6, and 8 in his original slate of experiments. Test 5 was intended to assess IBIS performance using cartridges fired from SIG Sauer firearms, which are known among examiners for having minimal breech face characteristics. An extensive set of SIG Sauer test fires was subsequently used in Tulleners’ joint study, De Kinder et al. (2004), described in the next section. Test 8 was meant to test the system using firearms known to have strong subclass characteristic carry-over, such as some Heckler and Koch and Lorcin firearms (see Section 3–B.1). Test 6 “would have taken some test-fired cartridge cases from selected weapons, buried one of the cartridge cases in a large database and then observe the correlation on these cartridge cases” (Tulleners, 2001:7-3, 7-4). The California Department of Justice was unable to complete the test as planned, though it arranged for a limited test along the same lines to be conducted by the New York City Police Department (NYPD). The California Criminalistics Institute submitted eight casings each fired from two 9mm SIG Sauer pistols. In each set, two rounds used Remington-Peters ammunition, and the other firings used Winchester, Federal, Hornady Vector, Fiocchi, CCI, and Sellier and Bellot ammunition. One of the Remington rounds was retained as the “evidence” casing, so that for each of the two pistols, seven sister images were mixed into the NYPD’s 9mm database, which then contained 3,673 items. For both pistols, four of the seven sister images were found in the top 15 ranks by either breech face or firing pin, and the second Remington round generally turned up as the top-ranked entry by either mark. The Hornady Vector, CCI, and Sellier and Bellot rounds “seemed to be the most difficult for comparison” (Tulleners, 2001:8-10). 4–G.3 Criticisms and Independent Review Rebutting the California study on behalf of ATF, Thompson et al. (2002:15, 16) argued most stridently that “all of [the performance test
OCR for page 131
Ballistic Imaging results] are skewed due to the selection of Federal Brand ammunition.” They argue that Federal is not the prescribed “ATF protocol ammunition in any of the calibers of interest, due to the primer surface generally being too hard in comparison to the ammunition being used in handguns.” Instead, they suggested Remington-Peters ammunition as a more suitable medium. In his review, De Kinder (2002b:9–11) rejected this argument, citing research by the Forensic Institute in The Netherlands on IBIS score results using 18 common primers suggesting that Federal showed medium performance in registering marks. (Unfortunately, the Dutch study did not include Remington ammunition, as it is not common in Europe.) More directly, De Kinder noted that hardness properties of primers are not well known but hardness measures for the six types of ammunition used in California’s Test 3 had been independently conducted by the Lawrence Livermore National Laboratory. The Federal brass primers were directly measured to be the least hard of the six (108 ± 5HV), including Remington-Peters’ nickel primers (157 ± 12HV).12 In its rebuttal to the California study, FTI (2002:5) argued that “the Evaluation has an overly pessimistic view of automated ballistics technology that discredits its conclusions.” In Test 1, FTI (2002:14–15), submits that too great a focus on the 38 percent of possible matches missing from the top 15 ranks unduly discounts the 48 percent that found the correct match in the top rank on one of the marks and the 62 percent that matched within the top 15 on either rank. “These results are sufficient to identify a significant number of cartridge cases that merit manual study and would have produced new cold hits.” More fundamentally, FTI (2002:13, 14) holds that the IBIS system was held to an unfair standard in the test. A firearms examiner manually compared the cases for FTI and concluded that he could not certify a match between eight of the Test 1 pairs and that “approximately half had markings that were somewhat unfavorable.” As a result, FTI suggested that at least the eight human-identified nonmatches be excluded from the statistics, arguing: It is immediately obvious that the performance of an automated examination could not, and should not, be more accurate than a microscope comparison by a firearms examiner. Thus, to the extent that the Evaluation included cartridge cases that had insufficient marks to be identified by a firearms examiner, the results cannot support the hypothesis, and the Evaluation must be without scientific value. 12 De Kinder also noted that the criticism of Federal ammunition was unusual, given that Federal had been chosen for a similar study by several of the same ATF authors (Thompson et al., 1996).
OCR for page 132
Ballistic Imaging On these points, De Kinder (2002b:14) concluded that the FTI arguments were an overreach. He countered that the passage quoted above “is the same type of expression as saying at the beginning of the 1990[s] that automated comparison of bullets and cartridge casings is impossible.” He preferred instead the revised statement that “the current scientific knowledge and state-of-the-art technology does not allow one to be more accurate than a microscope comparison by a firearms examiner.” De Kinder held that dropping the believed-“unmatchable” exhibits from analysis is “unacceptable,” particularly given that the study was oriented to studying the feasibility of an RBID. “All data points have to be taken into consideration” because “the goal of [an RBID] is not restricted to those cartridge cases that can be identified by a trained firearm examiner.” Generally, De Kinder (2002b) indicated approval of the conduct and interpretation of the major performance tests in the California study. He suggested the need for further study in a variety of areas.