Forensic Analysis Weighing Bullet Lead Evidence (2004) / Chapter Skim
Currently Skimming:
3. Statistical Analysis of Bullet Lead Data
3. Statistical Analysis of Bullet Lead Data
Pages 26-70
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 26... ...
A CIVL, a "compositionally indistinguishable volume of lead" which could be smaller than a production run (a "melt") is an aggregate of bullet lead that can be considered to be homogeneous.
|
From page 27... ...
First, as indicated in this chapter, we cannot guarantee uniqueness in the mean concentrations of all seven elements simultaneously. However, there is certainly variability between CIVLs given the characteristics of the manufacturing process and possible changes in the industry over time (e.g., very slight increases in silver concentrations over time)
|
From page 28... ...
Test statistics that measure the degree of closeness of the chemical compositions of two bullets are parameterized by critical values that define the specific ranges for the test statistics that determine which pairs of bullets are asserted to be matches and which are asserted to be non-matches. The error rates associated with false assertions of matches or non-matches are determined by these critical values.
|
From page 29... ...
, and cadmium (Cd) .4 The three replicates on each piece are averaged, and means, standard deviations, and ranges (minimum to maximum)
|
From page 30... ...
2-SD Overlap First, consider one of the seven elements, say arsenic. If the absolute value of the difference between the average compositions of arsenic for the CS bullet and the PS bullet is less than twice the sum of the standard deviations for the CS and the PS bullets, that is if lavg(CS)
|
From page 31... ...
If the two bullets match using this criterion for all seven elements, the bullets are deemed a match; otherwise they are deemed a non-match.7 Range Overlap The procedure for range overlap is similar to that for the 2standard deviation overlap, except that instead of determining whether 95 percent confidence intervals overlap, one determines whether the intervals defined by the minimum and maximum measurements overlap. Formally, the two bullets are considered as matching on, say, arsenic, if both max(CSl,CS2,CS3)
|
From page 32... ...
k =1, ..., K Then the compositional group average and the compositional group standard deviationsi2 are computed for this compositional group (assuming K members)
|
From page 33... ...
1 < 2sd(CG) for all seven elements, then the CS bullet is considered to be a match with that compositional group.
|
From page 34... ...
34 FORENSIC ANALYSIS: WEIGHING BULLET LEAD EVIDENCE 1 lo.
|
From page 35... ...
Again, we only care about distance and not direction, and for mathematical convenience we often work with the square of the distance function. The above extends to three dimensions: One needs an appropriate function of the standard deviations and correlations among the measurements, as well as a
|
From page 36... ...
These concepts extend directly to more than three measurements, though the physical realities are harder to picture. A specific, squared distance function, generally known as Hotelling's 12, is generally preferred over other ways to define the difference between sets of measurements because it summarizes the information on all of the elements measured and provides a simple statistic that has small error under common conditions for assessing, in this application, whether the two bullets came from the same CIVL.
|
From page 37... ...
Conversely, if the means of the CIVL are not the same, a decision that they are the same is also an error. The latter error may occur when the two bullets from different CIVLs have different compositions but are determined to be analytically indistinguishable due to the allowance for measurement error, or when the two CIVLs in question have by coincidence the same chemical composition.
|
From page 38... ...
For a univariate test of the type described here, critical values are often set so that there is a 5 percent chance of asserting a non-match when the bullets actually match, i.e., 5 percent is the false non-match rate. This use of 5 percent is entirely arbitrary, and is justified by many decades of productive use in scientific studies in which data are generally fairly extensive and of good quality, and an unexpected observation can be investigated to determine whether it was a statistical fluke or represents some real, unexpected phenomenon.
|
From page 39... ...
This 800-bullet data set provides individual measurements on three bullet lead samples which permits calculation of within-bullet means, standard deviations, and correlations for six of the seven elements measured with ICP-OES (As, Sb, Sn, Bi, Cu.
|
From page 40... ...
Further, a few of the specified measurements were not recorded, and only 854 bullets had all seven elements measured. Also, due to the way in which these bullets were selected, they do not represent a random sample of bullets from the population of bullets analyzed by the laboratory.
|
From page 41... ...
In summary, we will concentrate much of our analysis on the 1,837-bullet data set, understanding that it likely has bullets that are less alike than one would expect to see in practice. The 1,837-bullet data set was used primarily to validate the assumption of lognormality in the bullet means, and to estimate within-bullet standard deviations.
|
From page 42... ...
in bullets from 1,837-bullet data set: (a) log(As mean concentrations)
|
From page 43... ...
in bullets from 1,837-bullet data set: (a) log(Sb mean concentrations less than 0.051; (b)
|
From page 44... ...
log(B) mean concentrations)
|
From page 45... ...
in bullets from 1,837-bullet data set: (a) log(Ag mean concentrations)
|
From page 46... ...
In addition, the 1,837bullet data set suggests that, for the elements Sn and Sb, the distributions of bullet lead composition either are bimodal, or are mixtures of unimodal distributions. Further, some extremely large within-bullet standard deviations for copper and tin are not consistent with the lognormal assumption, as discussed below.
|
From page 47... ...
Within-Bullet Standard Deviations and Correlations From the 800-bullet data set of the average measurements on the logarithmic scale for each bullet fragment, one can estimate the within-bullet standard deviation for each element and the within-bullet correlations between elements. (We report results from the log-transformed data, but results using the untransformed measurements were similar)
|
From page 48... ...
The pooled within-bullet standard deviations on the logarithmic scale (or RSDs) for the 800-bullet and 1,837-bullet data sets are given in Table 3.2.
|
From page 49... ...
In particular, probabilities for tests, such as the 2-SD overlap procedure, that operate at the level of individual elements and then examine how many individual tests match or not, cannot be calculated by simply multiplying the individual element probabilities, since the multiplication of probabilities assumes independence of the separate tests. Since the 1,837-bullet data set used by the committee does not include multiple measurements per bullet (only summary averages and standard deviations)
|
From page 50... ...
Some of this understanding would result from decomposing the variability of bullet lead into its constituent parts, i.e., within-fragment variation (standard deviations and correlations) , between-fragment within-bullet variation, between-bullet withinwire reel variation, between-wire reel and within-manufacturer variation, and between-manufacturer variation.
|
From page 51... ...
Here we focus on comparing the within-measurement standard deviations obtained using the 800bullet data set with the within-lot standard deviations in the Randich data. The former includes five of the seven elements (As, Sb, Cu.
|
From page 52... ...
Again, further investigation is needed to determine whether this large within-CIVL variance for copper is a general phenomenon, and if so, how it should affect interpretations of bullet lead data. Randich et al.
|
From page 53... ...
We have already examined between-bullet standard deviations and correlations. This section is devoted to the average relative difference in chemical composition of bullets manufactured from different CIVLs.
|
From page 54... ...
ESTIMATING THE FALSE MATCH PROBABILITIES OF THE FBI'S TESTING PROCEDURES We utilize the notation developed earlier, where CSi represented the average of three measurements of the ith fragment of the crime scene bullet, and similarly for PSi. We again assume that there are seven of these sets of measures, corresponding to the seven elements.
|
From page 55... ...
To start, we discuss the FBI's calculation of the rate of false matching. FBI's Calculation of False Match Probability The FBI reported an estimate of the false match rate through use of the 2SD-overlap test procedure based on the 1,837-bullet data set.
|
From page 56... ...
, the FBI allowance of approximately 3.55c, being more than twice as wide raises a concern that the resulting false match and false non-match probabilities do not represent a tradeoff of these error rates that would be considered desirable. (Note that for the normal distribution, the probability drops off rapidly outside of the range of two standard deviations but not for longer-tailed distributions.)
|
From page 57... ...
, the measurement uncertainty were constant for all elements (for example, c, = 1.0 percent) , and the measurement errors for all seven elements were independent, the false match probability for seven elements would equal the product of the per-element rate seven times (for example, for ~ = 2.0, c, = 1.0, .8417 = 0.298 for the 2-SD-overlap procedure, and .3777 = 0.001 for the range-overlap procedure)
|
From page 58... ...
Table 3.8 for the 2-SD-overlap procedure for seven elements is rather disturbing in that for values of ~ around 3.0, indicating fairly sizeable differences in concentrations, and for reasonable values of c,, the false match probabilities can be quite substantial. (A subset of the 1,837-bullet data set showed only a few pairs of bullets where 6/c, might be as small as 3 for all seven elements.
|
From page 59... ...
Transform these seven numbers so that they have the same correlations as the estimated withinbullet correlations. Multiply the individual transformed values by the withinbullet standard deviations to produce a multivariate normal vector of bullet lead concentrations with the same covariance structure as estimated using the 200 Federal bullets in the 800-bullet data set.
|
From page 60... ...
The overall average and average standard deviation of the 42 average concentrations of the 42 "matching" bullets are given in the third and fourth lines of Table 3.11. In all cases, the average standard deviations are at least as large as, and usually 3-5 times larger than, the standard deviation of bullet 1,044, and larger standard deviations are associated with wider intervals and hence more false matches.
|
From page 61... ...
Again, we denote the three measurements on the CS and PS bullets CSi,C52,C53 and PSi,P52,P53, respectively. The basic question is whether three measurements of the concentrations of one of the seven elements from two bullets are sufficently different to be consistent with the following hypothesis, or are sufficiently close to be inconsistent with that hypothesis: that the mean values for the elemental concentrations for the bullets manufactured from the same CIVL with given elemental concentrations, of which the PS bullet is a member, are different from the mean values for the elemental concentrations for the bullets manufactured from a different CIVL of which the CS bullet is a member.
|
From page 62... ...
In practice we can check to see how close to normality we believe the bullet data or transformed bullet data are, and if they appear to be close to normality with no outliers we can have confidence that our procedure will behave reasonably. The spirit of the 2-SD overlap procedure is similar to the two-sample t-test for one element, but results in an effectively much larger critical value than would ordinarily be used because the "SD" is the sum of two standard deviations (SD(CS)
|
From page 63... ...
That is, given a specific 6, one cannot find a test statistic that has a simultaneously lower false match rate, given a specific 6, and lower false non-match rate. The setting of ta, which determines both error rates, is not a matter to be decided here, since it is not a statistical question.
|
From page 64... ...
In that case, the theoretically optimal procedure, assuming multivariate normality, is to add the squares of the separate t-statistics for the seven elements and to use the sum as the test statistic. The distribution of this test statistic is well-known, and false match rates and false non-match rates can be determined for a range of possible critical values and vectors of separation, 6.
|
From page 65... ...
This approach, which we will denote the "successive t-test approach" test statistics, is as follows: 1. estimate the within-bullet standard deviations for each element using a pooled within-bullet standard deviation sp from a large number of bullets, as shown above.
|
From page 66... ...
normally distributed, and if pooled estimates, over an appropriate reference set of bullets, are available to estimate within-bullet standard deviations and within-bullet correlations, and finally, if all seven elements are relatively active in discriminating between the CS and the PS bullets, then T2 is an excellent statistic for assessing match status. The successive t-test statistics procedure is somewhat less dependent on normality and can be used in situations in which a relatively small number of elements are active.
|
From page 67... ...
However, as in the univariate case, having an acceptable false match rate for separation vectors where the within-bullet standard deviations become unlikely to be a reasonably full explanation for differences in means would be very beneficial. It would also be useful to include a separation vector that demonstrated the performance of the procedure when not all mean concentrations for elements differ.
|
From page 68... ...
Further, estimated standard deviations should be charted regularly to
|
From page 69... ...
The tests should use pooled standard deviations and correlations, which can be calculated from the relevant bullets that have been analyzed by the FBI Laboratory. Changes in the analytical method (protocol, instrumentation, and technique)
|
From page 70... ...
"Statistical Treatment of Case Evidence: Analysis of Bullet Lead," Unpublished report, Dept. of Statistics, Iowa State University, 2002.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.