National Academies Press: OpenBook

Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements (1998)

Chapter: Appendix A Case Studies and System Descriptions

« Previous: 10 Increasing Access to Statistical Expertise for Operational Testing
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

APPENDICES

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

This page in the original is blank.

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

APPENDIX A
Case Studies and System Descriptions

In this appendix we present several examples, or case studies, that we refer to in the body of the report. We intend this appendix to help orient readers in the statistical community who are unfamiliar with the testing and evaluation of defense systems, as well as to provide additional background and support for recommendations addressed to those in the defense acquisition community.

LONGBOW APACHE HELICOPTER

The Longbow Apache (AH-64D) is a modified version of the Army's existing attack helicopter, the Apache (AH-64A). A key distinguishing feature of the new Longbow helicopter is a fire control radar (FCR) system that allows the helicopter to engage targets with radar-guided Hellfire missiles. Because the missiles can be used without visual or optical acquisition of the target, the FCR system is expected to increase the operational effectiveness of the helicopter in conditions of adverse weather and reduced visibility (e.g., due to smoke or fog) when the performance of laser, optical, and infrared sensors is degraded. In addition, the Longbow AH-64D includes an improved data modem that is intended to enable it to transfer digital target data to other attack helicopters, including the Apache AH-64A, thus enhancing the capabilities of integrated attack helicopter teams.

The Longbow is considered a major acquisition program (ACAT I). In constant fiscal 1994 dollars, the total procurement cost of the system is estimated

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

at $5.3 billion, and the total 20-year life-cycle cost is estimated at $14.3 billion.1 The total research, test, development, and evaluation costs are expected to be approximately $567 million. Because the Longbow is an ACAT I program, its operational testing and evaluation is subject to oversight by DOT&E, in the Department of Defense.

Operational testing of the Longbow comprised a series of gunnery tests with live ammunition at the Naval Air Warfare Center (in China Lake, California) in January and February 1995, and a series of force-on-force tests with simulated ammunition (at Fort Hunter Liggett, California) in February and March 1995. Separate air transportability and FCR conversion exercises were conducted as part of the force-on-force testing. These operational test events followed a period of developmental testing, training, and force development testing and experimentation. Both the gunnery and force-on-force tests involved a comparison of the performance of the Longbow AH-64D against the existing Apache AH-64A system.

The initial operational test and evaluation plan for the Longbow Apache involved a total of 46 measures of performance (MOPs). Of these, 21 were related to operational effectiveness and 26 were related to operational suitability. Of the 26 suitability MOPs, 11 concerned the Longbow's reliability, availability, and maintainability. Operational requirements in the initial plan were specified for 6 of the 11 measures of reliability, availability, and maintainability; see Table A-1 . Because the Longbow AH-64D is a modification of an existing system, many of its RAM requirements were specified on the basis of the comparable Apache AH-64A requirements and observed operating experience. For example, the availability of the Longbow AH-64D, excluding the FCR system, is required to at least match the projected baseline availability value of 0.916, which was computed using Army logistics records for the Apache AH-64A. The availability requirement for the Longbow with the FCR system is 0.824, a value that allows for a 10 percent degradation in availability to maintain the new FCR system.

The primary purpose of the gunnery test was to assess the effectiveness of the Hellfire radar-guided missile when used as part of the Longbow Apache system. The gunnery test collected data on the proportion of targets designated by the helicopter's FCR that were acquired by the missile's radar. The test also provided information about the range in which the missile could effectively engage targets. Such tests involving a small number of live-ammunition firings are needed to establish credible data on system performance. Live ammunition is not used in force-on-force tests for obvious reasons. The gunnery phase consisted of live missile firings from both the Longbow AH-64D and the Apache AH-64A

1  

The source of the estimates is Annex D of the Longbow Apache Test and Evaluation Master Plan (U.S. Department of Defense, 1993), which cites December 1993 estimates from the Longbow Program Office and the President's fiscal 1995 budget as the original sources.

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

TABLE A-1 Summary of Longbow RAM Requirements and Demonstrated Performance in Initial Operational Testing and Evaluation (in hours)

Measure

Required

Demonstrated

Objective

Reliability

 

 

 

Mean Time Between Mission Failure

15.3

22.18

21.0

Mean Time Between FCR System Failure

102.0

136.0

102.0

Availability

 

 

 

With FCR

0.824

0.914

 

Without FCR

0.916

0.925

 

Maintainability

 

 

 

FCR Mean Time to Repair

0.5

3.85

 

FCR Maintenance Ratio

0.023

0.173

 

teams against an array of enemy targets. Firings occurred under three distinct conditions: day mission with smoke obscuration, night mission with clear visibility, and night mission with smoke obscuration. The test and evaluation plan listed the required number of missile shots in each of the three scenarios as 12, 8, and 4, respectively, for the Longbow team and 5, 3, and 3, respectively, for the Apache baseline team. The panel does not know the basis for determining these gunnery test sample sizes.

Sample size for the force-on-force test was determined from considering the criterion that the modernized Apache AH-64D team demonstrate increased targets hit when compared with the AH-64A team. This criterion was interpreted as requiring a sample of sufficient size to detect a (one-sided) difference of 10 percentage points in the probabilities of hits of the Longbow team and the Apache team against the Red forces. Assuming a baseline hit probability of 0.5 yields a conservative sample size estimate that 325 missile shots by both the Longbow AH-64D and the baseline AH-64A are needed to estimate the difference in hit probabilities at 0.10 levels of both consumer's and producer's risk. If a higher risk level of 0. 15 is accepted by producer and consumer, then the sample size requirement can be reduced to 212 missile shots. The conclusion in the test and evaluation report is stated as follows (U.S. Department of Defense, 1993):

A sample size of 212 to 325 missile shots by the Longbow team, as well as by the AH-64A team in each cell of the event matrix, will detect a difference of 10 percent or more and give 85-90 percent confidence in the criteria conclusions. Based on the expected number of shots per mission, the number of missions [sic] under each condition can be estimated.

The force-on-force test was conducted in three major scenarios: close battle at day, close battle at night, and deep battle at night. (Close battles occur at the military front; deep battles involve strikes inside enemy lines.) A total of 320 missile shots were required in both night battle scenarios, and 256 missile shots were required in the day battle scenario.

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

The Longbow Apache test appears to be typical in that the size of the force-on-force test was primarily determined by considerations of effectiveness rather than suitability. A common approach in operational testing seems to be: Do the testing necessary to assess effectiveness, and accumulate as much of the requisite RAM data as feasible.

The primary data sources for measuring the reliability, availability, and maintainability performance of the Longbow system were the gunnery and force-on-force tests conducted as part of the initial operational test and evaluation plan. Secondary data sources included development testing, logistical demonstrations, and force development test and experimentation. According to the test and evaluation plan (U.S. Department of Defense, 1993:2-92): "Secondary data will be used to supplement [operational test] results by demonstrating historical performance, helping identify trends in anomalies found in the primary MOPs, and characterizing performance in conditions not encountered in the IOT." The test and evaluation plan also stated that, because of the different conditions obtained under gunnery and force-on-force testing, results from the two phases would not be merged; instead, they would be presented separately in the evaluation report. For the RAM-related MOPs, however, the operating hours appear to reflect the accumulated hours under both the gunnery and force-on-force phases.

The demonstrated RAM-related performance of the Longbow Apache AH-64D helicopter in comparison with its operational requirements is summarized in Table B-1. The mean time between mission failure (MTBF[M]) value of 22.18 hours is based on 11 observed mission failures of the Longbow AH-64D helicopter in a total of 244 operating hours. The mean time between FCR system failure (MTBF[S]) value of 136 hours is based on 2 failures in 272 system operating hours. The maintainability of the FCR system is summarized by two measures: (1) its mean time to repair of 3.85 hours, comprising six corrective maintenance actions and a total of 23.07 hours in repair time; (2) its maintenance ratio of 0.173, reflecting a total of 47.05 person-hours spent on system maintenance divided by the total of 272 system operating hours.

Computation of the availability measure is a bit more involved. The observed availability values (A0) of 0.914 and 0.925 for the Longbow AH-64D—including and excluding the FCR system, respectively—are computed according to the formula:

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

where OT denotes the total operating (flight) time of the Longbow system, TT denotes the total calendar time during the test period, MTTRe denotes the mean time expended on essential maintenance actions, ALDT denotes the total administrative and logistics downtime (e.g., time spent waiting for parts, maintenance personnel, or transportation), and MTBEMA denotes the mean time between essential maintenance actions. The sample values of these constituent parts observed during the initial operational testing and evaluation are described in the Longbow Apache operational evaluation report.

On the basis of the results in Table B-1, the Longbow Apache was judged to have met its reliability and availability requirements, but the FCR system did not meet its maintainability requirements. However, because the evaluator judged the FCR maintainability measures of secondary importance and because the realism of the 0.5-hour mean repair time requirement for the FCR was questioned, the Longbow system was determined to have demonstrated adequate RAM performance in its initial operational test.

C-130H HERCULES AIRCRAFT

The primary mission of the C-130H Hercules aircraft is to provide intratheater airlift—particularly, the tasks of normal airland, tactical airland, and tactical airdrop—in worldwide environments. The C- 130H is an upgraded version of the older C-130, with five previously untested modifications: a low-power color radar; rearranged cockpit instruments; an electrical system upgrade; new, flatscreen electronic flight instruments; and a mode advisory caution and warning system.

The purpose of the qualification operational test and evaluation was to evaluate the effectiveness and suitability of the modified C-130H. Instead of focusing on such parameters as range and payload (that are typically of primary interest for cargo aircraft in initial operational testing and evaluation), the qualifying testing and evaluation focused on the operation of the new modifications. The effectiveness evaluations essentially involved comparisons of the new modifications with the old subsystems to be replaced. Effectiveness data were gathered during two weeks of flight testing (approximately 60 flying hours) in March 1994 (at Little Rock Air Force Base).

Minimum reliability, availability, and maintainability test times were statistically determined according to the standard formula assuming exponentiality (see Milstar example, below). Required test times for the five new subsystems ranged from approximately 200 hours to 17,000 hours. A compromise test period of 5 months was selected with two justifications: the schedule would permit the issuance of a final report in time to influence fiscal year production decisions; and approximately 1,000 flying hours were expected to be logged during this 5-month period, which would meet the statistical test requirements for three of the five new subsystems.

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

To evaluate the availability and reliability of the C-130H, maintenance data from February to June 1994 were entered into the Core Automated Maintenance System at the Little Rock Air Force Base and at the Wyoming Air National Guard base. These data were subsequently downloaded into the Micro-Omnivore database of the Air Force Operational Test and Evaluation Center (AFOTEC). A test data scoring board maintained quality control of the maintenance events and checked the data for completeness and accuracy. The combined data represented approximately 2,200 flying hours.

The C-130H was found to be effective but not suitable. In some cases, observed values of the reliability, availability, and maintainability measures differed from operational requirements by as much as two orders of magnitude. AFOTEC's recommendation was to delay deployment of the C-130H until adequate technical orders, support equipment, and spares were available and training was complete. AFOTEC also recommended testing of the C-130H in extreme cold, desert, and tropical climates to stress the aircraft in these conditions.

B-1B BOMBER

The B-1B aircraft is a long-range, supersonic bomber with after-burning engines, capable of conducting high-speed, low-level flight profiles. An operational readiness assessment was mandated by Congress to determine whether the B-1B can achieve and sustain a desired readiness rate of 75 percent over a 6-month period. In the 2 years preceding the assessment, mission capable rates in the B-1 fleet averaged approximately 57 percent, primarily because of inadequate funding for repair of parts through interim contractor support and for stocking of repairable and new spare parts. In effect, the assessment was a test of the adequacy of the Air Force's planned upgrades in the levels of B-1 spare parts, logistics support equipment, and maintenance personnel.

The test was conducted at Ellsworth Air Force Base (South Dakota) from June 1 to November 30, 1994. The 28th Bomb Wing was selected as the military test unit on the basis of several considerations, including stability of force structure during the assessment period, impact of testing on field training operations, programmed aircraft maintenance and modification schedules, and overall unit experience in conventional operations. Aircraft operations during the 6-month evaluation period included planned peacetime training and a 2-week deployment to a remote location for exercises consistent with the B-1B's expected use in a conventional conflict.

The term "75 percent operational readiness rate" was defined as an average mission capable rate of 75 percent of possessed aircraft hours. An aircraft is considered mission capable if it is capable of performing at least one of its designated missions. Note that the mission capable rate is a measure of the performance of the entire B-1B support structure, encompassing more than the aircraft itself. In particular, the mission capable rate is not a direct measure of the

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

B-1B's inherent reliability or its ability to successfully complete assigned missions.

Data were collected for a total of 12 reliability, availability, and maintainability measures. The three reliability measures were break rate of system failures during assigned missions; mean time between corrective maintenance events; and rate of aircraft-to-aircraft or engine-to-aircraft cannibalization actions per 100 sorties flown. The four maintainability measures were the 12-hour fix rate, expressed as the percentage of inoperable aircraft that are returned to flyable status within 12 hours; maintenance person-hours per flying hour; mean man-hours to repair; and mean repair time, not including maintenance or supply delays. The five availability measures were the mission capable rate; the total not mission capable rate; two submeasures intended to isolate maintenance and supply problems; and the utilization rate, expressed as the average number of hours flown per aircraft per month.

With improved reliability, availability, and maintainability support, the 28th Bomb Wing achieved a readiness rate of 75 percent by the start of the operational readiness assessment and demonstrated a cumulative readiness rate of 84.3 percent during the 6-month evaluation period. Test results were regarded as evidence that, with 100 percent staffing, sufficient spare parts, and a robust base repair capability, a B-1B wing can achieve and sustain the required 75 percent readiness rate.

(According to AFOTEC presentation materials, data analyses for the 12 reliability, availability, and maintainability parameters involved trend analysis and analysis of variance—in addition to reporting mean values—but the final assessment report does not mention them.)

MILSTAR SATELLITE COMMUNICATION SYSTEM

Milstar is a constellation of four satellites and associated ground support, designed to provide secure, high-priority, strategic communication. The initial operational test and evaluation was an 18-month test, from February 1995 to July 1996, with oversight from DOT&E. This was a joint (or multiservice) test and evaluation conducted by the Air Force with involvement of other military services. Its purpose was to evaluate the operational effectiveness and suitability of the Milstar Low Data Rate System.

The testing and evaluation consisted of limited field exercises to test the endurance of the mobile constellation control station—defined as the Milstar terminal and satellite control station aligned in series configuration. The user requirement is that the mobile station system ''endure" for X days (a classified value) at a deployed location without outside support. The required number of test hours per platform (T) was determined statistically—assuming exponentially distributed failure times. Statistical means were to be used in reporting test data.

Field testing resources alone were judged insufficient to support evaluation,

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

and the use of a complementary model was proposed. For modeling purposes, the system was defined as two mobile constellations and four satellites. Modeling efforts were to be concentrated on the effect of reliability and sparing levels on the operation of the stations and satellites. Lack of a rigorous model validation, however, precluded the formal incorporation of modeling results into the system evaluation.

THE ATACMS/BAT SYSTEM

In the Army Tactical Missile System/Brilliant Anti-Tank (ATACMS/BAT) System, the Army Operational Test and Evaluation Command proposes to use a relatively novel test design in which a simulation model, when calibrated by a small number of operational field tests, will provide an overall assessment of the effectiveness of the system under test.

BAT submunitions use acoustic sensors to guide themselves toward moving vehicles and an infrared seeker to home terminally on targets. The submunitions are delivered to the approximate target area by the ATACMS, which releases the submunitions from its main missile. The ATACMS/BAT system is very expensive, costing several million dollars per missile.

The experimental design issue is how to choose the small number of operational field tests such that the calibrated simulation will be as informative as possible. Here we provide a description of the ATACMS/BAT operational testing program to illustrate one context in which to think about alternative approaches to operational test design.

Plans for Operational Testing

As noted above, the ATACMS/BAT operational test will not be a traditional Army operational test involving force-on-force trials, but will be similar to a demonstration test, using a model to simulate what might happen in a real engagement. To calibrate the simulation, various kinds of data will be collected, for example, from individual submunition flights and other types of trials, including operational test trials. This relatively novel approach has been taken because budgetary limitations on the sample size and the limited availability of equipment such as radio-controlled tanks for testing make it infeasible to develop a program of field testing that could answer the key questions about the performance of this system in a real operational testing environment.

Operational testing of the ATACMS/BAT system is scheduled to take place in 2000. According to the last approved Test and Evaluation Master Plan (U.S. Department of Defense, 1995b):

This portion of the system evaluation includes the Army ATACMS/BAT launch, missile flight, dispense of the BAT submunitions, the transition to independent flight, acoustic and infrared homing, and final impact on targets. Evaluation of

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

this discrete event also includes assessment of support system/subsystem RAM requirements, software, terminal accuracy, dispense effectiveness, kills per launcher load, and BAT effectiveness in the presence of countermeasures.

Initial operational test and evaluation is the primary source of data for assessing these system capabilities. There is no baseline system for comparison.

The number of armored vehicle kills (against a battalion of tanks) is the bottom-line measure of the system's success. Tank battalions vary in size, but typically involve about 150 vehicles moving in formation. (Unfortunately, every country moves its tanks somewhat differently.) Under the test scoring rules, no credit is given if the submunition hits the tank treads or a truck or if two submunitions hit the same tank.

There is one operational test site, and the Army has spent several million dollars developing it. There will be materiel constraints on the operational test. Only seven test events, with eight missiles, each of which has a full set of 13 BAT submunitions, are available for testing. Also, the test battalion will involve only 21 remotely controlled vehicles. Thus, the Army plans to use simulation as an extrapolation device, particularly in generalizing from 21 tanks to a full-size battalion.

Important Test Factors and Conditions

All stages of missile operation must be considered in an operational test, particularly acoustic detection, infrared detection, and target impact. Factors that may affect acoustic detection of vehicles include distance from target (location, delivery error), weather (wind, air, density, rain), vehicle signature (type, speed, formation), and terrain. For example, the submunitions are not independently targeted; they are programmed with logic to go to different targets. Their success at picking different targets can be affected by such factors as wind, rain, temperature, and cloud layers. Obviously, one cannot learn about system performance during bad weather if testing is conducted only on dry days. However, it is difficult to conduct operational tests in rain because the test instrumentation does not function well, and much data can be lost. Such factors as weather (rain, snow) and environment (dust, smoke) can also affect infrared detection of vehicles. Factors that affect the conditional probability of a vehicle kill, given a hit, include the hardness of the vehicle and the location of the hit. Possible countermeasures must also be considered: for example, the tanks may disperse at some point, instead of advancing in a straight-line formation or may try to use decoys or smoke obscuration.

The operational test design, or shot matrix, in the Test and Evaluation Master Plan lists eight test events that vary according to such factors as range of engagement, target location error, logic of targeting software, type of tank formation, aimpoint, time of day, tank speed and spacing, and threat environment; see Table A-2 .

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

Three levels are specified for the range of engagement: near minimum, near maximum, and a medium range specified as either "2/3s" or "ROC," the required operational capability. Target location errors vary with the method of control (either centralized or decentralized) and are quantified in terms of either a median level or one standard deviation (sigma) above the median. The logic of targeting software (primary or alternate) and type of tank formation (linear or dispersed) are combined into a single, two-level factor. Aimpoint distance is either short or long, and aimpoint direction is either left or right. The aimpoint factors are not expected to be important in affecting system performance. (The payload inventory is also unimportant in this context.) Tanks are expected to travel at lower speeds and in denser formations during night operations; therefore, tank speed and spacing are combined with time of day into a single two-level factor (day or night). Three different threat environments are possible: benign, Level 1, and Level 2 (most severe). Clearly, in view of the limited sample size, many potentially influential factors are not represented in the shot matrix.

Statistical Methods for Further Consideration

One approach to operational testing for the ATACMS/BAT would be to design a large fractional factorial experiment for those factors thought to have the greatest influence on system performance. The number of effective replications can be increased if the assumption that all of the included design factors are influential turns out to be incorrect. Assuming that the aimpoint factors are inactive, a complete factorial experiment for the ATACMS/BAT system would require 23 × 32 = 72 design points. However, fractional factorial designs with two- and three-level factors could provide much information while using substantially fewer replications than a complete factorial design. Of course, these designs are less useful when higher-order interactions among factors are significant. (For a further discussion of factorial designs, see National Research Council, 1995:Appendix B; Box and Hunter, 1961.)

Another complication is that environment (or scenario) is a factor with more than two settings (levels). In the extreme, the ATACMS/BAT operational test results might be regarded as samples from several different populations representing test results from each environment. Since it will not be possible to evaluate the test in several unrelated settings, some consolidation of scenarios is needed. It is necessary to understand how to consolidate scenarios by identifying the underlying physical characteristics that have an impact on the performance measures and to relate the performance of the system, possibly through use of a parametric model, to the underlying characteristics of those environments; see discussion in Appendix C of the panel's interim report (National Research Council, 1995).

Although the above fractional factorial approach has advantages with respect

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

TABLE A-2 Army Tactical Missile System Block II/Brilliant Anti-Tank Operational Test Shot Matrix

Testa

Range

Target Location Error (Method of Control)

Target/Logic

Aimpoint

Payload

Environmentb

DT/OT 1

ROC

ROC

Primary/linear

Long/right

9 Tt, 4 Tsw

Benign ROC

DT/OT 2

2/3s

Median centralized

Primary/linear

Short/right

13 Tt

Level 1 night

OT 1

Near max

1 sigma centralized

Primary/linear

Short/left

13 Tt

Level 1 and 2 day

OT 2

Near min

Median decentralized

Alternate/dispersed

Long/left

7 Tt, 6 Tsw

Level 1 day

OT 3c

2/3s

Median dentralized AMC

Primary/linear

C3 system determined

10 Tt, 3 Tsw

Level 1 night

OT 4c

2/3s

Median centralized AMC

Primary/linear

C3 system determined

10 Tt, 3 Tsw

Level 1 night

OT 5d

2/3s

1 sigma decentralized

Primary/linear

Long/on line

13 Tt

Level 1 and 2 night

OT 6d

2/3s

Median centralized

To be determined

To be determined

13 Tt

Level 1 day

NOTES:

AMC, Army Materiel Command

C3, command, control and communications

DT, developmental testing

OT, operational testing

ROC, required operational capability

See text for discussion.

a This shot matrix reflects a two-flight developmental testing/operational testing program of engineering and manufacturing development assets conducted by the Test and Evaluation Command, and a six-flight operational testing program of low-rate initial production Brilliant anti-tank assets conducted by the Test and Experimentation Command. This shot matrix is subject to threat accreditation.

b Flights will be conducted during daylight hours for safety and data collection. Night/day environments pertain to vehicle speed and spacing.

c OT 3 and OT 4 are ripple firings.

d Flight objectives may be revised based on unforeseen data shortfalls.

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

to understanding system performance equally in each scenario, there are some benefits of the current Army approach if one assumes that the majority of interest is focused on the "central" scenario, or the scenario of most interest. In the current approach, the largest number of test units are allocated to this scenario, while the others are used to study one-factor-at-a-time perturbations around this scenario, such as going from day to night or from linear to dispersed formation. This approach could be well suited to gathering information on such issues while not losing too much efficiency at the scenario of most interest. If it turns out that changing one or more factors has no effect, the information from these settings can be pooled to gain further efficiency at the scenario of greatest interest.

THE COMMON MISSILE WARNING SYSTEM

The common missile warning system (CMWS)2 is a warning system that alerts aircraft crew that missiles have been launched at the aircraft. The CMWS is intended to be effective against short-range infrared surface-to-air missile (SAM) threats. The system is composed of multiple sensors, coupled with a processing and control system, which examines the threat missile "plume" to determine whether the missile is a threat. If so, the CMWS triggers countermeasures and warns the aircrew. A key factor for the success of this system is the time between the reception of the warning and the missile strike. The basic requirement for the operational test of the system is the demonstration, at typical levels of statistical confidence,3 that the CMWS provides adequate warning across all threats and scenarios. However, there is no requirement that the test provide statistical confidence for assertions about the performance of the system at the threat- or scenario-level.

During test planning of the CMWS, it was initially assumed that the key factors related to warning time were altitude of the aircraft, horizontal distance (range) from the ground-based missile threat, aspect angle (the profile of the plane in the direction of the flight path of the missile), and the particular threat. (This assumption is currently being reviewed and is probably going to be revised once detailed simulation models of the relevant physical processes are developed.) Furthermore, using engineering and threat information, AFOTEC reduced the number of essential threats to four given the similar plumes and range and altitude of targets (threat envelopes) against which various threats are effective. Further, it is assumed that each of these four threats has equal priority, i.e., that their potential for use by various opponents is similar.

While the overall goal was the estimation of average warning time across threats and scenarios, it was also desired to estimate the average warning time for

2  

Much of the following is taken from Sheedy and Stolle (1996). In addition, a very useful informal summary was provided by Steve Ordonia.

3  

Unspecified in the documentation but ranging from 99% to 80%.

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

each of these individual threats across scenarios by producing a mathematical model that would predict warning time as a function of the above three factors. The hope was to support development of both an estimate of overall performance and these mathematical models for each threat for the least cost, i.e., for the fewest missile firings.

Additional information indicated that missile warning time was likely to be linearly related to altitude, and at most quadratically related to range and aspect angle. Therefore, missile warning time was measured at two different altitudes (low and high), three different ranges (short, medium, and long) and three different levels of aspect angle (rear, beam, and front), which would permit estimating the assumed relationship between these factors and warning time. The result was a total of 2 × 3 × 3 = 18 test scenarios of interest for each of the four threats, representing 72 threat-scenario combinations.

Fixed Grid Approach

The first approach to this problem defined the three test factors in terms of a "fixed" grid, i.e., identical across all threats. So, e.g., the two altitudes, low and high, were defined across threats as:

Low:

Alt ≤ 5,000 ft

High:

5,000 ft < Alt ≤ 10,000 ft.

The three ranges, in nautical miles (NM) were defined, again across threats, as:

Short:

Range ≤ 1 NM

Medium:

1 NM < Range ≤ 3 NM

Long:

3 NM < Range ≤ 6 NM.

Finally, the three aspect angles in degrees were defined, across threats, as:

Front:

315° ≤ θ < 45°

Beam:

45° ≤ θ < 135°; 225° ≤ θ < 315°

Rear:

135°≤ θ < 225°.

This fixed grid approach to the definition of test scenarios caused a problem, since some of the test scenarios in the resulting factorial design were not testable. For example, threat 2 could not be tested at long range. In addition, the low altitude, short range, beam scenario was testable for threat I but not testable for threat 2. The result was that only 32 of the 72 scenario/threat combinations were testable. (Note that other complications have been omitted in this description). The hope of an early design using this fixed grid was also to have estimates of performance in each scenario for each threat, which resulted in a suggested

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

FIGURE A-1 Threat Categorization: experimental design for fixed grid approach. NOTE: In the partial threat and no threat cells, no shots could be taken, as indicated, justifying the switch from a fixed grid approach to the scalable grid approach; see text for discussion. NM: Nautical miles.

SOURCE: "Common Missile Warning System Sample Size Strategy," 2-11-96 presentation, Air Force Operational Test and Evaluation Center

design of 5 shots per scenario and threat combination. (Assuming an extremely successful performance by the CMWS, this design would have had marginally acceptable power with respect to the assumption that the CMWS did not meet its required level of performance.) Given that some cells were designed to have an additional 10 shots, this design required 205 missile shots (see Figure B-1). It was decided that this was more shots than could be afforded. (Figure B-1 displays three altitutdes, which were later collapsed to two.)

The objectives of the test design were then revisited, and it was understood that no scenario- or threat- level analysis was requested, besides the development

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

of mathematical models at the threat level. Also, as was recognized by AFOTEC, there were real advantages for the test design to be a full factorial experiment (possibly with replications), i.e., to have each combination of levels of each test factor appear the same number of times in the test. This would facilitate development of the threat-level models and test evaluation.

Scaleable Grid Approach

Therefore, AFOTEC replaced the fixed grid approach with a scaleable grid for each threat. For example, the low and medium levels of altitude were shifted from 0-5,000 feet (low) and 5,000-10,000 feet (medium) to instead fluctuate based on the effective target altitude for the threat. (Details on the precise definition of the scaleable grid are not provided here.) This was also done for range and aspect angle. This approach is quite sensible in that it provides test scenarios that are well-distributed inside the feasible test region for each threat, which would not have been as true using the fixed grid approach.

As a result of using the scaleable grid, each of the 18 scenarios for each threat (which were defined differently for each threat) became testable, and therefore average missile warning time across scenarios and threats could be estimated for the CMWS (which now had a different interpretation than in the fixed grid situation), and a mathematical model of average missile warning time for each threat as a function of altitude, range, and aspect angle, and some two and three level interactions of these variables, could be easily developed. Finally, these models could be used to provide predictions and associated confidence intervals for additional scenarios that were within the test domain for each threat.

Test Sample Size Argument

The remaining question was whether one missile shot per scenario - threat combination would be sufficient to produce an estimate of average missile warning time across scenarios and threats that would pass a significance test at typical levels with high probability, thereby supporting that the CMWS satisfied its required level of performance. The argument used to justify test size was as follows. Assume that one missile shot is taken in each of the 18 scenarios for each of the four threats as a full factorial experiment over the three test factors. That would result in 72 missile shots. In addition, given that there was a requirement for investigation of the performance of the CMWS against multiple-threats in a given quadrant, the number of missile shots was increased by two for each threat. The resulting test would therefore have 20 shots per threat, or 80 overall shots.4 Characterizing a single missile test shot as being successful for the CMWS if the

4  

There were actually 96 test missile allocated. a 20 percent margin in case there were problems with the missiles.

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

warning provided was less than or equal to x seconds, where x was the required average warning time,5 the argument for the number of missile shots was that the test needed to demonstrate that the success rate was significantly greater than .80 across scenarios and threats. Using 80 total missile shots, the displayed number of failures result in the associated statistical confidence that the real success rate was greater than .8:

Total Missiles: 80

Number of Failures

Confidence

8

99%

9

97%

10

94%

11

90%

12

84%

(Note that this argument assumes a constant success rate across threats and scenarios, which is unlikely to obtain.)

Of course, this does not directly answer the question of how reliable estimates of the average warning time across scenarios and threats will be or address power considerations. However, the probability of a warning time less than x is certainly of interest, and there may have been little a priori information on what standard deviation might have been expected for the CMWS to permit inference about the ability to estimate average warning time. Therefore, this approach has the advantage of being a quantifiable answer to an important and related question.

Note that although there is no user requirement for estimates of average warning time at the threat level at typical levels of statistical confidence, this test design can produce estimates with reasonable levels of statistical confidence at that level. The analogous table for 20 missile shots is:

By Each Threat: Total Missiles 20

Number of Failures

Confidence

0

99%

1

93%

2

79%

5  

The value for x was not provided to enable the above documents to be unclassified.

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

Once the experiment is carried out, regression models will be fit for three purposes (note that it is unlikely, given the scaleable grid, that one could develop a single mathematical model that would be useful across threats):

  1. to compute confidence bounds on the average missile warning time for a particular threat,

  2. to investigate performance of the CMWS in alternative scenarios for a given threat within the test domain for that threat, and

  3. as input to a digital model to examine the impact of missile warning time on the success of the total mission.

An open question is whether the test size could be reduced by making use of preliminary versions of the regression models that will be developed, since these models will produce estimated average warning times that, with some reasonable assumptions, will have lower variances than one could get through direct measurement for an individual threat. Or, one could instead maintain the variance of the estimated average warning time while reducing the number of required test shots, through use of a design that would have permitted estimation of the four regression models.

SELECTED SYSTEM DESCRIPTIONS

This section contains brief descriptions of several of the remaining systems mentioned in the report. The panel's objective is to provide readers not actively involved in the defense community a basic understanding of these systems that can be used when interpreting the discussion and related examples in the report.

Air Defense Antitank System

Don Richardson, Institute for Defense Analyses

The Air Defense Antitank System (ADATS) is a short-range air defense weapon that was part of the Army's follow-on program to the Sgt. York division air defense gun. ADATS was intended to provide air defense protection to the Army's forward maneuver forces of tanks and armored vehicles. The ADATS system consisted of eight laser-guided missiles, a search and tracking radar, and infrared and optical sensors.

Advanced Amphibious Assault Vehicle

U.S. Department of Defense, 1998b

The Advanced Amphibious Assault Vehicle (AAAV) [a USMC ACAT ID program] is a high water-speed amphibious armored personnel carrier to replace the current family of Marine Corps assault amphibians, the AAV7A 1 series. . . . Armed with a medium-caliber machine gun and a cannon of 25-35 mm, the

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

AAAV will use GPS, a forward-looking infrared radar, and a night vision system for navigation, targeting, and intelligence gathering.

The AAAV must operate in all climates, over all terrain, and in all weather and lighting conditions. At sea, it must achieve a water speed of 20 knots in 3-ft significant wave height and cross a surf zone characterized by up to 8-ft plunging surf. The 20-knot water speed is significantly greater than the 7-knot speed provided by the current AAV7A1 amphibious tractor. . . .

The AAAV's land mobility characteristics must be comparable to the Marine Corps' M1A1 Abrams main battle tank. This requires a top speed of approximately 45 miles per hour, the capability to traverse the same terrain at the same speed as the tank during cross-country operations, and the capability to cross the same obstacles and terrain features (for example, trenches, hills, walls, and soft soils) as the tank.

AH-1 Cobra

Arthur Fries, Institute for Defense Analyses

The Bell AH-1 Cobra attack helicopter was first delivered to the Army in the summer of 1967. This single engine, twin-bladed aircraft was a modification of the UH-1 Iroquois. To reduce its susceptibility to detection and engagement by enemy forces, the width of the airframe was reduced. Hence the two man crew occupied tandem seats, vice the side-by-side seating arrangement in the UH-1. Two stub-wing pods were capable of carrying combinations of 2.75 inch rockets, mini-guns, and TOW anti-tank missiles. It was the Army's primary attack helicopter during the Vietnam conflict.

AQUILA

Arthur Fries, Institute for Defense Analyses

The AQUILA was a remotely piloted air vehicle (RPV) system developed by the Army in the 1980s and terminated late in that decade. It was designed to perform reconnaissance, target acquisition, artillery fire adjustment, and target designation for laser-guided munitions such as Copperhead artillery rounds and HELLFIRE missiles. The concept of operations was for the RPV to penetrate enemy territory 20 to 30 kilometers, where it might be acquired and engaged, or countered, by enemy systems such as air defense units or enemy radio-frequency jammers.

AN/ALQ-165 Airborne Self Protection Jammer

U.S. Department of Defense, 1998b

The AN/ALQ-165 Airborne Self Protection Jammer (ASPJ) [a canceled Navy ACAT I program] is an automated modular reprogrammable active radar frequency (RF) deception jammer designed to contribute to the electronic self

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

protection of the host tactical aircraft from a variety of air-to-air and surface-to-air RF threats. The ASPJ was designed to accomplish threat sorting, threat identification, and jamming management in a dense signal environment to counter multiple threats. The modular architecture supports internal integration with other avionics/weapons systems in a variety of aircraft. The basic system consists of five weapons replaceable assemblies/line replaceable units (WRAs/LRUs) which include two receivers, two transmitters, and one processor. Each WRA is interchangeable among different tactical aircraft. Additional transmitters can be installed on aircraft with larger radar cross sections to increase the effective radiated power.

C-17 Airlift Aircraft

U.S. Department of Defense, 1998b

The C-17 [an Air Force ACAT ID program] is a four engine turbofan aircraft capable of airlifting large payloads over intercontinental ranges without refueling. Its design is intended to allow delivery of outsize combat cargo and equipment directly into austere airfields. The C-17 will deliver passengers and cargo over intercontinental distances, provide theater and strategic airlift in both airland and airdrop modes, and augment aeromedical evacuation and special operations missions.

CH-46

Navy Fact Filehttp://www.chinfo.navy.mil/navpalib/factfile/aircraft/air-ch46.html

The CH-46 Sea Knight was first procured in 1964 to meet the medium-lift requirements of the Marine Corps in all combat and peacetime environments since that time.

[It is a] medium lift assault helicopter, primarily used to move cargo and troops. The CH-46D Sea Knight helicopter is used by the Navy for shipboard delivery of cargo and personnel. The CH-46E is used by the Marine Corps to provide all-weather, day-or-night assault transport of combat troops, supplies, and equipment. Troop assault is the primary function and the movement of supplies and equipment is secondary.

CH-47 Chinook

Arthur Fries, Institute for Defense Analyses

The Boeing CH-47 is a twin engine, medium lift helicopter. Each engine drives a three-bladed rotor. The CH-47A was first introduced into the Army in late 1962. Subsequently there have been three major modifications resulting in the CH-47B, C and D. It is used to lift both cargo and personnel. Cargo loads of

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

8 to 10,000 pounds can be carried internally or externally. CH-47s are normally found in corps level aviation support units, and provide most of the tactical cargo airlift within the corps and division areas.

CH-54 Skycrane

Arthur Fries, Institute for Defense Analyses

The Sikorsky CH-54 is the Army's heavy lift helicopter. Initial deliveries of this aircraft were made in late 1964. It is capable of lifting loads of 15 to 20,000 pounds. In Vietnam it was used to lift heavy equipment, such as dozers and lightly armored vehicles. Very few CH-54s were fielded, and none currently reside in the active forces.

Dragon

Rosser Bobbitt, Institute for Defense Analyses

The Dragon is a man-portable medium anti-tank missile system with command to line-of-sight guidance (gunner holds cross-hairs on target). Its maximum range is about 1,000 meters against the last, but not current, generation of threat main battle tanks. Recent modifications have extended its range to 1,500 meters and increased its penetration of the current generation of main battle tanks. It is being replaced in the active component of the U.S. Army by the Javelin anti-tank missile system with longer range and a fire-and-forget capability.

Family of Medium Tactical Vehicles

U.S. Department of Defense, 1998b

The Family of Medium Tactical Vehicles (FMTV) consists of fourteen wheeled tactical vehicles based on a common truck cab, chassis, and internal components and two tactical trailers. The components are primarily non-developmental items integrated in rugged tactical configurations. The light-medium tactical vehicles (LMTV) are 2.5-ton payload capacity models consisting of cargo, air drop cargo, and van variants. The medium tactical vehicles (MTV) are 5-ton payload capacity models consisting of cargo (with and without material handling crane), air drop cargo, tractor, wrecker, dump, air drop dump, fuel tanker, and expansible van variants.

FMTV supports Joint Vision objectives: focuses logistics through the transport of troops, water and ammunition distribution, and general cargo transport; information superiority through the provision of mobility to the new generation of automated systems, sophisticated management information systems, and communications links; and precision engagement as the prime mover for towed artil

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

lery and Patriot and as the chassis for the High Mobility Artillery Rocket System (HIMARS).

Javelin Antitank Missile

U.S. Department of Defense, 1998b

The Javelin [an Army ACAT ID program] is a manportable, fire-and-forget, antitank missile employed by dismounted infantry to defeat current and future threat armored combat vehicles. Javelin is intended to replace the Dragon system in the Army and the Marine Corps.

The Javelin consists of a missile in a disposable launch tube and a reusable Command Launch Unit with a trigger mechanism and day/night sighting device for surveillance, and target acquisition and built-in test capabilities. The missile locks on to the target before launch using an infrared focal plane array and onboard processing, which also maintains target track and guides the missile to the target after launch.

Joint Surveillance Target Attack Radar System

U.S. Department of Defense, 1998b

The Joint Surveillance Target Attack Radar System (JSTARS) [an Air Force ACAT ID program, contributes] a synoptic battlefield view to operational maneuver commanders. The system [is required] to perform battlefield surveillance, battle management for both air and land component forces, and indications and warning functions. . . . The JSTARS system is intended to meet the operational need for locating, classifying, and supporting precision engagement of time-sensitive moving and stationary targets.

The JSTARS system consists of the Air Force E-8C aircraft, an Army ground station, and the data link that connects the two elements. . . . The JSTARS system brings to the battlefield the technical capability to perform surveillance through interleaved high resolution synthetic aperture radar (SAR), moving target indicator (MTI), and the computer capability to integrate battlefield and geographic information into a near real-time picture of the ground battle.

M1 Abrams Tank

Bernard Kempinski, Institute for Defense Analyses

M1, M1A1 and M1A2 are versions of the Army's Abrams main battle tank. The Abrams is [a] tracked armored vehicle mounting either a 105mm or 120mm cannon, three machine guns and four crew men. Its mission is to close with and destroy enemy armor and troops. It is heavily protected and designed to absorb a high level of damage.

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
OH-58D Kiowa Warrior

U.S. Department of Defense, 1997

The OH-58D Kiowa Warrior [an Army ACAT IC program] is a two-[pilot,] single engine armed reconnaissance helicopter. . . . The principal difference between the Kiowa Warrior and its immediate OH-58D predecessor is a universal weapons pylon on both sides of the aircraft capable of accepting combinations of the semi-active laser Hellfire missile, the Air-to-Air Stinger (ATAS) missile, 2.75'' Folding Fin Aerial Rocket (FFAR) pods, and a 0.50 caliber machine gun. In addition to these weapons, the Kiowa Warrior upgrade includes changes designed to provide improvements in air-to-air and air-to-ground communications, mission planning and management, available power, survivability, night flying, and reductions in crew workload through the use of on-board automation and cockpit integration.

The primary mission of the Kiowa Warrior is armed reconnaissance in air cavalry troops and light attack companies. In addition, the Kiowa Warrior may be called upon to participate in the following missions or tasks: (1) Joint Air Attack (JAAT) operations; (2) air combat; (3) limited attack operations; and (4) artillery target designation.

Sensor Fuzed Weapon

U.S. Department of Defense, 1998b

The CBU-97/B Sensor Fuzed Weapon (SFW) [an Air Force ACAT ID program] is an anti-armor cluster munition to be employed by fighter/attack and bomber aircraft to provide . . . multiple kills per pass against armored and support vehicle combat formations. . . . SFW is currently delivered as an unguided, gravity weapon. After release, the TMD opens and dispenses the ten submunitions which are parachute stabilized. At a preset altitude sensed by a radar altimeter, a rocket motor fires to spin the submunition and initiate an ascent. The submunition then releases its four projectiles, which are lofted over the target area. The projectile's sensor detects a vehicle's infrared signature, and an explosively formed penetrator fires at the heat source.

Sergeant York

Elliot Parkin, Institute for Defense Analyses

The Sergeant York, a radar controlled twin 40 mm air defense gun for defense of armored forces, was evaluated by the Army during an April 1985 operational test at Fort Hunter Liggett, California. The operational test consisted of 30-minute force-on-force battles in which the Sergeant York defended an ar

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×

mored unit, conducting offensive or defensive operations, from enemy air attack. During these mock-combat battles, the Sergeant York could not effectively engage the standoff helicopted threat that enemy force successfully employed beyond the range of the Sergeant York's guns. The $4 billion program was subsequently canceled with the government writing off the $1.8 billion already spent before testing. The 65 systems that had already been delivered to the Army were scrapped.

Stinger

U.S. Department of Defense, 1998b

The Stinger missile . . . is the Army's system for short-range air defense that provides the ground maneuver commander force protection against low-altitude airborne targets such as fixed-wing aircraft, helicopters, unmanned aerial vehicles, and cruise missiles. The Stinger is launched from a number of platforms: Bradley Stinger Fighting Vehicle, Bradley Linebacker, Avenger (HMMWV), and helicopters as well as Man Portable Air Defense (MANPADS).

TOW

Rosser Bobbit, Institute of Defense Analyses

The TOW is an optically tracked wire-guided heavy anti-tank missile system introduced into the U.S. Army during the Vietnam War. It is fired from a ground mount, the High Mobility Multi-Purpose Vehicle (Hummer), Bradley Fighting Vehicle, and Cobra Attack Helicopters. Through continuous modification it has been kept up to date against current threat main battle tanks. It is being replaced by the Follow-on-to-TOW missile system, very similar in concept but with longer range and a fire-and-forget capability.

UH-1 Iroquois (Huey)

Arthur Fries, Institute of Defense Analyses

The UH-1 is a single engine, twin-bladed transport helicopter built by Bell Helicopter. It was first fielded in the Army in 1959, and served as its primary troop and light cargo lift helicopter until the subsequent introduction of the UH-60 Blackhawk. The UH-1 had a crew of three, including the crew chief, and seats for approximately six passengers. Thus, two Huey's could lift a full strength rifle squad. It could carry either internal or external cargo loads. It was the primary tactical lift carrier for the Army during the Vietnam conflict. It has also served as an airborne command and control aircraft.

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
UH-60 Black Hawk

U.S. Department of Defense, 1998b

The UH-60 Black Hawk [an Army ACAT IC program] is a single rotor medium-lift helicopter powered by twin General Electric T700-GE-701C turboshaft engines rated at 1,700 shp each. The Black Hawk helicopter provides utility and assault lift capability across a wide range of missions. The Black Hawk is the primary helicopter for air assault, general support, and aeromedical evacuation units. In addition, modified Black Hawks operate as command and control, electronic warfare, and special operations platforms.

Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 165
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 166
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 167
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 168
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 169
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 170
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 171
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 172
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 173
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 174
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 175
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 176
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 177
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 178
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 179
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 180
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 181
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 182
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 183
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 184
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 185
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 186
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 187
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 188
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 189
Suggested Citation:"Appendix A Case Studies and System Descriptions." National Research Council. 1998. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. Washington, DC: The National Academies Press. doi: 10.17226/6037.
×
Page 190
Next: Appendix B Abstracts of Background Papers »
Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements Get This Book
×
Buy Paperback | $57.00 Buy Ebook | $45.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

For every weapons system being developed, the U.S. Department of Defense (DOD) must make a critical decision: Should the system go forward to full-scale production? The answer to that question may involve not only tens of billions of dollars but also the nation's security and military capabilities. In the milestone process used by DOD to answer the basic acquisition question, one component near the end of the process is operational testing, to determine if a system meets the requirements for effectiveness and suitability in realistic battlefield settings. Problems discovered at this stage can cause significant production delays and can necessitate costly system redesign.

This book examines the milestone process, as well as the DOD's entire approach to testing and evaluating defense systems. It brings to the topic of defense acquisition the application of scientific statistical principles and practices.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!