Page 30

4

Evaluation Methodology

There is no formula or specific set of criteria on which to base a determination of the point at which the putative military utility of AP mines is exceeded by the humanitarian costs of their employment. This is a complex problem involving subjective judgment; for a nation to deny its military the use of a weapon that could assist in prevailing against opposing forces while reducing friendly casualties, there must be a compelling case that the weapon is not indispensable to legitimate military operations and that its utility is clearly outweighed by the resultant killing and wounding of innocent civilians. (Gard, 1999)

The committee considered a large number of potential alternatives to APL. Some are under development as part of the U.S. government's three-track initiative, and some are being developed by other nations. Still others were suggested by nongovernmental organizations and independent scientists committed to the elimination of APL. Members of the committee also suggested several possible alternatives; some are improvements to existing systems and some totally new. This chapter describes the methodology, baseline systems, criteria, and performance factors the committee used to rank potential alternatives.

METHODOLOGY

Information about the alternatives described in this report was gathered over a 10-month period. The sponsor, in meetings with the committee, stressed that all types of alternatives should be considered: materiel and nonmateriel, as well as existing, developmental, conceptual, and notional technologies. Although the committee cast as wide a net as possible, some potential alternatives may have been missed. At the time the data-gathering phase of the study ended, the Track III initiative had reduced the number of concepts under consideration to seven finalists. Three of these were undergoing concept evaluation by industry and were considered competition sensitive; therefore, these were not considered in this study.

Although independent modeling or simulations may have assisted the committee in its evaluations, this was not feasible within the committee's constraints of time and budget. Therefore, the committee developed a score sheet to measure the effectiveness of alternatives. Although the analysis is admittedly subjective, and the results are qualitative rather than quantitative, the scoring was helpful for ranking the alternatives.

The criteria for the score sheet were developed by the entire committee. Committee members with technical expertise provided detailed write-ups of each of the candidates. A subcommittee of technical and military operational experts then met and scored the alternatives against the criteria. The results were presented to the full committee, and adjustments were made, as necessary. The score sheets resulting from these assessments are provided in Chapter 5, Chapter 6, and Chapter 7.

The committee adopted the terminology used in previous studies by DOD to categorize types of alternatives. Materiel alternatives are weapon-like systems; nonmateriel alternatives are nonweapon solutions, such as changes in doctrine, organization, concepts, and training or weapons designed for another purpose. Only the materiel alternatives were scored.

BASELINE SYSTEMS

There are many very different types of mines. Mines used solely against a dismounted enemy are so different from the mine systems used against mounted targets that the committee divided its alternatives into two categories: those to be used against a dismounted threat and those to be used against a mounted threat. It then selected two baseline systems against which candidate systems could be measured for all criteria: the M14/16 nonself-destructing mines for use against a dismounted threat and the Volcano system (M87), a mixed system, for use against a mounted threat.

CRITERIA

The committee developed the criteria as a qualitative guide for assessing how well alternatives would fulfill the requirements for APL. These are availability, military effects, humanitarian concerns, overall technical risk, tactics and doctrine, and economic factors. The first three sets of criteria were most important to the committee's overall assessments and recommendations. The committee reiterates that the scores are qualitative and, therefore, somewhat subjective. They were used to help organize the committee's thinking, but not as a basis for pass/fail judgments.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 30
Page 30 4 Evaluation Methodology There is no formula or specific set of criteria on which to base a determination of the point at which the putative military utility of AP mines is exceeded by the humanitarian costs of their employment. This is a complex problem involving subjective judgment; for a nation to deny its military the use of a weapon that could assist in prevailing against opposing forces while reducing friendly casualties, there must be a compelling case that the weapon is not indispensable to legitimate military operations and that its utility is clearly outweighed by the resultant killing and wounding of innocent civilians. (Gard, 1999) The committee considered a large number of potential alternatives to APL. Some are under development as part of the U.S. government's three-track initiative, and some are being developed by other nations. Still others were suggested by nongovernmental organizations and independent scientists committed to the elimination of APL. Members of the committee also suggested several possible alternatives; some are improvements to existing systems and some totally new. This chapter describes the methodology, baseline systems, criteria, and performance factors the committee used to rank potential alternatives. METHODOLOGY Information about the alternatives described in this report was gathered over a 10-month period. The sponsor, in meetings with the committee, stressed that all types of alternatives should be considered: materiel and nonmateriel, as well as existing, developmental, conceptual, and notional technologies. Although the committee cast as wide a net as possible, some potential alternatives may have been missed. At the time the data-gathering phase of the study ended, the Track III initiative had reduced the number of concepts under consideration to seven finalists. Three of these were undergoing concept evaluation by industry and were considered competition sensitive; therefore, these were not considered in this study. Although independent modeling or simulations may have assisted the committee in its evaluations, this was not feasible within the committee's constraints of time and budget. Therefore, the committee developed a score sheet to measure the effectiveness of alternatives. Although the analysis is admittedly subjective, and the results are qualitative rather than quantitative, the scoring was helpful for ranking the alternatives. The criteria for the score sheet were developed by the entire committee. Committee members with technical expertise provided detailed write-ups of each of the candidates. A subcommittee of technical and military operational experts then met and scored the alternatives against the criteria. The results were presented to the full committee, and adjustments were made, as necessary. The score sheets resulting from these assessments are provided in Chapter 5, Chapter 6, and Chapter 7. The committee adopted the terminology used in previous studies by DOD to categorize types of alternatives. Materiel alternatives are weapon-like systems; nonmateriel alternatives are nonweapon solutions, such as changes in doctrine, organization, concepts, and training or weapons designed for another purpose. Only the materiel alternatives were scored. BASELINE SYSTEMS There are many very different types of mines. Mines used solely against a dismounted enemy are so different from the mine systems used against mounted targets that the committee divided its alternatives into two categories: those to be used against a dismounted threat and those to be used against a mounted threat. It then selected two baseline systems against which candidate systems could be measured for all criteria: the M14/16 nonself-destructing mines for use against a dismounted threat and the Volcano system (M87), a mixed system, for use against a mounted threat. CRITERIA The committee developed the criteria as a qualitative guide for assessing how well alternatives would fulfill the requirements for APL. These are availability, military effects, humanitarian concerns, overall technical risk, tactics and doctrine, and economic factors. The first three sets of criteria were most important to the committee's overall assessments and recommendations. The committee reiterates that the scores are qualitative and, therefore, somewhat subjective. They were used to help organize the committee's thinking, but not as a basis for pass/fail judgments.

OCR for page 30
Page 31 Availability for Implementation The first and most important criterion was the availability of the system, in keeping with the requirement in the Statement of Task that the committee identify technologies that would be available by 2006. Although 2006 was the critical milestone, many emerging technologies that most likely would not be available until after that date showed considerable promise. Ultimately, the committee divided the potential alternatives into three groups: systems available now (see Appendix C and Chapter 5 for detailed descriptions); systems currently under development that have a reasonable likelihood of being available by 2006 ( Chapter 6); and technologies unlikely to be available until after 2006 ( Chapter 7). Military Effects The second criterion was whether the alternative system could fulfill DOD's requirements (i.e., be militarily effective). The committee's choice of the criterion was neither arbitrary nor capricious. After several attempts to design military criteria for landmines, the committee determined that the capabilities described in the mission need statements1 were the most complete description available. A mission need statement is one of the first documents to describe the requirement for a new piece of military equipment. The mission need statements used in this study were validated by the Joint Requirements Oversight Council, a high-level element of the Office of the Joint Chiefs of Staff, and were considered adequate by DOD for industry to use as a basis for designing alternatives. Therefore, the committee considered them to be an accurate reflection of military service requirements. Although the performance standards in these documents are at a higher level than the requirements for technologies described in the study's Statement of Task (i.e., they would perform better than current APL, not similarly), they provide the most complete description of the requirements for mine alternatives known to the committee; an approved operational requirements document was not available. The standards can be read in their entirety in Appendix G. The numbering of each criterion coincides with the numbers on the score sheets. A score of zero shows that the alternative performed at the same level as the baseline system. A positive number indicates that the alternative performed better than the baseline system. A negative number shows that the alternative did not perform as well as the baseline system. Against Mounted Threats The Mixed Landmine System Alternatives mission need statement describes requirements for a mixed system for use against a mounted threat (see Appendix G). The committee used the Volcano (M87) as the baseline system against which alternatives were judged. A1 Enhance effects of close and deep friendly fires 0 = same as Volcano 1 = greater coverage than Volcano and/or technological improvements 2 = man-in-the-loop or improved sensors 3 = sensors with battlefield awareness −1 = nonlethal component or easily detectable −2 = longer sensor-to-shooter cycle −3 = longer sensor-to-shooter cycle and a small footprint −4 = longer shooter-to-sensor cycle, a small footprint, and may require a cease fire from other weapons to protect the delivery platform A2 Has multiple methods of delivery 0 = at least two methods of delivery (as many or more than Volcano) −1 = only one method of delivery A3 Provides a range of effects that inhibit mounted and dismounted maneuvers 0 = similar to Volcano, has both AT and APL components 1 = sensor that enables use of something other than an APL mine to counter dismounted targets 2 = multiple sensors/highly sophisticated sensors −1 = inhibits only personnel or only tanks; nonlethal against personnel −2 = nothing in place on ground to stop or slow enemy movement A4 Resists full spectrum of enemy breach methods, including dismounted methods 0 = has AP and AT components 1 = has man-in-the-loop to react to visual observation of breach 2 = unbreachable without great risk to troops; very difficult to breach 3 = multiple sensors, allowing other weapons to be brought to bear; larger area covered; automatically counters breach attempts −1 = nonlethal AP components; some AT components have antihandling devices −2 = has only AT or only AP N/A=no minefield to be breached A5 Provides early warning of ground attack 0 = might provide early warning within normal observation range 1 = will provide early warning because of man-in-the-loop (observes ground attack) 1 The two current mission need statements that relate to APL are Battlefield Shaping and Force Protection against Personnel Threats (alternatives to APL when used alone) and Mixed Landmine Systems Alternatives (alternatives for current mixed systems, combining both APL and AT mines) (see Appendix G for full text).

OCR for page 30
Page 32 2 = will provide reliable early warning via sensors at greater distance to observe attack −1 = will not provide early warning within normal observation range A6 Is safe for friendly forces All alternatives approved in the mission need statements will be as safe as Volcano by doctrine. DOD will not develop a weapon that poses a threat to its own forces. 0 = as safe as Volcano A7 Is effective in all types of terrain and weather 0 = terrain and weather have no effect −1 = cannot see target because of weather/terrain; reaction requires sensor input that can be affected by weather/terrain A8 Poses minimal residual hazard to own forces and noncombatants after military conflicts 0 = can self-destruct/self-deactivate; no ground munitions to explode (munitions have exploded in the air or on contact with the ground) 1 = can be command detonated; is strictly nonlethal A9 Is difficult to detect by enemy forces 0 = same as Volcano; may be visible when shot off but are not easily visible on ground 1 = smaller and less easily seen than Volcano 2 = nothing to be detected on the ground −1 = bigger, easier to see; sends a signal that can be detected −2 = visible and audible by the enemy before breaching A10 Poses minimal risk of fratricide 0 = has an APL component 1 = has no APL component; has nonlethal AP component 2 = man-in-the-loop allows identification of friend or foe before detonation; more accurate minefield location 3 = sensors tell own troops where munitions are; munitions can be turned off when own troops approaching; is strictly nonlethal A11 Effects modifiable for mounted and/or dismounted threat 0 = once in place munitions are either AT or AP 1 = has man-in-the-loop, therefore other systems can be used 2 = more sensors allow several options for bringing other weapons to bear A12 Has controllable activation/deactivation and duration before and after installation. 0 = set before emplacement, cannot be changed after 1 = set at time of use; man-in-the-loop 2 = can do many things with munitions (turn on/off; command destruct) A13 Is effective in nuclear, chemical, and biological environments 0 = not affected (same as Volcano) −1 = sensors or communications may be affected by electromagnetic pulse A14 Is easy and efficient to distribute 0 = remotely delivered; AT and AP components delivered together −1 = emplaced by hand; components delivered separately (i.e., requires two rounds) −2 = more components require hand emplacement than Volcano Against Dismounted Threats The Battlefield Shaping and Force Protection Against Personnel Threats mission need statement describes the capabilities required for protection against a dismounted threat (see Appendix G). The committee used the M14/M16 as the baseline system for evaluating these alternatives. A15 Can delay, disrupt, and/or canalize enemy movement/ maneuvers. 0 = same as M14/M16 1 = greater area covered; greater range; selection of when to detonate; line of sight 2 = detection of enemy forces further out via sensors allow time for use of other weapons −1 = nonlethal with strong effects that might affect enemy movement −2 = nonlethal with weaker effects not likely to affect enemy movement A16 Denies enemy access to terrain/facilities (including short-term/long-term deterrent for boundaries and DMZ areas) 0 = same as M14/M16 1 = greater area covered; greater range; selection of when to detonate; line of sight required 2 = detection of enemy forces further out via sensors, allowing time for use of other weapons −1 = nonlethal, with strong effects that might affect enemy movement −2 = nonlethal, with weaker effects that are not likely to affect enemy movement A17 Enhances effects of friendly force weapons, obstacles, and munitions (including AT mines) 0 = same as M14/M16 1 = greater coverage; man-in-the-loop 2 = better sensors −1 = nonlethal A18 Generates exploitable delays and opportunities (fixes or contains enemy)

OCR for page 30
Page 33 0 = same as M14/M16 1 = greater area covered; greater range; selection of when to detonate; line of sight required 2 = detection of enemy forces further out via sensors allowing time for use of other weapons −1 = nonlethal, with strong effects that might affect enemy movement −2 = nonlethal, with weaker effects that are not likely to affect enemy movement A19 Generates detection, alert, and/or early warning 0 = makes a noise, alerting friendly troops to enemy intrusion 1 = sensors alert to enemy intrusion 2 = more sophisticated sensors alerting of intrusion at greater distance (sooner) −1 = no noise is made autonomously; line of sight required; difficult to cover blind spots A20 Facilitates classification of target 0 = no classification 1 = man-in-the-loop 2 = man-in-the-loop with sophisticated sensor A21 Produces desired effects on enemy forces (nonlethal to lethal) 0 = solely lethal or nonlethal; no munition 1 = permits choice of munition (lethal or nonlethal) 2 = has inherent selection of effects, from lethal to nonlethal A22 Reduces casualties/risk for U.S. and/or allied forces 0 = same as M14/M16 1 = self-destruct/self-deactivating or man-in-the-loop 2 = nonlethal; more positive identification at greater ranges A23 Deters pursuit to facilitate breaking of contact under pressure 0 = no man-in-the-loop, weapon does not require operator −1 = man-in-the-loop; nonlethal Humanitarian Concerns The third criterion was fulfillment of the humanitarian intent of international agreements, such as the CCW and the Ottawa Convention. Although compliance with these agreements was not specifically included in the Statement of Task, the committee considered it implicit in the rationale for this study. Senator Leahy, a strong supporter of detmining, providing aid for victims of mines, and the Ottawa Convention, introduced legislation requesting this study (Rieser, 1999). Because humanitarian concerns were clearly one impetus for this study, the committee used those elements from the CCW and the Ottawa Convention that addressed the post conflict hazard of mines and possible injuries to noncombatants as humanitarian criteria. The five characteristics listed below, derived from the CCW and the Ottawa Convention, were applied to alternatives used to counter mounted and dismounted targets. Against Mounted Threats B1 Presents postconflict residual hazard (CCW) 0 = no self-destruct/self-deactivation mechanism; munitions persist on ground 1 = has self-destruct/self-deactivation; easily removable; purely nonlethal; no munitions persisting on ground B2 Location can be recorded (CCW) 0 = approximate locations are known 1 = more precise location known 2 = will self-record where they are 3 = no munition B3 Detectable to facilitate removal (CCW) 0 = sufficient metal content for detection 1 = no munitions B4 Discriminates between combatants and civilians (Ottawa) 0 = has no means of discrimination 1 = less than positive target identification (i.e., with a nonvisual sensor); observed fire 2 = discriminates by man-in-the-loop, visual identification; has nonlethal AP component B5 Does not explode on presence, proximity, or contact of a person (Ottawa) 0 = explodes on presence, proximity, or contact 1 = does not explode on presence, proximity, or contact; is nonlethal Against Dismounted Threats B1 Presents residual hazard (CCW) 0 = no self-destruct/self-deactivation mechanism 1 = is self-removing (via self-destruction/self-deactivation) or nonhazardous after war; is very easily removable; is strictly nonlethal B2 Location can be recorded (CCW) 0 = is hand emplaced 1 = has man-in-the-loop or sensor B3 Detectable to facilitate removal (CCW) 0 = sufficient metal content for detection 1 = no munitions B4 Discriminates between combatants and civilians (Ottawa) 0 = has no means of discrimination 1 = less than positive target identification (i.e., with a nonvisual sensor) 2 = discriminates by man-in-the-loop, visual identification

OCR for page 30
Page 34 B5 Does not explode on presence, proximity, or contact of a person (Ottawa) 0 = explodes on presence, proximity, or contact 1 = does not explode on presence, proximity, or contact; is strictly nonlethal Overall Technical Risk In assessing overall technical risk, the committee assigned the highest score to systems already in production and progressively lower scores to systems with technologies beyond the state of the art. This very simple scoring was helpful for assessing the more “futuristic” alternatives. C0 0 = in production −1 = capability has been demonstrated −2 = capability not prototyped, but uses technology within the state of the art −3 = technology is beyond the state of the art Changes in Tactics and Doctrine The consideration of tactics and operational concepts was specified in the Statement of Task. This criterion was given a simple yes or no on the score sheet to indicate whether changes in tactics or doctrine would be necessary to implement the alternative (see D0 on score sheet). Cost Because cost2 was not included in the Statement of Task and because the determinations by the committee were only estimates, scores for this criterion only indicate the committee's evaluation process. The following costs were considered for each alternative: E1 Research and development 0 = weapon already exists −1 = known, funded cost; relatively low cost; prototype exists or is technologically easy to achieve −2 = no prototype, but elements of the system exist; technology appears to be straightforward −3 = conceptual stage; requires a technology breakthrough E2 Procurement cost 0 = production base exists/has been produced −1 = production capability exists, anticipated product costs are low −2 = requires complex manufacturing process and/or high product cost −3 = manufacturing processes are conceptual, costs unknown 2 Despite the sponsor's admonition that cost not be considered a factor, it was included because the committee believed it might be a consideration for the future. However, cost estimates were rudimentary.