National Academies Press: OpenBook

Improving Metrics for the Department of Defense Cooperative Threat Reduction Program (2012)

Chapter: 2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report

« Previous: 1 Introduction
Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

2

Committee Assessment of the Department of Defense Cooperative
Threat Reduction Metrics Report

The first step in developing metrics is to establish what questions the metrics are meant to answer, and how the metrics will be used. Throughout this study, in conversations with the Department of Defense (DoD), other agencies, staff on Capitol Hill, and in the Executive Office of the President, and with officials in partner countries, the committee asked how the metrics will be used. The answers ranged based on the respondent, from those who sought measures of progress toward agreed goals, to cost effectiveness, to others who sought measures of how well the efforts align with the national defense mission. Given that different audiences want metrics for different purposes, it makes sense to begin by describing the committee’s perspective on the purposes of metrics.

Metrics are tools for evaluating the impact and effectiveness of programs and projects against strategic goals and for management within the program. The former are program metrics. The latter are project metrics. Different metrics may be appropriate for different stages of program and project development, but they all need to be tied together by the strategic objectives of the program as they fit with broader DoD Cooperative Threat Reduction objectives.

The committee evaluated the DoD CTR metrics described in the (CTR) DoD Metrics Report based on whether the metrics provide decision makers with the essential information to manage the effectiveness and impact of CTR programs. The committee drew on several frameworks for program planning, performance, and evaluation, but especially on capability based planning, to identify general steps needed for meaningful evaluation.15 Other frameworks are also valid and will either explicitly or implicitly include these elements. DoD must

 

•   state the objectives of the program and the project (i.e., the goals of the actual activities)

•   identify the capabilities it is trying to develop or maintain

•   define objectives for each capability and link those capabilities to metrics

•   ensure that the metrics reflect program effectiveness and impact

•   plan for and measure sustainment (see footnote 3 in the Summary)

It is generally good practice for the program to establish minimum performance levels and aspirational goals for each metric (see footnote 4 in the Summary). In Table 2-1, the committee

image

15 A much more elaborate metrics framework is emerging from analysts focused on the international development community. This highly structured approach treats the portfolio of projects as an experimental trial, with control projects or communities and strict adherence to a protocol established at the outset of the program or project. The committee chose a program evaluation framework more like the program planning, performance, and evaluation approaches already used in DoD because the latter are more familiar to DoD and, in the committee’s view, better suited to the CTR Program. The CTR Program engages in too few projects for meaningful control groups, and the political conditions, physical and social circumstances, budgets, and objectives for the program change, all of which make a trial-based approach incompatible with CTR today.

Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

TABLE 2-1 Summary of metrics from the DoD Metrics Report

CTR Program

What metrics are intended to measure

Metrics

Cooperative Biological Engagement Program (CBEP)

1. Secure and consolidate collections of especially dangerous pathogens (EDP) and their associated research at a minimum number of secure health and agricultural laboratories or related facilities.

2. Enhance partner country/region’s capability to prevent the sale, theft, diversion, or accidental release of biological weapons (BW)-related materials, technology, and expertise by improving biological safety and security (BS&S) standards and procedures.

3. Enhance partner country/region’s capability to detect, diagnose, and report endemic and epidemic, man-made or natural EDPs, bio-terror attacks, and potential pandemics.

4. Ensure the developed capabilities are designed to be sustainable within each partner country/region’s current operating budget.

5. Facilitate the engagement of partner country’s/regional scientific and technical personnel in research areas of interest to both the partner country/region and the United States.

6. Eliminate any BW-related infrastructure and technologies encountered in a partner/country region.

Examples of measures of effectivenessa

Partner country EDP collections and associated research are consolidated into a minimum number of locations.

Partner country EDPs and associated research are secured in a manner consistent with standards.

Partner country has BS&S laws and regulations governing work with EDPs.

Partner country disease detection and diagnosis capability meets U.S. and/or international guidelines for biosafety.

Partner country has preparedness and response plans.

Partner country disease surveillance system is capable of detecting and reporting suspect EDP cases to those responsible for human and animal health.

Chemical Weapons Elimination Program (CWE)

Destruction of agent through the Russian Federation’s safe, efficient operation of the destruction facility

Quantity of CW agent destroyed Number of munitions destroyed Project Metrics

Scheduled facility downtime

Unscheduled facility downtime

Facility achieved availability

Nuclear Weapons Safety and Security (NWSS) supports Nuclear Weapons Transport Security Program

1. Performance/capability assurance

2. Configuration management

3. Procedures and process

4. Training

5. Organization and personnel

6. Life-cycle management

7. Maintenance

8. Logistics

Examples of metrics

Number of nuclear weapons storage sites upgraded

Number of personnel trained

Establishment of regional technical centers

Note: Some metrics still under development

Weapons of Mass Destruction Proliferation Prevention Program (WMD-PPP)

1. Enhance partner country capability to perform effective risk management

2. Enhance partner country capability to perform border security command, control, communications and computers

3. Enhance partner country capability to perform border security surveillance

4. Enhance partner country capability to perform WMD detection

5. Enhance partner country capability to perform border security interdiction

6. Assist partner country with the development of a sustainment budget for all systems delivered under this program

7. Enhance partner country capability to support and maintain delivered equipment and/or systems

8. Enhance partner country capability to sustain delivered training

9. Enhance a partner country capability to capture and disseminate information including that concerning WMD incidents

10. Increase the awareness of partner countries as to their critical role in WMD non-proliferation (partner country buy-in)

11. Develop interagency, bilateral, regional, and multilateral cooperation

Examples of metrics

Miles of green (land) / blue (maritime) border with added security

Number of ports of entry enhanced

Number of border facilities (land and maritime) provided increased capability to detect WMD

Number of personnel trained and equipped to perform border security interdiction

Initial and refresher training system exists

Number of personnel exchanges (training, professional development)

Mean time to negotiate agreements

Number of regional relationships facilitated

a The terms “goal,” “objective,” “indicator,” “attribute,” and “metric” are used inconsistently among the different program descriptions in the DoD Metrics Report. CBEP uses the term “measure of effectiveness” to describe what is being measured and the term “indicator” to denote how it is being measured (i.e., what is being counted).

Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

attempts to summarize the disparate metrics from the DoD Metrics Report and Table 2-2 summarizes the committee’s assessments of DoD’s CTR Metrics Report with respect to the elements described above. Because of differences among the programs, inconsistencies in their terminologies, and the complexity of their metrics, the program metrics and indeed the assessments are difficult to represent in a uniform tabular structure, so the committee commends readers seeking deeper familiarity with the metrics to the DoD Metrics Report, which is reproduced in full in Appendix B. The committee did not address the Strategic Offensive Arms Elimination Program, which is not discussed in the DoD Metrics Report. DoD indicated that it plans to use long-standing “Nunn-Lugar Scorecard” metrics for that program.

METRICS AND OBJECTIVES

In its introduction, the DoD Metrics Report states that “We can measure the amount of equipment provided and the number of training events conducted or scientists engaged; however, we need better measures to show that these efforts actually result in changed practices or additional effectiveness.” The committee agrees and notes that the example given conflates inputs and project metrics (those that measure progress on the means to meet objectives) and outputs and programmatic or strategic metrics (those that measure progress relative to/toward the objectives themselves).

Effectiveness and efficiency are important criteria for the evaluation of government programs. A government program is effective if it achieves the objectives set forth by the partners engaged in the program.16 A government program is efficient if it minimizes the resources required to achieve its goals. A common approach is to manage and assess programs based on their cost-effectiveness, which is an efficiency measure. Although measures of effectiveness and impact are the most challenging and should continue to receive the most attention, measures of efficiency and cost-effectiveness are also necessary and should be included in future iterations of the CTR Program metrics.17

OVERALL ASSESSMENT

The DoD Metrics Report contains reasonable metrics for the CTR programs aimed at consolidation and elimination of weapons and weapons materials, and contains a reasonable starting point for developing metrics for the newer capacity-building programs.

image

16 It may not be possible to directly measure the higher-level outcomes from some CTR programs or projects, even with the best metrics. For example, DoD may never know how many illegal shipments were not interdicted at a border crossing where a CTR program provides assistance, or how many patients sick with an illness of interest did not go to the hospital. Recognizing this fact at the outset will help to avoid wasted time and effort and avoid false expectations.

17 While cost-effectiveness is important, particularly as a part of prioritization, it is essential that efficiency (which is relatively easy to measure) not drive out effectiveness (which is more difficult to measure). In other words, programs that are extremely cost-efficient may be pursuing the wrong result, or be solving the wrong problem. Balancing cost-effectiveness against other measures is a necessary part of developing prioritized metrics.

Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

Individual Program Objectives

The DoD Metrics Report describes both CTR’s highest-level objectives and DoD’s difficulties in developing metrics, and it does so clearly and succinctly in the introductory section of its report. However, the individual program objectives (those of the CWE, NWSS, CBEP and WMD-PPP) and their connections to threats to U.S. national security are not well articulated or not addressed. Each program has objectives, and they may be described well in other DoD documents (see, e.g., Nacht, 2009), but the DoD Metrics Report itself needs to articulate the objectives in a way that makes apparent the connection between the actions taken under the Program and the higher level objectives. Not making those connections poses a problem for the Program because proper metrics indicate progress toward objectives, or more specifically, progress in developing the capabilities to fulfill the objectives. Those connections are the logical starting point for the presentation and for the actual development and analysis of metrics. Again, this is not to say that there is no connection, but that the DoD Metrics Report does not describe the connection.

For example, describing the connection would clarify how building the capacity to better track respiratory disease in East Africa reduces threats to U.S. national security or why Ukraine’s interdiction of smuggled frozen chicken legs indicates a positive impact from a CTR program. A strong case can be made for each of these, but not without stating the Program objectives and connect them to program objectives.

For the programs that do not already have such objectives stated in the DoD Metrics Report, a concise statement of the objectives of each program and how the actions planned under the programs are intended to reduce threat or risk would provide a connection between the objectives and the metrics.

Partnership

CTR programs are intended to be and work best when they are carried out as partnerships (cooperative) with other countries. This includes joint development of the objectives with the partner countries. Most of the metrics in the DoD Metrics Report do not reflect such joint development. The report reads as if metrics are U.S. measurements of our partner’s progress toward U.S. goals. This can create misunderstandings between the United States and our partners, which can undermine CTR efforts.

Prioritization

DoD did not use a consistent framework for developing and articulating objectives, priorities, and metrics, and DoD did not prioritize among its metrics. As a consequence, the DoD Metrics Report mixes project management measures with higher-level program performance metrics for some of the CTR programs, and weights equally metrics that are critically important and others are not.

Cooperation

DoD plans to leverage other United States Government agencies’ experience, capabilities, and assets as CTR expands to new countries and as it continues existing programs.

Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

DoD also needs to communicate, coordinate, and cooperate with relevant agencies. Examination of other U.S. Government documents suggests that CBEP is the most consistently planned and coordinated across the other agencies.18 The committee notes that previous Academy reports and government agencies themselves have stated the importance of such coordination for program success, so a metric for it seems reasonable and valuable in theory. To the committee’s knowledge, there is no metric to reflect interagency coordination or a “whole of government” approach to working with a partner country, rather than a piecemeal approach. The committee does not have a solution to this problem (see Chapter 3 for further discussion).

In addition to working with these other agencies, DoD CTR can learn something from them. They, too, engage in capacity building programs, work on similar missions, and measure performance. U.S. Customs and Border Protection conducts a mission similar to the PPP along many thousands of kilometers of border and has developed metrics for its mission and operations. The U.S. Department of Agriculture’s Animal and Plant Health Inspection Service is a leader in an international surveillance network that has many parallels to the global network DoD leadership envisions for reducing biothreats. The U.S. Agency for International Development operates programs with partners across the world to foster democratic institutions. As a further example, a partner’s compliance with the International Health Regulations would reflect interests and activities of DoD, Health and Human Services, and Department of State, and the extent of their coordination and cooperation in achieving that outcome is another metric.

Even other organizations within DoD could provide different models: The Defense Security Cooperation Agency builds security capacities with partner nations. DoD’s CTR Program can learn from these other agencies successes and challenges (and even its own experience) if it incorporates a lessons-learned mechanism for the Program. Furthermore, the CTR Program will be working with these other agencies in a whole-of-government effort, and the others might already have mechanisms in place for measuring impact and effectiveness that would be useful to CTR.

Change

The DoD Metrics Report deliberately does not consider future missions or changes in objectives, but it is difficult to see how CTR metrics can be designed to respond to change if this is not discussed in DoD’s report. Some factors are under the Program’s control. Far more are outside of the Program’s control. Every responsible business, military operation, and government program builds in resilience that is the ability to deal with exogenous change. The CTR programs are susceptible to external change ranging from budget reductions to shifts in the host country’s political or economic environment.

The practical consequences of some of the shortcomings listed above, such as not articulating the connection to threat or risk, might not be large for projects in progress under longstanding agreements. But they are important for new projects and especially for new partnerships. A more detailed examination of the programs can be found in the next section.

image

18 See Weber, A. 2010.

Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

PROGRAM-BY-PROGRAM ASSESSMENT

The committee has summarized several important aspects of its assessment of the DoD Metrics Report in Table 2-2. The column headings represent a logical chain from objectives through metrics to sustainability and the minimum performance required, read from left to right.

Chemical Weapons Elimination (CWE)

The treatment of the CWE program in the DoD Metrics Report may be adequate for the current project under the program. The stated program goal looks very much like a project goal: “[T]o destroy nerve agent-filled munitions located in the Planovy [chemical weapons] storage facility in a safe, secure, and environmentally sound manner.” The United States and the Group of Eight Global Partnership against the Spread of Weapons and Materials of Mass Destruction (G8 Global Partnership) supported the design and construction of the chemical weapons destruction facility at Shchuch’ye, Russia, which began operating in 2009. At Russia’s request, the CWE program is providing technical support for the commissioning and operation of the facility. So the desired capability to support the program and project objectives is defined as adequate operation (by the Russians with U.S. technical support) of the Shchuch’ye destruction facility.

In its Metrics Report, DoD refers to the quantities of chemical weapons agent destroyed (neutralized and immobilized) as the program metrics. DoD’s only project level metrics are scheduled facility downtime and unscheduled facility downtime.

These metrics align well with the “destroy nerve agent” aspect of the stated program objectives, but the metrics only implicitly address the “safe, secure, and environmentally sound manner” aspect of the objectives. Worker safety and environmental emissions are also important indicators of proper operation of the facility, but are not mentioned in the DoD Metrics Report. In a facility that has strong regulatory controls and/or strong internal management controls on safety and environmental performance, problems with safety or environmental releases affect the operation of the facility, i.e., the facility will shut down some or all of its operations to correct problems. If the facility does not have strong controls in place, safety and environmental performance might not affect continued operation of the facility, so tracking facility downtime might not factor in safety and environmental performance. DoD is better positioned than the committee to assess this aspect of the Shchuch’ye facility and can justify its decision either to include such metrics or to exclude them. Likewise, the neutralization process is supposed to convert 99.99 percent of the agent to slightly hazardous components and then immobilize them for disposal. DoD does not explain the acceptable range of destruction percentage (is 90 percent destruction acceptable?) and whether it is important to track that parameter.

Finally, the committee notes that the Planovy Chemical Weapons Depot near Shchuch’ye contains approximately one seventh of the Russian stockpile of chemical weapons that awaits destruction. If DoD hopes that the support provided at Shchuch’ye will not only enable the Russian Federation to operate Shchuch’ye independently, but also affect progress at other depots, then DoD should include a metric for that objective too.

Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

TABLE 2-2. Program-by-program summary of objectives, capabilities, and metrics in the DoD Metrics Report. The committee’s assessment (shaded) is followed by a summary of the committee’s advice


Objectives (program/project) are specified? Desired capabilities identified? Linkage of capabilities to metrics Do metrics measure impacts and effectiveness? Do metrics address sustainability? Are minimum performance & aspirational goals specified?

Chemical Weapons Elimination (CWE) Assessment Not in DoD Report;a Some confusion between program and project objectives.a Adequately operating facility Shchuch’ye Yes Yes - for the operation of Shchuch’ye None Yes
CWE Advice N/A Consider whether development at other facilities is needed. This linkage needs to be established. Worker safety & plant emissions may also be appropriate & important metrics. This aspect could be developed. N/A

Nuclear Weapons Safety & Security (NWSS) Assessment Yes/Yes Yes, for sustainmentb Yes No, only measure input. Yes Yesa
NWSS Advice Clearly link overall objectives to threat reduction. N/A This linkage should be made more explicit, particularly as the program expands to other countries. Although there are constraints on measurement, improved metrics should have more focus on impact & effectiveness. N/A N/A

Cooperative Biological Engagement Program (CBEP) Assessment Yes/Yes Yes Yes Some, but not prioritized. Yes Yes
CBEP Advice Need to be more clearly linked to threat reductiona N/A Would benefit from linking capabilities metrics with threat reduction. Need to be prioritized, reduced in number, and direct metrics, where possible. N/A N/A

Weapons of Mass Destruction Proliferation Prevention Program (WMD-PPP) Assessment Yes/Yes Yes, but vague. Program metrics not linked to capabilities. Project, yes Program: no. Project: Some of them do, but not prioritized, not clearly linked to program level. Yes No
WMD-PPP Advice Recognize links to other related/similar programs; address integrated assistance efforts across the USG and other aid providers. Need to be clarified. Once capabilities are better articulated, they need to be better linked to the program and project metrics. More thought should be given to prioritization and impact and effectiveness. Sustainability metrics also need to be prioritized and focused on capabilities. The minimum performance and aspirational goals need to be specified.

a Not available in the DoD Metrics Report but available in other documents.

b NWSS has completed upgrades at agreed facilities, so the current program is focused on providing sustainment.

Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

Cooperative Biological Engagement Program (CBEP)

For the biosecurity mission, there are two main thrusts:19 (1) secure and consolidate especially dangerous pathogens, and any work with those pathogens, to a safe, secure facility or facilities; and (2) engage bioscience institutions20 and authorities in the partner country to establish a culture of responsible practice, detection, and international reporting of emergent pathogens and transfers of the pathogens. In principle, the first thrust is relatively easy to measure, if the work with those pathogens is known to the governments. The second thrust, the aim of which is in part to develop relationships of trust and practices that are trustworthy, is more difficult for the United States to measure (see Measuring Trust, Confidence, and Goodwill in Chapter 3). The metrics DoD developed for CBEP are numerous and mostly indirect with respect to trust, but relatively concrete and measureable. Some additional steps are needed to make these metrics effective tools for evaluating impact and effectiveness and managing the programs.

The DoD metrics for CBEP have nearly all of the elements that the committee thinks are needed for development of useful metrics. The CBEP objectives and desired capabilities are clearly stated, although their connection to threat reduction is not stated in the DoD Metrics Report. The metrics are linked to the desired partner country capabilities, and some of them measure impacts and effectiveness, although others do not.21 DoD also factors sustainability into its rating of the program’s performance.

The greatest shortcoming of this otherwise rather complete metrics structure is its lack of prioritization. DoD lists 49 metrics for CBEP and in its first application of the metrics, DoD weighted all of them equally. It is difficult to draw meaningful conclusions about overall progress from 49 metrics that are all weighted as equally important. Some of the metrics are far more important to DoD and to threat reduction than are others (see, e.g., Footnote 7 in this chapter), and not all 49 are needed, even after duplicated metrics are eliminated. The committee discusses metrics for the CBEP more extensively in Chapter 3, where the committee illustrates how an improved method for developing, prioritizing, and using metrics should be applied to CTR programs.

image

19 This is a summary of the four objectives DoD describes in its Metrics Report:

1.   Secure and consolidate collections of especially dangerous pathogens (EDPs) and their associated research at a minimum number of secure health and agricultural laboratories or related facilities;

2.   Enhance partner country/region’s capability to prevent the sale, theft, diversion, or accidental release of biological weapons related materials, technology, and expertise by improving biological safety and security standards and procedures;

3.   Enhance partner country/region’s capability to detect, diagnose, and report endemic and epidemic, man-made or natural EDPs, bio-terror attacks, and potential pandemics; and

4.   Ensure the developed capabilities are designed to be sustainable within each partner country/region’s current operating budget.

20 It seems reasonable to enhance this priority to include engaging bioscientists, not just bioscience institutions.

21 For example, one of the measures of effectiveness concerns the partner country laws and regulations for biological safety and security, and their conformance to U.S. standards. While the U.S. implementers may believe that having such laws and regulations is better than not having them, the actual practices are more important than the details of the laws. The practices may be poor despite good laws and regulations or good despite poor laws and regulations. As a result, a partner that otherwise has shown significant improvements with respect to biological safety and security might get a poor rating based on a regulatory system that is different from that in the United States.

Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

Nuclear Weapons Safety and Security (NWSS)

The NWSS section of the DoD Metrics Report gives candid assessments of the metrics it has developed (e.g., “Depending on the level of cooperation, DoD may not be able to track this metric independently.”), and overall gives the impression of a metrics effort in development. This reflects the current state of the NWSS program, but it also gives an incomplete picture of DoD’s thinking about measuring the impact and effectiveness of this program.

NWSS has completed upgrades at all of the facilities covered by the agreement between the United States and the Russian Federation, and DoD plans to continue to use the number of nuclear weapons storage sites upgraded as a metric of the impact and effectiveness of the program in Russia. Given that the upgrades are done, the current program is focused on (1) providing sustainment in the form of training, and on (2) the Nuclear Weapons Transportation Security Program. The DoD Metrics Report also states that “with the potential global expansion of the NWSS program to other partner countries, Defense Threat Reduction Agency DTRA has been working to develop enhanced metrics that better reflect how our efforts are contributing to overall threat reduction.”

The current NWSS program has severe constraints on measurements. The CTR program trains people who train the operators of the nuclear weapons storage sites, but the CTR training takes place at facilities away from the nuclear weapons storage sites, and CTR personnel do not have access to any of the sites. CTR supports supply of upgraded secure railcars for the transportation mission, but CTR personnel do not see them in operation. In both cases, the lack of access makes it difficult or impossible to measure the outcomes directly linked to the program’s objectives.

Recognizing these constraints, DoD has identified the objectives of the program and the capabilities desired to achieve those objectives. The DoD Metrics Report describes what DoD would like to measure and what might actually be possible. For example, DoD will track the number of railcar shipments supported and the number of secure railcars provided. DoD can measure the latter itself; it cannot measure the former. DoD included some metrics in its report because they were negotiated among DoD, Department of Energy, and the partner, Russia’s Ministry of Defense. Such mutually agreed metrics are important and notable, as discussed elsewhere in this report.

DoD presented additional information and ideas to the committee in November 2010.22 These ideas include a systematic method for weighting or prioritizing among the lower level factors that contribute to the metrics. Without endorsing the specific details presented to the committee, the committee agrees that DoD would benefit from further development of ideas like those in the November presentation.

Weapons of Mass Destruction Proliferation Prevention Program (WMD-PPP)

The WMD-PPP focuses on land and maritime border security, also known as green and blue borders respectively, in partner nations including Azerbaijan and Ukraine. Three program metrics are defined:

 

•   Miles of green (land) border provided sustainable security enhancements (in progress/complete)

image

22 See Appendix E.

Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

•   Miles of blue (maritime) border provided sustainable security enhancements (in progress/complete)

•   Number of ports of entry provided sustainable capability enhancements (in progress/complete)

In introducing these metrics, the DoD Metrics Report notes: “The ability to measure simply and objectively the impact that WMD-PPP assistance has had on threat reduction is challenging due to the nature of the program: we are providing a capability to our partners that gives them an ability to deter proliferation.”

DoD articulates both program and project objectives for the WMD-PPP. The difficulty for WMD-PPP program managers and decision makers results from the vagueness of the capabilities desired to achieve the goals or objectives. DoD clearly found it difficult to translate the desired capabilities into metrics. Miles of green and blue border with additional security enhancements, and the number of ports of entry where capabilities have been enhanced are ill defined. Are all enhancements equivalent? Is every kilometer of border equivalent in terms of threat reduction?

Securing a border is complicated. The first approximation of progress—the number of kilometers of green border deemed secure—might not accurately represent risk reduction for several reasons. First, not all kilometers of border represent equal risks. Second, adversaries can adapt to exploit the weakest link. Third, what constitutes a secure kilometer of border? Areas where security enhancements have been provided might still have known smuggling routes that regularly defeat security enhancements may be improperly counted in success measures. These challenges are well known to DTRA, and in implementing WMD-PPP, DTRA has not treated all borders as if they were equally important. For instance, some stretches of border may be sufficiently impassable due to the natural topography of the area as to render any additional security enhancement unnecessary while others may include commonly used smuggling routes. Metrics should instead focus on the risk posed and the likelihood of evading detection. In Ukraine, both DTRA and the Ukrainian State Border Guard Service conducted a simple risk ranking and agreed jointly on the highest risk stretch of border. They focused their prototype and demonstration facility efforts on that stretch of border. Thus the implementation is better than the metrics would suggest.

Interdiction events involving materials useful for WMD are rare and experts are cautiously optimistic that this is because smuggling of WMD material is rare, not just difficult to observe. Border protection services do not now have, and endeavor to never have, enough WMD incidents to make tracking them a useful statistical tool, so they use statistics from other border violations as proxies to indicate the effectiveness of their border controls. The definition of positive outcome must be sufficiently broad to prevent unintentionally causing a reduction of inspections or alarms that might prevent the detection of an illicit item. For instance, if the risk management device is using rules based targeting efforts or non-intrusive inspection equipment, identifying a shipment that comes from a legitimately high-risk source or known bad actor might be a positive outcome, even if the resulting inspection did not uncover illicit material. Similarly, the physical inspection of a container that uncovers a significant quantity of lead or similarly dense material may be a positive outcome to imaging analysis of a container x-ray that shows a density anomaly. Noting those inspections as negative outcomes may result in the overly narrowing of risk factors, thereby increasing the risk of missing an actual smuggling event.

Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

Progress in securing a border can be measured statistically (with some degree of accuracy, Chapter 3) if the country has sufficient control or awareness to even estimate the violations. Border protection services can track radiation alarms and smuggling incidents involving drugs, firearms, people, stolen goods, and goods avoiding tariffs.

It may be that partner countries do not need the ability to perform maritime interdictions themselves. Instead, one might measure the response time to conduct a maritime interdiction, understanding that another country’s team or a regional/multi-national team may be best situated and equipped to respond. Measuring the time for interdiction might more accurately measure the objective. For instance, the coastline of Georgia may be such that it needs fewer boarding teams than required to respond adequately to the much larger coastline of Ukraine. Another option might be to take the numerator/denominator approach, measuring the current capability against an assessed need.

At a project level, the metrics in the DoD Metrics Report are better linked to the capabilities desired. Some of the project metrics DoD developed for the WMD-PPP measure impacts and effectiveness, such as alerts resulting in a positive outcome, and others do not, such as the number of maintenance personnel trained. There are also a number of other programs of the United States and other countries that provide border control and law enforcement assistance that affect border control and trafficking that may provide useful insights for DoD.

METRICS AND EVALUATION

Metrics are inputs to evaluation, not evaluations themselves or substitutes for evaluations. The committee cautions those who use program metrics to use them judiciously. Some people will always trust quantitative or numerical metrics over qualitative metrics in a belief that they are somehow more rigorous or objective. But numerical metrics are not necessarily objective and especially when taken by themselves can be misleading because different countries will have different baselines and different objectives. Likewise, it would be unfortunate if otherwise appropriate attention to metrics results in funds being taken out of activities that are useful and shifted into activities that are measurable.

The WMD-PPP illustrates some of the caveats that must be kept in mind with quantitative metrics. In addition to the shortcomings to the “miles of border secured” metric described above, using a more nuanced quantitative evaluation of border security based on interdiction statistics requires interpretation beyond the numbers. A country that has poor control of its borders or poor ability to interdict illegal trafficking through its border crossings (either because of inability to detect illegal trafficking or because of corruption) may have statistics tracking its progress in securing its borders, but those statistics will require interpretation. Does a rise in interdictions indicate higher risk (more trafficking) or lower risk (more traffickers getting caught)? See Chapter 3 for a more general description of what metrics cannot do.

Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

This page intentionally left blank.

Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 17
Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 18
Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 19
Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 20
Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 21
Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 22
Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 23
Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 24
Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 25
Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 26
Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 27
Suggested Citation:"2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 28
Next: 3 Improvements to CTR Metrics »
Improving Metrics for the Department of Defense Cooperative Threat Reduction Program Get This Book
×
Buy Paperback | $38.00 Buy Ebook | $30.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Cooperative Threat Reduction (CTR) Program was created in 1991 as a set of support activities assisting the Former Soviet Union states in securing and eliminating strategic nuclear weapons and the materials used to create them. The Program evolved as needs and opportunities changed: Efforts to address biological and chemical threats were added, as was a program aimed at preventing cross-border smuggling of weapons of mass destruction. CTR has traveled through uncharted territory since its inception, and both the United States and its partners have taken bold steps resulting in progress unimagined in initial years. Over the years, much of the debate about CTR on Capitol Hill has concerned the effective use of funds, when the partners would take full responsibility for the efforts, and how progress, impact, and effectiveness should be measured.

Directed by Congress, the Secretary of Defense completed a report describing DoD's metrics for the CTR Program (here called the DoD Metrics Report) in September 2010 and, as required in the same law, contracted with the National Academy of Sciences to review the metrics DoD developed and identify possible additional or alternative metrics, if necessary. Improving Metrics for the DoD Cooperative Threat Reduction Program provides that review and advice.

Improving Metrics for the DoD Cooperative Threat Reduction Program identifies shortcomings in the DoD Metrics Report and provides recommendations to enhance DoD's development and use of metrics for the CTR Program. The committee wrote this report with two main audiences in mind: Those who are mostly concerned with the overall assessment and advice, and those readers directly involved in the CTR Program, who need the details of the DoD report assessment and of how to implement the approach that the committee recommends.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!