This National Academy of Sciences (NAS) study, requested under Congressional mandate, evaluates the “metrics to measure the impact and effectiveness of activities of the Cooperative Threat Reduction [CTR] Program of the Department of Defense [DoD]” (Section 1304, P.L. 111-84) described in a DoD September 2010 report (DoD, 2010; hereinafter the DoD Metrics Report). The NAS study committee found that the DoD Metrics Report provides reasonable metrics for some activities of the CTR Program which consolidate and eliminate weapons of mass destruction and weapons materials, and provides a solid starting point for developing metrics for newer, expanded capacity-building efforts under the CTR Program. The committee also identified shortcomings in the DoD Metrics Report and here provides recommendations to enhance DoD’s development and use of metrics for the CTR Program.
Improving Metrics
The committee has several recommendations to DoD on how it can improve its metrics for the CTR programs and better communicate its approach to metrics.
1. For each program described in the DoD Metrics Report, DoD should include a concise statement of its objectives and of how the program is intended to reduce threat or risk.
2. Objectives for projects and the overall CTR Program in a partner country are developed jointly between the United States and the partner country. An agreed set of metrics should also be built into projects from the outset. They may change, but the parties responsible for the projects should know at any given time the metrics that will be used to measure impact and effectiveness.
3. The committee judges that using a consistent framework to prioritize and refine metrics within a program would help DoD and other CTR decision makers. Using such a framework, DoD can identify the highest priority metrics, ensuring that the metrics are useable and useful, and allow decision makers to feed results back into the overall CTR processes of determining objectives and budgets. Any of several decision-making or prioritizing frameworks would be feasible, including the decision analysis technique of swing-weight analysis and the DoD capabilities based planning process.
4. DoD plans to leverage other U.S. Government agencies’ experience, capabilities, and assets as CTR expands to new countries and as it continues existing programs. DoD also needs to communicate, coordinate, and cooperate with relevant agencies.
5. DoD’s metrics and planning process should more explicitly factor in both planned and unplanned change over time. During the phases of active DoD involvement in a CTR project and afterward during sustainment, which is its own stage requiring resources (budgets, equipment, and trained people), clearer planning for how changes and metrics results will feed into decision making will make the metrics more credible and useful for both DoD and the partner country.
6. Capacity building programs need independent evaluation of how the capabilities being built perform in action. This can be accomplished by several means, ranging from expert observations of routine operations to comprehensive exercises that test the full scope of
capabilities. The level of effort can be tailored to the scope of the program, its resources, and its relative importance. DoD and its partners should build such independent evaluation into each project. The Defense Security Cooperation Program might be a good model for how to proceed.
