The panel has noted that there are deficiencies, some quite serious in magnitude, in current knowledge and practices in these subject areas vis-à-vis military testing and evaluation. These are some areas that could be emphasized by agencies such as the ARO, ONR, and AFOSR.
It is not appropriate to call for specific work in areas like experimental design, reliability, or software testing without having a clear notion about the potential value of different contributions in these areas. An effective program of sponsored research would require prior input from both the potential users of research findings and the research community. It would require specifying the innovations that practitioners need and the theoretical developments required to address the applications of interest.
There is a fundamental need for a collaborative effort between the military operational test and evaluation community and the statistical research community directed at defining new research initiatives. Developing such a research program, including preferred providers in government, industry, and academia would be a good continuing task for the Statistical Analysis Improvement Group that we recommend be established. One of its missions should be to advise the Undersecretary of Defense for Acquisition and Technology on the priorities of research on technical issues raised by the developmental and operational testing of defense systems and the potential sources for such research. At the same time, we suggest that the ARO, ONR, and AFOSR, consider increasing the priority of research on technical issues raised by the developmental and operational testing of defense systems. These applications are extremely important and might greatly benefit by new advances in statistical and related methodology. Sponsored research can be an extremely useful tool, for generating progress on many of the important issues described here, and it has not been used particularly effectively in recent years. Support of both external sponsored research and internal statistical methods research is critical if the services are to embrace improved statistical methods in designing and interpreting test and evaluation activities.