tem.1 NTCS-A is a commercial off-the-shelf evolutionary procurement system. (In evolutionary procurement, the system is developed in stages, with each stage undergoing separate testing.) This system experienced a number of problems that we believe are relatively widespread. First, because the system experienced frequent failures during testing, the goal of having the system run for a reasonable number of hours without failure was changed. Also, the large number of components for the system (approximately 40) created the potential for interaction problems each time one of the components was upgraded, since it would result in 40 different product enhancement and release cycles that would affect the whole system. Moreover, with little configuration control, the systems being tested in the operational testing and evaluation were materially different from systems being fielded; thus, the panel viewed the ability to control configuration as a key issue to be addressed.
The panel also had numerous interactions with the Air Force Operational Test and Evaluation Center (AFOTEC). AFOTEC places a great deal of emphasis on the examination of software code, approximately 2 percent of the code in large systems. We believe this is not an efficient use of analysts' time: the test sample of lines of code that are examined should never be based on a fixed percentage of total lines of code; furthermore, it is the software architecture that should be examined, not the code. We do applaud AFOTEC's efforts to communicate early in the process with software developers, and we concur that the use of software metrics, a measure based on code characteristics that evaluate code complexity, is useful for the purpose of producing estimates for support budgets.
The Army Operational Test and Evaluation Command has conducted impressive experiments in developing effective test processes, but these methods have not yet been institutionalized. Also, although the Army has developed an extensive software metrics program, it is of limited value because it is not connected to software failures in the field or to the development process.
On the basis of its review of these and other systems, the panel concludes that use of statistical science can significantly improve the test and evaluation of software and software-intensive systems. Papers that discuss some of the relevant research concerning statistical aspects of software testing and evaluation are Nair et al. (1998), Oshana (1997), and Poore and Trammell (1996). Before describing this use, it should be noted that there are several important software engineering issues involved in the defense system life cycle that are not in our
The Naval Tactical Command System-Afloat is described as an all-source, all-knowing system that is installed in most ships and many more shore sites. It is designed to provide timely, accurate, and complete all-source information management, display, and dissemination activities-including distribution of surveillance and intelligence data and imagery to support warfare mission assessment, planning, and execution—and is a current segment of a large strategy system known as the Joint Maritime Command Information System.