morale, state of alerting or fear, and anticipation of the scoring rules to be used), and require extra-statistical analysis.

Information one gains from simulation is obviously limited by the information put into the simulation. While simulations can be an important adjunct to testing when appropriately validated for the purpose for which they are used, no simulation can discover a system problem that arises from factors not included in the models on which the simulation is built. As an example, one system experienced unexpected reliability problems in field tests because soldiers were using an antenna as a handle, causing it to break. This kind of problem would rarely be discovered by means of a simulation.

As another example, consider Table 5-1, which represents the mean time between operational mission failures for the command launch unit of the Javelin (a man-portable anti-tank missile). Note that as troop handling grows to become typical of use in the field, the mean time between operational mission failure decreases. It is therefore reasonable to assume that the failure modes differ for the various test situations (granting that some were removed during the development process). However, since a simulation, designed to incorporate reliability, would most likely include failures typical of developmental testing (rather than operational testing), such a simulation could never replace operational tests. Thus the challenge is to identify the most appropriate ways simulation can be used in concert with field tests in an overall cost-effective approach to testing.

The panel is concerned that (1) rigorous validation of models and simulations for operational testing is infrequent, and external validation is at times used to overfit a model to field experience; (2) there is