Executive Summary

OVERVIEW: CONCLUSIONS AND RECOMMENDATIONS

The Department of the Navy must take a new look at modeling and simulation (M&S):

• The very nature of warfare is changing, perhaps drastically. The U.S. style of war is becoming technologically complex and dependent on distributed and interconnected systems. Modeling and simulation will be core tools for planning and conducting warfare as revolutionary changes in military affairs take place, especially since intuition based on past wars will become less helpful over time.
• Indeed, independent of Navy and Marine actions, M&S will be deeply embedded within joint command-and-control systems. Without enhanced efforts, the Navy and Marine Corps will not understand the strengths or limits of such models and simulations, nor be proficient with them.
• M&S will also become a core feature of system development and acquisition, as is the case already in leading-edge civilian industry. Because of its centrality, M&S should be seen as an enterprise technology in itself-part of the revolution in business affairs that is now a key element of the Department of Defense's DOD's overall strategy.

While the future of M&S should be exceedingly bright, the Department of the Navy will not be able to exploit its potential unless it attends to serious and chronic shortfalls-the most important of which relate not to software, but to the quality and content of the underlying models. Dramatic advances are being made in the DOD's M&S, but these advances are associated mainly with computer and software technologies. By contrast, too little attention has been focused on the content of the models themselves, or on the research base needed to create that content. Failure to address this shortfall will inevitably lead to less effective but more expensive combat forces and—quite possibly—serious operational failures. The escalating complexity of planned systems and operations creates profound integration challenges requiring superb M&S for success—and for the avoidance of downright failures.

All of this suggests that the Department of the Navy needs to make an attitude shift regarding M&S, which has never previously merited a high priority for leadership attention. Today, what is needed is a strategic commitment to exploiting M&S (Figure ES.1). This, of course, would lead to a strategy, policy, and investment actions. In this report the Panel on Modeling and Simulation identifies priorities for such matters. One priority involves the two principal joint simulation programs for training and analysis, the Joint Simulation System (JSIMS) and the Joint Warfare System (JWARS), respectively. Since first-generation versions will be quite imperfect and the systems may last 10 to 20 years, the panel recommends that the Department of the Navy take an active role to ensure that JSIMS and JWARS are produced as evolving systems that incorporate future research results. This is not simply a management issue, but rather something very challenging technically, since the architecture of the simulations must allow this evolution. Also, links must be created between the research and M&S worlds, links that do not now exist. And the research itself must be expanded substantially.

FIGURE ES.1 Looking at M&S strategically.

The research needed falls into two categories, warfare-area research and fundamental research. The former involves understanding the processes that take place in combat (e.g., those of littoral operations or long-range precision strike). This is military science, not programming, and commercial industry will not lead the way. The fundamental research needed involves theory and methodology that will make it possible to design and construct sound and reusable models and simulations that will also be comprehensible, flexible, and testable in specific contexts. Achieving these features will require major advances. It follows that the panel recommends that the Department of the Navy (working with other Services and DOD as appropriate) establish a robust but focused program in research, with both warfare-area research and research on fundamental theory and methods.

Unfortunately, research alone is not sufficient, and its fruits are often not harvested because researchers, M&S developers, warfighters, and other leadership figures are often disconnected. Accordingly, the panel recommends that the Department of the Navy establish processes that ensure early scientific review of models emerging from research, a competitive atmosphere in which "the market" of model users is both encouraged and assisted in constantly evolving their M&S to represent the best available knowledge (i.e., in assimilating improvements), and a general emphasis on quality, including the ability to represent uncertainty. Accomplishing this will require a multiyear commitment of senior leadership, because the baseline culture is very different from the one needed. Elements of a changed approach would include enhanced officer education and continual "beta testing" of models and simulations by organizations such as the war colleges and commands, testing that extends deeply into content, not merely software performance. However, such "operators" will need substantial assistance from the scientific community.

In addition to addressing the quality and content of models and simulation, the Department of the Navy needs to make investments that cut across usual organizational stovepipes and budget accounts. This relates to the promise of simulation-based acquisition. Accordingly, the panel recommends that the Department of the Navy treat simulation-based acquisition (SBA) as a key enabling technology with extraordinary long-term leverage and that it organize and invest consistently with that enterprise-technology view.

RICH OPPORTUNITIES FOR MODELING AND SIMULATION

Modeling and simulation (M&S) offers the promise of greatly enhancing future naval force capabilities and achieving major cost savings. To cite a few examples:

• Concept development. Next-century naval forces will require new operational concepts and force structures—in some cases radically different from current ones. Ours is an era of military ferment analogous to that of the 1920s or 1930s. M&S can help screen, design, and test concepts and force structures before irrevocable commitments to them are made. In principle, M&S could also be compelling enough to "force" needed changes of doctrine before disasters occur in war.
• Simulation-based acquisition (SBA). Representations of proposed system designs can be constructed and tested in simulated environments. These virtual prototypes can be used to refine system requirements and relate tradeoff and engineering decisions to these requirements. Subsequently, computer-based representations can be maintained as development and production occur, and as modifications are introduced throughout the life cycle. The results can be more affordable systems that are better attuned to an operator's needs, easier to assimilate, and easier to modify. The remarkable success of the Boeing 777-development merely tapped the surface of what will eventually be possible.
• Decision support. M&S can assist commanders in their planning for combat and other military operations. Key uses include developing and assessing proposed courses of action, mission planning and rehearsal, and dynamic situation assessment and adaptation in the course of battle. M&S-based decision support also has an important role in peacetime activities such as concept evaluation and resource allocation.

Models and simulations are being used for all of these functions today. Indeed, the breadth of M&S is enormous, as suggested by Figure ES.2. The best-known recent successes have been in training, but there is rapidly growing documentation on valuable applications throughout Figure ES.2's cube—as judged not only by those advocating M&S, but also by senior commanders. Documented examples of recent high-leverage payoffs from M&S are given in Chapter 2, but the future is more relevant in this study. It is especially significant for the future that M&S will be thoroughly embedded in command-and-control systems. There have already been numerous exercises in which differences between the real and the simulated have been blurred or made invisible to some participants. This will be increasingly the case as U.S. forces adopt the concepts sketched in the Joint Chiefs of Staff's Joint Vision 2010 (Shalikashvili, 1996).

FIGURE ES.2 The scope of DOD's M&S activities.  SOURCE:  Adapted from Defense Modeling and Simulation Office (1996d), Figure 2.1.

THE POTENTIAL FOR FAILURES AND DISASTERS

Cautions Amidst Enthusiasm

Despite this bullish introduction and the fact that M&S will surely "take off" in the commercial sector, the potential of M&S for the Department of the Navy and DOD may not be realized in the foreseeable future. Some of the DOD's most important and expensive M&S efforts may fail or—perhaps worse—end up saddling the DOD components with mediocre, inflexible, and sometimes misleading tools that impede innovation and improvement of content. There is also the potential for disasters due to overdependence on M&S for images and predictions that appear more valid than they actually are (the "Spielberg" effect). This could cost lives and undercut military operations.

One basic problem is that DOD's investments have been concentrated more on content-neutral computer and software technologies rather than on the content of the models.¹ On the technology side, much has happened, for example, object-oriented programming, high-performance computing, computer-aided design, and establishment of the communications protocols and infrastructure for everything from collaborative, distributed, multidisciplinary, simulation-based engineering design to large-scale distributed war games. The dramatic progress in technology will assuredly continue because of commercial developments and DOD efforts such as those of its Defense Modeling and Simulation Office (DMSO). An Undernourished Knowledge Base

In contrast, there has been curiously little investment in the knowledge base determining the substantive content and quality of much M&S—particularly higher-level M&S needed for mission- and campaign-level work. It is an open secret, and a point of distress to many in the community, that too much of the substantive content of such M&S has its origin in anecdote, the infamous "BOGSAT" (bunch of guys sitting around a table), or stereotypical versions of today's doctrinally correct behavior. There is a need for focused research on the phenomena of combat and other military activities, both historical and prospective. This is the realm of military science.

Another shortfall in knowledge relates to theories and methods for conceiving, designing, and building models (as distinct from software). Symptoms of the problem are evident if one observes that DOD's M&S often consists of nothing more than the computer code itself: there is no separable documented "model" to be reviewed and improved, nor any way to readily understand the assumptions generating the simulation's behavior. This can hardly be a comfortable basis for decision support.

Complex Systems

The inherent complexity of the systems and force operations that DOD is attempting to simulate introduces new difficulties. Too many forecasts are extrapolating unreasonably from the Boeing 777 experience, and from M&S successes in weapon-system and small-unit training, to imagined M&S systems of extraordinary complexity. Recent failures such as the automated Denver airport baggage system and the Federal Aviation Administration advanced air-traffic-control system suggest the difficulties associated with reliably modeling and engineering complex systems. The military operations envisioned for future forces in the information era involve exceedingly complex systems.

Complexity is a multifaceted concept, but to appreciate some of what is involved here, the panel notes that planners, commanders, and engineers are most familiar and comfortable with systems (and models) that are primarily static, linear, and deterministic. However, automated and integrated military systems (the systems-of-systems approach) involve systems and models that are dynamic, nonlinear, and heterogeneous interconnections of mixed subsystems, with a much more sophisticated treatment of uncertainty, including uncertainties about the opponent's intentions and actions.

The panel's principal observation here is that dealing with such complex systems effectively is a decades-long task. In the meantime, there is need for humility, multiple approaches and competition, patience, and hedges. And even long-run success will require profound changes in the way M&S is conceived and designed, as well as an across-the-board attention to its content and validity—in a sense that suitably recognizes uncertainties. The panel describes what is needed below.

Assimilation of M&S Technology

Finally, there are the problems of assimilation and exploitation. M&S is an enabling technology, but its value is cross-cutting, and it has no natural single home. Nor should it, since—in a partial defense of stovepiping—the majority of work must be dictated by the needs of individual applications. However, whenever such a cross-cutting technology is introduced or its use expanded, there are organizational and managerial challenges. The key to success is often having a strategy embraced by the organization's leadership.

Against this background diagnosis, the panel has observations and suggestions in each of a number of subjects. In essence, the recommendation is for a concerted and long-overdue effort to improve the research base for the Department of the Navy's and DOD's models, and to ensure that the results of research are in fact incorporated. The panel's recommendations include priorities and suggestions for a strategy, not simply general funding of research.

RECOMMENDATIONS ON JOINT MODELS

Concerns

One useful focus for the Department of the Navy's thinking about M&S is the set of joint systems now in development (most prominently JSIMS and JWARS). Taken together, these worthy programs (including the Service components) have a price tag approaching $1 billion. It is DOD's intention that JSIMS and JWARS will become the core for all future joint work on training and analysis, respectively. Although this is unlikely (a wider range of models will probably prove necessary), it may indeed be that JSIMS and JWARS will dominate the joint M&S scene for the next 20 years. Thus, it is important to the Department of the Navy that naval forces be adequately represented. Otherwise, valuable training opportunities will be compromised and the Navy and Marines will suffer in the competitions about doctrinal changes, future missions, and force-structure tradeoffs. More generally, the quality of joint work will suffer.

Unfortunately, it is likely that first-generation versions of JSIMS and JWARS will not be satisfactory—even with heroic efforts and even though the products will have many excellent features. There will be major shortcomings with respect to both content and performance. Consequently, the panel recommends that the Navy insist that DOD and the program offices adopt open-architecture attitudes that will promote rather than discourage substitution of improved modules as ideas arise from the research and operations communities, and that they build explicit and well-exercised mechanisms to ensure that such substitutions occur.

This recommendation may seem uncontroversial, and it calls for no more than what some of the programs (notably JSIMS) are projecting on viewgraphs, but the history of DOD modeling has often been to produce relatively monolithic and inflexible programs. Further, there has been great DOD emphasis in recent years on avoiding alleged redundancies, collecting "authoritative representations," and exercising configuration control. The panel observes widespread frustration among analysts and other substantive users of models, who see DOD's M&S efforts as driven by civilian and military managers who think models are commodities to be standardized, who sometimes seem to value standardization more highly than quality (harsh words, but too important to be omitted), and who have given near-exclusive emphasis to software technology issues. They and the panel believe that M&S should instead be seen as organic, evolving, and flexible systems with no permanent shape (but with standardized infrastructure, including many component pieces).

In fact, the visionary technical infrastructure being promoted by OSD's Defense Modeling and Simulation Office (DMSO) (and software technologists) will permit the open system approach and will permit competition among alternative models (e.g., alternative representations of ballistic-missile defense, mine warfare, or command, control, communications, computing, intelligence, surveillance, and reconnaissance (C4ISR)). Thus, while it would be easy for JWARS, JSIMS, and other systems to end up as rigid monoliths, with the right architecture and organizational structure DOD can have its cake and eat it: it can have "standard configurations" while still making it easy for users to substitute model components as new ideas and methods emerge. An important but more subtle aspect of this infrastructure is connecting model evolution to the R&D and operational communities concerned with both current and futuristic doctrine; and, significantly, nurturing a competition of ideas and models. In that way the evolution will be more like survival of the soundest than like continuation of what has previously been approved.

The panel underlines the problem of incorporating research results when they exist because, at present, the communities that do research and the programming of models often do not communicate well and there is little pressure to assure that the "best" models are reflected in M&S. Indeed, there is much pressure to avoid changes.

Technical Attributes Needed in Joint Models

Against this background of concerns, the panel recommends that the Navy advocate an approach to joint-model development that has a long-haul view and an associated emphasis on flexibility. The groundwork should be in current model-building efforts for the following, which will be important in selected applications in the years ahead:

• Multi-resolution modeling, not only of entities, but also of physical and command-and-control processes, with the objective of building integrated models of families with different levels of resolution.
• Explicit decision models representing the reasoning and behavior of commanders and different levels, and reflecting in natural ways courses of action, plans, and the adaptations that commanders make in the course of operations.
• Diverse representations of uncertainty, including use of probability distributions (and, sometimes, alternatives such as fuzzy-set concepts), even in aggregate-level models.
• Systematic treatment of important correlations (e.g., the "configural effects" of mine warfare and air defense).
• Explanation capabilities linking simulated behavior to situations, parameter values, rules and algorithms, and underlying conceptual models.
• Mixed modes of play that are interactive, selectively interruptible (e.g., for only higher-level commander decisions), and automated. (The panel regards the option for human play as critical for analytic applications as well as training, and the option of closed play, e.g., of the opponent, as critical for training.)
• Testing of new doctrinal concepts requiring new entities, attributes, and processes.
• Different types of models. The systems should accommodate model types as diverse as general state-space and simple Lanchester equations, entity-level "physics-based" models, and agent-based models with emergent behaviors. They should employ such varied tools for such uses as statistical analysis, generation of response surfaces, symbolic manipulators, inference engines, and search methods (e.g., genetic algorithms). The models must be able to deal not just with old-style head-on-head attrition warfare, but also with maneuver warfare on a nonlinear battlefield in the information era, and with operations in urban sprawl. They must reflect different command-and-control concepts.
• Tailored assembly. The systems should facilitate tailored creation of models, including relatively simple M&S for specific applications. That is, one should conceive of JSIMS and JWARS as tool kits with rapid-assembly and modification mechanisms. Excessive complexity obfuscates and paralyzes.

In some respects, the last item is the most important. Given the breakthroughs in software technology over the last two decades, it is feasible (though not easy)—and essential—for major M&S efforts to be designed for frequent adaptation, specialization, and module-by-module improvement. One should think of assembling the right model, not taking it from the shelf whole. Further, it should be possible to discard or abstract complexities irrelevant to the problem at hand. Doing so runs directly counter to the common inclination to seek high resolution for everything, but tailored simplifications are crucial in applications—especially when they involve "exploratory analysis" over diverse situations and assumptions rather than point calculations. The type of analysis is crucial when uncertainties are large-as they often are. In any case, the need for simplicity is generally much better understood by those who have conducted studies or exercises, or designed decision-support systems, than by those who develop software.

The panel notes, however, that there are limits to what can be accomplished by assembly or composition. The Department of the Navy and DOD should be skeptical about the notion that a single system (e.g., JWARS or JSIMS) will prove useful to a wide range of communities. It is one thing to assemble and tailor components for one study rather than another, or for one exercise rather than another. It is a very different matter to have the same system and library of components support a broad range of different functions (testing, exercises, force planning, etc.). Viewgraphs postulating such versatility do not constitute an existence proof.

RECOMMENDATIONS FOR RESEARCH

Research in Key Warfare Areas

As noted above, there has been relatively little recent investment in understanding the phenomenology of military operations at the mission and operational levels. Much of the basis for related M&S is still programmer hypothesis and qualitative opinions expressed by subject matter experts. This has not always been so. During and after World War II, operations research worked from a rich empirical base, but now the United States is entering a period of nonlinear, parallel, information-era warfare for which the intuition of scientists, operations researchers, and warriors is insufficient and unreliable. Further, it will be relying on complex systems working as designed in multifaceted joint campaigns. Success may be much less tolerant of errors in concept and execution than in days past. Indeed, some of the doctrinal concepts under discussion will involve very high risks.

Given, then, that improvement of the research base is essential, how might it be accomplished? Rather than merely urging general support for research, the panel recommends a managerially focused approach with priorities and mechanisms for assuring relevance and assimilation. Accordingly, the panel recommends that the Navy and Marine Corps select a few high-priority warfare areas and create research programs to support them. These programs should be organized so as to ensure close ties to operational and doctrinal-development communities, and to relevant training and exercise efforts that could be mined as a source of empirical knowledge (e.g., as suggested in Figure ES.3, which would exploit emerging capabilities for distributed interactive simulation). This is a nontrivial and potentially controversial suggestion, since the long-standing tradition has been to avoid—and even prohibit—extensive data collection for use beyond those being trained. The costs of such efforts would be small in comparison with those for buying and operating forces, or even procuring large models. Although the Department of the Navy (and DOD) need to make up for past failures to invest adequately in research, this is a domain in which a total of $20 million to $30 million per year can accomplish a great deal.

FIGURE ES.3 Using exercises as a source of empirical data for M&S. SOURCE: Reprinted, by permission, from Davis (1995b). Copyright 1995 by IEEE.

As a first list of warfare areas for focused research, the panel recommends the following, which have some overlaps:

• Expeditionary warfare and littoral operations;
• Joint task force operations with dispersed forces;
• Long-range precision strike against forces employing countermeasures;
• Theater-missile defense, including counterforce and speed-of-light weapon options, against very large ballistic-missile and cruise-missile threats; and
• Short-notice, early-entry operations with opposition.

Each of the above warfare areas has major knowledge gaps that could be narrowed by empirical and theoretical research closely tied to the "warrior communities."

This report describes key attributes of research programs for such warfare areas. An overarching theme is the need to take a holistic approach rather than one based exclusively on either top-down or bottom-up ideas. A second theme is that the research should be seen as focused military science, not model building per se. This will determine the type and range of people involved, and also the depth of the work.

Two examples may be useful here. The first is the challenge of developing command-control concepts for highly dispersed Marine Corps forces operating in small units far from their ship-based support and dependent on a constellation of joint systems. The Marine Corps is studying alternative concepts in the Hunter/Warrior experiments. Such experiments need to be accompanied by systematic research and modeling of different types, perhaps including new types of modeling useful in breaking old mind-sets. It is plausible, for example, that cellular-automata models could help illuminate behaviors of dispersed forces with varying command-control concepts ranging from centralized top-down control to decentralized control based on mission orders. To its great credit, the Marine Corps is exploring such possibilities, opting to accept some "hype and smoke" in the realm of controversial complex-system research in exchange for new perspectives and tools useful in doctrinal innovation. While the panel does not believe such simplified models will prove adequate in the long run, they can be very helpful in developing new hypotheses.

A Navy example involves mine and countermine warfare. From prior research based on sophisticated probabilistic modeling accounting for numerous "configural effects" (i.e., effects of temporal and spatial correlations), it is known that effective strategies for mine-laying or penetrating minefields are often counterintuitive. By exercising such models and simulation-based alternatives in an exploratory manner (as distinct from answering specific questions), it should be possible to develop decision aids of great value in training, acquisition, and operations. Such aids should not, however, focus only on "best estimate" single-number predictions; they should instead provide commanders with information about odds of success, as a function of information. If the aids are to be useful, they must be informed by an intimate understanding of operational commanders' needs.

Recommendations on Fundamental Research

While many research activities are best driven by applications, other critical areas of M&S research require more fundamental research that might be sponsored by the Office of Naval Research (ONR), other Service analogues, the Defense Advanced Research Projects Agency (DARPA), and the Director of Defense Research and Engineering (DDR&E). In what follows the panel suggests particular subjects for fundamental research, divided into theory, advanced methodologies, and infrastructure (including tools).

Modeling Theory

The panel recommends that research on the following be given priority:

• Multi-resolution modeling, integrated families of models, and aggregation-disaggregation. Multiple-resolution modeling depicts phenomena at different levels of detail. In some cases a single model can operate at different levels of resolution with appropriate kinds of consistency. More often it is possible—although unusual and difficult—to design integrated families of models that can be mutually calibrated using information available in many forms and resolutions, and to do so with full recognition of statistical averaging issues. Such families of integrated hierarchical models would be invaluable in all application areas and would substantially improve validity, traceability, and the design of exploratory analyses. How to design and build such multi-resolution models or families, however, is a frontier problem in modeling theory. A related subject is often called aggregation-disaggregation. This often refers to distributed simulations in which, within the course of a simulation, some of the entities must be disaggregated and reaggregated (e.g., a Marine Corps company might have to be disaggregated to engage simulated entity-level opponents and then reaggregated to continue its maneuver). While this is relatively straightforward from a software perspective, there are deep questions concerning the consistency of behaviors at different resolution levels.
• Agent-based modeling and generative analysis. Some of the most interesting new forms of modeling involve so-called "agent-based systems" in which low-level entities with relatively simple attributes and behaviors can collectively produce (or "generate") complex and realistic "emergent" system behaviors. This is potentially a powerful approach to understanding complex adaptive systems generally—in fields as diverse as ecology, economics, and military command-control. A fundamental step in developing particular models and simulations is deciding which attributes and interactions to represent, and in what detail. This choice should be the one that most adequately describes the phenomena one is trying to observe, but that choice is often not known until the subsystems are connected and the simulation is run. Thus, methods should be developed to allow one to iterate on the choice of the initial representations of subsystem models, based on results of their use in interconnected systems.
• Semantic consistency. Phenomenological representations in different simulations need to interact with one another in distributed simulations. Such interaction is meaningful only if the representations are "semantically consistent," that is, if there is a shared understanding of what concepts and data "mean." This requires commonality of context and definition (or well-understood translations). The Navy should track related research, adding to it for special purposes. It should also support DMSO efforts to develop common models of the mission space (CMMS), which will assist in establishing semantic consistency in particular contexts and in developing integrated families of models.

Advanced Methodologies

The general task of developing and using models and simulations, and the particular activity of forming phenomenological representations, would be aided by methodological advances. Particular topics are as follows:

• Characterization of uncertainty. No matter how careful one is in preparing for a simulation, certain attributes and interactions will have some measure of uncertainty. Often, uncertainties dominate the problem. Methods to track the propagation of uncertainties should be developed since they can lead to large uncertainties in the output of the simulation. This is a particular challenge in heterogeneous, nonlinear dynamical systems, where uncertainties in components can interact in nonintuitive and unpredictable ways. The so-called "butterfly effect" in chaotic systems is a well-known popular example.
• Exploratory analysis under uncertainty. Running a simulation for one set of fixed conditions is generally not satisfactory since there are often large uncertainties throughout the system. Even normal sensitivity analysis on a one-variable-at-a-time basis does not suffice because of interaction effects. An important research area, then, is developing ways to use modern computer power to explore the space of simulation outcomes and to search for interesting regimes (e.g., regimes representing high or low risks for an operation or for especially profitable, or unacceptable, performance of a weapon). This research has implications for the design of models (some of it closely related to multi-resolution modeling), search engines, and visualization methods. It has even more profound implications for analysis and decision making because it encourages decision makers to ask not about best-estimate outcomes, which are often no more likely than very different ones, but rather about how outcomes of a strategy would be likely to vary as a function of the many assumptions in "scenario space." This can help by focusing attention on the need to avoid "dangerous regimes" in the course of operations, by focusing attention on the search for crucial information, and by emphasizing the need for both hedging and adaptability. This approach, of course, is quite different from the search for mythical optimality.

Infrastructure, Tools, and Supporting Technology

With regard to infrastructure, the following research areas are of particular significance:

• Intellectual infrastructure. Scientific and engineering disciplines typically have a mathematical language in which to frame and solve their problems—e.g., the use of calculus for disciplines as diverse as aeronautical engineering and chemistry. In contrast, there is no widely understood and adopted theoretical basis for M&S. To some extent, object-oriented modeling (not programming) is helping here, but in practice it usually deals with only some of the problems. While mathematics and systems theory can form a common language, modeling assumptions and their consequences tend to be domain-specific and implicit. Even worse, the only underpinning to many simulations is the computer code in which they are written. To help create the needed intellectual infrastructure, the Department of the Navy and DOD should cooperate with industry and universities in encouraging the development of theory and the promulgation of standard texts and case studies. DOD's adoption of software engineering methodologies (e.g., in the JWARS effort) is useful here. It may also be useful for the Department of the Navy, other Service components, and OSD to cooperate in developing "virtual centers" exploiting the World Wide Web, and in establishing additional peer-reviewed journals and scientific conferences overseen by research institutions.
• Object repositories and interface standards to enhance reusability and composability. Object-oriented technology admits the possibility of assembling major parts of simulations to meet the demands of a particular application from sets of stored objects representing entities and processes. Realization of this capability requires being able to manage large numbers of objects and to ensure consistency despite involvement of multiple developers. Such a capability could reduce costs in simulation development and allow flexibility in simulation application. A key of OSD's strategy in this domain is embodied in the high-level architecture (HLA), which establishes standards for M&S that may be used in federations employing distributed simulation. Despite controversy and anxiety about program costs, the HLA is a needed step in the direction of increased modularity and interoperability. It will have many long-term benefits. The Navy should support and exploit the HLA initiative—recommending modifications as needed.
• Explanation/traceability capability. This capability applies to all phases of the management process. For example, it would help document the source code with multimedia techniques so that one could understand the phenomena being represented, and it would help explain the results of a simulation by displaying the logic trail that led to the results. Realization of this capability would figure centrally in achieving the verification, validation, and accreditation (VV&A) of simulations, both in the formal sense and to the satisfaction of individual users. This capability is important for field commanders, managers, and engineers.

For example, commanders using M&S to assess courses of action may need to know the following: On what assumptions do the simulation outcomes depend critically; should those assumptions be modified and the simulations rerun; what if the component commanders are given contingent orders? The implications of having both comprehensible models and effective explanation/traceability capability are so significant for VV&A that the Navy should undertake more general efforts. The Navy should use commercial products where appropriate and should foster commercial development where necessary, since the capabilities required will have more general value (e.g., to computer-aided design). Such commercial coupling is necessary because development of these capabilities will be expensive.

• Other tools. Many other tools are badly needed. These include tools for (1) automated scenario generation and experimental design and (2) postprocessing and data analysis.

Research Is Not Enough: Planning to Incorporate Its Fruits

There is one further challenge associated with research: assuring that its products are recognized and used appropriately. This is challenging because the research, M&S-building, and user communities are reasonably distinct. Those building M&S often are only minimally acquainted with cutting-edge research in either the phenomenology of warfare operations or modeling methodology. Further, their sponsoring organizations are often more interested in the stability of M&S (and the related gargantuan databases) than in improvements of "theory," the rewards of which may be less than immediately tangible. This state of affairs probably continues because so little higher-level M&S is bounced against empirical realities. The panel has several suggestions here:

• Requiring documentation of "conceptual models" (as distinct from details of the implementing programs and databases),
• Providing scientific review of such models to advise the Navy about the quality of the models in relation to scientific knowledge and best practices in the community, and
• Redefining the JWARS and JSIMS programs to have a continuing component responsible for reviewing, sponsoring, and incorporating research results. The function should be one of nourishing military science, not merely administration or auditing. Major changes are needed if there is to be a resurgence of in-depth study of military phenomena and the kind of open scientific discussion and debate that will lead to top-quality M&S.

RECOMMENDATIONS ON ASSIMILATING AND EXPLOITING M&S

The Need for Strategic Commitment

Finally, there are the organizational problems of assimilation and exploitation. As noted above, M&S is an enabling technology. However, whenever such a cross-cutting technology is introduced, there are organizational and managerial challenges. It is commonplace for the organization to measure the value of investments against the wrong yardsticks (e.g., saving money in narrow domains, as distinct from changing the very way the organization does business and improving effectiveness for mainstream missions). It is also common for investments to go awry because (1) the new technology is procured and used as an add-on without sufficient buy-in and influence by the organization's core responsibility and workers, (2) too much is done by committee without leaders and champions who understand the core business, or (3) the educational groundwork has not been laid. Despite much ongoing success, all of these M&S-related problems are visible in DOD's components, including the Navy and the Marine Corps.

Based on the history of technology assimilation and the specifics of the current situation with respect to M&S, the panel recommends that the Department of the Navy make a strategic commitment to the success of exploiting M&S. Such a commitment would have consequences for organizational structure and responsibility (although the panel makes no recommendations on such matters), investment mechanisms (e.g., assuring that investment funds are available without forcing program managers always to make tradeoffs within their own domains), and the establishment of clear policies and strategies that would make manifest the leadership's demand for constant improvements in "validity" (as understood in the context of sometimes-extreme uncertainties) and usefulness to decision makers. As discussed above, the panel believes that the appropriate strategy would place considerable emphasis on warfare areas and cross-cutting modeling challenges, rather than still more emphasis on computer and software technology. To put this more bluntly, if funding tradeoffs are needed within M&S budgets, then the panel recommends giving higher priority to research improving model content rather than programming or reprogramming of current models.

In addition to investing in research to improve the quality and content of models, the Department of the Navy must organize, plan, and invest strategically if it is to enjoy the potentially great benefits of simulation-based acquisition. In doing so, it should take a long view because, as in other aspects of M&S, there are substantial obstacles. Success will be evolutionary over a period of decades.

Education for Next-generation Officers

One element of a strategy should be increased education in M&S for next-generation officers. The effective exploitation of M&S depends on the experience, knowledge, and wisdom of its practitioners, hence upon their education. The panel recommends increased Navy investment in such education at all levels: for those who acquire and design M&S tools, and also for those who rely on them to guide acquisition, training, and operations. Some of the education should be in the form of enhanced master's and Ph.D.-level programs. Other aspects should include short courses tailored for officers needing refresher courses, technology updates, and preparation for next assignments involving M&S management.

1. There are some partial counterexamples, of course: e.g., "knowledge acquisition" for the "semiautomated forces" (SAFOR) used in the Joint Countermine Operational Simulation (JCOS) and the Army's tactical training system, new modeling of C⁴ISR effects in JWARS, development of object models, and research on configural effects. Even in these cases, however, the work has often been more like "computer modeling" than establishing an empirical and theoretical research base.