|
Items in cart [0] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
Executive Summary
T
he Air Force and the other military services are increasingly inter-
ested in using models of the behavior of humans, as individuals and
in groups of various kinds and sizes, to support the development of
doctrine, strategies, and tactics for dealing with state and nonstate adver-
saries, for use in analysis of the current political and military situation, for
planning future operations, for training and mission rehearsal, and even for
the acquisition of new systems. In this report we refer to this broad class
of models as individual, organizational, and societal (IOS) models. There
are many lines of research on such models, which span several disciplines,
have different goals, and often use different terminologies.
The National Research Council was asked by the U.S. Air Force to
review relevant IOS modeling research programs in the various research
communities, evaluate the strengths and weaknesses of the programs and
their methodologies, determine which have the greatest potential for military
use, and provide guidance for the design of a research program to effectively
foster the development of IOS models useful to the military. The formal
statement of task for the study includes the following specific items:
• Review the state of the art of the subset of the social sciences per-
ceived as having the greatest payoff in terms of informing future
computational model developments.
Review the state of the art in societal1 modeling applications serving
•
1 In
this study, the committee broadened the scope to include individual and organizational
models as well because of the inseparability of all three, given the intended usage.
OCR for page 2
BEHAVIORAL MODELING AND SIMULATION
the U.S. Department of Defense (DoD) and related agencies, with
special emphasis given to computational modeling and simulation-
based approaches.
• Review the state of the art in the three computational modeling
communities outside DoD (cognitive science and individual behav-
ioral modeling, network analysis and multiagent organizational
modeling, and multiresolution modeling and simulation) and iden-
tify strengths and shortcomings in each.
• Identify how gaps in societal behavioral modeling applications
serving DoD and related agencies might be filled by conceptual
models in the social sciences; computational modeling approaches
now under way in the social science community; and closer
linkages between the cognitive science community, the network/
organizational modeling community, and the multiresolution
modeling and simulation community.
• Develop a research and development roadmap to fill current appli-
cation gaps, for the near, mid-, and far term.
Today’s military missions have shifted away from force-on-force
warfare—fighting nation-states using conventional weapons—toward com-
bating insurgents and terrorist networks in a battlespace in which the atti-
tudes and behaviors of civilian noncombatants may be the primary effects
of military actions. These new missions call for agile, indigenously sensitive
forces capable of switching quickly and effectively from conventional com-
bat to humanitarian assistance and able to defuse tense situations without,
if possible, the use of force. IOS models are greatly needed for planning,
supporting, and training for these forces and for evaluating the technology
with which they fight. Models of human behavior in social units—teams,
organizations, cultural and ethnic groups, and societies—are needed to
understand, predict, and influence the behavior of these social units.
For example, models could be used to predict the effects of actions
intended to disrupt terrorist networks, to predict the response of insur-
gents and the local population to the presence of friendly forces in a given
area, or to predict the effects of alternative diplomatic, military, and eco-
nomic courses of action on the attitudes and behaviors of the population
in a region of interest. Models could also be used in training and mission
rehearsal to create simulation environments in which military units could,
for example, experience the effects of their actions on the (simulated)
behavior of a crowd that might either disperse or turn hostile. Models could
also be used to evaluate the likely results of proposed changes intended to
make military command and control organizational structures more agile
and adaptive, and to assess the effects of introducing new technology capa-
bilities on the performance of these organizations.
OCR for page 3
EXECUTIVE SUMMARY
CONCLuSIONS
We use a framework of modeling pitfalls, lessons learned, and future
needs to characterize our major conclusions in a way that will be most use-
ful to the sponsors in the design of future research programs. The problems
or pitfalls identified by the committee are organized in terms of five major
categories:
. Modeling strategy—matching the problem to the real world: Dif-
ficulties in this area are created either by inattention to the real
world being modeled or by unrealistic expectations about how
much of the world can be modeled and how close a match between
model and world is feasible.
. Verification, validation, and accreditation: These important func-
tions often are made more difficult by expectations that verifica-
tion, validation, and accreditation (VV&A)—as it has been defined
for the validation of models of physical systems—can be usefully
applied to IOS models.
. Modeling tactics—designing the internal structure of a model:
Problems are sometimes generated by unwarranted assumptions
about the nature of the social, organizational, cultural, and individ-
ual behavior domains, and sometimes by a failure to deliberately
and thoughtfully match the scope of the model to the scope of the
phenomena to be modeled.
4. Differences between modeling physical phenomena and human
behavior—dealing with uncertainty and adaptation: Problems arise
from unrealistic expectations of how much uncertainty reduction is
plausible in modeling human and organizational behavior, as well
as from poor choices in handling the changing nature of human
structures and processes.
5. Combining components and federating models: Problems arise
from the way in which linkages within and across levels of analysis
change the nature of system operation. They occur when creat-
ing multilevel models and when linking together more specialized
models of behavior into a federation of models.
To summarize, IOS modeling is a complex, emerging science with roots
in many different disciplines. Its advancement requires that researchers
maintain awareness of each other’s work and build on each other’s results,
yet the multidisciplinary nature of IOS modeling has created a fragmented
field. For the field to advance, researchers need better frameworks and
forums in which to compare, discuss, and evaluate their results. The field
currently features a multitude of complex models created using different
OCR for page 4
4 BEHAVIORAL MODELING AND SIMULATION
data and different theories to address different problems, making compara-
tive analysis nearly impossible. Common datasets and challenge problems
are needed in order to learn which modeling approaches and sets of vari-
ables are most useful for specific types of problems.
It seems clear there is no single right model and probably will never
be. The committee thinks that a federated modeling approach, in which
different models at different levels are linked together and component sub-
models can be swapped in and out, is promising for attacking complex IOS
modeling problems. Considerable research needs to be done to make this
federated vision a reality, however. Standards, architectures, methods, and
tools are needed to lower the barriers for developing, linking, and validat-
ing federated models.
Different modeling purposes require different types of models. In the
committee’s judgment, the purpose of the model should drive the appropri-
ate variables to be included in the model. To do this successfully requires
a clear specification of model purpose and criteria for usefulness for that
purpose, which in turn requires that model developers work closely with
the eventual users of the model.
The committee also recommends validation for action, in which the
purpose of the model drives its validation criteria. IOS models cannot
be validated “in general”—they must be validated for a specific use. A
cross-disciplinary community of interest needs to establish and promulgate
accepted standards for validation of IOS models. Triangulation methods
that combine expert judgment, qualitative results and theoretical work,
and quantitative results should be further refined and more widely used.
Common challenge problems and datasets are needed to facilitate docking
of models for comparative purposes.
Finally, models of human beings and their individual and collective
behaviors necessarily include a large amount of inherent uncertainty. This
uncertainty is not a flaw of the model and cannot be designed out of the
model. Human behavior is dynamic and adaptive over time, and it is
impossible at the moment (and into the foreseeable future) to make reliably
exact predictions about it. Researchers need to develop ways to estimate
the probability of plausible outcomes and express those estimates in ways
that are clear and meaningful to model users, who can then judge whether
the results meet their needs. It is important also to avoid raising expecta-
tions about the capabilities of IOS models beyond what can realistically be
expected.
RECOMMENDATIONS
Recommendations for an IOS modeling research and development
program fall into three broad categories: (1) large-scale, integrated cross-
OCR for page 5
5
EXECUTIVE SUMMARY
disciplinary research programs, focused around representative challenge
problems and common datasets; (2) research in six independent areas
that will advance the capabilities to address these integrated problems;
and (3) multidisciplinary conferences, workshops, and other information
exchange forums, with attendees to include not only model developers but
also government program managers and military decision makers.
Integrated Cross-Disciplinary Research Programs
We suggest the funding of multiple large-scale, multiyear research pro-
grams that focus on comparing and, if appropriate, integrating models from
different disciplines, different perspectives, and different levels of detail.
The goal would be to create a level playing field on which the capabilities of
different approaches could be compared and the strengths of each assessed.
The ultimate goal is to move IOS modeling science forward through the
process of comparison, docking, and integration.
It is essential for all participants in each program to focus on the same
well-defined challenge problem instantiated in a common testbed and to use
a common dataset. At the heart of each program would be a representative
problem that is critical for military operations, defined in detail. We have
chosen five representative problems as a starting point for choosing the
problems to be addressed.
The research teams for these efforts should be multidisciplinary, and the
program team should also include military users with operational experi-
ence in the domain for which the models are to be developed. These users
will be ultimate judges of whether model results are useful and will provide
advice on how the results can best be presented. The use of a common
challenge problem and a common testbed will facilitate docking of the dif-
ferent models for purposes of comparison. The development of challenge
problems should be a major focus early in the development of research
programs.
These integrated programs will encourage mutual education between
modelers and operational users. Results should be presented at workshops
for program participants and other interested parties and at public confer-
ences as well as published in the open literature.
Independent Research Thrusts
In support of the integrated programs we recommend, we have identi-
fied six independent areas in which research is needed. Progress in each of
these areas could increase the ability to develop the integrated modeling
capabilities that are needed to address military problems. In each area,
we suggest the funding of multiple research teams from multiple perspec-
OCR for page 6
BEHAVIORAL MODELING AND SIMULATION
tives, with periodic workshops for researchers to exchange results. We also
suggest that operational users as well as government program managers
participate in these workshops.
Thrust 1: Theory Development
Models should be conceptually correct and grounded in the underly-
ing fundamentals of what is known about individual human and group
social behavior. However, current theory in this area does not answer all
of the questions needed to structure models that address relevant issues.
Basic research is needed for theory development, especially for the low-
level social behaviors that are the building blocks for larger scale social
behavioral patterns. This theory development work must involve multiple
disciplines and perspectives with periodic workshops to exchange results.
Theory development challenge problems should be defined to guide
the work, but these can be nonmilitary and need not involve the level of
military detail necessary for the integrated problems discussed above. A
series of workshops should be conducted with researchers to identify key
theory gaps.
Academic institutions are key players for theory development, but
they need information, incentives, and funding to address these theoretical
issues. There is a need to educate researchers in military domains, establish
conferences and journals in which their results can be presented, provide
postdoctoral support, and provide funding that allows researchers to spend
time learning about military domains in depth.
Thrust 2: uncertainty, Dynamic Adaptability, and Rational Behavior
Models must deal with the inherent uncertainty and the dynamic adap-
tation that characterizes human behavior. Models must also be capable of
modeling both rational and nonrational behavior.
Basic research is needed in each of these areas. Issues include
• How should models capture the “uncertainty-in-the-small” associ-
ated with individuals and small groups? How can model structures
and parameters capture this variability, and how much of this vari-
ability must be included for the purposes of the model?
• How should models capture the “uncertainty-in-the-large” associ-
ated with populations and variations in population distributions?
How much variability must be included for the purposes of the
model?
• How can models capture adaptation and learning over time and as
the results of actions by others? For example, people have multiple
OCR for page 7
EXECUTIVE SUMMARY
overlapping identities and allegiances. How can these be captured
in a model, and how can one estimate the effects of actions and
events on the primacy of these multiple allegiances as they affect
decisions and actions?
• What are the factors that contribute to rational, adaptive behavior,
and what factors induce behavior that appears irrational? Models
of both rational and nonrational behavior must capture all the
key factors—cognitive, affective, cultural, and contextual—that
motivate and shape behavior of specific individuals in specific
situations.
Better techniques are needed for understanding the implications of
diversity and variability for model-based sensitivity analysis. Better auto-
mated technology is needed to put the model through its paces to explore
the parameter space effectively and produce robust results.
Thrust 3: Data Collection Methods
The difficulty of obtaining data is an ongoing challenge for IOS model-
ing. Research is needed to develop better data collection processes through
field studies, experiments, and potentially massively multiplayer online
games (MMOGs).
Although a variety of ethnographic data collection techniques are cur-
rently in use, they need to be better tailored to the needs of IOS models. For
field data collection, it is necessary to bring modelers and data collectors
together to develop data ontologies, joint specifications, and data collection
methodologies and tools that are specifically tuned to IOS models.
MMOGs are a potential untapped resource for collecting social and
behavioral data on a large scale. We recommend the creation of a MMOG
facility and the funding of basic research to determine if MMOGs can be
used to test, verify, and validate IOS models. We recommend that fund-
ing be put into developing the science of MMOGs. We note that funding
MMOGs is a risky endeavor, but we think that the potential benefits out-
weigh the risks.
Thrust 4: Federated Models
It is a fundamental conclusion of the committee that no single model-
ing approach can provide all the capabilities needed by DoD. We recom-
mend a federated approach in which modeling components are created
to be interoperable across levels of aggregation and detail. For example,
a federated model might include a detailed representation of a few key
individuals, linked to group-level models of different cultural groups and
OCR for page 8
BEHAVIORAL MODELING AND SIMULATION
terrorist organizations, linked to geographic sector–level models of the level
of unrest in a city. This approach is flexible and extensible, allowing the
addition or subtraction of models at different levels of detail as needed for
the problem to be addressed.
Combining model components to create federated models in the sense
being recommended requires deep semantic interoperability (i.e., theoreti-
cal consistency) and presents difficult challenges. To create semantic inter-
operability, it is necessary to recognize that the links among components
are themselves elements of the model. Research is needed on:
• How to ensure that the models being federated embrace compat-
ible assumptions regarding concept abstractions, entity resolution,
time scale resolution (tempo), uncertainty, adaptability, docking
standards, input-output, semantics, etc.
• How the components of the federated model should be encapsu-
lated and which elements must be exposed to other components.
• How specific classes of models should be linked (e.g., cognitive
models to social network models).
• How to ensure dynamic extensibility.
In addressing these issues, IOS modelers should maintain awareness of
research and development in model federation in the larger modeling and
simulation community.
Thrust 5: validation and usefulness
Current VV&A concepts and practices were developed for the physical
sciences, and we argue that different approaches are needed for IOS models.
Specifically, we recommend that a “validation for action” approach be used
that assesses the usefulness of a model for the specific purposes for which
it was developed. It is thus very important that the purpose(s) and criteria
for judging success be clearly stated a priori for all models. We recom-
mend organizing national workshops to agree on appropriate processes for
VV&A of IOS models and to outline a roadmap for developing improved
processes and standards. On the basis of the results of this workshop, we
recommend that a DoD-wide authority develop and disseminate VV&A
processes and standards for IOS models. Basing model validation on the
usefulness of the model for specific problems requires that model purposes
be clearly stated by model users and clearly understood by model devel-
opers. We suggest that, as part of developing a VV&A standard for IOS
models, clear guidelines be developed for specifying model purpose.
OCR for page 9
EXECUTIVE SUMMARY
Thrust 6: Tools and Infrastructure for Model Building
It is important to reduce the barrier to entry for developing models,
modeling tools, frameworks, and testbeds. Scientists should be able to build
and validate models without the large overhead currently associated with
many DoD modeling and simulation investments. It should be possible to
easily tailor existing models for specific purposes.
Sharing of IOS modeling knowledge across disciplines, as facilitated
by the conferences and workshops recommended below, will support this
goal. Work is also needed in developing an infrastructure for IOS modelers,
including a national network of possible collaborators, common databases
for model development and testing, and frameworks and toolkits for rapid
model development.
The limited data that exist for IOS models are often not accessible to
model developers. We recommend national web-accessible data repositories
that are open to researchers who seek to inform and test models. For mili-
tarily relevant domains in which some data are classified, we recommend
an investment in automated tools to sanitize the data.
We also recommend the development and maintenance of an online
web-based catalog of general approaches, models, simulations, and tools.
The notion is to develop something along the lines of the Defense Model-
ing and Simulation Office’s Modeling and Simulation Resource Repository
or the clearinghouse at Carnegie Mellon’s CASOS site (http://www.casos.
cs.cmu.edu). To be effective, the envisioned site needs careful consideration
in terms of organization, content, currency, and usability. This cannot be a
one-time effort. It needs significant startup funding and continued support
over its lifetime.
Multidisciplinary Conferences and Workshops
A number of the issues and problems identified by the panel were the
results of the failure of different disciplines to exchange information, or they
resulted from misunderstandings among government funders of model devel-
opment efforts, military users of models, and model developers. Because of
the diversity of this group, there is no natural forum for them to exchange
information. We recommend the organization of special-purpose workshops
around the integrated research programs recommended above, as well as
workshops for the independent research thrusts described above.
IOS modelers need to be educated on:
• The nature of the military decisions for which models are relevant.
• Desired model functionality.
OCR for page 10
0 BEHAVIORAL MODELING AND SIMULATION
• The most useful form for presenting model results.
• The value of work performed by others outside their discipline.
• Feasible and appropriate VV&A approaches for IOS models.
Operational users and managers need to be educated on:
• The value of multidisciplinary approaches and the need for review
of models from multiple perspectives.
• The inherent uncertainty associated with model predictions.
• The value of models for sensitivity and trade-off analysis (versus
the one right answer).
• The design of virtual experiments to assess results over a range of
conditions.
• Reasonable definitions of validation for IOS models, feasible
approaches for VV&A testing, and why these approaches differ
from those used for physics-based models.
The recommended workshops should involve model developers, opera-
tional military users of the models, and government personnel making
funding decisions regarding model development.
Roadmap for Future Research and Development
The committee recommends a use-driven research program to extend
the state of the art in IOS modeling, focused around a series of challenge
problems—clear specifications of the uses to which the model is to be put,
defined to be relevant to military needs, and expanded over time as progress
is made in modeling approaches, tools, and technologies. The purpose of
the model, as captured in the challenge problems, drives the theory to be
applied, the data to be used, and the model development. Model develop-
ment is made easier by modeling tools and infrastructure and relies on
federation standards to ensure the interoperability of model components.
Once the model is developed it is validated by asking the question: Is the
model useful for its intended purpose?
The recommended program proceeds in a cyclical fashion. Based on the
answers to the question “Is the model useful?” new models may need to be
developed, new theory and new data (and new types of data) may be needed,
and new interoperability standards, tools, and infrastructure may be required.
Depending on the results, the problem itself may need to be redefined, clari-
fied, or expanded. These challenge problems, combined with periodic work-
shops and conferences to compare and exchange results, serve as a unifying
force and a common ground for the fragmented field of IOS modeling, pro-
viding a foundation on which scientific progress can be made.
