|
Items in cart [0] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 271
8
Common Challenges in IOS Modeling
T
his chapter discusses broad issues and challenges that are encoun-
tered across the range of individual, organizational, and societal
(IOS) modeling approaches and methods, highlighting problems
that need to be solved for these modeling approaches to be most useful
for the military’s needs. We first describe issues of integration and inter-
operability, the challenges that confront modelers and simulation develop-
ers when they attempt to integrate multiple models and simulations, with
the goal of making them interoperable—that is, able to use output from
one model as input for another. Next we describe some of the challenges
(and potential benefits) of developing and using modeling frameworks
and tools that facilitate the development of IOS models. We then describe
issues of model verification, validation, and accreditation (VV&A), issues
that are especially challenging for the modeling of human behavior. Finally,
we discuss some of the challenges posed by the data requirements of IOS
models in light of the realities of the data and information available to
model developers and users. In each section we note some potential solu-
tions to the challenges.
INTEgRATION AND INTEROPERABILITy
In this section, we discuss the issues that confront modelers attempting
to integrate models developed with different internal structures, at different
levels of granularity, or with inconsistent inputs and outputs. The nature of
the challenges requires that the discussion be quite technically sophisticated
and use terminology and concepts that may be unfamiliar to many readers.
OCR for page 272
BEHAVIORAL MODELING AND SIMULATION
We have tried to define some of the terms in footnotes, but a simplified
discussion would not do justice to the subject matter.
Model Interoperability: Incompatibilities and Functionality gaps1
There are several fundamental issues (and associated hard problems)
that need to be addressed in undertaking the development of an inter-
operable framework of IOS models. First and foremost is the problem of
making existing or even new models interoperable, as these are developed
independently (i.e., with no coordination) by different software design
and development teams, in consultation with domain experts having vari-
ous levels of skills and expertise. A very common approach is to build a
wrapper around an existing model, thus converting it to an input-output
(I-O) black box, or to provide an intelligent agent operating autonomously,
which communicates with other models in the network. But this approach
is likely to introduce other types of gaps and incompatibilities between
models, some of which are identified in Table 8-1 and illustrated in Fig-
ure 8-1. We discuss here the need to identify an overall methodology to fill
these gaps, including various intelligent automated techniques, processes,
and guidelines, as well as aid from human subject matter experts and ana-
lysts whenever needed.
Interface Incompatibility
The first problem shown in Figure 8-1 (in the top row) concerns inter-
face incompatibility between two models that either already exist or are
being developed independently. If we intend to feed output from model A
about a certain object X as input to model B, then some mismatch between
the output and input may occur in terms of the assumptions about the
numbers and types of X’s attributes. This is often straightforward but
tedious to deal with, often merely involving translation from one descrip-
tive framework to another (e.g., from numerical values—1, 2, 3, . . . —to
“fuzzy” values—low, medium, high, . . .). A bigger problem ensues when
different levels of resolution are used to represent the same object in two
different models. If model A provides a high-resolution object representa-
tion of X (e.g., a map, enemy force estimates) for model B, and model B
needs a low-resolution representation (e.g., latitude/longitude of enemy
center of gravity), then some aggregation process must be conducted, usu-
1 Much of the work described in this section on model integration and interoperability was
performed by John Langton and Subrata Das at Charles River Analytics with support from the
Air Force Research Laboratory, Information Directorate (AFRL/IF) under contract FA8750-
06-C-0076, and adapted from Langton and Das (2007).
OCR for page 273
COMMON CHALLENGES IN IOS MODELING
TABLE 8-1 Gaps and Incompatibilities Between IOS Models
Type Definition
Interface Mismatch between the data types of different models or outputs of one
model and inputs of another, e.g., real number vs. Boolean
Ontological Different relationship structures, naming schemes, etc., in ontologies for
different models
Formalism Different logic and inferencing mechanisms and procedures for different
models
Subdomain Differing domains and dynamics between PMESII model dimensions, e.g.,
gaps economic vs. social
SOURCE: Langton and Das (2007).
About object of
type X
Interface
Model A Model B
Incompatibility
Input Output
Input
Output
Ontological
Incompatibility
Formalism Bel(X) p(X)
Incompatibility
Subdomain
Gaps Economy Social
FIguRE 8-1 Illustration of gaps and incompatibilities between IOS models.
8-1.eps
SOURCE: Langton and Das (2007).
ally based on one approximation method or another. The reverse process
is much more difficult, going from a low-resolution output to a high-
resolution input, since, in effect, missing input attributes have to be inferred
or approximated and filled in. A number of approaches can be used to
resolve the interface incompatibility. These are described in the section on
interoperability recommendations below.
OCR for page 274
4 BEHAVIORAL MODELING AND SIMULATION
Ontological Incompatibility
The second problem illustrated in the figure is ontological incompatibility
between models, which arises due to differing vocabularies and expressive
power in their respective ontologies.2 Different teams of engineers and subject
matter experts with a diverse range of expertise, knowledge, and cognitive
capabilities independently creating models will inevitably develop and use dif-
ferent underlying ontologies, which in turn will give rise to incompatibilities
across models. Initially, one might suggest the development of a common
ontology for the set of all possible models; however, many failed efforts in
this direction make it clear that developing a universal ontological standard
for model creation is impractical, if not theoretically impossible. Moreover,
if models are to be built rapidly, analysts should ideally be free to use a
model-building environment of their own choosing without assistance from
knowledge engineers. The analysts should not be constrained by a predefined
ontology to express their knowledge, which usually inhibits their expressive
flow. Hence, rather than proposing to develop a common ontology for the
model space, one approach is to focus on facilitating better mapping capa-
bilities between differing ontologies. For example, there are tools that can
map ontological terms from one domain to another by solving the problems
of synonymy and polysemy;3 these clearly offer hope for translating differ-
ing ontologies used in the models. In some cases of incompatibility between
the underlying ontological structures of the models (e.g., semantic networks
versus logical expressions), one domain can be mapped to another by pro-
viding a more expressive ontological structure for one of the models (e.g.,
semantic networks can be mapped to first-order logical sentences). Therefore,
some parts of the ontological incompatibility problem can be addressed via
automated techniques. A number of approaches can be used to resolve the
ontological incompatibility, described below.
Formalism Incompatibility
While ontological incompatibility creates problems due to multiple
ways of designating an entity, the formalism incompatibility shown in
Figure 8-1 is concerned with multiple ways of instantiating the object entity
2 An ontology, for the purposes discussed here, is “a systematic arrangement of all of the
important categories of objects or concepts which exist in some field of discourse, showing
the relations between them. When complete, an ontology is a categorization of all of the
concepts in some field of knowledge, including the objects and all of the properties, relations,
and functions needed to define the objects and specify their actions” (http://www.answers.
com/ [accessed July 2007]).
3 Synonymy refers to one referent (concept) with several words that can denote it (plain English
examples: big, large); polysemy refers to one word denoting multiple referents (plain
English examples: break; park).
OCR for page 275
5
COMMON CHALLENGES IN IOS MODELING
computationally represented in the model. For example, uncertainty can
be expressed not only in terms of probability values, but also via various
other formalisms, such as certainty factors, the Dempster-Shafer measure
of beliefs (Shafer, 1976), and numerous other qualitative dictionaries. These
are fundamentally incompatible with each other, both in terms of their
underlying conceptual representation of uncertainty and probabilistic rea-
soning, and in the sense of having different types of scales. Conversion
between two such formalisms often requires deep understanding of the
models and their formalisms, thus breaking the simple I-O black box idea
of encapsulation. Specialization of formalism is often appropriate to map
one approach to another. For example, probability theory is a special
case of Dempster-Shafer theory that allows beliefs to be expressed only
on singleton sets, facilitating development of a mapping from probability
models into Dempster-Shafer models.
Subdomain gaps
If one wants to feed the output from a model in one domain to
another, it will require an analyst or domain expert with knowledge of
both domains to bridge the subdomain gaps. This is due not only to the
ontological gaps between the domains being considered, but also to differ-
ing dynamics between the domains. Addressing this problem requires the
skills of experts from the respective domains or ideally ones who are expert
in both domains.
A number of approaches can be proposed to bridge such gaps, by high-
lighting possible correspondences between concepts and variables across
domains, described below. Recommendations are also made for more compre-
hensive approaches that could be part of a long-term development effort.
Figure 8-2 provides an illustration of model interoperability—focusing
on political, military, economic, social, information, and infrastructure
(PMESII)-related issues—with interactions among three layered models:
one focusing on the social structure, one on the community infrastructure,
and a third on the underlying information models, respectively from top
to bottom.
The infrastructure model in the middle models a stabilization and
reconstruction operations (SRO) model, developed by the AFRL/IF,
(Robbins, Deckro, and Wiley, 2005) using a system dynamics modeling
approach (see Chapter 4), and captures a sequence of influences among
variables, starting from the power supply at an electrical power substation.
The generated power is fed into an industrial water plant, which produces
water consumed by oil field work. An oil field produces crude oil to be
refined by a refinery. Refined fuel is used to generate power, which in turn is
supplied to various power substations, thus forming a loop. It is especially
OCR for page 276
High
Level of Anger
Level of Anger
Loss
Medium Influence Diagram
Among Population
Among Population
Low
Model
Fragment of Social
Model in Town Aligned
Refined
Refined Sufficient
Drinking
Drinking
with the USA Power Fuel
Power Fuel Food Supply
Water
Water
Drinking
Power Water Refined Fuel
Power Industrial Oil Oil Power
Fragment of
Substation Water Plant Field Refinery Generators
Infrastructure Industrial Refined
Crude
Power Fuel
Model Water
High Voltage Power SRO Model
Refined Fuel
Terrorist Group A Leads Leader X Angers
8-2.eps
Attr Attr Attr Concept Graph
Fragment of Behavioral Model
Information Model in Town
with Terrorist Stronghold Aggressive Diplomatic Quick to Anger
Attr Attr
Attr
Imminent Attack
Causes
in landscape view, smaller type is 5.73 pt
Use of Threatening Calling for Inviting Suicide Observable Intelligence
Phrases Jihad Bombers
FIGURE 8-2 Interoperability of three different PMESII models.
SOURCE: Langton and Das (2007).
OCR for page 277
COMMON CHALLENGES IN IOS MODELING
difficult to reason with these types of graphs, containing such loops span-
ning many variables, as it creates an additional burden for discounting the
variables’ self-influence.
The social model at the top of the figure captures the impact of these
infrastructure-related variables on the society, using influence modeling
technology (see Chapter 6). The model specifically captures the influence
of the four variables of power, drinking water, refined fuel, and sufficient
food supply on a variable representing the level of anger of the population
in a town aligned with coalition forces. The dynamics of the social model
are that short supply in any one of these three consumable products will
increase the level of anger among the local population. In fact, if a terrorist
organization became aware of the mid-layer SRO model sequence in the
infrastructure, then the power substation would assume heightened impor-
tance in the eyes of the terrorist strategists: an attack on a substation would
not only cripple other services in the loop, but would also drive the senti-
ment of the local population against the coalition. Note that the diamond
box represents the expected mission utility in line with the level of anger.
The utility (although difficult to quantify here) should go up when the anger
level is down and vice versa.
The behavioral information model at the bottom of the figure illus-
trates how a model of a terrorist leader can be built using a concept graph
approach (Sowa, 1984) in which concepts are represented by rectangles
(e.g., [Person: Leader X] and [Behavior: Aggressive]), and conceptual rela-
tions are represented by circles (e.g., has Attributes) and soft-cornered
rectangles (e.g., Leads, Causes). An analyst can query such a model to
determine who the terrorist leader is and the nature of the leader based
on various observable intelligence. Such a leader X, who leads the terror-
ist group A, can possess different types of behavior attributes, including
aggressive, diplomatic, quick to anger, etc. If the leader is quick to anger
and there are some stimuli to make the leader angry, then an attack on
friendly targets may be imminent. One such stimulus would be coalition
forces stopping the supply of oil to the region, as indicated by the link to
the SRO model above.
The key issue here is the interoperability among the models. Note
that although an I-O connection has been made between the two variables
Oil Refinery and Refined Fuel of the top two models, they are ontologi-
cally incompatible as defined earlier. However, they can be made compat-
ible by recognizing that the term “Oil” is synonymous with “Fuel,” and
“Refined” and “Refinery” have a common base word. Another difficult
compatibility problem is illustrated by the fact that there is no input for the
variable Sufficient Food Supply in the social model, illustrating the inter-
face incompatibility described earlier. One can envision, however, that this
“sufficiency” concept could be automatically computed from the supply of
OCR for page 278
BEHAVIORAL MODELING AND SIMULATION
food previously recorded in available databases to bridge this last gap. A
number of recommendations for resolving specific model incompatibilities
and functionality gaps are provided below. More general approaches to
resolving more than one of these gaps simultaneously are a current area of
study (Langton and Das, 2007).
Recommendations for Resolving gaps in Model Interoperability
A number of approaches can be taken to maintain, adapt, and integrate
diverse models in the context of the interoperability gaps just defined.
Dealing with Interface Incompatibility
Interface incompatibility generally refers to two or more models having
different types of data for their inputs and outputs and thus not being able
to interoperate without some form of data conversion. There are at least
three types of interface incompatibilities:
. I-O format incompatibilities: string versus binary, real versus integer,
fixed versus floating point, numeric versus Boolean, incompatible
scale, incompatible zero point, date-time format, color format.
. Logical incompatibilities: number of I-O points (e.g., three out-
puts versus four inputs—RGB to CMYK is a trivial example), I-O
timing (e.g., fast output versus slow input).
. Model persistence format incompatibilities: XML versus YAML,
OWL versus RDF, etc.
One way to deal with these issues is via a development interface that
provides a basic set of translation functions that can learn from user
interaction over time. A graphic user interface (GUI) would allow users
to explicitly modify, add, and remove interface translation functions, as
illustrated in Figure 8-3. Users could also specify these translation functions
within an ontology or the XML schema of a model, based on specifica-
tions derived, for example, from an evolved, global ontology. A full-scope
GUI would then allow users to explicitly modify, add, and remove inter-
face translation functions. A number of potential translation functions
are described below in the context of the type of incompatibilities each
addresses.
Dealing with I-O Format Incompatibilities
Many interface incompatibilities fall within this category, and most
solutions can be resolved by some combination of the following:
OCR for page 279
COMMON CHALLENGES IN IOS MODELING
About object of
type X
Model A Model B
Input Output
Input
Output
X1 Y1
Contextual
X Y Information
2 2
⇒
... ...
Xm Yn
Y j = f ( X1i ,..., X i , C)
k
Develop an interface for encoding
commonly used transformation functions
Ex: PROJECT(X3), X1*X2+X3, min(X1,
X2), gen(X3), fuzz(X3)
FIguRE 8-3 Resolving interface incompatibility.
8-3.eps
• Normalization: mapping any value to lie between 0 and 1 relative
to its minimum and maximum possible values.
• Weighting: scaling a value, typically in relation to other values.
• Fuzzification: randomly generating a number to lie within some con-
straining interval (e.g., some random number between 0.3 and 0.6).
• Discretization: “binning” values according to their range and a range
they must fall within—somewhat like rounding—sometimes taking
their distribution into account (e.g., 0.5 within a range between
0 and 1 can be discretized to 1 for a range of only 0 or 1).
XML schemas often exist to support model file persistence. These
schemas define the elements of a model along with the possible values
they can take on. XSLT can then be used along with a number of standard
OCR for page 280
0 BEHAVIORAL MODELING AND SIMULATION
translation functions for integrating inputs and outputs of two models on
relevant nodes or links. These functions can also be adapted according to
user interaction over time.
Dealing with Logical Incompatibilities
In some cases, one model may have more outputs than another’s inputs
or vice versa. When integrating models, we therefore need methods for
addressing these situations. For an overabundance of values, we can simply
use some form of aggregation. Again, a model schema or ontology can
specify how this aggregation should be performed, or the user could specify
this through the above-mentioned GUI. In the case in which we have only
one value but must map to more than one, we can simply duplicate the
value or partition it according to any context provided in the model schema
or ontology.
In some cases, the sample rate of inputs and outputs may differ. One
way of dealing with this is through smoothing and resampling.
Dealing with Model Persistence Format Incompatibilities
In essence, this issue really mirrors the greater task of integrating
models. The existence of a standard schema or ontology for different
models would immediately resolve this issue. However, we cannot now
depend on such a standard or on adherence to it. A partial solution may
be to evolve or derive a standard schema or ontology. In either case, most
effective solutions will entail the use of XML and XSLT for the translation
of one model format to another.
Dealing with Ontological Incompatibility
Ontological incompatibility refers to two models having different struc-
tures, including the entities they specify and the relationships between
them. For instance, a rules system model may have several pairs of nodes
connected by one link (precedent and consequent), whereas a Bayesian net
typically has more of a tree structure. Nodes can have different names,
graphs can be directed or undirected, and two models representing the same
system can be at different resolutions and thus include a different number
of nodes and links. The principal issue of this incompatibility is determin-
ing which entities, nodes, or links in different models should map to one
another for interoperation.
Syntactic heuristics: The labels and descriptions of nodes and links in
differing models can be compared on the basis of their raw string content.
If these string components match, then the nodes or links may be a match
OCR for page 281
COMMON CHALLENGES IN IOS MODELING
as well. For instance, “runway16” may map to “runway.” A threshold for
how many characters must match to infer a string match must be specified.
This type of matching can also include matching nodes/links based on the
range, cardinality,4 and other attributes of their possible values.
Semantic heuristics: Nodes and links from different models can be
compared on the basis of the semantics of their labels, descriptions, and
any other textual metadata specified in an XML file, XML schema, or
ontology. Elements from different models that have a semantic similarity can
then be mapped to one another for model integration. For instance, a node
with the name “airport” in one model may be mapped to a node with the
name “runway” in another model on the basis of the semantic similarity of
their labels. Semantic similarity is determined by the relations between two
words as derived from statistical usage, ontologies, thesauruses, dictionaries,
etc. There are both service-oriented architectures and application program
interface specifications for this purpose, including WordNet (Al-Halimi and
Kazman, 1998) and Lexical Freenet (Beeferman, 1998).
Relation mapping: Relation mapping can be used to address ontologi-
cal incompatibility by mapping nodes from one model to nodes of another
based on their relations (how they are connected) within their individual
models. With this information, we can then suggest potential mappings
between nodes of different models based on the similarity of their relations
within their respective models. Consider the nodes α of model A and β of
model B. Although these nodes may have very different names, they may
have very similar relations. For example, both could influence five other
nodes and be influenced by four other nodes. Based on their similarity, we
may be able to deduce that these nodes can be mapped together for model
integration. It is important to note that relations encompassing a node are
not merely all of its incoming and outgoing links; they also include features
identifying how the node affects any other nodes in the model. While this
approach should rarely be used to draw links automatically, it could be used
to make effective recommendations.
Model node aggregation: Model aggregation can be used to address
ontological incompatibility by identifying how sets of nodes in different
models with differing cardinalities may be mapped to one another. It may
be the case that a node α in model A maps to a subset of nodes N in model
B, resulting in incompatible ontologies. For example, consider α to be the
node airport and N to be the subset of nodes runway, plane, radar, and air
traffic control. The question is, which nodes should airport be mapped to
for model integration? We can use the semantic similarity of the labels on
the nodes of N (e.g., interfacing with WordNet for ontological inference)
4 In
mathematics, the cardinality of a set is a measure of the “number of elements of the set”
(Wikipedia, see http://en.wikipedia.org/wiki/Cardinality [accessed Feb. 2008]).
OCR for page 318
BEHAVIORAL MODELING AND SIMULATION
tion accounts for more variance in recall and substitution errors than per-
sonal attributes, such as gender, race, and age.
In contrast to such well-developed conceptual frameworks, broad
metaphors (brains as information-processing devices, organizations as
cultures) are not really subject to verification or falsification. Whether or
not they are used in a particular domain is likely to depend largely on face
validity and established precedent. In evaluating the usefulness of a broad
conceptual model, the yardstick is often not how well supported the model
is, but how much interesting research it inspires. Even when a verbal model
seems, in principle, to be subject to falsification, the underspecification of
relations and processes often means that a rather broad array of different
outcomes can be presented as consistent with the theory. As Harris (1976)
noted in his paper entitled “The Uncertain Connection Between Verbal
Theories and Research Hypotheses in Social Psychology,” theoretical terms
often are not defined, boundary conditions are unspecified and, under
various plausible interpretations of assumptions or conditions, several
well-known theories include internal contradictions and inconsistencies
(cited in Davis, 2000).
validation of Cultural Models
Cultural inventory models rely on ethnographic observation and are
therefore both time-consuming to develop and highly subjective. Having
multiple independent observers helps ameliorate the subjectivity problem,
but it is expensive.
Dominant trait models, such as the Hofstede dimensional models, can
involve two sets of data. The first set is used to derive the dimensions.
These can be validated by a number of different statistical methods, such
as factor analysis. Once these are fixed, another set of data is obtained to
score each new culture on the dimensions. These data have to be obtained
from willing natives of the culture, and the data have to be updated over
time because cultures change.
validation of Cognitive Models
While there is increasing emphasis on validation of cognitive architec-
tures, validation remains one of the most challenging aspects of cognitive
architecture research and development. “[Human behavioral representa-
tion] validation is a difficult and costly process [and] most in the commu-
nity would probably agree that validation is rarely, if ever done” (Campbell
and Bolton, 2005, p. 365). Campbell goes on to point out that there is no
general agreement on exactly what constitutes an appropriate validation
of a cognitive architecture. Since cognitive architectures are developed for
OCR for page 319
COMMON CHALLENGES IN IOS MODELING
a wide variety of reasons, there is a correspondingly wide set of validation
(and evaluation) objectives and metrics and associated methods. Lack of
established benchmark problems and criteria exacerbates this problem.
validation of Cognitive-Affective Architectures
In spite of the challenges associated with validation of emotion models
and cognitive-affective architectures, progress is being made in the area.
A promising trend in emotion modeling is the increasing emphasis on
including evaluation and validation studies in publications. As is the case
with cognitive architectures, no existing emotion models or cognitive-
affective architectures have been validated across multiple contexts and a
broad range of metrics. However, some important evaluation and valida-
tion approaches and studies exist and are discussed in detail in Chapter 5.
Cognitive-affective architecture validation has not yet reached the stage
of systematic comparisons that is beginning to be used for their cognitive
counterparts. However, given the recent emphasis on validation in the
computational emotion research community, such studies are likely to be
taking place in the near future.
validation of Agent-Based Models
Agent-based models (ABMs) are computational frameworks that permit
the theoretical exploration of complex processes through controlled repli-
cable experiments (see Chapter 6). In principle, these experiments could be
run entirely with artificially generated initial conditions, parameter values,
and functional forms. Nevertheless, their ultimate usefulness depends on
the extent to which they prove capable of shedding light on real-world
systems, that is, their ability to enhance understanding and guide decisions
and actions.
When validation of ABM frameworks is attempted, the validation
is generally restricted to small areas of performance. A typical approach
to validation is to run an experiment using an ABM framework, collect
data from this experiment, statistically analyze the results to generate the
response surface, and then contrast the response surface with real data. It
is easy, even with only a few variables, to generate such a quantity of data
from an ABM framework that there are no existing data with which to
compare them, no existing statistical package can handle them, and most
desktops cannot store them. Therefore, typically only small portions of the
overall response surface can be estimated at once. The size of the analyzed
response surface is thus often dictated by the user’s interests and the critical
policy or decision-making questions at issue (i.e., the action domain and the
scenarios relevant to that domain, as discussed above).
OCR for page 320
0 BEHAVIORAL MODELING AND SIMULATION
ABM researchers have recently begun to explore promising new
approaches to validation. For example, a number of them are now advo-
cating iterative participatory modeling (IPM) as an effective way to incre-
mentally achieve validation of the structural, institutional, and behavioral
aspects of the complex systems they study. For an introductory exposition
of IPM, see Barreteau (2003). The essential idea is to have multidisciplinary
researchers join with stakeholders in a repeated looping through a four-
stage modeling process: (1) field study and data analysis, (2) scenario
discussion and role-playing games, (3) ABM development and implementa-
tion, and (4) intensive computational experiments.
The new aspect of IPM relative to more traditional participatory model-
ing approaches is the emphasis on modeling as an open-ended collaborative
learning process. The modeling objective is to help stakeholders manage
complex problems over time through a continuous learning process rather
than to attempt the delivery of a definitive problem solution.
In addition, ABM researchers are also beginning to explore the poten-
tial benefits of conducting parallel experiments with real and computational
agents for achieving improved validation of their behavioral assumptions.11
A critical concern is how to attain sufficiently parallel experimental designs
so that information drawn from one design can usefully inform the other.
Recommendations for Developing and validating IOS Models
We have argued that IOS models should be validated beginning with
the purpose and then considering the action set, scenarios, and if-then
relations in the specific situation. The committee makes a number of sug-
gestions for modeling and simulations that will facilitate the validation of
a specific model.
Check with Multiple Experts
Four different experts should examine an IOS model: the users of the
model, the scenario experts, the if-then or domain experts, and the modelers
themselves. Modelers cannot examine a model by themselves; they tend to
focus on the verification with less emphasis on the purpose of the model.
For an action model, the user is very important to check the relevance and
feasibility of the action set. The scenario expert should examine the uncer-
tainties and unknowns. Domain experts are particularly knowledgeable
about the if-then relationships. However, their knowledge is not necessarily
11 See http://www.econ.iastate.edu/tesfatsi/aexper.htm for annotated pointers to ABM
research on parallel experiments with real and computational agents; see also the survey by
Duffy (2006).
OCR for page 321
COMMON CHALLENGES IN IOS MODELING
framed in this manner, so some adjustment may be required. For example,
domain experts know about “what is” and “what has been” but may be
less certain about “what might be” outcomes. However, they are likely
to point out errors in the models for what might be and limits of what is
known. Each expert can contribute to the validation of an action model.
It is unlikely that any single expert can ensure a valid action model alone.
The structure and content of the model provide a template for a procedure
by which multiple experts can validate different aspects of an integrated
action model.
Keep the Model as Simple as Possible for Its Purpose
An IOS model does not have to be complex. Parsimonious models are
preferred. The corn farmer action model is simple and does not capture the
complexity of weather forecasting or the chemistry of fertilizers. But it is
understandable and permits the farmer to make a decision and take action.
Action models that are intuitively understandable to decision makers (trans-
parent) are preferred. An action model that is disconnected from a deci-
sion maker’s intuition and from concepts he or she is familiar with does
not permit interplay between the decision maker and the model. In short,
complicated, nonintuitive action models require decision makers to accept
the implications of the models on blind faith. Action models should aid
decision makers, not replace them.
Examine “What Might Be” as Well as “What Is”
“What is” should mimic the real world within limits. “What is” models
are a basis for “what might be.” A model that has little or no correspon-
dence with the real world is not likely to be relevant for what might happen.
What might be is very important for action models—particularly in new
situations (Burton, 2003). Many of the relevant action-scenario combina-
tions have not been observed in the past. So the model must be relevant for
action beyond what is or what has been to new situations. For example,
it would be desirable if the illustrative village deployment action model
could be used reliably in other similar situations, say for the withdrawal
from a village as well as entry. But it is not likely that the model could
be used to help plan an action to disarm a resistance cell. Presumably
this would require a more detailed model of the functioning of the cell.
Whether it would be desirable to develop one model to handle both entry
and cell disarmament or two separate models would presumably depend
on economies of scope—Is there anything to be gained by considering both
issues jointly?—and on computational implementation costs. IOS models
should be developed and examined beyond what is to what might be. At
OCR for page 322
BEHAVIORAL MODELING AND SIMULATION
the same time, it is important to examine the limits of the model and not
use it in situations in which it might be inappropriate. As suggested above,
simplicity is desirable, but it must be balanced so that the action model is
useful for its purpose.
IOS models are likely to forecast a range of possible outcomes, some
more likely than others, and to incorporate many factors that are highly
uncertain and, indeed, unknowable at the time the model is developed.
How then can such models be validated? Popper, Lempert, and Bankes
(2005) argue that models used to explore policy alternatives for an uncer-
tain future should not be expected to yield predictions that can be tested but
rather should be used to explore and compare possible outcomes under a
variety of possibilities in order to select strategies that are robust—yielding
the best overall results across a variety of possible futures.
Postevent outcomes can also be used to evaluate models, although
models are not necessarily incorrect if the actual outcome that occurred
was not the one forecast to be most likely. Unlikely events do occur, and
many IOS applications do not permit the replication that would generate
a distribution of actual outcomes. A very useful approach would be to
develop multiple models that take different perspectives and use different
theories and data, merge their predictions to create zones of likelihood, and
compare their forecasts with the actual outcomes (see Docking below). As
with other validation approaches, the value of the model’s results depends
on its intended use, so the degree to which forecasts need to correspond to
reality will depend on the model’s purpose.
use Model Touching for validation
Model touching is comparison or juxtaposition of models. There are
many ways to bring models together. Here is a list:
• Bring experts (as described above) together to develop and examine
the model.
• Compare the action model with qualitative studies for the situation
or domain.
• Check with other studies that might be empirically based on data
from the field or from experiments.
• Compare with computational models that are based on field data.
Docking. Docking is the bringing together of two models—a metaphor
borrowed from space exploration. More precisely, docking is an evaluation
of the extent to which two or more different models of the same action
situation can be cross-calibrated so that they yield the same outcome (or
outcome probability distribution) given the same contingency condition
OCR for page 323
COMMON CHALLENGES IN IOS MODELING
(Axtell, Axelrod, Epstein, and Cohen, 1996). Docking goes beyond model
touching to compare in more detail. It can provide a better understanding
of the true connections relating the three key elements of an action model:
the actions, the scenarios, and the possible outcomes resulting under each
contingency condition (scenario-action pair). Docking gives confirmation
that we have a reasonable understanding of an action situation, and that
our conclusions are being driven by the intrinsic nature of the action situ-
ation and not by idiosyncratic aspects of the model implementation. One
possible approach is to compare how different models perform under the
same benchmark action-scenario combination, which can provide insight
into how different models define actions and how they structure if-then
relationships. That is, for an action model, take the same action possibili-
ties and the same unknown scenarios, then develop two separate if-then
relationship models. Develop and compare the outcome tables for the two
models. Are the outcomes the same? If not, why? One must go behind the
model outcomes and examine the details of the models to understand their
differences. Individuals who are expert in the subject are critical in judging
the models and their value. Docking should involve experts throughout
the process, as discussed above. Docking of multiple modeling approaches
against common benchmark problems using a panel of expert judges has
recently been used to provide considerable insight into individual cognitive
performance models (Gluck and Pew, 2005).
At this time, there is a need to develop benchmark scenario-action situ-
ations that can be used to dock two or more models. This effort will involve
action, scenario, and if-then experts. With these benchmarks, docking
studies can add greatly to the development of action models.
Given the current state of the art, the participation of experts in the
docking process is essential. The next best step in validation is to support
docking studies among experts who develop computation-based models.
Automated machine docking of two or more models is a very high-risk
endeavor at present. At a later stage of understanding, we may be able to
develop a computationally based approach to the docking of models. But
for now, experts and their judgment are mandatory.
Triangulation. Triangulation goes beyond docking and involves examining
the same action domain using an action model, an expert group using a
qualitative approach, and reference to quantitative studies in the domain.
An action model validated using multiple approaches is more likely to help
the decision maker take actions that meet the purpose. However, a large
number of triangulations are often possible. We do not know a priori what
the best triangulation is for a given situation, but it is quite likely that a
good triangulation will be situation dependent.
OCR for page 324
4 BEHAVIORAL MODELING AND SIMULATION
Exploratory testing of robustness. Miller (1998) proposes active nonlinear
tests for complex models to validate the model’s structure and robustness.
In this approach, automatic nonlinear search algorithms probe for extreme
outcomes that could occur within the set of reasonable model perturba-
tions. This multivariate sensitivity technique can find places where a com-
plex model “breaks,” that is, produces results that are outside a range of
reasonable predictions.
In summary, universal rules about what is the appropriate procedure
for validating IOS models are not possible. However, we recommend the
validation of models through a three-part triangulation process, based
on the purpose of the model. Validation should involve (1) participation
by multiple experts who can provide different perspectives on the action
domain, the scenarios, and the if-then rules incorporated in the model;
(2) docking of similar computational models against one another; (3) com-
parison to qualitative and theoretical studies and previous quantitative
results and exploratory testing for a range of outcomes. A good heuristic
would be to begin with the experts as discussed above and move as quickly
as possible to docking studies and exploratory testing.
DATA ISSuES AND CHALLENgES
Data can be used in two different ways in modeling. When models
are developed inductively from data, the quality of the data is extremely
important. In that case the data are broader in scope and limited only in a
very general manner. For example, an anthropologist sees different things
than an engineer in the same situation. For existing models, the data are
prescribed by the model, and the quality of data is extremely important.
Here again, the data yield values for the model parameters and make the
model specific to a given situation and problem. The data requirements are
driven by different modeling needs. For each situation, quality data are
needed and are important to the usefulness of the model.
This means that even the most promising, sophisticated, and elegant
models may be severely limited or hampered by specific data needs and
requirements. Thus, data issues are an essential component for assessing
the ultimate success for model development, validation, and applications.
A number of potential data factors need to be considered in the course of
conceptualizing and developing models. These include but are not limited
to the following.
• Primary/secondary: Data may already exist (secondary) or may
need to be collected (primary). Obviously, models using secondary
source data have some advantages because they require little or no
data collection. However, models using such forms of data may
OCR for page 325
5
COMMON CHALLENGES IN IOS MODELING
be limited by the nature and quality of the data that exist. This
might mean the model will be constrained by the type of data avail-
able, and such constraints may limit the model’s ability to address
important issues and problems. Models using primary sources of
data have more flexibility, given that they can determine exactly
what type of data needs to be collected. However, primary data
collection involves its own set of limitations that are reflected in
the factors described below.
• Observable/nonobservable: Some data are directly observable,
and this may facilitate ease of collection. Phenomena that are not
directly observable may require more extensive efforts to uncover
the necessary information (e.g., face-to-face interviews).
• Distant/close: Some forms of data can be collected at a distance.
This may involve the use of technology, such as cell phones or video
links. However, other types of data require actually being there on
the ground, as for face-to-face contact or interviews with subjects,
respondents, or informants.
• Representative/nonrepresentative: Often model assumptions require
data to be collected or compiled in some specific manner. The best
example of this is the explicit assumptions underlying classical
parametric statistical models that require random samples from
a population. There are other models that simply require units
of analysis to be representative of a given theoretically important
category of some type, and it may be the case that any unit of
analysis fitting the categorical criteria will suffice. An important
consideration is the extent to which units of analysis used in the
model need to be derived by either probabilistic or nonprobabilistic
methods (see Johnson, 1990).
• Passive/active: This is related to some of the factors above in that
some data can be collected casually or on the fly. Such data may still
require being there but may require only documenting or record-
ing naturally occurring events, conversations, or interactions. In
contrast, more direct and active methods of data collection may
be necessary and will involve, for example, actually interviewing
individuals at events or interviewing them about given conversa-
tions or interactions.
• Tacit/explicit: Some forms of data require little interpretation or
reading between the lines. Other types of data are implicit, and
there is a need to make them more explicit. This is particularly
true for some forms of human knowledge that are often tacit and
may require specific types of elicitation interviewing techniques to
extract the requisite information to be used in the model (Johnson
and Weller, 2002).
OCR for page 326
BEHAVIORAL MODELING AND SIMULATION
There are certainly other important factors to be considered in terms
of relating models to various data requirements. However, the factors
described above potentially reflect impediments to the utility and validity of
any proposed model. If, for example, models require data involving forms
that are tacit, active, representative, close, nonobservable, and, of course,
primary, then the data may be costly to obtain and may limit the models’
potential effectiveness given the data constraints. But this does not address
in any way issues of data quality concerning reliability and validity. We can
consider the factors above to reflect elements of how hard data might be
to collect or obtain. Although some of these factors are related to issues of
reliability and validity, they are not necessarily one and the same. Often the
data that are the most difficult to collect (i.e., on-the-ground face-to-face
interviews) are the data that have the most reliability and validity, whereas
data that are the easiest to obtain (i.e., secondary source data) may be the
most problematic. The extent to which one trusts the data will ultimately
determine the extent to which one trusts model outcomes or predictions.
In summary, even though quality data are extremely important, the
operationalization of quality is different for the different demands of the
model. One implication is that we need better quality data. Another impli-
cation is that we need a better understanding of how we can model,
describe, predict, and explain with less than quality data. This further sug-
gests that a better notion is needed of what is meant by quality data for the
various models and needs.
REFERENCES
AFRL/HE. (2002). Proceedings of the Eleventh Conference on Computer Generated Forces
and Behavioral Representation, May, Orlando, FL.
Al-Halimi, R., and Kazman, R. (1998). Temporal indexing through lexical chainin. In C.
Fellbaum (Ed.), WordNet: An electronic lexical database (pp. 333–351). Cambridge,
MA: MIT Press.
Allen, J.G. (2004). Commander’s automated decision support tools. Briefing to Proposers’
Symposium for DARPA’s Integrated Battle Command Program, Dec. 15, Washington,
DC.
Axtell, R., Axelrod, R., Epstein, J.M., and Cohen, M.D. (1996). Aligning simulation models:
A case study and results. Computational and Mathematical Organization Theory, (2),
123–141.
Bachman, J.A., and Harper, K.A. (2007). Toolkit for building hybrid, multi-resolution PMESII
models. (Final report #RI-RS-TR-2007-238.) Cambridge, MA: Charles River Analytics.
Available: http://stinet.dtic.mil/cgi-bin/GetTRDoc?AD=ADA475418&Location=U2&do
c=GetTRDoc.pdf [accessed Feb. 2008].
Barreteau, O. (2003). Our companion modelling approach. Journal of Artificial Societies
and Social Simulation, (2). Available: http://jasss.soc.surrey.ac.uk/6/2/1.html [accessed
April 2008].
OCR for page 327
COMMON CHALLENGES IN IOS MODELING
Beeferman, D. (1998). Lexical discovery with an enriched semantic network. In Proceedings
of the Workshop on Applications of WordNet in Natural Language Processing Systems,
Association for Computational Linguistics/Conference on Computational Linguistics
(ACL/COLING 1998), Montreal, Canada. Available: http://www.lexfn.com/doc/lexfn.
pdf [accessed Feb. 2008].
Box, G.E.P. (1979). Robustness in the strategy of scientific model building. In R.L. Launer
and G.N. Wilkinson (Eds.), Robustness in statistics: Proceedings of a workshop. New
York: Academic Press.
Brinkley, J. (2003, May 14). AFRL/HE instruction #-0. Wright Patterson Air Force Base,
OH: Air Force Research Laboratory/Human Effectiveness Directorate.
Burton, R.M. (2003). Computational laboratories for organization science: Questions, validity
and docking. Computational and Mathematical Organization Theory, (2), 91–108.
Burton, R.M., and Obel, B. (1995). The validity of computational models in organization
science: From model realism to purpose of the model. Computational and Mathematical
Organization Theory, (1), 57–71.
Campbell, G.E., and Bolton, A.E. (2005). HBR validation: Integrating lessons learned from
multiple academic disciplines, applied communities, and the AMBR project. In K.A.
Gluck and R.W. Pew (Eds.), Modeling human behavior with integrated cognitive archi-
tectures: Comparison, evaluation, and validation (pp. 365–395). Mahwah, NJ: Lawrence
Erlbaum Associates.
Carley, K.M., and Svoboda, D. (1996). Modeling organizational adaptation as a simulated
annealing process. Sociological Methods and Research, 5(1), 138–168.
Davis, J.H. (2000). Simulations on the cheap: The Latané approach. In D.R. Ilgen and C.L.
Hulin (Eds.), Computational modeling of behavior in organizations: The third scientific
discipline (pp. 217–220). Washington, DC: American Psychological Association.
Duffy, J. (2006). Agent-based models and human subject experiments. In L. Tesfatsion and
K.L. Judd (Eds.), Handbook of computational economics, volume : Agent-based com-
putational economics. Amsterdam, The Netherlands: Holland/Elsevier.
Epstein, J.M. (2006). Remarks on the foundations of agent-based generative social science. In
L. Tesfatsion and K.L. Judd (Eds.), Handbook of computational economics, volume :
Agent-based computational economics. Amsterdam, The Netherlands: Holland/Elsevier.
Fagiolo, G., Windrum, P., and Moneta, A. (2006). Empirical validation of agent-based models:
A critical survey. Journal of Artificial Societies and Social Simulation, 0(2). Available:
http://jasss.soc.surrey.ac.uk/10/2/8.html [accessed April 2008].
Fiske, A.P. (1992). The four elementary forms of sociality: Framework for a unified theory of
social relations. Psychological Review, (4), 689–723.
Gluck, K.A., and Pew, R.W. (Eds.). (2005). Modeling human behavior with integrated cog-
nitive architectures: Comparison, evaluation, and validation. Mahwah, NJ: Lawrence
Erlbaum Associates.
Haefner, J.W. (2005). Modeling biological systems: Principles and applications. New York:
Springer.
Harper, K.A., Pfautz, J.D., Ling, C., Tenenbaum, S., and Koelle, D. (2007). Human and system
modeling and analysis toolkit (HASMAT). (Final report #R06651.) Cambridge, MA:
Charles River Analytics.
Harris, R.J. (1976). The uncertain connection between verbal theories and research hypotheses
in social psychology. Journal of Experimental Social Psychology, (2), 210–219.
ITT Research Institute. (2001). Modeling and Simulation Information Analysis Center
(MSIAC). Verification, validation, and accreditation (VV&A) automated support tools:
A state of the art report, Part -Overview. Chicago, IL: Author.
Johnson, J.C. (1990). Selecting ethnographic informants. Thousand Oaks, CA: Sage.
OCR for page 328
BEHAVIORAL MODELING AND SIMULATION
Johnson, J.C., and Weller, S. (2002). Elicitation techniques in interviewing. In J. Gubrium
and J. Holstein (Eds.), Handbook of interview research (pp. 491–514). Thousand Oaks,
CA: Sage.
Judd, K.L. (2006). Computationally intensive analyses in economics. In L. Tesfatsion and K.L.
Judd (Eds.), Handbook of computational economics, volume : Agent-based computa-
tional economics. Amsterdam, The Netherlands: Holland/Elsevier.
Kaelbling, L.P., Littman, M.L., and Moore, A.W. (1996). Reinforcement learning: A survey.
Journal of Artificial Intelligence Research, 4, 237–285.
Langton, J.T., and Das, S.K. (2007). Final technical report for contract #FA50-0-C-00.
Air Force Research Laboratory, May 16, Rome Research Site, Rome, NY.
Marks, R.E. (2006). Validation and complexity. Working paper, Austrailian Graduate School
of Management, University of New South Wales, Sydney.
Miller, J.H. (1998). Active nonlinear tests (ANTs) of complex simulation models. Management
Science, 44(6), 820–830.
National Research Council. (1998). Modeling human and organizational behavior: Appli-
cation to military simulations. R.W. Pew and A.S. Mavor (Eds.). Panel on Modeling
Human Behavior and Command Decision Making: Representations for Military Simula-
tions. Commission on Behavioral and Social Sciences and Education. Washington, DC:
National Academy Press.
Popper, S.W., Lempert, R.J., and Bankes, S.C. (2005). Shaping the future. Scientific Ameri-
can Digital, April. Available: http://www.sciamdigital.com/index.cfm?fa=Products.
ViewIssue&ISSUEID_CHAR=8E3EADFC-2B35-221B-6A8455DA45AE8B50 [accessed
Feb. 2008].
Robbins, M., Deckro, R.F., and Wiley, V.D. (2005). Stabilization and reconstruction opera-
tions model (SROM). Presented at the Center for Multisource Information Fusion Fourth
Workshop on Critical Issues in Information Fusion: The Role of Higher Level Informa-
tion Fusion Systems Across the Services, University of Buffalo. Available: http://www.
infofusion.buffalo.edu/ [accessed Feb. 2008].
Rouse, W.B., and Boff, K.R. (2005). Organizational simulation. Hoboken, NJ: John Wiley
& Sons.
Sageman, M. (2004). Understanding terror networks [web page, Foreign Policy Research
Institute]. Available: http://www.fpri.org/enotes/20041101.middleeast.sageman.
understandingterrornetworks.html [accessed August 2007].
Sageman, M. (2006). The psychology of Al Qaeda terrorists: The evolution of the Global
Salafi Jihad. In C.H. Kennedy and E.A. Zillmer (Eds.), Military psychology: Clinical and
operational applications (Chapter 13, pp. 281–294). New York: Guilford Press.
Schelling, T.C. (1971). Dynamic models of segregation. Journal of Mathematical Sociology,
, 143–186.
Shafer, G. (1976). A mathematical theory of evidence. Princeton, NJ: Princeton University
Press.
Sowa, J.F. (1984). Conceptual structures: Information processing in mind and machine.
Reading, MA: Addison-Wesley.
Telvick, M. (Producer). (2005, January). Al Qaeda today: The new face of the global Jihad.
Arlington, VA: PBS Frontline.
Telvick, M. (2007). Al Qaeda today: The new face of the global jihad [web page, PBS Front-
line]. Available: http://www.pbs.org/wgbh/pages/frontline/shows/front/etc/today.html
[accessed August 2007].
U.S. Department of Defense. (1995). Modeling and simulation (M&S) master plan. (DoD
#5000.59-P.) Washington, DC: Office of the Undersecretary of Defense for Acquisition
and Technology.
