|
Items in cart [0] |
Questions? Call 888-624-8373 |
| Copyright © 2010. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 356
11
Recommendations for Military-
Sponsored Modeling Research
T
his report has reviewed the state of the art in individual, organi-
zational, and societal (IOS) modeling and the ability of current
modeling approaches to meet military needs; assessed the common
pitfalls and problems associated with this type of modeling; and pointed
out areas in which additional work is needed. This chapter summarizes the
committee’s recommendations for advancing behavioral modeling capabili-
ties to meet the military’s current and anticipated needs.
There are many challenges in advancing the science of human behav-
ioral modeling. The theory on which to base the models is often fragmented
and incomplete, failing to specify key links that are needed to answer the
questions of interest. Data for testing theories and models (or for deriving
empirically based models) are also sparse and often lacking in detail for
exactly those factors that are critical for the model. Because of the scale of
many behavioral models, it is rarely possible to generate useful data from
controlled laboratory experimentation (as, for example, is often possible for
models of individual cognition and behavior). Furthermore, there are often
no well-defined criteria for success in these modeling efforts and no widely
accepted definitions and methods for validation of IOS models. Finally, the
research and development efforts are being conducted in many different
disciplines. Modelers currently use different types of data, at different levels
of detail, to model different types of behavior in order to answer differ-
ent kinds of questions. Little effort is devoted at present to comparison or
integration of models from different perspectives.
How, then, can this fragmented field best advance? Our recommenda-
tions focus on cross-disciplinary information exchange and the compari-
5
OCR for page 357
5
RECOMMENDATIONS FOR MILITARY-SPONSORED MODELING RESEARCH
son and integration of models, structured around well-defined challenge
problems and common datasets, with independent research thrusts recom-
mended for those issues that are most critical.
Recommendations fall into three broad categories: (1) large-scale, inte-
grated cross-disciplinary research programs, focused around representative
challenge problems and common datasets; (2) research in six independent
areas that will advance the capabilities to address these integrated problems;
and (3) multidisciplinary conferences, workshops, and other information
exchange forums, with attendees to include not only model developers but
also government program managers and military decision makers.
INTEgRATED CROSS-DISCIPLINARy RESEARCH PROgRAMS
We suggest the funding of multiple large-scale, multiyear research pro-
grams that focus on comparing and, if appropriate, integrating models from
different disciplines, different perspectives, and different levels of detail.
This funding would provide incentives for researchers in diverse disciplines
to work together on military-relevant problems. The goal would not be to
pick the best model but rather to create a level playing field on which the
capabilities of different approaches could be compared and the strengths
of each assessed (see Gluck and Pew, 2005, for a description of a similar
research program conducted for individual cognitive models). The ultimate
goal is to move IOS modeling science forward through the process of com-
parison, docking, and integration.
It is essential for all participants in each program to focus on the same
well-defined challenge problem instantiated in a common testbed and to
use a common program dataset. At the heart of each program would be
a representative problem that is critical for military operations, defined in
detail. The five representative problems described in Chapter 2 provide a
possible starting point for choosing the problems to be addressed.
The definition of challenge problems is a difficult but essential step for
the recommended IOS modeling research program, and it should be the first
step in such a program. Initial grants should fund challenge problem devel-
opment, and continuation of the program should be contingent on success
in defining these problems. Operational users must be involved in defin-
ing the challenge problems, and the criteria for modeling success should
be clearly specified as part of problem definition. What type of model
results—discovery, understanding, or forecasting—will be relevant for the
problems being defined? What actions may be taken based on the model
results? Criteria for model usefulness in the challenge problems should be
clearly defined up front.
The research teams for these efforts should be multidisciplinary, and the
program team should also include military users with operational experi-
OCR for page 358
5 BEHAVIORAL MODELING AND SIMULATION
ence in the domain for which the models are to be developed. These users
will be ultimate judges of whether model results are useful (which we argue
is the ultimate criterion for validation; see Chapter 8) and will provide
advice on how the model results can best be presented for immediate com-
prehension and relevance. The use of a common challenge problem and
a common testbed will facilitate the docking of the different models for
purposes of comparison.
These integrated programs will encourage mutual education between
modelers and operational users. Researchers will learn about the military
domain and about user expectations. Users will learn about the scientific
limitations to understanding of the basics of human behavior and what is
feasible to represent in models (and implement in usable simulations) and
will develop an understanding of the level of uncertainty associated with
model forecasts or predictions. Results should be presented at workshops
for program participants and other interested parties and at public confer-
ences, published in the open literature for the research community at large,
and presented “up the chain” to the program managers who rely on these
models for operational, training, and mission rehearsal uses.
INDEPENDENT RESEARCH THRuSTS
In support of the integrated programs we recommend, we have identi-
fied six independent areas in which research is needed. Progress in each of
these areas could increase the ability to develop the integrated modeling
capabilities that are needed to address military problems. In each area, we
suggest the funding of multiple research teams approaching the work from
multiple perspectives, with periodic workshops for researchers to exchange
results. We also suggest that operational users as well as government pro-
gram managers participate in these workshops to draw on their areas of
expertise and to gain better insight as to model capabilities and limita-
tions. The funding structure of the programs should support and enable
the participation of individual researchers or smaller laboratories in both
academic institutions and industry and not be limited to large institutions,
as is often the case in collaborative projects supported by the Department
of Defense (DoD).
Thrust 1: Theory Development
Models should be conceptually correct and grounded in the underlying
fundamentals of what is known about individual human and group social
behavior. However, current theory in this area does not answer all of the
questions needed to structure models that address relevant issues. Basic
research is needed for theory development, especially for the low-level social
OCR for page 359
5
RECOMMENDATIONS FOR MILITARY-SPONSORED MODELING RESEARCH
behaviors (e.g., choosing friends) that are the building blocks for larger scale
social behavioral patterns (e.g., joining a terrorist group). Since affective
states and traits represent a key component of individual motivation and play
a critical role in interpersonal behavior and group and organizational deci-
sion making, basic research in emotion and emotion-cognition interactions
should be emphasized. This theory development work must involve multiple
disciplines and perspectives with periodic workshops to exchange results.
Theory development challenge problems should be defined to guide
the work, but these can be nonmilitary and need not involve the level of
military detail necessary for the integrated problems discussed above. A
series of workshops should be conducted with researchers to identify key
theory gaps. We recommend working backward from a set of operational
problems (as defined for the integrated programs) to identify areas in which
lack of theory is impeding modeling progress. These theory gaps can be
used to define theory challenge problems.
Academic institutions are key players for theory development, but
they need information, incentives, and funding to address these theoretical
issues. There is a need to educate academic researchers in military domains,
establish conferences and journals in which their results can be presented,
provide postdoctoral support, and provide funding that allows researchers
to spend time learning about military domains in depth. Funding for gradu-
ate students is a key part of this thrust as a cost-effective way to bring about
shared understanding and progress.
Thrust 2: uncertainty, Dynamic Adaptability, and Rational Behavior
Models must deal with the inherent uncertainty (nondeterminism) and
the dynamic adaptation (nonstationarity) that characterizes human behav-
ior. Models must also be capable of modeling both rational and nonrational
behavior.
Basic research is needed in each of these areas. Issues include
• How should models capture the “uncertainty-in-the-small” associ-
ated with individuals and small groups? How can model structures
and parameters capture this variability, and how much of this vari-
ability must be included for the purposes of the model?
• How should models capture the “uncertainty-in-the-large” associ-
ated with populations and variations in population distributions?
For example, to what extent should models be based on mean val-
ues versus capturing effects from the tails of a distribution? How
much variability must be included for the purposes of the model?
• How can models capture adaptation and learning over time and
in response to actions by others? For example, models of cultural
OCR for page 360
0 BEHAVIORAL MODELING AND SIMULATION
groups often assume that cultural identity is static and unitary. In
fact, people have multiple overlapping identities and allegiances
that vary in their influences over behavior. How can these be cap-
tured in a model, and how can one estimate the effects of actions
and events on the primacy of these multiple allegiances as they
affect decisions and actions?
• What are the factors that contribute to rational, adaptive behavior
and what factors induce behavior that appears nonrational? His-
torically, emotions and affective factors have often been adduced to
explain irrationality, but recent research in psychology and neuro-
science has demonstrated that emotions also play a critical role in
rational, adaptive behavior. Likewise, behaviors viewed as purely
cognitive—including habit, bounded rationality, the range of beliefs
unfamiliar to the observer, and ignorance, as well as behaviors with
strong cognitive and affective components, such as fanaticism—can
lead to what appears to be irrational behavior. Models of both
rational and irrational behavior must therefore capture all the key
factors—cognitive, affective, cultural, and contextual—that motivate
and shape behavior of specific individuals in specific situations.
Better techniques are needed for understanding the implications of
diversity and variability for model-based sensitivity analysis. Combinatorial
explosion of possible combinations of parameters is a challenge, and better
automated technology is needed to put the model through its paces to
explore the parameter space effectively and produce robust results.
Thrust 3: Data Collection Methods
The difficulty of obtaining data is an ongoing challenge for IOS model-
ing. Research is needed to develop better data collection processes through
field studies, experiments, and potentially by using massively multiplayer
online games (MMOGs).
Although a variety of ethnographic data collection techniques are cur-
rently in use, they need to be better tailored to the needs of IOS models. For
field data collection, it is necessary to bring modelers and data collectors
together to develop data ontologies, joint specifications, and data collection
methodologies and tools that are specifically tuned to IOS models.
MMOGs are an untapped resource for collecting social and behavioral
data on a large scale. We recommend the creation of an MMOG facility
that could serve as a testbed for exploratory research and model testing and
the funding of basic research to determine if MMOGs can be used to test,
verify, and validate IOS models. We recommend that funding be put into
developing the science of MMOGs rather than in developing additional
OCR for page 361
RECOMMENDATIONS FOR MILITARY-SPONSORED MODELING RESEARCH
artificial worlds. The research agenda for this facility should be developed
through workshops that convene both IOS modeling scientists and game
experts. Note that funding MMOGs is a risky endeavor, with no guarantee
that games that are useful for research purposes will find the widespread
interest necessary for extensive data generation, but we think that the
potential benefits outweigh the risks.
Given the critical role of emotions and affective personality factors
in organizational decision making and behavior, it is also important to
enhance the current methods for collecting affective data. Emotions and
moods are notoriously difficult to assess accurately, particularly in natural-
istic and field settings. Yet recent progress has been made in using multi-
modal approaches to affect assessment, including physiological monitoring
and indirect assessment of these transient states via diagnostic tasks and
performance tracking. We recommend that funding be allocated to the con-
tinued refinement of these methods and to the development of standardized
assessment instruments, particularly in naturalistic settings.
Thrust 4: Federated Models
It is a fundamental conclusion of the committee that no single model-
ing approach can provide all the capabilities needed by DoD. We recom-
mend a federated approach in which modeling components are created
to be interoperable across levels of aggregation and detail. For example,
a federated model might include a detailed representation of a few key
individuals, linked to group-level models of different cultural groups and
terrorist organizations, linked to geographic sector–level models of the level
of unrest in a city. This approach is flexible and extensible, allowing the
addition or subtraction of models at different levels of detail as needed for
the problem to be addressed.
Combining model components to create federated models in the
sense being recommended is not simply a matter of specifying and using
interface-level syntactic compatibility protocols. It requires deep semantic
interoperability (i.e., theoretical consistency). To create semantic inter-
operability, it is necessary to recognize that the links among components
are themselves elements of the model. Components created at different
levels of detail and for different purposes do not simply snap together to
produce meaningful results.
Assuming that the interface protocol issues will be solved by others
(e.g., enterprise database developers), research is needed to answer the
following questions:
• What is the best way to ensure that the models being federated
embrace compatible assumptions regarding concept abstractions,
OCR for page 362
BEHAVIORAL MODELING AND SIMULATION
entity resolution, time scale resolution (tempo), uncertainty, adapt-
ability, docking standards, input-output semantics, etc.?
• How should the components of the federated model be encapsu-
lated, and which elements must be exposed to other components?
• How should specific classes of models be linked (e.g., cognitive
models to social network models)?
• How can developers ensure dynamic extensibility?
These issues are not unique to IOS modeling. In addressing them, IOS
modelers should maintain awareness of research and development in model
federation in the larger modeling and simulation community.
Thrust 5: validation and usefulness
Current verification, validation, and accreditation (VV&A) concepts
and practices were developed for the physical sciences, and we argue that
different approaches are needed for IOS models. Specifically, we recom-
mend the use of a “validation for action” approach that assesses the use-
fulness of a model for the specific purposes for which it was developed.
Although promising work has been done in testing IOS models through
triangulation among multiple types of expertise and multiple data sources,
and some work has been done in docking different models for comparison,
these approaches are not widespread. We recommend organizing a national
workshop to agree on appropriate processes for VV&A of IOS models
and to outline a roadmap for developing improved VV&A processes and
standards. On the basis of the results of this workshop, we recommend
that a DoD-wide authority develop and disseminate VV&A processes and
standards for IOS models. These standards should be developed de novo,
not as an adjunct to conventional VV&A standards.
Basing model validation on the usefulness of the model for specific
problems requires that model purposes be clearly stated by model users and
clearly understood by model developers. This is an area in which mutual
education is needed. We suggest that, as part of developing a VV&A stan-
dard for IOS models, clear guidelines be developed for specifying model
purpose.
Thrust 6: Tools and Infrastructure for Model Building
It is important to reduce the barrier to entry for developing models,
modeling tools, frameworks, and testbeds. Scientists should be able to build
and validate models without the large overhead currently associated with
many DoD modeling and simulation investments. It should be possible to
tailor existing models easily for specific purposes.
OCR for page 363
RECOMMENDATIONS FOR MILITARY-SPONSORED MODELING RESEARCH
Sharing of IOS modeling knowledge across disciplines, as facilitated
by the conferences and workshops recommended below, will support this
goal. Work is also needed to develop an infrastructure for IOS modelers,
including a national network of possible collaborators, common databases
for model development and testing, and frameworks and toolkits for rapid
model development. There is also a need for web-based repositories of
information about existing models and, later, model components.
To facilitate the development and use of shared ontologies and model
components, funding must also be allocated to the refinement of existing
markup and modeling languages, as well as the development of new lan-
guages for particular domains or tasks.
The limited data that exist for IOS models are often not accessible to
model developers. We recommend the funding of national web-accessible
data repositories that are open to researchers who seek to inform and test
models. For militarily relevant domains in which some data are classified,
we recommend an investment in automated tools to sanitize potentially
sensitive military data.
Often, the IOS models themselves are not readily accessible or even
known to researchers or practitioners. Researchers are often unaware of
efforts under way in DoD that are not reported on in conventional confer-
ences and journals, and military developers are likewise unaware of progress
being made in the research community. Or if they are, the typical user can
face great difficulty in assessing the applicability of one approach or model
over another, given their particular problem at hand. The occasional studies
that attempt to survey the community and categorize development efforts
and associated models, such as this one (see, e.g., Table 2-1 and Table 8-3)
and its predecessor study (National Research Council, 1998), take small
steps in this direction, but they are not meant to be exhaustive surveys and
are only snapshots in time, become stale rather quickly, and fail to offer
easy electronic access directly to the rich and evolving world of IOS models
and associated simulations.
We therefore recommend the development and maintenance of an
online web-based catalog of general approaches, models, simulations, and
tools. The notion is to develop something along the lines of the Defense
Modeling and Simulation Office’s (DMSO’s) Modeling and Simulation
Resource Repository (MSRR) at http://www.msrr.dmso.mil/, maintained
by the Modeling and Simulation Information Analysis Center, or the clear-
inghouse at Carnegie Mellon’s CASOS site (see http://www.casos.cs.cmu.
edu). But to be effective, the envisioned site needs careful consideration in
terms of the following:
• Organization: The model ontology and site structure need to be
carefully thought out, both from the researcher’s perspective (e.g.,
OCR for page 364
4 BEHAVIORAL MODELING AND SIMULATION
foundational concepts underlying the particular model in the repos-
itory) and from the user’s perspective (domain of application, limi-
tations, simulation requirements, etc.).
• Content: Considerations need to be given to what is maintained on
the site, ranging from simple descriptive abstracts to full-fledged
downloadable simulations and “read me first” instructions.
• Currency: Once set up, the maintainers must devote effort to con-
stantly updating the site, by tracking changes to existing models,
adding new models that arise on the scene, and, certainly of equal
importance, removing defunct models, or at least moving them to
the archival section of the site to support historical surveys and
the like. Failure to maintain currency will be the death knell of the
repository, as it is with most websites today. One approach that
should be considered is a Wikipedia-based model.
• Usability: The site design needs to ensure ease of use for all autho-
rized visitors, including contributors, users, and occasional viewers.
Procedures need to be in place to vet content modifications or addi-
tions, to support ease of navigation and internally searching for
what the user is seeking, and to keep the site fresh and attractive
to the larger community.
It is clear that this cannot be a one-time effort like DMSO’s MSRR, nor
an unfunded academic effort like Carnegie Mellon’s CASOS site. It needs
significant startup funding and continued support over its lifetime.
MuLTIDISCIPLINARy CONFERENCES AND WORKSHOPS
A number of the issues and problems identified by the panel were
the results of the failure of different disciplines to exchange information,
or they resulted from misunderstandings among government funders of
model development efforts, military users of models, and model developers.
Because of the diversity of this group, there is no natural forum for them
to exchange information, as there would be in conferences and journals for
members of the same academic discipline or professional group. We there-
fore recommend the organization of special-purpose workshops around the
integrated research programs recommended above as well as workshops for
the independent research thrusts described above.
IOS modelers who are interested in working on military-relevant prob-
lems need to be educated on:
• The nature of the military decisions for which models are relevant
and of the operational situations in which the decisions must be
made.
OCR for page 365
5
RECOMMENDATIONS FOR MILITARY-SPONSORED MODELING RESEARCH
• Desired model functionality.
• The most useful form(s) for presenting model results.
• The value of work performed by others outside their discipline.
• Feasible and appropriate VV&A approaches for IOS models.
Operational users and managers need to be educated on:
• The value of multidisciplinary approaches and the need for review
of models from multiple perspectives.
• The inherent uncertainty associated with IOS model predictions.
• The value of models for sensitivity and trade-off analysis (versus
the one right answer).
• The design of virtual experiments to assess results over a range of
conditions.
• Reasonable definitions of validation for IOS models, feasible
approaches for VV&A testing, and why these approaches differ
from those used for physics-based models.
The recommended workshops should involve model developers, opera-
tional military users of the models, and government personnel who make
funding decisions regarding model development. Issues to be discussed
include methods for clearly specifying model purpose, criteria for judging
the usefulness of models (i.e., what does it mean to validate a model), rea-
sonable expectations for the certainty of model predictions, and methods
for most clearly communicating model results.
ROADMAP FOR RECOMMENDED RESEARCH
The committee’s recommendations are based on the concept of use-
driven research. As defined by Stokes (1997), use-driven research combines
elements from both basic and applied research. Like applied research,
use-driven research seeks to solve a practical problem—in this case, the
development of IOS models that can serve military purposes. But, like
basic research, it also asks “why” in a fundamental way—Why do some
methods work and others not work? What are the principles that underlie
success or failure?
Figure 11-1 illustrates the major elements of a use-driven research
program for IOS modeling. The process starts with challenge problem defi-
nition, which includes a clear specification of the use to which a model is
to be put. This specification should be based on the needs of the model
users, expressed in terms that are meaningful to the IOS modeling com-
munity. The challenge problem definition step is critical, and the funding
of the remainder of the program should be contingent on its success. The
OCR for page 366
BEHAVIORAL MODELING AND SIMULATION
Can additional
needs be met?
Problem
Definition
Theory Data
Development Collection Methods
Model What additional data
Is new theory needed?
Development are needed?
Federation Infrastructure
Standards and Tools
Model
Has required interoperability What new tool
been achieved? capabilities are needed?
Validation
Does the model meet
the user’s needs?
FIguRE 11-1 Elements of use-driven research for IOS modeling.
11-1.eps
purpose of the model drives the theory to be applied, the data to be used,
and the model development. Model development is made easier by model-
ing tools and infrastructure and relies on federation standards to ensure
the interoperability of model components. Once the model is developed
it is validated by asking the question: Is the model useful for its intended
purpose?
As shown in Figure 11-1, the problem specification and model develop-
ment process is cyclical. Based on the answers to the question “Is the model
useful?” new models may need to be developed, new theory and new data
(and new types of data) may be needed, and new interoperability standards,
tools, and infrastructure may be required. Depending on the results, the
problem itself may need to be redefined, clarified, or expanded.
Figure 11-1 lays out the areas in which research and development are
needed for IOS modeling and shows how they are interdependent. Fig-
ure 11-2 organizes the suggested research areas into a roadmap that shows
lines of activity and the interrelationships among them, repeating yearly in
a cyclical fashion to advance the state of the art in meeting military IOS
modeling needs.
As in all use-driven research, the recommended activities start with
a clear definition of the purposes to which IOS models are to be put. We
recommend that the initial activity for the program (the first six months) be
spent on developing a clear definition for selected representative challenge
problems (the problems listed at the end of Chapter 2 can provide a starting
point) in close collaboration with operational military users. Concurrent
with problem definition, the first six months should be spent in developing
OCR for page 367
Year 1 Year 2 Year 3 Year 4
Problem Set 1 Problem Set 2 Problem Set 3 Problem Set 4
DEFINE CHALLENGE
PROBLEMS
Develop Datasets for
Challenge Problems
Select
multiple
MODEL approaches
DEVELOPMENT
Docking and Docking and Docking and
comparison comparison comparison
VALIDATION
Theory Development
11-2.eps
Research on Data
Collection methods
Development of
Federation Standards
Development of
Infrastructure and Tools
Workshops
Workshops Workshops
CONFERENCES
landscape so that type can be a legible size
Input for new Challenge Problems
FIguRE 11-2 Roadmap for an IOS modeling research program.
NOTE: Only the first four years are shown.
OCR for page 368
BEHAVIORAL MODELING AND SIMULATION
datasets for these challenge problems. The challenge problems will provide
common themes that tie together the diverse research and model develop-
ment efforts.
There is currently no single approach that is clearly dominant for IOS
modeling. Our recommendation is to select and fund a number of modeling
teams that take different approaches for each challenge problem. In addi-
tion to the modeling teams, we recommend a series of specialized research
thrusts focused on theory development, data collection methods, federation
standards, and the development of infrastructure and tools. These thrusts
will be aware of the challenge problems and will use the problems to focus
their research, but their charter is broader and covers the entire field of
IOS modeling.
The modeling teams and the research thrusts will come together in a
conference at month 6, to learn about the challenge problems and the data-
sets associated with each problem. Conferences that involve the entire pro-
gram will be scheduled yearly, with workshops for the individual research
thrusts at the intervening 6-month intervals. The yearly conferences will
also provide the forum for the presentation of new challenge problems,
based on the results obtained in the prior year.
At the end of year 1, the models that have been developed for the
challenge problems will be presented and discussed at a validation work-
shop, and docking and comparison activities will follow during the next
6 months, with results to be reported to the whole program at the yearly
conference. These validation workshops should involve representative
model users for each challenge problem. These users will assess the extent
to which model results are useful for their intended purpose, as defined
in the challenge problems. This process will repeat in subsequent years.
As shown in Figure 11-1, the intention is that the results of the validation
effort will inform all of the research thrusts as well as model development
for the next cycle.
Although not shown in the timeline, it is assumed that a concurrent
effort will be focused on the development and maintenance of an online
web-based catalog of general approaches, models, simulations, and tools, as
described earlier. This will serve not only as a repository of current theories
and models, but also as a common record of the results of the execution
of the roadmap.
The roadmap structure proposed in Figure 11-2 is intended to provide
the field of IOS modeling with the common ground and forums for sharing
information that will allow it to advance in a systematic way. Develop-
ment and testing of models against a common set of challenge problems
will avoid the current proliferation of specialized models for specialized
purposes with no common framework for comparison and validation and
therefore no foundation for scientific progress. Figure 11-2 shows the
OCR for page 369
RECOMMENDATIONS FOR MILITARY-SPONSORED MODELING RESEARCH
research cycle repeating over a four-year period, but we recommend that
the program continue well beyond four years, with each year assessing the
progress that has been made and increasing the complexity of the challenge
problems based on the increasing capability of the modeling technology.
New participants should be added to the funded programs and conferences
each year, as new approaches and tools are developed and tested.
REFERENCES
Gluck, K.A., and Pew, R.W. (Eds.). (2005). Modeling human behavior with integrated cog-
nitive architectures: Comparison, evaluation, and validation. Mahwah, NJ: Lawrence
Erlbaum Associates.
National Research Council. (1998). Modeling human and organizational behavior: Appli-
cation to military simulations. R.W. Pew and A.S. Mavor (Eds.). Panel on Modeling
Human Behavior and Command Decision Making: Representations for Military Simula-
tions. Commission on Behavioral and Social Sciences and Education. Washington, DC:
National Academy Press.
Stokes, D.E. (1997). Pasteur’s quadrant: Basic science and technological innovation. Washing-
ton, DC: Brookings Institution Press.
OCR for page 370
