Human-System Integration in the System Development Process A New Look (2007) / Chapter Skim
Currently Skimming:
Part II Human-System Integration Methods in System Development, 6 Defining Opportunties and Context of Use
Part II Human-System Integration Methods in System Development, 6 Defining Opportunties and Context of Use
Pages 127-188
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 127... ...
Part II Human-System Integration Methods in System Development
|
From page 129... ...
• The range of shared representations that can be generated as output of each of four HSI activities. These representations provide shared views that can be inspected and evaluated by the system stakeholders, including domain practitioners, who will be the target users of the system.
|
From page 130... ...
representations • Role networks • Network representations • Fault Tree Analyses (FTA) and other • Cultural profiles and models produced technique variations • Concept maps • Futures tables • Lists of usability problems in the form of: • Graphs of workload as a function of • Opportunity maps written reports, presentations, or videos time or task progress • Profiles • Time and accuracy of user's performance • PERT charts • Scenarios • Gantt charts • Mockups and prototypes • Stories, storyboards, narratives, • Task description tables and use cases • Hierarchical network representations • Theatrical methods • Timeline representations • Workshop outcomes • Formal requirements and specifications • Task flow diagrams • Vision statements • Executable simulations and models • Descriptions of current or future • Concept maps end-user work environments • Goal-decomposition representations • Physiological test results • Stories, storyboard, narratives, • Subjective ratings and use cases • Checklists and rating scales • Flow models • Human Digital Modeling • Sequence diagrams • SA test results • Physical models • Alertness models • The products of multidimensional scaling and Pathfinder network scaling FIGURE II-1 HSI activities, participants, methods, and shared representations.
|
From page 131... ...
Everyone agrees that function allocation is, at the base level, a creative aspect of the overall design process. Everyone agrees that it requires hypothesis generation, evaluation and iteration.
|
From page 132... ...
• Do the users have situation awareness? • Are the users stressed?
|
From page 133... ...
specific behaviors • Cognitive measures of knowledge sharing and team situation awareness Table II-1 contains some examples of the kinds of measures that are likely to be of interest. Since each situation is different, the analyst must consider the context of use under which measurement or prediction is to be undertaken, the goals of the measurement, the characteristics of the users who will be tested or about whom performance will be inferred, and the level of detail of analysis required in order to select specific measures to be used.
|
From page 135... ...
They include methods that focus on the capabilities, tasks, and activities of users (e.g., task analysis methods that characterize the tasks to be performed and their sequential flow, cognitive task analysis methods that define the knowledge and mental strategies that underlie task performance) , as well as methods that examine the broader physical, social, and organizational context in which individu
|
From page 136... ...
. The chapter covers a variety of complementary approaches, ranging from participatory analysis methods, which include domain practitioners as active partners, to event data analysis methods, which promise the potential of more automated and less obtrusive ways of uncovering user activities and needs.
|
From page 137... ...
DEFINING OPPORTUNITIES AND CONTEXT OF USE HSI Activities, Participants, Methods, and Shared Representations HSI Activities Defining Requirements Evaluation Defining Opportunities and Design and Context of Use Who's involved? Domain practitioners Design experts and other stakeholders Representative - Organizational and Environmental set of methods Context Analysis - Field Observations and Ethnography - Task Analysis - Cognitive Task Analysis - Participatory Analysis - Contextual Inquiry - Event Data Analysis Sample shared • Organization charts representations • I/O system diagrams produced • Role networks • Cultural profiles and models • Futures tables • Opportunity maps • Profiles • Scenarios • Mockups and prototypes • Task description tables • Hierarchical network representations • Timeline representations • Task flow diagrams • Executable simulations and models • Concept maps • Goal-decomposition representations • Stories, storyboard, narratives, and use cases • Flow models • Sequence diagrams • Physical models • The products of multidimensional scaling and Pathfinder network scaling FIGURE 6-1 Representative set of methods and sample shared representations for defining opportunities and context of use.
|
From page 138... ...
. Context of use analysis methods can play an important role in mitigating the risks of these types of design failures by promoting a more complete understanding of needs and design challenges as part of the incremental commitment model.
|
From page 139... ...
As a consequence, crew operating philosophy and training were completely redefined so as to capitalize on the crew members' freedup mental resources (they could now provide an independent and diverse check on plant state) , resulting in improved shared situation awareness of the entire team.
|
From page 140... ...
As a result, any optimization scheme may turn out to be short-lived, because the conditions that were considered in crafting the optimization may themselves change. Here, we focus on a brief introduction to analyzing the enterprise and the environment as relevant contexts for system development, using the sociotechnical systems perspective and focusing on four general methods with associated sources of data and shared representations (Table 6-1)
|
From page 141... ...
The shared representations presented have been selected for their potential appreciation by a wide and diverse audience and are the shared representations that map to a sociotechnical systems approach to organizational context. Organization Charts: A widely known but often incorrectly or underused representation is the organizational chart, which depicts lines of authority and communication in an organization.
|
From page 142... ...
Role Network: A role network, based on a role analysis, is also a useful shared representation. A job within an organization is defined by the formal job description that is a contract or agreement between the individual and the organization.
|
From page 143... ...
Gold Project Scientist Payload Manager V,I,G,A,L V,G,A O V,I,G,A,L R.W. Farquhar Instrumentation Mission O Lead Engineers Congress Manager Andy Santo V,G,A Lead System V,M,T O Engineer Mission NASA V,G,A Operation Lead Outside Engineers Vendors APL Sub-system Lead Engineers FIGURE 6-3 Role network for National Aeronautics and Space Administration (NASA's)
|
From page 144... ...
, a project managed by the Applied Physics Lab for NASA. Cultural Profile: Shared representations related to culture include organizational culture and climate assessment tools.
|
From page 145... ...
that have standardized practices and a stable external environment. Some advantages of the functional organizational design include professional identity, professional development, and the minimization of redundancy.
|
From page 146... ...
FIGURE 6-4 Example of organizational design (used with permission)
|
From page 147... ...
The relationships in the role network are then evaluated. Internal and external customers of roles can be interviewed or surveyed for their perceptions of role effectiveness as well.
|
From page 148... ...
A work system design change supported by valid changes in policy and leadership and training is likely to be the best approach to achieving desired culture change and effective performance of a new or improved system. Stakeholder Analysis In addition to an enterprise context for systems, systems have environments that surround them and the enterprise of which they are a part.
|
From page 149... ...
Thus, analyzing these contexts is most useful at the beginning stages of the system design process. Analysis, design, and implementation that fail to include the perspectives of stakeholders can often lead to systems that fail in functional, organizational, or economic terms.
|
From page 150... ...
as well as CES staff. FIELD OBSERVATIONS AND ETHNOGRAPHY Overview Ethnographic Principles Ethnographic approaches can provide excellent in-depth reports on conditions of use in specific case studies or at specific locations or sites.
|
From page 151... ...
. Note that this stance is almost the exact opposite of participatory analysis and design (see below)
|
From page 152... ...
, which makes strategic use of the diversity among sites in an ongoing, hypothesis-testing manner. In grounded theory, the analyst begins with a general research question, rather than a specific theory and hypothesis.
|
From page 153... ...
Within the bounds of the chosen sampling approach (e.g., quota, purpose, convenience, snowball, or grounded theory) , ethnographic practices tend to take a small number of forms.
|
From page 154... ...
Several commercial tools have become de facto standards in this area; see recommendations from the American Anthropology Association for details.1 Contributions to the System Design Process There are major claims of the usefulness of ethnographic work to systems engineering and design (e.g., Hutchins, 1995)
|
From page 155... ...
Shared Representations Several types of intermediary products or shared representations may be used between ethnographers and their clients (e.g., systems engineers or developers) : • Experience models are documents or visualizations that help software professionals to understand patterns of human behavior, thought, and communication.
|
From page 156... ...
collaborative analysis of qualitative data (for related approaches to collaborative analysis of field data, see Holtzblatt, 2003; Holtzblatt et al., 2004)
|
From page 157... ...
In this section we focus on task analysis methods that are particularly suited for defining tasks and the behavioral sequence of activities necessary to accomplish the task. The next section, on cognitive task analysis, describes specific methods for uncovering and representing the knowledge and mental activities that underlie more cognitive performance (e.g., situation assessment, planning, decision making)
|
From page 158... ...
• Tabletop analyses of expected interaction given design descriptions. • Interviews with domain practitioners.
|
From page 159... ...
Examples of a task flow diagram and a task description table are provided in the risk analysis section of Chapter 8. The outputs of task analyses Withdraw money from ATM Specify account Specify amount to Take bank card Insert bank card Take money number be withdrawn back Type in digits corresponding to Press enter key Read display Confirm amount Press yes key amount FIGURE 6-5 An example of graphic representation of a portion of a hierarchical task analysis of the "withdraw money from ATM" task.
|
From page 160... ...
. Uses of Methods Task analysis methods are widely used to provide a step-by-step description of the activity under analysis.
|
From page 161... ...
COGNITIVE TASK ANALYSIS Overview Traditional task analysis approaches break tasks down into a series of external, observable behaviors. For tasks that involve few decision-making requirements (e.g., assembly line jobs, interacting with a consumer product that are expected to be easy to operate, such as ATM machines)
|
From page 162... ...
provide an excellent how-to handbook with detailed practical guidance on how to perform a cognitive task analysis. Comprehensive catalogues of CTA methods and additional guidance can also be found on two active web sites: http://www.ctaresource.com maintained by Aptima, Inc., and http://www.
|
From page 163... ...
Figure 6-6 is an example of a concept map that depicts the knowledge of cold fronts in Gulf Coast weather of an expert in meteorology. Cognitive task analysis techniques have been developed to explore how changes in technology and training are likely to impact practitioner skills, strategies, and performance vulnerabilities.
|
From page 164... ...
that used a future incident technique to explore the potential impact of contemplated future air traffic management architectures on the cognitive demands placed on domain practitioners. Controllers, pilots, and dispatchers were presented with a series of future incidents to jointly resolve.
|
From page 165... ...
In turn, cognitive task analyses will tend to specify the physical activities required to access and integrate the information needed for cognitive tasks (e.g., if a cognitive task requires integrating information across multiple displays, communicating with others, or traversing physically disparate locations, these will be captured in a cognitive task analysis)
|
From page 166... ...
It can also take the form of structured tables that catalogue the decision points that arise, why they are difficult, the knowledge and skill that enable experts to handle the situation, and the typical errors that less experienced personnel make. Other shared representations that are produced from a cognitive task analysis are concept maps that provide graphic depictions of the structure and content of knowledge of domain practitioners (both experts and less experienced individuals)
|
From page 167... ...
Contributions to System Design Phases Cognitive task analyses are particularly useful as part of early context of use analyses in support of understanding needs and envisioning opportunities. They can be used to help focus further analyses and design efforts on aspects of performance that are most cognitively challenging and error prone and identify leverage points at which the introduction of new technology can have the most positive impact on performance.
|
From page 168... ...
Finally, they require analysts who have a background in behavioral sciences and training in CTA methods to conduct and analyze the cognitive task analyses. While CTA methods can be resource intensive, the CTA tool kit contains a variety of methods that can be tailored to the needs and constraints of the particular application.
|
From page 169... ...
Participatory analysis provides methods for understanding the current state of work, including technologies and work practices. Participatory design provides methods for improving the state of work in various ways,
|
From page 170... ...
. Methods in participatory analysis assist users to describe their work, their work organizations, their work technologies, and their work contexts.
|
From page 171... ...
. These workshops combine methods from market research, ethnography, and participatory design.
|
From page 172... ...
Scenarios in Participatory Analysis Some participatory analysis methods focus on storytelling to create a level playing field, to put users and technologists on a common footing; in more formal terms, these techniques have been described and analyzed as scenario-based methods (Carroll, 1995, 2000; Carroll, Rosson, and Carroll, 2002, 2003)
|
From page 173... ...
Table 6-3 provides a synopsis of how each participatory method may be of use in the system development process. Shared Representations Each category of participatory analysis method produces its own characteristic shared representation.
|
From page 174... ...
The shared representations and issues related to ethnography are also relevant here. Strengths, Limitations, and Gaps Participatory analysis methods are often very fast to execute.
|
From page 175... ...
. Contextual inquiry is a set of methods for informing analysis with the users' context and working practices, and for interpreting those analyses into engineering requirements.
|
From page 176... ...
Shared Representations In the course of these three activities, the contextual inquiry approach creates five analytic models: 1. Flow model -- a high-level summary of work processes and communication patterns.
|
From page 177... ...
The trade-off is obvious: the participatory methods provide less formal, more open outcomes that emphasize the users' unique knowledge, but that may require further interpretation and analysis by systems professionals; the contextual inquiry methods produce more formal, more closed outcomes that are more ready for adoption by systems professionals. EVENT DATA ANALYSIS Overview Human performance in the context of complex systems is often measured in terms of outcome (number of errors, amount of time to complete task)
|
From page 178... ...
of users' behavior in context, as well as quantitative indices associated with that description. The richness of the event data affords a deeper look at the behavior behind effective or ineffective human performance and thus is valuable in reducing uncertainty and guiding human-system integration.
|
From page 179... ...
multidimensional scaling and Pathfinder network scaling, generate shared representations. For example, in Figure 6-7 the Pathfinder data-reduction procedure (Schvaneveldt, Durso, and Dearholt, 1989)
|
From page 180... ...
How can the space be used more effectively or efficiently? Uses of Methods In the context of cognitive work analysis, event data analysis can be especially useful for strategies analysis and social, organization, and cooperation analysis.
|
From page 181... ...
It also can be used to describe behavior in the context of a new design, thereby pitting old design against new, as might be helpful in the development and operation phases. Event data analysis encompasses a family of methods differing on a variety of dimensions.
|
From page 182... ...
Other events take a broader look at the collaboration among multiple users and nonhuman agents, which also occurs in the context of a larger system. Examples of individually oriented event data analysis include verbal protocol analysis (e.g., Ericsson and Simon, 1984)
|
From page 183... ...
For some applications it may be sufficient to generate a shared representation, and in others it may be more informative to carry the analysis through to assessment and diagnosis. Each of these steps depends on the intended use or application of event data analysis and is discussed in turn.
|
From page 184... ...
. Furthermore, event data have also been used to derive social networks, although certainly not the typical approach to social network analysis, which has relied more on human judgments regarding relationships (e.g., Tyler, Wilkinson, and Huberman, 2005)
|
From page 185... ...
Assessment and diagnosis move event data analysis from its purely descriptive status to serve an additional evaluative function. Contributions to System Design Phases Table 6-5 describes how EDA can be applied across the life-cycle phases.
|
From page 186... ...
Event data can be collected prior to a prototype if expert judgments are used in lieu of events. Application in the early stages will reduce risk by providing information about typical user behavior or normative patterns of collaboration.
|
From page 187... ...
training or education levels may be underestimated if the more demanding work-around activities are not recorded. Thus, event data analysis is one tool in the tool box and should be used in a balanced way with other, more qualitative tools.
|
From page 188... ...
88 HUMAN-SYSTEM INTEGRATION IN SYSTEM DEVELOPMENT from whom are critical and nontrivial. Without experienced decision makers, the analyst is in danger of experiencing the "garbage in–garbage out" dilemma and, depending on familiarity with the domain, may never recognize the limits of the data.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.