|
Items in cart [0] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 71
l
-
4
Assessment of the Aviation Safety Program
BACKGROUND
Program Information
The Aviation Safety Program (AvSP) is one of
three programs in the Aeronautics Technology Pro-
grams of NASA's Aerospace Technology Enterprise.
AvSP was created in 2000 as an outcome of a formal
process initiated by NASA to develop a research in-
vestment strategy in the area of aviation safety.
The goal of the AvSP is to protect air travelers and
the public. Its research and development strategy is to
increase safety by three primary methods:
Aviation system modeling. Identify and correct
problems using aviation system-level data,
Accident prevention. Identify interventions and
develop technologies to eliminate recurring
types of accidents, and
· Accident mtigation. Reduce injury and decrease
fatalities in survivable accidents.
These methods are applied in the three major re-
search and development components:
iG. Finelli, NASA Langley, "NASA Aviation Safety Program
Overview," presentation to panel, February 2003.
71
Vehicle Safety Technology, which includes
Single Aircraft Accident Prevention, Accident
Mitigation, and Synthetic Vision Systems,
Weather Safety Technology, which includes
Aircraft Icing and Weather Accident Preven-
tion, and
System Safety Technology, which includes
Systemwide Accident Prevention, Search and
Rescue,2 and Aviation System Monitoring and
Modeling.
A fourth research component, security research,
will be added in FY04. The committee did not evaluate
this component since no research and development
work is currently under way. The AvSP also has an
effort in Technical Integration, which is separate from
the three research projects.
Research and development for AvSP is performed
at NASA Langley Research Center, NASA Glenn Re-
search Center, NASA Ames Research Center, and
NASA Dryden Flight Research Center, with the pro-
gram headquarters at Langley. A program organization
chart is shown in Figure 4-1.
AvSP was funded at $156.2 million in FY03 under
2Search and Rescue is funded through AvSP but is implemented
through the Office of Space Flight. Since all programmatic devel-
opment and all technical research are performed under the Office
of Space Flight, the Aviation Safety Panel did not review this work.
OCR for page 72
72
AN ASSESSMENT OF NASA 'S AERONAUTICS TECHNOLOGY PROGRAMS
I _ l Technical Integration
~ Aviation Safety Program ~ · Effort
// \~
System Safety
Technology Project
A:
Vehicle Safety
Technology Project
Synthetic
Vision
Systems
1 ~
Commercial
and Business
Aircraft
Single Accident
Aircraft Mitigation
Accident
Prevention
I ~ r
Vet ~ Prevention
Flight Critical
System Design
Propulsion
System Safety
Technologies
Control Upset
Prevention and
Recover
General
Aviation
Enabling
Technologies
,
_
Weather Safety |
Technology Project
Aircraft Weather
Icing Accident
Prevention
Design and
Analysis
Tools
Aircraft Ice
Protection
Education
and Training
Aviation
Weather
Information
Weather
Information
Comm.
Turbulence
Prediction
and Warning
Systems
FIGURE 4-1 Aviation Safety Program organization chart.
the full-cost accounting scheme.3 Vehicle Safety Tech-
nology accounted for $83.9 million (54 percent of the
AvSP total), Weather Safety Technology accounted for
$31.6 million (20 percent of the total), and System
Safety accounted for $40.7 million (26 percent of the
total). NASA is in the process of transitioning to full-
cost accounting from a net accounting scheme; previ-
ously, NASA managers assessed their budgets by the
amount of funding available to them for contracts,
grants, and other types of procurements. Uncler the net
accounting scheme, Vehicle Safety Technology is bud-
geted at $19.8 million, Weather Safety Technology at
$14.7 million, and System Safety Technology at $18.4
million. In this report, specific subprojects and tasks
are discussed in net dollars only, as this was the only
information provided to the committee. The net budget
breakdowns by subproject are shown in Table 4-1.
3Full-cost accounting encompasses all costs, including research
and program management; institutional infrastructure costs, such
as research operations support; direct procurements; direct civil
service workforce, benefits, arid Gavel; service pools; center gen-
eral and adrninis~ative (G&A); and corporate G&A.
_
Systemwide Aviation
Accident System
Prevention Monitoring
and Modeling
1 1 ,- 1
Human
Performance
Models
Maintenance
Human
Factors
Crew Training
Program
Human
Factors
Data Analysis
Tool
Development
Extramural
Monitoring
Modeling and
Simulations
Intramural
Monitoring
Like other NASA programs, each AvSP project has
a 5-year lifespan. This does not imply that the program
ceases to exist after 5 years, however. Project plans are
reevaluated after each 5-year time period to phase in
new projects that build upon previous research and de-
velopment.
Review Process
The Aviation Safety Panel was formed in Decem-
ber 2002 as one of three panels that would review
NASA's Aeronautics Technology Program. The Avia-
tion Safety Panel met for the first time on February 26-
28, 2003, in Washington, D.C. At this first meeting,
the 10-person panel received technical briefings from
the program and project managers in AvSP on the over-
al1 program, specific projects, and individual tasks.
After the first meeting, panel members participated in
site visits to each of the relevant NASA facilities
(NASA Langley, NASA Glenn, and NASA Ames).
The purpose of the site visits was to obtain a deeper
understanding of the research and development in the
program, to speak directly with the principal investiga-
tors for each project task, and to observe the products
OCR for page 73
ASSESSMENT OF THE AVIATION SAFETY PROGRAM
TABLE 4-1 Net Budget for the Aviation Safety Program
Budget (million $)
ProjeetlSubproject Name
FY03 FY04
Vehicle Safety Technology
Single Aircraft Accident Prevention
Accident Mitigation
Synthetic Vision Systems
Weather Safety Technology
Aircraft Icing
Weather Accident Prevention
System Safety Teehnologya
System-Wide Accident Prevention
Aviation System Monitoring and Modeling
19.8
4.7
8.4
9.8
0.4
2.2
7.2
5.0
9.7
s.o
8.4
13.9
17.7
0.2
2.6
7.0
s.o
8.9
5.1
8.6
aSystem Safety includes Search and Rescue, which is not reviewed here.
SOURCE: Information provided to the NRC panel by G. Bond, Aviation Safety Program Office,
NASA Langley Research Center.
and facilities firsthand. The site visits are listed in Ap-
pendix C. Panel members visited on-site or spoke via
teleconference with NASA personnel from every AvSP
task. Panel members, who were experts in their fields,
also reviewed technical reports and journal articles and
followed up with individual principal investigators by
means of teleconference calls and written questions.
Before the first meeting, the NRC asked each prin-
cipal investigator to complete a short questionnaire
with 12 questions relating to the research and develop-
ment goals, products, roadblocks, users, and technical
outcomes. A blank questionnaire is shown in Appen-
dix D. The completed questionnaires were distributed
to the panel for review prior to the first panel meeting.
Thus, the panel members were already somewhat fa-
miliar with the programs and projects under review
before they were briefed in person by the NASA re-
searchers and program managers. The questionnaires
proved to be a valuable tool for the panel in performing
its program assessment.
Upon completion of the three site visits, the panel
met for a second time, again in Washington, D.C., on
May 27-29, 2003, to come to consensus on findings
and recommendations for the program. The panel dis-
73
cussed outstanding questions and issues of concern
with program staff from NASA. It also developed
crosscutting observations across the different projects
and tasks within AvSP. The panel then provided its
input to the Aeronautics Technology Programs parent
committee in the form of working documents. Four of
the ten panel members represented the panel on the
committee.
PORTFOLIO
The committee evaluated the appropriateness of the
AvSP research portfolio based on the amount of basic
research versus user-driven research; the presence of
gaps or incomplete areas of research; the balance be-
tween high-risk, high-payoff research and more evolu-
tionary work; and whether or not the portfolio ad-
dresses real-world problems.
The committee is concerned about the balance be-
tween fundamental and product-driven research in the
Aviation Safety Program. It observed a shift away from
essential basic research over recent years. Such basic
research is necessary for the development of future
safety products that will enable the AvSP to reduce
OCR for page 74
74
': 1
AN ASSESSMENT OF NASA 'S. AERONA UTlCS TECHNOLOGY PROGRAMS
accident rates. In some instances, the committee ob-
served ineffective work-arounds created out of neces-
sity to divert resources from funded, low-payback
projects to accomplish unfunded but critical basic re-
search. Furthermore, with a few notable exceptions
(such as the Aircraft Icing subproject and the Modeling
and Simulations task in the Aviation System Monitor-
ing and Modeling subproject), the committee felt that
this problem was widespread within the program.
The committee found examples of research that is
essentially complete and ready for transition (such as
Fault Tolerant Modular Architectures-, Personal Elec-
tronic Device electromagnetic susceptibility, virtual
and augmented reality for maintenance crews, and the
Performance Data Analysis and Reporting System).
The committee also found places where basic research
was lacking for example, high-temperature materi-
als for engines, weather display interfaces, turbulence
warning systems, and human factors work in many ar-
eas. The committee's findings and recommendations
regarding specific instances where research is too prod-
uct-driven or where additional basic research is needed
are presented in the discussion of each task.
Finding: Support for Basic Research. There has
been a shift away from essential basic research in
the Aviation Safety Program in recent years.
Program Recommendation: Support for Basic Re-
search. The Aviation Safety Program should rein-
state a core competency program dedicated to basic
research that is essentially unencumbered by short-
term, highly specified goals. Without a strong basic
research program, the more applied research even-
tually suffers from a lack of good ideas and trained
personnel. The criterion for starting or restarting
such an activity within a Center is that a need must
exist for knowledge that is not now available.
The committee noted specific gaps in the portfolio
at the subproject and task levels in subsequent sections.
It found one significant program-wide omission in the
research portfolio: rotorcraft.
Finding: Rotorcraft. Rotorcraft safety can be im-
proved with additional research in the areas of de-
cision aids, synthetic vision, training, workload, and
situational awareness.
Program Recommendation: Rotorcraft. The Avia-
tion Safety Program should reincorporate rotor-
craft research into its program. The research should
consider the most effective approaches for reducing
the workload of rotorcraft pilots and improving
their ability to conduct safe, low-speed, low-altitude
rotorcraft operations in obstacle-rich environments
and in adverse weather.
PROGRAM PLAN
The AvSP program plan emerged from a series of
strategic planning sessions on aviation safety in 1997
known as the Aviation Safety Investment Strategy
Team (ASIST). ASIST established a vision and priori-
tized the research and development investment areas
for the AvSP. The AvSP approach includes system
modeling, accident mitigation, and accident prevention,
with an emphasis on mitigating problems that contrib-
ute most heavily to accident and fatality rates. The
AvSP was established in 2000 with a 5-year program
plan.
Each individual task within the AvSP is structured
to last 5 years. This 5-year programming cycle is more
suitable for a product-oriented program. It is difficult,
if not impossible, for NASA to maintain core compe-
tencies with these 5-year programs. In addition, there
do not appear to be sufficient off-ramps to transition
research that has been completed before the 5-year time
window closes.
Finding: Use of Sunset Requirements. NASA func-
tions on a 5-year schedule to the detriment of solid
research.
Program Recommendation: Use of Sunset Require-
ments. The Aviation Safety Program should struc-
ture its program based on the natural duration of
each research effort and not compel conformity to a
5-year cycle for every task. NASA should eliminate
arbitrary time constraints on program completion
and schedule key milestones based on technology
maturity, task complexity, and resource limitations.
Research involving the human-machine interaction
and causes of human error should be a major focus of
any aviation safety research program. The AvSP con-
tains a wide array of human factors research, from syn-
thetic vision displays to aviation weather information
requirements to tools for aircraft maintenance teams.
In general, the committee found evidence of high qual-
OCR for page 75
ASSESSMENT OF THE AVIATION SAFETY PROGRAM
,
.j
ity in all of NASA's human factors research; however,
it also found that the work was not always well inte-
grated into a cohesive program without overlaps. The
committee approves of the efforts of the Aviation
Safety Program Office in pulling together some of the
disparate human factors tasks through cross-center
meetings and through the Program Human Factors task
of the Systemwide Accident Prevention subproject.
However, the committee did not observe any improve-
. . . .
ment In the ~ntertask communication or any synergy
from the human factors research within the program.
Aviation accident data make clear that human er-
ror is a much greater factor than hardware or software
failure or environmental conditions. Ideally, every
technology effort should be examined from a human
factors perspective at an early design phase to antici-
pate problems. However, the advice of human factors
professionals, who must necessarily draw on the softer
behavioral sciences, is often disregarded by the engi-
neering designers, who view it as negative or irrelevant.
NASA has traditionally supported research in human
factors, and the human factors group at NASA Ames
has truly been a national resource.
Finding: Human Factors Research. In recent years
the Aviation Safety Program's work in human fac-
tors has been eroding; senior in-house research staff
have left, and in order to get the work done, more
human factors professionals have found themselves
managing contractors, a task for which they often
are not well qualified. Crosscutting efforts to inte-
grate human factors have also suffered.
Program Recommendation: Human Factors Re-
search. Critical human factors expertise should be
better supported in order to maintain critical mass,
to foster basic research in this field, to identify gaps
in our understanding of safety, and to be available
to consult with various NASA projects.
Program Recommendation: Early Analysis of Hu-
man Factors. Project requirements should include
requirements for human factors analysis early in
the design phase.
The committee found that the considerable layers
of both line management and project management ob-
scure the lines of accountability in AvSP. In at least
one case, a person's subordinate in the research project
hierarchy is his or her superior in the line staff hierar-
75
_
chy. The committee felt that subproject- and task-level
plans, goals, metrics, and responsibility could not be
clearly traced back to an overarching plan and vision
for the AvSP. In other words, the planning appeared to
be more bottom-up than top-down. Additionally, the
committee heard from a number of technical civil ser-
vants in the program that too much of their time was
spent "doing management" (e.g., making PowerPoint
slides) and not enough doing science and technology.
In addition, it was not clear to the committee what
methods and metrics NASA uses to evaluate objec-
tively the status of its research projects against its own
stated goals. The program effort in Technical Integra-
tion (described in a subsequent section) would be a
natural place for such an evaluation.
Finding: Management Structure. The organiza-
tional structure is unnecessarily complex, making it
difficult to trace lines of responsibility. Subproject-
and task-level plans, goals, metrics, and responsi-
bility could not be clearly traced back to an
overarching plan and vision for the Aviation Safety
Program.
Program Recommendation: Management Struc-
ture. NASA should articulate a clear, long-range
plan for the Aviation Safety Program and a hierar-
chy of goals, and it should adopt a less complex
management system that enables program account-
ability and implementation to be clearly traced.
The committee suggests that NASA reexamine its
names for the AvSP activities (many terms sound like
they overlap or are ambiguous) so that the goals of each
major project are easily understood. This ambiguity is
particularly evident in the Single Aircraft Accident Pre-
. .
venhon su project.
TECHNICAL PERFORMANCE
The committee asked a variety of questions to as-
sess the technical quality of the work, to evaluate the
facilities and personnel, to find evidence of system-
level assessments, and to determine the balance be-
tween experimental and theoretical work. The commit-
tee also compared the quality of the AvSP work relative
to that of other work in industry, academia, and gov-
ernment, including international work.
The committee found the technical quality of the
AvSP to be very good. In some cases, particularly in
OCR for page 76
76
specific parts of the weather work, NASA personnel
can be considered among the world leaders in their re-
spective fields. The review committee found the facili-
ties to be adequate for achieving the research goals; in
some cases (such as the icing wind tunnel), the facili-
ties can be considered true national assets.
The committee identified several specific tasks and
subtasks that have achieved an outstanding level of
technical achievement:
.
i
AN ASSESSMENT OF NASA 'S AERONA UTICS TECHNOLOGY PROGRAMS
by NASA that are similar to or have considerable over-
lap with products developed by industry. (Specific ex-
amples will be discussed in the task-specific sections.)
Finding: Redundancy with Industry. Some prod-
ucts being developed by NASA are similar to or
have considerable overlap with products already
developed by industry.
Program Recommendation: Benchmarking Against
Industry. The Aviation Safety Program should com-
pare (benchmark) its research projects against
those of other research and development entities in
government and industry to ensure that NASA's
work is leading. If it is not, NASA should terminate
the work.
.
Structures health management subtask of the
Vehicle Health Management and Flight Criti-
cal System Design task in the Single Aircraft
Accident Prevention subproject,
Mode confusion subtask of the Vehicle Health
Management and Flight Critical System Design
task in the Single Aircraft Accident Prevention
subproject,
Scale model development and testing work in
the Single Aircraft Accident Prevention sub-
proJect,
· Design and Analysis Tools task in the Aircraft
Icing subproject,
Aircraft Ice Protection task in the Aircraft Icing
subproject, and
· Crew Training task in the Systemwide Acci-
dent Prevention subproject.
A number of outstanding products have been de-
veloped, but many of these (an example being the Per-
formance Data Analysis and Reporting System
(PDARS) trajectory monitoring tool) are ready for
handoff to industry. Much of the low-TRL research is
excellent, but its relevance and potential usefulness
seem not to have been made clear to potential users (a
good example is human performance models).
USER CONNECTIONS
User connectivity was evaluated in two separate
ways. First, the committee asked how well NASA per-
sonnel reflect and leverage work being conducted else-
where and how well NASA research results are ac-
cepted and adopted by the outside community. Second,
the committee asked how the research itself is con-
ducted --- for example, if it uses an appropriate mix of
internal and external personnel.
In comparing the work of the Aviation Safety Pro-
gram with other work in the community, the committee
found several instances of products being developed
Exploring the second aspect of user connectivity
(how well the program uses expertise from the outside
community), the committee found that the answer var-
ied from task to task. In some cases (particularly the
Vehicle Safety Technology project), the committee felt
the project would benefit from additional involvement
with the outside community. In particular, the commit-
tee believes NASA's fundamental research projects
would benefit from increased university participation.
In other cases (especially in the System Safety Tech-
nology project), the committee felt there were too few
in-house personnel and that too much of the research
was being conducted by contractors. This tends to
weaken the core competencies of NASA.
ASSESSMENT BY PROJECT
Technical Integration Pro jest
The AvSP has an effort in Technical Integration,
which is designed to provide program assessments,
develop systems-level implementation strategies, and
integrate research and development efforts across pro-
gram tasks, particularly in flight testing.
The committee believes the concept behind the
Technical Integration project is very important, provided
it plays a significant role in deciding what research to
undertake and when such research should be modified,
transitioned to industry, or discontinued. The commit-
tee understands that because the Technical Integration
project began after the current 5-year plan had begun, it
has been playing catch-up with regard to its status in the
overall program. However, the committee had difficulty
OCR for page 77
ASSESSMENT OF THE A VIA TlON SAFETY PROGRAM
, . ~
determining the effect of the Technical Integration ac-
tivities on current planning. The Technical Integration
project seemed to be running almost as an independent
activity somewhat disconnected from project manage-
ment. The committee also observed that subjective
evaluations are being made, mostly by NASA project
managers, and it believes that NASA should have more
input from customers and industry and from lower-level
managers, scientists, and engineers engaged directly in
the various efforts. For example, the market penetration
of AvSP products should be studied.
The Technical Integration effort as currently con-
stituted seems best suited for evaluating AvSP's near-
term products. However, the committee is concerned
about how Technical Integration will integrate project
"stovepipes" into a workable whole. For example,
there appears to be little integration of NASA Ames
human factors activities with the synthetic vision work
at NASA Langley. The committee also sees a need for
anticipatory or prospective integration of the Human
Performance Models task, the Monitoring and Simula-
tion task, and the other monitoring tools efforts.
Finding: Use of the Technical Integration Project.
The Technical Integration effort does not play the
role it needs to play in deciding what research to
undertake, in performing cost-benefit analyses for
projected and ongoing projects, and in deciding
when such research should be modified, trans-
itioned to industry, or discontinued.
Recommendation: Use of Systems Engineering.
NASA project managers should employ systems
engineering approaches to ensure proper integra-
tion of projects.
Recommendation: Use of a Quality Assurance Pro-
gram. NASA should institute a quality assurance
activity, separate and independent from project
management, the results of which should be re-
ported directly to the Aviation Safety Program
manager and to the Aerospace Technology Enter-
prise associate administrator.
As discussed in the assessment of the Airspace
System Program, above, there appear to be significant
overlaps in the various system modeling efforts within
NASA, and it may be feasible to consolidate or inte-
grate some projects. In particular, modeling research
by the AvSP should be coordinated with Virtual Air-
77
space Simulation Technologies, which is part of ASP.
NASA should also develop and implement a plan for
evolving current models, simulations, and analysis
tools into large-scale models.
The committee applauds the Technical Integration
support of the Commercial Aviation Safety Team and
the Joint Implementation Measurement Data Analysis
team.
Vehicle Safety Technology Pro tech
Background
The Vehicle Safety Technology project is designed
to strengthen aircraft against vehicle system and com-
ponent failures, loss of control, loss of situational
awareness, and postcrash and in-flight fires. The
project focuses on applications for the aircraft itself.
The majority of the research is conducted at NASA
Langley, with a relatively small amount of work in pro-
pulsion safety and fire prevention conducted at NASA
Glenn. The Vehicle Safety Technology project was
funded at $83.9 million (full-cost)/$19.8 million (net)
in FY03 and is divided into three subprojects: Single
Aircraft Accident Prevention, Accident Mitigation, and
Synthetic Vision Systems. In net dollars, Single Air-
craft Accident Prevention is funded at $10.4 million,
Accident Mitigation at $2.2 million, and Synthetic Vi-
sion at $7.2 million.
Portfo/io
The goals of the project are focused on the vehicle
itself, in applications related to the flight deck, flight
critical systems, propulsion, and airframe. The research
focuses on loss-of-control prevention and recovery;
flight critical systems; vehicle health monitoring; pro-
pulsion systems safety; fire mitigation, detection, and
prevention; and improving low-visibility conditions by
providing a synthetic picture of the outside world.
This is an ambitious project with many diverse
goals, applications, and areas of research expertise. The
folding of such diversity into a single project and the
integration of the results of each research effort present a
considerable challenge. As with all AvSP programs, the
projects within the Vehicle Safety Technology project
have a 5-year life span. The termination point for the
Vehicle Safety Technology Project tasks is scheduled to
be 2005, although many of the projects will probably be
continued in some form into the next phase of the AvSP.
OCR for page 78
78
The committee found the researchers stretched in many
directions in the Vehicle Safety Project and believed it
was unlikely that every subtask could achieve its stated
goals to an appropriate level of detail by the project ter-
mination point in 2005. Further, the fact that the names
of many research tasks seemed to be similar suggested
that some tasks could be combined.
Overall, the committee believes there is an appro-
priate balance of low-TRL work with more applica-
tion-driven research in the Vehicle Safety Technology
project. Across the AvSP as a whole, the committee
has some concerns about the increasing trend toward
product-driven research and development, so it was
pleased to see several fundamental, low-TRL tasks
within Vehicle Safety Technology, such as the design
work in the Control Upset Prevention and Recovery
(CUPR) task. The committee urges a continued em-
phasis on this basic research in the next phase of the
Aviation Safety Program. At the same time, the com-
mittee notes that several tasks for example, some of
the fire prevention work and fault-tolerant integrated
modular architectures have already attained a high
level of technology readiness and should be
transitioned to industry.
The committee is sensitive to the fact that by fo-
cusing on fewer concepts, the project eliminates other
worthy research ideas. However, despite recommend-
ing that the Vehicle Safety Technology project focus
on fewer tasks in greater detail, the committee also
found a significant omission in the array of activities in
this project namely, rotorcraft. The committee be-
lieves that NASA could have a significant impact on
rotorcraft safety by including the topic in this project.
The committee believes that NASA's decision to ter-
minate rotorcraft work is a mistake, as there are a num-
ber of real-world problems in rotorcraft safety that ap-
parently are not being addressed outside NASA.
Program Plan
The committee believes that NASA will make sig-
nificant impacts if it can mature the technologies in the
Vehicle Safety Technology project. However, the com-
mittee judges the program plan for technology matura-
tion to be overoptimistic.
Finding: Portfolio Breadth. Despite the encourag-
ing progress reported. to date, the time remaining is
insufficient to achieve the goals set forth in the pro-
gram plan. The breadth of the work in Vehicle
AN ASSESSMENT OF NASA 'S. AERONA UTlCS TECHNOLOGY PROGRAMS
Safety Technology is coming at the expense of tech-
nical depth.
Recommendation: Portfolio Breadth. The Aviation
Safety Program should narrow the scope of activi-
ties in the Vehicle Safety Technology project to in-
crease the depth of research activities and focus
them in fewer, more specific, higher-priority areas.
A few tasks within the Vehicle Safety Project have
already reached a high TRL, and the committee noted
that there were no appropriate off-ramps or transitions
for those tasks that have reached or will reach comple-
tion before the 2005 project end date. Specific instances
are noted in the commentary on the individual tasks,
below.
Technica/ Performance
The committee found the individual researchers to
be bright, aware of the relevant literature, and able to
answer both theoretical and application-related ques-
tions. The facilities are state of the art and appropriate
for carrying out the project.
The committee found evidence of several notewor-
thy research tasks within this project that have a high
level of technical achievement, such as the structures
health management subtask. Several other tasks per-
haps should be transitioned because they have already
completed their research objectives, such as fault-tol-
erant integrated modular architectures and some of the
fire mitigation work. In no case did the committee rec-
ommend research termination for lack of quality.
The committee is concerned about the functional
integration of the many diverse activities talking place
across the different NASA research centers. NASA
should develop software ant! hardware interface specifi-
cations that connect the various subsystems early on to
aid in the integration and definition of the scope and
plans for program research. These specifications can be
spiraled into more detail and refined accordingly as the
program evaluations progress. They form the basis for
integrating the work taking place between the NASA
centers and NASA contractors. These interfaces should
include interactions between all the vehicle subsystems,
including the controls arid display tasks.
Finding: Interim Integration Milestones. There ap-
pears to be a lack of interim task-level milestones to
track the progress of integration activities.
OCR for page 79
ASSESSMENT OF THE AVIATION SAFETY PROGRAM
Recommenclation: Interim Integration Milestones.
NASA should integrate the information that sys-
tems evolving from individual tasks such as Vehicle
Health Management and Flight Critical System
Design and Control Upset Prevention and Recovery
can provide to the flight-deck crew.
Recommendation: Interim Integration Milestones.
NASA should develop software and hardware in-
terface specification documents that address the
various subsystems early on to aid in the integra-
tion and definition of the scope and plans for pro-
gram research.
Recommendation: Interim Integration Milestones.
NASA should incorporate interim test and evalua-
tion milestones for its flight simulation facilities to
measure the impact of its design integration on on-
· ~ · ·. ~
going crew performance act~v~es.
User Connections
In general, the committee found that the research-
ers are collaborating with the appropriate outside
agents; by and large, there is the right degree of in-
volvement with industry, and the connectivity to the
research community is impressive. In a few cases, es-
pecially in areas with low-TRL work, the NASA re-
search could be augmented with university research.
Specific instances are noted below. It appears that uni-
versity involvement is relatively minor, notably in for-
mal methods of software verification and validation
and in some of the propulsion safety technologies.
Assessment by Subyroject
Single Aircraft Accident Prevention Subproject
The Single Aircraft Accident Prevention (SAAP)
subproject is designed specifically to develop and
implement technologies that enhance aircraft airwor-
thiness and resiliency against loss of control while in
flight. Again, the work focuses on onboard technolo-
gies for the individual vehicle. The subproject contains
three tasks: Vehicle Health Management and Flight
Critical System Design (VHM and FCSD), Propulsion
System Safety Technologies, and Control Upset Pre-
vention and Recovery (CUPR). The net budget for
SAAP is $10.4 million in FY03, with $4.8 million for
79
VHM and FCSD, $4.1 million for Propulsion Safety
Technologies, and $1.5 million for CUPR.
NASA's effort to expand and improve industry
knowledge of the aerodynamic performance envelopes
of transport-category aircraft appears to be on target for
reducing the incidence of loss-of-control accidents. This
subproject promises to improve the fidelity of flight
simulators used as tools for improving pilot performance
in manual recovery from extreme attitudes. The research
could lead to better avoidance of such conditions as well
as to systems that effect automatic recovery.
By their nature, many of the modeling and analysis
efforts do not have a well-defined end point, and there
is always room for improvement. The lack of a clear
completion point for some of the SAAP work was nev-
ertheless troubling, and the committee believes that
NASA should develop ways to "declare victory" and
make clear the degree to which the effort has succeeded
and the amount of research still needed to achieve suc-
cess. For example, the modeling of follower aircraft
interaction with wake vortices from lead aircraft is in
its infancy because of the complexity of the problem,
but it should continue to be pursued in future years. On
the other hand, the work in fault-tolerant integrated
modular architectures is at a high TRL and ready for
transition to industry.
Finding: Wake Vortex. While wake vortex interac-
tions have an obvious impact on capacity, there are
equally important safety considerations, and the
AvSP is not sufficiently involved in the wake turbu-
lence effort.
Recommendation: Wake Vortex. NASA should in-
clude wake turbulence interaction models in its
Control Upset Prevention and Recovery dynamics
modeling and simulation technologies work. Cur-
rent models used in airline training simulators are
quite crude and provide insufficient fidelity for ef-
fective pilot training purposes.
The committee was pleased to observe the excite-
ment in using the 1/20 scale model 757 for both flight
and wind tunnel tests of control upset and other tasks.
Such tests could integrate and coordinate the diverse
activities in SAAP.
The collaboration with other relevant parties (the
FAA, DoD, and industry) appears to be excellent in
this subproject.
OCR for page 80
so
Vehicle Health Management and Flight Critical System
Design Task
The goal of this task is to research technologies
to reduce loss-of-control accidents and system or
component failures on the vehicle itself. Even within
this single task, there is a large array of activities,
from structures health evaluation to software integ-
rity evaluation. Research is conducted in the follow-
ing areas:
Structures health management
Flight systems health management
Verification of neural networks
Mode confusion
Software safety
Requirements modeling
· Recoverable computing
Modular avionics
· Electromagnetic susceptibility of avionics
· Neutron particle effects on flight critical
systems
· Validation methods
The TRL also varies widely across the work in this
task: Some of the software work is at a relatively low
TRL. while some of the structures health monitoring
work is near product development stage. The net fund-
ing for the Vehicle Health Management and Flight
Critical System Design work is $4.8 million for FY03
and is scheduled to be $5.1 million for FY04.
The committee found a number of activities in this
task worthy of commendation. It was particularly im-
pressed by the specific research activity in two areas:
structures health management, particularly the fiber-
optic strain systems (FOSS), and pilot confusion over
automation control-display modes.
The structures health management activity is an
area that NASA should showcase in the program. It has
made significant progress in a relatively short amount
of time, and NASA has truly catalyzed breakthroughs
in this area. In general, the cost-benefit analysis work
done in this area is impressive, and it is clear that
NASA knows what it would take to install and Held its
systems. The committee found the task to be well
thought out in terms of the interaction between corro-
sion and other properties of aging materials and the
measurement and diagnosis of structural faults. The
FOSS work at NASA Langley is interesting with an
appropriate blend of fundamental and user-driven re-
AN ASSESSMENT OF NASA 'S AERONA UTICS TECHNOLOGY PROGRAMS
search. The potential payoff in this area is very high.
The mode confusion work also has a very high poten-
tial payoff, and the work being performed in this area is
novel and of high quality.
The committee also encourages NASA's contin-
ued involvement in the verification of flight systems,
particularly as software becomes more complex and
new issues must be addressed, bringing corresponding
increases in cost and development time. In addition,
NASA should continue to foster the introduction of
object-orientecl (OO) programming into the flight criti-
cal software area.
Flight critical software is software onboard an air
vehicle that is used to control the vehicle and whose fail-
ure would lead directly to the Toss of that vehicle. Be-
cause of the cost of recertification, this is an area in which
commercial companies are slow to invest, even though
all recognize the eventual payoff. The payoff of GO
techniques, while not directly related to safety, comes
from reuse, savings in cost and time, and increased effi-
ciencies in verification and validation activities.
It would be useful if NASA could determine or
demonstrate ways to reduce the risks and costs of re-
certifying software, and its activity in OO program-
ming with the FAA is a good step in that direction.
As the committee noted in its subproject discus-
sion, this task could benefit from fewer tasks. There is
such a broad range of activities within this subproject
that the committee found it unlikely they all can be
brought to meaningful closure, with an appropriate
TRL, by the task's end in 2005. Specifically, the com-
mittee believes NASA should reorganize that portion
of the SAAP that combines VHM (including the model-
based diagnostics of the propulsion arm) and the detec-
tion, identification, reconfiguration, and recovery part
of CUPR in a single anomaly detection, identification,
and reconfigurationlrecovery structure. This would
eliminate the appearance of redundant research efforts
and further enable functional integration.
Finagling: Portfolio Breadth. The Single Aircraft
Accident Prevention activities are linked by their
common goal (reducing system or component fail-
ures on the aircraft) but not necessarily by common
expertise or research methodologies. Similar activi-
ties appear to be taking place in multiple subtasks.
Recommendation: Portfolio Breadth. NASA should
restructure or descope this task.
OCR for page 81
l'
,
ASSESSMENT OF THE AVIATION SAFETY PROGRAM
In several specific areas the committee has doubts
about the utility of NASA's continued involvement.
While the committee understands NASA's desire to
offer a complete solution to the Flight Critical System
problem, the committee is not convinced that NASA
should be working in fault-tolerant integrated modular
architectures. Commercial companies are in this busi-
ness and competing heavily with one another. The TRL
of this technology is well above 6, and NASA is not
needed to foster innovation. The second area that the
committee questions is in personal electronic device
electromagnetic susceptibility. This work. seems more
appropriate for industry (i.e., airlines and airframers).
The committee understands that part of this work is
sponsored by an airline but believes that the effort
should have low priority in the NASA research plan.
Finding: Modular Architectures and Personal Elec-
tronic Devices. The work in fault-tolerant inte-
grated modular architectures and personal elec-
tronic device electromagnetic susceptibility is at a
high TRL and more appropriate for industry.
Recommendation: Modular Architectures and Per-
sonal Electronic Devices. NASA should terminate
its involvement in modular architecture develop-
ment and electromagnetic interference activities in
order to concentrate resources in other less-re-
searched areas of the program.
Propulsion System Safety Technologies Task
The purpose of the Propulsion System Safety Tech-
nologies task is to reduce propulsion system failures as
a factor in civil aircraft accidents through the predic-
tion, detection, and testing of propulsion system mal-
functions and failures. The propulsion system safety
team works in system health monitoring, crack-resis-
tant blades and disks, and engine containment. This
effort is conducted at NASA Glenn and has a net bud-
get of $4.1 million per year in FY03 and FY04.
The committee found the researchers to be knowl-
edgeable and familiar with the relevant work in the
outside community. In general, the task was well orga-
nized and had a more focused goal and approach than
the other tasks in SAAP. The committee found two ar-
eas worthy of notice: model-based diagnostics and en-
gine sensor technology, particularly the eddy current
sensors. Also, the committee found that NASA has
played a key role in integrating the various aspects of
81
crack-detection technologies sensors, algorithms, and
testing resources.
NASA's involvement in model-based diagnostics
shows promise for onboard diagnostics and is a worth-
while investment, but it could benefit from integration
with related subtasks in SAAP.
Finding: Integration of Related Activities. The
model-based diagnostics subtask is not well inte-
grated with related activities in Single Aircraft Ac-
cident Prevention.
Recommendation: Integration of Related Activities.
NASA should integrate model-based diagnostics
with the vehicle health monitoring activities in the
Vehicle Health Management and Flight Critical
System Design task when it plans the future of these
tasks.
NASA efforts in embedded technologies with eddy
current sensors offer good promise in the early detec-
tion of faults. Engine companies are also working on
these technologies, however.
Finding: Eddy Current Sensors. Some of the eddy
current sensor work may be redundant with the
work by industry.
Recommendation: Eddy Current Sensors. NASA
should perform additional experimental work and
operational testing on these resilient sensors and
other sensors under development by the engine
companies only if it is leading and not following the
· ~
engine compames.
In general' the work in engine disk crack detection
and engine materials research is well integrated and
following good experimental practices. The commit-
tee believes this work could be enhanced with addi-
tional research at high temperatures.
Finding: High-Temperature Engine Materials.
NASA lacks some basic research activities in alter-
native high-temperature engine materials.
Recommendation: High-Temperature Engine Ma-
terials. NASA should also foster progress into other
engine materials and heat-treating technology. This
work might benefit from additional university in-
volvement.
OCR for page 94
94
AN ASSESSMENT OF NASA 'S AERONAUTICS TECHNOLOGY PROGRAMS
that have served the aviation community well for years
and there are sure to be many others that show great
promise for future applications. Many of the programs
are user driven, both internally and externally. Unfor-
tunately, the expectation that projects can be completed
in 5 years appears to constrain NASA's ability to ar-
ticulate a consistent and clear vision for Tong-term core
research. It also seems to impact the ability to create
well-defined goals that lead to an integrated research
program. The goal of providing a general knowledge
foundation, as in the case of human performance mod-
eling, should be part of a core research program. Bal-
ancing the System Safety Technology suite of applied
research activities with more basic research would help
sustain essential core competencies within the associ-
ated groups.
A specific long-range goal such as creating a fully
integrated virtual National Air Space model by 2020 or
2050 for modeling total system safety and efficiency
would be helpful in focusing and balancing research
projects. Research initiatives should support such a
central long-range goal, and the AvSP should work
closely with the VAMS effort in the ASP program to
achieve this. Requirements could then be more easily
developed, including problem statements, standards,
and test procedures. These should be established in a
way that encourages innovation while maintaining fo-
cus and accountability. Adclitionally, a continuous sys-
tems analysis approach would be constructive in iden-
tifying research priorities and allocating resources to
projects with the greatest impact on safety.
Program Plan
Many of the safety tasks have articulated very desir-
able outcomes, but plans to achieve these outcomes were
often unclear or lacked measurable milestones. For ex-
ample, a number of outcomes are in the form of a per-
centage reduction in accidents or in fatalities. There ap-
pears to be no method in place in the research program
for evaluating such outcomes or for assessing progress.
This gap appeared to be driven by a disconnect between
the resources or time required to accomplish the target
outcomes and the availability of assets and time. The
committee acknowledges that some of this work is low-
TRL and difficult to relate directly to measurable
changes in accident mitigation. Nevertheless, the com-
m~ttee believes that NASA should develop interim m~le-
stones and metrics for internally evaluating the success
of the System Safety Technology project relative to in-
tended project deliverables. This should be done in con-
junction with the Technical Integration effort.
The process of research project selection, planning,
resourcing, programming, and accountability within
the matrix management scheme was complex and dif-
ficult for the committee to understand. Some program-
matic decisions appear sensible from a safety perspec-
tive but do not seem to relate to an overall research
plan.
Technica/ Performance
The committee was impressed with the technical
capabilities of the NASA Ames staff associates! with
the System Safety Technology project. NASA Ames
has an excellent reputation for "basic applied" research.
The committee encourages NASA to uphold this strong
reputation by sustaining basic research programs at
Ames, where scientific publication is a core value.
The committee is concerned that the balance of in-
house and contractor personnel is becoming heavily
weighted toward outsourcing to an extent that could
compromise the ability to maintain core competencies.
Additionally, heavy outsourcing forces scientific per-
sonnel to focus on management oversight rather than
on building internal scientific activities. This discour-
ages young researchers from joining the NASA team
or even remaining with NASA.
Basic research seems constrained in a number of
areas owing to either lack of access to data or lack of
resources to process available data. Good examples of
this are the highly respected Aviation Safety Reporting
System product and the Maintenance Human Factors
task. As long as there are barriers to accessing data,
basic research could languish.
User Connections
The committee was impressed with the establish-
ment of an integrated FAA/NASA Aviation Safety
R&D Plan, an Aviation Safety Working Group, and an
Aviation Safety Program Executive Council, all to en-
sure greater coordination of FAA/NASA research. Ef-
fective use of these groups will be vital to establishing
post-2004 safety research goals.
In some areas NASA seems to be pursuing tech-
nologies or tools that have reached maturity or are
complementary to items already available in industry
or other government agencies. This is true, for example,
with Performance Data Analysis and Reporting Sys-
OCR for page 95
ASSESSMENT OF THE AVIATION SAFETY PROGRAM
tern (PDARS), Aviation Performance Measuring Sys-
tem (APMS), and the virtual reality maintenance work.
It is critical that regular, open, candid product
benchmarking and communications occur among
NASA, FAA, industry, and other research entities in
order to avoid duplication, to ensure that valuable and
limited resources are effectively and efficiently allo-
cated, and to sustain world-class research standards and
products.
Assessment by Subproject
System wide Accident Prevention Subproject
SWAP is the AvSP subproject devoted to human
factors and its relationship to error mitigation. The fo-
cus of the research is primarily in error modeling, train-
ing procedures, and maintenance procedures. SWAP is
also responsible for identifying crosscutting issues in
human factors that relate to the AvSP as a whole or to
other subprojects and tasks under the AvSP purview.
SWAP is broken into four tasks: Human Performance
Models, Maintenance Human Factors, Crew Training,
and Program Human Factors. SWAP is funded at a net
value of $5.0 million in FY03 and $5.1 million in FY04.
Human Performance Models Task
The Human Performance Models task utilizes cog-
nition and perception models to detect and analyze hu-
man error and to develop tools for system design. The
task works primarily with five human performance
models: Air Man-Machine Integration Design and
Analysis System (AirMIDAS), ACT-R/PM, A-SA, D-
OMAR, and IMPRINT/ACT-R. Each model uses a dif-
ferent cognitive approach and each has a different ap-
plication to sources of pilot error. The Human
Performance Models task has also developed a track-
ing system, the Crew Activity Tracking System, to pre-
dict operator behavior and to interpret operator actions.
The Human Performance Models task of SWAP is
funded at a net value of $1.5 million in FY03.
The activities in this area are appropriately
weighted toward fundamental research. The goal is
clearly to create state-of-the-art modeling techniques.
While resources seem adequate for the stated goals, the
5-year program constraint appears to limit the long-
term potential of this core research area.
Error analysis appears to focus on error as devia-
tion from nominal procedure rather than considering
95
which deviations are dangerous and which are merely
alternative but still acceptable ways to accomplish the
task. These alternative methods may in some cases be
better than the nominal (e.g., under off-normal condi-
tions). Expanding the scope of work to include accept-
ability analysis may broaden the potential application
of this effort.
While application of NASA human performance
modeling research to other efforts at NASA, such as
synthetic vision research, would seem promising, there
is little to show as yet. There appears to be no connec-
tion with human performance modeling at other gov-
ernment agencies.
Finding: Collaboration with Other Agencies. There
appears to be no substantive interface with human
performance modeling at other government agen-
cies such as the Air Force Laboratory's Human Per-
formance Modeling Integration Program, the De-
partment of Defense, or government laboratories
such as the Human Emulation Laboratory at Sandia
National Laboratories. NASA is not part of the Hu-
man Performance Modeling Special Interest
Area.~2
Recommendation: Collaboration with Other Agen-
cies. NASA should conduct collaborative research
with both the Defense Advanced Research Projects
Agency and the DoD to leverage techniques devel-
oped by these other agencies for piloting, decision
making, estimating human error in automated sys-
tems, and vigilance.
There is a well-documented, short-term plan with
reasonable milestones, but the long-range vision and
plan for this initiative lack definitive goals and metrics.
Development of a method for comparison across mod-
els is encouraged, since current metrics vary from
model to model.
Finding: Human Factors Outreach. Much of the
Human Performance Models work was done by
human factors engineers for human factors engi-
neers. There is too little outreach to NASA engineers
in other disciplines who should be future users of
these models. Additionally, program deliverables
and their purposes were not clearly articulated.
12See .
OCR for page 96
96
Recommendation: Human Factors Outreach. The
NASA Human Performance Models group should
work with the managers of all internal aviation re-
search programs to identify each manager's need
for human performance models and to support the
testing of emerging models against human-in-the-
loop simulation as well as flight demonstration.
Recommendation: Human Factors Outreach.
NASA human factors programs should publish a
book or CD on the state of human performance
modeling to communicate what can realistically be
done in this type of modeling and to measure
progress in this research area.
Recommendation: Human Factors Outreach.
NASA should create, document, and apply more
clearly defined off-ramps for high-TRL Human
Performance Models.
NASA Ames has maintained an excellent reputa-
tion for sponsoring and convening human performance
modelers for several decades. It is essential to strive
for continual high quality since human lives are af-
fected by the accuracy of the safety estimates derived
from these models. The committee applauds the par-
ticipation of academia in the NASA Ames aviation
safety work but strongly urges outreach to the govern-
ment agencies listed in the finding on collaboration,
above. In addition, this group has only been able to
apply models to a limited number of real-world prob-
lems, such as taxiing errors. The committee feels that
these models can be tested and improved by applying
them to additional real-world problems.
NASA is developing tools in this area for others to
use. However, actual and potential users, both manag-
ers and researchers, should be more clearly iclentified
so their input can be solicited when research and appli-
cations are being identified and prioritized.
Maintenance Human Factors Task
The Maintenance Human Factors task is designed
to develop "guidelines, recommendations, and tools
directly to maintenance personnel and managers"~3
through a combination of research in understanding
human error in maintenance and developing mainte-
i3B. Karlki, NASA-Ames, "Maintenance Human Factors," ques-
tionnaire completed in January 2003 (see Appendix D).
AN ASSESSMENT OF NaSA 'S AERONAUTICS TECHNOLOGY PROGRAMS
nance tools and aids to enhance safety. The mainte-
nance program focuses on risk analysis, resource man-
agement, advanced displays, and human error
baselines. The effort is funded at $1.1 million net for
FY03 and FY04.
The importance of human error in the maintenance
of aircraft was underscored recently by the US Air-
ways Express Air Midwest Flight 5481 accident. The
National Transportation Safety Board concluded in
May 2003 that the probable cause of the accident, in
combination with several other factors, was improp-
erly set elevator control cables a maintenance over-
sight. In this case, maintenance personnel skipped criti-
cal steps outlined in the maintenance manual because
they felt the steps were superfluous.
This maintenance human factors initiative is criti-
cal to reducing maintenance errors as well as to pre-
venting injuries to personnel and damage to equipment.
Industry applauds the effort. There are many facets to
this program, but the resources seem limited relative to
the need. This research group has made significant con-
tributions in raising maintenance human factors aware-
ness within the aviation community. However, this type
of research is still in its infancy and just beginning to
receive enough attention to identify data sources from
which to generate statistically sound trend information.
Finding: Maintenance Data Collection. Sources
from which to collect data have been identified, but
barriers to collection and processing seem to be
slowing productive research.
Recommendation: Maintenance Data Collection.
NASA should develop a clear plan to include inspec-
tion data and information from maintenance tech-
nician training in its research data set.
There is a coherent short-term plan for each of the
projects, but the long-range strategic goals seem to be
disjointed. The process used for selection of the par-
ticular research topics was unclear to the committee.
All are potentially useful tools at some level but lack
the anchor of a long-term research mission. Specifi-
cally, the committee is uncertain how the virtual reality
and augmented reality work differs from or comple-
ments what industry uses already and how such work
will be applied to real-world maintenance error mitiga-
tion. There also does not appear to be a systems analy-
sis approach to setting priorities for the research effort.
OCR for page 97
ASSESSMENT OF THE AVIATION SAFETY PROGRAM
Finding: Goals for Maintenance Research. The Main-
tenance Human Factors task is an excellent activity
but seems to lack clearly defined long-range goals.
Recommendation: Goals for Maintenance Research.
NASA's Maintenance Human Factors task should set
clear and quantitative long-range goals and test its
research against these goals annually. This is an area
for long-term research and should be an area for de-
veloping enduring core competencies.
Finding: Virtual/Augmented Reality. The project to
create virtual and augmented reality tools for main-
tenance technicians seems to be operating without a
clear understanding of what is available today in
automation for maintenance technicians and the
realities of an all-weather, real-world airline main-
tenance operation.
Recommendation: Virtual/Augmented Reality.
NASA should formally assess the enhanced displays
for maintenance research work, including what is
currently in use by the airline industry, to deter-
mine a more focused and practical approach to vir-
tual and augmented reality tools for maintenance.
The external community of maintenance human
factors researchers was described as small, and there
were said to be very close connections between agen-
cies and academia. However, the allocation of roles and
responsibilities among FAA, NASA, the Navy Safety
Center, the Air Force Safety Center, and other research
entities was not clear.
Finding: Outreach to Community. There are a few
omissions in the links to the outside community in
this task.
Recommendation: Outreach to Community. NASA
should establish links to the Air Force Safety Cen-
ter as well as airframe and power plant training in-
stitutions. NASA should perform an active outreach
to the maintenance technician unions for program
planning, research vetting, and research participa-
tion. NASA should collaborate with the Professional
Aviation Maintenance Association on aviation
maintenance research and with maintenance tech-
nician schools such as the Stratford School to col-
lect data and provide research results to enhance
safety training.
97
C~ · ~ 7
rew 1 raining 1 ash;
The goal of this task is to develop training tech-
niques and tools to help pilots avoid making errors that
lead to accidents and to manage in-flight problems in
situations brought about by external circumstances
such as weather or system failures. The effort is funded
at $1.9 million net for both FY03 and FY04.
The Crew Training task of System Safety Technol-
ogy has served the commercial aviation training com-
munity for many years, producing excellent research
work that could occur nowhere else. The current scope
of activities is excellent; however, without a long-term
core research plan, the projects seem disjointed. Sim~-
larly, the individual subtasks in Crew Training are well
planned but do not amount to a core training research
program. Training research is inherently a long-term
activity. Given the inability to go beyond the 5-year
horizon for NASA program planning, the researchers
in this task have tried to build longer-term research into
the current 5-year plan; for example, they have devel-
oped an anchor procedure for solving issues relating to
flaps and auxiliary power units.
In general, the committee found this research ef-
fort to be productive and of high quality, with several
activities in this task judged to be outstanding. The re-
search in distributed team performance is clearly state
of the art and is vital in developing flight as well as
maintenance training programs. This group has also
developed a number of high-quality training tools that
have been distributed to the aviation community, par-
ticularly the tool known as "How to Train Automa-
tion." It is clear that core competencies within this
group must be preserved.
There is significant interaction and trust between
this group and the aviation community operations and
training personnel as well as labor unions. The ALPA
training council had a meeting at NASA Ames in
March 2003. Boeing will be at NASA Ames to review
its internal research and development with NASA.
These links keep NASA honest and enhance transition
to industry.
Finding: Outreach to Community. The Crew Train-
ing task's already excellent user connection could be
enhanced by greater interaction with entities outside
the NASA aviation community, including high-level
training decision makers at the officer level of major
airlines and general aviation companies. Users could
OCR for page 98
98
be more involved than they currently seem to be in
setting goals for NASA training research.
Finding: Use of Milestones and Feedback. Clear
metrics for understanding the impacts of NASA-
developed training materials were not apparent.
Recommendation: Use of Milestones and Feedback.
NASA should institute a crew training quality as-
surance program complete with feedback tools that
measure adherence to goals and objectives, exit cri-
teria, and status in regard to similar research being
performed throughout the world.
Or '' - '
. . .;' .
., ~ .
. , ~
AN ASSESSMENT OF NASA 'S AERONAUTICS TECHNOLOGY PROGRAMS
The committee identified several areas in which
the training work could be expanded to have additional
impact within the aviation community. In particular,
there are needs and opportunities for research on main-
tenance training that could be addressed in addition to
flight crew training. Rotary wing crew training could
also benefit from the research expertise gained through
this task.
The committee felt that some of the projects, such
as research on the effects of low blood sugar on safety,
would fit better in other venues like FAA's Civil Aero-
space Medical Institute, which already has a long-
stancling program in this area. Additionally, some of
the research results (such as "How to Train Automa-
tion") might better be disseminated through the FAA
to avoid potential or perceived conflicts between regu-
lator expectations and a respected research body such
as NASA.
Program Human Factors Task
The goal of the Program Human Factors task is to
identify crosscutting issues in human factors within the
AvSP and to make specific human factors recommen-
dations to other projects within the program. The effort
is funded at $500,000 for both FY03 and FY04, in net
dollars.
The cockpit integration of the various and dispar-
ate tasks of the aviation safety technologies is impor-
tant and should be continuously and thoroughly ad-
dressed. The Program Human Factors task at NASA
Ames is designed to cut across multiple subprojects, in-
cluding Synthetic Vision Systems, Weather Accident
Prevention, and Single Aircraft Accident Prevention.
Each of these subprojects is to perform its own internal
human factors research. However, it appears that many
key researchers in human factors are affiliated with
Ames, making this an appropriate group to evaluate the
overall safety program from a human factors perspec-
tive.
The group has completed a crosscutting look at is-
sues arising as a function of humans interacting with
synthetic vision. The study revealed that off-nominal
procedures were weak; the technology was built, but
procedures were poorly developed. This was the only
work looking at full integration of synthetic vision with
other existing and emerging technologies. This is a
critical, real-world issue being addressed by no one
else.
The committee noted that the objectives of this
task seem to have diminished over time, with corre-
sponding reductions in allocated resources. Coupled
with the 5-year life expectancy of research projects,
the end result is that the plan to carry out the program
seemed somewhat fragmented. The committee noted
that there is only a single in-house researcher; all oth-
ers come from outside contractors and academia. This
threatens the future of the human factors core compe-
tencies at Ames that are so essential to long-term re-
search.
Finding: Acceptance of Program Human Factors.
The other projects within the Aviation Safety Pro-
gram may be unresponsive to the recommendations
of the Program Human Factors task.
Recommendation: Acceptance of Program Human
Factors. NASA management should foster greater
accountability for the findings of the Program Hu-
man Factors research and findings to ensure coop-
eration within NASA so that human factors issues
identified in Synthetic Vision Systems, Single Air-
craft Accident Prevention, and Accident Mitigation
are well considered by and integrated into all ap-
propriate projects.
The program is somewhat disconnected from the
users. As with most of the Ames programs, the poten-
tial users are quite broadly defined and interaction with
users is not sufficiently documented. Human factors
engineering has to be assertive to make clear its rel-
evance, and thus the committee encourages coopera-
tion and outreach with both industry and NASA Lan-
gley and broad dissemination of research results.
NASA should benchmark against similar external
work, such as military projects like those at the De-
fense Advanced Research Projects Agency, Big Pic-
OCR for page 99
"at ' ';
;
1
ASSESSMENT OF THE AVIATION SAFETY PROGRAM
sure, and Quiet Knight and in forums such as the FAA
and the Society of Automotive Engineers and leverage
the results of that work. NASA researchers should
present results of their work at the Institute of Electri-
cal and Electronics Engineers ant! the American Insti-
tute of Aeronautics and Astronautics to improve out-
reach to potential users.
Finding: Human Factors for Commercial Carriers.
The research in this area only considers commer-
· ~ · ~
cla1 alr carriers.
Recommendation: Human Factors for General
Aviation and Rotorcraft. NASA's Program Human
Factors research should add general aviation and
rotorcraft to its work on human factors, as it could
have great impact in these areas.
Aviation System Monitoring and
Modeling Subproject
The ASMM subproject develops technologies to
view aviation safety from a systemwide perspective,
develops metrics for the safety of the NAS, and pre-
dicts systemwide effects of changes to the NAS. The
subproject is composed of four tasks: Data Analysis
Tool Development, Extramural Monitoring, Modeling
and Simulations, and Intramural Monitoring. ASMM
has $8.4 million in net funding for FY03 and $8.6 mil-
lion in FY04.
Data Analysis Tool Development Task
The Data Analysis Tool Development task ana-
lyzes both digital and textual data. This work tends to
be low-TRL. The task develops concepts that are then
instantiated in some of the AS MM modeling efforts.
This task emphasizes tool design and development over
modeling. Currently, the task focuses on two major ar-
eas: digital data analysis tools and textual data analysis
tools. The first set of tools, a system known as the
Profiler, takes digital data from a system like the Avia-
tion Performance Measuring System to generate and
evaluate flight signatures. From these signatures, the
researchers produce a list of atypical flights, identify
the atypical parameters, and summarize the results. The
second set of tools, known as PLADS (which stands
for the steps in the preprocessing: Phrase ID, Leave,
Augment, Delete, Substitute) and the Automatic Lan-
guage Analysis Navigator (ALAN), preprocesses and
99
processes the kind of text data that would be found in
the Aviation System Reporting System. The goal is to
identify atypical situations without any a priori infor-
mation merely by sifting through the flight data. This
task is funded at $1.7 million in FY03 and FY04.
The committee was impressed with the work of the
contractors and their knowledge of analytical methods.
However, it was concerned that the expertise for devel-
oping this system is contractor-based and is not part of
the NASA Ames knowledge base. The committee was
generally impressed with the Profiler work and its abil-
ity to identify atypical parameters from signatures. The
committee also found the statistical methods used to be
sound.
In the text area, NASA does not seem to have le-
veraged existing software in use by the Securities and
Exchange Commission, the Defense Advanced Re-
search Projects Agency, and the intelligence commu-
nity. Data mining in the textual domain is a widely stud-
ied problem, and the committee suggests that the
researchers build on existing methodologies. In addi-
tion, the text data research work should be dissemi-
nated and benchmarked at major text search venues
such as the Text Retrieval Evaluation Conference spon-
sored by the National Institute of Standards and Tech-
nology.
The committee was encouraged to see collabora-
tion with Office National cl'Etudes et de Recherches
Aerospatiales, the French research agency. Such col-
laboration should be extended further to other foreign
agencies to assure quality benchmarking, including the
Japan Aerospace Exploration Agency, the Depart-
amento do Aviaco Civil (Brazil), the National Aero-
space Laboratory (Netherlands), the State Research
Institute of Aviation Systems (Russia), and the Defence
Science and Technology Laboratory (United King-
dom).
Finding: Use of Milestones. The intended path to
technology maturation for these data mining tools
was not clear. In particular, it was unclear how data
mining research was divided among the low-TRL
too! development work in this task, the work on data
mining applications taking place in the Extramural
Monitoring task, and the work on Aviation Perfor-
mance Measuring System analysis in the Intramu-
ral Monitoring task.
Recommendation: Use of Milestones. NASA should
define clear goals and objectives, exit criteria, and a
OCR for page 100
100
set of milestones for technology transfer or for the
next level of development.
Extramural Monitoring Task
. . .
~ . -
,. . I.
. - ,.
. ~
Am ASSESSMENT OF NASA 'S AERONA UTICS TECHNOLOGY PROGRAMS
The Extramural Monitoring task strives to create a
database of information to serve as the repository of
aviation safety events and trends and the basis for avia-
tion safety decision making. In particular, the task
works with two databases: the National Aviation Op-
erations Monitoring Service (NAOMS) and Aviation
Safety Reporting System (ASRS), with most of the task
investment in NAOMS. The overall funding for Extra-
mural Monitoring is $2.0 million.
NAOMS consists of a longitudinal survey of air-
craft operators, gathering information about safety-re-
lated experiences of pilots, cabin crews, and mainte-
nance operators for both general aviation and air
carriers. NAOMS is a random survey in which staff
proactively question active operators in a telephone
call. It provides statistically reliable results about the
frequency of occurrence of safety-related incidents.
In contrast, the ASRS is a joint FAA-NASA re-
porting system that asks for the voluntary participation
of operators who have experienced a safety-related
problem. ASRS is funded by the FAA, although NASA
administers the program.
To encourage submissions to ASRS, NASA makes
sure that the reporter remains anonymous. The FAA
had agreed that an ASRS report cannot be used as evi-
dence to substantiate an alleged violation in an enforce-
ment action.~4
Only a small portion ($250,000 of $2.0 million) of
the Extramural Monitoring budget supports ASRS-re-
lated activities. That portion of the budget addresses
data mining techniques applied to the ASRS database.
The NAOMS approach is built on research and
implementation of national surveys such as those of
the Bureau of Labor Statistics. The NAOMS sampling
methods have been grounded in sound interview poll-
ing science; however, the interviews are conducted by
professional pollsters, not aviation experts. The com-
m~ttee has some concern about the level of accuracy
attained by pollsters who have no expertise in the area
in which they are conducting the telephone interview.
The committee is also concerned about potential
Remark Blazy, 1999. "We all know about ASRS, but what's an
ASRP?" FAAviation News Magazine, October.
redundancy between NAOMS data and data available
from the air carriers or through the ASRS database.
The NAOMS project seems to be developing a meth-
odology to establish trends in aviation safety perfor-
mance that are already available through other sources
within the industry and government. For example,
NAOMS appears to duplicate what many airlines are
already doing both voluntarily and in FAA-mandated
programs to track trends for example, in engine
shutdowns. The NAOMS program may become more
useful when applied to the general aviation commu-
nity, however. NASA's decision to collect its own
primary data in this case should rest on the type of
research NASA wants to perform and whether that
research can be supported by information obtained
from the airlines. At this point, the committee does
not see a compelling argument for independent data
collection. Greater interaction with the Air Transport
Association and the airlines might help to clarify the
usefulness of this effort.
The ASRS program has been around for many
years. It is highly trusted by the pilot community
and is growing in acceptance by the maintenance
technician group. Because the program provides lim-
ited immunity from certificate action by the regula-
tor for errors by pilots, mechanics, and dispatchers
(not willful acts), some tasks within the regulatory
community resent the program, while others within
the research community disparage its value because
the inputs are voluntary. In truth, the threat of a cer-
tificate action strongly encourages the submission
of an ASRS. Unfortunately, the ASRS program is
currently resourced to input less than 25 percent of
the reports received into the database. Direct follow-
up for additional information from the reporting par-
ties can rarely be accomplished. Significantly greater
volumes of data are anticipated from emerging Air-
plane Safety Action Partnership (ASAP) programs,
with no anticipated increase in research resources.
This could create a serious shortfall in data available
to researchers. While the committee is aware that
the funding for the database collection work is pro-
videcl by the FAA, not NASA, NASA is still respon-
sible for maintaining the ASRS program. The com-
mittee finds the defined ASRS activity for NASA to
be much larger than its resource allocation; one or
the other requires modification.
It is important that when gaps in the ASRS data
occur, phone calls should be made to fill in what is
missing. The lack of resources to handle ASRS in a
OCR for page 101
ASSESSMENT OF THE AVIATION SAFETY PROGRAM
statistically sound manner is a significant issue in un-
derstanding the safety trends in the NAS.
There are many opportunities to accomplish more
research with the data available through the ASRS sys-
tem. It was not clear if there were plans in this research
task to optimize the joint use of ASRS and NAOMS.
Finding: Aviation Safety Reporting System. Regret-
tably, the Aviation Safety Reporting System data-
base is only inputting about 25 percent of the sub-
mitted reports. Interviews to follow up on Aviation
Safety Reporting System submissions are very lim-
ited owing to the lack of resources. The industry
believes that the Aviation Safety Reporting System
database has been underutilized for some time. The
National Aviation Operations Monitoring Service
is consuming the majority of the resources in this
project area.
Recommendation: Aviation Safety Reporting Sys-
tem. NASA should combine the National Aviation
Operations Monitoring Service methodology and
resources with the Aviation Safety Reporting Sys-
tem program data to identify aviation safety trends.
Modeling and Simulations Task
The Modeling and Simulations task seeks to incor-
porate human performance models into an analysis of
systemwide operations to identify safety-related char-
acteristics and predict system response to safety inter-
ventions. This program is not responsible for model
development, but it incorporates models from other
research efforts (such as the AirMIDAS mode! devel-
oped in the SWAP program) into a larger, systems-
level approach. The task is funded at $1.5 million in
FY03 and $1.6 million in FY04.
The committee applauds NASA's efforts to inte-
grate the various performance models with models of
the aircraft and air traffic control systems. This is bold
and difficult work and is the kind of research in which
NASA should be engaged. The TRL is low, but that is
a quality of long-range research that can only be ac-
complished by NASA. The weaknesses of the program
seem to be a lack of interconnectivity and integration
of tools as well as a limited ability to include issues
such as clear air turbulence effects on traffic conflict
and quality of performance. There is also no collabora-
tion between the program and other programs that
model environmental safety and noise.
101
The Reconfigurable Flight Simulator and Object
Based Event Scenario Trees modeling programs are not
tied to NAS models built using the FAA Consolidated
Operations and Delay Analysis System and Aviation
System Performance Metrics. Nor was there a tie to the
Total Airspace and Airport Modeler, which has been
validated by Eurocontrol, or the Traffic Organization
and Perturbation Analyzer model (developed by the
National Aerospace Laboratory in the Netherlands),
which has been used to estimate the safety-capacity
relationship that may be affected by airports at high
operational workloads.
Finding: Outreach to the Modeling Community.
The modeling programs in this area have excellent
potential but appear to lack coordination with other
similar modeling programs.
Recommendation: Outreach to the Modeling Com-
munity. This task should benchmark its perfor-
mance against other modeling implementation ef-
forts and consolidate programs where possible to
achieve a master system performance, capacity, and
safety model.
Intramural Monitoring Task
Intramural Monitoring refers to internal quality
assurance and safety functions within each air carrier
and air traffic management organization. The Intramu-
ral Monitoring products are the Aviation Performance
Measuring System (APMS) and the Performance Data
Analysis and Reporting System (PDARS). The APMS
project is designed as a tool for analyzing aircraft flight
data. APMS provides envelope data for each flight pa-
rameter in typical flights, provides information about
atypical flights, and provides descriptive statistics on
phase-of-flight performance. PDARS is designed to
collect, process, and analyze air traffic management
data. It generates daily reports, shares data among fa-
cilities, supports exploratory and causal analysis, and
archives data for developing baselines. Its major
strength is in the seamless integration of data from
multiple sources. The overall task emphasis is on safety
risk management. The task received $3.18 million in
FY03 and expects $3.25 million in FY04.
Some committee members worried that APMS and
PDARS were not novel. The committee believes com-
peting and sometimes superior systems are already
used by airlines.
OCR for page 102
102
l
.. . .
.. ...
AL ASSESSMENT OF NaSA 'S AERONAUTICS TECHNOLOGY PROGRAMS
Overall, the APMS program has been in the refine-
ment stage for several years. A target and milestones
for technology transfer or the next level of develop-
ment were not clear. Benchmarking of APMS against
similar programs in other government arenas and
academia seems to be lacking.
The APMS tools to mine anomalous data are en-
tirely appropriate and useful for airline flight opera-
tional quality assurance (FOQA) programs. However,
there are significant barriers—among them litigation
issues to centralizing a general FOQA database at a
government agency in the near term. This creates bar~-
ers to close interaction with the industry.
To make this array of activities more complete,
emphasis and resources in this program need to shift
further to integrating APMS and other complementary,
commercially available FOQA software into an inte-
grated operational efficiency and risk model.
Finding: Aviation Performance Modeling System.
The APMS software is mature in its development
and is ready for the off-ramp to the marketplace.
Recommendation: Aviation Performance Modeling
System. NASA should redirect the APMS resources
to pursue integrated data risk model research. The
weather overlay work is a clear example of the kind
of research that needs to be emphasized.
Finding: Performance Data Analysis and Report-
ing System. As a safety analysis tool, PDARS was
well designed and is being utilized extensively by air
traffic control management. PDARS is useful for
airspace design, but it is at a fairly high TRL and is
ready to be turned over to industry. The committee
identified only one remaining gap in the research
activity- data source integration.
Recommendation: Performance Data Analysis and
Reporting System. NASA PDARS resources should
be used to integrate PDARS data with traffic and
weather information to feed NASA's modeling and
simulation activities. In addition, methods to inte-
grate the Flight Operations Quality Assurance
(FOQA) program and Airlines Safety Action Part-
nership and Aviation Safety Reporting System in-
formation into the higher level models should be
developed.
OCR for page 103
Appendixes
1 .
.
OCR for page 104
Representative terms from entire chapter:
synthetic vision
