Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 13
2
Overall Assessment of the Pioneering Revolutionary
Technology Program
The observations and recommendations presented
here are overarching issues of concern throughout the
PRT program. Detailed assessments of the individual
PRT programs (CICT, ECS, and ECT) can be found in
Chapters 3 through 5.
OVERALL ASSESSMENT
The committee' s overall assessment of the research
within PRT was made based on information from the
individual assessments of the three supporting panels.
Tasks judged by the committee and panels to be world-
class met the following criteria: (1) they gave evidence
of productivity (publications, software, presentations,
patents, mission-accepted technology); (2) they exhib-
ited strong linkage at the task level to actual flight
projects, flight engineers, or science customers;
(3) they possessed connectivity with other research
communities external to NASA; and (4) they were rec-
ognized by external peers as an authority in the subject
matter. In some cases, excellence was also observed
when basic research, facilities, systems analysis, flight
integration, and test and evaluation were vertically in-
tegrated or when programs had achieved success over
a period of 10 to 15 years and continue to do so. Exem-
plifying this long-term excellence were the Energetics
project and the Advanced Measurement and Detection
(AMD) element.
13
The panels and committee were very careful to use
the descriptor "world-class" only when a project or task
clearly met the set of criteria listed above and was
clearly a leader in the field. The word "revolutionary"
was used only in very specific instances, when it was
perceived the work promised to provide leaps in capa-
bility or technology over current methods.
While there are some important concerns about
management practices within the PRT portfolio, the
committee found that most of the Pioneering Revolu-
tionary Technology (PRT) research consisted of good
work that is important to the future of NASA and the
nation. Ten percent of the individual research tasks
were judged as work of the highest quality, represent-
ing truly world-class endeavors. The committee and
panels recommended that another 10 percent of the
program's research tasks be discontinued or
transitioned to mission applications. Tasks marked for
transition are typically of excellent quality and involve
successful work ready to be funded by a NASA mis-
sion or external partners. Tasks marked for discontinu-
ation were identified primarily based on a judgment
about the relative quality of the work or its value to
NASA and its alignment (or lack thereof) with PRT
program goals. With 80 percent of the program being
of good quality, but not world-class, there is an oppor-
tunity for improving PRT program research by focus-
ing more attention on several common issues, includ-
OCR for page 14
14
AN ASSESSMENT OF NASA 'S PIONEERING REVOLUTIONARY TECHNOLOGY PROGRAM
ing the need for research to be more results-oriented,
pervasive use of systems analysis, peer review, and in-
creasing collaboration between outside experts and the
program.
Finding: The committee judged approximately 90
percent of the PRT program to be good work, im-
portant to NASA and the nation. Of this 90 percent,
10 percent was deemed to be world-class.
COMMON THEMES
The committee noted six themes cutting across the
entire PRT program where special attention would
strengthen the program: systems analysis; benchmark-
ing and metrics; external peer review and competition;
stability and continuity; research portfolio balance; and
technology transition.
Systems Analysis
A crucial part of portfolio management, systems
analysis includes competitive task selection, ongoing
refinement, and redirection as technical progress is
made in a program. Systems analyses are engineering
analyses that integrate the effects of specific scientific
and engineering disciplines, components, and assem-
blies, and their interactions in order to predict the per-
formance or otherwise explain the behavior of hard-
ware (system) and enable trade studies to be performed
on assumptions, boundary conditions, and other con-
straints. Systems analysis also leads to an awareness of
the system-level impacts of individual technologies
under development. The committee observed gaps in
system-level awareness and systems analysis capabil-
ity throughout the PRT program, from top to bottom.
Methods for risk assessment of various types were nei-
ther widely used nor well understood. Yet, pockets of
systems analysis were found within the program, typi-
cally in the areas of excellence. For example, the Ener-
getics project within the ECT program has effectively
used excellent-quality systems analysis for much of its
work to guide research efforts toward the critical high-
est-payoff technical challenges on the system level. It
is the committee's understanding that the Technology
Assessment Analysis (TAA) process within the ECT
program is being developed to address a portion of this
need; however, there was no clear indication that the
TAA, as structured for FY2003, could ever develop
into a true portfolio analysis tool set.
Systems analysis capability that covers a range of
fidelity from back-of-the-envelope to refined para-
metric excursions of specific point designs should be
employed. Awareness of system-level impacts should
be encouraged down to the level of individual projects
and researchers to ensure that research goals retain their
original desired relevance. Such analyses should vary
in complexity; in some cases, a simple calculation suf-
fices, but in others a more advanced state-of-the-art
analysis is needed.
During the course of the review and in response to
the committee' s interim letter report (NRC, 2003), the
PRT program has made several changes in the area of
systems analysis. The ECT program's TAA, although
reduced in funding by approximately one-half, is fo-
cusing its work on four mission-based pilot studies cho-
sen by the various enterprises within NASA. However,
much additional work is necessary to develop a perva-
sive tool set to analyze technology portfolios and sys-
tems issues. The TAA effort and other systems analy-
sis issues are discussed in further detail in Chapter 5.
The CICT program has recruited an individual who will
be responsible for program-level coordination of CICT
system analysis activities and specific impact assess-
meets (Tu and VanDalsem, 2003~. Other more specific
impact assessments and related efforts are also under
way. Because these efforts are new, the committee can-
not comment on their quality or success.
Finding: Gaps in the awareness of potential system-
level impacts of individual technologies and in the
use of systems analysis for research and portfolio
management were observed throughout the PRT
program. Further emphasis and strengthening are
necessary in this area.
Recommendation: Systems analysis should be
strengthened as a crucial part of the portfolio man-
agement and project selection process to support in-
vestment decisions in the technology areas needing
development. This process should recognize the pri-
orities NASA has set for its missions and the poten-
tial impact the research projects have on enabling
and enhancing those missions. The process should
also be applied to individual tasks and used by indi-
vidual researchers as a mechanism for ensuring re-
search goals retain their original desired relevance.
However, it should not be so rigid as to disallow ser-
endipity and ideas of opportunity.
OCR for page 15
OCR for page 16
OCR for page 17
OCR for page 18
OCR for page 19
Representative terms from entire chapter:
revolutionary technology
OVERALL ASSESSMENT
Benchmarking and Metrics
Benchmarking establishes quantitative goals or
expectations that serve as technical measures of suc-
cess. These objective goals are expressed at the disci-
pline, component, subsystem, and system levels, tied
together by systems analysis. Excellent projects and
tasks within the PRT program have always developed
methodologies and goals from meaningful technical
benchmarks and subjected their research progress to
external assessment using appropriate metrics. These
benchmarks were supported by analyses, where appro-
priate, and developed from basic scientific principles.
The Space Communications project within CICT is an
excellent example of how setting and using proper
metrics can enhance a research program. The project' s
tasks had clearly defined goals for even the most basic
research. Both the Advanced Measurement and Detec-
tion (AMD) element and the Energetics project within
ECT also exemplify this characteristic. Both have well-
defined goals and objectives that derive from the pro-
gram needs of the relevant mission within the associ-
ated NASA program office. For example, the various
investigators leading superconducting-transition-edge,
sensor-array research in AMD have taken the task's
benchmarks from the Constellation X scientific mea-
surement requirements defined by Code SO
Each program element and task should, in conjunc-
tion with element and program managers, establish
technical benchmarks that are supported by analyses
from basic principles. These metrics should be tem-
pered with realistic engineering considerations and
should be used to devise consistent, science-based re-
search methodologies. Used correctly, these metrics
can enable a useful assessment of long-term progress
and results in the tasks, elements, and projects where
they are applied.
Finding: Tasks within the PRT program that devel-
oped methodologies and goals from specific techni-
cal benchmarks produced excellent work.
Recommendation: Each project, element, and task
within the PRT program should establish technical
benchmarks to enable assessment of progress and
Murder information on this example is available online at
16
AN ASSESSMENT OF NASA 'S PIONEERING REVOLUTIONARY TECHNOLOGY PROGRAM
In response to the committee's interim letter re-
port, the CICT program has placed more emphasis on
having internal work peer reviewed by external experts
in the field (Tu and VanDalsem, 2003~. This is a posi-
tive step for the program; however, the procedure for
the reviews has not been evaluated by the committee
for effectiveness.
Finding: The PRT program makes little use of ex-
ternal peer review to select and evaluate the inter-
nal research program.
Recommendation: The PRT program should incor-
porate external peer review in all aspects of the pro-
gram, including selection of internal research tasks,
ongoing progress reviews and working groups, and
final assessment of results.
Finding: The committee observed uneven involve-
ment of researchers in publishing in peer-reviewed
publications (either in journals or in the proceed-
ings of peer-reviewed conferences).
Recommendation: NASA management should en-
courage peer-reviewed publication in landmark
journals and peer-reviewed conference proceedings.
It is important that NASA maintain competencies
in areas critical to its mission, whether inside NASA or
out. This does not imply that research in these areas be
exempt from competition, even for technologies where
NASA is the only customer. In many cases, NASA will
be the most appropriate place for such research because
of its unique capabilities, infrastructure, or superior
skills (e.g., radioactive power sources, autonomous ro-
bots). In such cases, NASA will be competitive. In
other cases, academia, research labs, or industry may
be better placed to pursue research in designated areas.
A systematic use of competitive processes and ex-
ternal reviews will ensure that the highest quality re-
search is performed. However, even where research is
done outside NASA, it is critical that NASA maintain a
subject matter expertise in the relevant areas in order to
effectively direct and interact with external researchers
and integrate their work within NASA.
Finding: Broader external participation in the PRT
program can enhance productivity, cooperative
teaming, and quality of research. World-class pro-
grams within PRT exhibit these qualities.
Recommendation: All PRT research projects
should be subject to competition. Internal and
external competition should be separate to avoid
conflicts of interest and ensure fairness and coop-
eration. Clearly, NASA must maintain internal
technical expertise to ensure that research products
are effectively transitioned and integrated.
Based on the premise that competition for re-
sources improves the quality and relevance of both in-
house and external research, the committee developed
the following peer assessment process as an example.
The intent of this example is not to add to the adminis-
trative burden but to improve the technical quality and
productivity of those programs that are already consid-
ered good but could be made better. In fact, this
process could even replace or augment some of the in-
ternal review currently used by the program. The com-
mittee anticipates a process by which interactions with
external peer reviewers will be welcomed by the tech-
nologists, providing valuable collaborative discussions.
The peer process also benefits the junior researcher by
exposing her or him to means by which established re-
search leaders set and maintain high standards and con-
struct pathways to achieve difficult goals. In addition,
the process should bring a broader perspective on trans-
ferring technology to flight programs and techniques
to research groups. The process will also provide valu-
able input from the external scientific and technologi-
cal community into how internal NASA research is
prioritized and chosen.
The first step of such a process requires that NASA
management set top-level goals and strategic objectives
that establish expectations and specific directions for
Space R&T consistent with NASA's Strategic Plan
(Goldin, 2000; O'Keefe, 2002), just as they do now.
That would be followed by an allocation of budget re-
sources between in-house and externally funded re-
search in categories that are broad enough to promote a
healthy competition in ideas, concepts, and approaches.
Following a broad announcement of opportunity, a
separate competitive selection process would be fol-
lowed for internally funded projects just as is done now
for externally funded research. Nationally recognized
technical experts from universities, industry, and other
government laboratories and NASA personnel in other
in-house organizations and enterprises would assess the
proposals and report to NASA management on matters
of technical quality and appropriateness of content
compared with that of related work in their own institu-
OVERALL ASSESSMENT
lions. Competing proposals would be rank ordered and
funded by rank down to the limit of available resources.
For example, if 20 percent of the available resources
were competed for in this manner each year and the
other 80 percent were allocated to the continuation of
multiyear grants, there would be sufficient continuity
from year to year to ensure stability, while promoting
the infusion of new ideas and talent. The pace of the
rolling competition would be guided by NASA man-
agement.
The committee does note that a one-size-fits-all
process may not be appropriate for the PRT program
due to the program's size and diversity of technology.
The process should be adaptable over the wide range of
programs and technical areas and include both basic
researchers and hands-on technology developers.
Stability and Continuity
Changes in priority, organization, and funding will
always occur and should be expected in a dynamic re-
search program. However, the PRT program has un-
dergone frequent and sometimes disruptive restructur-
ing and reorganization, often based merely on advocacy
hype. Some of these changes appeared to be a destruc-
tive force rather than a natural reallocation of resources
as a part of research progress and maturation. For ex-
ample, portions of the program have been managed by
five different enterprises within NASA during the past
10 years. A link can be made between the stability of a
project (or lack thereof) in this regard and the project' s
technical performance over a long time horizon. This is
especially so for the more challenging basic research
tasks, for which fundamental advances in science and
engineering are required and long time horizons are
necessary to adequately investigate and transition
ideas.
The committee recognizes that certain current pro-
gram time spans are imposed by the Office of Manage-
ment and Budget (OMB). However, the OMB con-
straints apply 5-year time horizons, whereas the PRT
program has experienced reorganization at 1- and 2-
year intervals. Even more significant, during the course
of this 12-month review process, the entire PRT port-
folio was renamed the Mission Science and Measure-
ments theme. Portions of the ECS program were in flux
throughout the entire course of the review. Moreover,
if current plans for the FY2005 ECT program are
implemented, the program will have undergone three
top-level organizational changes within the course of
17
this review. While the committee understands that
many of the research projects within these programs
will continue, this is yet another example of constant
churning in the program.
NASA should strive to redirect programs based on
sound technical issues and progress. It should avoid
organizational churning and stutter-step reprogram-
ming motivated by advocacy or external pressure.
NASA management and the technical team must to-
gether provide stability and continuity for the manage-
ment of inevitable change. A well-structured process is
needed for selecting and maturing technology through
development and transition to application.
Projects in the PRT program show why a process
for effectively managing change is needed. The Ad-
vanced Measurement and Detection (AMD) element
within ECT is exemplary in its well-structured process
for selecting and maturing technology through instru-
ment development and transition to application. More
detailed information on this process can be found in the
Annex to Chapter 5, on the ECT program. This process
has led to the successful integration of instruments in
NASA missions despite the management and organi-
zational change endured by the element. This element
within ECT and other programs of excellence within
NASA have on their own adopted management prac-
tices that can accommodate frequent reorganizations at
the top. They have achieved progress in spite of those
reorganizations, not because of them.
Finding: The PRT program components have un-
dergone frequent and sometimes disruptive restruc-
turing and reorganization.
Recommendation: To provide stability and conti-
nuity despite inevitable program changes, NASA
should further develop and utilize more structured
processes for selecting and developing technology
from basic research to application. Program redi-
rection should be based primarily on technical is-
sues and progress. Projects should be provided with
stable funding and assured stable organization to
the extent possible.
Research Portfolio Balance
The committee observed that the PRT program
tasks consisted of tasks apparently assembled from a
bottom-up selection and lacking top-down connection
to the NASA Strategic Plan (Goldin, 2000; O'Keefe,
18
AN ASSESSMENT OF NASA 'S PIONEERING REVOLUTIONARY TECHNOLOGY PROGRAM
2002~. Clearly, the connection between the top-down,
mission-driven technology needs of the NASA pro-
grams in the NASA mission codes (Codes M, S. U. and
Y) and the bottom-up technology planning need to be
better established. Top-level PRT and program goals
and objectives (Hanks, 2002) are well connected to the
NASA Strategic Plan; however, the program goals are
not generally well connected to the actual individual
tasks or even, in some cases, to missions. This is due,
in part, to both the restructuring of the program and the
apparent lack of acceptance of the NASA-wide plan by
researchers. This disconnect can be rectified by engag-
ing individual researchers in a more collaborative plan-
ning process. The Space Communications project in
CICT and the Advanced Measurement and Detection
element in ECT are two positive examples where the
top-down, bottom-up connection is evident.
Finding: The NASA strategic plan is not well con-
nected top to bottom.
Recommendation: NASA should use a more col-
laborative process in strategic planning and the ex-
ecution of goals in order to involve researchers, cus-
tomers, and managers in the strategic planning
process.
In an ideal collaborative planning process, tech-
nology development plans (including tasks, priorities,
and investment levels) are created and accepted by all
the stakeholders. Periodic reviews should be used to
assess progress and make appropriate project adjust-
ments. Designing, executing, funding, and assessing a
research portfolio as substantial as that of PRT must
weigh a number of factors to determine the balance of
projects and tasks that would best achieve NASA's
mission. There is no single best balance, and the defi-
nition of a tuned portfolio will change over time, but
once the portfolio is initially defined through a strate-
gic planning and competitive selection process that
balances need and opportunity, further adjustments
should be expected after assessing such factors as the
following:
An example of a potentially disruptive technology
in the PRT program is CICT's work in neural net flight
controls. The research has been demonstrated to work
very well in a simulator; however, it is so novel and
unusual that the regulatory processes to field it may be
problematic. The AMD work in radio frequency/
terahertz and focal planes for astrophysics and plan-
etary exploration will open up new mission possibili-
ties.
Determining an optimum balance among these fac-
tors is not possible until a well-defined method for de-
veloping a program architecture is in place. As a result,
the committee felt it inappropriate to suggest a balance.
However, the committee did feel it appropriate to com-
ment on the amount of revolutionary technology re-
search in the program. The committee recognizes that a
large portion of the PRT program appropriately con-
tains evolutionary technology. Only a few stretch, high-
risk research efforts were observed those that, if suc-
cessful, disrupt conventional thinking and open up new
approaches, missions, and systems. Although the pro-
gram is investing in some so-called revolutionary areas
(such as nanotechnology and quantum computing), the
committee notes that a research topic perceived as
emerging or revolutionary by the scientific and techni-
cal community does not necessarily mean that the re-
search itself is of excellent quality or great potential
relevance to NASA. Also, the committee noted that
some excellent research, very relevant to NASA mis-
sions, is more evolutionary than revolutionary and sup-
ports a core technical competency that is unique to
NASA capabilities and needs. For this reason, the com-
mittee urges NASA to select research projects on the
basis of the quality of the research and its relevance to
NASA, independent of whether it is perceived as revo-
lutionary. That said, the committee also believes that
the PRT portfolio should exhibit more tolerance for
taking on stretch goals (properly grounded in physics)
that could yield high-payoff results in areas where
NASA can have a unique impact.
Finding: Few efforts within the PRT program were
considered to be high-risk, high-payoff efforts. Most
of the work, much of it high in quality, was evolu-
Relative funding levels for the three programs, tionary.
CICT, ECS, and ECT.
The balance between fundamental scientific re-
search and engineering, user-driven research.
The proportion of evolutionary (low-risk) and
revolutionary (disruptive, high-risk) research.
Recommendation: The PRT program should en-
courage more stretch goals in revolutionary areas
that could yield high-payoff and mission-enabling
results.
OVERALL ASSESSMENT
Technology Transition
The committee observed that some useful technol-
ogy becomes caught between the end of PRT support
(at a low technology readiness level [TRL]) and the
start of user support (at a mid- to high TRL). Every
effort should be made to work with the user enterprises
of NASA and industry to prevent such breaks in fund-
ing. As successful research efforts mature, transition
funding should come jointly from PRT and the user
enterprises and industry. The committee notes that a
few projects within PRT have an effective process for
transitioning new technology to the successful produc-
tion of mission hardware. The JPL autonomous robot-
ics work in CICT and the AMD element in the ECT
program both transition technology successfully. For
AMD, an enduring, well-defined process exists that
allows a natural transition through m~-TRL instrument
development programs such as the Planetary Instru-
ment Definition and Development Program (PIDDP)
and the Instrument Incubator Program (IIP). Recent
examples include the m~croshutter array that is now
baselined for the future James Webb telescope and the
m~crothermopile array for the Mars Climate Sounder
instrument on the Mars Reconnaissance Orbiter. Cost-
sharing of transitional research is a goal of the ECT
program and is used quite frequently. This practice
should be continued and expanded beyond ECT.
Finding: Promising technology often fails in transi-
tion, when the PRT program concludes, often with
19
good reason, that it is mature enough for applica-
tion but before a mission organization has accepted
ownership.
Recommendation: Provisions for cost-sharing of
transitional research between the PRT program
and mission organizations at NASA and in industry
should be pursued as an explicit milestone in the
TRL maturation process.
REFERENCES
Goldin, Daniel.2000. National Aeronautics and Space Administration Stra-
tegic Plan 2000, September. Washington, D.C.: National Aeronautics
and Space Administration.
National Research Council (NRC). 2003. Interim Report of National Re-
search Council Review of NASA's Pioneering Revolutionary Technol-
ogy Program. Washington, D.C.: The National Academies Press. Avail-
able online at . Accessed
September 3, 2003.
BRIEFINGS
Brantley Hanks, "Pioneer Revolutionary Technologies: OAT Strategic Pro-
gram Area Overview," presentation to the committee and the panels on
June 11, 2002.
Sean O'Keefe, NASA Headquarters, "NASA Vision," briefing to Maxwell
School of Citizenship and Public Affairs on April 12, 2002. Available
online at . Ac-
cessed September 4, 2003.
Eugene Tu and Bill VanDalsem, NASA Ames Research Center, "CICT
Actions in Response to the NRC Review of NASA's Pioneering Revo-
lutionary Technology Program Interim Report, dated January 16,
2003," material provided to the committee on April 21, 2003.
