Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
2 Overall Assessment of the Pioneering Revolutionary Technology Program The observations and recommendations presented here are overarching issues of concern throughout the PRT program. Detailed assessments of the individual PRT programs (CICT, ECS, and ECT) can be found in Chapters 3 through 5. OVERALL ASSESSMENT The committee' s overall assessment of the research within PRT was made based on information from the individual assessments of the three supporting panels. Tasks judged by the committee and panels to be world- class met the following criteria: (1) they gave evidence of productivity (publications, software, presentations, patents, mission-accepted technology); (2) they exhib- ited strong linkage at the task level to actual flight projects, flight engineers, or science customers; (3) they possessed connectivity with other research communities external to NASA; and (4) they were rec- ognized by external peers as an authority in the subject matter. In some cases, excellence was also observed when basic research, facilities, systems analysis, flight integration, and test and evaluation were vertically in- tegrated or when programs had achieved success over a period of 10 to 15 years and continue to do so. Exem- plifying this long-term excellence were the Energetics project and the Advanced Measurement and Detection (AMD) element. 13 The panels and committee were very careful to use the descriptor "world-class" only when a project or task clearly met the set of criteria listed above and was clearly a leader in the field. The word "revolutionary" was used only in very specific instances, when it was perceived the work promised to provide leaps in capa- bility or technology over current methods. While there are some important concerns about management practices within the PRT portfolio, the committee found that most of the Pioneering Revolu- tionary Technology (PRT) research consisted of good work that is important to the future of NASA and the nation. Ten percent of the individual research tasks were judged as work of the highest quality, represent- ing truly world-class endeavors. The committee and panels recommended that another 10 percent of the program's research tasks be discontinued or transitioned to mission applications. Tasks marked for transition are typically of excellent quality and involve successful work ready to be funded by a NASA mis- sion or external partners. Tasks marked for discontinu- ation were identified primarily based on a judgment about the relative quality of the work or its value to NASA and its alignment (or lack thereof) with PRT program goals. With 80 percent of the program being of good quality, but not world-class, there is an oppor- tunity for improving PRT program research by focus- ing more attention on several common issues, includ-
14 AN ASSESSMENT OF NASA 'S PIONEERING REVOLUTIONARY TECHNOLOGY PROGRAM ing the need for research to be more results-oriented, pervasive use of systems analysis, peer review, and in- creasing collaboration between outside experts and the program. Finding: The committee judged approximately 90 percent of the PRT program to be good work, im- portant to NASA and the nation. Of this 90 percent, 10 percent was deemed to be world-class. COMMON THEMES The committee noted six themes cutting across the entire PRT program where special attention would strengthen the program: systems analysis; benchmark- ing and metrics; external peer review and competition; stability and continuity; research portfolio balance; and technology transition. Systems Analysis A crucial part of portfolio management, systems analysis includes competitive task selection, ongoing refinement, and redirection as technical progress is made in a program. Systems analyses are engineering analyses that integrate the effects of specific scientific and engineering disciplines, components, and assem- blies, and their interactions in order to predict the per- formance or otherwise explain the behavior of hard- ware (system) and enable trade studies to be performed on assumptions, boundary conditions, and other con- straints. Systems analysis also leads to an awareness of the system-level impacts of individual technologies under development. The committee observed gaps in system-level awareness and systems analysis capabil- ity throughout the PRT program, from top to bottom. Methods for risk assessment of various types were nei- ther widely used nor well understood. Yet, pockets of systems analysis were found within the program, typi- cally in the areas of excellence. For example, the Ener- getics project within the ECT program has effectively used excellent-quality systems analysis for much of its work to guide research efforts toward the critical high- est-payoff technical challenges on the system level. It is the committee's understanding that the Technology Assessment Analysis (TAA) process within the ECT program is being developed to address a portion of this need; however, there was no clear indication that the TAA, as structured for FY2003, could ever develop into a true portfolio analysis tool set. Systems analysis capability that covers a range of fidelity from back-of-the-envelope to refined para- metric excursions of specific point designs should be employed. Awareness of system-level impacts should be encouraged down to the level of individual projects and researchers to ensure that research goals retain their original desired relevance. Such analyses should vary in complexity; in some cases, a simple calculation suf- fices, but in others a more advanced state-of-the-art analysis is needed. During the course of the review and in response to the committee' s interim letter report (NRC, 2003), the PRT program has made several changes in the area of systems analysis. The ECT program's TAA, although reduced in funding by approximately one-half, is fo- cusing its work on four mission-based pilot studies cho- sen by the various enterprises within NASA. However, much additional work is necessary to develop a perva- sive tool set to analyze technology portfolios and sys- tems issues. The TAA effort and other systems analy- sis issues are discussed in further detail in Chapter 5. The CICT program has recruited an individual who will be responsible for program-level coordination of CICT system analysis activities and specific impact assess- meets (Tu and VanDalsem, 2003~. Other more specific impact assessments and related efforts are also under way. Because these efforts are new, the committee can- not comment on their quality or success. Finding: Gaps in the awareness of potential system- level impacts of individual technologies and in the use of systems analysis for research and portfolio management were observed throughout the PRT program. Further emphasis and strengthening are necessary in this area. Recommendation: Systems analysis should be strengthened as a crucial part of the portfolio man- agement and project selection process to support in- vestment decisions in the technology areas needing development. This process should recognize the pri- orities NASA has set for its missions and the poten- tial impact the research projects have on enabling and enhancing those missions. The process should also be applied to individual tasks and used by indi- vidual researchers as a mechanism for ensuring re- search goals retain their original desired relevance. However, it should not be so rigid as to disallow ser- endipity and ideas of opportunity.
OVERALL ASSESSMENT Benchmarking and Metrics Benchmarking establishes quantitative goals or expectations that serve as technical measures of suc- cess. These objective goals are expressed at the disci- pline, component, subsystem, and system levels, tied together by systems analysis. Excellent projects and tasks within the PRT program have always developed methodologies and goals from meaningful technical benchmarks and subjected their research progress to external assessment using appropriate metrics. These benchmarks were supported by analyses, where appro- priate, and developed from basic scientific principles. The Space Communications project within CICT is an excellent example of how setting and using proper metrics can enhance a research program. The project' s tasks had clearly defined goals for even the most basic research. Both the Advanced Measurement and Detec- tion (AMD) element and the Energetics project within ECT also exemplify this characteristic. Both have well- defined goals and objectives that derive from the pro- gram needs of the relevant mission within the associ- ated NASA program office. For example, the various investigators leading superconducting-transition-edge, sensor-array research in AMD have taken the task's benchmarks from the Constellation X scientific mea- surement requirements defined by Code SO Each program element and task should, in conjunc- tion with element and program managers, establish technical benchmarks that are supported by analyses from basic principles. These metrics should be tem- pered with realistic engineering considerations and should be used to devise consistent, science-based re- search methodologies. Used correctly, these metrics can enable a useful assessment of long-term progress and results in the tasks, elements, and projects where they are applied. Finding: Tasks within the PRT program that devel- oped methodologies and goals from specific techni- cal benchmarks produced excellent work. Recommendation: Each project, element, and task within the PRT program should establish technical benchmarks to enable assessment of progress and Murder information on this example is available online at <http:// constellation.gsfc.nasa.gov/docs/technology/sxt.html>, accessed August 8, 2003. 15 results. These benchmarks should include measur- able, objective targets for research and should be developed in the context of the research's applica- tion. External Peer Review and Competition Interaction with external peers comes in a number of different forms, all of which should be encouraged throughout the research life cycle. Before research is initiated, external peer reviews are used fairly effec- tively in the competitively selected external portion of the PRT program but only sparingly, if at all, in com- petitively selecting in-house research projects. Further- more, there is limited involvement of external peers in evaluating the technical quality of ongoing in-house work to decide what should continue, be redirected, or be terminated. Finally, as mentioned in the subsequent chapters on individual programs, the encouragement of publication in peer-reviewed technical journals is inconsistent. As observed by the panels, there is a clear correlation between excellent-quality work and tan- gible results presented in peer-reviewed publications and manifested in deliverable flight hardware and soft- ware. For example, in the Resilient Materials Struc- tures (RMS) element within ECT, about 80 percent of the publications are from two of the nine tasks. Both tasks were judged by the panelists to be of excellent quality. The PRT program should institutionalize an exter- nal peer review process in all aspects of the research and technology enterprise: task selection (including the in-house portion of the program), ongoing progress re- views, and final assessment of results. This peer re- view process would Increase the quality of program planning pro- cesses, Increase communication across groups within NASA, Provide another means of recognizing and re- warding research talent in NASA, Increase communication with researchers out- side the agency, and Reduce unintentional overlaps of research with ongoing academic and commercial research. It is important for the credibility and success of such a review that an appropriate number of nonadvo-cate re- views and reviewers be used.
16 AN ASSESSMENT OF NASA 'S PIONEERING REVOLUTIONARY TECHNOLOGY PROGRAM In response to the committee's interim letter re- port, the CICT program has placed more emphasis on having internal work peer reviewed by external experts in the field (Tu and VanDalsem, 2003~. This is a posi- tive step for the program; however, the procedure for the reviews has not been evaluated by the committee for effectiveness. Finding: The PRT program makes little use of ex- ternal peer review to select and evaluate the inter- nal research program. Recommendation: The PRT program should incor- porate external peer review in all aspects of the pro- gram, including selection of internal research tasks, ongoing progress reviews and working groups, and final assessment of results. Finding: The committee observed uneven involve- ment of researchers in publishing in peer-reviewed publications (either in journals or in the proceed- ings of peer-reviewed conferences). Recommendation: NASA management should en- courage peer-reviewed publication in landmark journals and peer-reviewed conference proceedings. It is important that NASA maintain competencies in areas critical to its mission, whether inside NASA or out. This does not imply that research in these areas be exempt from competition, even for technologies where NASA is the only customer. In many cases, NASA will be the most appropriate place for such research because of its unique capabilities, infrastructure, or superior skills (e.g., radioactive power sources, autonomous ro- bots). In such cases, NASA will be competitive. In other cases, academia, research labs, or industry may be better placed to pursue research in designated areas. A systematic use of competitive processes and ex- ternal reviews will ensure that the highest quality re- search is performed. However, even where research is done outside NASA, it is critical that NASA maintain a subject matter expertise in the relevant areas in order to effectively direct and interact with external researchers and integrate their work within NASA. Finding: Broader external participation in the PRT program can enhance productivity, cooperative teaming, and quality of research. World-class pro- grams within PRT exhibit these qualities. Recommendation: All PRT research projects should be subject to competition. Internal and external competition should be separate to avoid conflicts of interest and ensure fairness and coop- eration. Clearly, NASA must maintain internal technical expertise to ensure that research products are effectively transitioned and integrated. Based on the premise that competition for re- sources improves the quality and relevance of both in- house and external research, the committee developed the following peer assessment process as an example. The intent of this example is not to add to the adminis- trative burden but to improve the technical quality and productivity of those programs that are already consid- ered good but could be made better. In fact, this process could even replace or augment some of the in- ternal review currently used by the program. The com- mittee anticipates a process by which interactions with external peer reviewers will be welcomed by the tech- nologists, providing valuable collaborative discussions. The peer process also benefits the junior researcher by exposing her or him to means by which established re- search leaders set and maintain high standards and con- struct pathways to achieve difficult goals. In addition, the process should bring a broader perspective on trans- ferring technology to flight programs and techniques to research groups. The process will also provide valu- able input from the external scientific and technologi- cal community into how internal NASA research is prioritized and chosen. The first step of such a process requires that NASA management set top-level goals and strategic objectives that establish expectations and specific directions for Space R&T consistent with NASA's Strategic Plan (Goldin, 2000; O'Keefe, 2002), just as they do now. That would be followed by an allocation of budget re- sources between in-house and externally funded re- search in categories that are broad enough to promote a healthy competition in ideas, concepts, and approaches. Following a broad announcement of opportunity, a separate competitive selection process would be fol- lowed for internally funded projects just as is done now for externally funded research. Nationally recognized technical experts from universities, industry, and other government laboratories and NASA personnel in other in-house organizations and enterprises would assess the proposals and report to NASA management on matters of technical quality and appropriateness of content compared with that of related work in their own institu-
OVERALL ASSESSMENT lions. Competing proposals would be rank ordered and funded by rank down to the limit of available resources. For example, if 20 percent of the available resources were competed for in this manner each year and the other 80 percent were allocated to the continuation of multiyear grants, there would be sufficient continuity from year to year to ensure stability, while promoting the infusion of new ideas and talent. The pace of the rolling competition would be guided by NASA man- agement. The committee does note that a one-size-fits-all process may not be appropriate for the PRT program due to the program's size and diversity of technology. The process should be adaptable over the wide range of programs and technical areas and include both basic researchers and hands-on technology developers. Stability and Continuity Changes in priority, organization, and funding will always occur and should be expected in a dynamic re- search program. However, the PRT program has un- dergone frequent and sometimes disruptive restructur- ing and reorganization, often based merely on advocacy hype. Some of these changes appeared to be a destruc- tive force rather than a natural reallocation of resources as a part of research progress and maturation. For ex- ample, portions of the program have been managed by five different enterprises within NASA during the past 10 years. A link can be made between the stability of a project (or lack thereof) in this regard and the project' s technical performance over a long time horizon. This is especially so for the more challenging basic research tasks, for which fundamental advances in science and engineering are required and long time horizons are necessary to adequately investigate and transition ideas. The committee recognizes that certain current pro- gram time spans are imposed by the Office of Manage- ment and Budget (OMB). However, the OMB con- straints apply 5-year time horizons, whereas the PRT program has experienced reorganization at 1- and 2- year intervals. Even more significant, during the course of this 12-month review process, the entire PRT port- folio was renamed the Mission Science and Measure- ments theme. Portions of the ECS program were in flux throughout the entire course of the review. Moreover, if current plans for the FY2005 ECT program are implemented, the program will have undergone three top-level organizational changes within the course of 17 this review. While the committee understands that many of the research projects within these programs will continue, this is yet another example of constant churning in the program. NASA should strive to redirect programs based on sound technical issues and progress. It should avoid organizational churning and stutter-step reprogram- ming motivated by advocacy or external pressure. NASA management and the technical team must to- gether provide stability and continuity for the manage- ment of inevitable change. A well-structured process is needed for selecting and maturing technology through development and transition to application. Projects in the PRT program show why a process for effectively managing change is needed. The Ad- vanced Measurement and Detection (AMD) element within ECT is exemplary in its well-structured process for selecting and maturing technology through instru- ment development and transition to application. More detailed information on this process can be found in the Annex to Chapter 5, on the ECT program. This process has led to the successful integration of instruments in NASA missions despite the management and organi- zational change endured by the element. This element within ECT and other programs of excellence within NASA have on their own adopted management prac- tices that can accommodate frequent reorganizations at the top. They have achieved progress in spite of those reorganizations, not because of them. Finding: The PRT program components have un- dergone frequent and sometimes disruptive restruc- turing and reorganization. Recommendation: To provide stability and conti- nuity despite inevitable program changes, NASA should further develop and utilize more structured processes for selecting and developing technology from basic research to application. Program redi- rection should be based primarily on technical is- sues and progress. Projects should be provided with stable funding and assured stable organization to the extent possible. Research Portfolio Balance The committee observed that the PRT program tasks consisted of tasks apparently assembled from a bottom-up selection and lacking top-down connection to the NASA Strategic Plan (Goldin, 2000; O'Keefe,
18 AN ASSESSMENT OF NASA 'S PIONEERING REVOLUTIONARY TECHNOLOGY PROGRAM 2002~. Clearly, the connection between the top-down, mission-driven technology needs of the NASA pro- grams in the NASA mission codes (Codes M, S. U. and Y) and the bottom-up technology planning need to be better established. Top-level PRT and program goals and objectives (Hanks, 2002) are well connected to the NASA Strategic Plan; however, the program goals are not generally well connected to the actual individual tasks or even, in some cases, to missions. This is due, in part, to both the restructuring of the program and the apparent lack of acceptance of the NASA-wide plan by researchers. This disconnect can be rectified by engag- ing individual researchers in a more collaborative plan- ning process. The Space Communications project in CICT and the Advanced Measurement and Detection element in ECT are two positive examples where the top-down, bottom-up connection is evident. Finding: The NASA strategic plan is not well con- nected top to bottom. Recommendation: NASA should use a more col- laborative process in strategic planning and the ex- ecution of goals in order to involve researchers, cus- tomers, and managers in the strategic planning process. In an ideal collaborative planning process, tech- nology development plans (including tasks, priorities, and investment levels) are created and accepted by all the stakeholders. Periodic reviews should be used to assess progress and make appropriate project adjust- ments. Designing, executing, funding, and assessing a research portfolio as substantial as that of PRT must weigh a number of factors to determine the balance of projects and tasks that would best achieve NASA's mission. There is no single best balance, and the defi- nition of a tuned portfolio will change over time, but once the portfolio is initially defined through a strate- gic planning and competitive selection process that balances need and opportunity, further adjustments should be expected after assessing such factors as the following: An example of a potentially disruptive technology in the PRT program is CICT's work in neural net flight controls. The research has been demonstrated to work very well in a simulator; however, it is so novel and unusual that the regulatory processes to field it may be problematic. The AMD work in radio frequency/ terahertz and focal planes for astrophysics and plan- etary exploration will open up new mission possibili- ties. Determining an optimum balance among these fac- tors is not possible until a well-defined method for de- veloping a program architecture is in place. As a result, the committee felt it inappropriate to suggest a balance. However, the committee did feel it appropriate to com- ment on the amount of revolutionary technology re- search in the program. The committee recognizes that a large portion of the PRT program appropriately con- tains evolutionary technology. Only a few stretch, high- risk research efforts were observed those that, if suc- cessful, disrupt conventional thinking and open up new approaches, missions, and systems. Although the pro- gram is investing in some so-called revolutionary areas (such as nanotechnology and quantum computing), the committee notes that a research topic perceived as emerging or revolutionary by the scientific and techni- cal community does not necessarily mean that the re- search itself is of excellent quality or great potential relevance to NASA. Also, the committee noted that some excellent research, very relevant to NASA mis- sions, is more evolutionary than revolutionary and sup- ports a core technical competency that is unique to NASA capabilities and needs. For this reason, the com- mittee urges NASA to select research projects on the basis of the quality of the research and its relevance to NASA, independent of whether it is perceived as revo- lutionary. That said, the committee also believes that the PRT portfolio should exhibit more tolerance for taking on stretch goals (properly grounded in physics) that could yield high-payoff results in areas where NASA can have a unique impact. Finding: Few efforts within the PRT program were considered to be high-risk, high-payoff efforts. Most of the work, much of it high in quality, was evolu- Relative funding levels for the three programs, tionary. CICT, ECS, and ECT. The balance between fundamental scientific re- search and engineering, user-driven research. The proportion of evolutionary (low-risk) and revolutionary (disruptive, high-risk) research. Recommendation: The PRT program should en- courage more stretch goals in revolutionary areas that could yield high-payoff and mission-enabling results.
OVERALL ASSESSMENT Technology Transition The committee observed that some useful technol- ogy becomes caught between the end of PRT support (at a low technology readiness level [TRL]) and the start of user support (at a mid- to high TRL). Every effort should be made to work with the user enterprises of NASA and industry to prevent such breaks in fund- ing. As successful research efforts mature, transition funding should come jointly from PRT and the user enterprises and industry. The committee notes that a few projects within PRT have an effective process for transitioning new technology to the successful produc- tion of mission hardware. The JPL autonomous robot- ics work in CICT and the AMD element in the ECT program both transition technology successfully. For AMD, an enduring, well-defined process exists that allows a natural transition through m~-TRL instrument development programs such as the Planetary Instru- ment Definition and Development Program (PIDDP) and the Instrument Incubator Program (IIP). Recent examples include the m~croshutter array that is now baselined for the future James Webb telescope and the m~crothermopile array for the Mars Climate Sounder instrument on the Mars Reconnaissance Orbiter. Cost- sharing of transitional research is a goal of the ECT program and is used quite frequently. This practice should be continued and expanded beyond ECT. Finding: Promising technology often fails in transi- tion, when the PRT program concludes, often with 19 good reason, that it is mature enough for applica- tion but before a mission organization has accepted ownership. Recommendation: Provisions for cost-sharing of transitional research between the PRT program and mission organizations at NASA and in industry should be pursued as an explicit milestone in the TRL maturation process. REFERENCES Goldin, Daniel.2000. National Aeronautics and Space Administration Stra- tegic Plan 2000, September. Washington, D.C.: National Aeronautics and Space Administration. National Research Council (NRC). 2003. Interim Report of National Re- search Council Review of NASA's Pioneering Revolutionary Technol- ogy Program. Washington, D.C.: The National Academies Press. Avail- able online at <http://www.nap.edu/catalog/10605.html>. Accessed September 3, 2003. BRIEFINGS Brantley Hanks, "Pioneer Revolutionary Technologies: OAT Strategic Pro- gram Area Overview," presentation to the committee and the panels on June 11, 2002. Sean O'Keefe, NASA Headquarters, "NASA Vision," briefing to Maxwell School of Citizenship and Public Affairs on April 12, 2002. Available online at <http ://www.gsfc.nasa.gov/indepth/nasavision.html>. Ac- cessed September 4, 2003. Eugene Tu and Bill VanDalsem, NASA Ames Research Center, "CICT Actions in Response to the NRC Review of NASA's Pioneering Revo- lutionary Technology Program Interim Report, dated January 16, 2003," material provided to the committee on April 21, 2003.
