Committee to Review the NASA Institute for Advanced Concepts Statement of Task, Objective 2—Evaluate the method by which grantees were selected and recommend changes, if needed.
The NASA Institute of Advanced Concepts (NIAC) devoted considerable effort to the selection of grants in order to make the process fair and effective. Critical to the success of this selection process was whether the projects were chosen on solid grounds using appropriate selection procedures, whether the grantees had the credentials and capabilities to undertake the work, and whether the activities had breadth in their scope.
A general outline of the proposal selection process, based on peer review, is shown in Figure 2-1. The peer review process was created by the NIAC staff, with advice from the NIAC Science Council, whose members included accomplished technical leaders as well as researchers in emerging technical areas, in order to ensure a fair and objective process. The reviewers were asked to rank proposals with respect to the basic elements of the NIAC project philosophy:
Revolutionary and new concepts; not duplicative of concepts previously studied by NASA;
An inspiration for a great leap in performance or capabilities of aerospace endeavors, achievable within the NIAC time frame of 10-40 years in the future; and
Largely independent of existing technology or a unique combination of systems and technologies.
The selection process was also based on the specifics associated with each award phase. For Phase I awards, the reviewers analyzed each proposal with respect to the following specific questions:
How well have the benefits been qualified in the context of a future aeronautics and/or space mission appropriate to the NASA charter and responsibilities?
How well is the concept described in a system or architecture context?
Is the concept revolutionary rather than evolutionary? To what extent does the proposed activity suggest and explore creative and original concepts that may initiate a revolutionary paradigm change?
Is the concept substantiated with a description of applicable scientific and technical disciplines necessary for development?
How well conceived and organized is the study work plan, and does the team have appropriate key personnel and proven experience?
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 21
2 Grantee Selection Process Committee to Review the NASA Institute for Advanced Concepts Statement of Task, Objective 2⎯ Evaluate the method by which grantees were selected and recommend changes, if needed. The NASA Institute of Advanced Concepts (NIAC) devoted considerable effort to the selection of grants in order to make the process fair and effective. Critical to the success of this selection process was whether the projects were chosen on solid grounds using appropriate selection procedures, whether the grantees had the credentials and capabilities to undertake the work, and whether the activities had breadth in their scope. PEER REVIEW PROCESS A general outline of the proposal selection process, based on peer review, is shown in Figure 2-1. The peer review process was created by the NIAC staff, with advice from the NIAC Science Council, whose members included accomplished technical leaders as well as researchers in emerging technical areas, in order to ensure a fair and objective process. The reviewers were asked to rank proposals with respect to the basic elements of the NIAC project philosophy: 1. Revolutionary and new concepts; not duplicative of concepts previously studied by NASA; 2. An inspiration for a great leap in performance or capabilities of aerospace endeavors, achievable within the NIAC time frame of 10-40 years in the future; and 3. Largely independent of existing technology or a unique combination of systems and technologies. The selection process was also based on the specifics associated with each award phase. For Phase I awards, the reviewers analyzed each proposal with respect to the following specific questions: 1. How well have the benefits been qualified in the context of a future aeronautics and/or space mission appropriate to the NASA charter and responsibilities? 2. How well is the concept described in a system or architecture context? 3. Is the concept revolutionary rather than evolutionary? To what extent does the proposed activity suggest and explore creative and original concepts that may initiate a revolutionary paradigm change? 4. Is the concept substantiated with a description of applicable scientific and technical disciplines necessary for development? 5. How well conceived and organized is the study work plan, and does the team have appropriate key personnel and proven experience? 21
OCR for page 21
FIGURE 2-1 Selection process from receipt of proposals to notification of winners. SOURCE: Modified from NASA Institute for Advanced Concepts, 5th Annual Report (2002-2003) Atlanta, Ga., July 2003, p. 24. For Phase II awards, the reviewers analyzed each proposal with respect to the following specific questions: 1. Does the proposal continue the development of a revolutionary architecture or system in the context of a future NASA mission? Is the proposed work likely to provide a sound basis for NASA to consider the concept for a future mission or program? 2. Is the concept substantiated with a description of applicable scientific and technical disciplines necessary for development? 3. Has a pathway for development of a technology roadmap been adequately described? Are all of the appropriate enabling technologies identified? 4. Are the programmatic benefits and cost versus performance of the proposed concept adequately described and understood? Does the proposal show the relationship between the concept’s complexity and its benefits, cost, and performance? Reviewers were given forms to evaluate concepts using a numerical rating from 0 (worst) to 9 (best) for all these specific components, as well as written strengths and weaknesses for each of the components. In addition, the reviewers provided an overall opinion as to the viability of the proposal for NIAC funding. 22
OCR for page 21
BOX 2-1 Reasons for Rejection During Internal Review • The concept is evolutionary, rather than revolutionary. • The proposal is duplicative of concepts previously studied by NASA. • The proposal does not describe an architecture or system. • The concept was not placed into a NASA aeronautics and/or space mission context. • The concept is inadequately substantiated with a description of the scientific principles that underpin it. • The concept uses existing technology or a combination of systems and technologies without introducing a significantly unique or innovative concept. • The concept continues the development of technologies that by their very nature are narrowly focused on the development and performance of subsystems or components. • The concept incrementally extends the performance of an aerospace system or previously studied concept, or develops a new specialized instrument or high-performance material. • The concept emphasizes an incremental system development, technology demonstration, or other supporting development program that is closely linked to an existing NASA program or mission and would be a near-term progression of the existing program or mission. • The concept is based solely on technically unsubstantiated science fiction. • A program, workshop plan, or literature search is proposed as a way to solve a problem or attain a goal with no specifically described architecture or system. • The proposal solely describes research experiments on fundamental processes or theoretical derivations with no connection to an overall architecture or system. Virtually all the reviews followed the process that was outlined in NIAC’s first annual report: A group of forty-four reviewers took part in the peer review of the 119 proposals submitted in response to CP98-01. These peer reviewers represented a cross-section of senior research executives in private industry, senior research faculty in universities, specialized researchers in both industry and universities, and aerospace consultants. Peer reviewers for CP98-02 were drawn from a similar community of scientists and engineers. For CP98-01 each proposal received at least three peer reviews. Each reviewer was asked to evaluate the proposal according to the criteria stated in the Call-for-Proposals. Forms were created to help guide the reviewer through the process of assigning a numerical ranking and providing written comments. Each reviewer was required to sign a non-disclosure and no-conflict-of-interest agreement. A small monetary compensation was offered to each reviewer. Depending on the capabilities of each reviewer, the proposals and all required forms were transmitted to the reviewer over the Internet, by diskette or by paper copy. Each reviewer was given approximately thirty days to review the proposals and return the completed evaluation forms. The ANSER Corporation provided valuable assistance to the peer review process through a search of its archives, knowledge bases and additional resources. These information databases were used to provide additional background on prior and ongoing advanced concept research efforts sponsored by NASA and non-NASA sources.1 The review process began with an initial internal evaluation of proposals for competitiveness. This internal review was conducted by the NIAC senior technical staff (director, associate director, and senior science advisor), selected Universities Space Research Association (USRA) technical staff, and consultants, as needed. See Box 2-1 for a list of reasons for rejection during internal review. To help ensure that a proposed concept did not duplicate previously studied concepts, NIAC accessed the NASA Technology Inventory Database and other public NASA databases to search for related NASA-funded projects. 1 NASA Institute of Advanced Concepts, Annual Report (1st; 1998-1999), Atlanta Ga., 1999, p. 10. 23
OCR for page 21
For the proposals that were selected for full review, external reviewers were assigned by NIAC senior technical staff based on the known qualifications of the reviewers in the database. Additional reviewers were recruited, as needed. After peer review, the proposals were generally ranked by a review panel into three categories based on the submitted ad hoc reviews: “reject,” “support if funds are available,” and “consider with a fourth review.” The third option was used when the three reviewers had divergent opinions. The proposals were then ranked from top to bottom and a cut line was placed at the point where the funds were exhausted. Proposals on both sides of the margin were discussed again until a final ranking was established. With this process completed, the documentation was taken to NASA Headquarters to obtain concurrence. The NIAC director was required to present the research grant selections to the NASA chief technologist and representatives of the NASA strategic enterprises before any awards were announced. Technical concurrence by NASA ensured consistency with NASA’s charter, strategy, and budget limits before any grants were announced or issued. Overall, the review process was efficient, effective, and in keeping with practices used by other federal funding agencies. Finding 2.1: The process for selecting NIAC grantees was well documented, was disciplined, met the charter of NIAC, and was generally commensurate with the practices of other federal funding agencies. Just as important as the award selection process was the selection of reviewers. NIAC strove to incorporate a wide spectrum of reviewers across education level, technical specialty, and organizational demographics. According to NIAC’s 2004 annual report: The NIAC leadership has developed an efficient and proven method for identifying and selecting the most qualified and appropriate external review panel members to evaluate proposals submitted to the Institute. Over the last five years, NIAC has continuously recruited experts across a broad cross-section of technical expertise and a total of one hundred seventy eight individuals have been used, thus far, for peer review. In order to ensure a continuous refreshment of the available expertise representing newly emerging technologies within the scientific community, the NIAC leadership continually recruits additional reviewers for each new peer review cycle. NIAC peer reviewers recruited by USRA include senior research executives in private industry, senior research faculty in universities, specialized researchers in both industry and universities, and aerospace consultants. For identifying prospective peer reviewers, several resources are used in combination. Because Phase I proposals are necessarily less technically specific and will be judged more for the validity of the concept itself, evaluation of these proposals requires experts regarded as “big picture” people (i.e., individuals whose careers have exposed them to a variety of technical disciplines and an understanding of complex systems employing many different technologies). An example of this type of individual might be a vice president of a major aerospace corporation. Phase II proposals, however, which offer far more technical detail, will typically require a more specific group of evaluators. One significant resource that the Institute employed successfully was the personal knowledge of the NIAC Director, Associate Director, and Senior Science Advisor of many qualified experts in a wide variety of fields related to NIAC. Some of these experts had a prior association with NIAC, some served previously as NIAC reviewers, and some participated in Grand Challenges/Visions workshops. Others may have been suggested by NIAC Science Council members. An additional resource of qualified peer reviewers could be found in the authors of publications cited in the proposals to be reviewed. These researchers often represented the forefront of knowledge in a specific emerging technology directly relevant to the proposed study.2 2 NASA Institute for Advanced Concepts, Annual Report (6th; 2003-2004), Atlanta, Ga., 2004, p. 21. 24
OCR for page 21
BOX 2-2 Demographics for NIAC Peer Reviewers in 2003 Technical Specialist1 Affiliation University 71 Science For-profit industry 30 Physics 47 Not-for-profit industry 21 Biological sciences 29 Government laboratory 19 Chemistry 5 Consultant 22 Physical sciences 22 Engineering Level of Experience Propulsion 39 Less than 10 years 2 Power 26 10 to 20 years 31 Design 18 20 to 30 years 40 Systems analysis 52 More than 30 years 90 Computing/information technology 6 Biotechnology 5 Highest Academic Achievement Materials/structures 9 B.S. 19 M.S. 24 Ph.D./M.D. 130 1 These numbers do not add up to 163 because many of the reviewers preferred to describe their area of technical specialty and experience in a more multidisciplinary manner. All of these mechanisms for finding reviewers were commensurate with general practices at the National Science Foundation and other federal funding agencies. The total number of reviewers as of 2003 was 163. This number grew slightly for the subsequent years of NIAC activity. While information after 2003 is not available, it is likely that the demographics of the additional reviewers were similar to those presented in Box 2-2. As noted, there was a good distribution of reviewers from university, industry, laboratories, and other categories, as well as a good distribution of technical specialties. The level of experience of the reviewers was heavily weighted toward persons that have been in the field for a few decades⎯excluding, to a very large extent, beginning investigators. There was no evidence of an effort to achieve a balance of gender and ethnicity in the pool of reviewers, and statistics were thus not collected by NIAC. DIVERSITY OF GRANTEES Over the 9 years of NIAC’s existence, a total of 1,066 Phase I proposals and 129 Phase II proposals were received at NIAC and evaluated for possible funding. The total number of evaluated submissions and awards for the entire life of NIAC can be visualized in Figure 2-2. The majority of submissions were from small businesses. Especially in the early years of NIAC, a significant number of principal investigators submitted multiple proposals, even within the same solicitation. There may have been a lack of dissemination of the solicitations within a broader community. This is understandable in the early years of the program, when the news of this new source of funding was not widely available. From what can be ascertained, dissemination of the NIAC solicitations was accomplished via a number of different paths, possibly enough to counteract the small number of submitters in the early years of NIAC. Notices were sent to the NIAC email distribution list, generated from responses by individuals who signed up on the NIAC Web site to receive the call; announcements on professional society Web 25
OCR for page 21
sites or newsletters (e.g., American Institute for Aeronautics and Astronautics, American Astronautical Society, the American Astronomical Society and the American Society of Gravitational and Space Biology); announcements on the USRA and NIAC Web sites; Web links from NASA mission directorate Web pages; Web link from the NASA coordinator’s Web page; announcements to a distribution list for historically black colleges and universities, minority institutions, and small disadvantaged businesses, provided by NASA; distribution of announcements to an Earth-sciences list provided by NASA Goddard Space Flight Center; and announcements distributed at technical society meetings. Distribution of the NIAC Student Fellows Prize (NSFP) announcement also occurred through the Space Grant College Directors and the USRA Council of Institutions. From the 1,066 evaluated submissions, a total of 126 Phase I grants were awarded by NIAC, as listed in Appendix E. This represents a success rate of 11.8 percent, which is comparable to other federal funding agencies. Figure 2-2 illustrates the total number of awards by category for the life of NIAC. Phase II solicitations were limited to NIAC fellows (investigators that had received a Phase I award). A total of 129 proposals were submitted during the life of NIAC. Following the same trend as in the Phase I submissions and awards, the majority of grants were awarded to small businesses and universities (Figure 2-2). From the 129 submissions, a total of 42 Phase II awards were approved by NIAC. This represents a success rate of 32.6 percent. Figure 2-2 illustrates the total number of Phase II awards by category for the life of NIAC. Finding 2.2: The process for selecting NIAC grantees led to a variety of involved organizations, principally from universities and small businesses. In summary, the committee found the methods by which NIAC grantees were selected to be generally effective, efficient, and appropriate to the objectives of the NIAC program, including the advertisement of the solicitations, the selection of reviewers, the screening of proposals, and the use of electronic processes to streamline the overall evaluation. 26
OCR for page 21
Phase I Reviewed Submissions Phase I Awards Phase II Reviewed Submissions Phase II Awards FIGURE 2-2 Reviewed submissions and awards over the life of NIAC. 27
OCR for page 21