Major Award Decisionmaking at the National Science Foundation (1994) / Chapter Skim
Currently Skimming:
3 AWARDING MAJOR PROJECTS: CRITERIA AND REVIEW PROCEDURES
3 AWARDING MAJOR PROJECTS: CRITERIA AND REVIEW PROCEDURES
Pages 61-88
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 61... ...
than is necessary for small research grants. Major projects: · represent a substantial investment of NSF resources in a research area, with greater political costs and financial liabilities in the case of failure; · are more complex than traditional individual investigator or small-group research projects and involve more varied criteria, some less exclusively scientific and technical than those of individual investigator proposals; and · promise a substantial impact not only on their research community but also on the institutions and localities in which they are located.
|
From page 62... ...
As a result of the committee's report, the NSF director adopted the term "merit review" to emphasize the greater role of expanded criteria, especially for "center-based activities, research groups, and shared facilities" (NSF, 1986b:D-2~. NSF's current policy is that all formal proposals for grant funding are subject to peer review by appropriate experts external to the agency, with minor exceptions listed in a policy document approved by NSB (e.g., small travel grants, the Small Grants for Exploratory Research program)
|
From page 63... ...
Current Review Criteria According to the NSF Proposal and Award Manual (NSF, 1993c:~-3) , it is agency policy to give every research proposal "proper consideration in accordance with established criteria approved by the NSB.
|
From page 64... ...
Proposal and Award Manual, is an addition to the criteria adopted by the NSB in 1981. The criteria listed on the back of the evaluation form sent to external reviewers (NSF Form 1)
|
From page 65... ...
Criterion 3 also relates to major goal oriented activities that the Foundation carries out such as those directed at improving the knowledge base underlying science and technology policy, furthering international cooperation in science and engineering, and addressing areas of national need. Criterion 4, effect on the infrastructure of science and engineering, permits the evaTuation of research proposals in terms of their potential for improving the scientific and engineering enterprise and its educational activities in ways other than those encompassed by the first three criteria.
|
From page 66... ...
They are supposed to be listed in program announcements and solicitations; specified in the cover letters sent to peer reviewers; and used by site visitors, panels, and NSF staff in summarizing the basis for their recommendations. In addition, NSF still has "special guidelines for organization and research administration" for large projects, which were based on the 1974 criteria for creating or changing institutional structures: a.
|
From page 67... ...
That statement reaffirmed NSF's practice of managing its basic research program "in such a way as to permit the development of science along lines dictated by the internal needs of science itself" (NSB, 1967:31. To achieve this goal, NSF looked to fund unsolicited proposals from university-based researchers who met certain criteria: promise of significant scientific results; past record of performance of the investigators; potential scientific impact of the proposed work; degree of novelty, originality, or uniqueness involved; educational value of the proposed research in terms of effects on students; and relevance of the proposed work to potential applications.
|
From page 68... ...
In 1974, NSB revised and elaborated the selection criteria it had developed for individual investigator proposals and made them applicable to all proposals, from small research projects to large-scale national facilities. The criteria were grouped in four categories relating to competent performance, intrinsic scientific merit, utility or relevance, and future scientific potential of the nation.
|
From page 69... ...
The criteria may include, for example, managerial capacity, technology transfer, humar1 resource development, participation of underrepre s Additional criteria are not necessarily confined to large facilities and centers that are the subject of major awards. For example, proposals for materials research groups (MRGs)
|
From page 70... ...
Peer reviewers often confine themselves to those criteria they are most qualified to judge, usually technical merit and capability of the proposer, and it is not always clear who is assessing the other criteria or how they are eventually integrated by staff in the final decision. The pane!
|
From page 71... ...
Technical merit is the likelihood that a proposed project will achieve the projected research objectives, whatever they may be. Thus, technical merit has been and should remain the primary standard met by all NSF awards, including major awards, because it best assures effective use of the funding devoted to federally supported research.
|
From page 72... ...
Because the additional considerations must be taken fully into account by NSF officials in recommending the best overall proposal, reviewers should be appointed specifically to assess each of them and provide input to the process. This recommendation would be more easily implemented in conjunction with the two-phase review process recommended later in this report, which provides a mechanism for systematically ensuring that both high technical merit and the other criteria are taken into account in reaching a final decision on the overall merit of major award proposals.
|
From page 73... ...
did not find a single female principal investigator, and it seems likely that minority investigators were equally rare. Given the gradual shift in NSF resources toward this category of award, special attention should be devoted to opening the door of opportunity for leadership on major award projects at least as wide as it has been opened for women and minorities to be principal investigators on individual investigator grants.
|
From page 74... ...
Nonfederal cost sharing may be a legitimate criterion for a major award, but in accord with NSB policy, it should be required only for programmatic reasons (e.g., to ensure industrial relevance of the proposed work, to strengthen its linkages to an interdisciplinary research center, or to show that a university is committed to running a high-quality national facility)
|
From page 75... ...
Although the panel's recommendation is current NSB policy, it is reaffirmed here because the temptation to leverage over funds to make NSF dollars go further is very strong. The level of nonfederal cost sharing in NSF awards has been increasing over the past decade.
|
From page 76... ...
In a merit review system, this means that individuals with the expertise to assess each criterion must be appointed and deployed as needed for that specific solicitation (e.g., ad hoc reviews by individual experts, perhaps site visits, and one or more pane} reviews)
|
From page 77... ...
reviews may be supplemented with site visits by teams of outside experts and NSF staff. NSB members may be site visitors as well.
|
From page 78... ...
If the proposals involve substantial size or complexity, broad disciplinary or multidisciplinary content, or potential applications to significant national problems, some reviewers should have a broader or more generalized knowledge of the research subf~elds involved. Some reviewers are supposed to have a broad knowledge of the infrastructure of the national science and engineering enterprise to evaluate proposals for their impact on scientific and engineering education and human resource development, distribution of resources to institutions and geographical areas, and other social goals.
|
From page 79... ...
in the case of the Ear~quake Engineering Research Center, for example, for having too few earthquake researchers and too few Westerners on the review panel (GAO, 1987~.6 6 The General Accounting Office (GAO) audit report on the decision concluded that the composition of the panel had met NSF's requirement that there be reviewers with both special and general knowledge of the scientific subfields involved in the proposal.
|
From page 80... ...
Award Decisionmaking Among the key actors in the decision process are NSF program directors. They make initial recommendations for the review format and specific reviewers to be used for each proposal or competitive set of proposals, which division directors and assistant directors review and approve.
|
From page 81... ...
. the peer review process should continue to preserve the traditional benefits of peer evaluation of intrinsic scientific merit.
|
From page 82... ...
may take other considerations into account, such as the balance among research topics or the amount available in the program budget. In the case of individual investigator grants, recommendations by program directors can be approved at the division level.If the proposal is large enough, as in the case of major awards, the award decision is reviewed by the appropriate assistant director and at the NSF director's level before being sent for approval to NSB with a memorandum from the director.
|
From page 83... ...
Given the importance and complexity of major awards, the role of outside peer evaluation in weighing and balancing the various criteria, both technical and nontechnical, is especially important throughout the process. The recommendations in this report thus aim to create a merit review system in which outside peer review not only plays its usual major role in evaluating technical excellence but also plays the major role in evaluating secondary criteria and identifying the most meritorious proposal based on all criteria.
|
From page 84... ...
Recommendation 6: A Two-Phase Merit Review Process For major awards, the peer review part of the merit review process should be conducted in two phases. The first phase would be a strictly technical review; to help assure the primacy of technical merit, only those proposals judged to be technically superior would be forwarded to the second phase for any further consideration.
|
From page 85... ...
In research center competitions where there are numerous proposals, it is relatively easy to forward a set of top-rated proposals to the second phase, explicitly ranked in order of technical merit. In fact, NSF already uses a form of the two-phase merit review process for Me Engineering Research Centers, Science and Technology Centers, and Supercomputer Centers.
|
From page 86... ...
Phase 2 is a review of all other merit criteria (e.g., human resource development and equal opportunity, geographic or institutional distribution, technological relevance, industrial involvement, outreach and technology transfer activities)
|
From page 87... ...
. In addition to providing an overall summary rating based on technical criteria, Phase ~ reviewers should be encouraged to comment, if they wish, on the other merit review criteria.
|
From page 88... ...
Phase 2 panels should be asked to provide a careful qualitative discussion of how they balanced the nontechnical criteria against the technical ones. We believe that in the typical case the discussion will revolve around whether proposals ranked second or third on the technical merit scale are the most meritorious overall and should be funded.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.