Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Space Studies Board Jump to Top Search: NewsJump to Science in the Subscribe to our FREE e- Headlines newsletter! NATIONAL ACADEMY OF SCIENCES NATIONAL ACADEMY OF ENGINEERING INSTITUTE OF MEDICINE NATIONAL RESEARCH COUNCIL June 18, 2004 Current Operating Status On Peer Review in NASA Life Sciences Programs On July 26, 1995, the Committee on Space Biology and Medicine Chair Mary Jane Osborn and Space Studies Board Chair Claude R. Canizares sent the following letter to Dr. Joan Vernikos, director of NASA's Life and Biomedical Sciences and Applications Division. The Committee on Space Biology and Medicine is pleased to respond to your letter of February 21, 1995, requesting that it evaluate the status of NASA life sciences research programs and peer review within the Human Exploration and Development of Space Strategic Enterprise in light of the recommendations of the recent NASA Federal Laboratory Review (NFLR). As you requested, the committee has focused on aspects of the organization of life sciences research within NASA and on the appropriateness of the external peer review system that the Division of Life and Biomedical Sciences put into effect in 1994. Specifically, the committee was asked to consider the following questions: 1. Are life sciences research and operational support clearly differentiated in the present organization and funding processes? Will these distinctions be clarified and accommodated by changes recommended by NFLR? Are any changes in distribution of life sciences research programs and resources within NASA that might result from implementation of NFLR recommendations likely to affect the scientific strength of the research program either positively or negatively? 2. Is research merit review being applied appropriately to both intra- and extramural research? Would implementation of NFLR recommendations strengthen or weaken merit review and thereby the scientific program? Are procedures for evaluating the effectiveness and equity of merit review currently defined and in place? Would implementation of NFLR recommendations affect these evaluations either positively or negatively? The committee devoted its meeting of April 12-14, 1995, to these questions. Published materials available for its consideration included the NFLR report (NASA Federal Laboratory Review, NASA Federal Laboratory Review Task Force of the NASA Advisory Council, February 1995) and the recent letter (March 29, 1995) from the Space Studies Board to Dr. France Cordova on the role of NASA http://www7.nationalacademies.org/ssb/csbmvern.html (1 of 7) [6/18/2004 9:43:47 AM]
Space Studies Board scientists and alternative management models for the science enterprise. In addition, the committee heard presentations from you, Dr. Frank Sulzman, Dr. Earl Ferguson, and Dr. Harry Holloway on the current organization of life sciences research and operational support programs within NASA and issues arising out of ongoing restructuring and downsizing of NASA; Dr. Ron White presented a detailed analysis of the results of the newly implemented 1994 external peer review process. This letter summarizes the committee's discussion and conclusions on the above issues. The committee has not been able to fully address all of the questions raised in your letter. Rather than directly assessing specific recommendations contained in the NFLR report, the committee preferred to treat the issues underlying the NFLR, independent of the various possible interpretations of the NFLR recommendations themselves. Those issues involved the appropriate use of peer review for NASA life sciences research, as well as potential organizational and administrative changes for NASA life sciences research. In addition, your need for a prompt response in light of the rapid evolution of the restructuring process precluded a more extensive study of some of the issues. Recognizing that it has not commented on many detailed matters treated in the NFLR report, the committee hopes that the following observations on some of the major issues will be helpful to you. Organizational and Administrative Issues for NASA Life Sciences Research Organizational differentiation between life sciences research and operational support depends on a clear understanding of the terms. The committee applied the following definitions to distinguish among fundamental research, operational (strategic) research, and operational support: q Fundamental research studies ways in which gravity or the space environment affects living organisms, including humans, and seeks to understand the basic mechanisms underlying such effects. Fundamental research is generally best addressed by investigator-initiated proposals drawn from the entire national and international research community, both within and outside of NASA. q Operational (strategic) research in the life sciences addresses problems related to the presence of humans in space and their short- and long- term ability to survive and function in that environment. In many instances, operational research may be performed most effectively at NASA centers. q Operational support consists of the existing technology that is necessary for spaceflight missions and includes in-house operations necessary for research experiments to be conducted in space. Operational support is necessarily concentrated at NASA centers. By these definitions, life sciences research and operational support are not always clearly differentiated in NASA's organizational and management structure. Blurring http://www7.nationalacademies.org/ssb/csbmvern.html (2 of 7) [6/18/2004 9:43:47 AM]
Space Studies Board is especially evident with respect to operational research, which arises from specific operational needs and seeks to answer operational questions or solve operational problems. The problem arises when a clear distinction is not maintained between operational research and operational support for NASA life sciences, with the result that research activities are inadequately reviewed as support activities. Careful design and conduct of such research are essential if meaningful data are to be obtained (particularly when the experiments must be carried out in space). The committee believes that rigorous peer review is the best way to assess operational research protocols and guarantee that quality and benefit are maximized. Further, it cannot be expected that the limited number of center-based NASA life scientists can include all areas of expertise that may be required to address the full spectrum of operational research problems. Moreover, additional downsizing of the intramural scientific work force, likely to result from stringent budget constraints, can only increase the dependence of NASA centers on the external scientific community. The committee recognizes the imperative to downsize NASA headquarters and decentralize aspects of program management but is convinced that strong centralized planning, coordination, and oversight of all NASA life sciences research will continue to be necessary to ensure quality and cost-effectiveness, to facilitate advantageous interactions among centers, and to minimize potential redundancies in center programs. The committee is particularly concerned about transfer of elements of program management such as project selection to centers whose in- house programs include life sciences research. On-site science program managers charged with making funding decisions affecting both in-house NASA scientists and external applicants would inevitably be subject to conflicts of interest that could seriously damage the credibility of the selection process and the relationship with the larger external research community. The committee also notes that life sciences research is a major priority for the International Space Station Alpha (ISSA). Effective establishment of research goals and priorities and coordination of research efforts across the entire research community are essential to prepare for the new opportunities and for efficient exploitation of those opportunities. Core expertise in the space life sciences resides in both the intra- and extramural scientific communities; neither is adequate alone. The planning and coordination required to set and meet research goals for ISSA utilization are best accomplished centrally, especially given the international dimensions of the scientific enterprise aboard the space station. NASA centers should to the extent possible develop and maintain focus and coherence in their intramural research programs. However, the essential role of NASA scientists as the interface between external investigators and mission development and operations may impose requirements for a breadth of expertise that works against development of a critical mass in any given subfield of research. The committee supports recommendations that Ames Research Center (ARC) explore means of providing a broader intellectual environment for NASA scientists. The committee adds a similar recommendation for Johnson Space Center (JSC). http://www7.nationalacademies.org/ssb/csbmvern.html (3 of 7) [6/18/2004 9:43:47 AM]
Space Studies Board Peer Review of Life Sciences Research All life sciences research in NASAâwhether intramural or extramural, fundamental or operationalâwill best serve NASA's mission and scientific goals if it is of the highest quality. The committee believes the time-tested process of external peer review is by far the best mechanism for carrying out merit review in order to ensure consistent scientific excellence. Applying the same peer review process to in- house and extramural research is also important to maintaining the credibility of intramural NASA research in life sciences and the respect of the extramural scientific community. Properly constituted external peer review does not in itself constitute a threat to the integrity of core intramural research programs and resources. The committee defines as "core" (1) the research, conducted at centers and primarily of the operational research type, that is essential to accomplishing the goals of the center's mission and strategic plan; and (2) unique facilities and resources that are necessary for carrying out NASA-supported research activities, either intra- or extramural. Clearly, panels designated to review proposals concerned with operational research questions and with experiments to be carried out in space should include appropriate expertise, and much of the necessary practical experience and expertise will be found among the scientists at NASA. The committee sees no reasons based on intractable conflict of interest or other considerations that would preclude expert NASA scientists from serving on peer review panels together with their extramural colleagues, as long as appropriate guidelines for conflict of interest were observed. (For instance, NASA scientists would be unable to serve on a panel reviewing their own work.) In addition to the valuable practical perspective that NASA scientists would bring to proposal evaluations, such service would no doubt be a positive factor in gaining full acceptance of external peer review by the intramural community of NASA life scientists. Unique core facilities and resources at NASA centers and other sites are important to extramural as well as intramural research activities and as such are an important focus of interaction between NASA life scientists and their external colleagues. To ensure their optimal utility as research resources for the entire life sciences community, such facilities should also receive periodic peer review including both external and internal users as reviewers. Especially in times of budgetary constraint and downsizing, questions regarding the continued effectiveness and ultimate lifespan of technological support facilities should be addressed by hard- headed examination and the broadest possible input. Although a significant amount of NASA-sponsored life sciences research is supported outside the Office of Life and Microgravity Sciences and Applications (OLMSA), the committee has little information about the nature of these programs or current mechanisms and criteria for evaluation and selection of this research. The committee believes, however, that the same principles of external peer review should be applied to all NASA life sciences research whatever the specific program http://www7.nationalacademies.org/ssb/csbmvern.html (4 of 7) [6/18/2004 9:43:47 AM]
Space Studies Board of origin. Criteria and mechanisms should be developed for evaluating both the ongoing operations of peer review and the long-range efficacy of the system in fostering excellence in research in space biology and medicine. Appropriate criteria and procedures appear to be in place for evaluation of the OLMSA's new peer review process for life sciences with respect to equity and efficiency of operation. These include detailed analyses of scores and funding success rates on the basis of applicant demographics and solicitation of and response to feedback from applicants as well as review panel chairs and panel members. The committee was very favorably impressed by data summarizing the initial 1994 experience with the new process, which gave strong evidence that the system was equitable and effective in its operation and was being applied appropriately to both intramural and extramural research proposals. The committee strongly supports continuation of the OLMSA's new peer review process. Continued effort should be directed to shortening the time from submission of proposals to their review and especially to reducing the interval between review and final funding decisions. Evaluation of the long-range efficacy of the current peer review system in fostering scientific excellence must necessarily await accumulation of sufficient experience over time to judge final outcomes. The committee suggests that a minimum of 3 to 5 years will be necessary to permit meaningful conclusions to be drawn. Development of useful criteria and mechanisms for analysis of outcomes is often a complex and difficult process, but the experience of the National Institutes of Health and the National Science Foundation suggests that it will be important to identify appropriate criteria as soon as possible in order to be able to collect data appropriate to the desired analyses. Possible criteria include, for fundamental research, publication of results in peer-reviewed journals of accepted quality and analysis of impact as indicated by frequency of citation and other means. Criteria for evaluating operational research should assess the impact of the research findings on operational problems, for example, improvement of protocols and procedures for flight, improvement in physiological responses of astronauts to the space environment, achievement of spin-offs, or improvement in the cost- effectiveness of operations. The committee wishes to thank the NASA personnel who provided the information used in this review. The committee hopes that the above guidance will be useful to you in the coming months as NASA continues its restructuring and streamlining plans. CSBM and SSB Membership COMMITTEE ON SPACE BIOLOGY AND MEDICINE Mary Jane Osborn, University of Connecticut Health Center, Chair Norma M. Allewell, University of Minnesota Charles J. Arntzen, Texas Medical Center http://www7.nationalacademies.org/ssb/csbmvern.html (5 of 7) [6/18/2004 9:43:47 AM]
Space Studies Board Robert E. Cleland, University of Washington Mary F. Dallman, University of California at San Diego Francis (Drew) Gaffney, Vanderbilt University Medical Center Marc D. Grynpas,* Samuel Lunenfeld Research Institute James R. Lackner, Brandeis University Anthony P. Mahowald, University of Chicago Elliot Meyerowitz, California Institute of Technology Kenna D. Peusner, George Washington University Medical Center Steven E. Pfeiffer, University of Connecticut Health Center Gideon A. Rodan, Merck Research Laboratories Fred D. Sack,* Ohio State University Richard Setlow, Brookhaven National Laboratory Sandra J. Graham, Study Director Victoria P. Friedensen, Senior Program Assistant *term ended June 30, 1995 SPACE STUDIES BOARD Claude R. Canizares, Massachusetts Institute of Technology, Chair John A. Armstrong, IBM Corporation (retired) James P. Bagian, Environmental Protection Agency Daniel N. Baker, University of Colorado Lawrence Bogorad, Harvard University Donald E. Brownlee, University of Washington John J. Donegan, John Donegan Associates, Inc. Anthony W. England, University of Michigan Daniel J. Fink, D.J. Fink Associates, Inc. Martin E. Glicksman, Rensselaer Polytechnic Institute Ronald Greeley, Arizona State University Bill Green, former member, U.S. House of Representatives Noel W. Hinners, Lockheed Martin Astronautics Janet G. Luhmann, University of California at Berkeley John H. McElroy, University of Texas at Arlington Roberta Balstad Miller, CIESIN Berrien Moore III, University of New Hampshire Mary Jane Osborn, University of Connecticut Health Center Simon Ostrach, Case Western Reserve University Carlé M. Pieters, Brown University Marcia J. Rieke, University of Arizona Roland W. Schmitt, Rensselaer Polytechnic Institute (retired) John A. Simpson, University of Chicago Robert E. Williams, Space Telescope Science Institute Marc S. Allen, Director http://www7.nationalacademies.org/ssb/csbmvern.html (6 of 7) [6/18/2004 9:43:47 AM]
