Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
NIH Extramural Center Programs: Criteria for Initiation and Evaluation Executive Summary ABSTRACT. The National Institutes of Health (NIH) promotes advances in biomedical research primarily by supporting extramural research at colleges and universities and other nonprofit research institutions. Center awards are one mechanism of extramural support, constituting approximately 9 percent of NIH’s budget. Centers are very diverse in structure and purpose, and definitions of centers and metrics for measuring their productivity are not uniformly applied across institutes, making it difficult to evaluate their effectiveness. The committee finds, however, that extramural centers offer an attractive mechanism for supporting research that benefits from a multidisciplinary, team-based approach, especially research aimed at understanding complex biomedical systems, and for translating basic scientific discoveries into useful clinical applications. The committee makes recommendations to improve the classification and tracking of center programs, clarify and improve the decision process and criteria for initiating center programs, resolve the occasional disagreements over the appropriateness of centers, and evaluate the performance of center programs more regularly and systematically. The report concludes by noting that recent changes in the nature of biomedical research, which involve opportunities to understand complex biological systems through collaborations among multiple investigators in different fields and different institutions and by assembling large-scale research infrastructures and databases, will probably result in the expanded use
OCR for page 2
NIH Extramural Center Programs: Criteria for Initiation and Evaluation of centers and other mechanisms that support collaborative research by interdisciplinary teams. The United States currently enjoys a remarkably productive system of biomedical research. The basis of this highly successful enterprise is the partnership of academia, government, and industry in making discoveries leading to better understanding and improved ways of preventing and treating disease and promoting health. Congress has set national biomedical research priorities and provided generous research funding. A key part of the system is the National Institutes of Health (NIH), which is the largest single source of support for biomedical research in the nation and the world. NIH uses a variety of ways to identify and support high-quality research. The main approach is to invite investigators from throughout the country to submit their best ideas for research projects, have the proposals rated by an appropriate peer review group, and give grants for the projects considered the most promising by the peer reviewers. The research model historically has been a single investigator, working with one or two collaborators and several postdoctoral fellows, graduate students, and technicians on a specific project of three to five years duration, after which the investigator must apply for a renewal of the grant or a new award by proposing follow-up studies or a new project. These individual investigator-initiated grants are still the mainstay of NIH’s extramural research program, accounting for nearly two-thirds of the research grants awarded by NIH in fiscal year (FY) 2002 and the majority of the funding for the grants.1 Another research model is the multi-investigator research center. In academia, centers have evolved as a structure to facilitate collaborations by multiple investigators on a research problem of common interest. NIH has supported research centers for many years as a means of encouraging interdisciplinary basic, clinical, and population-based research on scientific problems not being adequately addressed by individual investigator grants alone. Center programs are also popular with the public, organizations representing patients, and Congress, because they can bring focus, visibility, and more funding (private and public) to research on a specific disease. Every year Congress, with the encouragement of patient advocacy groups, urges 1 In FY2002, NIH awarded 43,500 research grants, of which 63 percent were R01 (individual investigator) research project grants. R01 grants accounted for 53.4 percent of the funding ($16.8 billion). Calculated from NIH table “NIH Research Grant Awards by Fiscal Year and Activity, Fiscal Years 2000-2002” (http://grants.nih.gov/grants/award/research/rgbyact0002.htm).
OCR for page 3
NIH Extramural Center Programs: Criteria for Initiation and Evaluation NIH to establish centers for research on some particular diseases and conditions. NIH usually responds by creating a center program or by taking other initiatives it determines would be more effective in advancing research at that time. Sometimes, however, Congress directs NIH to establish centers despite NIH’s view that the funding could be better spent through other mechanisms of research support. The occasion of several of these congressional mandates in recent years led to this study of the conditions in which the establishment of new center programs is appropriate. Clearly, in the view of some, congressional intervention may have a positive effect, and such input should be anticipated and integrated into the process. A Senate amendment to the Muscular Dystrophy Community Assistance, Research and Education Amendments of 2001 specified that the “Secretary of Health and Human Services shall enter into a contract with the Institute of Medicine for the purpose of conducting a study and making recommendations on the impact of, need for, and other issues associated with Centers of Excellence at the National Institutes of Health.”2 Although the legislation refers to “centers of excellence,” the term is a general label that is not specific to a particular kind of center supported by NIH. As a result, when NIH contracted with the Institute of Medicine (IOM) in the autumn of 2002, the National Research Council established the Committee to Assess Centers of Excellence at NIH and charged it to: …conduct a one-year study of the use of research centers at the National Institutes of Health. The study will focus on the criteria and procedures used in deciding to adopt the use of centers, how they are designed and administered, comparisons with other mechanisms of research support, their impacts and costs, and how they are evaluated as a mechanism (as well as how individual centers are evaluated). The emphasis will be on how NIH uses centers as a program mechanism, compared with other mechanisms, rather than on how individual centers are chosen for awards. The committee will prepare a consensus report with findings and recommendations for improving the use of the center mechanism, given the many factors that must be taken into account in a specific area of research, including the state of the science, presence of promising research opportunities, burden of disease, need for interdisciplinary approaches, alternative mechanisms (e.g., research project grants, program project grants, and contracts), and adequacy of the research infrastructure. The report will include recommended criteria and processes for deciding whether research centers should be created. 2 P.L. 107-84, Muscular Dystrophy Community Assistance, Research and Education Amendments of 2001, Sec. 7, Study on the Use of Centers of Excellence at the National Institutes of Health.
OCR for page 4
NIH Extramural Center Programs: Criteria for Initiation and Evaluation CURRENT USE OF CENTER AWARDS Center grants and cooperative agreements (hereafter called grants) have constituted between 8 percent and 9 percent of the total NIH budget each year for more than a decade. In February 2003 NIH reported that it planned to fund 1,209 extramural research center grants at an annual cost of $2.4 billion in FY2003. Investigators in every state won center grants in FY2002. Every NIH institute funds center grants, with FY2001 totals ranging from 13 center awards (National Institute of Dental and Craniofacial Research) to more than 300 (National Center for Research Resources). The median number of awards funded by an institute was 33. Grants for specialized centers were most numerous in FY2001, totaling 383. The next most numerous were the 318 awards for “core” grants that fund center infrastructure but no actual research. Specialized centers were the most expensive, with grants averaging $2.2 million in FY2001 (not counting General Clinical Research Centers, also averaging $2.2 million, and Primate Centers, averaging $7.5 million). Close analysis of NIH Requests for Applications (RFAs) and Program Announcements (PAs) led the committee to conclude that these data, though generally useful, have some shortcomings. Specifically, not all centers are funded by center awards and some programs without the word “center” in their names are center programs in terms of goals, structure, and activities. The committee suggests a set of definitions for three types of center awards: Center Infrastructure awards or “core” grants support administrative and technical services required by a group of investigators whose research is funded by independently obtained research grants. The primary goal is to focus research on a particular set of questions and increase efficiency. Research Center awards fund not only common services but research projects as well. In some cases, they may also support additional activities such as community education, screening and counseling programs, and educating medical and allied health professionals about state-of-the-art diagnostic, prevention, and treatment techniques. Research Resource Center awards develop and provide specialized research resources, services, and tools to researchers across the country. Examples of their products include animal models, microarrays for genomic analyses, and islet cells for transplantation in patients with diabetes. The committee sees a need for NIH to adopt this set of definitions or develop a similar classification scheme and then consistently identify all activities meeting one of those definitions as centers, regardless of the name of the program or the current funding mechanism. The committee also
OCR for page 5
NIH Extramural Center Programs: Criteria for Initiation and Evaluation noted that the large category of research resource centers, those with the primary mission of developing and supplying scientists with research tools and resources such as specially bred animals, are not central to the congressional concerns that led to the current study and are therefore not included in subsequent discussion of centers in this report. Finding. NIH does not consistently apply either the term center or the budget mechanism category for center awards to extramural research centers. This inconsistency makes it difficult to describe accurately the extent of research funding devoted to support of centers or evaluate the relative effectiveness of center awards or how well center programs complement other NIH-funded activities. Recommendation 1. NIH should adopt or develop a coherent classification system with functional criteria that should be uniformly applied across all institutes for the categorization of all NIH centers. The three functional categories of centers offered above by the committee represent one possible system of classification. All activities that fit in one of the categories in the classification system adopted or developed by NIH should be identified as centers, regardless of the name of the program or mechanism of funding. INITIATION AND MANAGEMENT OF CENTER PROGRAMS The impetus for NIH support of research centers comes from many sources. Centers established since the beginning of 2002 have been suggested by external advisory groups, NIH institute strategic plans, scientific workshops supported by NIH, NIH program staff, a federal interagency coordinating group, advocacy organizations, Congress, a national commission, and an IOM report. New initiatives, such as center programs, are considered for adoption in NIH’s annual planning and budgeting process. That process is elaborate and open, involving input from external advisory groups, meetings with voluntary health associations and patient advocacy groups, and review by an institute’s national advisory council. It is a very decentralized process that varies from institute to institute, but it is generally informal in terms of procedures and criteria for adopting new programs. The process for deciding on the design and management of a new program initiative is also decentralized to the institutes, including the requirements and criteria for reviewing applications for center funding and the choice of the award to be used to support the centers that are approved for funding. The current criteria for center programs can be gleaned from the discussion of program purposes and descriptions of center activities and com-
OCR for page 6
NIH Extramural Center Programs: Criteria for Initiation and Evaluation ponents in recent PAs and RFAs for center programs, which include the following rationales: The scientific opportunities and/or health problems that the center program would address have high priority at the institute. Centers would facilitate activities that are most effectively undertaken by teams of investigators working in close proximity, such as: multidisciplinary collaborations for problems that require diverse scientific backgrounds; multi-investigator teams capable of a wide scope of research activities; translating the results of basic research into clinical practice; supporting existing and stimulating new investigator-initiated applications for research program grants; training graduate students, postdoctoral fellows, and other health professionals in cross-disciplinary or translational research; attracting experienced researchers into a new area of research; networking with other centers in the program to conduct coordinated research activities beyond the capacity of any single center, for example, by recruiting larger numbers of patients into common research protocols. The centers program would provide critical research resources that are difficult or too expensive to develop in most individual laboratories. The centers would build infrastructure to promote institutional development of a field of research (e.g., nursing, population research), state-of-the-art biomedical and behavioral research at minority-serving institutions or institutions in regions with little NIH research funding, and community education and outreach programs. The committee makes several recommendations aimed at making the process for assessing the appropriateness of research centers more explicit and consistent, developing and applying a uniform set of key questions to ask in deciding to establish a new center program (a list of such questions is suggested), and providing a mechanism to adjudicate disagreements over the need for centers as they arise. Finding. Proposals to establish center programs originate from many sources within and outside NIH, including scientific workshops, internal program reviews, national advisory councils and other advisory bodies, NIH professional staff, professional scientific societies, citizen groups, the
OCR for page 7
NIH Extramural Center Programs: Criteria for Initiation and Evaluation executive branch, and Congress. Although each of the institutes has a planning process for setting priorities and developing programs, the procedures and criteria for assessing the appropriateness of centers in an area of research are not explicit or uniform. The national advisory councils are currently required to review all initiatives, but given the small amount of time they can devote to the task, effective arrangements for soliciting external advice in the approval process and clear and consistent criteria for program approval (see next recommendation) are critical elements of center program initiation. Recommendation 2. NIH should make explicit its process for deciding whether establishment of a new center program is appropriate to meet a specified goal. The key elements of the process, which should be consistent across institutes, necessarily involve broad input from the extramural scientific community and incorporation of the views of the public. NIH needs to inform Congress and advocacy groups of the process and of the opportunities they will have to provide input. Finding. The rationale for initiating a center program stated in concept papers or in the PAs and RFAs does not always indicate why a program of centers is a better means for achieving program goals than other mechanisms of research support. The scientific rationale for adding centers to the mix of funding mechanisms in a specific area is not usually made explicit, and the comparative advantage of using centers to accelerate progress is not always shown. Recommendation 3. A uniform set of key questions to ask in establishing each program of centers, such as those listed in Box ES-1, below, should be developed and adopted by NIH. The recommendation to establish any program of centers should be supported by positive responses to the relevant questions on the list that NIH adopts. Finding. NIH is occasionally urged to establish centers by Congress or by groups advocating greater federal action on a specific disease or other health issue about which NIH scientists believe the knowledge base or the number of active researchers, or both, are too small to support an effective center program. Even if the process and criteria for reaching this conclusion are made more open and explicit, and involve broad input from the scientific and advocacy communities (see Recommendation 2), differences among stakeholders and scientific experts may still exist. Congressional hearings may not provide the optimal forum for resolving these differences. A need exists for an advisory mechanism to assist Congress and NIH when there is continuing disagreement about the need for centers for a specific disease or other health issue.
OCR for page 8
NIH Extramural Center Programs: Criteria for Initiation and Evaluation BOX ES-1 Suggested Criteria for Initiation of Center Programs Center program may be the right mechanism if… Another mechanism is more appropriate if… Both Center Infrastructure (Core-Type) and Research Center Programs Should Meet the Following Criteria. Importance of the problem: Is the area of research important enough to warrant a concentration of resources? The area of research has been declared a high priority by the institute in its planning process. The area of research is a lower priority for the institute. Need for core resources: Do shared resources in this area provide economies of scale? The area of research relies heavily on specialized resources not provided in normal university services but difficult to include in research project or program project (i.e., R01 or P01) budgets. The area of research can proceed with standard university services, or individual investigators can access the services cost-effectively. Center Infrastructure (Core-Type) Programs Should Meet the Following Additional Criteria. Concentrations of projects: Do enough investigators at one university or in close proximity already have funded projects in this area? The proposed program or award can justify in detail that there are enough users for the shared services. The number of documented potential users is well below the capacity of the proposed shared resources. Research Center Programs Should Meet the Following Additional Criteria. Sufficient number of investigators: Are there enough people working in the field to support the level of effort proposed? There are, or potentially are, plenty of strong investigators in the area, so that there will be real competition for the center awards. There are few investigators, or little potential for more investigators working in development with noncenter grants.
OCR for page 9
NIH Extramural Center Programs: Criteria for Initiation and Evaluation Need for strategic focus: Does this research area need some coordination among projects to build toward or accelerate important findings? Scattered findings from a number of research groups are leaving critical gaps in the knowledge base. These groups need to articulate a larger-scale, more coordinated research program to make or accelerate progress. Individual grants or P01s are already moving the field forward rapidly based on shared understanding of the critical methods and problems. Need for interdisciplinary interaction: Would the research problem benefit from an interdisciplinary approach that is not happening now? Current grant-supported research is largely single-discipline, and credible, independent advisors recommend an interdisciplinary approach. Current R01 and P01 research is already interdisciplinary. Need to identify research problems with translational potential: Does the clinical community perceive that their problems are not being addressed? The clinical or other practice community can provide a significant body of questions that research could address to help them solve problems but is not currently producing. The list of questions should be articulated, and the match against existing knowledge should be documented with a literature search. The clinical or other practice community is already absorbing a high level of research knowledge and has significant influence on the research agenda of basic research related to the problem. Need to stimulate translational activities: Does the basic science community perceive that their findings are not being taken up? There is a large body of knowledge that is not being translated into clinical or public health practice. The program should be able to quantify the size of that body of knowledge with publications. Basic research related to the problem is already being fully utilized in clinical research, drug development, clinical practice, or public health. Need to provide distinctive training environments Researchers trained in existing modes in the field are being prepared too narrowly to meet the challenges of problemsolving in this area, or are missing critical skills. Existing training is giving Ph.D.s and physicians the key skills and knowledge they need for their career paths.
OCR for page 10
NIH Extramural Center Programs: Criteria for Initiation and Evaluation Recommendation 4. In those occasional instances in which disagreement continues over the need to establish a new center program, the NIH director or congressional committee chairman should request that an advisory committee be appointed by the Secretary of Health and Human Services to review the evidence in support of a developing initiative for a centers program and assess whether the proposed program meets the prestated criteria for the establishment of centers. EVALUATION OF CENTER PROGRAMS NIH, like other federal research agencies, takes several approaches to evaluating the performance of its programs. One is technical review by a panel of external experts knowledgeable in the area of research involved and perhaps users of the research results. Another approach is formal evaluation based on data collected by external contractors. As described in Chapter 3, a proposal to establish a program of research centers in the first place undergoes a prospective review process, which varies from institute to institute but generally involves an external committee or workshop to obtain input on the goals and design of the proposed centers, as well as the required review and approval by a chartered external advisory body (usually the institute’s national advisory council) and clearance of the RFA or PA inviting applications for center support through the Office of Extramural Programs in the Office of the Director of NIH. Research programs may also undergo retrospective evaluation. Each ongoing as well as proposed center program must justify itself in the annual planning and budgeting process. In addition, some of the institutes engage in a formal “visiting committee” process, that is, an external panel of experts, usually a subcommittee of the institute’s national advisory committee for a major program division that reviews the division’s programs on a regular schedule. This process can lead to changes in center programs or proposals to initiate new center programs. From time to time, institute staff members, the institute director, or the national advisory council decides that a center program should be reviewed by staff, an external committee of experts, or a combination of staff and outside experts for its continuing effectiveness and/or relevance. Examples of reports from such ad hoc exercises are summarized in Appendix F. Generally the reports are based on the experience and expert judgment of committee members, because for reasons discussed below, objective measurement and analysis of a program’s performance, especially in terms of outcomes and impact, are difficult to obtain and frequently require resources and technical skills beyond that provided to the committees.
OCR for page 11
NIH Extramural Center Programs: Criteria for Initiation and Evaluation Finding. NIH does not have formal regular procedures or criteria for evaluating center programs. From time to time, institutes conduct internal program reviews or appoint external review panels, but usually these ad hoc responses are done in response to a perception that the program is no longer effective or appropriate rather than as part of a regular evaluation process. Most of these reviews relied on judgment by experts rather than systematically collected objective data, although some formal program evaluations have been performed by outside firms using such data. Recommendation 5. Every center program should be given a formal external retrospective review for its continued effectiveness on a regular basis (at least every five to seven years). The review should be coordinated at an organizational level above the centers program itself. The review should be performed by people at arms-length distance from the program and with the appropriate expertise to judge the varied activities of the centers. The views of interested publics, including the scientific and advocacy communities, as well as NIH officials and grantees, should be solicited as a matter of course. The program should be evaluated against its original objectives and with regard to the changed circumstances of its field. The review should include consideration of the question, “Are centers still the most appropriate means of making progress in this field?” and the criteria should be consistent with those adopted or developed in response to Recommendation 3 for establishing the center program in the first place. The review should use multiple sources of evidence to evaluate the effectiveness of the program, and its conclusions should be evidence-based. The review might consider, for example, the scientific impact (e.g., publication counts and impacts, important discoveries, development and sharing of research tools); impact on human health (e.g., changes in health status); and impact on human resources (e.g., career paths of pre- and postdoctoral students and investigators). A program evaluation plan should be developed as part of the design and implementation of new center programs, and data on indicators used in the evaluation plan should be collected regularly and systematically. Data collected from the centers should conform to a common format. Many of the indicators should also be useful for program monitoring and progress reporting. One set of potential indicators is provided in Box ES-2. Each institute’s plan for evaluating center programs should be linked to its strategic planning process.
OCR for page 12
NIH Extramural Center Programs: Criteria for Initiation and Evaluation BOX ES-2 Potential Indicators for Evaluating NIH Center Programs Goal: Increased basic and clinical research in program’s area of focus. Indicators: Increased number of studies in each category being funded, especially new studies; increased number and impact of publications and presentations of center research. Goal: More multidisciplinary research. Indicators: Increased number of collaborations established; increased number and percent of center studies, especially center scientific publications authored by teams of scientists from two or more university departments; greater number of disciplines represented among center-affiliated staff. Goal: More translational research. Indicators: Increased number of publications in clinically oriented journals; patent applications; licenses issued; and clinical trials under way or completed. Goal: Increased or more effective support, or both, for independently funded investigators. Indicators: Larger number of studies supported, especially new studies; more types and amounts of support supplied; characteristics of core facilities, materials, and services available; increased number of publications of center-affiliated investigators. Goal: Increased attention to program’s area of focus by centers’ home institutions, scientific community, and general public. Indicators: Increased institutional support for center operations (space, faculty and staff, recognition on institutional organizational charts and publications); additional research funding from NIH and other public agencies, nonprofit organizations, and commercial industry. CONCLUDING COMMENTS Center programs are a small but important element in NIH’s array of tools to address its dual mission of pursuing fundamental knowledge about the nature and behavior of living systems and of bridging the gap between basic science discoveries and the application of that knowledge to extend healthy life and reduce the burdens of illness and disability. Judging from the pronouncements of a number of leaders in the field, including the present director of NIH, support for centers and similar mechanisms might well grow significantly over the next decade. Elias Zerhouni, the NIH director, recently summarized the results of a series of
OCR for page 13
NIH Extramural Center Programs: Criteria for Initiation and Evaluation Goal: Successful recruitment of established researchers to the program’s area of interest. Indicators: More scientists with previous publications in the area joining the center; increased number of new grants or other funding obtained by these new investigators; number of publications, patents, or other products of work at the centers. Goal: Development of new investigators. Indicators: More trainees associated with the programs’ centers; current positions of former trainees; research grants subsequently won by these trainees at program centers or elsewhere; larger number of trainees who are elected members or fellows of professional societies. Goal: Expanded education of health professionals. Indicators: Increased number of courses, seminars, and workshops offered by program centers; larger number of health professionals attending. Goal: Expanded education of the general public. Indicators: Increased number of publications in the popular press, radio or television appearances by center staffs, increases in patient load for relevant health problems; increased percentage of patients agreeing to participate in clinical research. Goal: Demonstration of state-of-the-art prevention, diagnosis, and treatment techniques. Indicators: Increased number of seminars, grand rounds, workshops, and other educational programs conducted; larger number of local and regional practitioners participating in such programs. meetings he convened to develop a Roadmap for Medical Research, defined as a short list of initiatives that would make the biggest impact on biomedical research.3 The meetings, which included leading extramural scientists 3 On September 30, 2003, after this report was drafted, NIH made public the initiatives it is planning to implement the “Roadmap for Medical Research” (Zerhouni, 2003). As expected, new center programs will play a prominent role in implementing the Roadmap objectives. In September and October 2003, NIH issued eight RFAs under the Roadmap, five of them for new center programs. They include: National Centers for Biomedical Computing (U54); National Technology Centers for Networks and Pathways (U54); Exploratory Centers for Inter
OCR for page 14
NIH Extramural Center Programs: Criteria for Initiation and Evaluation as well as NIH institute directors, resulted in three major themes, at least two of which have been driving forces behind the establishment of center programs in the past, the need for more multidisciplinary team-based research and the need to reengineer the national clinical enterprise for faster translation of scientific discoveries into clinical reality. Although certain obstacles to collaborative research discussed in the last chapter must be overcome, this committee expects to see center programs continue and expand, because they are well suited for addressing certain kinds of research priorities, especially turning important scientific discoveries into clinically useful applications. REFERENCES Kaiser, J. 2003. Speeding up Delivery: NIH aims top push for clinical results [News]. Science 2003 (5642):28-29. Zerhouni E. 2003. Opening Statement on the FY2004 President’s Budget Request, “FY 2004 Appropriations Overview,” Hearing before the House Appropriations Subcommittee on Labor, Health and Human Services, and Education, April 2, 2003. [Online]. Available: http://www.nih.gov/about/director/budgetrequest/FY2004budgetrequest.htm [accessed May 20, 2003]. disciplinary Research (P20); Development of High Resolution Probes for Cellular Imaging (P20); and Centers for Innovation in Membrane Protein Production (P50). At least three more center programs are planned: Nanomedicine Centers, Bioactive Small Molecule Library and Screening Centers, and Regional Translational Research Centers. Funding over six years is planned to be $2.1 billion, or $350 million a year on average, although funding in the first year (FY2004) will be approximately $128 million (Kaiser, 2003). According to the five center program RFAs, up to $45 million has been set aside for center awards in FY2004.
Representative terms from entire chapter: