Assessing the Markey Research Program Grants

Three different approaches were used to assess the Research Programs Grants. First, all grantees were required to submit annual progress reports to the Trust. No specific format for the annual reports was imposed with the consequence that progress reports varied greatly in what and how they reported. The progress reports of some grantees provided a detailed insight into the outcomes of the research conducted, as well as a diary of the process used to reach these outcomes. The progress reports of some grantees were less detailed and provided only thumbnail descriptions of activities conducted by the recipient organization. Despite the unevenness of the progress reports, the committee was able to use them to document some milestones for the grantees, including data on staffing changes, construction and renovation, and purchase of major equipment.

In addition, the committee and NRC staff made 19 site visits, conducted 12 telephone interviews with principal investigators, and received two letter reports. The selection of institutions for site visits was based on an intersection of several constructs. First, the committee recognized that there was neither the time nor the resources to visit all awardees. Second, the committee wanted to visit sites that received both large and small awards and sites that were infrastructure development and investigator initiated awardees. Third, the committee wanted to restrict site visits to those programs for which the principal investigator was still actively engaged with the program. One site was unable to participate as all staff with any institutional knowledge of the grant had left the institution.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 27
Funding Biomedical Research Programs: Contributions of the Markey Trust Assessing the Markey Research Program Grants Three different approaches were used to assess the Research Programs Grants. First, all grantees were required to submit annual progress reports to the Trust. No specific format for the annual reports was imposed with the consequence that progress reports varied greatly in what and how they reported. The progress reports of some grantees provided a detailed insight into the outcomes of the research conducted, as well as a diary of the process used to reach these outcomes. The progress reports of some grantees were less detailed and provided only thumbnail descriptions of activities conducted by the recipient organization. Despite the unevenness of the progress reports, the committee was able to use them to document some milestones for the grantees, including data on staffing changes, construction and renovation, and purchase of major equipment. In addition, the committee and NRC staff made 19 site visits, conducted 12 telephone interviews with principal investigators, and received two letter reports. The selection of institutions for site visits was based on an intersection of several constructs. First, the committee recognized that there was neither the time nor the resources to visit all awardees. Second, the committee wanted to visit sites that received both large and small awards and sites that were infrastructure development and investigator initiated awardees. Third, the committee wanted to restrict site visits to those programs for which the principal investigator was still actively engaged with the program. One site was unable to participate as all staff with any institutional knowledge of the grant had left the institution.

OCR for page 27
Funding Biomedical Research Programs: Contributions of the Markey Trust Finally, in order to preserve resources, the committee concluded that, whenever possible, site visits should be made in clusters, minimizing travel time and expenses. Telephone interviews were used to augment the site visits. In two cases, NRC staff were unable to schedule a time for the telephone interview with the principal investigator, who submitted a letter report in place of the telephone interview. From some grantees telephone interviews and site visits and/or letter reports were obtained. Ultimately, data were obtained from 25 recipients. These data provided the committee with valuable insights into how funds were used in a particular institution. The committee found, however, that it was difficult to generalize the insights garnered from these sources because of the diverse nature of the problems studied by grantees and the variety of awards made. The committee came to the conclusion that Research Program Grants were awarded to a heterogeneous group of investigators at a number of different universities within differently configured research centers. The committee recognized that, at a minimum, Markey Research Program Grants awards could be classified into two categories: infrastructure development and investigator-initiated awards. The infrastructure development awards were used to create, expand, or enhance an existing department, center, or program or to develop new centers that focused on a particular aspect of the biological sciences. For several recipients, the awards resulted in the development of multidisciplinary departments within the biological sciences. Investigator-initiated awards focused on one or more particular research projects tied to a particular investigator or team of investigators. In addition, the committee classified Research Program Grants awards on a second dimension—the size of the award. The award amounts varied from less than $1 million to more than $13 million. The committee, somewhat arbitrarily, designated $4 million in total funding as the boundary between large and small awards. These awards are shown in Table A. Infrastructure development awards tended to be made during the initial years of the Trust’s philanthropy and were, in general, large awards—that is, in excess of $4 million. Investigator-initiated awards tended to be made during the concluding years of the Trust’s philanthropy and were, in general, smaller awards—less than $4 million. However, there was sufficient variability in these awards that they overlapped in size with those for infrastructure development. Site visits were made to evaluate both infrastructure development and investigator-initiated awards and to recipients of both large and small awards. A good example of such an intersection of dimensions is one of the earliest awards made by the Markey Trust to the University of California, San Francisco. This award of nearly $14 million enabled the university to

OCR for page 27
Funding Biomedical Research Programs: Contributions of the Markey Trust TABLE A Grant Recipient Award Amount Years of Funding Beginning Ending Large - Infrastructure Development California Institute of Technology $13,000,000 1986 1991 Case Western Reserve University $5,500,000 1988 1997 Cold Spring Harbor Laboratory $4,500,000 1991 1996 Columbia University $6,500,000 1988 1996 Cornell University Medical College $4,000,000 1992 1997 Duke University $8,000,000 1990 1994 Florida State University $4,500,000 1991 2000 Fox Chase Cancer Center $4,000,000 1991 1996 Harvard Medical School $11,000,000 1988 1993 Johns Hopkins University $7,150,000 1988 1996 Northwestern University $5,890,000 1989 1993 Purdue University $6,990,000 1988 1997 Stanford University $12,613,550 1986 1997 The Whitehead Institute for Biomedical Research $7,650,000 1988 1993 University of California, Los Angeles $4,350,000 1988 1997 University of California, San Diego $4,320,000 1988 1998 University of Colorado Health Sciences Center $5,000,000 1991 1996 University of Miami $6,270,000 1988 1999 University of Virginia $6,100,000 1990 1996 Washington University in St. Louis $12,100,000 1988 1994 Yale University $12,100,000 1988 1997 Large - Investigator Initiated Princeton University $4,500,000 1992 1997 The Scripps Research Institute $5,000,000 1992 1996 The University of Michigan $8,250,000 1989 1997 University of California, Berkeley $8,500,000 1989 1994 University of Chicago $9,219,223 1986 1992 University of Pennsylvania $4,720,402 1988 1996 University of Rochester School of Medicine/Dentistry $4,000,000 1991 1997 University of Washington $7,500,000 1990 1997 Vanderbilt University $5,500,000 1991 1996 Small - Infrastructure Development Carnegie Institute of Washington $2,700,000 1988 1997 Carnegie-Mellon University $1,925,000 1986 1992 Children’s Memorial Medical Center $1,000,000 1995 1997 Cincinnati Children’s Hospital Medical Center $3,500,000 1992 1996

OCR for page 27
Funding Biomedical Research Programs: Contributions of the Markey Trust Grant Recipient Award Amount Years of Funding Beginning Ending Harvard University $1,600,000 1995 1998 Harvard University, School of Public Health $3,500,000 1991 1996 Massachusetts General Hospital $3,000,000 1993 1997 New York University $2,600,000 1991 1997 Public Health Research Institute $2,500,000 1992 1996 Stanford University $1,200,000 1995 1997 The Burnham Institute $1,500,000 1992 1996 The Children’s Hospital, Boston $2,475,000 1988 1993 The Salk Institute for Biological Studies $2,600,000 1994 1996 The University of Utah $2,500,000 1993 1997 Thomas Jefferson University $3,500,000 1990 1994 University of California, Santa Cruz $2,500,000 1992 1999 University of Colorado, Boulder $1,500,000 1995 1997 University of Massachusetts Medical Center $1,500,000 1995 1997 University of North Carolina at Chapel Hill $1,500,000 1995 1997 University of Oregon $3,300,000 1988 1995 University of Texas-Houston Health Sciences Center $1,000,000 1995 1997 University of Texas-Southwestern Medical Center $2,280,000 1986 1992 University of Texas-Southwestern Medical Center $1,045,000 1988 1994 University of Texas Medical Branch at Galveston $1,000,000 1995 1996 University of Vermont $2,300,000 1991 1999 Wisconsin University-Madison $990,000 1988 1992 Small - Investigator Initiated Albert Einstein College of Medicine of Yeshiva Univ. $2,310,000 1988 1995 Baylor College of Medicine $1,400,000 1994 1999 Brandeis University $3,200,000 1988 1996 Brown University $1,300,000 1994 1998 Cornell University $1,200,000 1995 1999 Dana Farber Cancer Institute $1,500,000 1995 1997 Dartmouth Hitchcock Medical School $1,500,000 1994 1997 Eleanor Roosevelt Institute for Cancer Research $1,475,000 1988 1993 Georgetown University $1,000,000 1995 1997 Johns Hopkins University $1,300,000 1995 1997 Joslin Diabetes Center $3,500,000 1993 1999 Kennedy Krieger Institute $500,000 1995 1997

OCR for page 27
Funding Biomedical Research Programs: Contributions of the Markey Trust Grant Recipient Award Amount Years of Funding Beginning Ending Massachusetts Institute of Technology $3,850,000 1991 1999 Memorial Sloan-Kettering $2,700,000 1991 1994 Mount Sinai Medical Center $3,000,000 1993 1997 Neurosciences Institute $1,375,000 1988 1995 Oregan Health Sciences University $1,300,000 1995 1997 Rice University $1,200,000 1995 1997 Rush Presbyterian St. Lukes Medical Center $1,000,000 1995 1998 Schepens Eye Research Institute $1,000,000 1995 1997 SUNY-Buffalo $1,000,000 1995 1997 Temple University $2,500,000 1990 1996 Texas A&M University $1,000,000 1995 1998 Tufts University $2,000,000 1993 1996 University of Alabama at Birmingham $1,500,000 1991 1995 University of California, Davis $1,600,000 1995 2002 University of California, Irvine $1,000,000 1995 1999 University of Florida $1,600,000 1995 2000 University of Illinois Urbana Champagne $3,000,000 1992 1998 University of Kentucky $1,900,000 1995 1998 University of Maryland Biotechnology Institute $1,000,000 1995 1997 University of Miami $1,000,000 1995 1998 University of Pittsburgh $1,000,000 1995 1997 University of Southern California $1,800,000 1994 1998 University of Wisconsin-Madison $3,000,000 1992 2003 Worchester Foundation $1,000,000 1994 1997 establish the Program in Biomedical Science, which restructured research and graduate education in the School of Medicine. Although this award was originally classified as a General Organizational Grant by Markey Trustees, the committee saw in this award the genesis of the infrastructure development and concluded that for analytical purposes it should be considered a Research Program Grants award. This program subsequently received a site visit by the committee. Although the PIBS program at UCSF was site visited by the committee and the committee considered it an exemplary program, technically it was not a Research Program Grants award. Consequently, data on the UCSF awards are not included in Appendix D. The Markey Trust awarded the first General Organizational Grant award to UCSF in 1988. In subsequent years, the Trustees changed the focus of General Organizational

OCR for page 27
Funding Biomedical Research Programs: Contributions of the Markey Trust Grants to training in translational research. For a more thorough assessment of General Organizational Grants in general and the PIBS program in particular, see Bridging the Bed-Bench Gap: Contributions of the Markey Trust, published by National Academies Press in 2004. A third source of information came from analysis of the Lucille P. Markey Charitable Trust Records. As noted earlier, as the Trust was entering its final years, it arranged for all Trust documents to be archived at the Rockefeller Archive Center in Sleepy Hollow, New York. Following the conclusion of the Trust in 1997, all documents were transferred to the center, classified, and microfilmed. The archived Lucille P. Markey Charitable Trust Records currently consist of 153 reels of microfilm with approximately 800 frames on each reel. They are a rich source of information on all aspects of the Trust and will be made available to the public in 2007. The NRC staff searched the archive for information on the process used by the Trust to (1) define the rationale and focus of the Research Program Grants awards, (2) develop the solicitation process, (3) develop the mechanism and protocols for funding these awards, (4) establish the selection process for the awards, and (5) gain understanding of the decisions that led to the 18 supplemental grants. LIMITATIONS OF THIS ASSESSMENT Examination of these three data sources provided insights into the purpose of the awards, the processes of selecting grantees, and the impact of grant funds on researchers and recipient institutions. The committee realized that it was impossible to systematically collect data that would enable an assessment of individual programs. The committee had considered reviewing all publications that emerged from Markey funded projects; examining patents and licenses produced with Markey funding; and tabulating subsequent extramural funding that was produced by the Research Program Grants as one way to assess individual programs. However, the committee came to the conclusion that the data needed for such an assessment were inconsistent or missing or both—not only between programs, but also within them. For several additional reasons the committee’s ability to conduct an evaluation was limited: The grantees, by design, were not homogeneous; rather they represented a broad spectrum of large and small universities, academic medical centers, and research institutes. There was no single, overriding principle that directed funding for Research Program Grants. Rather, the Trustees were directed by the guideline “for the purposes of supporting and encouraging basic medical research,” and this guideline was broadly and variously interpreted.

OCR for page 27
Funding Biomedical Research Programs: Contributions of the Markey Trust The focus of Research Program Grants changed over time from a few large awards to many smaller awards. Another part of the reason for this change was a conscious decision by Trustees to make a larger number of smaller awards to newly expanding programs in biomedical research. Another part of the reason for this change was pragmatic, based on the decreased availability of funding as the Trust neared completion. Grant applications were relatively short and were more conceptual than comprehensive. Consequently, goals, objectives, milestones, and activities were not stated in sufficient detail to be evaluated…. Systematic data were not collected on grantees’ progress toward completing the goals and milestones specified in a grant. Although grantees were required to submit annual progress reports in order to receive subsequent-year funding, there was no specific format for these progress reports and they varied considerably in length and detail both longitudinally and across grantees. No comparison group of similar recipients of grants from other sources could be identified. The committee concluded that neither the data nor the existing resources would permit a rigorous evaluation of the program. Unlike grant programs funded by NIH and NSF, the Markey Research Program Grants were not guided by a systematic, uniform solicitation on which proposals were based. In addition, the focus of the Research Program Grants was fluid—some grants were made to create or build programs, some were for research, and some were for infrastructure development—reflecting both the availability of funds and changes in the goals of the Trustees. Finally, the Trustees did not request data from the grantees that would permit the evaluation of Research Program Grants. Although the Trustees recognized the need for an assessment of the Trust’s funding activities, the decision to conduct an assessment was made toward the end of the Trust’s tenure. Ideally, program evaluation would have been built in from the very beginning of the program, with variables for measuring program outcomes identified and collected from its onset. In any case, the outcome of basic research is a very long term prospect and may not lend itself to easy assessment. Thus, intermediate or other variables, such as publication rates, new faculty and postdocs, or funding rates, might be used as proxies. Foundations or government agencies that wish to assess their funding programs will have to determine what kinds of information would be useful to guide future decisions about such funding schemes and build their evaluation around such measures. The committee recognized that systematic collection of key variables would be essential to monitor grantees and assess the outcomes of grants

OCR for page 27
Funding Biomedical Research Programs: Contributions of the Markey Trust such as those awarded through the Research Program Grants. The committee considered a number of issues, including: Should there be a minimum data set (MDS) of variables collected from all grantees and, if so, what variables should be included in the MDS; How frequently should data be collected from grantees; and What mechanism should be used to collect these MDS data? The committee found guidelines for these issues from the practices used by NIH and NSF for renewal or awards. Both agencies have established systematic reporting mechanisms for the renewal of grants. NSF uses the Annual Progress Report (through Fastlane) and NIH uses the Grant Progress Report (PHS 2590) to collect these interim data. While the content of the two mechanisms differ in details, both agencies collect monitoring data in four areas: Detailed budget information, including costs for personnel (itemized for all staff employed on the project), consultants, equipment (itemized), supplies (itemized by category), travel, alterations/renovations, construction, other costs not elsewhere classified, and overhead. A listing of key personnel, their role on the project, and their annual effort; A narrative describing the project’s progress including specific aims, studies and results, significance of findings, and plans for the coming year; and A listing of publications generated by the project. The committee believes that these data elements could serve as the basis for establishing an MDS for annual progress reports grantees submit to funders. Both NIH and NSF utilize electronic transmittal of progress reports (NSF uses Fastlane and NIH uses SNAP, the Streamlined Noncompetitive Award Process). The committee recognizes the advantage of electronic transmittal, but believes that submission in a standardized protocol via the Internet would meet the needs of most philanthropic funders. In addition, the committee recognizes that establishing an MDS in itself is not sufficient to assess the outcomes of programs like the Research Program Grants. This requires identification of the possible products of such programs and the more careful, in-depth, and systematic measurement of these outcomes by the evaluation team. Moreover, the committee recognizes that if the Markey Trustees had adopted an MDS and established goals and outcomes for grantees, its evaluation of the Research Program Grants would have been much more straightforward; although

OCR for page 27
Funding Biomedical Research Programs: Contributions of the Markey Trust the committee also recognizes that the implementation of the MDS and the identification, measurement, and collection of specific program products and outcomes will inevitably lead to greater bureaucratic requirements and less flexibility in program administration, characteristics identified as strengths of the Markey Trust’s framework.