National Academies Press: OpenBook

Funding Biomedical Research Programs: Contributions of the Markey Trust (2006)

Chapter: Assessing the Markey Research Programs Grants

« Previous: Background of the Research Program Grants
Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×

Assessing the Markey Research Program Grants

Three different approaches were used to assess the Research Programs Grants. First, all grantees were required to submit annual progress reports to the Trust. No specific format for the annual reports was imposed with the consequence that progress reports varied greatly in what and how they reported. The progress reports of some grantees provided a detailed insight into the outcomes of the research conducted, as well as a diary of the process used to reach these outcomes. The progress reports of some grantees were less detailed and provided only thumbnail descriptions of activities conducted by the recipient organization. Despite the unevenness of the progress reports, the committee was able to use them to document some milestones for the grantees, including data on staffing changes, construction and renovation, and purchase of major equipment.

In addition, the committee and NRC staff made 19 site visits, conducted 12 telephone interviews with principal investigators, and received two letter reports. The selection of institutions for site visits was based on an intersection of several constructs. First, the committee recognized that there was neither the time nor the resources to visit all awardees. Second, the committee wanted to visit sites that received both large and small awards and sites that were infrastructure development and investigator initiated awardees. Third, the committee wanted to restrict site visits to those programs for which the principal investigator was still actively engaged with the program. One site was unable to participate as all staff with any institutional knowledge of the grant had left the institution.

Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×

Finally, in order to preserve resources, the committee concluded that, whenever possible, site visits should be made in clusters, minimizing travel time and expenses. Telephone interviews were used to augment the site visits. In two cases, NRC staff were unable to schedule a time for the telephone interview with the principal investigator, who submitted a letter report in place of the telephone interview. From some grantees telephone interviews and site visits and/or letter reports were obtained. Ultimately, data were obtained from 25 recipients.

These data provided the committee with valuable insights into how funds were used in a particular institution. The committee found, however, that it was difficult to generalize the insights garnered from these sources because of the diverse nature of the problems studied by grantees and the variety of awards made. The committee came to the conclusion that Research Program Grants were awarded to a heterogeneous group of investigators at a number of different universities within differently configured research centers.

The committee recognized that, at a minimum, Markey Research Program Grants awards could be classified into two categories: infrastructure development and investigator-initiated awards. The infrastructure development awards were used to create, expand, or enhance an existing department, center, or program or to develop new centers that focused on a particular aspect of the biological sciences. For several recipients, the awards resulted in the development of multidisciplinary departments within the biological sciences. Investigator-initiated awards focused on one or more particular research projects tied to a particular investigator or team of investigators. In addition, the committee classified Research Program Grants awards on a second dimension—the size of the award. The award amounts varied from less than $1 million to more than $13 million. The committee, somewhat arbitrarily, designated $4 million in total funding as the boundary between large and small awards. These awards are shown in Table A.

Infrastructure development awards tended to be made during the initial years of the Trust’s philanthropy and were, in general, large awards—that is, in excess of $4 million. Investigator-initiated awards tended to be made during the concluding years of the Trust’s philanthropy and were, in general, smaller awards—less than $4 million. However, there was sufficient variability in these awards that they overlapped in size with those for infrastructure development. Site visits were made to evaluate both infrastructure development and investigator-initiated awards and to recipients of both large and small awards.

A good example of such an intersection of dimensions is one of the earliest awards made by the Markey Trust to the University of California, San Francisco. This award of nearly $14 million enabled the university to

Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×

TABLE A

Grant Recipient

Award Amount

Years of Funding

Beginning

Ending

Large - Infrastructure Development

California Institute of Technology

$13,000,000

1986

1991

Case Western Reserve University

$5,500,000

1988

1997

Cold Spring Harbor Laboratory

$4,500,000

1991

1996

Columbia University

$6,500,000

1988

1996

Cornell University Medical College

$4,000,000

1992

1997

Duke University

$8,000,000

1990

1994

Florida State University

$4,500,000

1991

2000

Fox Chase Cancer Center

$4,000,000

1991

1996

Harvard Medical School

$11,000,000

1988

1993

Johns Hopkins University

$7,150,000

1988

1996

Northwestern University

$5,890,000

1989

1993

Purdue University

$6,990,000

1988

1997

Stanford University

$12,613,550

1986

1997

The Whitehead Institute for Biomedical Research

$7,650,000

1988

1993

University of California, Los Angeles

$4,350,000

1988

1997

University of California, San Diego

$4,320,000

1988

1998

University of Colorado Health Sciences Center

$5,000,000

1991

1996

University of Miami

$6,270,000

1988

1999

University of Virginia

$6,100,000

1990

1996

Washington University in St. Louis

$12,100,000

1988

1994

Yale University

$12,100,000

1988

1997

Large - Investigator Initiated

Princeton University

$4,500,000

1992

1997

The Scripps Research Institute

$5,000,000

1992

1996

The University of Michigan

$8,250,000

1989

1997

University of California, Berkeley

$8,500,000

1989

1994

University of Chicago

$9,219,223

1986

1992

University of Pennsylvania

$4,720,402

1988

1996

University of Rochester School of Medicine/Dentistry

$4,000,000

1991

1997

University of Washington

$7,500,000

1990

1997

Vanderbilt University

$5,500,000

1991

1996

Small - Infrastructure Development

Carnegie Institute of Washington

$2,700,000

1988

1997

Carnegie-Mellon University

$1,925,000

1986

1992

Children’s Memorial Medical Center

$1,000,000

1995

1997

Cincinnati Children’s Hospital Medical Center

$3,500,000

1992

1996

Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×

Grant Recipient

Award Amount

Years of Funding

Beginning

Ending

Harvard University

$1,600,000

1995

1998

Harvard University, School of Public Health

$3,500,000

1991

1996

Massachusetts General Hospital

$3,000,000

1993

1997

New York University

$2,600,000

1991

1997

Public Health Research Institute

$2,500,000

1992

1996

Stanford University

$1,200,000

1995

1997

The Burnham Institute

$1,500,000

1992

1996

The Children’s Hospital, Boston

$2,475,000

1988

1993

The Salk Institute for Biological Studies

$2,600,000

1994

1996

The University of Utah

$2,500,000

1993

1997

Thomas Jefferson University

$3,500,000

1990

1994

University of California, Santa Cruz

$2,500,000

1992

1999

University of Colorado, Boulder

$1,500,000

1995

1997

University of Massachusetts Medical Center

$1,500,000

1995

1997

University of North Carolina at Chapel Hill

$1,500,000

1995

1997

University of Oregon

$3,300,000

1988

1995

University of Texas-Houston Health Sciences Center

$1,000,000

1995

1997

University of Texas-Southwestern Medical Center

$2,280,000

1986

1992

University of Texas-Southwestern Medical Center

$1,045,000

1988

1994

University of Texas Medical Branch at Galveston

$1,000,000

1995

1996

University of Vermont

$2,300,000

1991

1999

Wisconsin University-Madison

$990,000

1988

1992

Small - Investigator Initiated

Albert Einstein College of Medicine of Yeshiva Univ.

$2,310,000

1988

1995

Baylor College of Medicine

$1,400,000

1994

1999

Brandeis University

$3,200,000

1988

1996

Brown University

$1,300,000

1994

1998

Cornell University

$1,200,000

1995

1999

Dana Farber Cancer Institute

$1,500,000

1995

1997

Dartmouth Hitchcock Medical School

$1,500,000

1994

1997

Eleanor Roosevelt Institute for Cancer Research

$1,475,000

1988

1993

Georgetown University

$1,000,000

1995

1997

Johns Hopkins University

$1,300,000

1995

1997

Joslin Diabetes Center

$3,500,000

1993

1999

Kennedy Krieger Institute

$500,000

1995

1997

Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×

Grant Recipient

Award Amount

Years of Funding

Beginning

Ending

Massachusetts Institute of Technology

$3,850,000

1991

1999

Memorial Sloan-Kettering

$2,700,000

1991

1994

Mount Sinai Medical Center

$3,000,000

1993

1997

Neurosciences Institute

$1,375,000

1988

1995

Oregan Health Sciences University

$1,300,000

1995

1997

Rice University

$1,200,000

1995

1997

Rush Presbyterian St. Lukes Medical Center

$1,000,000

1995

1998

Schepens Eye Research Institute

$1,000,000

1995

1997

SUNY-Buffalo

$1,000,000

1995

1997

Temple University

$2,500,000

1990

1996

Texas A&M University

$1,000,000

1995

1998

Tufts University

$2,000,000

1993

1996

University of Alabama at Birmingham

$1,500,000

1991

1995

University of California, Davis

$1,600,000

1995

2002

University of California, Irvine

$1,000,000

1995

1999

University of Florida

$1,600,000

1995

2000

University of Illinois Urbana Champagne

$3,000,000

1992

1998

University of Kentucky

$1,900,000

1995

1998

University of Maryland Biotechnology Institute

$1,000,000

1995

1997

University of Miami

$1,000,000

1995

1998

University of Pittsburgh

$1,000,000

1995

1997

University of Southern California

$1,800,000

1994

1998

University of Wisconsin-Madison

$3,000,000

1992

2003

Worchester Foundation

$1,000,000

1994

1997

establish the Program in Biomedical Science, which restructured research and graduate education in the School of Medicine. Although this award was originally classified as a General Organizational Grant by Markey Trustees, the committee saw in this award the genesis of the infrastructure development and concluded that for analytical purposes it should be considered a Research Program Grants award. This program subsequently received a site visit by the committee.

Although the PIBS program at UCSF was site visited by the committee and the committee considered it an exemplary program, technically it was not a Research Program Grants award. Consequently, data on the UCSF awards are not included in Appendix D. The Markey Trust awarded the first General Organizational Grant award to UCSF in 1988. In subsequent years, the Trustees changed the focus of General Organizational

Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×

Grants to training in translational research. For a more thorough assessment of General Organizational Grants in general and the PIBS program in particular, see Bridging the Bed-Bench Gap: Contributions of the Markey Trust, published by National Academies Press in 2004.

A third source of information came from analysis of the Lucille P. Markey Charitable Trust Records. As noted earlier, as the Trust was entering its final years, it arranged for all Trust documents to be archived at the Rockefeller Archive Center in Sleepy Hollow, New York. Following the conclusion of the Trust in 1997, all documents were transferred to the center, classified, and microfilmed. The archived Lucille P. Markey Charitable Trust Records currently consist of 153 reels of microfilm with approximately 800 frames on each reel. They are a rich source of information on all aspects of the Trust and will be made available to the public in 2007. The NRC staff searched the archive for information on the process used by the Trust to (1) define the rationale and focus of the Research Program Grants awards, (2) develop the solicitation process, (3) develop the mechanism and protocols for funding these awards, (4) establish the selection process for the awards, and (5) gain understanding of the decisions that led to the 18 supplemental grants.

LIMITATIONS OF THIS ASSESSMENT

Examination of these three data sources provided insights into the purpose of the awards, the processes of selecting grantees, and the impact of grant funds on researchers and recipient institutions. The committee realized that it was impossible to systematically collect data that would enable an assessment of individual programs. The committee had considered reviewing all publications that emerged from Markey funded projects; examining patents and licenses produced with Markey funding; and tabulating subsequent extramural funding that was produced by the Research Program Grants as one way to assess individual programs. However, the committee came to the conclusion that the data needed for such an assessment were inconsistent or missing or both—not only between programs, but also within them. For several additional reasons the committee’s ability to conduct an evaluation was limited:

  • The grantees, by design, were not homogeneous; rather they represented a broad spectrum of large and small universities, academic medical centers, and research institutes.

  • There was no single, overriding principle that directed funding for Research Program Grants. Rather, the Trustees were directed by the guideline “for the purposes of supporting and encouraging basic medical research,” and this guideline was broadly and variously interpreted.

Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×
  • The focus of Research Program Grants changed over time from a few large awards to many smaller awards. Another part of the reason for this change was a conscious decision by Trustees to make a larger number of smaller awards to newly expanding programs in biomedical research. Another part of the reason for this change was pragmatic, based on the decreased availability of funding as the Trust neared completion.

  • Grant applications were relatively short and were more conceptual than comprehensive. Consequently, goals, objectives, milestones, and activities were not stated in sufficient detail to be evaluated….

  • Systematic data were not collected on grantees’ progress toward completing the goals and milestones specified in a grant. Although grantees were required to submit annual progress reports in order to receive subsequent-year funding, there was no specific format for these progress reports and they varied considerably in length and detail both longitudinally and across grantees.

  • No comparison group of similar recipients of grants from other sources could be identified.

The committee concluded that neither the data nor the existing resources would permit a rigorous evaluation of the program. Unlike grant programs funded by NIH and NSF, the Markey Research Program Grants were not guided by a systematic, uniform solicitation on which proposals were based. In addition, the focus of the Research Program Grants was fluid—some grants were made to create or build programs, some were for research, and some were for infrastructure development—reflecting both the availability of funds and changes in the goals of the Trustees. Finally, the Trustees did not request data from the grantees that would permit the evaluation of Research Program Grants. Although the Trustees recognized the need for an assessment of the Trust’s funding activities, the decision to conduct an assessment was made toward the end of the Trust’s tenure. Ideally, program evaluation would have been built in from the very beginning of the program, with variables for measuring program outcomes identified and collected from its onset. In any case, the outcome of basic research is a very long term prospect and may not lend itself to easy assessment. Thus, intermediate or other variables, such as publication rates, new faculty and postdocs, or funding rates, might be used as proxies. Foundations or government agencies that wish to assess their funding programs will have to determine what kinds of information would be useful to guide future decisions about such funding schemes and build their evaluation around such measures.

The committee recognized that systematic collection of key variables would be essential to monitor grantees and assess the outcomes of grants

Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×

such as those awarded through the Research Program Grants. The committee considered a number of issues, including:

  • Should there be a minimum data set (MDS) of variables collected from all grantees and, if so, what variables should be included in the MDS;

  • How frequently should data be collected from grantees; and

  • What mechanism should be used to collect these MDS data?

The committee found guidelines for these issues from the practices used by NIH and NSF for renewal or awards. Both agencies have established systematic reporting mechanisms for the renewal of grants. NSF uses the Annual Progress Report (through Fastlane) and NIH uses the Grant Progress Report (PHS 2590) to collect these interim data.

While the content of the two mechanisms differ in details, both agencies collect monitoring data in four areas:

  • Detailed budget information, including costs for personnel (itemized for all staff employed on the project), consultants, equipment (itemized), supplies (itemized by category), travel, alterations/renovations, construction, other costs not elsewhere classified, and overhead.

  • A listing of key personnel, their role on the project, and their annual effort;

  • A narrative describing the project’s progress including specific aims, studies and results, significance of findings, and plans for the coming year; and

  • A listing of publications generated by the project.

The committee believes that these data elements could serve as the basis for establishing an MDS for annual progress reports grantees submit to funders. Both NIH and NSF utilize electronic transmittal of progress reports (NSF uses Fastlane and NIH uses SNAP, the Streamlined Noncompetitive Award Process). The committee recognizes the advantage of electronic transmittal, but believes that submission in a standardized protocol via the Internet would meet the needs of most philanthropic funders.

In addition, the committee recognizes that establishing an MDS in itself is not sufficient to assess the outcomes of programs like the Research Program Grants. This requires identification of the possible products of such programs and the more careful, in-depth, and systematic measurement of these outcomes by the evaluation team. Moreover, the committee recognizes that if the Markey Trustees had adopted an MDS and established goals and outcomes for grantees, its evaluation of the Research Program Grants would have been much more straightforward; although

Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×

the committee also recognizes that the implementation of the MDS and the identification, measurement, and collection of specific program products and outcomes will inevitably lead to greater bureaucratic requirements and less flexibility in program administration, characteristics identified as strengths of the Markey Trust’s framework.

Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×
Page 27
Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×
Page 28
Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×
Page 29
Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×
Page 30
Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×
Page 31
Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×
Page 32
Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×
Page 33
Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×
Page 34
Suggested Citation:"Assessing the Markey Research Programs Grants." National Research Council. 2006. Funding Biomedical Research Programs: Contributions of the Markey Trust. Washington, DC: The National Academies Press. doi: 10.17226/11627.
×
Page 35
Next: Outcomes of the Research Program Grants »
Funding Biomedical Research Programs: Contributions of the Markey Trust Get This Book
×
Buy Paperback | $47.00 Buy Ebook | $37.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

During an interval of 15 years, the Lucille P. Markey Charitable Trust spent over $500 million on four programs in the basic biomedical sciences that support the education and research of graduate students, postdoctoral fellows, junior faculty, and senior researchers. The Markey Trust asked the NRC to evaluate these programs with two questions in mind: “Were these funds well spent?” and “What can others in the biomedical and philanthropic communities learn from the programs of the Markey Trust, both as an approach to funding biomedical research and as a model of philanthropy?” One of five resulting reports, this volume examines the Research Program Grants, which awarded $323 million to support investigators with a major commitment to the life sciences and to assist in the establishment, reorganization, or expansion of significant biomedical research centers or programs. Using information from Markey archives, materials from grant recipients, and site visits to a sample of institutional grant recipients, the authoring committee describes the impact that Markey grants made on the centers and programs funded by these grants, along with the unique aspects of the Markey approach to funding that may be applicable to other funders of biomedical research programs.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!