Enhancing Philanthropy's Support of Biomedical Scientists Proceedings of a Workshop on Evaluation (2006) / Chapter Skim
Currently Skimming:
Burroughs Wellcome Fund Evaluation Strategy--Martin Ionescu-Pioggia and Georgine Pion
Burroughs Wellcome Fund Evaluation Strategy--Martin Ionescu-Pioggia and Georgine Pion
Pages 21-42
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 21... ...
. CABS is a five-year $500,000 bridging award targeted at helping talented postdoctoral scientists obtain tenure-track faculty positions and achieve research independence.
|
From page 22... ...
During this period, the board made its overall approach to evaluation explicit. This paper describes the overall evaluation strategy, the evaluation efforts that have been conducted, and how the results of these evaluations have affected BWF decisions about both program design and investment in their continued operation.
|
From page 23... ...
The impact of decision makers increases as the pyramid ascends, and in this regard, the pyramid reflects BWF's position that advisory committee judgment carries more weight than formal data-based evaluations in the decision-making process. As can be seen, formal data-based evaluation studies form only one of several activities used by BWF to oversee and evaluate its programs.
|
From page 24... ...
In discussing BWF's evaluation strategy and how individual evaluation studies would fit into this strategy, the board identified several factors that should guide the conduct of these efforts. That is, BWF should: • only initiate studies when there is a clear rationale for their conduct; • conduct outcome evaluations that are prospective (when possible)
|
From page 25... ...
By Other Organizations: • Adoption of similar program by the Kauffman Foundation BWF-HHMI 2002 • BWF staff monitoring By BWF: Lab Management and review • Offering of the course in the Course: BWF • Survey of course subsequent year portion ~ $480,000 participants • Development and • Analysis of Web site data dissemination of course guide Making the Right Move for use by any postdoctoral or individual responsible for postdoctoral training • Creation of the Partners program MBL Frontiers in • BWF staff monitoring By BWF: Reproduction (FIR) and review • Continuation of funding for Course: $1.2 million • Outcome survey of another three years former course participants and analysis of extant data continues
|
From page 26... ...
With a competitive award rate of 10.7 percent, BWF has made 217 awards, for an investment of approximately $107 million in young scientists' careers. Evaluation Efforts During the time BWF discussed evaluation of the program, there were neither similar programs (except for the previously mentioned one by the Markey Charitable Trust)
|
From page 27... ...
The study design purposively included a small number of critical outcome markers, based on the program's goals or "markers of success" involved in becoming an independent investigator. Among them were: • receipt of a tenure-track faculty position at a research-intensive university, • time elapsed from degree receipt to first faculty appointment and total length of postdoctoral training, • percentage of time spent in research, • amount of start-up funding offered by the hiring faculty institution, • receipt of NIH R01 funding and time to receipt of first R01, and • number and quality of publications.
|
From page 28... ...
Results are shared with grant recipients. A later analysis compared awardees' laboratory start-up packages to national averages for newly hired faculty at public and private universities and in average or high-end research areas (see Figure 2)
|
From page 29... ...
tracking study, a second study, termed the "comparative study," was launched to compare the performance of CABS recipients with individuals who had applied to the program but had not been selected to receive an award. In designing the study, emphasis was placed on examining only those research-related career outcomes for which data were available from existing sources (e.g., NIH grants and publications)
|
From page 30... ...
Use of Evaluation Results The evaluation data gathered from both BWF staff and formal evaluation studies have informed CABS program decisions and helped finetune the award to help meet the needs of recipients and validated BWF's method of advisory committee application review. They also have been used by outside organizations involved with issues related to early-career investigators.
|
From page 31... ...
• Based on outcome surveys and staff interviews with grantees, the patent policy for CABS recipients was revised to allow patents to either become the property of the investigator or to defer to institutional policy, supplemental grants were developed to support collaborative research and support awardee-initiated symposia at professional meetings, and the required minimum of one additional year of postdoctoral study for grantees who obtained faculty positions between the time of application and the start date of the award was waived. • Review of grantee progress reports and results of the tracking study revealed that grantees often had other sources of research support.
|
From page 32... ...
STUDENT SCIENCE ENRICHMENT PROGRAM The board also recommended during its initial terrain mapping that BWF evaluate its Student Science Enrichment Program (SSEP) program.
|
From page 33... ...
Ratings from the evaluators on program success based on student feedback data and observations of each project also suggest the program is accomplishing its goals. An interesting facet of the SSEP evaluation initiative is that it includes ongoing evaluation capacity building for all sites to strengthen evaluation competency within funded programs and improve the quality of outcome data provided by each site to the summary SSEP outcome database.
|
From page 34... ...
At the policy level, the BWF-HHMI laboratory management course provides a national and an international model for training new investigators in nonscientific management topics. Evaluation Efforts BWF developed the evaluation for the 2002 course, which involved surveys of course participants after each session, at the completion of the course, and at six months and one year after the course.
|
From page 35... ...
N Downloads Chapter 10,619 Getting Funded 8,948 Data Management and Lab Notebooks 7,380 Obtaining and Negotiating a Faculty Position 4,016 Lab Leadership -- Revised: Staffing Your Laboratory 3,698 Time Management 3,291 Mentoring and Being Mentored 3,174 Getting Published and Increasing Your Visibility 2,814 Lab Leadership -- Revised: Defining and Implementing Your Mission 2,682 The Investigator Within the University Structure 2,649 Project Management 2,207 Collaborations 2,070 Course Overview and Lessons Learned 1,616 Tech Transfer
|
From page 36... ...
Evaluation of the Course Following an initial three years of funding and one three-year renewal, BWF wanted to know whether this course was meeting its goals given the $800,000 investment at the second renewal. BWF staff, in conjunction with the evaluation consultant and course directors, designed a tracking study to determine outcomes from the first six years of the course, which may be the first formal outcome study of "small-dose training treatments" like a six-week course.
|
From page 37... ...
Evaluation Efforts Board terrain mapping conclusions, combined with CABS outcomes, led BWF to conclude that it receives better results when it manages programs it develops rather than supporting external endeavors. During the terrain mapping exercise, BWF wanted to know whether outcomes of the AAOGF fellowship merited continued support for the fellowship.
|
From page 38... ...
If evaluations of externally funded grants are planned, evaluation capacity building should be done with the recipient from the outset. COLLABORATIVE EVALUATION ACTIVITIES OF THE HEALTH RESEARCH ALLIANCE The Health Research Alliance (HRA)
|
From page 39... ...
SUMMARY Albeit very modest, BWF's investment in monitoring and assessing the outcomes of its programs has supplied useful data to inform the fund's decisions about program design and continued investment in programs. In developing plans for new studies, staff plan to work closely with the BWF board, as was the case with the first CABS tracking study, to develop evaluations that address the outcomes of most interest and that are acceptable to the board at the onset.
|
From page 40... ...
These reactions argue for design simplicity and for close collaborations with governance in planning outcome evaluations. As this paper was being written, the BWF Board of Directors conducted a thorough review of the fund's evaluation strategy eight years after the original approach was formulated.
|
From page 41... ...
:507–516. Burroughs Wellcome Fund and Howard Hughes Medical Institute.
|
From page 42... ...
2003. Bridging postdoctoral training and a faculty posi tion: Initial outcomes of the Burroughs Wellcome Fund Career Awards in the Bio medical Sciences.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.