Cover Image

PAPERBACK
$35.50



View/Hide Left Panel

Program Assessment in HHMI-Sponsored Medical Student Research Training Programs

Min K. Lee, Barbara Ziff, and William R. Galey


For over 20 years, the Howard Hughes Medical Institute has been involved in research training for medical students. The HHMI-NIH Medical Research Scholars Program was established in 1985 as a partnership with the National Institutes of Health (NIH) to provide an opportunity for students from U.S. medical and dental schools to conduct a full year of research at the NIH laboratories in Bethesda, Maryland. Currently, 42 students are provided salary, travel and educational support, and research facilities. The students are housed in a facility on the NIH campus that once was a convent for cloistered nuns; therefore the program has become known as the “Cloister Program.” Its sister program, the Research Training Fellowships for Medical Students program, provides support for U.S. medical and dental students to conduct a year of research at their home institution or nearly any other academic institution in the United States, except the NIH. The primary goal of both programs is to enhance the number and quality of physician-scientists in this country, but HHMI is confident that the year of research experience will also enhance the careers of the alumni who choose purely clinical careers.

As part of HHMI’s ongoing program evaluation, both anonymous and nonanonymous feedback is gathered from alumni regarding their perceptions of the program at the beginning, during, and end of their research training experiences. To determine the effectiveness of this program, the career development of the former program participants is also followed. Two major challenges hampering this effort are (1) the long period between program participation and career initiation and (2) the



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 99
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation Program Assessment in HHMI-Sponsored Medical Student Research Training Programs Min K. Lee, Barbara Ziff, and William R. Galey For over 20 years, the Howard Hughes Medical Institute has been involved in research training for medical students. The HHMI-NIH Medical Research Scholars Program was established in 1985 as a partnership with the National Institutes of Health (NIH) to provide an opportunity for students from U.S. medical and dental schools to conduct a full year of research at the NIH laboratories in Bethesda, Maryland. Currently, 42 students are provided salary, travel and educational support, and research facilities. The students are housed in a facility on the NIH campus that once was a convent for cloistered nuns; therefore the program has become known as the “Cloister Program.” Its sister program, the Research Training Fellowships for Medical Students program, provides support for U.S. medical and dental students to conduct a year of research at their home institution or nearly any other academic institution in the United States, except the NIH. The primary goal of both programs is to enhance the number and quality of physician-scientists in this country, but HHMI is confident that the year of research experience will also enhance the careers of the alumni who choose purely clinical careers. As part of HHMI’s ongoing program evaluation, both anonymous and nonanonymous feedback is gathered from alumni regarding their perceptions of the program at the beginning, during, and end of their research training experiences. To determine the effectiveness of this program, the career development of the former program participants is also followed. Two major challenges hampering this effort are (1) the long period between program participation and career initiation and (2) the

OCR for page 99
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation natural “scattering” of participants after their involvement in our programs as they pursue further training. In addition, HHMI depends on its alumni to voluntarily provide career information. This paper describes the information collected about the alumni, the method and verification of the collected information, and two approaches used to assess HHMI programs. DATA COLLECTED ON HHMI ALUMNI The primary information collected from HHMI awardees, either at the time of their award and in subsequent years, is: name (and any name changes thereafter) and social security number; personal and professional addresses, e-mail, and telephone numbers; permanent or parental addresses, telephone, and e-mail contact information; medical school information and dates of matriculation and graduation; location and dates of residency and other training information; professional appointments; publications and awards; research funding (NIH and other sources); and curricula vitae. MEANS OF DATA COLLECTION This information is collected from a number of sources. The first is the application materials, where permanent and current name and address information as well as medical school entrance information is obtained. A significant amount of immediately useful information about the program is gained from end-of-year interviews and an anonymous online exit survey. While most of this information helps in evaluating and modifying program elements, it provides limited information on awardee career outcomes. Subsequent to the end of the participants’ involvement in the program, HHMI begins to follow the career development and successes of each individual. HHMI tried to follow each person through an annual Web-based update survey that asks the individual to add or modify only information that is new or changed. Unfortunately, this approach has had only limited success, as a significant number of alumni are either unmotivated or cannot find the time to complete the survey on an annual basis.

OCR for page 99
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation To provide alumni with an incentive to supply updated information, HHMI has recently begun to offer them its popular program alumni directory upon completion of the online survey. A surprising amount of information on our alumni is collected from informal interactions with them. These include national scientific meetings, when they return to visit the program facilities, and during staff recruitment trips. HHMI often initiates the contact, but occasionally alumni will spontaneously contact program staff. Most recently, a Regional HHMI Trainee Alumni Network was established in which alumni from all HHMI graduate and medical research trainee programs are invited to attend and meet each other at regional gatherings. HHMI attempts to have program staff attend these events, which often include a combination of scientific presentations and career development and networking opportunities for alumni. These meetings have been very informative and HHMI believes they will also increase participation in the annual online updates. Finally, it has been found useful to congregate alumni members into focus groups to solicit their input and evaluation on specific issues. Similar to the involvement of individuals in the alumni regional programs, the focus groups also seem to renew interest in updating individual information. Occasionally, contact is lost with some alumni. When this happens, HHMI tries to relocate them using online search tools such as Google, Web sites of state medical boards of licensure, medical school faculty directories, hospital staff directories, and personal and practice Web pages. On a few occasions commercial locator services have been used. In short, numerous formal and informal tools have been used to determine the career outcomes of HHMI alumni. A TALE OF TWO ASSESSMENT APPROACHES In recent years, HHMI has found it appropriate to evaluate (1) whether the two programs have been successful in increasing the likelihood that alumni will become physician-scientists, (2) the level of success attained by participants of each of the two programs, and (3) if the outcomes of the two programs are different. This was done to help the institute decide what, if any, changes should be made to either or both programs to increase their effectiveness and to help institutional leadership decide whether the programs were achieving their purpose and should continue to be supported. Further, the institute wanted to inform the research, medical, and medical education communities as to the success or failure of the general paradigm these programs represent of “year-out research training programs” as a model for training much-needed physician-scientists.

OCR for page 99
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation Approach #1: One approach was to enlist the aid of an independent organization to evaluate the program through the level of success enjoyed by alumni representing a segment of graduation years (1987–1995 and 1991–1995) of each HHMI program (1). These individuals were studied because it was thought that enough time had passed to be able to determine if their early professional careers had been influenced by their program. The approach was to compare the awardees of the two HHMI-sponsored programs with Medical Science Training Program (MSTP) and non-MSTP M.D.-Ph.D. graduates and general M.D. graduates. Assessments were made employing logistic regression analyses to control for academic and demographic variables (such as MCAT (Medical College Admission Test) scores and medical school rank) that could influence the selection to the various programs. Results from the above study showed that, while women and underrepresented minorities were represented proportionately to their percentage in the overall pool of medical students in the two HHMI-sponsored programs, they were not represented to the same degree in M.D.-Ph.D. programs. The data also showed that alumni from the HHMI programs were more likely or equally likely to receive a faculty appointment in a medical school that had significant research responsibility. In addition, they were as likely as non-MSTP M.D.-Ph.D. students to receive NIH postdoctoral grant support (an indicator of future success at obtaining NIH research support). This study concluded that the HHMI training programs are “an attractive strategy for training physician-scientists” (Fang and Meyer, 2003). Data for the analysis was obtained from the Association of American Medical Colleges (AAMC) Medical Student databases, the AAMC Faculty Roster Database, and the Medline and CRISP databases. Approach #2: The second approach to evaluate the success of HHMI program alumni was to utilize the data collected by Cloister staff from the interactions described above. The results of the simple studies documenting the early careers of the former awardees are presented in Table 1. The data in Table 1 show that alumni of the two HHMI programs are essentially equivalent. Roughly 60 percent were engaged in research 10 years after participation in the programs and over 50 percent went on to early careers in academic medicine. The data also show that 7 to 8 percent went on to study for a Ph.D. degree and that alumni of both programs published an average of over one paper a year even through their clinical training years. In addition, anecdotal testimonies were available from the accumulated information. The following quote is from an alumnus who is an associate professor in a major eastern medical school:

OCR for page 99
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation TABLE 1 Intramural Study of HHMI-Supported Medical Student Research Training Programs, 1985–1992 Classes Classes 1985a–1992 99% Reporting Med Fellowsa Cloister Currently doing research (25% or more of time) 59% 58% Hold an academic appointment 52% 51% Went on for Ph.D. 8% 7% Work in industry 3% 5% Average number of publications per year 1.2 1.1 aMed fellows program began in 1989. It is not an overstatement to say that the Research Scholars Program changed my career. Before entering the program, after my third year of medical school, I was going to be a head and neck surgeon. Surgery was exciting, interesting, and challenging. During my tenure in the laboratory, I realized that science was the same. There is no doubt that my year as a Research Scholar opened doors in both the medical and scientific communities. EVALUATION OF THE TWO APPROACHES TO PROGRAM ASSESSMENT Although both the internal and external program evaluations showed both HHMI training efforts to be equivalent and effective, they present differing views of the two programs and their relationship to other training opportunities. Because HHMI’s internal program evaluation has such detailed and up-to-date information on the career activities of alumni, it provides quantitatively different information from the external evaluation process that utilizes more publicly accessible information. For instance, although the data were collected by both evaluations at essentially the same time, the internal analysis identified over 306 publications by a cadre of alumni, whereas the external assessment identified only 109 publications for the same group of individuals. A close look showed a number of differences, including: Common names made it necessary to use author search criteria that selected not only on the name of the individual but also the institution of affiliation used in the publication. This resulted in some publica-

OCR for page 99
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation tions not being attributed to a particular alumnus if he or she published using a slightly different institutional name or listed an institutional affiliation (such as a hospital or an institute) that differed from the institutional affiliation of the individual’s “academic appointment” used in the electronic database search. Publications of some individuals were not found because some alumni had changed their names due to marriage. Occasionally, individual publications were not found because they were attributed to an institution where the individual had a former affiliation rather than the current “academic appointment.” Apparently some publications were not attributed to the alumni because the publication was identified with a collaborator rather than the alumnus. Discrepancies were also identified between the database-oriented external evaluation and the internally derived data in determining “faculty positions” of alumni. The in-house-derived data obtained from program files showed that for a group of program alumni, 85 had obtained a faculty position, while the data derived from the AAMC Faculty Roster Schedule (FRS) found 75. The externally derived data were further compromised when it was found that of the 75 academic positions identified, 18 individuals had already left their positions and 28 attained faculty positions had not yet been listed in the FRS database by the academic institutions. SUMMARY Both the internal and external evaluations of the two HHMI medical student research training programs indicate that the two programs are successful and roughly equivalent. There are a number of advantages and disadvantages for each approach to the evaluation of alumni success. The external review process is inherently respected for its presumed lack of bias and hence may have more external acceptance. Consequently, such studies may have greater public impact. Furthermore, the use of large databases such as those used in the study by Fang and Meyer allow much more adjustment for confounding factors, comparisons with other trainee populations, and powerful statistical analyses. Finally, the cost is significantly less, is nonreoccurring, and requires relatively little institutional staff effort. Alternatively, the internal evaluation approach makes use of already-existing alumni data, is likely to be more accurate and complete, and is able to evaluate grant funding from non-NIH sources. However, for this to be true, a major effort must be made to assure that the data collected are

OCR for page 99
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation comprehensive and up to date. It was perceived that although the external review was intellectually more rigorous, it appeared that decision-makers evaluating the program saw the sophisticated statistical analyses as too complicated and perhaps obfuscatory. The simple results and personal testaments provided by the less involved internal analyses seemed to have more impact. ACKNOWLEDGMENT The authors thank Dr. Anh-Chi Le for her help in preparing the presentation and manuscript. REFERENCE Fang, D., and R. E. Meyer. 2003. Effect of two Howard Hughes Medical Institute research training programs for medical students on the likelihood of pursuing research careers. Academic Medicine 78:1271–1280.