National Academies Press: OpenBook

Evaluation of the Markey Scholars Program (2006)

Chapter: 3 Evaluation Methodology

« Previous: 2 Markey Scholar Awards in Biomedical Sciences
Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×

3
Evaluation Methodology

The committee was asked to assess whether the Markey Trust’s funds were well spent and what others could learn from the programs of the Markey Trust both as an approach to funding biomedical research and as a model of philanthropy. For purposes of this report, the committee was particularly interested in determining if there was evidence that the Markey Scholar award gave a particular and singular advantage to Scholars that enabled them to reach scientific independence, career progression, and positions of leadership in the scientific community earlier than a similar group of postdoctoral fellows without the Markey award. The committee recognized that there were two aspects of the Markey award that could account for differences between the Markey Scholars and a comparable group of postdoctoral fellows. One was the process used to select Markey Scholars, a process designed to ensure that only outstanding researchers were selected. The other was the size, structure, and duration of the award itself. The committee concluded that it was unable to differentiate the impacts of these two factors, and that they could only evaluate the Markey award program generally.

To do this, the committee identified measures that would indicate scientific independence, research productivity and professional success, and selected the following five as particularly salient:

  1. Appointment to a tenure-track position in a top-ranked Research I university

  2. Promotion and tenure to associate and full professor

  3. Publishing rate

Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×
  1. Citations per individual and per article

  2. Success in achieving extramural funding, especially a traditional NIH research program award (R01)

In addition, the committee determined that the use of a comparison group would strengthen its ability to assess the relative impact of the Markey award on the Scholars’ outcomes. The committee established three criteria to use in searching for possible comparison groups for the evaluation of the Markey Scholars. First, they specifically sought programs that provided transitional funding for postdocs, not funding to faculty (persons with faculty status were ineligible for the Markey Scholar award). Second, they sought programs that provided initial funding for approximately the same time as the Markey Scholars, 1985 through 1991. Finally, they sought programs that provided generous funding for seven years that included both stipend/salary funding as well as funds for laboratory expenses. The committee could find no programs that met these criteria. The NIH K22 awards did not begin until 1998. The Burroughs Wellcome awards, based on the Markey award, did not begin until 1995. The American Heart Association fellow-to-faculty awards did not begin until 2002. Career development awards made by Pew, Searle, American Cancer Society, American Heart Association, Sloan, Keck, and Beckman are all oriented toward funding faculty and generally speaking provide funding for only 3 or 4 years. Markle awards were restricted to physicians.

The committee concluded that the best comparison group would be candidates who were considered for the Markey Award but who did not receive it. After examining the Scholar selection process, the committee decided that two comparison groups could be identified. The first comparison group was composed of candidates who were top-ranked but not selected (referred to in this report as “top-ranked candidates”). The second comparison group consisted of candidates who were competitive, but not top-ranked (referred to in this report as “competitive candidates”).

SOURCES OF DATA

The evaluation of the Markey Scholars Program presented the committee with some interesting considerations. First, the outcome of the evaluation would not inform the Markey Trust, which no longer existed. Rather the evaluation would inform others in the philanthropic community that supported the training and research of biomedical scientists. Second, whatever approach to evaluation the committee selected, the approach would be a hybrid: a combination of both a prospective and a retrospective assessment.

Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×

All, or nearly all, of the Markey Scholars had completed their awards by the time of the initial meeting of the National Academies committee in 1998. However, the Markey Trustees realized that any evaluation of the Scholars program would require data collection well beyond the end of the Markey Trust. They envisioned an evaluation that would be completed in 2007 (Lucille P. Markey Charitable Trust Records) and commissioned the National Research Council of the National Academies to conduct a prospective evaluation that would culminate about 10 years after the Trust ceased operation.

The committee identified five sources of information that would inform its evaluation of the Markey Scholars. Three sources provided quantitative data:

  1. Curriculum vitae of Markey Scholars and comparison group members

  2. Citations of articles published by Markey Scholars and comparison group members

  3. Data on extramural funding of Markey Scholars and comparison group members

Two additional sources provided qualitative information:

  1. Data collected by the Markey Trust

  2. Interviews with Scholars and comparison group members.

Quantitative Data

Curriculum vitae (CV) submitted by Scholars and comparison group members at the time of their interview. All Scholars and comparison group members were asked to submit a complete CV at the time of their interview. The CV analysis was used to extract data on career progress such as rank, tenure, promotions, honors and awards. It was also used to extract data on productivity such as licenses, patents, and journal articles.

For the publication analysis, counts were made of journal articles listed on an individual’s CV for a 14-year period beginning with the year in which the individual received or would have received the Markey Scholar Award. The 14-year range was chosen because it was assumed that all individuals in the study would have achieved their first professional position (i.e., became faculty members if they stayed in academia) within 4 years and the evaluation sought to follow up 10 years later. Consequently, all Scholars, top-ranked, and competitive candidates were compared over an interval of the same length.

Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×

Data on citations. While journal articles provide an indicator of productivity, citations for these articles provide a measure of overall scholarly impact. The number of citations for journal articles written by Scholars and comparison group members were obtained from the Institute for Scientific Information (ISI). NRC staff submitted permutations of the names of all Scholars and comparison group members in ISI format (e.g., J Smith and JB Smith for John B. Smith), indicating the years of interest for the biological and biomedical sciences broadly defined. Using current CVs for Scholars and for both candidate groups, NRC staff linked journal articles from the CV for the 14-year period used in counting publications (see above and Figure 3-1) to the ISI citation database which provided citations for these articles for the period through 2004. So, for example, using articles published by Scholars and comparison group members in cycle 1 for the period 1985-1998, we counted citations on those articles for the period 1985-2004. (Since an insufficient number of up-to-date CVs were submitted by comparison group members in cycles 6 and 7, the citation analysis was carried out for just cycles 1-5.)

Data on extramural funding. To measure the extent of extramural funding received by Scholars and comparison group members, the committee examined the number of NIH grants received by them. Generally speaking, for biomedical scientists, NIH is the largest and most important source of extramural funds. Obtaining an R01 grant is considered an important rite of passage for biomedical researchers. Also data on the number and type of NIH grants are listed in the Computer Retrieval of Information on Science Projects (CRISP) database (National Institutes of Health, 2006a). NRC staff searched the CRISP database for the relevant 10 years for each Scholar and comparison group member to obtain data on the number and type of NIH grants.

The committee used these three quantitative data sources to obtain 12 outcome variables to track the progress of Scholars and comparison group members. These outcome variables are:

  1. Current rank—if in academia—and prestige of academic institution.

  2. Tenure status

  3. Number of years to tenure

  4. Current position (if not in academia)

  5. Years to current position

  6. Number of honors and awards

  7. Number of journal articles

  8. Number of citations

  9. Total number of NIH grants

  10. Number of years to obtain first NIH grant

Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×
  1. Number of R01 grants

  2. Number of years to obtain first R01 grant

Qualitative Information

Extant data collected by the Markey Trust on Markey Scholars and candidates for the award. The Markey Trustees collected a great deal of information from all the Markey Scholars and a lesser quantity of data from unsuccessful candidates for the award. At a minimum, these data consisted of the application package described earlier. For Scholars, additional data elements consisted of annual progress reports, equipment and supply expense statements, correspondence with the Trust’s Miami office, and comments and assessments from the Scholar Selection Committee. Further information was available from the archived records of the Lucille P. Markey Charitable Trust Records.1

Ethnographic interviews of all Scholars and comparison group members conducted approximately 10 years after the assumption of the first professional position. The committee recognized that it was not possible to determine the critical decision points or the thought processes that led to those decisions from CV analysis or citation data alone. Consequently, the committee decided to conduct a relatively short—35 to 45 minute—telephone interview with all Markey Scholars and comparison group members. These interviews were conducted by a consultant who was a biomedical scientist and also trained in interviewing techniques. All interviews were recorded and transcribed. Rather than use a structured interview schedule, the committee elected to conduct ethnographic interviews2 (Spradley, 1983).

1

As the Trust was entering its final years, it arranged for all Trust documents to be archived at the Rockefeller Archive Center in Sleepy Hollow, New York. Following the conclusion of the Trust in 1997, all documents were transferred to the center, classified, and microfilmed. The archived Lucille P. Markey Charitable Trust Records currently consist of 153 reels of microfilm with approximately 800 frames on each reel. They are a rich source of information on all aspects of the Trust and will be made available to the public in 2007.

2

Ethnographic interviews employ descriptive and structural questions. Descriptive questions are broad and general and allow people to describe their experiences, their daily activities, and objects and people in their lives. These descriptions provide the interviewer with a general idea of how individuals see their world. Structural questions are used to explore responses to descriptive questions. They are used to understand how the respondent organizes knowledge. Interviews begin with descriptive questions. Responses to the descriptive questions enable the interviewer to discover what is important to the subject and lead to structural questions. Good ethnographic interviews use the following six guidelines:1. Ask for use instead of meaning.2. Use open-ended questions rather than dichotomous questions that trigger a yes or no response.3. Restate what the client says by repeating the client’s exact words; do not paraphrase or interpret.4. Avoid double-barreled or multiple questions.5. Avoid leading questions that tend to orient the person to respond in a particular direction.6. Avoid using “why” questions.

Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×

TABLE 3-1 Schedule of Markey Scholar and Comparison Group Interviews

Scholars and comparison group members were classified into cohorts corresponding to the award cycles of Markey Scholars. They were interviewed approximately 10 years after they achieved their first professional position (i.e., as an assistant professor if they remained in academia) which was assumed to have occurred within four years of the point in time in which an individual received or would have received the Markey Scholar Award. These intervals were based on the committee’s analysis of the postdoctoral experience of Scholars and were designed to ensure that all Scholars had completed their postdoctoral experience and assumed a professorial or professional position. In practice, Scholars’ years in the postdoctorate ranged from 1 to 4 years, so the interviews were completed approximately 14 years after they received the Markey Award. For comparison group members, the intervals were similar: 3 or 4 years in the postdoctorate followed by 10 years of work. Therefore, all Scholars and comparison group members were interviewed at approximately the same time in the progression of their career and had the same number of years to achieve rank, publish, and obtain extramural funding. In Table 3-1, we show the start and end points for the range of years (cycle=light gray) and the year interviewed for each of the seven cycles (indicated in the darker gray) of Markey Scholars (and comparison groups).

Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×

Analysis of the ethnographic interviews gave the committee a more nuanced view of the pathways Scholars and comparison group members chose. The interview tracked information in 12 categories:

  1. The effect of Markey award on independence as a postdoctoral fellow.

  2. The effect of the Markey award on the Scholar’s future plans when they were postdoctoral fellows.

  3. What factors influenced the first academic position.

  4. Experiences as a junior faculty member compared with those who did not have immediate support.

  5. The influence the Markey award played on future funding opportunities.

  6. Expectations for teaching responsibilities at first faculty position.

  7. Size and composition of lab.

  8. The effect of the Markey award on networking capabilities.

  9. Impact on patents, licenses, commercial interests or consultancies.

  10. Current and future interests in biomedical research.

  11. Retrospective analysis of current research and status.

  12. Orientation towards clinical research.

The focus of the interview questions was tailored specifically for Markey Scholars and comparison group members. Copies of the interview schedules used for both groups are found in Appendix F.

RESPONSE FROM SCHOLARS AND COMPARISON GROUP MEMBERS

The response to our efforts to collect CVs from and interview Scholars was almost 100 percent. With a small number of exceptions, we were able to obtain the information we sought from nearly all of the Scholars. This was expected as these individuals have had a strong connection to the Markey program over the years and our attempts to follow up with them occurred at approximately five years after the end of their Markey support.

The effort to collect information from comparison group members, by contrast, was more difficult. There were 177 nominees who were reviewed and top-ranked (the first comparison group) and 209 nominees who were classified as competitive but not top-ranked (the second comparison group). NRC staff attempted to contact all of these nominees who were reviewed by the Scholar Selection Committee. The services of Equifax, a credit reporting agency, were used to obtain current contact information for reviewed nominees. Equifax was able to supply some

Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×

TABLE 3-2 Number of Comparison Group Members, by Level of Contact

Level of Contact

Top-Ranked Candidates

Competitive Candidates

Total

Total number of reviewed nominees

177

209

386

Number with some contact data

140

154

294

Number who responded to staff inquiry

99

96

195

Number who submitted recent data

63

64

127

Number who were interviewed

51

50

101

current information—home address and/or home telephone number—on 294 (76 percent) of reviewed nominees—140 from top-ranked candidates and 154 from competitive candidates (Table 3-2). The Equifax inquiry was compromised because social security numbers were not available for any comparison group members from the first cycle of nominations and Equifax depends on the social security number to identify unique persons. As a consequence, the number of locatable persons from the first cycle of nominations for both top-ranked candidates and competitive candidates was much lower than anticipated and lower than for any other cycle. NRC staff attempted to obtain current location information on the candidates who could not be located by Equifax by using web searches and checking for affiliations of authors in relevant journals. This effort proved to be futile as only a few nominees could be accurately located and only three of the nominees who could be located responded to our inquiry; all three of those were from the top-ranked candidate group.

NRC staff attempted to contact these 294 comparison group members with moderate success. Successful contact—verifying current status and location, describing the evaluation of the Scholars and its importance for future funding policy in the biomedical sciences, and requesting participation in the evaluation—was made with 195 comparison group members (66 percent). For approximately half of the 99 nonrespondents (n = 48), the contact information we received from Equifax was incorrect (e.g., postmaster returns). Consequently, we had contact information for 99 top-ranked candidates and 96 competitive candidates. These 195 persons formed the basis for our comparison groups.

All 195 comparison group members returned current status data and, at least initially, agreed to participate in the evaluation: to submit a CV and participate in an ethnographic interview. But in fact, only 127 (66 percent) of the comparison group members actually submitted a CV and only 101 were interviewed. We were able to get some data—a CV and/or interview—from 127 of the 195 comparison group members for whom we had contact information. The percentage of top-ranked and competitive

Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×

candidates interviewed out of the total number of candidates (the cooperation rate) is 29 and 24 percent respectively.

The committee recognizes that the lower than desired response rate for top-ranked and competitive candidates might affect the validity of the evaluation. Also, the committee is especially concerned about the very low number of candidates in the initial cycles of Markey funding. These very low numbers preclude potential analyses by funding cycle.

The committee recognizes the possibility of a selection bias in the two comparison groups in that some top-ranked and competitive candidates were not available to participate in the study. Unfortunately, the committee did not have the resources to contact a sample of these nonrespondents to determine if their characteristics were similar to the characteristics of those comparison group members who did respond. However, the committee reasoned that professionally successful comparison group members would be less likely to self-select out of the study.

The committee also worried that the lower than expected number of female Scholars and comparison group members may have had some impact on the study. The percentage of females at all stages of the Markey Pathway (Table 2-4) was lower than expected. In 1988 (the middle of the Markey award period), for example, 37 percent of persons graduating with doctorates in the biological sciences were female, while only 24 percent of candidates for the Markey award were female (National Science Foundation, 1997). The committee was less concerned about the higher number of candidates with Ph.D. degrees than those with M.D. or M.D./Ph.D. degrees. In 1988, there were over 4,000 Ph.D. graduates in biological sciences (National Science Foundation, 1997). However, of the nearly 16,000 M.D.s that graduated in 1988, only 214 were in M.D./Ph.D. programs and only 540 anticipated a research fellowship (American Association of Medical Colleges, 1988).

The committee is aware that the outcome measures used in this study may have been affected by both the low response rates of persons in the two comparison groups and the lower than expected number of females among Markey Scholar candidates. The latter factor, however, would not have influenced the analytical comparisons since the percent of Markey Scholars who are women (19.5%) is similar to the percent of women in each of the two comparison groups (18.8%), as shown in Table 2-4.

Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×
Page 21
Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×
Page 22
Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×
Page 23
Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×
Page 24
Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×
Page 25
Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×
Page 26
Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×
Page 27
Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×
Page 28
Suggested Citation:"3 Evaluation Methodology." National Research Council. 2006. Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press. doi: 10.17226/11755.
×
Page 29
Next: 4 Outcomes for the Markey Scholars »
Evaluation of the Markey Scholars Program Get This Book
×
Buy Paperback | $44.00 Buy Ebook | $35.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

One of five in a series evaluating the grant programs of the Lucille P. Markey Charitable Trust, this report examines the Markey Scholars Awards in Biomedical Sciences. The Scholars program awarded more than $50 million to outstanding young investigators as postdoctorates and junior faculty. Using analysis of curriculum vitae, data on citations and grants, and interviews, the report examines the career outcomes of Scholars relative to those of individuals who applied for the Scholars award. The authoring committee concludes that the Scholars program was a success and provides a template for current programs designed to address the career transitions of young investigators.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!