Cover Image

PAPERBACK
$35.50



View/Hide Left Panel

Program Evaluation at the Robert Wood Johnson Foundation

Nancy Fishman

EVALUATION TRADITION

The Robert Wood Johnson Foundation has a tradition of evaluation to gain outside perspectives on its programs—whether demonstration programs or scholars programs. The foundation’s goals for its evaluation strategy are to learn from its programs in order to inform the field, be accountable to the Board of Trustees, and help with future programming. Independence is valued in the evaluations, and an evaluator who works outside the program is generally funded, though the foundation is also attempting to make grantees more a part of the data collection effort and analysis.

In doing evaluations or assessments the foundation hopes to improve the program, find out what works under what circumstances, and create an evidence base for social change, practice in the field, and foundation grant making.

For programs that fund scholars and fellows, the goals for evaluation are often centered around formative feedback, some measure of impact on the field, and some measure of impact on the individual. Measuring impact for these programs is a difficult task and success is not always achieved.

The foundation has gone through a strategic planning process and restructured the way it manages its scholars/fellow programs. These programs are now in a large portfolio of programs that is referred to as the Human Capital Portfolio. In the past the foundation invested in pro-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 79
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation Program Evaluation at the Robert Wood Johnson Foundation Nancy Fishman EVALUATION TRADITION The Robert Wood Johnson Foundation has a tradition of evaluation to gain outside perspectives on its programs—whether demonstration programs or scholars programs. The foundation’s goals for its evaluation strategy are to learn from its programs in order to inform the field, be accountable to the Board of Trustees, and help with future programming. Independence is valued in the evaluations, and an evaluator who works outside the program is generally funded, though the foundation is also attempting to make grantees more a part of the data collection effort and analysis. In doing evaluations or assessments the foundation hopes to improve the program, find out what works under what circumstances, and create an evidence base for social change, practice in the field, and foundation grant making. For programs that fund scholars and fellows, the goals for evaluation are often centered around formative feedback, some measure of impact on the field, and some measure of impact on the individual. Measuring impact for these programs is a difficult task and success is not always achieved. The foundation has gone through a strategic planning process and restructured the way it manages its scholars/fellow programs. These programs are now in a large portfolio of programs that is referred to as the Human Capital Portfolio. In the past the foundation invested in pro-

OCR for page 79
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation grams to strengthen the health and health care workforce, build scholarly work in new targeted fields, and foster leaders and leadership, but that work was scattered across interest areas in the foundation. The Human Capital Portfolio brings these efforts together to increase learning, enhance coordination, and promote greater effectiveness from these diverse programs. This provides the foundation with a tremendous opportunity to learn from its programs. The fellows/scholars programs in this portfolio include: Developing Leadership in Reducing Substance Abuse Generalist Physician Faculty Scholars Health Policy Fellowships Program Harold Amos Medical Faculty Scholars Innovators Combating Substance Abuse Investigator Awards in Health Policy Research Program The Robert Wood Johnson Clinical Scholars Program The Robert Wood Johnson Community Health Leadership Program The Robert Wood Johnson Executive Nurse Fellows Program The Robert Wood Johnson Health and Society Scholars Program Scholars in Health Policy Research Program Summer Medical and Dental Education Program ASSESSMENT GOALS Traditionally, assessments of the foundation’s scholar programs have funded a grantee not connected with the program to collect information from a variety of sources during a short period after the program has been in existence for a number of years. This type of assessment is considered a midcourse review of the program and does not include ongoing data collection over the life of the program. In general, information is obtained from scholars, the program office that administers the program, the national advisory committee, and a few key stakeholders in the fields of interest. Both quantitative and qualitative data are obtained. In retrospect, James Knickman, vice president of research and evaluation at the foundation, thinks that “the external assessments have provided excellent insights for improving our initiatives and they give our staff and board an objective look at the value to date and potential value of these long-term investments.” Beyond the foundation’s more traditional approach to evaluation and in lieu of randomizing scholars to its programs, a number of evaluation innovations have been attempted. Most evaluations include interviews with scholars, foundation staff, employers, and others involved in the

OCR for page 79
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation program. One evaluation, discussed below, surveyed runners-up for a program. Recently, an evaluation was structured to include people outside the realm of the project but who may interact with its scholars. An additional strategy is to contract with a number of senior scholars to review the work of the participants and comment on (1) the impact of the program on personal and professional growth, (2) the contribution of work to the participant’s academic discipline, and (3) the influence of the participant on the health policy debate. APPROACHES TO ASSESSMENT This paper looks at the experience of the foundation’s two longest standing human capital programs: the Clinical Scholars Program and the Health Policy Fellowship Program. The purpose and history of these programs are reviewed, along with evaluation methods. This section will briefly describe the two programs mentioned above and discuss the goals, methods, and nature of the evaluation outcomes. In general, midcourse assessments are done in anticipation of a request for renewal to the Board of Trustees. This is true of both of the following programs: The Health Policy Fellowships Program (HPFP) is the second-oldest and longest-running program of the Robert Wood Johnson Foundation (RWJF). HPFP was created in March 1973 and is now in its thirty-second year of operation. As of spring 2005, 183 fellows from more than 80 universities, colleges, and other health-related organizations have participated in the program. RWJF has committed more than $20 million in support of HPFP. The program has been managed since its inception by a small staff at the National Program Office (NPO) at the Institute of Medicine (IOM), part of the National Academies. During its early years, the HPFP’s primary objective was to prepare academic health professionals to assume governmental positions related to health policy. RWJF later added the goal of helping the academic health centers (AHCs) that sent fellows to HPFP. RWJF believed that providing academic health professionals with a Washington experience would help the AHCs become better equipped to interact with the federal government, to have an impact on health policy, and to be more responsive to society’s need for preventive and primary care services (as opposed to just specialty care). Over time, and especially after a formal evaluation in 1992, program administrators realized that this was an unrealistic goal given HPFP’s small size and thus shifted the emphasis again toward developing the health policy and leadership skills of individual fellows. Beginning in the mid-1990s, eligibility was extended beyond academicians to include community-based health care professionals.

OCR for page 79
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation As stated in the 2003 program brochure: “The Health Policy Fellowships Program provides an outstanding opportunity for exceptional midcareer health professionals and behavioral and social scientists with an interest in health to take part in and better understand the health policy process at the federal level. Fellows actively contribute to the formulation of national health policies and accelerate their careers as leaders in health policy.” HPFP also aims to enrich the substance of the health care policy debate at both the federal and the state levels. The program has had two outside grantees assess it and a number of in-depth reports from the national program office. 1980 Assessment Goal—The first assessment in 1980 had a stated goal of examining the impact of HPFP on the individual fellows, their home institutions, and the congressional committees to which they were assigned. Methods—The evaluator interviewed alumni of the program, academic officials who had been invited to submit nominations, congressional staff members, persons to whom the alumni reported, and persons requesting information about HPFP. The process used by the board to select fellows and the support provided by IOM staff to fellows were not included within the scope of the evaluation. Results—Overall the evaluation reported that the program was beneficial to both fellows and their home institutions. The evaluation produced a list of 10 operational issues that could be considered by foundation staff and the national program office. These issues included the nomination procedure, qualifications for the fellows, and the nature of the institutional linkage after the fellowship. Comment—Although the goal was stated as measuring impact, the assessment emphasized short-term impacts. The formative nature of the assessment supplied a number of useful suggestions on how to improve the program. As Daniel Zwick, the evaluator, commented in his report, the program impacts were likely to be seen over time and indirectly, and therefore it was premature to assess long-term impact. A number of changes were made to the program in response to this evaluation, particularly the manner in which nominations were handled. 1992 Assessment Goal—This evaluation addressed the following three questions: Should RWJF continue the Health Policy Fellowships Program? How could the fellowships be improved while keeping the current goals and

OCR for page 79
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation structure intact? And if RWJF were to remake HPFP from scratch, what alternative goals and structures might make sense at this time? Methods—Responses to these questions were based on a comprehensive study and analysis of the history, structure, and performance of the HPFP during the 19 years since it was founded. The study included personal interviews with more than 30 individuals knowledgeable about HPFP; surveys of all alumni, unsuccessful finalists in the competition for HPFP (as a comparison group), and leaders of a sample of AHCs; reviews of program documents; and inquiries into the histories and performance of analogous fellowships. Products—This evaluation included a survey of unsuccessful finalists and provided insight as to the potential impact of the fellowships on the rates of academic promotion, publication, service in government-appointed offices, involvement in academic health policy activities, and community and government affairs related to health care. The extensive evaluation report included impact analyses on fellows, their home institutions, and their fellowship placement. Results were included in a report to the foundation and a journal article. Table 1 is an example of the data collected. Comments—This is one of the few times that a foundation evaluation has contacted unsuccessful finalists for interviews. Although the sample has obvious potential for bias, there is concrete information to be gained. This study did assure the foundation that alumni of this program were more oriented toward health care policy than their peers. TABLE 1 Leadership and Administrative Roles of Fellows and Finalistsa Leadership/Administrative Role Fellows Finalists Chairman of Department 33 percent 25 percent CEO of Hospital 4 0 President of Professional Organization 21 6b Any leadership role in professional organization 73 78 Dean of Professional School 15 10 Member of IOM 7 6 Member of NAS 2 2 Any leadership role 57 37b aBlumenthal, D., G. Meyer, J. Edwards. 1992. The Robert Wood Johnson Foundation Health Policy Fellowship: An Evaluation Report, Health Policy Research and Development Unit Massachusetts General Hospital, December 3. bStatistically significant (p ≤ 0.05).

OCR for page 79
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation 1999 Assessment Goal—This assessment was conducted by the staff of the HPFP program office to provide information for an upcoming renewal proposal. Methods—Surveys were sent to (1) all HPFP alumni, (2) key congressional staff persons who had mentored at least two fellows in their offices in recent years, and (3) other selected individuals who had maintained a special relationship with HPFP. In some cases they conducted structured interviews with individuals in these groups, either in addition to or in place of the individuals completing the survey. More than 75 percent of HPFP alumni and 60 percent of senior Capitol Hill staff members responded to the survey. Survey questions focused not on the intrinsic merits of the fellowship but instead on how, and to what extent, HPFP was being affected by major changes in the health care marketplace and policy environment. The survey asked respondents to select the four issues they believed to be most important to the future of the HPFP and to rank those issues in order of priority. Respondents were also asked to provide ideas or suggestions on how to address any of the issues listed and to identify any other issues they thought were relevant to the assessment but were not listed in the survey. Products—Based on the survey’s findings, a report was developed by the National Program Office, Issues for the Future of the Robert Wood Johnson Health Policy Fellowships Program, which identified seven major issues relevant to the future of HPFP. Comments—As another renewal for this program approaches, the foundation is in the process of thinking about what it needs to know about this program. To start thinking about how to frame this project, interviews will take place with he program officer in charge of the program, senior management at the foundation, and the staff of the National Program Office. THE CLINICAL SCHOLARS PROGRAM The Robert Wood Johnson Clinical Scholars Program provides postdoctoral training for young physicians interested in research and leadership careers in health policy and academic medicine. Originally authorized by the Board of Trustees in 1972, it is the oldest national program of the foundation. The program aims to produce scholarly physician-leaders with the understanding and skills necessary to have a major influence on health care policy and to help create and build the field of health services research. The core curriculum introduces scholars to basic nonbiology disciplines and methods used in health care research, along with other

OCR for page 79
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation courses that reflect each institution’s strengths and faculty interests. As of spring 2004, there were seven participating schools (University of California, Los Angeles [UCLA], School of Medicine; University of Chicago Division of the Biological Sciences; the Pritzker School of Medicine, University of Michigan Medical School; University of North Carolina at Chapel Hill School of Medicine; University of Washington School of Medicine; and Yale University School of Medicine). The last group of students participating in the program at these seven schools entered in July 2004 and completes the program in July 2006. Beginning in July 2006, four schools will train clinical scholars (UCLA School of Medicine, University of Michigan Medical School, University of Pennsylvania Health System, and Yale University School of Medicine). There have been two assessments of this program. 1991 Assessment Goal—In 1991, RWJF awarded a grant for a year-long assessment, aimed at answering the following questions: What are the contributions of the Clinical Scholars Program to the careers of the scholars and to the fields in which they work? What is the current and likely future demand for individuals with clinical scholar training? What is the demand by young physicians for clinical scholar training? How can the Clinical Scholars Program be improved? This included considerations of the following: (1) the policy requiring each site to adopt priority areas, (2) the feasibility of linking priority areas to RWJF goals, (3) expansion or contraction of the number of program sites, (4) expansion of the program fellowship to a (perhaps optional) third year, and (5) policies for encouraging the diffusion of clinical scholars more widely within the medical research community. Methods—The assessment team reviewed documents and program records, interviewed key informants, and conducted information-gathering visits at each program site. Products—This assessment produced a report to the foundation. The program was as a whole found to be highly successful. A number of recommendations were made to strengthen the structure and function of the program. Among other outcome indicators, the reviewers sited an analysis demonstrating that nearly one-fourth of all articles written by authors young enough to have been eligible for the program and that

OCR for page 79
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation were published in the journal Medical Care from January to September 1991 were authored by former clinical scholars. Comments—This review was very positive in general and was used for a number of years as a reference and guide for program change. 2002 Assessment Goal—This project was not, per se, a classic program assessment and was promoted more as a method to understand the market for the program for future cohorts. Methods—The project consisted of two surveys. The first, fielded in December 2000, queried current and former participants in the program about (1) why they had applied to the program, (2) their experience in the program, (3) their experience after completing the program, (4) how the program affected their careers, and (5) their suggestions about possible improvements to the program. Project staff sent 862 surveys to current and former scholars. The overall response rate was 49 percent. The second survey, fielded in 2001, queried medical residents about their interests in fellowship training in general and the Clinical Scholars Program in particular. It elicited information from residents about (1) their career goals and options; (2) whether they were considering applying to a fellowship program after residency; (3) if so, the type and characteristics of a fellowship program to which they might apply; and (4) personal or other circumstances that will or may affect their career paths. To avoid biasing the survey in regard to residents’ perceptions of the program, the introduction to the survey described it as a survey about “career decisions,” and the Clinical Scholars Program was mentioned as one type of program to which residents might want to apply. Project staff mailed 400 surveys to second-year residents identified through a list obtained from the American Medical Association, and distributed another 5,380 surveys to 1,076 residency directors, asking them to request that their second- and third-year residents complete and return them. The fielding yielded 513 surveys from the targeted respondents (an 8.9 percent response rate). Products—Much useful information was gained, and the report fed into a process that led to a major change in the program. However, the response rate for the residents reflected the difficulty of gaining information from people not in a program. By querying clinical scholars on their career goals from all years of the program some interesting trends could be seen (see Table 2). Comment—These data, along with input from a national advisory committee, were used to formulate major changes to the program, includ-

OCR for page 79
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation TABLE 2 Career Goals at Entry into the Clinical Scholars Program by Decade Subject 1970s N=97 (%) 1980s N=121 (%) 1990s N=161 (%) Current N=45 (%) Total N=424 (%) Primary Care 42 34 27 28 33 Policy 9 7 14 27 12 HSR 13 12 13 16 13 Public Health 12 9 7 2 8 Subspecialty 18 22 27 22 23   SOURCE: Showstack, J., A. A. Rothman, N. Greene. 2002. Final Report: Survey of the Market for the Clinical Scholars Program, Institute for Health Policy Studies University of California, May 9, 2002. ing the reduction in the number of sites, addition of a leadership component, and a focus on community participatory research. John Showstack and Arlyss Anderson Rothman made these comments in their report: The data show clearly that many residents are unaware of possible options for fellowship training. Although the responses suggest that residency faculty provide some information, it does not appear that residency programs offer counseling for participants, and even basic information about fellowship opportunities appears to reach only a portion of residents. Among the subset of residents who are considering fellowship training the vast majority say they will choose a subspecialty fellowship (and, presumably, leave primary care). After devoting many years of education and training to the profession of medicine, these bright young physicians appear to be fatigued, financially extended, and to have received little information regarding fellowship training. In addition, fellowship programs appear to do little marketing to inform and attract potential fellows, especially programs for generalists. These characteristics may lead to small applicant pools for generalist fellowships, and a group of talented young physicians who make less than fully informed decisions regarding their future. These data provide a strong framework and incentive for developing fellowship marketing programs if there is a desire to increase the applicant pool for a generalist or primary care fellowship. USE OF EVALUATION DATA A core issue for these, and all of the foundation’s human capital programs, is whether they really changed the career trajectory of the partici-

OCR for page 79
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation pants. Do the best and the brightest need us? Can we ever really understand the impact of these programs? Although important, these questions are often sidestepped because of a lack of information on the cohort that did not participate. The information from the foundation’s evaluations is most often formative in nature and used to help inform the next phase of the program. Sometimes that includes major changes, but more often the program is fine-tuned to reflect what the staff learned from the evaluation. Often, the foundation staff and Board of Trustees have multiple goals for a scholars program. Generally, the interest in building a field with trained scholars takes precedent over building leadership capacity within a group of scholars, but these two goals most often coexist. This fact provides a challenge for the evaluation team. Often, time needs to be spent with foundation staff helping to articulate the major goals for the program. New Approaches To improve the flow of information both to and from our alumni scholars, RWJF is in the process of establishing an online tracking initiative. This project will establish a system that keeps information about RWJF scholars and fellows up to date and available to both the national program offices and the foundation. A Web-based system is being planned that will query scholars/fellows once a year and be accessible to program offices to update information throughout the year. This information will be used to help identify topic areas that these scholars are working in and hopefully the scholars in the foundation’s current work. In addition, this information will allow the foundation to track the career trajectories of this group of grantees. Consistent monitoring of a few key variables over time (e.g., position, publications) will provide tremendous insight into the impact of RWJF’s programs. ACKNOWLEDGMENTS Many thanks are offered to the evaluators of these programs and the Grants Results Report Unit at RWJF for extensive information on its programs and evaluations.