Biomedical and Behavioral Research Scientists: Their Training and Supply Volume III: Commissioned Papers (1989) / Chapter Skim
Currently Skimming:
1. Evaluating the National Research Service Award Program: A Review and Recommendation for the Future
1. Evaluating the National Research Service Award Program: A Review and Recommendation for the Future
Pages 1-40
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 1... ...
Then, previous evaluation efforts will be discussed in terms of whether they addressed these questions and provided answers that could be viewed with a reasonable degree of confidence. The ''match"' between the questions of interest and the avaiJability.of sound evidence for answering these questions as gleaned from previous evaluations to date will serve as the basis for recommending future evaluation priorities.
|
From page 2... ...
. Consequently, previous evaluations in the NRSA program differ in terms of the specific program of interest, the target populations examined, the training activities invest ved, and the outcomes studied.
|
From page 3... ...
In addition, the evaluation questions of most interest to a particular group often depend on the extent of its nature with the NRSA program. For example, agency staff whose major responsibility lies in administering institutional training grants may be most enthusiastic about collecting data that could improve their ability to monitor and guide programs; in contrast, scientific societies' demands may stem primarily from their desire to develop stronger arguments for increased NRSA funding in their respective discipliners)
|
From page 4... ...
DEMAND FOR THE PROGRAM AND LEVELS OF PARTICIPATION Having definitive data on the need for research training support and levels of participation in NRSA programs are "bottom line" demands of all major stakeholders. For example, both the authorizing and reauthorizing legislation for the NRSA (e.g., P.L.
|
From page 5... ...
Many of these questions have been spurred by disappointment in the low rates of participation by women and ethnic minorities in science, coupled with a concern that the nation's pool of scientists and engineers may prove inadequate to meet future challenges (e.g., Office of Technology Assessment 1985; Vetter, 1989~. PROGRAM CHARACTERISTICS AND OPERATION Of primary interest to federal agency staff who administer NRSA programs and policies are questions related to program functioning.
|
From page 6... ...
100-607~; NAS is directed to "assess Current NRSA programs] and other current training programs available for .
|
From page 7... ...
When questions as to the effectiveness of NRSA programs do surface, they typically center around issues of relative effectiveness. For example, data that can "tease out" the effects of NRSA programs in producing biomedical researchers relative to the performance of other research training programs with similar goals are deemed more salient than evaluations aimed at understanding whether NRSA training is more effective than no research training at all or research training that is entirely financed by the individual through loans or other personal sources.
|
From page 8... ...
Although such questions are important for guiding and improving future evaluations of the NRSA program and can indeed be addressed by wel1-designed studies, it must be kept in mind that providing answers is neither the sole responsibility of NRSA nor the evaluation efforts connected with this program. Where Improvements in Evaluation Activities Are Needed?
|
From page 9... ...
presented information on one or more outcomes for programs or program elements, and 38 percent (N = 6) attempted to address in some way the effectiveness of NRSA programs or distinct components.
|
From page 10... ...
. Answers to these questions also are requested in the charge for evaluation specified in the authorizing and reauthorizing legislation for the NRSA program: to "assess current MESA programs]
|
From page 11... ...
Of particular interest here is predoctoral and postdoctoral research training paid by research grants to individual investigators. Available data suggest that the use of this mechanism in supporting research training is not infrequent; research assistantships paid by federal and other grants were a source of predoctoral support for 16 percent of the 1987 Ph.D.
|
From page 12... ...
study of NIH postdoctoral appointments. Data on race/ethnicity typically have appeared only in internal program reports or evaluations of the Minority Access to Research Careers 12
|
From page 13... ...
universities, which includes those who received NRSA support, also is collected by the SED. However, gaining access to and merging these files with PHS data require resources, and monies typically become available only when large-scale evaluation studies, which focus on outcomes, are commissioned.
|
From page 14... ...
" This same situation is even more characteristic of applicants to NRSA programs, although the issue is somewhat more complicated. "Applicants" include individuals who apply for NRSA fellowships and institutions that apply for NRSA training grants.
|
From page 15... ...
, in their evaluation of MARC undergraduate training grants, also obtained qualitative information gleaned from site visits on such operational issues as the activities on which MARC funds were spent, departmental composition, and recruitment practices. At the same time, a profile of how programs function in terms of recruitment, selection, and actual training activities-issues currently of interest to major constituencies -- is not available for the majority of NRSA programs.
|
From page 16... ...
For example, congressional policymakers want to know if NRSA programs are more effective than research training programs administered by NSF. Agency staff want answers to such questions as "Is predoctoral or postdoctoral training more effective in producing researchers in the clinical sciences?
|
From page 17... ...
~; and employment data from reports submitted by awardees after completion of their NRSA appointments in order to fulfill payback requirements. This reliance on archival data has at least partly resulted from the constraints imposed by limited funding for evaluation, short timelines for reporting, and OMB regulations for data collection efforts contracted by federal agencies.
|
From page 18... ...
In general, the results indicated that NIH awardees distinctly outperformed their comparisons in terms of greater involvement in research (e.g., receipt of additional postdoctoral research training, time spent in research, and grant application/award activity)
|
From page 19... ...
Awardees in the biomedical sciences also were much more likely to have spent the first year after their doctorate in postdoctoral study (65.2 percent versus 47.9 percent) in contrast to behavioral science Ph.D.s (15.6 percent versus 10.4 percent)
|
From page 20... ...
While such results do suggest that NIH predoctoral support, at least in the biomedical sciences, increases the probability that an individual will have a research career in health-related areas, it nonetheless remains a small role. Analyses regressing years since Ph.D., the quality of the predoctoral institution, and total months of NIH predoctoral support on number of NIH research grant applications, average priority score awarded to 20
|
From page 21... ...
Given that individuals who choose to undergo the additional years of training involved in postdoctoral appointments may share certain characteristics that are distinct from those of individuals who do not engage in postdoctoral study, comparing outcomes of NIH postdoctoral awardees with those who had their postdoctoral training sponsored by other agencies is more 21
|
From page 22... ...
found, NTH awardees continued to outperform in terms of grant application activity those individuals whose postdoctoral training was supported via another source (56.9 percent versus 34.5 percent) ; Also, they were more likely to have been awarded a grant (40.0 percent versus 22.3 percent)
|
From page 23... ...
Looking at 1974 M.D.s only, there were differences between this group and NIH postdoctoral trainees and fellows. For example, those-M.D.s with NIH-supported postdoctoral training also were slightly more apt to have applied for NIH/AD~HA research grants (18.6 percent versus 12.0 percent)
|
From page 24... ...
GAPS IN UNDERSTANDING OUTCOMES AND EFFECTS OF NRSA PROGRAM COMPONENTS The NRSA program is comprised of several different programs and/or components. For example, there are two basic award mechanisms -- individual fellowships and departmental training grants.
|
From page 25... ...
Other distinct NRSA program components also have been evaluated. One program involves the institutional training grants supported by MARC t Honors Undergraduate Research Training Grants)
|
From page 26... ...
. GAPS IN BASIC EVALUATIVE DATA FOR PROGRAMS IN SPECIFIC FIELDS/RESEARCH PROBLEMS: The large majority of evaluation studies have focused on NRSA programs administered by NIH.
|
From page 27... ...
RECOMMENDATTON8 FOR FUTURE EVALUATION ACTIVITIES The major "gaps" described above imply that any portfolio of future evaluation efforts for the NRSA program should consider the following issues: The quality of the major data bases on NRSA appointments should be assessed so as to ensure that information on program recipients covers the key characteristics of most interest, is accurate, and is collected uniformly on all NRSA components. O A core set of evaluative data to be collected for all research training programs funded by NRSA should be identified.
|
From page 28... ...
ENSURING A CORE SET OF EVALUATION ACTIVITIES FOR ALL NRSA PROGRAMS To date, evaluation activities for programs in certain fields or areas (e.g., the behavioral sciences and health services research) have been minimal and haphazard.
|
From page 29... ...
Only in this manner can the scope and breadth of NRSA activities be examined and strategies developed to improve programs aimed at all fields, levels, and types of research training supported by NRSA funds. INITIATING EVALUATION ACTIVITIES THAT PROVIDE BETTER INFORMATION ON PROGRAM PARTICIPATION AND OPERATION At present, it is difficult to characterize what is happening in NRSA programs, regardless of the type, level, or field of training.
|
From page 30... ...
Finally, if such efforts were extended to a sample of predoctoral students or postdoctoral associates without traineeships in the same academic program and possibly to students in similar programs in departments without training grants, we would begin to develop a better sense of the strength and integrity of various NRSA "treatments" (e.g., whether NRSA programs provide training experiences sufficiently different from those received by students in the same program or in other programs)
|
From page 31... ...
Needless to say, the ability to ultimately provide answers to questions seeking evidence on effectiveness is hampered by our lack of knowledge about how individuals are selected to receive NRSA support. INCREASED ATTENTION TO A8SE88ING PROGRAM OUTCOMES Those associated with NRSA programs, particularly at the program administrative level, want to know what works best in research training, either in terms of specific training models (e.g., broad versus specialized training)
|
From page 32... ...
found that recipients of the KO-6 awards were more likely to receive PHS training grants -- a relevant outcome, based on the request made by Congress for obtaining information on the number of awardees who engage in health research or teaching as a career (P.L.
|
From page 33... ...
In conjunction with these research programs, efforts need to be targeted at developing new measures of scientific productivity and quality of research and testing their feasibility. Work on assessing the quality of training programs, apart from the accomplishments of their graduates, is another component that requires attention, particularly if future evaluation efforts aimed at judging the quality of training grant recipients (i.e., programs/departments within institutions)
|
From page 34... ...
Further, questions have been raised about the fact that the total expenditures per graduate for MST programs have been increasing over the last decade and are now significantly higher than for graduates supported by other predoctoral training grants (National Research Council, 1983~. Well-designed evaluations of the MST program, focusing on the outcomes of its graduates and its effectiveness on certain outcomes and as compared to other research training alternatives for physician-scientists, would do much towards addressing these issues.
|
From page 35... ...
1987. An Evaluation of the NIH Research Career Development Award.
|
From page 36... ...
Minority Access to Research Careers: An Evaluation of the Honors Undergraduate Research Training Program. Washington, D.C.: National Academy Press.
|
From page 37... ...
1982. Postdoctoral training in bioscience: Allocation and outcomes.
|
From page 38... ...
1989a. Minority Access to Research Careers Program: Briefing on Program Activities, FY 1980 to 1986.
|
From page 39... ...
1979. Evaluation of the Annual Manpower Survey of NIH Research Grants.
|
From page 40... ...
Zumeta, W
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.