Summary

In an effort to improve the scientific foundation of the Environmental Protection Agency (EPA) decision-making process, EPA’s Office of Research and Development (ORD) created the Science To Achieve Results (STAR) program in 1995. The STAR program is a competitive, peer-reviewed, extramural research grants program intended to increase the agency’s access to the nation’s best scientists and engineers in academic and other nonprofit research institutions. It supports research pertaining to human health and the environment and is designed to maximize the independence of the researchers supported and to provide an equal opportunity for all researchers to qualify for support.

The STAR program, currently managed by ORD’s National Center for Environmental Research (NCER), is integrated into EPA’s overall research program through extensive planning and consultation with the agency’s other research centers and laboratories and its program and regional offices. The research sponsored by the STAR program allows the agency to fill information gaps that are not addressed completely by its intramural research program and to respond to new issues that the EPA laboratories are not able to address.

The research support awarded by the STAR program is of three main kinds: grants awarded to individual investigators or small groups of investigators, grants awarded to multidisciplinary (and sometimes multi-institutional) research centers, and fellowships to support graduate work (at the master’s and Ph.D. levels) in environmental sciences. The program has been funded at about $100 million per year over the last few years and accounts for 15-20% of ORD’s research budget. The program has leveraged its funds by forming partnerships with other agencies that support



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Summary In an effort to improve the scientific foundation of the Environmental Protection Agency (EPA) decision-making process, EPA’s Office of Research and Development (ORD) created the Science To Achieve Results (STAR) program in 1995. The STAR program is a competitive, peer-reviewed, extramural research grants program intended to increase the agency’s access to the nation’s best scientists and engineers in academic and other nonprofit research institutions. It supports research pertaining to human health and the environment and is designed to maximize the independence of the researchers supported and to provide an equal opportunity for all researchers to qualify for support. The STAR program, currently managed by ORD’s National Center for Environmental Research (NCER), is integrated into EPA’s overall research program through extensive planning and consultation with the agency’s other research centers and laboratories and its program and regional offices. The research sponsored by the STAR program allows the agency to fill information gaps that are not addressed completely by its intramural research program and to respond to new issues that the EPA laboratories are not able to address. The research support awarded by the STAR program is of three main kinds: grants awarded to individual investigators or small groups of investigators, grants awarded to multidisciplinary (and sometimes multi-institutional) research centers, and fellowships to support graduate work (at the master’s and Ph.D. levels) in environmental sciences. The program has been funded at about $100 million per year over the last few years and accounts for 15-20% of ORD’s research budget. The program has leveraged its funds by forming partnerships with other agencies that support

OCR for page 1
similar kinds of research. Since it was established, the components and management of the program have adapted in response to changing agency needs, experience gained in operating the program, and external reviews. CHARGE TO THE COMMITTEE In 2000, EPA asked the National Research Council to conduct an independent assessment of the STAR program. In response, the Research Council established the Committee to Review EPA’s Research Grants Program, which prepared this report. The committee was given the following task: The NRC committee will conduct a program review of EPA’s Science To Achieve Results (STAR) competitive extramural research grants program. Using information to be obtained from EPA, STAR grant recipients, and other sources, the committee will assess the program’s scientific merit, its demonstrated or potential impact on the agency’s policies and decisions, and other program benefits that are relevant to EPA’s mission. The committee will recommend ways to enhance the program’s scientific merit, impact, and other benefits. In the context of other relevant research conducted or funded by EPA, and in comparison with other basic and applied research grant programs, this assessment will address the STAR program’s research priorities, research solicitations, peer-review process, ongoing research projects, results and dissemination of completed research, and other aspects to be identified by the committee. In undertaking its review, the committee held three public sessions in which it heard presentations about the STAR program by EPA officials and others. The EPA officials represented NCER and other EPA research and program offices. The public sessions included presentations by representatives of other federal agencies that support extramural research and by experts in evaluating research programs. Committee members also interviewed STAR project officers and STAR grant and fellowship recipients and attended STAR sponsored workshops and meetings. NCER staff provided the committee with substantial amounts of information regarding the operation and financing of the program.

OCR for page 1
THE COMMITTEE’S EVALUATION The committee’s evaluation of the STAR program focused on the program’s quality, relevance, and performance as described in the recent Office of Management and Budget (OMB) guidelines on evaluating research programs. The committee used metrics that grew out of its review of information available from EPA and of metrics used by EPA and other organizations. The metrics, which are both quantitative and qualitative, assisted the committee in forming judgments regarding the scientific merit of the program and its impact on the agency. The committee recognizes that the STAR program is still too young to provide all the information needed for a full evaluation of the extent, impact, and value of its activities. Evaluation of research results is difficult and requires substantial elapsed time; for a given topic, it can take 3-5 years from the initiation of laboratory or field experiments to the analysis and publication of results. Considerably more time must elapse to realize the impact of published research on the scientific and regulatory communities. Nevertheless, the committee judged that it had sufficient information to evaluate how the STAR program operates and its value to the nation’s overall environmental research and management efforts. To effectively communicate its findings in this summary, the committee developed and addressed a series of specific questions that it believed would be of greatest interest to the audience of this report. On the basis of its evaluation, the committee unanimously arrived at the following conclusions and recommendations. Should the STAR program continue to be part of the ORD research program? Finding. EPA requires a strong and balanced science and technology research program to fulfill its mission properly. The STAR program is an important part of the overall EPA research program. Several previous reports by EPA and the National Academies have addressed the question of whether EPA should have its own research program or rely on research results developed elsewhere. Those reports all concluded emphatically that EPA needs its own strong research program to meet the needs of its mission. The committee concurs with that conclusion.

OCR for page 1
The STAR program is EPA’s preeminent program that solicits independent scientific and technologic research from the nation’s best academic and other nonprofit research institutions. The program has established and maintains a high degree of scientific excellence. By funding the majority of its research efforts through broadly advertised, competitive grants, the STAR program provides the agency access to independent information, analyses, and perspectives. The research portfolio of the STAR program is derived directly from the strategic plans of EPA and ORD and from ORD’s more-detailed research strategies that address particular topics. It is an integrated part of EPA’s research program. The STAR program provides the agency access to a broad community of researchers, allows it to fund research at the cutting edge of science, and assists it in addressing information gaps that it does not have the internal resources to address properly. The STAR program also encourages its grantees to disseminate their research results widely to promote their rapid and widespread use. For all those reasons, STAR research effectively expands the nation’s scientific foundation for protecting human health and the environment. Moreover, by expanding environmental research and analysis capabilities in many of the nation’s academic and other nonprofit research institutions and by supporting young scientists interested in environmental research, the STAR program actively expands the nation’s environmental-science infrastructure. Recommendation. The STAR program should continue to be an important part of EPA’s research program. What is the unique contribution of the STAR program? Finding. The STAR program funds important research that is not conducted or funded by other agencies. The STAR program has also made commendable efforts to leverage funds through establishment of research partnerships with other agencies and organizations. The STAR program provides EPA with access to independent research that is directly relevant to its mission. The program makes strong efforts to ensure that the results of its research are expeditiously communicated to relevant EPA program offices and to other potential users. The STAR pro

OCR for page 1
gram gives primary potential users of research results a unique role in helping to plan the research and to identify the specific high-quality proposals that will be of greatest value to them. The exploratory and core research that the program sponsors alerts the agency to possible emerging issues, providing more opportunity for the agency to consider how it might best address them. Much of the research funded by STAR would not have been undertaken without the program, because it is not conducted or funded by other agencies. For instance, EPA is one of the few agencies that provide extramural funding for examining the impacts of endocrine disruptors on ecosystem processes. The STAR ecologic-indicators program is the primary source of support of research on the development of water-quality indicators for biologic monitoring. The interdisciplinary centers that STAR has supported also represent an innovative approach to supporting research that will be most relevant for environmental decision making in several important topics. Finally, the STAR program has been successful in working with other agencies that have similar or complementary research interests through research partnerships and in obtaining supplementary funding. That not only leverages additional funds for research projects of interest to STAR but also helps to increase the partner agencies’ awareness of the pertinent issues and information needs of EPA. The STAR program’s ability to establish partnerships has increased as more funds have been allocated to it. Recommendation. STAR should continue to partner with other government and nongovernment organizations to support research of mutual interest and of relevance to EPA’s mission, explore innovative approaches for carrying out this research, and sponsor a diverse portfolio of research that alerts the agency to emerging issues and provides independent analyses of issues that the agency is currently addressing. Does the STAR program have adequate processes to ensure that it is sponsoring high-quality and relevant research? Finding. The procedures that STAR has established for soliciting and selecting the highest-quality research proposals compare favorably with the procedures established by other research agencies. STAR’s procedures for incorporating mission relevance into its research-planning process and in

OCR for page 1
the selection of proposals to fund exceed those practiced by most other agencies. The STAR program has developed a grant-award process that compares favorably with and in some ways exceeds that in place at other agencies that have extramural research programs, such as the National Science Foundation (NSF) and the National Institute of Environmental Health Sciences. An unusually high degree of planning goes into identifying the specific research subjects to be supported. The agency also puts considerable time and thought into preparing effective research solicitations and into funding projects that are relevant to its mission and program needs. EPA spends substantial effort in defining its research agenda, and the STAR program submits its proposed requests for applications (RFAs) to extensive review within the agency. Those efforts are intended to ensure that the research requests are focused on the issues most important to EPA. However, the STAR program makes insufficient use of outside experts in planning its research agenda and in identifying the most important gaps in scientific knowledge. As a result, some of its early RFAs were not as well focused as they should have been. In soliciting research proposals, STAR makes a substantial effort to reach out to the broad scientific community and to attract the most capable scientists. The RFAs are distributed widely through EPA’s Web site, the Federal Register, announcements at professional meetings, and e-mail distributions to individuals or institutions that sign up on the STAR Web site. When the desired research is outside EPA’s traditional research fields and might therefore include scientists not already involved with the agency’s research program, STAR often solicits the help of other agencies that traditionally work with these scientists to ensure that they are aware of the funding opportunities. The STAR program has established a rigorous peer-review process. Such peer-review processes are a key part of the foundation on which excellence is achieved in all research programs, including those of the National Institutes of Health (NIH) and NSF. The agency has taken effective steps to ensure that the process does not suffer from conflicts of interest and is independent. EPA provides a “firewall” that shields the peer-review process from the influence of the project officers and staff who oversee the individual-investigator, fellowship, and center awards. Recommendation. The STAR program should continue to improve the focus of its RFAs, and when the agency does not have the capacity inter

OCR for page 1
nally to adequately define the state of the science in a particular research field, STAR should consider greater use of external experts to assist in identifying the highest-priority research and data gaps. Is the STAR program producing high-quality research results? Finding. Although it is still too early for comprehensive evaluations of the research results of the STAR program, some STAR research efforts have already substantially improved the scientific foundation for decision making, and the results produced by STAR investigators have been widely published in peer-reviewed journals. Evaluating the quality of research results is difficult and necessarily involves substantial judgment on the part of scientists with expertise in the research fields being reviewed. In addition, because of the relative youth of the STAR program, only about 40% of STAR research projects funded to date have been completed. However, many STAR projects have resulted in articles in highly respected, peer-reviewed journals—a traditional measure of research quality. These STAR research results have already helped to improve our understanding of the causes, exposures, and effects of environmental pollution—information critical to improving the scientific foundation for decision making. For instance, STAR-funded research on particulate matter has helped to improve our understanding of the biologic mechanisms by which inhaled ambient particles cause health effects and the nature of some of those effects. These data are critical to future regulatory decisions regarding our nation’s ambient air quality. A limited bibliometric analysis by the committee indicated that the citation rate of STAR-supported research is comparable with that of research in the same fields funded by other research organizations and undertaken by other investigators. For instance, in 1997, the average number of citations of STAR-funded ecologic research was 10.5, compared with 10.3 citations of the work of all other investigators in ecology. The committee also reviewed the backgrounds and accomplishments of a sample of STAR-funded principal investigators. The review indicated that the STAR program was funding many scientists with outstanding credentials; they have impressive research track records and are leaders in their fields; are editors of journals or officers in societies and have received awards of distinction; and were attracted to the STAR program from fields outside EPA’s mission.

OCR for page 1
On the basis of the STAR program’s process for awarding grants, the quality of the individuals and institutions funded by the program, and the highly competitive nature of the awards, the committee is confident that the products of STAR grants are of the highest quality. Recommendation. EPA should continue its efforts to attract “the best and the brightest” researchers to compete for STAR funding. Are the STAR program results useful for EPA decisions and processes? Finding. The STAR portfolio effectively supports EPA’s mission, Government Performance and Results Act goals, and ORD strategic plans. Specific STAR research projects have yielded significant new findings and knowledge critical for regulatory decision making. The STAR program is too young to be able to document fully the extent to which its research results are being used to inform development of new regulations and environmental-management decisions. Even with respect to projects that have been completed, there is often a substantial delay between when the research results are produced and the agency decides to undertake rule-making or other actions to address the issues that were studied. However, some STAR projects have already yielded information important for environmental decision making. For example, STAR-sponsored research in endocrine disruptors, particulate matter, and ecologic assessment has resulted in groups of peer-reviewed publications of immediate use in understanding causes, exposures, and effects of environmental pollution. Those results are directly relevant to EPA’s mission to “protect human health and to safeguard the natural environment—air, water, and land—upon which life depends.” For instance, STAR-funded research on particulate matter has helped to improve our understanding of the biologic mechanisms by which inhaled ambient particles cause health effects. Research on ecologic indicators has led to the development of a dynamic, economically linked model to evaluate the driving forces and ecologic consequences of land-use change. In research fields in which EPA does not already have substantial expertise, the committee suggests that the program consider bringing in outside experts to assist in assessing the state of the science while the research program is being planned and then to synthesize the contributions of the

OCR for page 1
STAR-supported research when it has been completed. Such assessments would help EPA to target RFAs and then analyze the utility of the completed research in providing critical knowledge or otherwise strengthening and improving the foundation for environmental decision making. To ensure the usefulness of STAR research results, it is also important for the STAR program to maintain a balanced research portfolio, including balances between “core” and “problem-driven” research and between human health and ecologic research. Recommendation. The STAR program and ORD should develop mechanisms for documenting the extent to which its research is being used to support the agency’s environmental decision making, should consider using outside experts to help document systematically the “state of the science” before research is initiated, and should synthesize the results of the research when it is completed to identify the specific contributions that STAR and ORD research has made to providing critical information. Is the STAR program effective in providing results relevant to the appropriate audiences? Finding. The STAR program has been commendably aggressive in experimenting with innovative approaches to communicating the results of its funded research to a wide variety of users and audiences, but its success in these efforts has been uneven. The STAR program supports research of potential value to a variety of users and audiences, both in and outside EPA. Much of the research is aimed primarily at the scientific community and those responsible for providing technical support for environmental-management decisions. For the scientific community, the primary communication product is peer-reviewed journal articles, and the program has been successful in encouraging the preparation of these articles. The program, however, also has other potential users, at least for the results of some of its research. They include other federal agencies; industry; state, tribal, and local governments; nonprofit environmental organizations; and international environmental agencies. The audience for some projects appears to be local communities (for instance, communities that have received Environmental Monitoring for Public Access and Community Tracking, or EMPACT, grants) or the general public; disseminating results to such audiences is much more difficult.

OCR for page 1
The STAR program has experimented aggressively with a wide variety of communication mechanisms. Information is available to the public on EPA’s Web site concerning every step of the STAR process, from the initial solicitation of research proposals, through the award of grants, to the final research results. STAR researchers are required to prepare annual progress reports, which are made available to the public in summary form. The STAR program also produces several series of reports that summarize research results in selected topics. In all those efforts, the program appears to substantially exceed the dissemination efforts of most other researchsponsoring organizations, both in and outside the federal government. Nevertheless, the STAR program could substantially improve its dissemination efforts by directing its communication efforts more effectively to specific users and audiences. The program does not always clearly identify the users and audiences for its research results. Often, the research results are produced, and then EPA assesses how to communicate them. The dissemination process would be much more effective and efficient if the potential audiences were clearly identified before the research began and if the focus were maintained throughout the research process and the preparation of reports. In some cases, the effective dissemination of results should be primarily STAR’s responsibility. In other cases, STAR’s contributions will be a component of a larger research effort, and the primary dissemination responsibility should lie with ORD or EPA. In all cases, however, dissemination efforts are likely to be more effective if the intended audiences are clearly defined from the beginning of the STAR grants process. Recommendation. The STAR program should clearly identify the intended audiences for proposed research results as early in the process as possible and should identify the audiences in RFAs. When appropriate, EPA should consider involving representatives of the intended audiences from outside the agency in helping to define the relevant research results and the strategy for their dissemination. Should the fellowship program continue to be part of the ORD research program? Finding. The STAR fellowship program is a valuable mechanism for enabling a continuing supply of graduate students in environmental sciences

OCR for page 1
and engineering to help build a stronger scientific foundation for the nation’s environmental research and management efforts. The fellowship program was established to “encourage promising students to obtain advanced degrees and pursue careers in environmentally related fields” and to develop the next generation of environmental scientists. It is the only federal fellowship program exclusively designed for students pursuing advanced degrees in environmental sciences. It has achieved its goals, as evidenced by the extraordinary competition for the fellowships and the rigorous, independent selection process. Of the fellowship applications that STAR receives annually, only 125 fellowships, or 10% of all applicants, receive funding. Of the more than 100 former EPA fellowship recipients that were contacted by the committee, over 95% indicated high satisfaction with the program, and nearly 90% have remained in the environmental field, thus successfully contributing to the long-term program goals. Recommendation. Given the nation’s continuing need for highly qualified scientists and engineers in environmental research and management, the STAR fellowship program should be continued and funded. Are the STAR program’s funds adequate to achieve its objectives? Finding. STAR is only able to fund less than 15% of the proposals received for its individual investigator and center grants, and its funding has not kept pace with the rate of inflation. NIH and NSF strive to fund, on the average, 25-30% of the proposals received. STAR’s budget allows it to fund only 10-15% of the proposals it receives and only about 60% of those rated “excellent” or “very good” by its independent quality peer-review panels. By that measure, STAR does not have sufficient funds to recognize all the best proposals received. To be effective in its partnerships with other agencies, STAR must have sufficient funding to allocate to subjects of mutual interest to make it worth the extra administrative effort that partnerships require. The partnerships benefit STAR as a result of both the funds they leverage and the reputation they bring to the program. Although the STAR program’s budget grew rapidly in its first 3 years, it has not kept pace with general inflation in the last few years. That is

OCR for page 1
particularly true of the STAR fellowship program. The effect of that budgetary situation is exacerbated by the fact that costs of research have outpaced general inflation for more than a decade. Therefore, at present, STAR funds buy less research than the same amount of money could have bought several years ago. It is appropriate to consider the funding of the STAR program in the context of the overall funding for all of ORD, which also has not kept pace with inflation. STAR currently represents about 18% of ORD’s total funding. The committee considers that percentage to be a reasonable recognition of the value of independent peer-reviewed research to the agency. Recommendation. STAR program funding should be maintained at 15-20% of the overall ORD budget, even in budget-constrained times. However, budget planners should clearly recognize the constraints of not having inflation escalators to maintain the level of effort of the entire program. How should the STAR program be evaluated? Finding. There are no easy answers when it comes to identifying metrics for evaluating research programs, and the best approach for evaluating the STAR program is to establish a structured system of reviews by panels of experts. The STAR program has undergone a substantial—some might say excessive—number of reviews. Most of the reviews have focused on the program’s procedures; it is too early in the program’s life to be able to evaluate the research products fully. Too many reviews can be disruptive to the program and can divert the program’s attention and resources from its primary purpose. The committee, in its own evaluation of STAR, assessed the quality, relevance, and performance of the program, as set forth in recent OMB research and development criteria, by using qualitative and quantitative metrics selected on the basis of its review of information available from EPA and metrics used by EPA and other organizations. That is one approach for reviewing the STAR program and similar programs. Several examples of qualitative and quantitative metrics that were used for evaluating the STAR program are these: Does the STAR program have a clearly

OCR for page 1
defined plan for regular, external reviews of its research quality, and has this plan been effectively carried out? Has the program made significant contributions to advancing the state of the science in particular research topics? Does the program award grants expeditiously? Does the program have a schedule for the products it intends to produce and how well is it adhering to the schedule? The committee’s judgment is that quantitative metrics, although outwardly simpler to use, are not necessarily more informative than qualitative metrics. In some cases, quantitative metrics can be misleading, and emphasizing inappropriate metrics can distort the research outputs of a program. Qualitative metrics are less likely to have such effects, but they need to be interpreted carefully. The committee judges that expert review by a group of people with appropriate expertise is the best method of evaluating broad research programs, such as the STAR program. Expert review is appropriate for evaluating both the processes and the products of the STAR program. The types of experts needed depend on the level of review being conducted—individual projects or programmatic levels. Both qualitative and quantitative metrics can provide valuable support for such expert reviews. In planning for future reviews, the committee recommends that STAR and ORD consider an evaluation structure for the STAR program that has four levels: level 1 should examine the individual research projects, level 2 should focus on topics or groups of research projects on the same subject, level 3 should address the STAR program as a whole, and level 4 should tackle the question of how the STAR program relates to the broader institutions of ORD and EPA. The primary mechanism of review at levels 2-4 should be the panel of independent experts with the appropriate scientific, management, and policy backgrounds; the panels’ evaluations can use metrics appropriate to the specific level of review. Such a structured review strategy could replace the number of ad hoc, unplanned, and uncoordinated reviews. Recommendation. STAR and ORD should establish a structured program of reviews by panels of independent experts and should collect the appropriate information to support these reviews.