5
Taking the Measure of STAR

This chapter contains the committee’s evaluation of the Environmental Protection Agency (EPA) Science To Achieve Results (STAR) program. For its evaluation, the committee selected a set of metrics and recommends that EPA consider them as it adopts evaluative criteria for future evaluations of the STAR program. The evaluation is structured according to the guidelines that the president’s Office of Management and Budget (OMB) issued for government agencies to use in assessing their research programs for FY 2004 (OSTP/OMB 2002). As described in Chapter 4, the OMB guidelines set forth three major criteria for evaluating research programs: quality, relevance, and performance. The committee considered that following the OMB guidelines would be most valuable to EPA as it strives to comply with the requirements of the Government Performance and Results Act (GPRA) of 1993.

Under each of the three OMB criteria, the committee has developed metrics related to processes and products of the STAR grants program. Specifically, the committee reviewed numerous documents, including materials from EPA, OMB, and the National Research Council (NRC) and materials provided by people in academe, to derive these metrics. The process metrics are used to evaluate the adequacy of the operation or procedures of the STAR program, and the product metrics are used to evaluate the outputs of the program, such as the number of publications or the influence or effect the program has had or may have. The committee evaluated the STAR fellowship program independently from the grants program because it is a small part of the STAR program and operates somewhat independently.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 108
5 Taking the Measure of STAR This chapter contains the committee’s evaluation of the Environmental Protection Agency (EPA) Science To Achieve Results (STAR) program. For its evaluation, the committee selected a set of metrics and recommends that EPA consider them as it adopts evaluative criteria for future evaluations of the STAR program. The evaluation is structured according to the guidelines that the president’s Office of Management and Budget (OMB) issued for government agencies to use in assessing their research programs for FY 2004 (OSTP/OMB 2002). As described in Chapter 4, the OMB guidelines set forth three major criteria for evaluating research programs: quality, relevance, and performance. The committee considered that following the OMB guidelines would be most valuable to EPA as it strives to comply with the requirements of the Government Performance and Results Act (GPRA) of 1993. Under each of the three OMB criteria, the committee has developed metrics related to processes and products of the STAR grants program. Specifically, the committee reviewed numerous documents, including materials from EPA, OMB, and the National Research Council (NRC) and materials provided by people in academe, to derive these metrics. The process metrics are used to evaluate the adequacy of the operation or procedures of the STAR program, and the product metrics are used to evaluate the outputs of the program, such as the number of publications or the influence or effect the program has had or may have. The committee evaluated the STAR fellowship program independently from the grants program because it is a small part of the STAR program and operates somewhat independently.

OCR for page 108
The committee was not able to evaluate some of the metrics completely, particularly those related to products, primarily because STAR has not been in operation long enough to produce a sufficient number of products to allow a complete evaluation. Other reasons for incomplete evaluations are that EPA has not collected sufficient information relevant to a metric and that some of the metrics are not intended to be addressed solely by the STAR program, but cover issues that are broader and that must be addressed by ORD and EPA. The set of metrics used by the committee in evaluating the STAR grants program is presented in Table 5-1. The goals and objectives of the fellowship program differ from the grants program, and it is evaluated more briefly at the end of this chapter. For each metric, this chapter summarizes the pertinent information provided to (or otherwise obtained by) the committee and then presents the committee’s conclusions about how adequately the STAR program appears to be addressing it with respect to the available information—except that, as stated above, for some product metrics the STAR program is too young to have produced sufficient products to permit a complete evaluation. RESEARCH PROGRAM Quality Evaluating research quality is extremely difficult. It cannot be measured with a simple metric, such as a thermometer or a yardstick, or by the number of reports or number of pages produced. As indicated in Chapter 4, even more sophisticated measures, such as the number of citations in the technical literature, need to be carefully interpreted. The STAR program has tended to focus more on the quality of its process than on the quality of its products. That is understandable and appropriate. It is understandable because, being a relatively young program, STAR has had to focus on trying to get the process right and is only now beginning to accumulate a sufficient number of products to support a quality evaluation and because most of the external reviews of the program have tended to focus on process issues. It is appropriate because, as indicated in Chapter 4, a high-quality pro-cess is generally a necessary condition for producing high-quality products. Frequent evaluations of the quality of the process will also provide an early warning of possible problems in the quality of the products. The causes of inadequate products usually lie in inadequacies of procedures that occurred many years previously.

OCR for page 108
TABLE 5-1 Metrics Used in the Committee’s Evaluation of the STAR Grants Program QUALITY Process Does the program have an effective process to ensure receipt of high-quality proposals for its grant awards? Does the program have an effective process to ensure the selection of high-quality proposals? Does the program have a mechanism for encouraging high-quality research? Does the program have a clearly defined plan for regular, external reviews of its research quality, and has this plan been effectively carried out? Product Is the STAR program sponsoring high-quality research? Has the program made significant contributions to advancing the state of the science in particular topics? Do bibliometric and citation analyses demonstrate excellence in the program’s research? RELEVANCE Process Does the STAR portfolio support EPA’s mission, GPRA goals, and ORD’s strategic plans, research strategies, and multiyear plans? Are the processes used to define the research initiatives that will be supported by the STAR program sufficient to target the topics of most important uncertainty, highest impact, or highest priority? Does the program have a “clear plan for external reviews of the program’s relevance” (OSTP/OMB 2002), and has this plan been effectively carried out? Does the program have an effective process for identifying and communicating with the potential audiences and users of the research results? Product Is the STAR portfolio appropriately mixed between core and problem-driven research and between human health and ecologic research? Does the program have a good plan for integrating and synthesizing results, and has this plan been carried out effectively? Have the program’s results been used in EPA, state, or international decision-support documents?

OCR for page 108
RELEVANCE Product Have the research results in one or more subjects significantly improved the scientific foundation for decision-making? Can a link between STAR research and improved protection of human health and ecologic systems be identified? PERFORMANCE Process Is the STAR budget appropriate to fulfill the program’s mission? Is the program effectively complementing ORD’s other research efforts? Is the program well balanced? Does the program award grants expeditiously? Does the program have a process to demonstrate the communication of individual grant results in the professional literature? Is there a process in place for reviewing the performance of individual investigators and research centers? Product Is the program funding relevant research that otherwise would not be funded? Does the program have a schedule for the products it intends to produce, and how well is it adhering to that schedule? To what extent are site-specific studies designed to be replicated at other locations? Although a high-quality process may be necessary for producing high-quality products, it is not sufficient. It is time for the STAR program to begin to implement product evaluations that will ensure that both its process and its products have the high quality that the nation needs to support an effective and efficient environmental-management program. Process Metrics Does the program have an effective process to ensure receipt of high- quality proposals for its grant awards? EPA provided the committee, through presentations and interviews,

OCR for page 108
with a substantial amount of information (summarized in Chapter 2) concerning how the STAR program attempts to elicit good proposals and how it evaluates those it receives. There are two basic steps in establishing a good process for satisfying this metric. The first is to identify the topic to be addressed by the research in a way that stimulates good responses. The second is to advertise the availability of research support broadly in the research communities most qualified to undertake high-quality research on the desired topic. With respect to the first step, EPA puts substantial effort into defining its research agenda, and the STAR program submits its proposed requests for applications (RFAs) to extensive review within the agency. That effort is intended to ensure that the RFAs are focused on the most important issues and that they define the research requirements properly. However, although substantial effort is devoted to the process in the agency, neither the research plans nor the proposed RFAs are externally peer-reviewed by subject-matter experts, except when the research is being supported jointly by another organization, in which case, representatives of the other organization participate in drafting and reviewing the RFA. With respect to notifying potential researchers of the funding opportunities, the National Center for Environmental Research (NCER) makes a substantial effort to reach out to a broad scientific community and to recruit the most capable scientists. NCER disseminates its RFAs widely through its Web site, the Federal Register, announcements at professional meetings, and e-mail distributions to individuals or institutions that sign up on the STAR Web site. When the desired research falls outside EPA’s traditional research fields and may therefore involve scientists that are not already tied into the agency’s research program, STAR solicits the help of other agencies that traditionally work with these scientists to ensure that they are aware of the funding opportunities. The committee concludes that the processes established by the STAR program compare favorably with and in many cases substantially exceed those established by other research-supporting organizations. Subjecting the research plans and draft RFAs to independent peer review would strengthen the process even more, but the committee recognizes that doing so might reduce the agency’s current flexibility to respond quickly to new research needs and might unduly delay the process of issuing RFAs. However, EPA should consider using external peer reviewers for RFAs when they do not have the in-house expertise.

OCR for page 108
Does the program have an effective process to ensure the selection of high-quality proposals? The process for selecting proposals was described to the committee by the person responsible for managing it and by several STAR project officers, and some committee members had participated in the process previously. The STAR program has established a rigorous peer-review process to evaluate the quality of proposals. Such peer review is the foundation on which excellence is achieved in all research programs, such as those of the National Institutes of Health (NIH) and the National Science Foundation (NSF). The agency has taken strong steps to ensure that this process does not suffer from conflicts of interest and is independent. The program’s procedures provide for a firewall that shields the peer-review process from any influence or potential conflicts of the project officers and staff who oversee the individual investigator, fellowship, and center awards. For instance, project officers can provide the names of potential reviewers to the Science Review Administrators (SRAs), also known as peer-review officers, but it is the sole responsibility of the SRAs to select reviewers and to make reviewer assignments. Project officers may attend peer-review meetings as observers but may not provide any comments that would affect peer review (Bryan 2002). The program selects peer reviewers from a large number of sources, including people who have served on previous panels, keyword searches of databases (such as, Community of Science, the National Library of Medicine’s PUBMED, and faculty listings), keyword searches of NCER’s peerreview panelist information system, input from project officers and program-office scientists, and lists of the attendees of pertinent technical conferences (Bryan 2002). The agency is unusual in that it pays the members of its peer-review panels a daily stipend, presumably to provide additional encouragement for experts to participate. Although lower than the consulting fees that such experts might earn in the private sector, the stipends do reduce the financial disincentives associated with serving on the panels. The committee received some comments suggesting that in the program’s early days, some members of the peer-review panels might not always have had the necessary qualifications to be effective members, but this problem appears to have disappeared as the program has matured. The committee heard no suggestion that the process of selecting peer reviewers was influenced by conflicts of interest.

OCR for page 108
The final selection of proposals to be funded is influenced by a “relevance review” that is carried out by agency staff. That review is limited to proposals that have been rated as “excellent” or “very good” by the quality peer-review panel. The committee concludes that, given EPA’s desire to avoid appearances of conflicts of interest by completely separating the selection of peer-review panel members from the influence of project officers, the agency has established a rigorous, independent peer-review process for selecting the highestquality proposals. Does the program have a mechanism for encouraging high-quality research? To gain a better understanding of how the STAR program encourages high-quality research, the committee received briefings from NCER staff on its processes, reviewed material that was publicly available, attended several progress-review meetings, and discussed the program’s procedures with STAR grant recipients. STAR has implemented several mechanisms for encouraging highquality research by its investigators. Investigators are required to submit annual progress reports that describe the research being undertaken and its progress. The progress reports are reviewed by STAR project officers, and summaries are posted on the NCER Web site. In addition, the STAR program sponsors progress review workshops on research topics. Principal investigators of all STAR grants receiving support pertinent to the topic are expected to attend. The meetings are also attended by other EPA Office of Research and Development (ORD) staff conducting relevant research; all meetings are open to the public. At the meetings, principal investigators must present their research progress to their colleagues and EPA staff, opening it to peer review. The meetings also provide an opportunity for researchers to share ideas and coordinate research efforts. Some of the meetings have apparently been much more successful than others in accomplishing their objectives. The committee concludes that EPA has established procedures for reviewing the quality of research in progress that in several ways exceed those adopted by most other research-supporting organizations. Those procedures could be enhanced by ensuring that progress-review workshops were held at the most expeditious times and were run most efficiently to stimulate peer review and collaboration. The program should review the success of its past progress-review meetings and organize its future meet

OCR for page 108
ings to emulate the ones that were most successful in accomplishing their objectives. Does the program have a clearly defined plan for regular, external reviews of its research quality, and has this plan been effectively carried out? As mentioned in Chapter 2, numerous reviews have been done on the processes and operation of the STAR grants program, and not on the products of the grants (EPA/BOSC 1998; EPA 2000; EPA/NSF 2000; EPA/SAB/BOSC 2000; GAO 2000; EPA/SAB 2001; EPA 2002a). The majority of the reviews have been conducted by EPA’s Science Advisory Board (SAB) or Board of Scientific Counselors (BOSC). The committee is concerned that too many and too frequent reviews of the STAR program have the potential to be damaging in that they may divert necessary financial and personnel resources from the program. The committee believes strongly that because of the nature of research, which takes a considerable amount of time and many projects to advance the state of knowledge, too frequent reviews of STAR add little value to the understanding of the operation and results of the program. The committee recommends that the STAR program establish a schedule of product reviews at the appropriate level (as discussed in Chapter 4). By establishing such a schedule, the program may protect itself from the apparent excess of external reviews that have been imposed on it in the past. Product Metrics Is the STAR program sponsoring high-quality research? The committee was presented with some anecdotes concerning the quality of the research being sponsored by the STAR program, but not with any systematic reviews or evidence concerning the quality of the program’s products. Evaluating the quality of research products is very difficult, involving substantial judgment on the part of scientists who have expertise in the research topic being reviewed. As indicated in Chapter 4, the committee considers that the most effective method for evaluating research is the use of independent expert review committees focusing on specific topics (a level 2 review). The committee recommended that the STAR program establish a schedule for such reviews (see Chapter 4).

OCR for page 108
Lacking product evaluations, the committee reviewed the backgrounds and accomplishments of a sampling of the STAR principal investigators. Although a rigorous sampling procedure was not conducted and thus the results of the review are only indicative and not definitive, this review indicated that the STAR program is funding many scientists who have outstanding research credentials. The scientists have impressive track records and are leaders in their fields. Many are editors of journals or officers in societies and have received awards of distinction. Some were attracted to the STAR program from fields outside EPA’s mission, so it can be said that the program has been successful in attracting the best and the brightest. Many of the investigators, however, have long been active in the relevant fields, and the STAR program has enabled them to continue to make contributions. The investigator mix also included young investigators who will be the leaders of the future. The committee notes that EPA’s rate of funding of individual investigator and center awards tends to be lower than that of other federal grants programs, such as those sponsored by NSF and NIH; this reflects the competitiveness of the program. As indicated in Chapter 2, data from FY 19992001 indicate that EPA funds an average of 10-15% of the proposals it receives. In contrast, as indicated in Chapter 3, agencies like NIH and NSF strive to fund at least about 25-30% of the proposals received. STAR is able to fund only about 60% of proposals rated as “excellent” or “very good” by its independent quality peer-review panels. It funds no proposals that receive a lower ranking (“good” or lower). On the basis of the STAR program’s process for awarding grants, the quality of the people and institutions being funded by the STAR program, and the highly competitive nature of its awards, the committee is confident that the products of these grants will be of the highest quality. Has the program made significant contributions to advancing the state of the science in particular topics? Although the STAR program does not systematically identify the significant contributions it makes to filling important gaps in the state of the science in particular topics, examples were presented to the committee by EPA staff and by investigators supported by STAR funding. Some of the committee members were also familiar with the advances that STAR-funded research was making to some particular topics.

OCR for page 108
The planning process that leads to the preparation of RFAs is to a large extent focused on ensuring that STAR grants will fill information gaps in topics of greatest interest to the agency. That focus is maintained throughout the research process. The committee was presented with several examples of STAR-supported research efforts that had made significant contributions to scientific understanding in particular topics (see Boxes 5-1, 5-2, and 5-3). For example, STAR-sponsored research in endocrine disruptors, particulate matter, and ecologic assessment has resulted in peer-reviewed groups of publications of immediate interest in understanding causes of, exposures to, and effects of environmental pollution. To determine whether STAR research has filled a critical knowledge gap or otherwise strengthened and improved the foundation for decision making, it would be useful to assess the state of the science in a particular issue before STAR-funded projects are completed and then synthesize the results of the research after the projects are completed. Such assessments would help EPA to target RFAs at the front end, as well as to analyze net results at the back end. A particular issue could be assessed by a panel convened by ORD, by STAR, or by others in the field (such as the Ecological Society of America, or the Society of Environmental Toxicology and Chemistry). There have been several successful examples of state-of-the-science documents, including the particulate-matter (PM) reports produced by the NRC’s Committee on Research Priorities for Airborne Particulate Matter (NRC 1998, 1999b, 2001). In general, better integration of research results by STAR and ORD and the state-of-the-science assessments mentioned above should provide most of the information necessary to report on this metric. The committee recognizes of course that this metric involves substantial subjective judgment and that it is often difficult to identify the effect of any particular set of research results. The judgments implicit in this metric can probably be best rendered by the use of expert review, as suggested in Chapter 4. The committee concludes that STAR-supported research is making significant contributions to advancing the state of the science in many of the topics that it is addressing. The committee suggests that the program undertake a more systematic effort to identify the contributions and the success in filling the knowledge gaps identified in the research planning process by preparing research synthesis reports when research is completed, as recommended elsewhere in this chapter.

OCR for page 108
BOX 5-1 Results of STAR Endocrine-Disruptors Program Determined that exposure to high concentrations of polybrominated biphenyls prenatally and in breast milk may affect puberty in girls (Blanck et al. 2000). Discovered a new (third) estrogen receptor in vertebrates and demon-strated that estrogens and xenoestrogens can act on cells at the membrane level (Hawkins et al. 2000). Developed and refined an in vivo model using medaka to identify endocrine disrupting chemicals (Cooke and Hinton 1999). Identified androgenic compounds (male-hormone mimics) in paper-mill effluent by using a screening assay in fish (Jenkins et al. 2001). Developed integrated array of computational tools undergoing valida-tion by EPA’s Office of Pollution Prevention and Toxic Substances (OPPTS) for setting priorities for science and technology programs (Xing et al. 1999). Determined concentrations of phytoestrogens in human amniotic fluid and effects of exposure to them in animal models (Hughes et al. 2001). Source: Adapted from P. Preuss, EPA, presentation to National Research Council committee, March 18, 2002. Do bibliometric and citation analyses demonstrate excellence in the program’s research? Although EPA encourages its grantees to provide the STAR program with information on the articles and other publications that stem from the research it supports, the program has no mechanism for monitoring such publications after a grant is completed or for conducting bibliometric and citation analyses that would demonstrate the influence of the research on other work being conducted. The committee did sponsor an ad hoc bibliometric analysis to gain a better understanding of the value of such an approach (IISCO, Atlanta, GA, unpublished material, 2002) (see Chapter 4). As indicated in Table 4-1, the bibliometric analysis conducted for the committee indicated that the citation rate of publications that result from STARsupported research is similar to that of other research publications in the topics for which the analyses were undertaken. As stated in Chapter 4, the committee considers that bibliometric and citation analyses are important quantitative metrics for gauging the quality of research but cautions that these types of analyses have many limitations.

OCR for page 108
ing the effects of endocrine disruptors on wildlife (E. Francis, EPA, Washington, D.C., personal commun., August 6, 2002). In addition, STAR’s ecologic-indicators program is the primary source of support for research on the development of water-quality indicators for biologic monitoring (B. Levinson, EPA, Washington, D.C., personal commun., August 4, 2002). Does the program have a schedule for the products it intends to produce, and how well does it adhere to that schedule? Although no such schedule was presented to the committee, OMB has requested that agencies prepare such a schedule in its guidelines for evaluating research programs under GPRA (OSTP/OMB 2002), and such specification of program outputs is a normal component of budget preparations. Those documents, however, are not usually made available to the public. Because of the STAR program’s heavy reliance on its Web site for communicating its activities to the public, placing such a schedule on its Web site might substantially enhance the efficiency of the program’s communication efforts. The committee, on the basis of discussions with EPA officials, learned that EPA program offices may not always be as aware of recent postings on NCER’s Web site as they should be. If STAR listed its expected products on NCER’s Web site and allowed users of the site to indicate that they would like to be notified when an expected product became available, the likelihood of being informed about issues could be substantially increased. The committee recommends that the program post a schedule of expected products on its Web site and allow members of the public to indicate whether they would like to be notified when products become available. This not only would provide the public with an opportunity to observe the performance of the program but also would improve the efficiency of its communication efforts. To what extent are site-specific studies designed to be replicated at other locations? Some of the STAR research efforts are focused on developing analytic tools or measures of environmental health for particular geographic areas. A major purpose of such grants is to develop approaches that could be adopted in other locations. The committee was presented with information about the extent to which biologic sampling protocols were replicated at 822 reference sites in

OCR for page 108
13 western states—work supported by the U.S. Forest Service and EPA (Gilman 2002). The STAR program also supported a pilot program in the mid-Atlantic states to “develop methods to transfer STAR grant results to environmental decision-makers, test these methods in the mid-Atlantic region, and evaluate the feasibility of using the methods in other regions” (Bradley 2002). At the time the committee was briefed on this effort, limited results were available, although the program had identified a number of regional and local decision-makers who “have found the results of the STAR program valuable” (Bradley 2002). The pertinent questions that reviews of such projects should be addressing are the extent to which the research projects were intended to be replicable, what efforts have been made to promote such replications, the extent to which replications have occurred, and what steps could be taken to promote their occurrence. The committee recommends that the program continue its efforts to collect information to answer those questions. FELLOWSHIP PROGRAM The STAR fellowship program is a small component of the overall STAR program whose goals and objectives differ from those of the main research grants program. The fellowship program is an important contribution to the nation’s effort to train and “encourage promising students to obtain advanced degrees and pursue careers in environmentally related fields” (EPA 2002b) and to develop the next generation of environmental scientists. The program is the only federal fellowship program designed exclusively for students pursuing advanced degrees in environmental sciences and engineering. It is highly competitive: only 10% of applicants receive funding. Because the fellowship program has been important in encouraging and maintaining a strong interest in environmental science and engineering, the committee considers that the program should be continued and funded. Until 2002, the STAR program was funding 100-125 fellows per year. However, the president’s FY 2003 budget did not contain funding for the program, so no new fellowships were awarded in 2002. (NOTE: The FY 2003 Omnibus Appropriations Act, signed February 2003, appropriated $9.75 million for the STAR fellowship.) Although the program publishes on the NCER Web site information about all the students receiving fellowships, it does not gather systematic information to track the status of past and currently funded fellows to assess the impact of the STAR program on their careers. To gather information on

OCR for page 108
the influence of the fellowship program, the committee contacted more than 100 STAR fellows who were initially funded in 1995 and 1996 and who would have completed their graduate work. Of the fellows contacted, over 95% indicated high satisfaction with the program. Additional information gathered by the committee permitted it to evaluate the following metrics. Does STAR have a process for ensuring the selection of high-quality fellows? The committee gathered information on how the STAR program selects fellows through discussions with people within EPA’s peer-review division. The STAR program’s process for selecting high-quality fellows ensures the competitiveness of the program. The program publishes the announcement for the fellowships widely, including posting it on the NCER Web site and distributing it via e-mail to obtain a large pool of applicants. Prospective fellows submit applications that are evaluated in an independent peerreview process that is similar to the one for reviewing individual investigator and center grants (see Chapter 2 for details). Fellowship applications are evaluated according to criteria that includes their academic and employment records, course of proposed research, and potential for success. To receive continued funding, STAR fellowship recipients are required to remain in good academic standing. A fellowship may be terminated if the EPA project officer determines that the fellow is not performing up to the standards of the program. On the basis of the wide dissemination of the fellowship applications, the peer-review process used for selecting applicants, and the low percentage of applicants who receive support, the committee concludes that the STAR fellowship program ensures a process for selecting high-quality applicants. What percentage of fellowship recipients obtain their advanced degrees? Although the program apparently does not collect the information required to assess this metric, the committee’s contacts with more than 100 STAR fellows who received funding in the early years of the program indicated that nearly all have already completed their research and graduated in their degree programs. That is a testament to the quality of the process for selecting fellows and to the success of the program. The committee suggests that if the fellowship program continues, EPA may want to collect metrics on the fellows, including the number graduating and positions held, to document the success of the program.

OCR for page 108
How many fellowship recipients who have completed their graduate work are working in environmental science? The program has not collected the information required to assess this metric, but nearly 90% of the fellows contacted by committee members were employed in the environmental science field. About 10% of those were working in government, over 55% chose to remain in academe, and the remainder were working in industry, in consulting, or in nonprofit organizations. Many stated that the fellowship was extraordinarily valuable in assisting them to advance their work in environmental science (see Box 5-4). The committee suggests that if the fellowship program continues, EPA may want to collect information on the careers pursued by fellowship recipients to document the success of the program. How many fellowship recipients have completed their degree programs with at least one peer-reviewed publication? Nearly 80% of the STAR fellows contacted by committee members indicated that they had at least one peer-reviewed publication as a result of their research funded through the fellowship program. That is an indication of the quality of the fellows and the quality of their research. The committee suggests that if the fellowship program continues, EPA may want to collect information on publications and other products of fellowship recipients to document the success of the program. CONCLUSIONS • The committee conducted an evaluation of the quality, relevance, and performance of the STAR program, as set forth in the recent OMB research and development criteria, using metrics that grew out of its review of information available from EPA and of metrics used by EPA and other organizations. The metrics, which are both quantitative and qualitative, assisted the committee in forming judgments regarding the scientific merit of the program and its impact on the agency. • The committee was able to evaluate the program’s process better than its results. Evaluation of research results requires a substantial lapse in time, in that it takes 3-5 years, or more, from the initiation of laboratory or field experiments until the analysis and publication of research results. Considerably more time must elapse to view the impact of the published research on the scientific and regulatory community. Advances in research

OCR for page 108
BOX 5-4 Responses from STAR Graduate-Student Fellows Regarding the Value of the STAR Fellowship “Yes, it was valuable in the sense that it helped my mentor pay me as a graduate student. It was valuable to me as an initial award on my CV that I have followed up with additional fellowships throughout the years. This gives me a track record of receiving funding. This track record has and will continue to assist me in my future endeavors to obtain funding/jobs. I believe there are not that many opportunities for graduate students to obtain independent funding and the STAR program was valuable in that sense.” “Yes, The STAR fellowship gave me the freedom to expand my thesis topic into very interesting directions that led to several publications in diverse journals.” “It was an invaluable resource. It freed me from teaching so that I could focus on my Ph.D. research. I believe I was able to finish sooner and investigate more than I would have otherwise. It enabled me to buy textbooks to learn about the field that I was working in. It enabled me to attend conferences that I would not have otherwise attended. By attending these conferences I met others in the field, including other graduate students I am still in contact with, and I gained confidence in myself and my abilities. After graduation, I received a National Science Foundation postdoc and went to Rensselaer Polytechnic Institute (RPI) for three years. Without the EPA fellowship, I might not have had the confidence or the ability to apply for the postdoc and write a successful proposal.” “Yes, very much so. I am presently working on the same system answering additional questions raised during my Ph.D. Not only did the fellowship allow me to complete my dissertation, it has in part spawned an entire research program in the evolution of development of stickleback here at the university. We presently have approximately 10-12 people working on this project, and are collaborating with labs at SUNY Stony Brook, Stanford, and Clark University.” “It was very valuable; in particular having a research budget gave me a lot more independence and flexibility in designing and carrying out my dissertation research. The program is also well known and respected. When I interviewed for jobs after graduation, I think my having received the STAR fellowship strengthened my position considerably.” tend not to be accomplished by the completion of individual research grants, but rather through the combined impacts of multiple research projects on a specific field.

OCR for page 108
• The committee recognizes that there have already been a substantial—some might say excessive—number of reviews of the STAR program. Most have focused on the administration and operation of the program. The committee is concerned that reviewing STAR too frequently has the potential to be damaging, in that it may divert necessary financial and personnel resources away from the program. The committee therefore recommends that STAR and ORD consider an evaluation structure for conducting future reviews by independent panels of experts, comprising individuals with the appropriate scientific, management, and policy backgrounds. Expert reviews are the best method of evaluating the quality of a research program, and having a structured framework would probably reduce the number of ad hoc, unplanned, and uncoordinated reviews. • The STAR program funds important research that is not conducted or funded by other agencies. The STAR program has also made commendable efforts to leverage funds through establishment of research partnerships with other agencies. • Although it is still too early for comprehensive evaluations of the research results of the STAR program, some STAR research efforts have already substantially improved the scientific foundation for decision making and the results produced by STAR investigators have been widely published in peer-reviewed journals. • It is appropriate for EPA’s research efforts to incorporate a balance between core and problem-driven research and a balance between ecologic and health-effects research. STAR research improves the knowledge base required to make sound environmental decisions, and this includes both core and problem-driven research. A balance between human-health and ecologic research is necessary, particularly because much of what is termed ecologic research actually elucidates the processes by which environmental stressors affect both humans and other ecosystems. • The committee encourages the STAR program to continue funding research that explores future environmental problems within its overall research portfolio. Research devoted to potential environmental threats may help to avoid or reduce the impact of such threats or at the very least put into place the scientific capacity to address them. • Although the STAR program has used several methods to report on the results of individual grants and centers, it has yet to produce documents that summarize the “state of the science” or provide a synthesis of research results and describe how the results of a group of grants have moved scientific understanding forward. The production of such reports, using outside experts where appropriate, can be extremely useful for targeting gaps in

OCR for page 108
knowledge and communicating the state of the science to the program’s diverse users and audiences. The committee considers the increased production of such reports to be an important improvement that should be made in the STAR program. The appropriate type of state of the science or research synthesis document will depend on the intended audience. Because the STAR supported research often complements that being done elsewhere in ORD, and sometimes in other agencies, the integration and synthesis of research results is a larger issue that in many cases cannot be taken solely by the STAR program and must be addressed by ORD or EPA. • The STAR program has been commendably aggressive in experimenting with innovative approaches to communicating the results of its funded research to a wide variety of users and audiences, but its success in these efforts has been uneven. EPA and the STAR program have various mechanisms for communicating with the STAR user community. However, the STAR program has not developed an effective strategy for communicating to a wider user community, including state, tribal, local, and international environmental agencies and the public; most of the emphasis has been on the scientific community and the program offices. In some cases, the effective dissemination of results should be primarily STAR’s responsibility. In other cases, STAR’s contributions will be only one component of a larger research effort, and the primary dissemination responsibility should lie within ORD or EPA. • The fellowship program is an important and valuable component of the STAR program for EPA and the nation. It ensures a continuing supply of graduate students in environmental science and engineering who provide a strong foundation for the nation’s environmental research and management efforts. The program has been important in encouraging and maintaining strong interest in environmental science and engineering. RECOMMENDATIONS • The committee recommends that NCER institute a structured system of program-level reviews as its primary mechanism for evaluating the STAR program. The improved information-collection efforts (discussed in Chapter 5) should be used to support such reviews. • The committee recommends that STAR and ORD continue to work to produce state-of-the-science and research-synthesis documents. These are important for identifying critical information gaps and communicating the state of knowledge on a particular issue to the many users and audiences interested in this information.

OCR for page 108
• The committee commends EPA for its efforts to communicate with its diverse users and audiences and recommends that STAR and EPA continue and, where appropriate, expand such outreach efforts. The likely audiences of research results should be identified early in the research planning process, explicitly identified in RFAs, and considered throughout the research implementation process; and a coherent strategy should be developed for disseminating research results when they become available. • STAR program funding should be maintained at 15-20% of the overall ORD budget, even in budget-constrained times. However, budget planners should clearly recognize the constraints of not having inflation escalators to maintain the level of effort of the entire program. • EPA should continue its efforts to attract “the best and the brightest” researchers to compete for STAR funding. • Given the nation’s continuing need for highly qualified scientists and engineers in environmental research and management, the STAR fellowship program should be continued and funded. REFERENCES Blanck, H.M., M. Marcus, P.E. Tolbert, C. Rubin, A.K. Henderson, V.S. Hertzberg, R.H. Zhang, and L. Cameron. 2000. Age at menarche and tanner stage in girls exposed in utero and postnatally to polybrominated biphenyl. Epidemiology 11(6):641-647. Bradley, P. 2002. Pilot Program Overview. Presentation at the First Meeting on the Review of EPA’s Research Grants Program, March 19, 2002, Washington, DC. Bryan, E. 2002. Peer Review Process for EPA’s National Center for Environmental Research’s Science to Achieve Results Program. Presentation at the Third Meeting on the Review of EPA’s Research Grants Program, June 6, 2002, Washington, DC . Butler, T.J., G.E. Likens, and B.J.B. Stunder. 2001. Regional-scale impacts of Phase I of the Clean Air Act Amendments in the USA: The relation between emissions and concentrations, both wet and dry. Atmos. Environ. 35(6):1015-1028. Cass, G.R., L.A. Hughes, P. Bhave, M.J. Kleeman, J.O. Allen, and L.G. Salmon. 2000. The chemical composition of atmospheric ultrafine particles. Philosophical Transactions: Mathematical, Physical and Engineering Sciences 358 (1775):2581-2592. Clarke, R.W., P.J. Catalano, P. Koutrakis, G.G. Murthy, C. Sioutas, J. Paulauskis, B. Coull, S. Ferguson, and J.J. Godleski. 1999. Urban air particulate inhalation alters pulmonary function and induce pulmonary inflammation in a rodent model of chronic bronchitis. Inhal. Toxicol. 11(8):637-656.

OCR for page 108
Cooke, J.B., and D.E. Hinton. 1999. Promotion by 17beta-estradiol and beta-hexa-chlorocyclohexane of hepatocellular tumors in medaka, Oryzias latipes. Aquat. Toxicol. 45(2):127-145. Deegan, L.A., J.T. Finn, and J. Buonaccorsi. 1997. Development and validation of an estuarine biotic integrity index. Estuaries 20(3):601-617. EPA (U.S. Environmental Protection Agency). 1998a. Ecological Research Strat-egy. EPA/600/R-98/086. Office of Research and Development, U.S. Environ-mental Protection Agency, Washington, DC [Online]. Available: http://www. epa.gov/ordntrnt/ ORD/WebPubs/final/eco.pdf [accessed Jan. 13, 2003]. EPA (U.S. Environmental Protection Agency). 1998b. Research Plan for Endo-crine Disruptors. EPA/600/R-98/087. Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC [Online]. Available: http://www.epa.gov/ORD/WebPubs/final/revendocrine.pdf [accessed Feb. 20, 2003]. EPA (U.S. Environmental Protection Agency). 1999. Airborne Particulate Matter Research Strategy. EPA/600/R-99/045. Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC. [Online]. Available: http://www.epa.gov/ORD/resplans/Pmstrat7.pdf [accessed Feb. 20, 2003]. EPA (U.S. Environmental Protection Agency). 2000. Evaluation Report: A Deci-sion Making and Valuation for Environmental Policy Interim Assessment. National Center for Environmental Research, Office of Research and Develop-ment, U.S. Environmental Protection Agency [Online]. Available: http://es. epa.gov/ncer/science/economics/reviews.html [accessed Jan. 13, 2003]. EPA (U.S. Environmental Protection Agency). 2002a. Report to ORD’s Board of Scientific Counselors, Self-Study Update. Prepared by the National Center for Environmental Research, Office of Research and Development, U.S. Environ-mental Protection Agency. January 2002. EPA (U.S. Environmental Protection Agency). 2002b. Fall 2002 Science to Achieve Results Fellowships for Graduate Environmental Study. 2002 RFA. National Center for Environmental Research, Officer of Research and Devel-opment, U.S. Environmental Protection Agency. EPA/BOSC (U.S. Environmental Protection Agency Board of Scientific Counsel-ors). 1998. Program Review of the National Center for Environmental Re-search and Quality Assurance (NCERQA). Final Report of the Ad Hoc Sub-committee on the Review of NCERQA. Board of Scientific Counselors, Office of Research and Development, U.S. Environmental Protection Agency, Wash-ington, DC. April 30, 1998. EPA/NSF (U.S. Environmental Protection Agency and National Science Founda-tion). 2000. Interim Assessment for the Decision Making and Valuation for Environmental Policy Grants Program. Final Report. Prepared for National Science Foundation and U.S. Environmental Protection Agency, by Aspen Systems Corporation. April 17, 2000. EPA/SAB (U.S. Environmental Protection Agency Science Advisory Board). 2001. The Science to Achieve Results (STAR) Water and Watersheds Grants

OCR for page 108
Program: An EPA Science Advisory Board Review. A Review by the Ecolog-ical Processes and Effects Committee (EPEC) of the EPA Science Advisory Board. EPA-SAB-EPEC-02-001. Science Advisory Board, U.S. Environmen-tal Protection Agency, Washington, DC [Online]. Available: http://www.epa. gov/science1/fiscal02.htm [accessed Jan. 13, 2003]. EPA/SAB (U.S. Environmental Protection Agency Science Advisory Board). 2002. Interim Review of the Particulate Matter (PM) Research Centers of the USEPA: An EPA Science Advisory Report. A Review by the PM Research Centers Interim Review Panel of the Executive Committee of the U.S. EPA Science Advisory Board (SAB). EPA-SAB-EC-02-008. Science Advisory Board, U.S. Environmental Protection Agency, Washington, DC. May 2002 [Online]. Available: http://www.epa.gov/science1/fiscal02.htm [accessed Jan. 13, 2003]. EPA/SAB/BOSC (U.S. Environmental Protection Agency Advisory Board and Board of Scientific Counselors). 2000. A joint SAB/BOSC report: Review of the Science to Achieve Results (STAR) program. EPA-SAB-EC-00-008. Science Advisory Board, Board of Scientific Counselors, U.S. Environmental Protection Agency [Online]. Available: http://www.epa.gov/sab/pdf/ec0008. pdf [accessed Jan. 13, 2003]. GAO (U.S. General Accounting Office). 2000. Environmental Research: STAR Grants Focus on Agency Priorities, But Management Enhancements Are Possi-ble: Report to the Chairman, Subcommittee on VA, HUD, and Independent Agencies, Committee on Appropriations, House of Representatives. GAO/RCED-00-170/B-142370. U.S. General Accounting Office, Washington, DC. Gilman, P. 2002. Presentation at the First Meeting on the Review of EPA’s Re-search Grants Program, March 19, 2002, Washington, DC. Hawkins, M.B., J.W. Thornton, D. Crews, J.K. Skipper, A. Dotte, and P. Thomas. 2000. Identification of a third distinct estrogen receptor and reclassification of estrogen receptors in teleosts. PNAS 97(20):10751-10756. Hughes, C., W. Foster, S. Chan, L. Platt, S. Thompson, S. Hubbard, A. DuBose, and L. Tyrey. 2001. Extrapolation of rodent studies on amniotic fluid contam-inants to human populations. Human and Ecological Risk Assessment 7(5):979-1002. Jenkins, R., R.A. Angus, H. McNatt, W.M. Howell, J.A. Kemppainen, M. Kirk, and E.M. Wilson. 2001. Identification of androstenedione in a river containing paper mill effluent. Environ. Toxicol. Chem. 20(6):1325-1331. Juliano, J., and M.D. Sobsey. 1997. Simultaneous concentration of Crypto-sporidium, bacteria and viruses from water by hollow fiber ultrafiltration. In: Proceedings of the 1997 Water Quality Technology Conference, American Water Works Association, Denver, CO, 1997. Locci, A.B., and J.F. Koonce. 1999. A theoretical analysis of food web constraints on walleye dynamics in Lake Erie. Pp. 497-510 in State of Lake Erie: Past,

OCR for page 108
and Future, M. Munawar, T. Edsall, and I.F. Munawar, eds. Leiden, The Netherlands: Backhuys Publishers. Lovett, E, R.W. Clarke, R.L. Verrier, P. Koutrakis, J. Lawrence, J.M. Antonini, and J.J. Godleski. 1999. Rat cardiovascular dysfunction prior to death during exposure to concentrated ambient air particles. The Toxicologist 48(1-S):297. Michalek, J.L., and J.L. Colwell. 2000. Monitoring development in Southwest Florida (1973-1995) using Landsat data and a hybrid change detection tech-nique. Natural Areas Journal April 2000. NRC (National Research Council). 1998. Research Priorities for Airborne Particu-late Matter. 1. Immediate Priorities and a Long-Range Research Portfolio. Washington, DC: National Academy Press. NRC (National Research Council). 1999a. Evaluating Federal Research Programs: Research and the Government Performance and Results Act. Washington, DC: National Academy Press. NRC (National Research Council). 1999b. Research Priorities for Airborne Partic-ulate Matter. 2. Evaluating Research Progress and Updating the Portfolio. Washington, DC: National Academy Press. NRC (National Research Council). 2001. Research Priorities for Airborne Particu-late Matter. 3. Early Research Progress. Washington, DC: National Acad-emy Press OSTP/OMB (Office of Science Technology and Policy/Office of Management and Budget). 2002. FY 2004 Interagency Research and Development Priorities. Memorandum for the Heads of Executive Departments and Agencies, from John Marburger, Director, Office of Science and Technology Policy, and Mitchell Daniels, Director, Office of Management and Budget, The White House, Washington, DC. May 30, 2002. Pope, C.A., R.T. Burnett, M.J. Thun, E.E. Calle, D. Krewski, K. Ito, and G.D. Thurston. 2002. Lung cancer, cardiopulmonary mortality, and long-term exposure to fine particulate air pollution. JAMA 287(9):1132-1141. Preuss, P.W. 2002. National Center for Environmental Research, History, Goals, and Operation of the STAR Program. Presentation at The First Meeting on Review EPA’s Research Grant Program, March 18, 2002, Washington, DC. Voinov, A., R. Costanza, L. Wainger, R. Boumans, F. Villa, T. Maxwell, and H. Voinov. 1999. Patuxent landscape model: Integrated ecological economic modeling of a watershed. Environmental Modelling and Software 14(5):473-491. Xing, L., W.J. Welsh, W. Tong, R. Perkins, and D.M. Sheehan. 1999. Comparison of estrogen receptor alpha and beta subtypes based on comparative molecular field analysis (CoMFA). SAR QSAR Environ. Res. 10(2-3):215-237. Zanobetti, A., and J. Schwartz. 2001. Are diabetics more susceptible to the health effects of airborne particles? Am. J. Respir. Crit. Care Med. 164(5):831-833.