Appendix B
Previous Reviews of the STAR Program

This appendix summarizes the conclusions and recommendations of the various external reviews that have been conducted on the Environmental Protection Agency (EPA) Science To Achieve Results (STAR) program and the agency’s official responses to the reviews.

1997 REVIEW BY ORD BOARD OF SCIENTIFIC COUNSELORS

The first review of the STAR program was conducted in 1997 by EPA’s Office of Research and Development (ORD) Board of Scientific Counselors (BOSC). BOSC conducted a series of management and programmatic reviews of ORD’s three major laboratories and two centers. One of the centers was the National Center for Environmental Research and Quality Assurance (NCERQA), now known as the National Center for Environmental Research (NCER), which had responsibility for overseeing and administering the STAR program. BOSC concluded that NCERQA was an organization of vital importance to ORD, EPA, and the national objective of improving fundamental knowledge for environmental assessment and management. It further concluded that NCERQA had played a key role in refocusing and shaping the new vision for ORD.

In the course of its review of NCERQA, BOSC made several recommendations that touched on the STAR program (EPA/BOSC 1998), including the following:



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 153
Appendix B Previous Reviews of the STAR Program This appendix summarizes the conclusions and recommendations of the various external reviews that have been conducted on the Environmental Protection Agency (EPA) Science To Achieve Results (STAR) program and the agency’s official responses to the reviews. 1997 REVIEW BY ORD BOARD OF SCIENTIFIC COUNSELORS The first review of the STAR program was conducted in 1997 by EPA’s Office of Research and Development (ORD) Board of Scientific Counselors (BOSC). BOSC conducted a series of management and programmatic reviews of ORD’s three major laboratories and two centers. One of the centers was the National Center for Environmental Research and Quality Assurance (NCERQA), now known as the National Center for Environmental Research (NCER), which had responsibility for overseeing and administering the STAR program. BOSC concluded that NCERQA was an organization of vital importance to ORD, EPA, and the national objective of improving fundamental knowledge for environmental assessment and management. It further concluded that NCERQA had played a key role in refocusing and shaping the new vision for ORD. In the course of its review of NCERQA, BOSC made several recommendations that touched on the STAR program (EPA/BOSC 1998), including the following:

OCR for page 153
“That support and expectations of Project Officers be addressed as a priority issue in NCERQA management planning. More prioritization, and in some cases streamlining of grant and fellowship management, appear necessary. That NCERQA continue to expand its cooperation and connections with other federal, private, and international environmental research organizations through joint solicitations, Web site links, and the exchange of ideas and research results. That RFA workshop proceedings be expanded to include a record of discussions, exchange of ideas, integration across research projects and their relevancy for environmental decision making. That NCERQA continue to support and expand its Web site as a central location for information on the Center, ORD, and other organizations performing related research. That NCERQA require investigators to discuss the relevance of their research to EPA as a part of their project summaries.” NCERQA, in a written response (Noonan 1999), addressed all the recommendations and described the actions taken up to that point. The response was judged to be satisfactory. 2000 REVIEW BY A SPECIAL JOINT COMMITTEE ESTABLISHED BY EPA’S SCIENCE ADVISORY BOARD AND ORD’S BOARD OF SCIENTIFIC COUNSELORS In the latter part of 1999, EPA’s Science Advisory Board (SAB) and ORD’s BOSC established a joint committee to review the STAR program (EPA/SAB/BOSC 2000). That committee concluded that the STAR program is of vital importance to EPA’s mission and to the national objective of improving the knowledge base for environmental assessment and management. The committee concluded further that the STAR program was structured and managed to generate high-quality science by well-qualified scientists on relevant topics as identified in the EPA strategic plan. The committee’s recommendations fell along two general lines: staff resources and information transfer. The committee felt that greater staff resources were required for maximizing the public’s return on investment in the STAR program. Coincidentally, that staffing need was identified in BOSC’s 1998 review of NCERQA (EPA/BOSC 1998). Concerning information transfer, the committee felt that greater emphasis and attention needed to be placed on developing and implementing the

OCR for page 153
tools, management processes, and procedures for ensuring that the information from and results of the STAR program are rapidly and effectively transferred to agency users. Ten recommendations were made: “The Agency should provide additional information in RFAs on research goals and objectives and on budget and relevancy criteria that will be used to evaluate proposals. The Agency should take steps to accelerate the peer review process for STAR results. The Agency should select several STAR research grants as case examples and evaluate the effectiveness of the coordination with the relevant client offices and the degree to which the awards are supporting the Agency’s strategic goals. The Agency should consider means of strengthening communica-tions between Agency program staff and STAR grant recipients. The Agency should assess how well the needs and issues of the regional offices are factored into the STAR planning process and consider additional mechanisms for ensuring adequate regional involvement in STAR Program activities. The Agency should request feedback on the success of the program review workshops and should expand the workshop proceedings to include a record of discussions regarding the relevancy of STAR results to the Agency’s research and regulatory agenda and to environmental decision making. The Agency should develop and implement a process for periodi-cally assessing the Agency’s portfolio in terms of its use of different funding instruments and the reliance on different R&D performers. The Agency should continue and expand its partnerships with other agencies and funding organizations. The Agency should seek assistance from program evaluation and decision analysis experts to help ORD develop a monitoring and evaluation system for the STAR Program. The Agency should budget sufficient resources to secure the ser-vices of a qualified, highly respected, and independent organization to conduct and publish an evaluation of the STAR Program’s results, effectiveness and impact.” ORD responded to the SAB and BOSC review in a written response from Norine Noonan (Noonan 2000a) in which she acknowledged that the agency had implemented two of the recommendations, including allocating additional staff resources to the STAR program and hiring an additional

OCR for page 153
staff person to assist with the communication of results. In addition, she noted that work had begun on a pilot “state-of-the-science” report on ecologic indicators. 2000 GENERAL ACCOUNTING OFFICE REPORT The General Accounting Office (GAO) published a report in September 2000 (GAO 2000) that reviewed the STAR grants program with emphasis on three issues: whether funding amounts awarded for the grants align with EPA’s strategic goals, ORD’s research priorities, and program-office priorities; the extent to which the completed focused grants have provided research that is being used by EPA’s program offices; and how ORD could enhance its management of the program to help to ensure that it meets its objectives. Broadly speaking, GAO found that STAR grant funding had generally been aligned with EPA’s, ORD’s, and the program offices’ broadly defined priorities. It found, however, that EPA’s program officials varied in the extent to which they believed that the grants’ results were useful to them. GAO noted further that ORD could enhance its management of the program to help to ensure that it meets its objectives. GAO made three recommendations to enhance the effectiveness of the STAR program. The administrator of EPA and the assistant administrator of ORD, it said, should take the following actions: “Track and monitor the grants to ensure that interim and final re-search results are delivered on time and are made available as soon as possible for use by the program offices. Take the additional steps needed to disseminate and communicate STAR research results to the appropriate program officials better. That would require continuing and expanding the efforts already under way to consult with program officers in determining the most effective communication methods. Develop program criteria to evaluate the effectiveness of each type of grant—exploratory grants, focused grants, and fellowships. In addition, the criteria should assist EPA in drawing an overall conclusion on whether the grants satisfy the program’s overall objectives.” In a letter dated October 17, 2000, the assistant administrator responded to the GAO report (Noonan 2000b) and stated that steps were being taken

OCR for page 153
to implement the recommendations of the report and the recommendations made in the EPA SAB-BOSC study. 2001 SCIENCE ADVISORY BOARD REVIEW OF THE WATER AND WATERSHEDS PROGRAM In autumn 2001, the Ecological Processes and Effects Committee (EPEC) of EPA’s SAB reviewed the water and watersheds component of STAR (EPA/SAB 2001). The EPEC review was the first to focus on the quality and utility of the research funded by STAR in a particular subject. At the time the review was conducted, the STAR water and watersheds program had provided about $36 million over a 5-year period, and research results of multiple grant cycles were available. EPEC commended the STAR water and watersheds program for the quality of its research, for refining decision tools (primarily computer models), for producing a crop of young researchers with experience in an important environmental field, and for legitimizing transdisciplinary research in the academic community. The committee recommended that the water and watersheds program be continued with the following midcourse corrections (EPA/SAB 2001): Refine the requests for proposals so that the resulting research focuses more sharply on information gaps and policy-relevant research topics. Begin to synthesize results from the collective body of research funded by the program and disseminate this information in useful forms to the rest of EPA and its partners in state and local agencies. In addition, EPEC provided examples of metrics that could be used to judge the success of the program in the future. 2000 JOINT REVIEW BY EPA AND NSF In 2000, the National Science Foundation (NSF) and EPA convened experts from outside the agencies and users of socioeconomic research, including EPA program and regional staff, to conduct an interim assessment of the Decision Making and Valuation for Environmental Policy grants program (DMVEP) (EPA/NSF 2000; EPA 2000). NSF and EPA, which

OCR for page 153
managed the program jointly, were interested in determining whether it was producing results and communicating them effectively. At the time of the review, the DMVEP program had been in operation for 5 years, and about $2 million had been given to support it each year. The experts who conducted the review concluded that the DMVEP program fills a critical research niche that is not addressed by other research programs and commended the program for advancing the state of knowledge in an underfunded field and helping to develop a new field of study. The experts recommended the following midcourse corrections: “Increase outreach and communication efforts, to improve aware-ness both of funding opportunities and of research findings; Continue to support research on both monetary and non-mone-tizable ecosystem valuation; and Encourage research on group and institutional—as well as individ-ual—valuation and decision making for environmental policy.” Although EPA did not respond formally to those recommendations, it did implement many of the recommendations in the 2001 and 2002 DMVEP solicitations. 2002 SCIENCE ADVISORY BOARD REVIEW OF PARTICULATE-MATTER CENTERS In February 2002, the PM Research Centers Interim Review Panel of EPA’s SAB convened a review of the particulate-matter research centers program. At the time of the review, the agency had been funding five PM centers since 1999 at about $8 million per year; about half the grant period had elapsed. The review was intended to provide the agency with guidance as to whether it should continue the concept of the PM research centers beyond FY 2004 or whether there was a better mechanism of generating the research results to inform EPA’s decision making on PM (EPA/SAB 2002). The PM Research Centers Interim Review Panel concluded that the PM centers program had produced benefits beyond those normally found in individual investigator-initiated grants and that the program merited continuation. The panel identified several advantages that centers offer over traditional investigator-initiated awards, including enhanced flexibility and adaptability, leading to improved timeliness; ability to conduct higher-risk

OCR for page 153
pilot and validation efforts; study designs enhanced by intracenter multidisciplinary integration; and improved leveraging of research resources. The panel offered several recommendations for improving the centers’ program: “Focusing the centers’ efforts on the most critical PM needs in the new RFAs. Development of an informal, but overarching mechanism for pro-viding scientific advice to the centers’ program. Enhanced opportunities for cross-fertilization of ideas with EPA’s intramural researchers and the larger extramural community. Enhanced interaction between the research conducted at the centers and ongoing intensive air quality monitoring efforts. Providing mechanisms and resources for inter-center integration” (EPA/SAB 2002). 2002 BOARD OF SCIENTIFIC COUNSELORS EVALUATION In November 2002, ORD’s BOSC issued the results of its program review of NCER, one in a series of programmatic reviews the board was conducting of the ORD laboratories and centers in response to a request made in 2000 by Assistant Administrator for Research and Development Henry Longest III (EPA/BOSC 2002). The review was a follow up to the 1997 review by the same organization and relied in part on a “self-study” report prepared by the NCER staff in response to a series of 19 questions put to the organization by BOSC (EPA 2002). The review was carried out by a specially constituted subcommittee of BOSC. The subcommittee concluded that “the National Center for Environmental Research (NCER) has a strong and dynamic research program that is well connected to the Office of Research and Development (ORD) and the Environmental Protection Agency (EPA) Program Offices.” It emphasized NCER’s “strong and creative leadership” and the staff’s “enthusiasm and professionalism.” The report had the following 16 findings and recommendations (EPA/BOSC 2002): “Recommendation 1: NCER should proceed with development of its Strategic Plan as soon as possible. The plan can serve as the cornerstone for measuring the health of NCER and determining its future resource requirements.

OCR for page 153
Recommendation 2: The communications of NCER decisions and actions surrounding research initiation and prioritization decision processes would be enhanced with the development of written documentation of the processes surrounding RFA prioritization, setting of funding levels for research topics, and initiating programmatic changes in the course of MYPs. Building on some successful efforts to date, additional effort is needed to refine and focus RFA solicitations to ensure that proposers, reviewers, and EPA staff understand the scope and focus of the research area to be addressed and its importance. Recommendation 3: NCER should develop a strategy or model to gauge the balance between the number of RFAs issued and available funding. Recommendation 4: The structured, integrative process being used by NCER to identify research topics and conduct research has considerable merit. However, streamlining measures should be taken to reduce the time demands on staff while preserving the essence of an integrative process. Recommendation 5: Progress of NCER in partnering with other federal agencies has been excellent, firmly establishing it as a leader in environmental research and significantly raising its visibility. This is certainly an accomplishment for which NCER should be commended. Recommendation 6: Although NCER’s budget for social science research is small, NCER is making progress in integrating social science into other STAR program research initiatives. A broader range of social science research is needed, in addition to the focus on economics. Recommendation 7: NCER should address the issue of research balance of human and ecological areas more routinely, and clarify the rationale (not just the methodology) for the balance selected. This should address the context of balance within EPA, and more globally, considering EPA’s contributions among other federal research programs. Recommendation 8: NCER has achieved commendable progress in developing effective systems for managing its programs and motivating its staff. The BOSC encourages its continuation through, for example, careful tracking of new hires to ensure their continued enthusiasm.

OCR for page 153
Recommendation 9: Although NCER’s unique qualities and capabilities within EPA are apparent and well recognized by those familiar with EPA, there was concern that other federal research programs and legislators may not appreciate these areas of distinction. It is vital that NCER document exactly how it differs from other, related federal and non-federal programs and why it can accomplish things that these other agencies cannot. Recommendation 10: NCER’s leadership clearly recognizes the need for effective communications (and for the benefits of improved program coordination that result from such communications), as evidenced by the enviable record of publications and information bulletins from the STAR program, numbers of “hits” on its Web site, and ambitions to develop new tools. The ambition of outreach to each of the important and diverse audiences is noble. However, it is clear that full realization of the goal remains to be achieved, and working toward such a goal will have a large impact on personnel in terms of the nature and amount of effort NCER has to mount. NCER can best achieve its goals by: (1) intensifying communications between NCER and its many audiences, (2) initiating those communications earlier in the research planning process, and (3) assuring that NCER’s research results reach those who are in a position to apply them to health and/or environmental improvement. NCER should continue to expand its proactive program of education and outreach to be sure that the results of NCERfunded research take their proper role in guiding EPA’s regulatory programs, and that Congress exercises its duties in oversight and support. Recommendation 11: NCER’s lack of performance measures is linked to the absence of a strategic plan and a single agency to provide a benchmark for comparison. NCER should complete its Strategic Plan and develop reference points by gathering benchmarks from an eclectic group of agencies whose activities intersect those of NCER. Recommendation 12: NCER programs can focus on priority areas of research of interest to EPA while meeting the concerns and

OCR for page 153
interests of entities outside the Agency. We encourage NCER to continue to maintain project planning and management approaches that allow integration of broad stakeholder interests and priorities. Recommendation 13: The case studies presented in the Self-Study Report indicated that budget limitations were the cause of a research program not meeting the expectations of a Program Office (sponsor). NCER is admonished to be cognizant of others barriers (e.g., a too narrowly defined project that overlooks chronic health impacts) that also can contribute to a sponsor’s expectations not being met. Recommendation 14: NCER has a commendable leadership role in seeking development of metrics for quantitative evaluations of research quality and impact. This is a significant area, and is a challenge of sufficient importance to justify allocation of additional personnel time and research funds. Recommendation 15: NCER and ORD have provided a number of strategic opportunities and quality tools that can be used to feed NCER research results back into the EPA research planning process, providing a means to impact the establishment of research priorities and integration with EPA’s mission. It appears that this process is effective but may not yet be optimized, pending more efficient communication and timemanagement considerations. Recommendation 16: NCER is doing a good job with its current resources and recognizing its needs. As the Center shifts the responsibilities of staff, the following are offered as suggestions: Link resource type and quantity to activities defined by the Strategic Plan; Develop innovative approaches to solve paperwork bottlenecks that currently are personnel intensive (the digital processes at NSF and other agencies are recommended starting places); and Develop performance measures for internal and external communication plans.” EPA had not prepared a response to those recommendations by the beginning of December 2002.

OCR for page 153
REFERENCES EPA (U.S. Environmental Protection Agency). 2000. Evaluation Report: A Deci-sion Making and Valuation for Environmental Policy Interim Assessment. National Center for Environmental Research, Office of Research and Develop-ment, U.S. Environmental Protection Agency [Online]. Available: http://es. epa.gov/ncer/science/economics/reviews.html [accessed Jan. 13, 2003]. EPA (U.S. Environmental Protection Agency). 2002. Report to ORD’s Board of Scientific Counselors, Self-Study Update. Prepared by the National Center for Environmental Research, Office of Research and Development, U.S. Environ-mental Protection Agency. January 2002. EPA/BOSC (U.S. Environmental Protection Agency Board of Scientific Counsel-ors). 1998. Program Review of the National Center for Environmental Re-search and Quality Assurance (NCERQA). Final Report of the Ad Hoc Sub-committee on the Review of NCERQA. Board of Scientific Counselors, Office of Research and Development, U.S. Environmental Protection Agency, Wash-ington, DC. April 30, 1998. EPA/BOSC (U.S. Environmental Protection Agency Board of Scientific Counsel-ors). 2002. Second Program Review of the National Center for Environmental Research (NCER). Board of Scientific Counselors, Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC. November 19, 2002. EPA/NSF (U.S. Environmental Protection Agency and National Science Founda-tion). 2000. Interim Assessment for the Decision Making and Valuation for Environmental Policy Grants Program. Final Report. Prepared for National Science Foundation and U.S. Environmental Protection Agency, by Aspen Systems Corporation. April 17, 2000. EPA/SAB (U.S. Environmental Protection Agency Science Advisory Board). 2001. The Science to Achieve Results (STAR) Water and Watersheds Grants Program: An EPA Science Advisory Board Review. A Review by the Ecolog-ical Processes and Effects Committee (EPEC) of the EPA Science Advisory Board. EPA-SAB-EPEC-02-001. Science Advisory Board, U.S. Environmen-tal Protection Agency, Washington, DC [Online]. Available: http://www.epa. gov/science1/fiscal02.htm. [accessed Jan. 13, 2003]. EPA/SAB (U.S. Environmental Protection Agency Science Advisory Board). 2002. Interim Review of the Particulate Matter (PM) Research Centers of the USEPA: An EPA Science Advisory Report. A Review by the PM Research Centers Interim Review Panel of the Executive Committee of the U.S. EPA Science Advisory Board (SAB). EPA-SAB-EC-02-008. Science Advisory Board, U.S. Environmental Protection Agency, Washington, DC. May 2002 [Online]. Available: http://www.epa.gov/science1/fiscal02.htm [accessed Jan. 13, 2003].

OCR for page 153
EPA/SAB/BOSC (U.S. Environmental Protection Agency Science Advisory Board and Board of Scientific Counselors). 2000. A Joint SAB/BOSC Report: Review of the Science to Achieve Results (STAR) Program. EPA-SAB-EC-00-008. Science Advisory Board, Board of Scientific Counselors, U.S. Envi-ronmental Protection Agency. [Online]. Available: http://www.epa.gov/sab/ pdf/ec0008.pdf [accessed Jan. 13, 2003]. GAO (U.S. General Accounting Office). 2000. Environmental Research: STAR Grants Focus on Agency Priorities, But Management Enhancements Are Possi-ble: Report to the Chairman, Subcommittee on VA, HUD, and Independent Agencies, Committee on Appropriations, House of Representatives. GAO/RCED-00-170/B-142370. U.S. General Accounting Office, Washington, DC. Noonan, N.E. 1999. Letter from N. Noonan, Assistant Administrator, Office of Research and Development, U.S. Environmental Protection Agency, Washing-ton, DC, to C. Denson, Chair of BOSC, University of Delaware, Newark, DE. May 19, 1999. Noonan, N.E. 2000a. An SAB/BOSC Report: Review of the Science to Achieve Results (STAR) Program (EPA-SAB-EC-00-008). Letter from N. Noonan, Assistant Administrator, Office of Research and Development, U.S. Environ-mental Protection Agency, Washington, DC, to C. Denson, Chair, Board of Scientific Counselors, University of Delaware, Newark, DE. May 26, 2000. Noonan, N.E. 2000b. Response to GAO Final Report, Environmental Research: STAR Grants Focus on Agency Priorities, but Management Enhancements are Possible (GAO/RCED-00-170). Letter from N. Noonan, assistant administra-tor, Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC, to N. Gelb, director, Annual Planning and Budget Division, Office of the Chief Financial Officer, GAO, Washington, DC. Octo-ber 7, 2000.