The BOSC Safe Pesticides/Safe Products (SP2) Subcommittee will conduct a retrospective and prospective review of ORD’s SP2 Research Program, and evaluate the program’s relevance, quality, performance, and scientific leadership. The BOSC’s evaluation and recommendations will provide guidance to the Office of Research and Development to help:
plan, implement, and strengthen the program;
compare the program with programs designed to achieve similar outcomes in other parts of EPA and in other federal agencies;
make research investment decisions over the next five years;
prepare EPA’s performance and accountability reports to Congress under the Government Performance and Results Act; and
respond to assessments of federal research programs such as those conducted by the
Office of Management and Budget (OMB highlights the value of recommendations from independent expert panels in guidance to federal agencies).
Independent expert review is used extensively in industry, federal agencies, Congressional committees, and academia. The National Academy of Science has recommended this approach for evaluating federal research programs.
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 119
Appendix H Charge to the BOSC Subcommittee on Safe Pesticides/Safe Products Research1 OBJECTIVE The BOSC Safe Pesticides/Safe Products (SP2) Subcommittee will con- duct a retrospective and prospective review of ORD’s SP2 Research Program, and evaluate the program’s relevance, quality, performance, and scientific lead- ership. The BOSC’s evaluation and recommendations will provide guidance to the Office of Research and Development to help: • plan, implement, and strengthen the program; • compare the program with programs designed to achieve similar out- comes in other parts of EPA and in other federal agencies; • make research investment decisions over the next five years; • prepare EPA’s performance and accountability reports to Congress un- der the Government Performance and Results Act; and • respond to assessments of federal research programs such as those con- ducted by the Office of Management and Budget (OMB highlights the value of recom- mendations from independent expert panels in guidance to federal agencies). BACKGROUND INFORMATION Independent expert review is used extensively in industry, federal agen- cies, Congressional committees, and academia. The National Academy of Sci- ence has recommended this approach for evaluating federal research programs. 1 (EPA 2007). 119
OCR for page 119
120 Evaluating Research Efficiency in EPA Because of the nature of research, it is not possible to measure the creation of new knowledge as it develops–or the pace at which research progresses or scientific breakthroughs occur. Demonstrating research contributions to out- comes is very challenging when federal agencies conduct research to support regulatory decisions, and then rely on third parties–such as state environmental agencies–to enforce the regulations and demonstrate environmental improve- ments. Typically, many years may be required for practical research applications to be developed and decades may be required for some research outcomes to be achieved in a measurable way. Most of ORD’s environmental research programs investigate complex en- vironmental problems and processes—combining use-inspired basic research with applied research, and integrating several scientific disciplines across a con- ceptual framework that links research to environmental decisions or environ- mental outcomes. In multidisciplinary research programs such as these, progress toward outcomes can not be measured by outputs created in a single year. Rather, research progress occurs over several years, as research teams explore hypotheses with individual studies, interpret research findings, and then develop hypotheses for future studies. In designing and managing its research programs, ORD emphasizes the importance of identifying priority research questions or topics to guide its re- search. Similarly, ORD recommends that its programs develop a small number of performance goals that serve as indicators of progress to answer the priority questions and to accomplish outcomes. Short-term outcomes are accomplished when research is applied by specific clients, e.g., to strengthen environmental decisions. These decisions and resulting actions (e.g., the reduction of contami- nant emissions or restoration of ecosystems) ultimately contribute to improved environmental quality and health. In a comprehensive evaluation of science and research at EPA, the Na- tional Research Council recommended that the Agency substantially increase its efforts to both explain the significance of its research products and to assist cli- ents inside and outside the Agency in applying them. In response to this recom- mendation, ORD has engaged science advisors from client organizations to serve as members of its research program teams. These teams help identify re- search contributions with significant decision making value and help plan for their transfer and application. For ORD’s environmental research programs, periodic retrospective analysis at intervals of four or five years is needed to characterize research pro- gress, to assess how clients are applying research to strengthen environmental decisions, and to evaluate client feedback about the research. Conducting pro- gram evaluations at this interval enables assessment of: research progress, the scientific quality and decision-making value of the research, and whether re- search progress has resulted in short-term outcomes for specific clients. A description of the OSTP/OMB Research and Development Investment Criteria is included in Appendix I.
OCR for page 119
121 Appendix H BACKGROUND FOR ORD’S SP2 RESEARCH PROGRAM AND DRAFT CHARGE QUESTIONS BACKGROUND The purpose of the SP2 Research Program is to provide EPA’s Office of Prevention, Pesticides, and Toxic Substances (OPPTS) with the scientific infor- mation it needs to reduce or prevent unreasonable risks to humans, wildlife, and non-target plants from exposures to pesticides, toxic chemicals, and products of biotechnology. The SP2 Research Program specifically addresses OPPTS’ high priority research needs that are not addressed by any of ORD’s other research programs. The research program is focused on three Long Term Goals: Long Term Goal 1: OPPTS and/or other organizations use the results of ORD’s research on methods, models, and data as the scientific foundation for: A) prioritization of testing requirements, B) enhanced interpretation of data to improve human health and ecological risk assessments, and C) deci- sionmaking regarding specific individual or classes of pesticides and toxic substances that are of high priority. The ultimate outcomes are the develop- ment of improved methods, models, and data for OPPTS’ use in requiring test- ing, evaluating data, completing risk assessments, and determining risk man- agement approaches. More specifically the outcomes are the development by ORD and implementation by OPPTS of more efficient and effective testing paradigms that will be better informed by predictive tools (chemical identifica- tion, improved targeting, less cost, less time, and fewer animals); improved methods by which data from the more efficient and effective testing paradigms can be integrated into risk assessments; and that OPPTS uses the result of ORD’s multidisciplinary research approaches, that it specifically requests, for near term decisionmaking on high priority individual or classes of pesticides and toxic substances. Long Term Goal 2: OPPTS and/or other organizations use the results of ORD’s research as the scientific foundation for probabilistic risk assess- ments to protect natural populations of birds, fish, other wildlife, and non- target plants. Results of this research will help the Agency meet the long term goal of developing scientifically valid approaches to extrapolate across species, biological endpoints and exposure scenarios of concern, and to assess spatially explicit, population-level risks to wildlife populations and non-target plants and plant communities from pesticides, toxic chemicals and multiple stressors, while advancing the development of probabilistic risk assessment. Long Term Goal 3: OPPTS and/or other organizations use the results of ORD’s biotechnology research as the scientific foundation for decisionmak- ing related to products of biotechnology. OPPTS will use the results from this research program to update its requirements of registrants of products of bio- technology and to help evaluate data submitted for its review.
OCR for page 119
122 Evaluating Research Efficiency in EPA The scope of the SP2 research program has been developed in partnership with OPPTS. ORD keeps abreast of complementary research ongoing in other federal agencies and scientific organizations. However, no other programs have similar goals, in terms of scope and mission, as the SP2 research program that provides OPPTS with the tools it needs to carry out its regulatory mandates. EPA’s SP2 research is multi-disciplinary, including: 1) research across all as- pects of the risk assessment/risk management paradigm, i.e., in effects, expo- sure, risk assessment, and risk management; and 2) as related to humans, wild- life, and plants. Comparison of potential benefits is conducted from a scientific perspective through coordinating and collaborating with other research pro- grams, participating at national and international scientific for a, and keeping abreast of state of the science. EPA’s SP2 program includes many areas that are of unique importance in helping OPPTS meet its legislative mandates, such as requiring industry to submit data on pesticides, toxic substances, and products of biotechnology. The SP2 program also includes other research areas that serve to improve the basic scientific understanding regarding these agents that OPPTS and other parts of the Agency need to evaluate data submissions, conduct risk assessments, and make informed management decisions. Furthermore, ORD’s intramural program is complemented by an extramural program implemented through the Science to Achieve Results (STAR) program. The research directions to address the key areas of scientific uncertainty are captured in the current version of the SP2 Multi-Year Plan (MYP). The MYP includes research activities implemented and planned for the period 2007 through 2015. The research described in the MYP assumes annual intramural and extramural resources of approximately 126 FTEs and $24.8 million, includ- ing payroll, travel and operating expenses. DRAFT CHARGE Program Assessment (Evaluate Entire Research Program) The responses to the program assessment charge questions below should be in a narrative format, and should capture the performance for the entire re- search program and all the activities in support of the program’s Long Term Goals (LTGs). Program Relevance 1. How consistent are the Long Term Goals (LTGs) of the program with achieving the Agency’s strategic plan and ORD’s Multi-Year Plan? 2. How responsive is the program focus to program office and regional re- search needs?
OCR for page 119
123 Appendix H 3. How responsive is the program to recommendations from outside advi- sory boards and stakeholders? 4. How clearly evident are the public benefits of the program? Factors to consider: the degree to which the research is driven by EPA priorities; the degree to which this research program has had (or is likely to have) an impact on Agency decisionmaking; and the extent to which research program scientists participate on or contribute to Agency workgroups and trans- fer research to program and regional customers. Program Structure 1. How clear a logical framework do the LTGs provide for organizing and planning the research and demonstrating outcomes of the program? 2. How appropriate is the science used to achieve each LTG, i.e., is the program asking the right questions, or has it been eclipsed by advancements in the field? 3. Does the MYP describe an appropriate flow of work (i.e., the sequenc- ing of related activities) that reasonably reflects the anticipated pace of scientific progress and timing of client needs? 4. Does the program use the MYP to help guide and manage its research? 5. How logical is the program design, with clearly identified priorities? Factors to consider: the appropriateness of the key science questions; the appropriateness of the Long Term Goals in providing a logical framework for organizing the SP2 program to best meet the Agency’s needs; the degree of clar- ity to the path of annual research products aimed at accomplishing each of the LTGs; the scientific soundness of the approaches used; the appropriateness of the research products identified in the MYP as the means to meet the highest priority research for each LTG; and the adequacy/sufficiency/necessity of the sets of APMs under the APGs to accomplish the intended goals. Program Performance 1. How much progress is the program making on each LTG based on clearly stated and appropriate milestones? Factors to consider: the scientific soundness of the approaches used; the degree to which scientific understanding of the problem has been advanced; the degree to which scientific uncertainty has been reduced; the impact and use of research results by EPA program and regional offices and by other organiza- tions; and the extent of the bibliography of peer reviewed publications.
OCR for page 119
124 Evaluating Research Efficiency in EPA Program Quality 1. How good is the scientific quality of the program’s research products? 2. What means does the program employ to ensure quality research (in- cluding peer review, competitive funding, etc.)? 3. How effective are these processes? Factors to consider: the impact and use of research results by EPA pro- gram and regional offices and other organizations; the degree to which peer re- viewed publications from this program are cited in other peer reviewed publica- tions, the immediacy with which they are cited, and their impact factor; the processes used to peer review intramural research designs and products (e.g., division-level or product-level reviews by independent panels); and the proc- esses used in the competitive extramural grants program. Scientific Leadership 1. Please comment on the leadership role the research program and its staff have in contributing to advancing the current state of the science and solv- ing important research problems. Factors to consider: the degree to which this program is identified as a leader in the field; the degree to which peer reviewed publications from this program are cited in other peer reviewed publications, the immediacy with which they are cited, and their impact factor; the degree to which SP2 scientists serve/are asked to serve on national/international workgroups, officers in profes- sional societies, publication boards; the degree to which SP2 scientists lead na- tional/international collaborative efforts, organize national/international confer- ences/symposia, and are awarded for their contributions/leadership; and benchmarking of scientific leadership relative to other programs, agencies, and countries. Coordination and Communication 1. How effectively does the program engage scientists and managers from ORD and relevant program offices in its planning? 2. How effectively does the program engage outside organizations, both within and outside government, to promote collaboration, obtain input on pro- gram goals and research, and avoid duplication of effort? 3. How effective are the mechanisms that the program uses for communi- cating research results both internally and externally? Factors to consider: the extent to which program/regional office scien- tists/managers are involved in planning the research; research activities of other
OCR for page 119
125 Appendix H federal agencies, industry, academic institutions, other countries; the degree of collaboration and coordination with other research organizations; and the means that are used to communicate results to OPPTS and to the external scientific community (e.g., through peer reviewed publications, scientific meetings, seminars). Outcomes 1. How well-defined are the program’s measures of outcomes? 2. How much are the program results being used by environmental deci- sion makers to inform decisions and achieve results? Factors to consider: the extent to which the MYP identifies the past or an- ticipated impact of the research activities; and the extent to which the research has contributed/or is anticipated to contribute to Agency and other decision- making. Summary Assessment (Rate Program Performance By LTG) A summary assessment and narrative should be provided for each LTG. The assessment should be based on 3 of the questions included above, which are: 1. How appropriate is the science used to achieve each LTG, i.e., is the program asking the right questions, or has it been eclipsed by advancements in the field? 2. How good is the scientific quality of the program’s research products? 3. How much are the program results being used by environmental deci- sion makers to inform decisions and achieve results? Elements to Include for Long-Term Goal 1 The appropriateness, quality, and use of ORD science by OPPTS and other organizations to inform decisions and achieve results with respect to 1) prioritization testing requirements, 2) enhancing the interpretation of data to improve human health and ecological risk assessments, and 3) making decisions regarding specific individual or classes of high priority pesticides and toxic sub- stances. The extent to which ORD is asking the right questions, conducting the right science, and providing products that are responsive to OPPTS’s and other organizations’ needs.
OCR for page 119
126 Evaluating Research Efficiency in EPA Elements to Include for Long-Term Goal 2 The appropriateness, quality, and use of ORD science by OPPTS and other organizations to inform decisions and achieve results with respect to prob- abilistic risk assessments to protect natural populations of birds, fish, other wild- life, and non-target plants. The extent to which ORD is asking the right ques- tions, conducting the right science, and providing products that are responsive to OPPTS’ and other organizations’ needs Elements to Include for Long-Term Goal 3 The appropriateness, quality, and use of ORD science by OPPTS and other organizations to inform decisions and achieve results with respect to prod- ucts of biotechnology. The extent to which ORD is asking the right questions, conducting the right science, and providing products that are responsive to OPPTS’ and other organizations’ needs. For each LTG, the BOSC SP2 Subcommittee will assign a qualitative score that reflects the quality and significance of the research as well as the ex- tent to which the program is meeting or making measurable progress toward the goal—relative to the evidence provided to the BOSC. The scores should be in the form of the following adjectives that are defined below and intended to pro- mote consistency among BOSC program reviews. The adjectives should be used as part of a narrative summary of the review, so that the context of the rating and the rationale for selecting a particular rating will be transparent. The rating may reflect considerations beyond the summary assessment questions, and will be explained in the narrative. The adjectives to describe progress are: • Exceptional: indicates that the program is meeting all and exceeding some of its goals, both in the quality of the science being produced and the speed at which research result tools and methods are being produced. An excep- tional rating also indicates that the program is addressing the right questions to achieve its goals. The review should be specific as to which aspects of the pro- gram’s performance have been exceptional. • Exceeds Expectations: indicates that the program is meeting all of its goals. It addresses the appropriate scientific questions to meet its goals and the science is competent or better. It exceeds expectations for either the high quality of the science or for the speed at which work products are being produced and milestones met. • Meets Expectations: indicates that the program is meeting most of its goals. Programs meet expectations in terms of addressing the appropriate scien- tific questions to meet its goals, and that work products are being produced and milestones are being reached in a timely manner. The quality of the science be- ing done is competent or better.
OCR for page 119
127 Appendix H • Not Satisfactory: indicates that the program is failing to meet a substan- tial fraction of its goals, or if meeting them, that the achievement of milestones is significantly delayed, or that the questions being addressed are inappropriate or insufficient to meet the intended purpose. Questionable science is also a rea- son for rating a program as unsatisfactory for a particular long term goal. The review should be specific as to which aspects of a program’s performance have been inadequate. REFERENCES EPA (U.S. Environmental Protection Agency). 2007. Review of the Office of Research and Development’s Safe Pesticides/Safe Products (SP2) Research at the U.S. Envi- ronmental Protection Agency. Board of Scientific Counselors, U.S. Environmental Protection Agency, Washington, DC.