Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 109
Appendix F Draft Board of Scientific Counselors Handbook for Subcommittee Chairs: Draft Proposed Charge Questions for BOSC Reviews1 PROGRAM ASSESSMENT (EVALUATE ENTIRE RESEARCH PROGRAM) The responses to the program assessment charge questions below will be in a narrative format, and will capture the performance for the entire research program and all the activities in support of the program’s Long Term Goals (LTGs). The Long term Goals should be consistent with EPA’s Strategic Plan and mutually agreed upon by ORD and OMB. Program Relevance 1. How consistent are the Long Term Goals (LTGs) of the program with achieving the Agency’s strategic plan and ORD’s Multi-Year Plan? 2. How responsive is the program focus to program office and regional re- search needs? 3. How responsive is the program to recommendations from outside advi- sory boards and stakeholders? 4. How clearly evident are the public benefits of the program? Program Structure 1. How clear a logical framework do the LTGs provide for organizing and planning the research and demonstrating outcomes of the program? 1 EPA 2007. 109
OCR for page 109
110 Evaluating Research Efficiency in EPA 2. How appropriate is the science used to achieve each LTG, i.e., is the program still asking the right questions, or has it been eclipsed by advancements in the field? 3. Does the MYP describe an appropriate flow of work (i.e., the sequenc- ing of related activities) that reasonably reflects the anticipated pace of scientific progress and timing of client needs? 4. Does the program use the MYP to help guide and manage its research? 5. How logical is the program design, with clearly identified priorities? Program Performance 1. How much progress is the program making on each LTG based on clearly stated and appropriate milestones? Program Quality 1. How good is the scientific quality of the program’s research products? 2. What means does the program employ to ensure quality research (in- cluding peer review, competitive funding, etc.? 3. How effective are these processes? Scientific Leadership 1. Please comment on the leadership role the research program and its staff have in contributing to advancing the current state of the science and solv- ing important research problems. Coordination and Communication 1. How effectively does the program engage scientists and managers from ORD and relevant program offices in its planning? 2. How effectively does the program engage outside organizations, both within and outside government, to promote collaboration, obtain input on pro- gram goals and research, and avoid duplication of effort? 3. How effective are the mechanisms that the program uses for communi- cating research results both internally and externally? Outcomes 1. How well-defined are the program’s measures of outcomes? 2. How much are the program results being used by environmental deci- sion makers to inform decisions and achieve results?
OCR for page 109
Appendix F 111 SUMMARY ASSESSMENT (RATE PROGRAM PERFORMANCE BY LTG) The responses to the three summary assessment charge questions below will rate the performance for each LTG. For each LTG, a qualitative score will be assigned that reflects the quality and significance of the research as well as the extent to which the program is meeting or making measurable progress to- ward the goal—relative to the information and evidence provided to the BOSC. The scores will be given in the form of adjectives that are clearly defined and which are intended to promote consistency among reviews. The adjectives will be used as part of a narrative summary of the review of each LTG so that the context of the rating and the rationale for selecting a particular rating will be transparent. The rating may reflect considerations beyond the summary assess- ment questions, and will be explained in the narrative. The adjectives to describe progress are: • Exceptional: indicates that the program is meeting all and exceeding some of its goals, both in the quality of the science being produced and the speed at which research result tools and methods are being produced. An excep- tional rating also indicates that the program is addressing the right questions to achieve its goals. The review should be specific as to which aspects of the pro- gram’s performance have been exceptional. • Exceeds Expectations: indicates that the program is meeting all of its goals. It addresses the appropriate scientific questions to meet its goals and the science is competent or better. It exceeds expectations for either the high quality of the science or for the speed at which work products are being produced and milestones met. • Meets Expectations: indicates that the program is meeting most of its goals. Satisfactory programs live up to expectations in terms of addressing the appropriate scientific questions to meet its goals, and that work products are being produced and milestones are being reached in a timely manner. The qual- ity of the science being done is competent or better. • Not Satisfactory: indicates that the program is failing to meet a substan- tial fraction of its goals, or if meeting them, that the achievement of milestones is significantly delayed, or that the questions being addressed are inappropriate or insufficient to meet the intended purpose. Questionable science is also a rea- son for rating a program as unsatisfactory for a particular long term goal. The review should be specific as to which aspects of a program’s performance have been inadequate. For each program review, the summary assessment charge questions be- low will be tailored to the specific review and LTG:
OCR for page 109
112 Evaluating Research Efficiency in EPA 1. How appropriate is the science used to achieve each LTG, i.e., is the program still asking the right questions, or has it been eclipsed by advancements in the field? 2. How good is the scientific quality of the program’s research products? 3. How much are the program results being used by environmental deci- sion makers to inform decisions and achieve results? REFERENCES EPA (U.S. Environmental Protection Agency). 2007. Draft Board of Scientific Counsel- ors Handbook for Subcommittee Chairs. Board of Scientific Counselors, U.S. En- vironmental Protection Agency, Washington, DC.