Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 109
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
Appendix F
Draft Board of Scientific Counselors Handbook for Subcommittee Chairs: Draft Proposed Charge Questions for BOSC Reviews1
PROGRAM ASSESSMENT (EVALUATE ENTIRE RESEARCH PROGRAM)
The responses to the program assessment charge questions below will be in a narrative format, and will capture the performance for the entire research program and all the activities in support of the program’s Long Term Goals (LTGs). The Long term Goals should be consistent with EPA’s Strategic Plan and mutually agreed upon by ORD and OMB.
Program Relevance
How consistent are the Long Term Goals (LTGs) of the program with achieving the Agency’s strategic plan and ORD’s Multi-Year Plan?
How responsive is the program focus to program office and regional research needs?
How responsive is the program to recommendations from outside advisory boards and stakeholders?
How clearly evident are the public benefits of the program?
Program Structure
How clear a logical framework do the LTGs provide for organizing and planning the research and demonstrating outcomes of the program?
1
EPA 2007.
OCR for page 110
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
How appropriate is the science used to achieve each LTG, i.e., is the program still asking the right questions, or has it been eclipsed by advancements in the field?
Does the MYP describe an appropriate flow of work (i.e., the sequencing of related activities) that reasonably reflects the anticipated pace of scientific progress and timing of client needs?
Does the program use the MYP to help guide and manage its research?
How logical is the program design, with clearly identified priorities?
Program Performance
How much progress is the program making on each LTG based on clearly stated and appropriate milestones?
Program Quality
How good is the scientific quality of the program’s research products?
What means does the program employ to ensure quality research (including peer review, competitive funding, etc.?
How effective are these processes?
Scientific Leadership
Please comment on the leadership role the research program and its staff have in contributing to advancing the current state of the science and solving important research problems.
Coordination and Communication
How effectively does the program engage scientists and managers from ORD and relevant program offices in its planning?
How effectively does the program engage outside organizations, both within and outside government, to promote collaboration, obtain input on program goals and research, and avoid duplication of effort?
How effective are the mechanisms that the program uses for communicating research results both internally and externally?
Outcomes
How well-defined are the program’s measures of outcomes?
How much are the program results being used by environmental decision makers to inform decisions and achieve results?
OCR for page 111
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
SUMMARY ASSESSMENT (RATE PROGRAM PERFORMANCE BY LTG)
The responses to the three summary assessment charge questions below will rate the performance for each LTG. For each LTG, a qualitative score will be assigned that reflects the quality and significance of the research as well as the extent to which the program is meeting or making measurable progress toward the goal—relative to the information and evidence provided to the BOSC. The scores will be given in the form of adjectives that are clearly defined and which are intended to promote consistency among reviews. The adjectives will be used as part of a narrative summary of the review of each LTG so that the context of the rating and the rationale for selecting a particular rating will be transparent. The rating may reflect considerations beyond the summary assessment questions, and will be explained in the narrative. The adjectives to describe progress are:
Exceptional: indicates that the program is meeting all and exceeding some of its goals, both in the quality of the science being produced and the speed at which research result tools and methods are being produced. An exceptional rating also indicates that the program is addressing the right questions to achieve its goals. The review should be specific as to which aspects of the program’s performance have been exceptional.
Exceeds Expectations: indicates that the program is meeting all of its goals. It addresses the appropriate scientific questions to meet its goals and the science is competent or better. It exceeds expectations for either the high quality of the science or for the speed at which work products are being produced and milestones met.
Meets Expectations: indicates that the program is meeting most of its goals. Satisfactory programs live up to expectations in terms of addressing the appropriate scientific questions to meet its goals, and that work products are being produced and milestones are being reached in a timely manner. The quality of the science being done is competent or better.
Not Satisfactory: indicates that the program is failing to meet a substantial fraction of its goals, or if meeting them, that the achievement of milestones is significantly delayed, or that the questions being addressed are inappropriate or insufficient to meet the intended purpose. Questionable science is also a reason for rating a program as unsatisfactory for a particular long term goal. The review should be specific as to which aspects of a program’s performance have been inadequate.
For each program review, the summary assessment charge questions below will be tailored to the specific review and LTG:
OCR for page 112
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
How appropriate is the science used to achieve each LTG, i.e., is the program still asking the right questions, or has it been eclipsed by advancements in the field?
How good is the scientific quality of the program’s research products?
How much are the program results being used by environmental decision makers to inform decisions and achieve results?
REFERENCES
EPA (U.S. Environmental Protection Agency). 2007. Draft Board of Scientific Counselors Handbook for Subcommittee Chairs. Board of Scientific Counselors, U.S. Environmental Protection Agency, Washington, DC.
