Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 173
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health* This is a document prepared by the National Academies’ Committee for the Review of NIOSH Research Programs,1 also referred to as the Framework Committee. This document is not a formal report of the National Academies—rather, it is a framework proposed for use by a number of National Academies committees that will be reviewing research in various research programs and health-outcomes programs. This version will be posted on the website of the National Academies and NIOSH for review. It is a working document that will be subject to change by the Framework Committee aimed at improving its relevance on the basis of responses received from evaluation committee members, NIOSH, stakeholders, and the general public before and during the course of the assessments conducted by independent evaluation committees of up to 15 research programs and health-outcomes programs. * Version of 12/19/05. 1 Members of the National Academies’ Committee for the Review of NIOSH Research Programs include: David Wegman (Chair; University of Massachusetts Lowell School of Health and Environment), William Bunn, III (International Truck and Engine Corporation), Carlos Camargo (Harvard Medical School), Susan Cozzens (Georgia Institute of Technology), Letitia Davis (Massachusetts Department of Public Health), James Dearing (Kaiser Permanente), Fred Mettler, Jr. (University of New Mexico School of Medicine), Franklin Mirer (Hunter School of Health Sciences), Jacqueline Nowell (United Food and Commercial Workers International Union), Raja Ramani (Pennsylvania State University), Jorma Rantanen (Finnish Institute of Occupational Health), Rosemary Sokas (University of Illinois at Chicago School of Public Health), Richard Tucker (Tucker and Tucker Consultants, Inc. and University of Texas at Austin), and James Zuiches (North Carolina State University).
OCR for page 174
OCR for page 175
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health CONTENTS Acronyms used in this document I. Overview of Charge I.A. NIOSH Strategic Goals and Operational Plan I.B. Information from Other Evaluations I.C. Evaluation Committees I.D. Evaluation Committees’ Information Needs II. Summary of Evaluation Process II.A. The Evaluation Flow Chart (Figure 2) II.B. Steps in Program Evaluation II.C. Assessing Relevance II.D. Assessing Impact III. Evaluation of NIOSH Research Programs—the Process III.A. Analysis of External Factors Relevant to the NIOSH Program III.A.1. Overview III.A.2. Considerations for Discussion III.B. Evaluating NIOSH Research Programs (Addressing Charges 1 and 2) III.B.1. Identifying Period of Time to be Evaluated III.B.2. Identification of Major Challenges (Circle in Figure 2) III.B.3. Analysis of NIOSH Program Strategic Goals and Objectives (Box A in Figure 2) III.B.4. Review of Inputs (Box B in Figure 2) III.B.5. Review of Activities (Box C in Figure 2) III.B.6. Review of Outputs (Box D in Figure 2) III.B.7. Review of Intermediate Outcomes (Box E in Figure 2) III.B.8. Review of End Outcomes (Box F in Figure 2) III.B.9. Review of Other Outcomes III.B.10. Summary Evaluation Ratings and Rationale III.C. Identifying Significant Emerging Research (addressing Charge 3) IV. Evaluation Committee Report Template V. Framework Committee Final Report Figure 1 The NIOSH operational plan presented as a logic model Figure 2 Flow chart for the evaluation of the NIOSH research program
OCR for page 176
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health Table 1 NORA High-Priority Research Areas by Category Table 2 Examples of NIOSH Program Research and Transfer Activities Table 3 Examples of a Variety of Scientific Information Outputs Table 4 Evaluation Committee Worksheet to Assess Research Programs and Subprograms
OCR for page 177
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health ACRONYMS ABLES Adult Blood Lead Epidemiology and Surveillance ACOEM American College of Occupational and Environmental Medicine AOEC Association of Occupational and Environmental Clinics BLS Bureau of labor Statistics CDC Centers for Disease Control and Prevention EC Evaluation Committee FACE Fatality Assessment Control and Evaluation FC Framework Committee HHE Health Hazard Evaluations MSHA Mine Safety and Health Administration NEISS National Electronic Injury Surveillance System NIOSH National Institute for Occupational Safety and Health NORA National Occupational Research Agenda NORA1 National Occupational Research Agenda 1996-2005 NORA2 National Occupational Research Agenda 2005-forward OSHA Occupational Safety and Health Administration OSHAct Occupational Safety and Health Act of 1970 OSH Review Commission Occupational Safety and Health Review Commission PART Performance Assessment Rating Tool PEL Permissible Exposure Limits SENSOR Sentinel Event Notification System of Occupational Risks TMT Tools, Methods, or Technologies
OCR for page 178
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health In September 2004, the National Institute for Occupational Safety and Health (NIOSH) contracted with the National Academies to conduct a review of NIOSH research programs. The goal of this multiphase effort is to assist NIOSH in increasing the impact of its research efforts in reducing workplace illnesses and injuries and improving occupational safety and health. The National Academies agreed to conduct this review and assigned the task to the Division on Earth and Life Studies and the Institute of Medicine. The National Academies appointed a committee of 14 members, including persons with expertise in occupational medicine and health, industrial health and safety, industrial hygiene, epidemiology, civil and mining engineering, sociology, program evaluation, communication, and toxicology; representatives of industry and of the workforce; and a scientist experienced in international occupational-health issues. The Committee on the Review of NIOSH Research Programs, referred to as the Framework Committee (FC), held meetings during 2005 on May 5-6 and July 7-8 in Washington, DC, and on August 15-16 in Woods Hole and Falmouth, MA. This document is not a report of the National Academies; rather, it presents the evaluation framework developed by the FC to serve as a guideline and structure for NIOSH program reviews by Evaluation Committees (ECs) to be appointed by various divisions and boards of the National Academies. The ECs will use this framework in reviewing as many as 15 NIOSH research programs during a 5-year period. This is a working document. It is shared with NIOSH and the public. The framework and criteria may be modified by the FC on the basis of responses it receives from the ECs and other sources. It is incumbent upon the ECs to consult with the FC if portions of the evaluation framework presented here are inappropriate for the specific program under review. I. OVERVIEW OF CHARGE At the first meeting of the FC, Lewis Wade, NIOSH senior science advisor, emphasized that the reviews should focus on evaluating NIOSH’s research programs impact and relevance to health and safety in the workplace. In developing a framework, the FC was asked to address the following: Evaluation committee assessment of progress in reducing workplace illnesses and injuries facilitated by occupational safety and health research through (a) an analysis of relevant data about workplace illnesses and injuries for the program activity, and (b) an evaluation of the effect that NIOSH research has had in reducing illnesses and injuries. The evaluation committees will rate the performance of each
OCR for page 179
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health program for impact of the program in the workplace. Impact may be assessed directly or, as necessary, using intermediate outcomes to estimate impact. Qualitative narrative evaluations may also be appropriate under certain circumstances. Evaluation committee assessment of progress in targeting new research to the areas of occupational safety and health most relevant to future improvements in workplace protection. Evaluation committee identification of significant emerging research areas which appear especially important in terms of their relevance to the mission of NIOSH. Those three charges constitute the scope of work of the individually appointed, independent ECs formed by the National Academies. I.A. NIOSH Strategic Goals and Operational Plan As a prelude to understanding the NIOSH strategic goals and operational plan, NIOSH research efforts should be understood in the context of the Occupational Safety and Health Act (OSHAct) under which it was created. The OSHAct identifies workplace safety and health to be a national priority and gives employers the responsibility for controlling hazards and preventing workplace injury and illness. The act creates an organizational framework for doing this, with complementary roles and responsibilities assigned to employers and employees, OSHA, the States, the OSH Review Commission, and NIOSH. As one component of a national strategy the act recognizes NIOSH’s roles and responsibilities to be supportive and indirect—NIOSH’s research, training programs, criteria and recommendations are all intended to be used to inform and assist those actually responsible for hazard control (OSHAct Section 2b and Sections 20 and 22). Section 2b of the OSHAct describes thirteen interdependent means of accomplishing the national goal, one of which is “by providing for research … and by developing innovative methods … for dealing with occupational safety and health problems.” Sections 20 and 22 give the responsibility for this research to NIOSH. In addition, NIOSH is given related responsibilities including: the development of criteria to guide prevention of work-related injury or illness, development of regulations reporting on the employee exposures to harmful agents, the establishment of medical examinations programs or tests to determine illness incidence and susceptibility, publication of a list of all known toxic substances, the assessment of potentially toxic effects or risk associated with workplace exposures in specific settings, the conduct of education programs for relevant professionals to carry out
OCR for page 180
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health the OSHAct purposes, and assisting the Secretary of Labor regarding education programs for employees and employers in hazard recognition and control. The NIOSH mission is “to provide national and world leadership to prevent work-related illness, injury, disability, and death by gathering information, conducting scientific research, and translating the knowledge gained into products and services”. To fulfill its mission, NIOSH has established the following strategic goals:2 Goal 1: Conduct research to reduce work-related illnesses and injuries. Track work-related hazards, exposures, illnesses, and injuries for prevention. Generate new knowledge through intramural and extramural research programs. Develop innovative solutions for difficult-to-solve problems in high-risk industrial sectors. Goal 2: Promote safe and healthy workplaces through interventions, recommendations, and capacity-building. Enhance the relevance and utility of recommendations and guidance. Transfer research findings, technologies, and information into practice. Build capacity to address traditional and emerging hazards. Goal 3: Enhance global workplace safety and health through international collaborations. Take a leadership role in developing a global network of occupational health centers. Investigate alternative approaches to workplace illness and injury reduction and provide technical assistance to put solutions in place. Build global professional capacity to address workplace hazards through training, information sharing, and research experience. In 1994, NIOSH embarked on a national partnership effort to identify research priorities to guide occupational health and safety research for the next decade. The National Occupational Research Agenda (NORA) identified 21 high-priority research areas (see Table 1). NORA was intended not only for NIOSH but for the entire occupational health community. Approaching the 10-year anniversary of 2 See also http://www.cdc.gov/niosh/docs/strategic/.
OCR for page 181
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health TABLE 1 NORA High-Priority Research Areas by Category Category Priority Research Area Disease and injury Allergic and irritant dermatitis Asthma and chronic obstructive pulmonary disease Fertility and pregnancy abnormalities Hearing loss Infectious diseases Low-back disorders Musculoskeletal disorders of upper extremities Trauma Work environment and workforce Emerging technologies Indoor environment Mixed exposures Organization of work Special populations at risk Research tools and approaches Cancer research methods Control technology and personal protective equipment Exposure-assessment methods Health-services research Intervention-effectiveness research Risk-assessment methods Social and economic consequences of workplace illness and injury Surveillance research methods NORA, NIOSH is working with its partners to update the research agenda. In the second decade of NORA, an approach based on industry sectors will be pursued. NIOSH and its partners will form sector research councils that will work to establish sector-specific research goals and objectives. Emphasis will be placed on moving research to practice in workplaces through sector-based partnerships. Figure 1 is the NIOSH operational plan presented as a logic model3 of the path from inputs to outcomes for each NIOSH research program. The FC adapted the model to develop its framework. NIOSH will provide similar logic models relevant to each research program evaluated by an EC. I.B. Information from Other Evaluations The FC is aware that several NIOSH programs have already been subjected to evaluation by internal and external bodies. Those evaluations range from overall assessments of NIOSH, such as the recent 2005 Performance Assessment Rating 3 Developed by NIOSH with the assistance of the RAND Corporation.
OCR for page 182
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health FIGURE 1 The NIOSH operational plan presented as a logic model
OCR for page 183
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health Tool (PART) review,4 to evaluation of research program elements such as any external scientific program reviews. The ECs should review all available prior reviews. Although it is important to consider all prior reviews in the present evaluation to aid in understanding the evolution of the programs and program elements, the ECs’ evaluations of NIOSH’s programs are independent of the prior reviews and evaluations. I.C. Evaluation Committees Individual ECs will be formed through a process consistent with the rules of the National Academies for the formation of balanced committees. The committees will be composed of persons with expertise appropriate to evaluating specific NIOSH research programs and may include representatives of stakeholder groups (such as labor unions and industry) and experts in technology transfer and program evaluation. The committees will conduct appropriate information-gathering sessions to obtain information from the sponsor (a NIOSH research program), stakeholders affected directly by the NIOSH research, and relevant independent parties. Each EC will consist of about 10 members, will meet about three times, and will prepare a report. The National Academies will deliver the report to NIOSH within 9 months after the individual EC is formed. EC reports will be subjected to the National Academies report-review process. I.D. Evaluation Committees’ Information Needs The ECs are expected to conduct information-gathering as appropriate on Background and resources of the program: History of program, including results of previous reviews. Program funding, by year, for the current year and the last 10 years. Program funding, by objective or subprogram. Extramural-grant awarding, cooperative agreement and contracting process, solicitation of research ideas, and advisory activities. Program goals and objectives. Internal NIOSH processes and research: Intramural surveillance, research, and transfer activities. Process to solicit and approve intramural research proposals. 4 PART focuses on assessing program-level performance and is one of the measures of success for the Budget and Performance Integration initiative of the president’s management agenda (see CDC Occupational Safety and Health at http://www.whitehouse.gov/omb/budget/fy2006/pma/hhs.pdf).
OCR for page 206
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health End outcomes, for purposes of this evaluation, are changes related to health, including decreases in injuries, illnesses, deaths, and decreases in exposures or risk factors resulting from the research in the specific program or subprogram. Quantitative data are preferable to qualitative, but qualitative analysis may be necessary. Sources of quantitative data include Bureau of Labor Statistics (BLS) data on fatal occupational injuries (Census of Fatal Occupational Injuries) and nonfatal injuries and illnesses (Annual Survey of Occupational Injury and Illnesses). NIOSH intramural surveillance systems, such as the National Electronic Injury Surveillance System (NEISS), the coal worker x-ray surveillance program, and agricultural worker surveys conducted by NIOSH in collaboration with the US Department of Agriculture. State-based surveillance systems, such as the NIOSH-funded ABLES, and the SENSOR programs (for asthma, pesticides, silicosis, noise-induced hearing loss, dermatitis, and burns). Selected state workers-compensation programs. OSHA, which collects exposure data, in the Integrated Management Information System. The FC is unaware of surveillance mechanisms for many occupationally related chronic illnesses such as cancers arising from long exposure to chemicals and other stressors. For many outcomes, incidence and prevalence are best evaluated by investigator-initiated research. The strengths and weaknesses of the various sources of outcome data should be recognized by the ECs. Quantitative accident, injury, illness, and employment data and databases are subject to error and bias and should be used by the ECs for drawing inferences only after critical evaluation and examination of whatever corroborating data are available. For example, it is widely recognized that occupational illnesses are poorly documented in the BLS Survey of Occupational Injuries and Illnesses, which captures only incident cases among active workers. Most illnesses that may have a relationship to work are not exclusively so related, and it is difficult for health practitioners to diagnose work-relatedness; few are adequately trained to make this assessment. Many of these illnesses have long latency and do not appear until years after people have left the employment in question. Surveillance programs may systematically undercount some categories of workers, such as contingent workers. Challenges posed by inadequate or inaccurate measurement systems should not drive programs out of difficult areas of study, and the ECs will need to be aware of such a possibility. In particular, contingent and informal working arrangements that place workers at greatest risk are also those on which
OCR for page 207
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health surveillance information is almost totally lacking, so novel methods for measuring impact may be required. In addition to measures of illness and injury, levels of exposure to chemical and physical agents and to safety and ergonomic hazards can be useful. Exposure or probability of exposure can serve as an appropriate proxy for disease or injury when a well-described occupational exposure-health association exists. In such instances, decreased exposure can be accepted as evidence that the end outcome of reduced illness has been achieved. That is particularly necessary in cases (such as exposure to asbestos) in which latency between exposure and disease outcome (lung cancer) makes effective evaluation of the relevant end outcome infeasible. As an example of how exposure levels can serve as a proxy, the number of sites that exceed an OSHA Permissible Exposure Limit (PEL) or an American Conference of Governmental Industrial Hygienists threshold limit value is a quantitative measure of improvement of occupational health awareness and reduction of risk. In addition to exposure level, the number of people exposed and the distribution of exposure levels are important. Those data are available from multiple databases and studies of exposure. Apart from air monitoring, such measures of exposure as biohazard controls, reduction in requirements for use of personal protective equipment, and reduction of ergonomic risks are important. Clearly, the commitment of industry, labor, and government to health and safety are critical external factors. Several measures of this commitment can be useful for the EC: monetary commitment of the groups, attitude, staffing, and surveys of relative level of importance. To the extent that the resources allocated to safety and health are limiting factors, the ECs should explicitly assess NIOSH performance in the context of constraints. Questions to Guide the Evaluation Committee What are the amounts and qualities of end-outcomes data (such as injuries, illness, exposure and productivity affected by health)? What is the temporal trend in those data? Is there objective evidence of improvements in occupational safety or health? To what degree has the NIOSH program or subprogram been responsible for improvements in occupational safety or health? If there is no time trend in the data, how do findings compare with data from other comparable US groups or the corresponding populations in other countries? Is there evidence that external factors have affected outcome measures?
OCR for page 208
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health Has the program been responsible for outcomes outside the United States that have not been described in another category? Assessment For this part of the assessment, the EC should provide a qualitative assessment discussing the evidence of reductions in injuries and illnesses or their appropriate proxies (impacts). III.B.9. Review of Other Outcomes There may be health and safety impacts not yet appreciated, and other beneficial social, economic, and environmental outputs, including potential NIOSH impacts outside the United States. Many NIOSH study results and training programs may be judged to be important, or there may be evidence of implementation of NIOSH recommendations, outside the United States. Questions to Guide the Evaluation Committee Is the program likely to produce a favorable change that has not yet occurred or not been appreciated? Has the program been responsible for other social, economic, security, or environmental outcomes? Has the program’s work had an impact on occupational health and safety in other countries? Assessment Evaluation by the EC may consist of a discussion of other outcomes, including positive changes that have not yet occurred; other social, economic, security, or environmental outcomes; and the impact that NIOSH has had on international occupational safety and health. It might also consider the incorporation of international research results into the NIOSH program of knowledge transfer for industry sectors. III.B.10. Summary Evaluation Ratings and Rationale An EC should use its expert judgment to rate the relevance and impact of the research program and its important subprograms by first summarizing its assessments of the subprograms and overall program according to the several items listed in Table 4. Table 4 is only a worksheet intended as an aid to the EC in its evalua-
OCR for page 209
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health TABLE 4 Evaluation Committee Worksheet to Assess Research Programs and Subprograms Please respond to each with “major or important,” “moderate,” “likely,” “limited,” or “none.” Background Context for Program Impact 1.1 Evidence of reduction of risk factors in the workplace (intermediate outcome) and evidence that external factors affected reduction 1.2 Evidence of reduction in workplace exposure, illness, or injuries (end outcome) and evidence that external factors affected reduction Addressing Charge 1 Activity Category Program Subprogram 1 … … n 1.3 Contributions of NIOSH research and transfer activities to changes in work-related practices Research Transfer 1.4 Contributions of NIOSH research and transfer activities to reductions in workplace exposure, illness, or injuries Research Transfer 1.5 Evidence of external factors preventing application of NIOSH research results Research Transfer 1.6 Contribution of NIOSH research to enhancement of capacity in government or other research institutions Research Transfer 1.7 Contributions of NIOSH research to productivity, security, or environmental quality (beneficial side effects) Research Transfer Addressing Charge 2 2.1 Relevance of current and recently completed research and transfer activities to future improvements in workplace safety and health Research Transfer 2.2 Progress in targeting research to areas of study most relevant to future improvements in occupational safety and health Research Transfer
OCR for page 210
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health tion. Its purpose is to encourage the EC to summarize its work in one place and to concentrate on the subprograms and the items that will contribute to the final impact and relevance scores. To set the context for this step in the evaluation of the impact of the research program in preparation to respond to charge 1, the EC will first need to consider the available evidence of changes in work-related risks and adverse effects and external factors related to the changes. That information should be organized as a prose response to items 1.1 and 1.2 in Table 4. Next, the EC should review the responses to the questions in Sections III.B.6 through III.B.8 and systematically rate the impact of the research program and its subprograms by responding to items 1.3-1.7 in Table 4. To complete the table, the EC response should use one of the following five terms: “major or important,” “moderate,” “likely,” “limited,” or “none” (since 1995). The EC should evaluate separately the impact of the research and the impact of transfer activities. High ratings on items 1.3-1.7 require the committee’s judgment that the program has contributed to outcomes. For example, outcomes have occurred earlier than they would have or are better than they would have been in the absence of the research program, or outcomes would have occurred in the absence of external factors beyond NIOSH’s control or ability to plan around. The EC should then assess the relevance of the research program and subprograms in preparation for addressing charge 2. The EC should review the responses to the questions in Sections III.B.2 through III.B.5 and rate the relevance of the research program and its subprograms by responding to items 2.1 and 2.2 in Table 4. The same five terms should be used (“major or important,” “moderate,” “likely,” “limited,” or “none”) to evaluate separately the relevance of the research and the relevance of the transfer activities. Transfer activities occur in two contexts: (1) NIOSH efforts to translate intellectual products into practice and (2) efforts by stakeholders to take advantage of NIOSH products. Final Program Ratings To provide the final assessment of the research program for charge 1 (impact) and charge 2 (relevance), the ECs will use their expert judgment, their responses to the questions in Table 4, and any other appropriate information to arrive at one overall rating for the impact of the research program and one for its relevance to the improvement of occupational safety and health. In light of substantial differences among the types of research programs that will be reviewed and the challenge to arrive at a summative evaluation of both impact and relevance, however, the FC chose not to attempt to construct a single algorithm to produce the two final ratings.
OCR for page 211
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health Having completed Table 4, the EC should undertake its final assessment of the impact and relevance of the program. Final program ratings will consist of the numerical scores and prose descriptions of why the scores were given. As explained below, the ECs will summarize their responses to charges 1 and 2 by rating the relevance and impact of the NIOSH research program on five-point scales in which 1 is the lowest and 5 the highest rating. The FC has made an effort to establish mutually exclusive rating categories in the five-point rating scale; when the basis of a rating fits more than one category, the highest applicable score should be assigned. ECs will need to consider the impact and relevance of both NIOSH completed research and research in progress. In general, the assessment of impact will consider research completed, and the assessment of relevance will include research in progress related to likely future improvements. When assessing the relevance of the program, the EC should keep in mind how well the program has considered the frequency and severity of the problems being addressed, whether appropriate attention has been directed to both genders, vulnerable populations or hard-to-reach workplaces, and whether the different needs of large and small businesses have been accounted for. The FC has some concern that the impact scoring system proposed below might be considered a promotion of the conventional occupational-health research paradigm that focuses on health-effect and technology research and not give much emphasis to socioeconomic and policy research and to surveillance and diffusion research (as opposed to activities) needed to effect change. Clearly, not all intermediate outcomes occur in the workplace. There are important outcomes much farther out on the causal chain that NIOSH can affect, and not all these can be defined as well-accepted intermediate outcomes. NIOSH, for example, has an important role to play in generating knowledge that may contribute to changing norms in the insurance industry, in health-care practice, in public-health practice, and in the community at large. The ECs may find that some of these issues need to be addressed and considered as important to influence the external factors that limit application of more traditional research findings. Given the rapidly changing nature of work and the workforce and some of the intractable problems in manufacturing, mining, and some other fields, the ECs are encouraged to think beyond the traditional paradigm. Rating of Impact 5 = Research program has made a major contribution to worker health and safety on the basis of end outcomes or well-accepted intermediate outcomes.
OCR for page 212
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health 4 = Research program has made a moderate contribution on the basis of end outcomes or well-accepted intermediate outcomes; research program generated important new knowledge and is engaged in transfer activities, but well-accepted intermediate outcomes or end outcomes have not been documented. 3 = Research program activities or outputs are going on and are likely to produce improvements in worker health and safety (with explanation of why not rated higher). 2 = Research program activities or outputs are going on and may result in new knowledge or technology, but only limited application is expected. 1 = Research activities and outputs are NOT likely to have any application. NA = Impact cannot be assessed; program not mature enough. Rating of Relevance 5 = Research is in highest-priority subject areas and highly relevant to improvements in workplace protection; research results in, and NIOSH is engaged in, transfer activities at a significant level (highest rating). 4 = Research is in high-priority subject area and adequately connected to improvements in workplace protection; research results in, and NIOSH is engaged in, transfer activities. 3 = Research focuses on lesser priorities and is loosely or only indirectly connected to workplace protection; NIOSH is not significantly involved in transfer activities. 2 = Research program is not well integrated or well focused on priorities and is not clearly connected to workplace protection and inadequately connected to transfer activities. 1 = Research in the research program is an ad hoc collection of projects, is not integrated into a program, and is not likely to improve workplace safety or health. III.C. Identifying Significant Emerging Research (Addressing Charge 3) Among the most challenging aspects of conducting research for the purpose of prevention of injury and illness is identifying new or emerging needs or trends and formulating an active research response that appropriately uses scarce resources in anticipation of those needs. Each EC should review the procedures that NIOSH has in place to identify needed research relevant to the NIOSH mission. Each EC should review the success that NIOSH has had in identifying and addressing research to emerging issues. The review should include examination
OCR for page 213
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health of leading indicators from appropriate federal agency sources, such as the Environmental Protection Agency, the Department of Labor, the National Institute of Standards and Technology, the National Institutes of Health, the Department of Defense, and the Department of Commerce. Those indicators should track new technologies, products, and processes and disease or injury trends. One source of inputs deserving particular attention is the NIOSH HHE reports. NIOSH’s HHE program is a separate legislatively mandated program that offers a potential mechanism to identify emerging research needs that could be incorporated as an input in each of the programs evaluated. The ECs should consider whether appropriate consideration has been given to findings from the HHE investigations as they are related to the research program under review. Some additional indicators might include NIOSH and the NIOSH-funded FACE, the AOEC reports, the US Chemical Safety Board investigations, SENSOR and other state-based surveillance programs, and others. In addition, appropriate federal advisory committees and other stakeholder groups should be consulted to provide qualitative information. The EC members should use their expert judgment both to evaluate what NIOSH has identified as emerging research targets (charge 2) and to respond to charge 3 by providing recommendations to NIOSH for additional research that NIOSH has not yet identified. An EC’s response to charge 3 will consist primarily of recommendations for research in subjects that the EC considers important and of the committee’s rationale. Questions to Guide the Evaluation Committee What information does NIOSH review to identify emerging research needs? What is the process for review? How often does the process take place? How are NIOSH staff scientists and NIOSH leadership engaged? What is the process for moving from ideas to formal planning and resource allocation? How are stakeholders involved? What advisory or stakeholder groups are asked to identify emerging research targets? How often are such groups consulted, and how are suggestions followed up? What new research targets have been identified for future development in the program under evaluation? How were they identified?
OCR for page 214
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health Were there lessons learned that could help to identify other emerging issues? Does the EC agree with the issues identified and selected as significant and with the NIOSH response, or were important issues overlooked? Is there evidence of unwise expenditure of resources on unimportant issues? IV. EVALUATION COMMITTEE REPORT TEMPLATE The following outline flows from the FC’s review of the generalized logic model prepared by NIOSH, the request for information from NIOSH programs, and the assessment model described earlier in this report. Introduction: This section should be a brief descriptive summary of the history of the program (and subprograms) being evaluated, with respect to pre-NORA, NORA 1, and current and future plans of the research program presented by NIOSH. It presents the context for the research on safety and health; goals, objectives, and resources; groupings of subprograms; and any other significant or pertinent information. (A list of the NIOSH materials reviewed should be provided in an appendix to the EC report.) Evaluation of programs and subprograms (charges 1 and 2): Evaluation summary (includes a brief summary of the evaluation with respect to impact and relevance, scores for impact and relevance, and summary statements addressing charges 1 and 2). Strategic goals and objectives: Describes assessment of the subprograms and overall program for relevance. Review of inputs: Describes adequacy of inputs to achieve goals. Review of activities: Describes assessment of the relevance and quality of the activities. Review of research program outputs: Describes assessment of relevance, quality, and potential usefulness of the research program. Review of intermediate outcomes and causal impact: Describes assessment of the intermediate outcomes and the causal attribu-
OCR for page 215
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health tion to NIOSH; includes the likely impacts and recent outcomes in the assessment. Review of end outcomes: Describes the end outcomes related to health and safety and provides an assessment of the type and degree of causal attribution to NIOSH. Review of other outcomes: Discusses other health and safety impacts that have not yet occurred; other beneficial social, economic, and environmental outcomes; and international dimensions and outcomes. Summary of ratings and rationale (see Table 4). Identification of needed research (charge 2): The EC should assess the progress that the NIOSH program has made in targeting new research in the fields of occupational safety and health. There should be a discussion of the assessment process and results. Emerging research areas (charge 3): The EC should assess whether the NIOSH program has identified significant emerging research areas that appear especially important in terms of their relevance to the mission of NIOSH. The EC should respond to NIOSH’s perspective and add its own recommendations. Recommendations for program improvement: On the basis of the review and evaluation of the program, the EC may provide recommendations for improving the relevance of the NIOSH research program to health and safety conditions in the workplace and the impact of the research program on health and safety in the workplace as related to the research program under review. Appendix A: List of the NIOSH and related materials collected in the process of the evaluation V. FRAMEWORK COMMITTEE FINAL REPORT At the conclusion of all individual program reviews, the FC will prepare a final report summarizing the findings of all the evaluating committees and providing NIOSH with an overall evaluation. All program ratings will be summarized and might be plotted graphically or with a Web chart. The following is a proposed outline of the FC’s final report:
OCR for page 216
Respiratory Diseases Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health Summary of national needs identified by the research programs reviewed. On the basis of the best available evidence, place those needs in the context of the overall estimated potential work-related disease and injury burden. Discuss the choices made and alternatives that might be the focus of current or future attention. Comment on programs not selected by NIOSH for evaluation by the National Academies. Assessment of how well the program goals. Were matched to the research program needs. Were adjusted to new information and inputs as the field of interest changed or program results became available. Assessment of NIOSH overall performance in the research programs reviewed. Distribution of available inputs. Activities and outputs. Intermediate outcomes. Summary assessment of significant differences among the programs. International impact. Leveraging of the NIOSH research activity with respect to other public and private research programs. Assessment of relative importance of external factors in permitting or preventing intermediate or end outcomes; attention paid to accounting for and planning within the constraints of external factors (not simply assigning lack of progress to external factors). Overall assessment of NIOSH impact on progress in reducing occupational injury and illness. Breakthrough knowledge. International impact. Addressing disparities. Targeting residual risks and intractable risks. Coordinating NIOSH research activity with respect to other public and private research programs. Impact on occupational safety and health. Summary, Conclusions, and Recommendations.