National Academies Press: OpenBook

Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health (2006)

Chapter: Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health

« Previous: 4 Recommendations for Program Improvement
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

A
Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health*

This is a document prepared by the National Academies’ Committee for the Review of NIOSH Research Programs,1 also referred to as the Framework Committee. This document is not a formal report of the National Academies—rather, it is a framework proposed for use by a number of National Academies committees that will be reviewing research in various research programs and health-outcomes programs. This version will be posted on the website of the National Academies and NIOSH for review. It is a working document that will be subject to change by the Framework Committee aimed at improving its

*

Version of 12/19/05.

1

Members of the National Academies’ Committee for the Review of NIOSH Research Programs include: David Wegman (Chair; University of Massachusetts Lowell School of Health and Environment), William Bunn, III (International Truck and Engine Corporation), Carlos Camargo (Harvard Medical School), Letitia Davis (Massachusetts Department of Public Health), James Dearing (Ohio University), Fred Mettler, Jr. (University of New Mexico School of Medicine), Franklin Mirer (United Automobile, Aerospace, and Agricultural Implement Workers of America), Jacqueline Nowell (United Food and Commercial Workers International Union), Raja Ramani (Pennsylvania State University), Jorma Rantanen (Finnish Institute of Occupational Health), Rosemary Sokas (University of Illinois at Chicago School of Public Health), Richard Tucker (Tucker and Tucker Consultants, Inc., and University of Texas at Austin), Joseph Wholey (University of Southern California School of Policy, Planning, and Development), and James Zuiches (Washington State University).

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

relevance on the basis of responses received from evaluation committee members, NIOSH, stakeholders, and the general public before and during the course of the assessments conducted by independent evaluation committees of up to 15 research programs and health-outcomes programs.

All public comments submitted to the Committee for the Review of NIOSH Research Programs will be included in the Public Access File for this study as provided in the National Academies Terms of Use (www.nationalacademies.org/legal/terms.html). Please keep in mind that if you directly disclose personal information in your written comments, this information may be collected and used by others.

For inquiries related to this document, or for the most current document version, please contact Evan Douple (edouple@nas.edu) or Sammantha Magsino (smagsino@nas.edu) of the National Academies.

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

CONTENTS

   

 Acronyms

 

 

   

 I.  Overview of Charge

 

 

   

  I.A.  NIOSH Strategic Goals and Operational Plan

 

 

   

  I.B.  Information from Other Evaluations

 

 

   

  I.C.  Evaluation Committees

 

 

   

  I.D.  Evaluation Committees’ Information Needs

 

 

   

 II.  Summary of Evaluation Process

 

 

   

  II.A.  The Evaluation Flow Chart (Figure 2)

 

 

   

  II.B.  Steps in Program Evaluation

 

 

   

  II.C.  Assessing Relevance

 

 

   

  II.D.  Assessing Impact

 

 

   

 III.  Evaluation of NIOSH Research Programs—the Process

 

 

   

  III.A.  Analysis of External Factors Relevant to the NIOSH Program

 

 

   

  III.A.1.  Overview

 

 

   

  III.A.2.  Considerations for Discussion

 

 

   

  III.B.  Evaluating NIOSH Research Programs (Addressing Charges 1 and 2)

 

 

   

  III.B.1.  Identifying Period of Time to be Evaluated

 

 

   

  III.B.2.  Identification of Major Challenges (Circle in Figure 2)

 

 

   

  III.B.3.  Analysis of NIOSH Program Strategic Goals and Objectives (Box A in Figure 2)

 

 

   

  III.B.4.  Review of Inputs (Box B in Figure 2)

 

 

   

  III.B.5.  Review of Activities (Box C in Figure 2)

 

 

   

  III.B.6.  Review of Outputs (Box D in Figure 2)

 

 

   

  III.B.7.  Review of Intermediate Outcomes (Box E in Figure 2)

 

 

   

  III.B.8.  Review of End Outcomes (Box F in Figure 2)

 

 

   

  III.B.9.  Review of Other Outcomes

 

 

   

  III.B.10.  Summary Evaluation Ratings and Rationale

 

 

   

  III.C.  Identifying Significant Emerging Research (Addressing Charge 3)

 

 

   

 IV.  Evaluation Committee Report Template

 

 

   

 V.  Framework Committee Final Report

 

 

Figure 1

 

The NIOSH operational plan presented as a logic model

 

Figure 2

 

Flow chart for the evaluation of the NIOSH research program

 

Table 1

 

NORA High-priority Research Areas by Category

 

Table 2

 

Examples of NIOSH Program Research and Transfer Activities

 

Table 3

 

Examples of a Variety of Scientific Information Outputs

 

Table 4

 

Evaluation Committee Worksheet to Assess Research Programs and Subprograms

 

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
   

ACRONYMS

ABLES Adult Blood Lead Epidemiology and Surveillance

ACOEM American College of Occupational and Environmental Medicine

AOEC Association of Occupational and Environmental Clinics

BLS Bureau of Labor Statistics

CDC Centers for Disease Control and Prevention

EC Evaluation Committee

FACE Fatality Assessment Control and Evaluation

FC Framework Committee

HHE Health Hazard Evaluations

MSHA Mine Safety and Health Administration

NEISS National Electronic Injury Surveillance System

NIOSH National Institute for Occupational Safety and Health

NORA National Occupational Research Agenda

NORA1 National Occupational Research Agenda 1996-2005

NORA2 National Occupational Research Agenda 2005-Forward

OSHA Occupational Safety and Health Administration

OSHAct Occupational Safety and Health Act of 1970

OSH Review Occupational Safety and Health Review Commission

Commission

PART Performance Assessment Rating Tool

PEL Permissible Exposure Limits

SENSOR Sentinel Event Notification System of Occupational Risks

TMT Tools, Methods, or Technologies

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

In September 2004, the National Institute for Occupational Safety and Health (NIOSH) contracted with the National Academies to conduct a review of NIOSH research programs. The goal of this multiphase effort is to assist NIOSH in increasing the impact of its research efforts in reducing workplace illnesses and injuries and improving occupational safety and health. The National Academies agreed to conduct this review and assigned the task to the Division on Earth and Life Studies and the Institute of Medicine.

The National Academies appointed a committee of 14 members, including persons with expertise in occupational medicine and health, industrial health and safety, industrial hygiene, epidemiology, civil and mining engineering, sociology, program evaluation, communication, and toxicology; representatives of industry and of the workforce; and a scientist experienced in international occupational-health issues. The Committee on the Review of NIOSH Research Programs, referred to as the Framework Committee (FC), held meetings during 2005 on May 5-6 and July 7-8 in Washington, DC, and on August 15-16 in Woods Hole and Falmouth, MA.

This document is not a report of the National Academies; rather, it presents the evaluation framework developed by the FC to serve as a guideline and structure for NIOSH program reviews by Evaluation Committees (ECs) to be appointed by various divisions and boards of the National Academies. The ECs will use this framework in reviewing as many as 15 NIOSH research programs during a 5-year period. This is a working document. It is shared with NIOSH and the public. The framework and criteria may be modified by the FC on the basis of responses it receives from the ECs and other sources. It is incumbent upon the ECs to consult with the FC if portions of the evaluation framework presented here are inappropriate for the specific program under review.

I.
OVERVIEW OF CHARGE

At the first meeting of the FC, Lewis Wade, NIOSH senior science advisor, emphasized that the reviews should focus on evaluating NIOSH’s research programs impact and relevance to health and safety in the workplace. In developing a framework, the FC was asked to address the following:

  1. Evaluation committee assessment of progress in reducing workplace illnesses and injuries facilitated by occupational safety and health research through (a) an analysis of relevant data about workplace illnesses and injuries for the program activity, and (b) an evaluation of the effect that NIOSH research has had in reducing illnesses and injuries. The evaluation committees will rate the performance of each program for impact of the

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

program in the workplace. Impact may be assessed directly or, as necessary, using intermediate outcomes to estimate impact. Qualitative narrative evaluations may also be appropriate under certain circumstances.

  1. Evaluation committee assessment of progress in targeting new research to the areas of occupational safety and health most relevant to future improvements in workplace protection.

  2. Evaluation committee identification of significant emerging research areas which appear especially important in terms of their relevance to the mission of NIOSH.

Those three charges constitute the scope of work of the individually appointed, independent ECs formed by the National Academies.

I.A.
NIOSH Strategic Goals and Operational Plan

As a prelude to understanding the NIOSH strategic goals and operational plan, NIOSH research efforts should be understood in the context of the Occupational Safety and Health Act (OSHAct) under which it was created. The OSHAct identifies workplace safety and health to be a national priority and gives employers the responsibility for controlling hazards and preventing workplace injury and illness. The act creates an organizational framework for doing this, with complementary roles and responsibilities assigned to employers and employees, OSHA, the States, the OSH Review Commission, and NIOSH. As one component of a national strategy the act recognizes NIOSH’s roles and responsibilities to be supportive and indirect—NIOSH’s research, training programs, criteria and recommendations are all intended to be used to inform and assist those actually responsible for hazard control (OSHAct Section 2b and Sections 20 and 22).

Section 2b of the OSHAct describes thirteen interdependent means of accomplishing the national goal, one of which is “by providing for research … and by developing innovative methods … for dealing with occupational safety and health problems.” Sections 20 and 22 give the responsibility for this research to NIOSH. In addition, NIOSH is given related responsibilities including: the development of criteria to guide prevention of work-related injury or illness, development of regulations reporting on the employee exposures to harmful agents, the establishment of medical examinations programs or tests to determine illness incidence and susceptibility, publication of a list of all known toxic substances, the assessment of potentially toxic effects or risk associated with workplace exposures in specific settings, the conduct of education programs for relevant professionals to carry out the OSHAct purposes, and assisting the Secretary of Labor regarding education programs for employees and employers in hazard recognition and control.

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

The NIOSH mission is “to provide national and world leadership to prevent work-related illness, injury, disability, and death by gathering information, conducting scientific research, and translating the knowledge gained into products and services”. To fulfill its mission, NIOSH has established the following strategic goals:2

  • Goal 1: Conduct research to reduce work-related illnesses and injuries.

    • Track work-related hazards, exposures, illnesses, and injuries for prevention.

    • Generate new knowledge through intramural and extramural research programs.

    • Develop innovative solutions for difficult-to-solve problems in high-risk industrial sectors.

  • Goal 2: Promote safe and healthy workplaces through interventions, recommendations, and capacity-building.

    • Enhance the relevance and utility of recommendations and guidance.

    • Transfer research findings, technologies, and information into practice.

    • Build capacity to address traditional and emerging hazards.

  • Goal 3: Enhance global workplace safety and health through international collaborations.

    • Take a leadership role in developing a global network of occupational health centers.

    • Investigate alternative approaches to workplace illness and injury reduction and provide technical assistance to put solutions in place.

    • Build global professional capacity to address workplace hazards through training, information sharing, and research experience.

In 1994, NIOSH embarked on a national partnership effort to identify research priorities to guide occupational health and safety research for the next decade. The National Occupational Research Agenda (NORA) identified 21 high-priority research areas (see Table 1). NORA was intended not only for NIOSH but for the entire occupational health community. Approaching the 10-year anniversary of NORA, NIOSH is working with its partners to update the research agenda. In the second decade of NORA, an approach based on industry sectors will be pursued. NIOSH and its partners will form sector research councils that will work

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

TABLE 1 NORA High-Priority Research Areas by Category

Category

Priority Research Area

Disease and injury

Allergic and irritant dermatitis

Asthma and chronic obstructive pulmonary disease

Fertility and pregnancy abnormalities

Hearing loss

Infectious diseases

Low-back disorders

Musculoskeletal disorders of upper extremities

Trauma

Work environment and workforce

Emerging technologies

Indoor environment

Mixed exposures

Organization of work

Special populations at risk

Research tools and approaches

Cancer research methods

Control technology and personal protective equipment

Exposure-assessment methods

Health-services research

Intervention-effectiveness research

Risk-assessment methods

Social and economic consequences of workplace illness and injury

Surveillance research methods

to establish sector-specific research goals and objectives. Emphasis will be placed on moving research to practice in workplaces through sector-based partnerships.

Figure 1 is the NIOSH operational plan presented as a logic model3 of the path from inputs to outcomes for each NIOSH research program. The FC adapted the model to develop its framework. NIOSH will provide similar logic models relevant to each research program evaluated by an EC.

I.B.
Information from Other Evaluations

The FC is aware that several NIOSH programs have already been subjected to evaluation by internal and external bodies. Those evaluations range from overall assessments of NIOSH, such as the recent 2005 Performance Assessment Rating

3

Developed by NIOSH with the assistance of the RAND Corporation.

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

FIGURE 1 The NIOSH operational plan presented as a logic model.

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

Tool (PART) review,4 to evaluation of research program elements such as any external scientific program reviews. The ECs should review all available prior reviews. Although it is important to consider all prior reviews in the present evaluation to aid in understanding the evolution of the programs and program elements, the ECs’ evaluations of NIOSH’s programs are independent of the prior reviews and evaluations.

I.C.
Evaluation Committees

Individual ECs will be formed through a process consistent with the rules of the National Academies for the formation of balanced committees. The committees will be composed of persons with expertise appropriate to evaluating specific NIOSH research programs and may include representatives of stakeholder groups (such as labor unions and industry) and experts in technology transfer and program evaluation. The committees will conduct appropriate information-gathering sessions to obtain information from the sponsor (a NIOSH research program), stakeholders affected directly by the NIOSH research, and relevant independent parties. Each EC will consist of about 10 members, will meet about three times, and will prepare a report. The National Academies will deliver the report to NIOSH within 9 months after the individual EC is formed. EC reports will be subjected to the National Academies report-review process.

I.D.
Evaluation Committees’ Information Needs

The ECs are expected to conduct information-gathering as appropriate on

  • Background and resources of the program:

    • History of program, including results of previous reviews.

    • Program funding, by year, for the current year and the last 10 years.

    • Program funding, by objective or subprogram.

    • Extramural-grant awarding, cooperative agreement and contracting process, solicitation of research ideas, and advisory activities.

  • Program goals and objectives.

  • Internal NIOSH processes and research:

    • Intramural surveillance, research, and transfer activities.

    • Process to solicit and approve intramural research proposals.

4

PART focuses on assessing program-level performance and is one of the measures of success for the Budget and Performance Integration initiative of the president’s management agenda (see CDC Occupational Safety and Health at http://www.whitehouse.gov/omb/budget/fy2006/pma/hhs.pdf).

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
  • NIOSH-funded extramural research:

    • Requests for proposals, cooperative agreements and research contracts distributed.

    • Awardee products, including close-out reports, surveillance, research, and transfer activities, peer-reviewed publications, and patents.

  • Products and technology transfer:

    • Data related to program publications, conferences, recommendations, patents, and so on.

    • Past and planned mechanisms for transferring outputs to outcomes.

    • Interventions, recommendations, and information-dissemination and technology-transfer activities designed to get research findings used to improve occupational safety and health.

    • Outcomes of research, alerts, standard-setting, investigations, and consultations; for example—documented reductions in risk after program-supported interventions, employer and industry behavior changes made in response to research outputs, and worker behavior changes in response to research outputs.

  • Impact on worker safety and health—data necessary to evaluate program impact on health outcomes (work-related injuries and illnesses) and exposures.

  • The most severe or most frequent adverse health and safety outcomes or exposures in the research program and the most accessible improvements with respect to health and safety.

  • Interactions within NIOSH and with other stakeholders:

    • The role of program research staff in NIOSH policy-setting, Occupational Safety and Health Administration (OSHA) and Mine Safety and Health Administration (MSHA) standard-setting, and voluntary standard-setting and other government policy functions.

    • Other institutions and research programs with overlapping or similar portfolios and an explanation of the relationship between the NIOSH work and staff and those of other institutions.

    • Stakeholder perspectives (OSHA, MSHA, union and workforce, industry, and so on).

    • Key partnerships with employers, labor, other government organizations, academic institutions, nonprofit organizations, and international organizations.

    • International involvement and perspective.

  • Systems to identify emerging problems and emerging research, including plans.

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

II.
SUMMARY OF EVALUATION PROCESS

The ECs are charged with assessing the relevance, quality, and impact of NIOSH research programs. In conducting their evaluations, the ECs should ascertain whether NIOSH is doing the right things (relevance) and doing them right (quality) and whether these things are improving health and safety in the workplace (impact).

II.A.
The Evaluation Flow Chart (Figure 2)

To address its charges, the FC has developed a flow chart (Figure 2) that breaks the NIOSH logic model into discrete, sequential program components to be characterized or assessed by the ECs. The components to be assessed are as follows:

  • Major program-area challenges.

  • Strategic goals and objectives.

  • Inputs (such as budget, staff, facilities, the institute’s research management, the NIOSH Board of Scientific Counselors, the NORA process, and NORA work groups).

  • Activities (efforts by NIOSH staff, contractors, and grantees, such as hazard and health-outcome surveillance, exposure-measurement research, health-effects research, intervention research, health services, other research, and technology-transfer activities).

  • Outputs (the products of NIOSH activities, such as publications, reports, conferences, databases, tools, methods, guidelines, recommendations, education and training, and patents).

  • Intermediate outcomes (responses by NIOSH stakeholders to NIOSH products, such as public or private policy change, training and education in the form of workshop or seminar attendance, self-reported use or repackaging of NIOSH data by intermediary stakeholders, adoption of technologies developed by NIOSH, implemented guidelines, licenses, and reduction of workplace hazardous exposures and other risk factors).

  • End outcomes (such as reduction of work-related injuries or illnesses, or hazardous exposures in the workplace).

Drawing on the program logic model, the flow chart, and EC members’ expertise, the ECs will delineate important determinants of a NIOSH research program’s agenda and the consequences of the NIOSH research activity. Determinants are conceptualized as inputs and external factors. Examples of external factors are the research activities of industry and other federal agencies and the political and

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

FIGURE 2 Flow chart for the evaluation of the NIOSH research program.

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

regulatory environment, which can affect all components of the research program (Figure 2). For purposes of this review, the results of inputs and external factors are the program research activities, outputs, and associated transfer activities that may result in intermediate outcomes and possibly eventual end outcomes.

The FC has used the NIOSH logic model to develop the flow chart to define the scope and steps of an EC evaluation. The FC’s vision of how a program evaluation should occur is incorporated in a summary manner in the flow chart and discussed extensively in later sections. For example, the FC identified two types of outcomes: (a) intermediate outcomes, which represent implementations (what external stakeholders, such as employers, do in reaction to the products of NIOSH work, including new regulations, widely accepted guidelines, introduction of control technologies in the workplace, changes in employer or worker behaviors, and changes in diagnostic practices of health-care providers), and (b) end outcomes, which are improvements (reductions in work-related injuries, illnesses, and hazardous exposures). For the purpose of evaluation, the FC does not differentiate between NIOSH’s “intermediate customer” and “final customer” activities (Figure 1); instead it combines them into a single category (Box E, Review and Assessment of Intermediate Outcomes, Figure 2). Training and development programs were appropriately defined as outputs by NIOSH in the logic model, but the FC finds more value in focusing on response to such offerings as intermediate outcomes (Box E) in the flow chart. The number of workers exposed to training activities represents a type of implementation of NIOSH outputs in the workplace. In evaluating each program or major subprogram, the EC must collect, analyze, and evaluate information on items described in each of the boxes of Figure 2. Further details on the evaluation are described in Section III of this document.

II.B.
Steps in Program Evaluation

The FC has concluded that useful evaluation requires: (a) a disciplined focus on a small number of questions or hypotheses typically related to program goals, performance criteria, and performance standards; (b) a rigorous method for answering the questions or testing the hypotheses; and (c) a credible procedure for developing qualitative and quantitative assessments. The evaluation process developed by the FC is summarized here and described in detail in Section III of this document.

  1. Gather appropriate information from NIOSH and other sources.

  2. Determine timeframe that the evaluation will cover (see III.B.1).

  3. Identify program-area major challenges and objectives (see III.B.2). All NIOSH research programs, whether based on health outcomes or sectors,

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

are designed to be responsive to the safety and health problems in today’s or tomorrow’s workplace. In the NIOSH vision, mission, values, and goals, each research program should have its own objectives. The ECs will provide an independent assessment of the major program challenges and determine whether they are consistent with the research program’s stated goals and objectives.

  1. Identify subprograms and major projects in the research program. It is important for each EC to determine how necessary it is to disaggregate a program to achieve a manageable and meaningful evaluation of its components and the total program. Each research program may need to be broken down into several recognizable subprograms or major projects if an effective evaluation is to be organized. It may be advantageous for an EC to disaggregate a program into subprograms that NIOSH identifies.

  2. Evaluate the program and subprogram components sequentially as discussed in Section III, using the flow chart (Figure 2) as a guide (Sections III.B.3 through III.B.8). This will involve qualitatively assessing each phase of a research program by using the questions and guidance provided by the FC and professional judgment.

  3. Evaluate the research program’s potential outcomes not yet appreciated (Section III.B.9).

  4. Evaluate and score the program outcomes and important subprogram outcomes specifically for contributions to improvements in workplace safety and health. A worksheet is provided with specific items for consideration (Section III.B.10).

  5. Evaluate and score the overall program for impact (Section III.B.10). Final program ratings will consist of a numerical score and discussion of its rationale.

  6. Evaluate and score the overall program for relevance (Section III.B.10). Final program ratings will consist of a numerical score and discussion of its rationale.

  7. Identify significant emerging research areas (Section III.C). On the basis of the expert judgment of EC members and information gathered from stakeholders (such as labor, industry, academe, and government agencies) and from appropriate NIOSH sentinel-event field-investigation activities, the EC will respond to Charge 3 by identifying and describing emerging research that appears especially important in its relevance to the mission of NIOSH. The EC will assess the extent to which NIOSH’s program is responsive to today’s and tomorrow’s needs and determine whether there are any gaps in response.

  8. Prepare report by using the template provided in Section IV as a guide.

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
II.C.
Assessing Relevance

FC members identified numerous possible factors to consider in assessing the relevance of NIOSH research programs, such as:

  • The severity, frequency, or both of the health and safety outcomes addressed and the number of people at risk (magnitude) for these outcomes.

  • The extent to which NIOSH research programs have identified and addressed gender issues and the concerns related to vulnerable populations. Vulnerable populations are defined as groups of workers who have (1) biological, social, or economic characteristics that place them at increased risk of developing work-related conditions and/or (2) inadequate data collected about them. Vulnerable populations include disadvantaged minorities, disabled individuals, low-wage workers, and non-English speakers for whom language or other barriers present health or safety risks.

  • The extent to which NIOSH research programs have addressed the health and safety needs of small businesses.

  • The “life stage” of the problems being addressed. As the health effects are understood, emphasis should shift to intervention research, and from efficacy to effectiveness to research on the process of dissemination of tested interventions. Gaps in the spectrum of prevention need to be addressed; for example, research on exposure assessment may be necessary before the next intervention steps can be taken.

  • The structure, in addition to the content, of the research program. A relevant research program is more than a set of unrelated research projects; it is an integrated program involving an interrelated set of surveillance, research, and transfer activities.

  • Appropriate consideration by NIOSH of stakeholder inputs.

II.D.
Assessing Impact

Causal attribution is a major aspect of program evaluation. It is necessary for the ECs to assess, to the extent possible, NIOSH’s contribution to end outcomes. Data on reductions in work-related injuries, illnesses, and hazardous exposures will be available for some programs. In some cases, they may be quantifiable. It is possible, however, to evaluate the impact of a NIOSH research program whether the outcomes are intermediate outcomes or end outcomes. Intermediate outcomes may be used as proxies for end outcomes in assessing impact if there is no direct evidence of improvements in health and safety as long as the ECs qualify their findings. The ECs will describe the realized or potential benefits of NIOSH’s pro-

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

grams. Examples of realized intermediate outcomes include: new regulations, widely accepted guidelines, work practices, and procedures, all of which may contribute measurably to enhancing health and safety at the work place.

The contribution of a NIOSH program to technology now in use or being implemented is another important part of impact assessment. NIOSH’s contribution can be assessed as major or important, moderate, likely, limited, or none. If technology development is in progress or has been abandoned, for whatever reason, the benefits are only potential or consist of knowledge gain.

III.
EVALUATION OF NIOSH RESEARCH PROGRAMS—THE PROCESS

III.A.
Analysis of External Factors Relevant to the NIOSH Research Program

As depicted in the logic model (Figure 1), the end outcome of reduced injuries, illnesses, or exposures is effected through stakeholder activities and outputs. All those involve the use of NIOSH outputs by stakeholders in industry, labor, other government agencies, and so on. It is evident that actions beyond NIOSH’s control—by industry, labor, and other entities—have important bearings on the incorporation in the workplace of NIOSH’s outputs to enhance health and safety. The implementation of research findings may depend on existing or future policy considerations.

III.A.1.
Overview

External factors may be considered as forces beyond the control of NIOSH that may affect the evolution of the program. External factors dominate the evolution of the path from NIOSH inputs to occupational health and safety outcomes (Figure 1). External factors can also be considered inputs to the evaluation of each aspect (planning, implementation, transfer, and others) of NIOSH research programs (Figure 2).

Identification of external factors by the ECs is essential to providing a context for NIOSH program evaluation. External factors may best be assessed through the expert judgment of EC members regarding the knowledge base, the research program, and implementation of interventions as these relate to the needs in the occupational health or safety area targeted by the research program. The ECs, however, may choose additional approaches to assess external factors.

The FC recommends the ECs ask NIOSH to identify and describe external factors early in the evaluation sequence. Factors external to NIOSH might have

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

been responsible for achieving some outcomes, and they might also have presented formidable obstacles. The ECs must address both possibilities.

III.A.2.
Considerations for Discussion

Some external factors may involve constraints on research activity related to target populations, methodological issues, and resource availability. For example, evaluators might examine whether

  • Projects addressing a critical health need are technologically feasible. A workforce with appropriate size and duration, magnitude, and distribution of exposure for measuring a health effect may not exist. For example, no population of workers has been exposed for 30 years to formaldehyde at the current OSHA Permissible Exposure Limit (PEL), so the related cancer mortality cannot yet be directly assessed.

  • Research is inhibited because NIOSH investigators are unable to access an adequate study population. Under current policy, NIOSH must either obtain an invitation by management to study a workplace or seek a judicial order to provide authority to enter a worksite. (Cooperation under court order may well be insufficient for effective research.)

  • Research is inhibited because the work environment, materials, and historical records cannot be accessed even with management and workforce cooperation.

  • Adequate or established methods do not exist for assessing the environment.

  • Records needed for historical-exposure reconstruction cannot be accessed or do not exist.

  • Intervention research is inhibited because an appropriate employer partner cannot be identified to institute the intervention.

  • The NIOSH contribution to a certain area of research is reduced because other institutions are working in the same area.

  • NIOSH resources are inadequate to tackle the key questions.

Evaluation of the impact of NIOSH research outputs on outcomes may require consideration of external factors that might have impeded or aided implementation, measurement, and so on. For example, evaluators might consider whether

  • Regulatory end points are unachievable because of obstacles to regulation or differing priorities of the regulatory agencies. For example, recommen-

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

dations for improved respiratory protection programs for health-care workers might not be implemented because of enforcement policies or lack of acceptance by the administration of health-care institutions.

  • A feasible control for a known risk factor or exposure is not implemented because the costs of implementation are too high or the economic incentives under current circumstances do not favor such actions.

  • Improvements in end points are unobservable because baseline and ongoing surveillance data are not available. For example, the current incidence of occupational noise-induced hearing loss is not known although surveillance for a significant threshold shift is feasible. (NIOSH conducts surveillance of work-related illnesses, injuries, and hazards, but comprehensive surveillance is not possible with existing resources.)

  • Reductions in adverse effects of chronic exposure cannot be measured. For example, 90% of identified work-related mortality is from diseases, such as cancer, that arise only after decades of latency from first exposure; therefore, effects of reducing exposure to a carcinogen cannot be observed in the timeframe of most interventions.

  • A regulation is promulgated that requires a technology that was developed but not widely used.

III.B.
Evaluating NIOSH Research Programs (Addressing Charges 1 and 2)
III.B.1.
Identifying Period of Time to Be Evaluated

Through study of materials presented by the NIOSH research program and other sources, an EC will become familiar with the history of the research program being evaluated and its major subprograms, program goals and objectives, resources, and other pertinent information.

It is useful for the ECs to consider three general timeframes in conducting their reviews:

  • 1970-1995, the period from the founding of NIOSH to the initiation of the NORA process (pre-NORA period).

  • 1996-2005 (NORA 1 period).

  • Current period and forward (NORA 2 period).

It will be important for the ECs to get a general sense of the history of the NIOSH research program and its impact, but their efforts should be focused on

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

the impact and relevance of NIOSH programs from 1996 on. It is recognized that many of the intermediate and end outcomes since 1996 are the consequence of research outputs accomplished earlier. Both the relevance of the research program targets of NORA 1 and the proposed NORA 2 objectives for the next decade should be considered.

NIOSH is in the midst of a substantial restructuring of the NORA agenda, and expert judgment about relevance and prospective impact of current research programs will be most useful to the agency. The timeframes provided here are only for general guidance; the exact dates of the period to focus on in reviewing programs will depend on the specific research program under review.

III.B.2.
Identification of Major Challenges (Circle in Figure 2)

Early in its assessment process, an EC should independently identify the major challenges for its research program. These would be the matters the EC believes should have priority in the research program being evaluated. In arriving at a list of challenges, the EC should rely on surveillance findings, including NIOSH investigations of sentinel events (through health-hazard or fatality-assessment programs), and its own expert judgment. Those should be supplemented with determinations or recommendations by appropriate advisory sources regardless of whether these sources have contributed to NIOSH program deliberations. This process will allow the EC to compare its assessment of challenges to be addressed by NIOSH with NIOSH program goals, and to evaluate the congruence between the two as a measure of relevance (Charge 2).

III.B.3.
Analysis of Research Program Strategic Goals and Objectives (Box A in Figure 2)

The research program goals and objectives should be evaluated, with a focus on how each research program’s goals are related to NIOSH’s agency-wide strategic goals and to the major current challenges and emerging problems identified in the step above. Differences may exist between the importance or relevance of an issue and the influence NIOSH-funded research might have in addressing the issue. The EC should recognize that NIOSH research priorities may be strategic rather than based on the assessment of the state of knowledge.

Some aspects of the NIOSH research program’s strategic goals and objectives would have been already subjected to evaluation by internal or external bodies. Research program relevant evaluations that should be requested include the NIOSH annual program review by the Leadership Team; the NORA research pro-

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

gram proposal pre-award external review, NORA post-award program external review, and external scientific program review.

Questions to Guide the Evaluation Committee

  1. Are the strategic goals and objectives of the program well defined and clearly described?

  2. In the last decade, how well were program goals and objectives aligned with NORA 1 priorities?

  3. How do the current strategic goals and objectives of the program relate to the current NIOSH strategy, including NORA 2?

  4. Are the research program goals, objectives, and strategies relevant to the major challenges in the research program and likely to address emerging problems in the research program (as determined by the EC)?

    1. Did past program goals and objectives (research and dissemination/ transfer activities) focus on the most relevant problems and anticipate the emerging problems in the research program?

    2. Are the current program goals and objectives targeted to the most relevant problems and likely to address emerging problems in the research program?

  1. How does the program identify emerging research areas?

    1. What information is reviewed by NIOSH?

    2. What advisory or stakeholder groups are asked to identify emerging areas?

    3. What new research areas have been identified in the program?

    4. Were important areas overlooked?

Assessment

The EC will provide a qualitative assessment discussing the relevance of the area’s goals, objectives, and strategies as related to the research program’s major challenges and emerging problems.

III.B.4.
Review of Inputs (Box B in Figure 2)

Inputs are categorized as planning or production inputs in the NIOSH logic model. Planning inputs include stakeholder inputs, surveillance and intervention data, and risk assessments. Production inputs include intramural and extramural funding, staffing, management structure, and physical facilities.

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

Inputs for program evaluation include existing intramural and extramural information and, potentially, surveys or case studies that might have been developed specifically to assess progress in reducing workplace illnesses and injuries and to provide information relevant to targeting research appropriately to future needs. The ECs should request the relevant planning and production inputs from NIOSH.

Planning Inputs

Planning inputs can be qualitative or quantitative. Sources of qualitative inputs include

  • Federal Advisory Committee Act panels (Board of Scientific Counselors, Mine Safety and Health Research Advisory Committee, National Advisory Committee on Occupational Safety and Health, and so on).

  • NORA research partners, initial NORA stakeholder meetings, later NORA Team efforts (especially strategic research plans), and the NORA Liaison Committee and federal liaison committee recommendations.

  • Other federal research agendas, industry, labor, academe, professional associations, industry associations, and Council of State and Territorial Epidemiologists.

  • OSHA and MSHA strategic plans.

Attention should be given to how comprehensive the inputs have been and to what extent gaps have been identified or considered.

Sources of quantitative inputs include

  • Intramural surveillance information, such as descriptive data on exposures and outcomes (appropriate data may be available from a number of NIOSH divisions and laboratories).

  • Health Hazard Evaluations (HHEs).

  • Reports from the Fatality Assessment Control and Evaluation (FACE) program.

  • Extramural health-outcome and exposure-assessment data from (1) OSHA and MSHA (inspection data) and the Bureau of Labor Statistics, U.S. Department of Defense, and U.S. Department of Agriculture (fatality, injury, and illness surveillance data); (2) state government partners, including NIOSH-funded state surveillance programs, such as Sentinel Event Notification System of Occupational Risks (SENSOR), Adult Blood

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

Lead Epidemiology and Surveillance (ABLES), and state-based FACE; and (3) non-government organizations, such as the Association of Occupational and Environmental Clinics (AOEC) and the American College of Occupational and Environmental Medicine (ACOEM).

  • Appropriate data from NIOSH-funded, investigator-initiated extramural research.

Production Inputs

For each research program under review, NIOSH should specify an identifiable portion of the NIOSH intramural budget, staff, facilities, and management that has been allocated by divisions and offices that play a major role in the research program. Production inputs should be described primarily in terms of intramural research projects and staff, relevant extramural projects (particularly cooperative agreements and contracts), and HHEs and related staff. Consideration should also be given to budget inputs for program evaluation and to leveraged funds provided by partners, such as National Institutes of Health and the Environmental Protection Agency joint requests for applications or program announcements and OSHA, MSHA, and Department of Defense contracts with NIOSH to conduct work.

Assessment of those inputs should include consideration of (1) the degree to which allocation of funding and personnel has been reasonably consistent with the resources needed to conduct the research and (2) the extent to which funding for the relevant intramural research program activity has been limited by lack of discretionary spending beyond salaries (travel, supplies, external laboratory services, and so on). The assessments, therefore, should consider the adequacy of the qualitative and quantitative planning inputs and the use and adequacy of production inputs, particularly (1) and (2) above.

Questions as a Guide for the Evaluation Committee

  1. Were the planning, production, and other input data adequate?

  2. How well were the major planning, production, and other program inputs used to promote the major activities?

  3. Were the sources of inputs and the amount and quality of inputs adequate?

  4. Was input obtained from stakeholders representing vulnerable working populations and small businesses?

  5. Were production inputs (intramural and extramural funding, staffing, management, and physical infrastructure resources) consistent with goals and objectives of the program?

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

Assessment

The EC will provide a qualitative assessment that discusses the quality, adequacy, and use of inputs.

III.B.5.
Review of Activities (Box C in Figure 2)

Activities are defined as the efforts and work of the program, its staff, and its grantees and contractors. For purposes of the present evaluation, activities of the NIOSH program under review should be divided into research and transfer activities. Research activities may be further categorized as surveillance, health-effects research, intervention research, health-services research, and other research (see sample classification of research activities in Table 2). Transfer activities include information dissemination, training, technical assistance, and education designed to translate research outputs into content and formats designed for application in the workplace to produce improvements in occupational safety and health. Depending on the scope of the program under review, activities may also be grouped by research program objectives or subprograms.

Conventional occupational-health research focuses appropriately on health effects and technology. A focus on socioeconomic and policy research and on surveillance and diffusion research is also needed to effect change because not all relevant intermediate outcomes occur in the workplace. There are important outcomes farther out on the causal chain that NIOSH can affect and thereby influence health and safety in the workplace. Some examples of types of research that might also prove important in addressing NIOSH’s mission are

  • Socioeconomic research on cost shifting between worker compensation and private insurance.

  • Surveillance research to assess the degree of significant and systematic underreporting of select injuries and illnesses on OSHA logs.

  • Research on methods to build health and safety capacity in community health centers that serve low-income and/or minority-group workers, and to improve recognition and treatment of work-related conditions.

  • Transfer research to change health and safety knowledge in teenagers while they are in high school to improve the likelihood of reduced injuries when they enter the workforce.

  • Community-based participatory research on differences between recently arrived immigrants and US-born workers regarding perceptions of acceptable health and safety risks to target programs to meet the workforce training needs of immigrant workers.

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

TABLE 2 Examples of NIOSH Program Research and Transfer Activities

Surveillance

(including hazard and health surveillance and evaluation of surveillance systems)

Health-effects research

Epidemiologic research

Toxicologic research

Laboratory-based physical and safety risk factor research

Development of clinical screening methods and tools

Exposure-assessment research

Intervention research

Control technology

Engineering controls and alternatives

Administrative controls

Personal protective equipment

Work organization research

Community-based participatory research

Policy research (such as alternative approaches to targeting inspections)

Diffusion and dissemination research

Training effectiveness

Information-dissemination effectiveness

Diffusion of technology

Health-services and other research

Access to occupational health care

Infrastructure research—delivery of occupational-health services, including international health and safety

Socioeconomic consequences of work-related injuries and illnesses

Worker compensation

Technology-transfer and other transfer activities

Information dissemination

Training programs

The ECs should review the list of research and transfer activities (projects) for the research program under review that have been completed, are in progress, or have been planned. Surveillance activities should be included in this review. An EC should request that the NIOSH program under review provide a list of activities, grouping the projects into research activities as in Table 2, and specify whether they are intramural or extramural. For extramural projects, the key organizations and principal investigators’ names should be requested, as should whether the projects were in response to a request for proposal or a request for application. For

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

an intramural project, the EC should ask NIOSH to provide a list of key collaborators (other government agency, academe, industry, and/or union partners).

The ECs should evaluate each of the research activities outlined in Table 2 to the extent that each forms an important element of the program research. In the case of a sector research program (for example, mining, construction) in which health-effects research is not being reviewed, the ECs should determine what research inputs are being used by the program to develop its targets and then assess the value of the inputs.

Questions to Guide the Evaluation Committee in Assessing Research Activities

  1. What are the major subprograms or groupings of activities within the program?

  2. Were the activities consistent with program goals and objectives?

  3. Were the research activities relevant to the major challenges in the research program?

    1. Did they address the most serious outcomes?

    2. Did they address the most common outcomes?

    3. Did they address the needs of both genders, vulnerable working populations, and small businesses?

  1. Were the research activities appropriately responsive to the input of stakeholders?

  2. To what extent were partners involved in the research activities?

  3. Are the resource allocations appropriate, and appropriate at this time, for the research activities?

  4. To what extent did peer reviews (internal, external, and precourse or midcourse) affect the activities?

  5. Is there adequate monitoring of quality assurance procedures to ensure credible research data, analyses, and conclusions?

Questions to Guide the Evaluation Committee in Assessing Transfer Activities

  1. Is there a coherent planned program of transfer activities?

  2. Are the program’s information dissemination, training, education, technical assistance, or publications successful in reaching the workplace or relevant stakeholders in other settings? How widespread is the response?

  3. To what extent did the program build research and education capacity (internal or external)?

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

Assessment

For this part of the assessment, the EC will provide a qualitative assessment discussing relevance and quality. This evaluation must include consideration of the external factors identified in Section III.A that constrain choices of research projects. The EC will consider the appropriateness of resource allocations with respect to issues’ importance and the extent to which the issue is being addressed. A highly relevant and high-quality program would be comprehensive, address high-priority needs, produce high-quality results, be highly collaborative, and be of value to stakeholders. Programs may be progressively less relevant or of lower quality as those key elements are not up to the mark or are missing. The discussion should cover those aspects in sufficient detail to arrive at a qualitative assessment of the activities. Assessment of the transfer activities must include considerations of program planning, coherence, quality, and impact.

III.B.6.
Review of Outputs (Box D in Figure 2)

As shown in Figure 1, research inputs and activities lead to outputs. An output is a direct product of a NIOSH research program that is logically related to the achievement of desirable and intended outcomes. Outputs are created for researchers, practitioners, intermediaries, and end-users, such as consumers. Outputs can be in the form of publications in peer-reviewed journals, recommendations, reports, Web-site content, workshops and presentations, databases, educational materials, scales and methods, new technologies, patents, technical assistance, and so on. Outputs of NIOSH’s extramurally funded activities should also be considered. Examples of major outputs are provided in Table 3.

Depending on the intended audience, outputs may be tailored to communicate information most effectively to increase the likelihood of comprehension, knowledge, attitude formation, and behavioral intent. The extent of use of formative evaluation data (data gathered prior to communication for the purpose of improving the likelihood of the intended effects) or intended user feedback in the design of the output can be considered an indicator of output quality.

In addition to outputs themselves, many related indicators of the production, reference to, and utility of outputs can be conceptualized and made operational. Examples include the extent of collaboration with other organizations in the determination of research agendas, the conduct of research, the dissemination of research results, and interorganizational involvement in the production of outputs. Coauthorship is a measure of the centrality of NIOSH researchers in the broader research community.

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

TABLE 3 Examples of a Variety of Scientific Information Outputs

Peer-reviewed publications by NIOSH staff

Total number of original research articles by NIOSH staff

Total number of review articles by NIOSH staff (including best-practice articles)

Complete citation for each written publication

Complete copies of the “top five” articles

Collaboration with other public- or private-sector researchers

Publications in the field of interest with other support by investigators also funded by NIOSH (for example, ergonomic studies with other support by an investigator funded by NIOSH to do ergonomics work, in which case NIOSH should get some credit for seeding interest or drawing people into the field)

Peer-reviewed publications by external researchers funded by NIOSH

Total number of NIOSH-funded original research articles by external researchers

Total number of NIOSH-funded review articles by external researchers (including best-practices articles)

Complete citation for each written report

Complete copies of the “top five” articles

Collaboration with other government or academic researchers

NIOSH reports in the research program

Total number of written reports

Complete citation for each written report

Complete copies of the “top five” reports

Sponsored conferences and workshops

Total number of sponsored conferences

Total number of sponsored workshops

For each sponsored conference or workshop, describe:

Title, date, and location

Partial vs complete sponsorship (if partial, who were cosponsors?)

Approximate number of attendees and composition of participants

Primary “products” of the event (such as publication of conference proceedings)

NIOSH’s assessment of value or impact

Databases

Total number of major databases created by NIOSH staff

Total number of major databases created by external researchers funded by NIOSH grants,

For each database:

Title, objective (in one to four sentences), and start and stop dates

Partial vs complete sponsorship (if partial, who were cosponsors?)

Study or surveillance-system design, study population, and sample size

Primary “products” of the database (such as number of peer-reviewed articles and reports)

Complete copies of the “top two” publications and/or findings, to date, from each database

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

Recommendations

Total number of major recommendations

For each:

Complete citation (article, report, or conference where recommendation was made)

Summary in one to four sentences

Percentage of target audience that has adopted recommendation 1, 5, and 10 years later

Up to three examples of implementation in the field

Identifications of “top five” recommendations to date

Tools, methods, or technologies (TMT)

Total number of major TMT (includes training and education materials)

For each:

Title and objective of TMT (in one to four sentences)

Complete citation (if applicable)

Percentage of target audience that has used TMT 1, 5, and 10 years later

Up to three examples of implementation in the field

Identification of “top five” TMT to date

Patents

Total number of patents

For each:

Title and objective patent (in one to four sentences)

Complete citation

Percentage of target audience that has used product 1, 5, and 10 years later

Up to three examples of implementation in the field

Identification of “top five” patents to date

Miscellaneous

Any other important program outputs

The EC should ask NIOSH to provide information on all relevant outputs for the specific program for the chosen time period.

Questions to Guide the Evaluation Committee

  1. What are the major outputs of the research program?

  2. Did the research program produce outputs that addressed the high-priority areas?

  3. To what extent did the program generate important new knowledge or technology?

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
  1. Are there peer-reviewed publications that are widely cited and considered to report “breakthrough” results?

  2. Were outputs relevant to both genders, vulnerable populations, and health disparities?

  3. Were outputs relevant to health and safety problems of small businesses?

  4. Are products user-friendly in terms of readability, simplicity, and design?

  5. To what extent did the program help to build the internal or extramural institutional knowledge base?

  6. Did the research produce effective cross-agency, cross-institute, or internal-external collaborations?

Assessment

For this part of the assessment, the EC should provide a qualitative assessment discussing relevance, quality, and usefulness. A highly ranked program will be one with outputs that address needs in high-priority areas, contain new knowledge or technology that is effectively communicated, contribute to capacity-building both inside and outside NIOSH, and are relevant to the pertinent populations. The discussion should cover those aspects in sufficient detail to support the qualitative assessment of the outputs.

III.B.7.
Review of Intermediate Outcomes (Box E in Figure 2)

Intermediate outcomes, for the purposes of this evaluation, are related to the program’s association with behaviors and changes at individual, group, and organizational levels in the workplace. An intermediate outcome reflects an assessment of worth by stakeholders outside NIOSH (such as managers in industrial firms) about NIOSH research or its products.

Intermediate outcomes include the production of standards, or regulations based in whole or in part on NIOSH research (products adopted as public policy or as policy or guidelines by private organizations or industry); attendance at training and education programs sponsored by other organizations; use of publications by workers, industry, and occupational safety and health professionals in the field; and citations of NIOSH research by industrial and academic scientists.

More difficult-to-collect intermediate outcomes that may be valid indicators of quality or utility include self-report measures by users and relevant nonusers of NIOSH outputs. These indicators include the extent to which key intermediaries find value in NIOSH databases for the repackaging of health and safety information, the extent to which NIOSH recommendations are in place and attended to in

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

workplaces, and employee or employer knowledge of and adherence to NIOSH-recommended practices.

A research program might be evaluated in terms of whether it is recognized as a national center of excellence, is one of the larger and best research programs in the country, is recognized only in terms of particular staff or a particular laboratory, duplicates other, larger facilities, or is not unique or has little capability or capacity.

Questions to Guide the Evaluation Committee

  1. Has the program resulted in stakeholder training or education activities that are being used in the workplace or in school or apprentice programs? If so, what is the response to what is being done, and how widespread is the response?

  2. Has the program resulted in standards, regulations, public policy, or voluntary guidelines that have been transferred to or created by the workplace in response to NIOSH outputs?

  3. Has the program resulted in new control technology or administrative control concepts that are feasible for use or have been adopted in the workplace to reduce risk factors?

  4. Has the program resulted in new personal protective equipment that is feasible for use or has been adopted in the workplace to reduce risk factors or exposures?

  5. Has the program contributed to changes in health care practices to improve recognition and management of occupational health conditions?

  6. Has the program resulted in research partnerships with stakeholders leading to changes in the workplace?

  7. To what extent did the program’s stakeholders find value in NIOSH’s products (as shown by document requests, web hits, conference attendance, and so on)?

  8. Has the program resulted in changes in employer or worker practices associated with the reduction of risk factors?

  9. Does the program or a subprogram provide unique staff or laboratory capability that is a necessary national resource? If so, is it adequate or does it need to be enhanced or reduced?

  10. Has the program resulted in interventions that protect both genders, vulnerable workers or address the needs of small businesses?

  11. To what extent did the program contribute to increased capacity at worksites to identify or respond to threats to safety and health?

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

Assessment

Only a qualitative assessment of product development, usefulness, and impact is required at this point in the EC report. Some thought should be given to the relative value of intermediate outcomes, and the FC recommends applying the well-accepted hierarchy-of-controls model. The discussion could include comments on how widely products have been used or programs implemented. The qualitative discussion should be specific as to the various products developed by the program and the extent of their use by specific entities (industry, labor, government, and so on) for specific purposes. Whether the products have resulted in changes in the workplace or in the reduction of risk factors should be discussed. The recognition accorded to the program or the facilities by its peers (such as recognition as a “center of excellence” by national and international communities) should be considered in the assessment. A program to be highly ranked should have a high level of performance in most of the relevant questions in this section. Whether the impact was caused by NIOSH alone or in combination with external agents should also be considered in the evaluation. An aspect of the evaluation can be whether the impact would have probably occurred without NIOSH’s efforts.

III.B.8.
Review of End Outcomes (Box F in Figure 2)

End outcomes are defined by measures of health and safety and of impact on process and programs. The FC recognizes that a major challenge in assessing the causal relationship between NIOSH research and specific occupational health and safety outcomes is that NIOSH does not have direct responsibility or authority for implementing its research findings in the workplace. Furthermore, the benefits of NIOSH research program outputs can be realized, potential, or limited to knowledge gain. For example, negative studies contribute to the knowledge base and the generation of important new knowledge is a recognized form of outcome, in the absence of measurable impacts.

Outcome impact depends on there being a “receptor” for research results, including regulatory agencies, consensus and professional organizations, and employers. The ECs should consider questions related to the various stages that lead to outputs, such as

  1. Did NIOSH research identify a gap in protection or a means of reduction of risk?

  2. Did NIOSH convey that information to potential users in a usable form?

  3. Was the research applied?

  4. Did the results work?

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

End outcomes, for purposes of this evaluation, are changes related to health, including decreases in injuries, illnesses, deaths, and decreases in exposures or risk factors resulting from the research in the specific program or subprogram. Quantitative data are preferable to qualitative, but qualitative analysis may be necessary.

Sources of quantitative data include

  • Bureau of Labor Statistics (BLS) data on fatal occupational injuries (Census of Fatal Occupational Injuries) and nonfatal injuries and illnesses (Annual Survey of Occupational Injury and Illnesses).

  • NIOSH intramural surveillance systems, such as the National Electronic Injury Surveillance System (NEISS), the coal worker x-ray surveillance program, and agricultural worker surveys conducted by NIOSH in collaboration with the US Department of Agriculture.

  • State-based surveillance systems, such as the NIOSH-funded ABLES, and the SENSOR programs (for asthma, pesticides, silicosis, noise-induced hearing loss, dermatitis, and burns).

  • Selected state workers-compensation programs.

  • OSHA, which collects exposure data, in the Integrated Management Information System.

The FC is unaware of surveillance mechanisms for many occupationally related chronic illnesses such as cancers arising from long exposure to chemicals and other stressors. For many outcomes, incidence and prevalence are best evaluated by investigator-initiated research.

The strengths and weaknesses of the various sources of outcome data should be recognized by the ECs. Quantitative accident, injury, illness, and employment data and databases are subject to error and bias and should be used by the ECs for drawing inferences only after critical evaluation and examination of whatever corroborating data are available. For example, it is widely recognized that occupational illnesses are poorly documented in the BLS Survey of Occupational Injuries and Illnesses, which captures only incident cases among active workers. Most illnesses that may have a relationship to work are not exclusively so related, and it is difficult for health practitioners to diagnose work-relatedness; few are adequately trained to make this assessment. Many of these illnesses have long latency and do not appear until years after people have left the employment in question. Surveillance programs may systematically undercount some categories of workers, such as contingent workers. Challenges posed by inadequate or inaccurate measurement systems should not drive programs out of difficult areas of study, and the ECs will need to be aware of such a possibility. In particular, contingent and informal working arrangements that place workers at greatest risk are also those

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

on which surveillance information is almost totally lacking, so novel methods for measuring impact may be required.

In addition to measures of illness and injury, levels of exposure to chemical and physical agents and to safety and ergonomic hazards can be useful. Exposure or probability of exposure can serve as an appropriate proxy for disease or injury when a well-described occupational exposure-health association exists. In such instances, decreased exposure can be accepted as evidence that the end outcome of reduced illness has been achieved. That is particularly necessary in cases (such as exposure to asbestos) in which latency between exposure and disease outcome (lung cancer) makes effective evaluation of the relevant end outcome infeasible.

As an example of how exposure levels can serve as a proxy, the number of sites that exceed an OSHA Permissible Exposure Limit (PEL) or an American Conference of Governmental Industrial Hygienists threshold limit value is a quantitative measure of improvement of occupational health awareness and reduction of risk. In addition to exposure level, the number of people exposed and the distribution of exposure levels are important. Those data are available from multiple databases and studies of exposure. Apart from air monitoring, such measures of exposure as biohazard controls, reduction in requirements for use of personal protective equipment, and reduction of ergonomic risks are important.

Clearly, the commitment of industry, labor, and government to health and safety are critical external factors. Several measures of this commitment can be useful for the EC: monetary commitment of the groups, attitude, staffing, and surveys of relative level of importance. To the extent that the resources allocated to safety and health are limiting factors, the ECs should explicitly assess NIOSH performance in the context of constraints.

Questions to Guide the Evaluation Committee

  1. What are the amounts and qualities of end-outcomes data (such as injuries, illness, exposure and productivity affected by health)?

  2. What is the temporal trend in those data?

  3. Is there objective evidence of improvements in occupational safety or health?

  4. To what degree has the NIOSH program or subprogram been responsible for improvements in occupational safety or health?

  5. If there is no time trend in the data, how do findings compare with data from other comparable US groups or the corresponding populations in other countries?

  6. Is there evidence that external factors have affected outcome measures?

  7. Has the program been responsible for outcomes outside the United States that have not been described in another category?

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

Assessment

For this part of the assessment, the EC should provide a qualitative assessment discussing the evidence of reductions in injuries and illnesses or their appropriate proxies (impacts).

III.B.9.
Review of Other Outcomes

There may be health and safety impacts not yet appreciated, and other beneficial social, economic, and environmental outputs, including potential NIOSH impacts outside the United States. Many NIOSH study results and training programs may be judged to be important, or there may be evidence of implementation of NIOSH recommendations, outside the United States.

Questions to Guide the Evaluation Committee

  1. Is the program likely to produce a favorable change that has not yet occurred or not been appreciated?

  2. Has the program been responsible for other social, economic, security, or environmental outcomes?

  3. Has the program’s work had an impact on occupational health and safety in other countries?

Assessment

Evaluation by the EC may consist of a discussion of other outcomes, including positive changes that have not yet occurred; other social, economic, security, or environmental outcomes; and the impact that NIOSH has had on international occupational safety and health. It might also consider the incorporation of international research results into the NIOSH program of knowledge transfer for industry sectors.

III.B.10.
Summary Evaluation Ratings and Rationale

An EC should use its expert judgment to rate the relevance and impact of the research program and its important subprograms by first summarizing its assessments of the subprograms and overall program according to the several items listed in Table 4. Table 4 is only a worksheet intended as an aid to the EC in its evaluation. Its purpose is to encourage the EC to summarize its work in one place and to concentrate on the subprograms and the items that will contribute to the final impact and relevance scores.

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

TABLE 4 Evaluation Committee Worksheet to Assess Research Programs and Subprograms

Please respond to each with “major or important,” “moderate,” “likely,” “limited,” or “none.”

Background Context for Program Impact

1.1

Evidence of reduction of risk factors in the workplace (intermediate outcome) and evidence that external factors affected reduction

1.2

Evidence of reduction in workplace exposure, illness, or injuries (end outcome) and evidence that external factors affected reduction

Addressing Charge 1

Activity Category

Program

Subprogram

1

n

1.3

Contributions of NIOSH research and transfer activities to changes in work-related practices

Research

 

 

 

 

 

 

Transfer

 

 

 

 

 

1.4

Contributions of NIOSH research and transfer activities to reductions in workplace exposure, illness, or injuries

Research

 

 

 

 

 

 

Transfer

 

 

 

 

 

1.5

Evidence of external factors preventing application of NIOSH research results

Research

 

 

 

 

 

 

Transfer

 

 

 

 

 

1.6

Contribution of NIOSH research to enhancement of capacity in government or other research institutions

Research

 

 

 

 

 

 

Transfer

 

 

 

 

 

1.7

Contributions of NIOSH research to productivity, security, or environmental quality (beneficial side effects)

Research

 

 

 

 

 

 

Transfer

 

 

 

 

 

Addressing Charge 2

 

 

 

 

 

 

2.1

Relevance of current and recently completed research and transfer activities to future improvements in workplace safety and health

Research

 

 

 

 

 

 

Transfer

 

 

 

 

 

2.2

Progress in targeting research to areas of study most relevant to future improvements in occupational safety and health

Research

 

 

 

 

 

 

Transfer

 

 

 

 

 

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

To set the context for this step in the evaluation of the impact of the research program in preparation to respond to charge 1, the EC will first need to consider the available evidence of changes in work-related risks and adverse effects and external factors related to the changes. That information should be organized as a prose response to items 1.1 and 1.2 in Table 4.

Next, the EC should review the responses to the questions in Sections III.B.6 through III.B.8 and systematically rate the impact of the research program and its subprograms by responding to items 1.3-1.7 in Table 4. To complete the table, the EC response should use one of the following five terms: “major or important,” “moderate,” “likely,” “limited,” or “none” (since 1995). The EC should evaluate separately the impact of the research and the impact of transfer activities. High ratings on items 1.3-1.7 require the committee’s judgment that the program has contributed to outcomes. For example, outcomes have occurred earlier than they would have or are better than they would have been in the absence of the research program, or outcomes would have occurred in the absence of external factors beyond NIOSH’s control or ability to plan around.

The EC should then assess the relevance of the research program and subprograms in preparation for addressing charge 2. The EC should review the responses to the questions in Sections III.B.2 through III.B.5 and rate the relevance of the research program and its subprograms by responding to items 2.1 and 2.2 in Table 4. The same five terms should be used (“major or important,” “moderate,” “likely,” “limited,” or “none”) to evaluate separately the relevance of the research and the relevance of the transfer activities. Transfer activities occur in two contexts: (1) NIOSH efforts to translate intellectual products into practice and (2) efforts by stakeholders to take advantage of NIOSH products.

Final Program Ratings

To provide the final assessment of the research program for charge 1 (impact) and charge 2 (relevance), the ECs will use their expert judgment, their responses to the questions in Table 4, and any other appropriate information to arrive at one overall rating for the impact of the research program and one for its relevance to the improvement of occupational safety and health. In light of substantial differences among the types of research programs that will be reviewed and the challenge to arrive at a summative evaluation of both impact and relevance, however, the FC chose not to attempt to construct a single algorithm to produce the two final ratings.

Having completed Table 4, the EC should undertake its final assessment of the impact and relevance of the program. Final program ratings will consist of the numerical scores and prose descriptions of why the scores were given. As explained below, the ECs will summarize their responses to charges 1 and 2 by rating

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

the relevance and impact of the NIOSH research program on five-point scales in which 1 is the lowest and 5 the highest rating. The FC has made an effort to establish mutually exclusive rating categories in the five-point rating scale; when the basis of a rating fits more than one category, the highest applicable score should be assigned. ECs will need to consider the impact and relevance of both NIOSH completed research and research in progress. In general, the assessment of impact will consider research completed, and the assessment of relevance will include research in progress related to likely future improvements. When assessing the relevance of the program, the EC should keep in mind how well the program has considered the frequency and severity of the problems being addressed, whether appropriate attention has been directed to both genders, vulnerable populations or hard-to-reach workplaces, and whether the different needs of large and small businesses have been accounted for.

The FC has some concern that the impact scoring system proposed below might be considered a promotion of the conventional occupational-health research paradigm that focuses on health-effect and technology research and not give much emphasis to socioeconomic and policy research and to surveillance and diffusion research (as opposed to activities) needed to effect change. Clearly, not all intermediate outcomes occur in the workplace. There are important outcomes much farther out on the causal chain that NIOSH can affect, and not all these can be defined as well-accepted intermediate outcomes. NIOSH, for example, has an important role to play in generating knowledge that may contribute to changing norms in the insurance industry, in health-care practice, in public-health practice, and in the community at large. The ECs may find that some of these issues need to be addressed and considered as important to influence the external factors that limit application of more traditional research findings. Given the rapidly changing nature of work and the workforce and some of the intractable problems in manufacturing, mining, and some other fields, the ECs are encouraged to think beyond the traditional paradigm.

Rating of Impact

5

=

Research program has made a major contribution to worker health and safety on the basis of end outcomes or well-accepted intermediate outcomes.

4

=

Research program has made a moderate contribution on the basis of end outcomes or well-accepted intermediate outcomes; research program generated important new knowledge and is engaged in transfer activities, but well-accepted intermediate outcomes or end outcomes have not been documented.

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

3

=

Research program activities or outputs are going on and are likely to produce improvements in worker health and safety (with explanation of why not rated higher).

2

=

Research program activities or outputs are going on and may result in new knowledge or technology, but only limited application is expected.

1

=

Research activities and outputs are NOT likely to have any application.

NA

=

Impact cannot be assessed; program not mature enough.

Rating of Relevance

5

=

Research is in highest-priority subject areas and highly relevant to improvements in workplace protection; research results in, and NIOSH is engaged in, transfer activities at a significant level (highest rating).

4

=

Research is in high-priority subject area and adequately connected to improvements in workplace protection; research results in, and NIOSH is engaged in, transfer activities.

3

=

Research focuses on lesser priorities and is loosely or only indirectly connected to workplace protection; NIOSH is not significantly involved in transfer activities.

2

=

Research program is not well integrated or well focused on priorities and is not clearly connected to workplace protection and inadequately connected to transfer activities.

1

=

Research in the research program is an ad hoc collection of projects, is not integrated into a program, and is not likely to improve workplace safety or health.

III.C.
Identifying Significant Emerging Research (Addressing Charge 3)

Among the most challenging aspects of conducting research for the purpose of prevention of injury and illness is identifying new or emerging needs or trends and formulating an active research response that appropriately uses scarce resources in anticipation of those needs. Each EC should review the procedures that NIOSH has in place to identify needed research relevant to the NIOSH mission.

Each EC should review the success that NIOSH has had in identifying and addressing research to emerging issues. The review should include examination of leading indicators from appropriate federal agency sources, such as the Environmental Protection Agency, the Department of Labor, the National Institute of Standards and Technology, the National Institutes of Health, the Department of Defense, and the Department of Commerce. Those indicators should track new technologies, products, and processes and disease or injury trends.

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

One source of inputs deserving particular attention is the NIOSH HHE reports. NIOSH’s HHE program is a separate legislatively mandated program that offers a potential mechanism to identify emerging research needs that could be incorporated as an input in each of the programs evaluated. The ECs should consider whether appropriate consideration has been given to findings from the HHE investigations as they are related to the research program under review.

Some additional indicators might include NIOSH and the NIOSH-funded FACE, the AOEC reports, the US Chemical Safety Board investigations, SENSOR and other state-based surveillance programs, and others. In addition, appropriate federal advisory committees and other stakeholder groups should be consulted to provide qualitative information.

The EC members should use their expert judgment both to evaluate what NIOSH has identified as emerging research targets (charge 2) and to respond to charge 3 by providing recommendations to NIOSH for additional research that NIOSH has not yet identified. An EC’s response to charge 3 will consist primarily of recommendations for research in subjects that the EC considers important and of the committee’s rationale.

Questions to Guide the Evaluation Committee

  1. What information does NIOSH review to identify emerging research needs?

    1. What is the process for review?

    2. How often does the process take place?

    3. How are NIOSH staff scientists and NIOSH leadership engaged?

    4. What is the process for moving from ideas to formal planning and resource allocation?

  1. How are stakeholders involved?

    1. What advisory or stakeholder groups are asked to identify emerging research targets?

    2. How often are such groups consulted, and how are suggestions followed up?

  1. What new research targets have been identified for future development in the program under evaluation?

    1. How were they identified?

    2. Were there lessons learned that could help to identify other emerging issues?

    3. Does the EC agree with the issues identified and selected as significant and with the NIOSH response, or were important issues overlooked?

    4. Is there evidence of unwise expenditure of resources on unimportant issues?

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×

IV.
EVALUATION COMMITTEE REPORT TEMPLATE

The following outline flows from the FC’s review of the generalized logic model prepared by NIOSH, the request for information from NIOSH programs, and the assessment model described earlier in this report.

  1. Introduction:

    This section should be a brief descriptive summary of the history of the program (and subprograms) being evaluated, with respect to pre-NORA, NORA 1, and current and future plans of the research program presented by NIOSH. It presents the context for the research on safety and health; goals, objectives, and resources; groupings of subprograms; and any other significant or pertinent information. (A list of the NIOSH materials reviewed should be provided in an appendix to the EC report.)

  2. Evaluation of programs and subprograms (charges 1 and 2):

    1. Evaluation summary (includes a brief summary of the evaluation with respect to impact and relevance, scores for impact and relevance, and summary statements addressing charges 1 and 2).

    2. Strategic goals and objectives: Describes assessment of the subprograms and overall program for relevance.

    3. Review of inputs: Describes adequacy of inputs to achieve goals.

    4. Review of activities: Describes assessment of the relevance and quality of the activities.

    5. Review of research program outputs: Describes assessment of relevance, quality, and potential usefulness of the research program.

    6. Review of intermediate outcomes and causal impact: Describes assessment of the intermediate outcomes and the causal attribution to NIOSH; includes the likely impacts and recent outcomes in the assessment.

    7. Review of end outcomes: Describes the end outcomes related to health and safety and provides an assessment of the type and degree of causal attribution to NIOSH.

    8. Review of other outcomes: Discusses other health and safety impacts that have not yet occurred; other beneficial social, economic, and environmental outcomes; and international dimensions and outcomes.

    9. Summary of ratings and rationale (see Table 4).

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
  1. Identification of needed research (charge 2):

    The EC should assess the progress that the NIOSH program has made in targeting new research in the fields of occupational safety and health. There should be a discussion of the assessment process and results.

  2. Emerging research areas (charge 3):

    The EC should assess whether the NIOSH program has identified significant emerging research areas that appear especially important in terms of their relevance to the mission of NIOSH. The EC should respond to NIOSH’s perspective and add its own recommendations.

  3. Recommendations for program improvement:

    On the basis of the review and evaluation of the program, the EC may provide recommendations for improving the relevance of the NIOSH research program to health and safety conditions in the workplace and the impact of the research program on health and safety in the workplace as related to the research program under review.

Appendix A: List of the NIOSH and related materials collected in the process of the evaluation.

V.
FRAMEWORK COMMITTEE FINAL REPORT

At the conclusion of all individual program reviews, the FC will prepare a final report summarizing the findings of all the evaluating committees and providing NIOSH with an overall evaluation. All program ratings will be summarized and might be plotted graphically or with a Web chart.

The following is a proposed outline of the FC’s final report:

  1. Summary of national needs identified by the research programs reviewed.

    1. On the basis of the best available evidence, place those needs in the context of the overall estimated potential work-related disease and injury burden.

    2. Discuss the choices made and alternatives that might be the focus of current or future attention.

    3. Comment on programs not selected by NIOSH for evaluation by the National Academies.

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
  1. Assessment of how well the program goals.

    1. Were matched to the research program needs.

    2. Were adjusted to new information and inputs as the field of interest changed or program results became available.

  1. Assessment of NIOSH overall performance in the research programs reviewed.

    1. Distribution of available inputs.

    2. Activities and outputs.

    3. Intermediate outcomes.

    4. Summary assessment of significant differences among the programs

    5. International impact.

    6. Leveraging of the NIOSH research activity with respect to other public and private research programs.

    7. Assessment of relative importance of external factors in permitting or preventing intermediate or end outcomes; attention paid to accounting for and planning within the constraints of external factors (not simply assigning lack of progress to external factors).

  1. Overall assessment of NIOSH impact on progress in reducing occupational injury and illness.

    1. Breakthrough knowledge.

    2. International impact.

    3. Addressing disparities.

    4. Targeting residual risks and intractable risks.

    5. Coordinating NIOSH research activity with respect to other public and private research programs.

    6. Impact on occupational safety and health.

  1. Summary, Conclusions, and Recommendations.

Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 125
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 126
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 127
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 128
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 129
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 130
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 131
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 132
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 133
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 134
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 135
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 136
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 137
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 138
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 139
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 140
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 141
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 142
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 143
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 144
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 145
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 146
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 147
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 148
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 149
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 150
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 151
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 152
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 153
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 154
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 155
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 156
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 157
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 158
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 159
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 160
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 161
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 162
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 163
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 164
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 165
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 166
Suggested Citation:"Appendix A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health." Institute of Medicine and National Research Council. 2006. Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. doi: 10.17226/11721.
×
Page 167
Next: Appendix B Methods Section: Committee Information Gathering »
Hearing Loss Research at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health Get This Book
×
Buy Paperback | $66.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National Institute for Occupational Safety and Health (NIOSH) was established by the Occupational Safety and Health Act of 1970 (U.S. Congress, 1970). Today the agency is part of the Centers for Disease Control and Prevention of the U.S. Department of Health and Human Services. NIOSH is charged with the responsibility to "conduct . . . research, experiments, and demonstrations relating to occupational safety and health" and to develop "innovative methods, techniques, and approaches for dealing with [those] problems" (U.S. Congress, 1970). Its research targets include identifying criteria for use in setting worker exposure standards and exploring new problems that may arise in the workplace. Prevention of occupational hearing loss has been part of the NIOSH research portfolio from the time the agency was established. A principal cause of occupational hearing loss is the cumulative effect of years of exposure to hazardous noise. Exposure to certain chemicals with or without concomitant noise exposure may also contribute to occupational hearing loss. Hearing loss may impede communication in the workplace and contribute to safety hazards. Occupationally acquired hearing loss may also have an adverse effect on workers' lives beyond the workplace. No medical means are currently available to prevent or reverse it, although hearing aids are widely used and research on other treatments is ongoing. Occupational hearing loss is a serious concern, although the number of workers affected is uncertain.

In September 2004, NIOSH requested that the National Academies conduct reviews of as many as 15 NIOSH programs with respect to the impact and relevance of their work in reducing workplace injury and illness and to identify future directions that their work might take. The Hearing Loss Research Program was selected by NIOSH as one of the first two programs to be reviewed.

Hearing Loss Research at NIOSH examines the following issues for the Hearing Loss Research Program: (1) Progress in reducing workplace illness and injuries through occupational safety and health research, assessed on the basis of an analysis of relevant data about workplace illnesses and injuries and an evaluation of the effect that NIOSH research has had in reducing illness and injuries, (2) Progress in targeting new research to the areas of occupational safety and health most relevant to future improvements in workplace protection, and (3) Significant emerging research areas that appear especially important in terms of their relevance to the mission of NIOSH.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!