The Health Hazard Evaluation Program at NIOSH: Reviews of Research Programs of the National Institute for Occupational Safety and Health (2009)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1 Introduction CHARGE TO THE COMMITTEE In 2005, the National Institute for Occupational Safety and Health (NIOSH) requested that the National Academies review up to 15 of its research programs. The Committee to Review the NIOSH Health Hazard Evaluation (HHE) Program was formed in 2007 to carry out an independent evaluation of the HHE Program under the following charge: The National Academies will appoint an ad hoc committee to carry out an independent evaluation of the impact, relevance, and future directions of the NIOSH Health Hazard Evaluation (HHE) Program. The committee will evaluate not only what the NIOSH HHE program is producing, but will also determine whether it is appropriate to credit NIOSH activities with changes in workplace practices, hazardous exposures, and/or occupa- tional illnesses and injuries, or whether the changes are the result of other factors unrelated to NIOSH. In conducting its assessment, the committee will evaluate 1. The impact of the program on •  reducing worker risk and preventing occupational illness in investi- gated workplaces; 13 

14 T h e H e a l t h H a z a r d E v a l u a t i o n P r o g r a m a t NIOSH •  transferring program-generated information to relevant employers and employees beyond the investigated workplaces; •  NIOSH research and policy-development programs; and •  activities of regulatory agencies, occupational safety and health the professionals and organizations, state and local health agencies, and others in the occupational health community, as achieved by trans- ferring program-generated hazard and prevention information. 2. The relevance of the program in addressing current and emerging workplace health hazards. The evaluation committee will provide quantitative assessments in the form of integer scores (on a scale of 1-5) for both the relevance and impact of the program in addressing workplace health hazards, to be accompanied by qualitative assessments of all the categories above. The committee will develop its own methodology for the evaluation, guided by the methodology and framework developed by the Com- mittee to Review NIOSH Research Programs where appropriate. The HHE Program does not conduct traditional research as do other NIOSH programs. It is mandated to respond to requests for assistance to identify specific workplace conditions that pose health hazards to workers (NIOSH, 2007b). The charge to this committee was therefore modified from that given for other evalu- ations in this series. Definitions of Impact and Relevance In developing its evaluation methodology, the committee carefully studied the charge and determined definitions for impact and relevance. For this evaluation, impact is defined more broadly than for the evaluations of other NIOSH programs. This committee is asked to determine whether the HHE Program contributes to occupational health in the workplace, as well as whether the program positively affects policy, other research programs, and the health and safety community in general. The charge to the committee specifically includes transfer of information as part of impact, although the methodology and evaluation framework (hereafter called the Framework Document) developed by the Committee to Review NIOSH Research Programs (hereafter called the Framework Committee) tends to include transfer in its definition of relevance. The committee considers transfer activities important for both relevance (Is the program doing the right things?) and impact (Is what the program is doing effective?). 

Introduction 15 The Framework Document defines a relevant program as an integrated pro- gram involving interrelated surveillance, research, and transfer activities. The evaluation criteria include the severity or frequency of hazards addressed and the number of people at risk; the extent to which gender-related issues and those of vul- nerable populations (hereafter described as “underserved populations”; see Box 1-1 for definition) are addressed; the extent to which the health and safety needs of small businesses are addressed; the “life stage” of problems being addressed; and the structure and content of the program. The criteria for relevance as defined in the Framework Document are as ap- plicable to the HHE Program as they are to traditional NIOSH research programs. The committee is to determine whether occupational health issues addressed by the HHE Program are the most serious in gravity or the most frequent among the spectrum of issues to which resources might be applied. It is also to determine whether the HHE Program addresses the needs of underserved populations and small businesses. HHEs are not the same as investigator-initiated research such as that sponsored by the National Institutes of Health. HHEs are considered part of the federal government’s public health surveillance activities, similar to outbreak investigations of infectious diseases in communities, and thus are exempt from the requirement of review by institutional review boards to which other forms of human subjects research are subject. For this reason, the committee considers the extent to which HHEs respond to the needs of the requesters to be an additional measure of relevance. In all its criteria, the Framework Document directs evaluation committees to consider how program research is relevant or has impact. This committee finds that substituting program activities for “program research” is an adequate modifica- tion in most cases, including within the Framework Committee’s scoring criteria. Box 1-2 provides the scoring criteria used by the committee in the determination of rankings for relevance and impact.  Public health surveillance is the “ongoing, systematic collection, analysis, interpretation, and dis- semination of data regarding a health-related event for use in public health action to reduce morbid- ity and mortality and to improve health. Data disseminated by a public health surveillance system can be used for immediate public health action, program planning and evaluation, and formulating research hypotheses” (MMWR, 2001b:2).  The life stage of an issue is determined by identifying the type of activities that should be under- taken at a given time to correct an issue. For example, as health effects are understood, efforts should shift from efficacy to intervention and intervention-effectiveness research. 

16 T h e H e a l t h H a z a r d E v a l u a t i o n P r o g r a m a t NIOSH BOX 1-1 Vulnerable or Underserved Populations Vulnerable populations are defined in the Framework Document (Appendix A) as groups of workers who have biologic, social, or economic characteristics that place them at increased risk for work-related conditions or on whom inadequate data have been col- lected. They include disadvantaged minorities, disabled persons, low-wage workers, and non-English speakers for whom language or other barriers present health or safety risks. Undocumented workers also fall within this category. This evaluation committee uses the term “underserved populations” when referring to vulnerable populations. THE EVALUATION PROCESS Information Gathering The committee reviewed material submitted by the HHE Program in the form of an “evidence package” that describes the HHE Program’s legislative founda- tions, organizational structure and management, resources, activities, and outputs (NIOSH, 2007b). Contributions to improvements in worker health are summa- rized in the form of narratives related to specific workplace issues. Results of pre- vious evaluations, and lists and examples of HHE Program reports are provided. HHE Program staff also responded during meetings and in writing to numerous questions from the committee and participated in a telephone conference with committee members to discuss HHE Program emergency response activities. Ap- pendix B is a list of materials provided by the HHE Program to assist the evalua- tion process. To deliberate its charge, the committee met four times. Two meetings were held in Washington, D.C. (September and December 2007), and two in Irvine, California (January and February 2008). The first three meetings included briefings from NIOSH and HHE Program staff and stakeholders. Meeting agendas, includ- ing names and affiliations of those who provided input, are given in Appendix C. Committee members discussed pertinent issues with several stakeholders on two panels representing the workplace and public sectors during the December 2007 meeting. In January 2008, the committee heard from stakeholders regarding four specific HHEs (NIOSH, 2004a, 2005b, 2005f, 2007i). Meetings were held on both the East and the West Coast to facilitate stakeholder participation and attendance. Subsets of the committee held numerous phone conferences and a meeting in Cincinnati, Ohio, to prepare the final report. 

Introduction 17 BOX 1-2 Scoring Criteria The following is the scoring criteria used by the committee in its evaluation of relevance and impact of the HHE Program. The criteria are based on those developed by the Framework Committee (see Appendix A), but have been slightly modified to accommodate the more applied nature of the HHE Program. Scoring Criteria for Relevance 5 =  ctivities are in high-priority subject areas and NIOSH is significantly engaged in ap- A propriate transfer activities for completed projects/reported results. 4 =  ctivities are in priority subject areas and NIOSH is engaged in appropriate transfer A activities for completed projects/reported results. 3 =  ctivities are in high-priority or priority subject areas, but NIOSH is not engaged in ap- A propriate transfer activities; or activities focus on lesser priorities but NIOSH is engaged in appropriate transfer activities. 2 =  ctivities are focused on lesser priorities and NIOSH is not engaged in or planning some A appropriate transfer activities. 1 =  ctivities are not focused on priorities and NIOSH is not engaged in transfer activities. A Scoring Criteria for Impact 5 =  ctivities have made major contribution(s) to worker health and safety on the basis of A end outcomes or well-accepted intermediate outcomes. 4 =  ctivities have made some contributions to end outcomes or well-accepted intermedi- A ate outcomes. 3 =  ctivities are ongoing and outputs are produced that are likely to result in improvements A in worker health and safety (with explanation of why not rated higher). Well-accepted outcomes have not been recorded. 2 =  ctivities are ongoing and outputs are produced that may result in new knowledge or A technology, but only limited application is expected. Well-accepted outcomes have not been recorded. 1 =  ctivities and outputs do not result in or are NOT likely to have any application. A NA =  mpact cannot be assessed; program not mature enough. I The committee found a general paucity of quantitative data showing the impact of HHE Program activities. For this reason, the committee made great ef- forts to hear from those who could reasonably be expected to benefit from HHE Program activities. The committee invited individuals representing small and large businesses; researchers in academe; local, state, and federal agencies; labor unions; and workers’ rights organizations and advocacy groups to participate in 

18 T h e H e a l t h H a z a r d E v a l u a t i o n P r o g r a m a t NIOSH discussions regarding the relevance and impact of HHE Program activities. All were asked to speak of their positive and negative experiences with the HHE Program and the usefulness and effectiveness of any resulting HHE recommendations. HHE Program stakeholders unable to travel to meetings were invited to participate in discussions via teleconference. The committee found it necessary to rely on this anecdotal evidence when no other evidence was available, but it is careful to provide citations in the report when doing so. To receive a broader range of stakeholder input, the committee also requested public input via an online questionnaire (see Appendix C). The questionnaire was announced via e-mail to more than 500 stakeholders. Responses could be submit- ted online, by e-mail, or by standard mail, and the option to respond anonymously was available. Questionnaire announcements were distributed to individuals sug- gested by the committee, as well as to those on lists provided by the HHE Program and generated by other committees in this evaluation series. The HHE Program provided links to the questionnaire on its Internet site, and the request for input was also announced through a variety of means including a NIOSH listserv for state health and labor departments that partner with NIOSH on occupational health surveillance; a listserv for occupational health practitioners run by the University of North Carolina; the American Industrial Hygiene Association weekly e-newsletter; and NIOSH eNews, a monthly newsletter (NIOSH eNews, 2008). The committee received 57 responses. A summary of stakeholder recommendations and identified emerging issues is provided in Appendix D (derived from Stakeholder Response Table, 2008). The committee found stakeholder insights thoughtful and invaluable during deliberations. Additional stakeholder input related to HHE Program emergency response activities was solicited (see Appendix C). Two responses were received as a result of that effort. Period Evaluated Because the HHE Program has evolved since its inception and because there have been multiple evaluations of this program in the past (see Chapter 2), the committee focuses on current program processes and activities. The committee chooses to emphasize the period between 1997 and 2007 in its evaluation; however many program activities before and after this time frame are considered. REPORT ORGANIZATION This report is organized into five chapters. This chapter orients the reader by outlining the committee review process. Chapter 2 provides a short history of 

Introduction 19 the HHE Program and its legislative authority and a description of the program’s inputs, activities, and outputs. The committee’s evaluation of the HHE Program can be found in Chapters 3 and 4, which are organized in a manner consistent with the program’s logic model (presented in Chapter 2). In Chapter 3, the com- mittee describes the relevance of the program’s strategic plan, its use of resources, program activities, outputs, and emergency response activities. The committee’s findings are organized and described under each of these categories. In Chapter 4, the committee evaluates the impact of the program. The chapter includes four sections corresponding to the four parts of the committee’s charge with respect to impact: (1) reduction in risk and prevention of occupational illness in investigated workplaces; (2) transfer of information beyond investigated worksites; (3) influ- ence on NIOSH research and policy; and (4) impact on others in the occupational health community. A fifth section discusses the impact of program emergency response activities. Finally, in Chapter 5, the committee describes its vision of what the HHE Program of the future would be and summarizes recommendations made in Chapters 3 and 4.