Summary

As part of a multifaceted look at the inherited and evolving portfolio of the National Personal Protective Technology Laboratory (NPPTL), the laboratory asked the National Academies to undertake a special look at the informational underpinnings of the NPPTL program to promote effective use of respirator equipment in the workplace. The primary focus of the committee inquiry was to be on a landmark survey conducted by the Bureau of Labor Statistics (BLS) and the National Institute for Occupational Safety and Health (NIOSH)—the 2001 Survey of Respirator Use and Practices (SRUP).

NIOSH had commissioned and collaborated with BLS to conduct a nationwide respirator survey in 2001. The purpose of this survey was to evaluate respirator use and practices in the workplace to help guide NIOSH respirator certification and research. The survey results were published in September 2003 in a report entitled “Respirator Usage in Private Sector Firms.”1 The survey findings suggest that there may be certain aspects of respiratory protection that are not compliant or are only partially compliant with Occupational Safety and Health Administration and Mine Safety and Health Administration regulations.

SCOPE OF THE STUDY

The purpose of this review has been to critique the survey and render judgment on the fitness and relevance of the survey methodology to provide valid information that would guide respirator protection decision making into the future. The National Academies formed the Committee on the Review of the NIOSH/BLS Respirator Use Survey Program to broadly represent the range of interests involved, with members drawn from industry, employee organizations, and academe. The committee members were selected on the basis of their expertise in occupational health and safety, industrial hygiene, respirator and filter technology, survey design and methodology, and statistical data analysis. This review is one part of a larger, more extensive, NPPTL-sponsored review by the National Academies of several scientific and technical issues relevant to the development, certification, deployment, and use of personal protective equipment (PPE),2 standards, and related systems to ensure workplace safety and health.

This review addresses the following issues:

  • The adequacy and appropriateness of the survey instrument, considering both the content and the format of the instrument;

  • The adequacy and appropriateness of the survey methodology, including the choice of sample, the sampling method, survey follow-up, and ultimate response rate;

  • The methods of estimating the resultant survey data and the adequacy of the data to address policy concerns with respirator usage;

  • The extent and adequacy of data analysis and publication;

  • The appropriateness of conclusions reached from the data;

  • The possibility of extending the utility of the data through additional statistical analysis; and

1

This report can be found on the world wide web in its entirety at http://www.cdc.gov/niosh/docs/respsurv/pdfs/respsurv2001.pdf.

2

PPE encompasses both protective ensembles (garments, boots, gloves, hoods, and respiratory protection) and operational equipment (equipment needed to sustain operations and provide general support during chemical, biological, radiological, and nuclear response operations). See the following web site for more information: http://saver.tamu.edu/assessments.php?s=2&c=1.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Measuring Respirator Use in the Workplace Summary As part of a multifaceted look at the inherited and evolving portfolio of the National Personal Protective Technology Laboratory (NPPTL), the laboratory asked the National Academies to undertake a special look at the informational underpinnings of the NPPTL program to promote effective use of respirator equipment in the workplace. The primary focus of the committee inquiry was to be on a landmark survey conducted by the Bureau of Labor Statistics (BLS) and the National Institute for Occupational Safety and Health (NIOSH)—the 2001 Survey of Respirator Use and Practices (SRUP). NIOSH had commissioned and collaborated with BLS to conduct a nationwide respirator survey in 2001. The purpose of this survey was to evaluate respirator use and practices in the workplace to help guide NIOSH respirator certification and research. The survey results were published in September 2003 in a report entitled “Respirator Usage in Private Sector Firms.”1 The survey findings suggest that there may be certain aspects of respiratory protection that are not compliant or are only partially compliant with Occupational Safety and Health Administration and Mine Safety and Health Administration regulations. SCOPE OF THE STUDY The purpose of this review has been to critique the survey and render judgment on the fitness and relevance of the survey methodology to provide valid information that would guide respirator protection decision making into the future. The National Academies formed the Committee on the Review of the NIOSH/BLS Respirator Use Survey Program to broadly represent the range of interests involved, with members drawn from industry, employee organizations, and academe. The committee members were selected on the basis of their expertise in occupational health and safety, industrial hygiene, respirator and filter technology, survey design and methodology, and statistical data analysis. This review is one part of a larger, more extensive, NPPTL-sponsored review by the National Academies of several scientific and technical issues relevant to the development, certification, deployment, and use of personal protective equipment (PPE),2 standards, and related systems to ensure workplace safety and health. This review addresses the following issues: The adequacy and appropriateness of the survey instrument, considering both the content and the format of the instrument; The adequacy and appropriateness of the survey methodology, including the choice of sample, the sampling method, survey follow-up, and ultimate response rate; The methods of estimating the resultant survey data and the adequacy of the data to address policy concerns with respirator usage; The extent and adequacy of data analysis and publication; The appropriateness of conclusions reached from the data; The possibility of extending the utility of the data through additional statistical analysis; and 1 This report can be found on the world wide web in its entirety at http://www.cdc.gov/niosh/docs/respsurv/pdfs/respsurv2001.pdf. 2 PPE encompasses both protective ensembles (garments, boots, gloves, hoods, and respiratory protection) and operational equipment (equipment needed to sustain operations and provide general support during chemical, biological, radiological, and nuclear response operations). See the following web site for more information: http://saver.tamu.edu/assessments.php?s=2&c=1.

OCR for page 1
Measuring Respirator Use in the Workplace The potential for obtaining additional information that is useful to NIOSH from current and future survey results. Each of these issues was considered by the committee to constitute a task to be accomplished during its review. To the extent that information was available to it, the committee assessed each of these issues and made recommendations, when appropriate. OBJECTIVE OF THE SURVEY The objective of SRUP was “to provide information to develop educational interventions for specific populations and to increase the frequency and effectiveness of respirator use in the workplace.”3 The survey was designed to provide estimates of the number of establishments and employees who used respirators in a recent 12-month period by type of respirator and type of use, and to collect data on the characteristics of the respirator program at the establishment; medical fitness to wear respirators; characteristics of respirator training at the establishment; usefulness of NIOSH approval labels and respirator manufacturers’ instructions; substances protected against by the use of respirators; and fit-testing methods used for respirators. The target population of the survey was private-sector establishments with employment covered by unemployment insurance programs that were included in the 1999 Survey of Occupational Injuries and Illnesses (SOII). The data to illuminate these survey objectives had never been systematically collected before from such a large number of establishments covering so many industries and size classes. FINDINGS The 2001 SRUP was critiqued with the thought that the analysis and recommendations would help guide a subsequent survey of this scope and nature and that the critique would be based on published documentation provided by BLS and NIOSH. This was, in many ways, a landmark survey for both agencies. Although it built on the extensive sampling capacity of BLS that came with years of experience in conducting the SOII, it was experimental in its selection of questions and sample units. Finding 1: The survey was an important first step in collecting respiratory protection data from a probability sample. As such, it was a worthwhile learning experience for both NIOSH and BLS. It soon became obvious to BLS management that the survey suffered from inadequate funding for its scope and size. One casualty of the lack of funding was the inability of BLS to follow up with a full set of documentation. Finding 2: There was insufficient documentation and detail for the committee to reconstruct key aspects of the methodology and to fully understand the survey design and implementation. The lack of documentation was particularly true for the sample design, sample weighting, content development, and handling of missing data through a type of imputation procedure known as the “hotdeck,” in which missing lines of data are replaced by sampled complete data records. Appropriate documentation and access to this documentation are essential to evaluating and reproducing a survey of this type. Development of the questionnaire was a joint responsibility between NIOSH and BLS. NIOSH participated in the development of the survey questionnaire by providing BLS with direction on technical subjects such as regulations, respirator types and uses, and specific substances that require respirator use. Although the survey was appropriately subjected to cognitive experimentation and field testing, the resulting questions tended to be focused more on items that were measurable from the perspective of the employer respondents, and the questions tended to elicit information on regulatory compliance rather than respirator certification and use. Finding 3: The survey questionnaire was not adequately related to the initial survey objectives. The committee found that it was difficult to evaluate the adequacy of pretesting because, in general, the documentation about the details of the testing, the resulting instrument revisions, and the efficacy of those revisions were inadequate. Although the testing appears, overall, to have uncovered a large number of problems, it is difficult to determine the effectiveness of the solutions without explicit examples or results of their retesting. The field test provided valuable insights that enhanced the survey operations. For example, the overall field-test response rate of 80 percent was fairly close to the reported survey response rate of 75.5 percent. However there were many issues with the pretesting, including the following: Finding 4: The field test paid little attention to exploring validation procedures that might have provided information on the quality of data collected or motivated the need for a formal quality assessment of the data, and thus missed an opportunity to improve understanding of the quality of the SRUP data. Finding 5: Many features of the survey were not user friendly or optimally designed to aid navigation. 3 Bureau of Labor Statistics and National Institute for Occupational Safety and Health; Respirator Usage in Private Sector Firms, 2001; Washington, D.C., 2003, p. 1.

OCR for page 1
Measuring Respirator Use in the Workplace Finding 6: There were several material weaknesses in the procedures for instrument testing. The committee observes that the SRUP was, in essence, a sample of a sample. The population sampled for the SRUP is technically the subset of the SOII target population defined by industry types. This was a potential strength of the survey. However, the opportunity was largely missed. Finding 7: NIOSH did not set specific precision objectives for key estimates of population subgroups from the SRUP. It is customary to establish reliability objectives for key data elements in the design of a survey. Reliability objectives are sensitive to sample size and survey operations, and, ultimately, to the cost of the survey in terms of resources and respondent burden. In the absence of such objectives it is difficult to assess the adequacy of the sample sizes for the various populations subgroups (e.g., by region, by type of business). Several aspects of the sample design resulted in less than optimal estimation in practice. Finding 8: The choice of stratification variables for the SRUP sample design appears to have been appropriate, since many of the survey’s reported findings by type and size of industry were quite different (e.g., respirator use). The rationale for using the allocation for the SRUP sub-sample among strata was to maximize the yield of companies with higher rates of respirator usage. This allocation, which made each stratum sample proportional in size to the expected number of establishments in the stratum that use respirators, may not have been optimal to improve the precision either of overall survey estimates or of estimates for population subgroups defined by the strata, but it did serve to increase the proportion of sample companies that reported respirator use. Finding 9: More could have been done with the characteristic of the SRUP being a subset of the SOII to build strength into the estimates in that a sample that is the second phase of a “two-phase” or “double” sample can gain power from the first-phase sample. The first-phase sample can improve stratification or estimation in the second phase if utilized appropriately. To do so, the strata should be defined consistently (if not identically) in both phases of sampling to permit analysis of the SRUP sample as a two-phase sample. Although the documentation is not clear, there may not have been attention to the necessary consistency of stratification between the SOII (first-phase sampling) and the selection of industries for the SRUP (second phase). This oversight limits the ability of the two-phase sample to be analyzed as such, with sample weights accounting for sampling and nonresponse in both phases of sampling and the stratification in the two phases accounting for determining the precision of survey estimates. Finding 10: The SRUP used a basic collection design that is fairly typical of many establishment mail surveys conducted by federal agencies. However, the SRUP data collection design did not use several state-of-the-art techniques that would likely have produced a higher response rate and enhanced data quality. Several of these techniques to be considered for use if there is another SRUP-type survey in the future include the following: Identifying the best-qualified survey respondent in advance of the questionnaire survey mailing; Sending a pre-notice letter in advance of the questionnaire mailings and thank-you or reminder postcards a week after the questionnaire mailings; Personalizing all mailings; Following the guidelines in Dillman4 for the format and content of the mailings; Using real stamps instead of “business reply” postage on the return envelopes included in the two questionnaire mailings; Making refusal conversion attempts on persons who refuse to participate during the nonrespondent follow-up calls; and Implementing measures to evaluate the quality of the data, such as assessments of response and nonresponse bias. Finding 11: The reported overall SRUP response rate of 75.5 percent paints an incomplete picture of the impact of nonresponse on all key SRUP findings. In addition to findings on the percentage of respirator use among all companies in the sample, the survey findings focused heavily on companies with required respirator use, and the response rate among these companies is not reported. Because of the setup of the survey questionnaire, the respondents who identified themselves as being in the group of establishments that had mandatory respirator usage were obliged to answer many more questions than those who only had voluntary use of respirators. There was a penalty for reporting mandatory use, so it is quite likely that companies having mandatory use had a higher rate of nonresponse. This would lead to a corresponding increase in the potential for nonresponse bias. Moreover, the net effect of nonresponse in a two-phase sample is the product of the response rates in the two phases, instead of just the response rate in the second 4 Dillman, D.A. Mail and Internet Surveys: The Tailored Design Method, New York: John Wiley & Sons, 2000.

OCR for page 1
Measuring Respirator Use in the Workplace phase corresponding here to the response outcome for the SRUP subsample. Finding 12: The performance of extensive machine editing and error resolution follow-up contacts showed commendable attention to resolving inconsistent and invalid responses. In keeping with its reputation for careful, professional work, BLS (and its contractor) did many things in processing the survey data in exemplary fashion. The care taken in resolving issues was particularly commendable. However, no assessment was made of response and nonresponse bias. Finding 13: Although a standard error was calculated for each estimate from this survey, these measures of error were not computed correctly. BLS used SAS Version 6.0 to produce the computations, which yielded appropriately weighted estimates but did not account for stratification in estimating the standard error of estimates.5 Also of concern was the finding that standard errors were not published along with the data in the main report. Sophisticated data users would be able to obtain the standard error estimates for every data item by requesting them from BLS, but it would have been preferable to have included them with the published tables in the report. Finding 14: A large number of analytical studies were conducted by NIOSH staff following the release of the initial SRUP report. However, more could have been done to disseminate survey findings and data to key stakeholders such as users, policy and decision makers, and the industrial hygiene and safety engineering communities. For the most part, NIOSH dissemination activities were concentrated on providing useful information to the professional PPE community. A more active outreach program to employer groups and employee organizations would have better popularized these findings. Finding 15: The failure to conduct the sample matching earlier in the process constituted a missed opportunity to improve the quality and richness of the data. The initiative to enrich the data with potentially useful analytical data from the SOII should be applauded, but it came very late in the process. Although the inability to control for differences in the data related to different time periods was recognized, different conditions in the reporting establishments rendered the results of the match somewhat speculative. This sample unit matching was conducted after the processing of survey data was concluded and primarily for the purpose of data analysis. It is unfortunate that the sample matching did not come earlier as a planned feature in the processing cycle. RECOMMENDATIONS Despite these findings about the adequacy of the survey operation and concerns about the missed opportunities to fully realize the potential of this survey, the committee applauds the agencies for undertaking this pioneering data collection in order to improve understanding of respirator use in industry. RECOMMENDATION 1: NPPTL should continue to address and explicitly articulate data needs to evaluate and improve the respirator certification program so as to ensure the efficient availability and advancement of protective technologies for employees. The committee is encouraged that the NPPTL leadership continues to place appropriate emphasis on its statutory mission of respirator certification, even as it enriches its program with attention to other objectives. It is proper to maintain this emphasis even as NPPTL moves forward in other areas. RECOMMENDATION 2: Discussion and explicit articulation of information needs related to PPE performance and utilization should be the subject of continuous review and periodic updating by NPPTL as PPE technology evolves and the method of meeting those needs changes. An intensive effort of this type may be impractical to sustain on an ongoing basis, but it should be a necessary function of NPPTL’s surveillance efforts. Imposing a top-to-bottom consideration of information needs in conjunction with program objectives is a hallmark of performance-based management. This is consistent with the NPPTL value creation system and government-wide performance management initiative. RECOMMENDATION 3: NPPTL’s future data-gathering activities should seek, within a sound scientific approach, to derive explanations for observations on the use of PPE in the field. Beyond meeting current and emerging program objectives, however, increasing recognition of the research role of NPPTL should be embedded in the future approach to the task. NIOSH is, in essence, a research organization and should give a sharper research focus to its data-gathering activities. RECOMMENDATION 4: In the future, the resource-intensive data-gathering framework for NPPTL surveillance efforts should focus on the evolving mission categories of surveillance, certification, research, technology, and standards. 5 Later versions of SAS account for stratification, but these versions may not have been available at the time of the SRUP analysis.

OCR for page 1
Measuring Respirator Use in the Workplace The committee provides an example of such a framework in this report for consideration by NPPTL leadership. The framework is tied to NPPTL mission categories. RECOMMENDATION 5: NPPTL should develop for implementation an ongoing survey of employer and employees whose overarching goal is to obtain needed information on the use of respirators and other PPE in the United States. To ensure the scientific quality and broad utility of this series of surveys, NPPTL should ensure that this system of periodic surveys: Is sufficiently flexible to meet NPPTL’s changing information needs; and Attains the highest standards of current survey research by: Following a responsive sample design, Assuring the reporting accuracy of all of its key survey measures, Widely disseminating its findings, Making its data easily available to outside researchers, and Documenting all of its methods in detail. The major recommendation arising from this review is based on the conclusion that data on respirator use are best provided by employers and employees in the context of the work setting. This suggests the need for a very different approach to conducting any future survey operation. After considering several options, the committee developed a proposal for an employee-within-establishment survey. The recommendation to conduct an employee-within-establishment survey should not be interpreted as suggesting this as an exclusive approach to all future data collection. To the extent that qualitative indicators are needed, focus groups might suffice. It may also continue to be useful to collect information about aspects of establishment respirator program management from employers, so it would be useful to continue to conduct focused employer-only surveys periodically. RECOMMENDATION 6: The most efficient data collection approach for the establishment survey is a mixed-mode design involving three phases: (1) an initial round of telephone screening calls, (2) a mail survey approach, and (3) telephone follow-up calls to nonrespondents. RECOMMENDATION 7: Instead of addressing the survey to someone with unknown expertise in the area of interest, surveillance personnel at NPPTL should conduct a quick telephone screening of sampled establishments in advance of the mail survey to identify the best-qualified respondent and to learn from that person whether the establishment is eligible for the full survey (that is, required use of respirators in the past 12 months). Telephone screening calls have the following objectives: (1) to identify and remove from the sample those establishments that are no longer in business, (2) to identify establishments that are PPE users, (3) to identify and close out as “completed” non-PPE user establishments, and (4) to make contact with the best-qualified respondent at PPE user establishments. The calls should be made by well-trained telephone interviewers in the survey contractor’s centralized call center. The interviewer should ask to speak to the person most knowledgeable about possible PPE use at the establishment.

OCR for page 1
Measuring Respirator Use in the Workplace This page intentionally left blank.