Questionnaire Development

The translation of survey objectives into the design of a questionnaire to obtain the required data was a dual responsibility of NIOSH and BLS. NIOSH participated in the development of the survey questionnaire by providing BLS with direction on technical subjects such as regulations, respirator types and uses, and specific substances that require respirator use. In a departure from the usual BLS arrangements, NIOSH actually developed the initial draft of questions for the survey.5


Finding 3: The survey questionnaire was not adequately related to the initial survey objectives.


The questions that were developed and asked in the survey did not relate to the original objectives for performing the survey. The survey became focused more on items that were measurable from the perspective of the employer respondents, and the questions elicited information focused more on regulatory compliance than on respirator use.

The BLS somewhat modified the NIOSH version of the questionnaire prior to the pretests. The BLS modifications were developed using Dillman’s Total Design Methods,6 as well as other methods emphasizing formal design principles. The pretest version of the survey used during the cognitive interviews looked vastly different from the initial version, although it retained much of the language, item structure, and content. The stated BLS goal was to test the “semantic and linguistic features” during pretesting and obtain respondent reaction to the newly redesigned respirator survey form.

Assessment of instrument testing for this establishment survey was guided by a number of general principles. Unfortunately, there is no consensus set of accepted pretesting best practices. While a number of common procedures are widely used in establishment surveys—including cognitive testing, usability testing, and various types of field tests—there is no generally accepted procedure or combination of procedures for establishment survey instrument testing.7

Pretesting is a required step in the preparation of BLS questionnaires. In the absence of a commonly accepted template, the SRUP pretesting program involved the integration of multiple methods, including expert review, cognitive interviewing, and field testing.

Cognitive Testing

A total of 12 cognitive interviews were conducted with 11 establishments during two rounds of pretesting. Nine establishments participated in the first round of interviews, which focused on the development of a satisfactory field-test instrument. The final three interviews were completed during a second round of testing after the field-test results were analyzed.8

The 11 companies that participated in the cognitive interviews were divided by BLS into three groups:9

  1. Five large companies that reported having heavy usage of respirators among their employees: these large companies included a steel firm, a utilities company, a ship repair company, a construction firm, and a paint producing and distribution company (this firm participated in two separate interviews). All five sites had one or two safety coordinators or managers and/or industrial hygienist(s) at the company who participated in the interview.

  2. Four midsize companies, two of which reported minimal or occasional respirator use and one that reported moderate respirator usage: these midsize companies included a sheet metal production company, two construction companies, and a scaffolding company. All four sites had a safety coordinator or manager and/or industrial hygienist at the company who participated in the interview. Three respondents participated in the cognitive interview with one of the construction firms.

  3. Two small companies, one of which reported minimal use of respirators and a second that reported a significant amount of respirator usage: one of those companies was a scrap metal company, and the other was an auto body repair and paint shop. Neither company had a safety coordinator, manager, or industrial hygienist at the company; as a result the company owner or office manager completed the interview.

5

Fisher, Sylvia Kay, Kelley Frampton, and Ramona Tran, Pretesting the Survey of Respirator Uses and Practices (SRUP): Cognitive and field testing of a new establishment survey, Proceedings of the Annual Meeting of the American Statistical Association, August 5-9, 2001.

6

Dillman, D.A., Mail and Internet Surveys: The Tailored Design Method, New York: John Wiley & Sons, 2000.

7

For an example of a comprehensive development and testing program for an establishment survey, see the following: Goldenberg, K.L., A.E. Anderson, D.K. Willimack, S.R. Freedman, R.H. Rutchik, and Z.L.M. Moy, Experiences implementing establishment survey questionnaire development and testing at selected U.S. government agencies, presented at the International Conference on Questionnaire Development, Evaluation and Testing Methods, Charleston, S.C., 2002; Willimack, D.K., L. Lyberg, J. Martin, L. Japec, and P. Whitridge, Evolution and adaptation of questionnaire development, evaluation and testing methods for establishment surveys, in Presser et al., eds., Methods for Testing and Evaluating Survey Questionnaires, New York: John Wiley & Sons, 2004.

8

Fisher, Sylvia Kay, Kelley Frampton, and Ramona Tran, Pretesting the Survey of Respirator Uses and Practices (SRUP): Cognitive and field testing of a new establishment survey, Proceedings of the Annual Meeting of the American Statistical Association, August 5-9, 2001.

9

The establishments selected for cognitive interviews were primarily large- and medium-sized firms in which interviews were generally with “safety experts with extensive training in industrial hygiene and safety issues….” This may have led to some underestimation of the difficulties that actual survey respondents might be expected to experience with the instrument.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement