4
Development and Evaluation of Materials
The first question regarding the development of new materials is how to prioritize which gaps to fill. The committee uses the term “materials” in this report to mean not only written information, but also communications methods such as audiocassettes, videocassettes, workshops, meetings, comic books, television and radio programming. In general, workshop participants were of the opinion that priority should not be on translation of technical health and safety documents. Rather, emphasis would be better placed on developing materials as part of a strategic initiative to reach Spanish-speaking workers, their employers, and their communities with practical information that can assist in injury and illness prevention in the workplace. Materials development in Spanish is important but only in the context of a larger outreach, education, and action strategy. A thought repeatedly expressed during the workshop was: “Good Spanish-language materials are necessary but not sufficient.”
A more comprehensive approach requires that those who develop new, effective materials be able to plan a comprehensive education plan that takes into account the organizational, societal, economic, and other factors that define the environment in which workplace change must take place. Ideally, materials development would be in the context of a plan that considers the significant barriers immigrant workers may face in taking action for workplace safety. These include fear based on immigration status, linguistic and language barriers, and different work experiences in their countries of origin. An effective approach also requires that we draw on cultural strengths, such as strong community institutions and health beliefs.
Much is known about cultural appropriateness and cultural competence in other aspects of public health. In the opinion of the workshop participants, the occupational safety and health community needs to become more familiar with the literature and experiences in such fields as tobacco control, human immunodeficiency virus and acquired immune deficiency syndrome education, and other significant public health campaigns that have addressed cultural and linguistic issues successfully. Partnerships with community and other public health professionals with health education and communications expertise would be useful.
Finally, workshop participants noted that it would be beneficial if materials development in Spanish drew on sound principles of effective adult education, social marketing, and public health education. Some of the key precepts that are well established include:
-
involvement of the target population in developing the educational message and outreach plan;
-
pilot testing of materials with the target population to assess, for example, appropriate messages, literacy level and readability, terminology, and graphics;
-
use of workers’ own narratives and images in materials;
-
development of action-oriented (problem-solving), participatory, and hands-on educational materials;
-
addressing of different learning styles by using a variety of methods;
-
use of multiple means of communication and education (e.g., radio, face-to-face, comic books); and
-
design materials that allow them to be used with some flexibility in a variety of instructional and learning situations.
When existing materials are being translated into Spanish, many of these same principles will apply. Workshop participants agreed that materials should be adapted for use by a Spanish-speaking audience, rather than simply translated literally. There was substantial discussion in the workshop about the preference for materials developed originally in Spanish (or in Spanish and English) rather than translated. However, because this is not always feasible, the importance of high-quality translation was highlighted. In particular, translations should be performed by native Spanish speakers who are qualified as translators and not just by anyone who speaks Spanish; translations should then be reviewed by native Spanish speakers and pilot tested with the intended audience. When translating materials, the education level of the audience should be taken into account. Unfortunately, there are far too many examples of embarrassingly bad translations, according to the workshop participants.
Approaches to materials development are most effective if grounded in broader intervention efforts that are based on sound principles of program planning. These principles include targeting interventions based on the best available surveillance data; thoroughly assessing needs; involving stakeholders in the process; establishing clear, measurable objectives; designing a participatory education program; pilot testing with the target population; evaluating program effectiveness; and revising efforts based on findings. Workshop participants thought it is essential that leadership and staff in organizations that fund materials development and testing understand the resource needs. The CDC Guide to Community Based Public Health Practice includes methodological guidelines for designing and funding sound evidence-based intervention research.3
Once an educational intervention is well tested, a wide-ranging dissemination program will be needed. It is essential to customize dissemination program to each audience and situation. Given the seriousness of the situation, society cannot afford to continually “reinvent the wheel.” Therefore, it is important that results be shared so that resources are maximized.
Workshop participants discussed the desire for a concerted effort to promote evaluation of Spanish-language materials. Evaluation should be built into the initial planning stages of materials development and educational interventions of all kinds. Materials development is greatly enhanced when target populations, objectives, and methods for measuring accomplishments are defined. However, during the push to develop and use materials, the evaluation phase does not always occur, because all too often these development projects lack evaluation resources.
Workshop participants thought that it is important to have a solid evaluation process of multiple phases, including (1) pilot testing and focus groups with intended audiences during the development phase; (2) feedback from end users (such as trainees) to determine comprehension of the materials; and (3) evaluation to determine which practices work best. In addition, there was discussion of how to review and rate existing materials in Spanish. One suggestion was that an organization should create a comprehensive clearinghouse of safety and health material on the Internet as a single source for this information in Spanish. Others responded that Internet sites might be burdensome and difficult to navigate, and the content quality might be questionable, depending on the skill and background of the person or group authoring a Web site. In addition to the task of developing acceptable criteria, the task of going through all the material may become overwhelming. As discussed in a later section, Hispanics still have limited access to the Internet.
There are, however, some positive examples of information compilation sites. The National Institute of Environmental Health Sciences clearinghouse on hazardous waste materials
management4 and the Building Trades Master Trainers forum5 are sites with no official evaluation of materials. On the other hand, the National Agricultural Safety Database6 and Electronic Library for Construction Occupational Safety and Health7 have advisory boards to review content.
Workshop participants agreed that rating criteria are important. Possible rating criteria discussed include:
-
whether materials for Hispanic workers are presented in a manner that is “culturally and linguistically competent,” comfortable to the worker, and not offensive;
-
whether materials are grammatically correct but in the vernacular of the workers, even when it may be a mix of Spanish and English (these variations can be identified as such);
-
whether materials are created for the appropriate literacy levels of the workers;
-
whether the information provided is technically sound; and
-
whether the subject matter is compatible with audience priorities and needs.
Workshop participants agreed that it is essential that a solid materials evaluation process be implemented so that materials can be judged as efficacious before being introduced to the workforce. Likewise, a program of continual evaluation is important to assure that the target population understands the materials and the materials continue to improve. Agreement on a common set of evaluation criteria would be useful both for the development of new materials and for distribution of existing materials through some type of clearinghouse.
4 |
See <http://www.niehs.nih.gov>. Date accessed November 5, 2002. |
5 |
See <http://www.buildingtrades.org>. Date accessed November 5, 2002. |
6 |
See <http://www.cdc.gov/niosh/nasd/nasdhome.html>. Date accessed November 5, 2002. |
7 |
See <http://www.cdc.gov/niosh/elcosh>. Date accessed November 5, 2002. |