Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
EAncillary, Emerging, arid Related Projects The second type of question raised by CDC program personnel following the release of the first edition of the panel's report involved ancillary, emerging, or related projects of the National AIDS Infonnation and Ed- ucation Program (NATEP) and the Center for Prevention Services (CPS). These projects include: (1) evaluating matenals distributed through the National AIDS ~fonnation Clealinghouse, (2) evaluating referral ser- vices for individuals testing REV antibody positive in CPS's nationwide counseling and testing program, and (3) monitoring counseling and test- ing services provided in private (non-CDC funded) sites. (See Appendix D for a discussion of technical matters relating to some of the report's suggestions.) Evaluating Clearinghouse Materials In Chapter 3, the panel focused on evaluation strategies for the most prominent intervention of the NAIEP the multiphase America Responds to AIDS campaign of public service announcements (PSAs). The panel noted that CAMP, which oversees the media campaign, also has funded several other media projects, such as the mass mailing of AIDS in- formational brochures to households nationwide and the national AIDS hotline. An important NAlEP project not mentioned in Chapter 3 is the National AIDS Formation Clearinghouse (NAIC), which has been sponsored since October 1987. The pane] considered a simple strategy for evaluating the Clearinghouse materials. Background and Objectives NAlC acts as a centralized resource for information about AIDS-related organizations and educational materials. Its three main services include: 335
336 ~ APPENDIX E (1) two online information data basesone for resources and one for educational materialsthat are each supplemented with technical and reference assistance by staff specialists; (2) outreach services, in which staff members provide conference support and program building; and (3) direct distnbution of selected educational matenals. According to its March 1989 User Guide, NAIC "evaluates its ser- vices and resources on an ongoing basis" and assists in "the development and assessment of resources." The "evaluation" and "assessment" ac- tivities refer not to the impact of services and resources but rather to their quality (quality being measured subjectively as well as objectively). Much of the assessment is done by internal staff, although external re- viewers may sometimes be used. For example, projects and organizations are entered into the Resources Database only after they "meet NAIC crite- na" and undergo "a rigorous process of internal validation." In addition, national organizations and state HIV coordinators are sometimes asked to provide a measure of external review. In-house staff also review publications, software, and audiovisuals before they are entered into the Educational Materials Database; like resources, educational materials are "indexed according to stringent guidelines" and undergo "a thorough quality control review" (User Guide, March 1989~. Finally, NAIC has an internal review board that recommends the matenals for direct dism- bution by the Cleannghouse, although CDC makes the final decision on these matenals. The internal board bases its recommendations on judg- ments about the matenals' accuracy, appropriateness to target audience, and currency. CDC has proposed that, in addition to the internal review board, NAIC form an external review committee to review publications listed in the Educational Matenals Database (except certain publications such as meeting proceedings and d~rectones). The external board would Took at materials on a regularly scheduled basis to assure their currency and to identify gaps in needed matenals. Because materials can become outdated, periodic reviews are needed to ensure currency. Similarly valuable is the need for NAIC to ensure quality control in the validation process and to assess the accuracy and appropriateness of educational matenals. Although these activities fre- quently entail subjective judgment, they contribute to an assessment of how well services are delivered. They do not, however, answer the questions "Does it work?" or "What works better?" Does It Work? As discussed in Chapter 1, the panel believes that the most efficient way to answer "Does it work?" is with a randomized field experiment that
OTHER PROJECTS ~ 337 compares a group that receives an intervention with a control group that does not receive it. In Chapter 4, the pane} discusses conditions under which having a no-treatment control is appropriate: scarce resources with which to provide a service; interventions that are of unproven value; and availability of related services elsewhere. (This subject is further elaborated In Appendix D.) In the case of the Clearinghouse services and resources, scarcity is not an issue, and availability of alternative services is questionable; therefore, the pane] believes that having a no-treatment control group is inappropriate. What can be answered, however, is "What works better?" What Works Better? This question is appropriate for evaluating those materials in the Clear- ~nghouse that NATC recommends or intends to recommend to CDC for direct distnbution. To answer the question of "What works better?," the pane] recommends a small-scale strategy similar to the copy testing that was proposed for a formative evaluation of the media campaign. A simple experiment can be set up to randomly assign groups to receive different treatments and then compare the results. To contain costs and personnel resources, focus groups or convenience samples of certain risk groups can be randomly assigned to receive different informational brochures about a given subject matter (e.g., needle use). Self-administered ques- tionna~res of knowledge, attitudes, and beliefs can be designed to center On particular subject matters and on general AIDS information; results can then be analyzed for cognitive differences between groups. (A more elaborate experiment would follow up the groups at a fixed later date (2 weeks, 4 weeks) to analyze differences in self-reported behavior.) Several comparative tests can be made. One test might compare the effect of different levels of readability on cognition for example, brochures written at the 7th and the lOth grade levels.2 Another test might compare materials containing different amounts of information. For ex- ample, one brochure might discuss risks and risk reduction involving needle use, while another provides the same information along with ad- ditional information related only peripherally to the central theme, such as morbidity and moronity data. Another type of test might compare the effects of bilingual publications and English-only materials. The pane] also considered Me possibility of comparing NAIC out- reach services (conference planning and program building). Ultimately 1 Convenience sampling is discussed in Appendix D. 2Hochhauser (1987) analyzed 16 AIDS prevention brochures written for He general population and found that, on average, they were written at a 14th grade level (second year of college). This level may be too high to foster a good understanding of recommended material.
338 ~ APPENDIX E we decided Mat He measurement of success is tenuous (e.g., attendance at a conference may be excellent, but the conference may be poor) and cannot be well controlled (e.g., weather can affect attendance at different sites). For these reasons, the pane] believes that evaluation efforts should be confined to materials recommended for direct distribution by NAlC. Evaluating Referral Services of the Counseling anti Testing Program In Chapter 5, the pane} noted that "the purpose of the counseling and testing program has evolved and may still be evolving," in part because "advances in He treatment of asymptomatic HIV-infected individuals may increase the demand for testing and counseling services." Shortly after He report was issued, representatives from the CPS stated that the program's purpose indeed is shifting. Just as it had once evolved from screening the nation's blood supply to disease prevention, it is now moving to the provision of follow-up referral services for seropositive ~ndividuals.3 This shift raises two questions: "What is the quality of follow-up referral services?" and "Are these services available?" These concerns faD within the context of the original question posed in Chapter 5, "How well are services delivered?" Out of the five aspects of service delivery that were identified to gather answers to this question, two are relevant to the new questions: the adequacy of the referrals that are actually provided and the accessibility of referral services. The pane} recognizes Hat both adequacy and accessibility are subjective terms that cad for judgment on the part of the evaluator. At the same time, the pane! also believes that some standards can be agreed upon as essential, so that an evaluation can deterrence if those standards have been met. In Chapter 5, the pane! described "adequacy" as correspondence with client needs, such as the need for confidentiality. The needs of seropos- itive clients were briefly discussed and were said to include information and counseling about the medical and psychological management of in- fection and pamer notification. To these requirements, the pane! now adds the need for information about financial management (given the enormous cost of medical treatment for HIV disease), legal rights, con- traception, and drug treatment, when applicable. Depending on individual circumstances, such information might include facts about insurance, re- ferrals to Medicaid, the welfare system, gynecological services, drug treatment facilities, and, if discrimination is an issue, legal aid. As a useful first step to developing standards by which to judge the quality of Bindweed, provision of case management services is likely to be another important component in the continuity and success of the counseling and testing program's essential prevention messages.
OTHER PROJECTS ~ 339 referrals, the panel suggests that CDC identify and draw up a checklist of appropriate referral services, just as the agency has identified and prescnbed checklists for pretest and pastiest counseling. Judgment about accessibility is also somewhat subjective, but a basis can be constructed that will help answer the question "How well are services delivered?" To evaluate the accessibility of counseling and testing services, the pane} recommended four sources of information. Three of the sources (client surveys, site inventories, case studies) are useful for determining accessibility to referral services as well; a fourth (population surveys) could be useful, but the return on investment is not likely to be as high as the others. . Client Surveys. Surveys of He clients of HIV testing sites can reveal whether clients returned for referral services, whether such services were offered, and what the referral needs were. Client surveys can elicit information about referrals made for specific populations as well as about the scope and relevance of the referrals. In fact, a client survey at a pilot stage is warranted to help establish client needs arid to identify the types of referrals that should anDear on a checklist of potential services. · Site Service Inventories. Information about the accessibil- ity of referral services can be gathered in the site service inventories recommended in Chapter 5. These inventories can monitor the services that sites provide to persons testing positive for ~V. They can also demonstrate whether clients returned for referrals and whether and what referrals were made, using the checklist recommended above. The inven- tones can be analyzed according to site ties, risk factors, local vanations, and so on. If socioeconomic status were also collected (see footnote 4, Chapter 5), referral needs might be further illuminated. · Case St~ies. The pane] recommends using independent observers of clinician-client interactions to gather informa- tion about the accessibility of referral services. The pane] does not recommend the use of "professional customers," as we did for the direct observation of counseling and testing services. The distinction is made because the relevant pop- ulation of referees (seropositive individuals) would require "professional seropositives" to evaluate the referral services. Although adding the REV rates of a handful of professional customers should not overly disturb a facility's surveillance
340 ~ APPENDIX E counts, the introduction of only seropositive individuals as professional customers could inflate the counts. The panel notes that CDC is currently fending a few con- fidential test sites to ask case managers to follow up on the services offered seropositive individuals. Such case studies ought to be important sources of information not only about HIV medical and legal referrals but also about indirectly related services, such as drug treatment or contraceptive services, that may be eclipsed by HIV services. · Population group surveys of high-risk individuals. The pane! believes that these types of surveys should be given the lowest priority. Although they might shed light on the accessibility of referral services for seropositive individu- als, the number of persons actually infected is too small for population surveys to be recommended. Nonetheless, if this approach is considered, the pane] hopes that the dis- cussion in Appendix D on probability-based surveys and convenience sampling will be helpful. Monitoring Services at Non-CDC Testing Sites Surveillance ~nforrnation from test labs indicates that some people are being tested for HIV in private sector sites, but CDC does not know who or how many people are doing so and what services they obtain. To learn about counseling and testing services provided at non-CDC- funded sites, CDC can build on the population group surveys previously recommended as evaluation sources. The pane} suggested this type of survey in Chapter 5 to evaluate site accessibility and barriers to using counseling and testing services, but items on such surveys can be readily modified or added to determine whether, what, and where services are being obtained elsewhere. Because survey respondents may not know whether the test site they visit is funded by CDC, one item should ask the site location so that it can later be cross-checked by CDC program personnel. As discussed in Chapter 5, population surveys can be targeted to par- ticular communities or high-risk populations, although sampling frames for some high-nsk groups may be difficult to construct. Alternatively, a survey of the general population Would be helpful. As with research to evaluate CDC-funded sites, such general population surveys would help identify the services people desire and how available Hey perceive them to be. Either a new survey could be launched which would be expensiveor questions could be added to the National Heath Interview Survey (NHIS) about the reasons people seek or avoid testing.
OTHER PROJECTS ~ 341 The pane] notes that CDC has had problems with timely access to the NHIS data in the past. Both the National Center for Health Statistics and the Office of Management and Budget have lengthy approval processes for items to be added to their surveys. In addition, there are delays in sharing data once they are collected. As we did in Chapter 6, the pane} wishes to register its concern about both of these problems and express its hope that, at least in the latter case, a greater cooperation between NCHS and other divisions within CDC can be effected. Finally, He pane] considered, but decided not to recommend, client surveys of non-CDC-funded sites. The panel felt that it would be too difficult to induce cooperation win a representative sample of private sites to conduct confidential client surveys. The parted recognized, however, that such research could potentially shed light on the adequacy and completion rates of private counseling and testing services and could provide a useful comparison with services provided by CDC-funded sites. REFERENCES Hochhauser, M. (1987) Readability of AIDS education materials. Presented at the Annual Meeting of the American Psychological Association, New York, August 30. National AIDS Information Clearinghouse (1989) User Guide. March. l
