4
Methodologic Considerations in Evaluating the Evidence

QUESTIONS TO BE ADDRESSED

The committee was charged with the task of summarizing the strength of the scientific evidence concerning the association between herbicide exposure during Vietnam service and each of a set of diseases or conditions suspected to be associated with such exposure. For each disease, the committee determined, to the extent that available scientific data permit meaningful determinations,

  1. whether a statistical association with herbicide exposure exists, taking into account the strength of the scientific evidence and the appropriateness of the statistical and epidemiologic methods used to detect the association;

  2. the increased risk of each disease among those exposed to herbicides during Vietnam service; and

  3. whether there exists a plausible biologic mechanism or other evidence of a causal relationship between herbicide exposure and the disease.

The law establishing the committee (Public Law 102-4, codified as 38 USC Sec. 1116) did not provide a specific list of diseases and conditions suspected to be associated with herbicide exposure. The committee staff and members responsible for the 1994 report Veterans and Agent Orange (hereafter VAO) (IOM, 1994) developed such a list based on diseases and conditions that had been mentioned in the scientific literature or in other documents that came to their attention through extensive literature searches. The VAO list has been supplemented over time in response to developments in the literature. The information used by the committee was developed through a comprehensive search of relevant data bases. Public and commercial data bases covering biological, medical,



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 124
Veterans and Agent Orange: Update 1998 4 Methodologic Considerations in Evaluating the Evidence QUESTIONS TO BE ADDRESSED The committee was charged with the task of summarizing the strength of the scientific evidence concerning the association between herbicide exposure during Vietnam service and each of a set of diseases or conditions suspected to be associated with such exposure. For each disease, the committee determined, to the extent that available scientific data permit meaningful determinations, whether a statistical association with herbicide exposure exists, taking into account the strength of the scientific evidence and the appropriateness of the statistical and epidemiologic methods used to detect the association; the increased risk of each disease among those exposed to herbicides during Vietnam service; and whether there exists a plausible biologic mechanism or other evidence of a causal relationship between herbicide exposure and the disease. The law establishing the committee (Public Law 102-4, codified as 38 USC Sec. 1116) did not provide a specific list of diseases and conditions suspected to be associated with herbicide exposure. The committee staff and members responsible for the 1994 report Veterans and Agent Orange (hereafter VAO) (IOM, 1994) developed such a list based on diseases and conditions that had been mentioned in the scientific literature or in other documents that came to their attention through extensive literature searches. The VAO list has been supplemented over time in response to developments in the literature. The information used by the committee was developed through a comprehensive search of relevant data bases. Public and commercial data bases covering biological, medical,

OCR for page 124
Veterans and Agent Orange: Update 1998 toxicological, chemical, historical, and regulatory information were examined. The majority of these data bases were bibliographic, providing citations to scientific literature. Committee staff examined the reference lists of major review articles, books, and reports for relevant citations. Reference lists of individual articles were also scanned for pertinent citations. Internet search engines were used to scan for information posted on the Internet. Literature identification continued through September 30, 1997. The input received both in written and oral form from veterans and other interested persons at public hearings and in written submissions served as a valuable source of additional information. Appendix A gives additional detail on the search strategies used to generate reference sources. Information submitted to the committee by interested persons is also listed in Appendix A. This second biennial update concentrates on evaluating the evidence published following the completion of work on Veterans and Agent Orange: Update 1996 (hereafter Update 1996) (IOM, 1996) and VAO. For each health outcome, the new evidence is reviewed in detail. Conclusions are based on the totality of the accumulated evidence, not just on recently published studies. In other words, new evidence is not interpreted alone but is put into the context of evidence addressed in the two previous reports. In addition to bringing earlier work up to date, the committee has addressed other areas of concern identified by the Department of Veterans Affairs (DVA). Specifically, the committee was asked to do the following: Pay particular attention to the relationship between exposure to herbicides and the subsequent development of diabetes. Chapter 11 contains an extended discussion of this topic. Special attention has also been given to adverse reproductive outcomes, which are addressed in Chapter 9. This area attracted heightened interest after a finding, reported in Update 1996, of limited/suggestive evidence of an association between herbicide exposure and spina bifida in the children of veterans. Examine the issue of the latency between exposure to herbicides and development of adverse health outcomes. In response to DVA's request, Chapter 8 (1) proposes a methodology to address issues concerning the timing of herbicide exposure and the risk of cancer; (2) reviews the literature on herbicide exposure and cancers classified in the sufficient and limited/suggestive evidence of an association categories for results that describe how the timing of exposure affects relative risk; and (3) describes timing-of-exposure characteristics of the Vietnam veterans and summarizes the implications of these factors for their risk of cancer. Discuss the classification of chondrosarcomas of the skull. This subject is addressed in Chapter 7 as part of the discussion of bone cancer. Offer advice on herbicide exposure assessment for Vietnam veterans. Chapter 5 of the report contains an extended discussion of this issue. In addition, a separate effort by another Institute of Medicine (IOM) committee is facilitating

OCR for page 124
Veterans and Agent Orange: Update 1998 the development and evaluation of models of herbicide exposure for use in studies of Vietnam veterans. That committee authored and disseminated a Request for Proposals for exposure assessment research in 1997 (IOM, 1997) and began scientific oversight of the research in 1998. Address the potential for using data combination methodologies to informatively reexamine existing data on the health effects of herbicide or dioxin exposure. The committee conducted a workshop on this topic in August 1997, which brought together experts in these methodologies and researchers who have developed and analyzed data sets evaluating the health of Vietnam veterans and individuals exposed to herbicides or dioxin. The results of this effort will be addressed in a separate publication. The committee's judgments have both quantitative and qualitative aspects; they reflect both the evidence examined and the approach taken to evaluate it. In VAO, the committee delineated how it approached its task, so that readers would be able to assess and interpret the committee's findings. By offering this information, the committee wished to make the report useful to those seeking to update its conclusions as new information was obtained. The committees responsible for Update 1996 and the present effort have adopted the original committee's approach. The remainder of this chapter outlines the types of evidence that the committee identified; the approaches used in evaluating published reports, both singly and collectively; and the nature of the committee's conclusions. Details of the analysis and specific conclusions concerning each health outcome appear in subsequent chapters. Descriptions of methodology and specific information on how the committee interpreted the questions asked in the original legislation can be found in Chapter 5 of VAO. Is Herbicide Exposure Statistically Associated with the Health Outcome? The committee necessarily focused on a pragmatic question: What is the nature of the relevant evidence for or against a statistical association between exposure and the health outcome? The evidentiary base that the committee found to be most helpful derived from epidemiologic studies of populations—that is, investigations in which large groups of people are studied to determine the association between the occurrence of particular diseases and exposure to the substances at issue. To determine whether an association exists, epidemiologists estimate the magnitude of an appropriate quantitative measure (such as the relative risk or the odds ratio) that describes the relationship between exposure and disease in defined populations or groups. The committee reports the measure of effect used by the study. However, the use of terms such as ''relative risk," "odds ratio," or "estimate of relative risk" is not consistent in the literature. In this report, the committee intends relative risk to refer to the results of cohort studies and odds ratio (an estimate of relative risk) to refer to the results of case-control

OCR for page 124
Veterans and Agent Orange: Update 1998 studies. Values of relative risk greater than 1 may indicate a positive or direct association—that is, a harmful association—whereas values between 0 and 1 may indicate a negative or inverse association—that is, a protective association. A "statistically significant" difference is one that, under the assumptions made in the study and the laws of probability, would be fairly unlikely to occur if there was no true difference. Determining whether an observed statistical association between exposure and a health outcome is "real" requires additional scrutiny because there may be explanations for the observed association other than the exposure. These include error in the design, conduct, or analysis of the investigation; bias, or a systematic tendency to distort the measure of association so that it may not represent the true relation between exposure and outcome; confounding, or distortion of the measure of association because another factor related to both exposure and outcome has not been recognized or taken into account in the analysis; and chance, the effect of random variation, which produces spurious associations that can, with a known probability, sometimes depart widely from the true relation. Therefore, in deciding whether an association between herbicide exposure and a particular outcome existed, the committee examined the quantitative estimates of risk and evaluated whether these estimates might be due to error, bias, confounding, or chance, or were likely to represent a true association. In pursuing the question of statistical association, the committee recognized that an absolute conclusion about the absence of association may never be attained. As in science generally, studies of health outcomes following herbicide exposure are not capable of demonstrating that the purported effect is impossible or could never occur. Any instrument of observation, including epidemiologic studies, has a limit to its resolving power. Hence, in a strict technical sense, the committee could not prove the absolute absence of a health outcome associated with herbicide or dioxin exposure. Nevertheless, for some outcomes examined, there was limited or suggestive evidence consistent with no association. The committee was able to conclude in some cases that within the limits of the current resolving power of the existing studies, there is no association with herbicide exposure. What Is the Increased Risk of the Outcome in Question Among Those Exposed to Herbicides in Vietnam? This question, which is pertinent principally (but not exclusively) if there is evidence for a positive association between exposure and a health outcome, concerns the likely magnitude of the association in Vietnam veterans exposed to herbicides. The most desirable evidence in answering this type of question involves knowledge of the rate of occurrence of the disease in those Vietnam veterans who were actually exposed to herbicides, the rate in those who were not exposed (the "background" rate of the disease in the population of Vietnam

OCR for page 124
Veterans and Agent Orange: Update 1998 veterans), and the degree to which any other differences between exposed and unexposed groups of veterans influence the difference in rates. When exposure levels among Vietnam veterans have not been adequately determined, which has been the case in most studies, this question becomes difficult to answer. The committee found the available evidence sufficient for drawing conclusions about the association between herbicide exposure and a number of health outcomes. However, the lack of good data on Vietnam veterans per se, especially with regard to exposure, had complicated the assessment of the increased risk of disease among individuals exposed to herbicides during service in Vietnam. By considering the magnitude of the association observed in other cohorts, the quality and results of studies that have been made of veterans, and other principles of epidemiologic research, the committee has formulated a qualitative judgment regarding the risk of disease among Vietnam veterans. Indeed, most of the evidence on which the findings in this report are based comes from studies of people exposed to dioxin or herbicides in occupational and environmental settings rather than from studies of Vietnam veterans. When the available data do not permit a meaningful statement regarding risk among Vietnam veterans, no conclusion is drawn and the reader is referred to the appropriate section in an earlier report for additional discussion. Is There a Plausible Biologic Mechanism? Chapter 3 details the cellular and animal experimental evidence that provides the basis for the assessment of biologic plausibility, that is, the extent to which a statistical association is consistent with existing biological or medical knowledge. As with the epidemiologic evidence, the chapter concentrates on studies published during 1995-1997 but considers all relevant studies in drawing conclusions. The likelihood that a given chemical exposure-health outcome relationship reflects a true association in humans is addressed in the context of research regarding the mechanism of interaction between the chemical and biological systems, evidence of tumorigenicity in animal studies, evidence of an association between exposure and health outcome occurrence in humans, and/or evidence that a given outcome is associated with occupational or environmental chemical exposures. It must be recognized, however, that given the limitations of existing biological and medical knowledge, the lack of data in support of a plausible biologic mechanism does not rule out the possibility that a causal relationship does exist. ISSUES IN EVALUATING THE EVIDENCE Toxicologic Studies A valid surrogate animal model for the study of a human disease must reproduce with some degree of fidelity the manifestations of the disease in hu-

OCR for page 124
Veterans and Agent Orange: Update 1998 mans. Whole animal studies or animal-based experimental systems continue to be used to study herbicide toxicity because they allow for rigid control of chemical exposures and close monitoring of health outcomes. Because many of the chemical exposures presently associated with certain diseases in humans have been confirmed in experimental studies (Huff, 1993; Huff et al., 1994), data derived from such studies are generally accepted as a valuable guide in the assessment of biological plausibility. As discussed in Chapter 3, many of the toxic effects of the herbicides used in Vietnam have been ascribed to 2,3,7,8-tetrachlorodibenzo-p-dioxin (TCDD), a contaminant of some of these herbicides. This has not, however, simplified the risk assessment process because the toxicologic profile of TCDD is rather complex. In general, there is consensus that most of the toxic effects of TCDD involve interaction with the aryl hydrocarbon receptor (AhR), a protein that binds TCDD and other aromatic hydrocarbons with high affinity. Attempts to establish correlations between the effects of TCDD in experimental systems and in humans are particularly problematic, because species differences in susceptibility to TCDD have been documented. Although studies in which transformed human cell lines are employed to study AhR biology minimize the inherent error associated with species extrapolations, caution must be exercised because the extent to which transformation itself influences toxicity outcomes has yet to be fully defined. In addition, while it is generally accepted that genetic susceptibility plays a key role in determining the adverse effects of environmental chemicals, the impact of different genetic backgrounds on AhR function is not yet completely understood. Evidence continues to accumulate that the AhR is not exclusively responsible for the toxic effects of TCDD. Thus, the dose-response relationships that arise from multiple toxicokinetic and toxicodynamic interactions must also be considered. Future attempts to define the quantitative relationship between receptor occupancy and biologic response to TCDD must consider that multiple biochemical changes may influence the overall cellular response. These considerations, and the available scientific information regarding non-TCDD biological effects, are discussed in greater detail in Chapter 3. Epidemiologic Studies Environmental and/or occupational exposures to herbicides or TCDD provide data on human responses that can be compared directly to data obtained in experimental studies. Higher-than-background body burdens of dioxin have been documented in many of these groups, and details describing the major findings from these studies are reviewed in Chapters 7-11 of this report. In general, the elevated risks of cancers at various sites reported in epidemiologic studies are consistent with the known biological actions of the agents present in herbicide formulations. Although its full potential has yet to be realized, the application of

OCR for page 124
Veterans and Agent Orange: Update 1998 molecular and cellular measurements to epidemiologic research promises to increase our understanding of the association between herbicide exposure and disease occurrence. This may provide a significant advantage in the assessment of biologic plausibility, because biologically based epidemiologic data allow for more accurate identification and quantification of exposures. For instance, the analytical data available from individuals known to have been exposed to herbicides during the Vietnam War constitute a valuable resource for the study of TCDD-related disease, with documented TCDD body burdens providing a quantitative bridge between experimental studies and human epidemiology. Taken together, experimental studies and epidemiologic investigations provide complementary perspectives from which to view human health effects of exposure to herbicides. However, it must be recognized that the ultimate test of associations between exposure and disease occurrence lies in data obtained from human populations. To obtain additional information pertinent to the evaluation of the potential effects of herbicide exposure of veterans, this and previous committees decided to review studies of other groups potentially exposed to the herbicides contained in Agent Orange, to other herbicides, and to dioxin, the contaminant believed to cause many of the purported adverse effects of Agent Orange. These study populations include industrial and agricultural workers, Vietnamese citizens, and people exposed environmentally as a result of residing near the site of an accident or a toxic waste dump. The committee felt that reviewing the studies of such groups would help in determining (1) whether these compounds could be associated with particular health outcomes in veterans and (2) the nature of any dose-response relationships, although the committee acknowledged that such findings may have only an indirect bearing on the association in veterans themselves. It is also important to note that the categories of association described below relate to the association between exposure to chemicals and health outcomes in human populations, not to the likelihood that any individual's health problem is associated with or caused by the herbicides in question. The Role of Case Studies and Other Studies with no Comparison Groups With the exception of one condition, the committee did not specifically consider case studies or other published studies lacking a control or comparison group. The one exception involved studies of acute and subacute transient peripheral neuropathy. The committee elected to consider case histories from occupational cohorts and descriptive reports of the Seveso accident when evaluating the association between exposure and these conditions because their transient nature precluded using case-control and other types of studies with comparison populations.

OCR for page 124
Veterans and Agent Orange: Update 1998 Publication Bias The phenomenon known as publication bias is also of concern to the committee. It has been well documented (Begg and Berlin, 1989; Berlin et al., 1989; Dickersin, 1990; Dickersin et al., 1992; Easterbrook et al., 1991; Stern and Simes, 1997) in biomedical research that studies with a statistically significant finding are more likely to be published than studies with nonsignificant results. Thus, evaluations of disease-exposure associations that are based solely on the published literature could be biased in favor of a positive association. In general, however, for reports of overall associations with exposure, the committee did not consider the risk of publication. bias to be high among studies of herbicide exposure and health risks. The committee took this position because there are numerous published studies showing no positive association; because it examined a substantial amount of unpublished material; and because the committee felt that publicity surrounding the issue of exposure to herbicides, particularly regarding Vietnam veterans, has been so intense that any studies showing no association would be unlikely to be viewed as unimportant by the investigators. In short, the pressure to publish such "negative" findings would be considerable. Nevertheless, publication bias of a more specific and subtle form may still have had a bearing on the committee's evaluation of the evidence. In particular, the relationship between timing and duration of exposure and subsequent changes in risk of disease was a major concern. This more subtle bias would arise if decisions to publish specific findings relative to timing of exposure were based on the statistical significance of those findings. For example, a study of production workers by the National Institute for Occupational Safety and Health (NIOSH) (Fingerhut et al., 1991) found a more substantial increase in risk among those whose exposure had begun more than 20 years previously and had lasted for more than a year. Many other studies did not publish data relevant to the issues of duration of, and time since, exposure. In most cases, it is impossible to know whether or not such issues were examined and simply not discussed in a publication solely because no "interesting" associations were found. Even decisions to examine or not examine such relationships may have been based on the investigators' perception that such analyses would or would not lead to any statistically significant findings. The Role of Judgment The evaluation of evidence to reach conclusions about statistical associations goes beyond quantitative procedures at several stages: assessing the relevance and validity of individual reports; deciding on the possible influence of error, bias, confounding, or chance on the reported results; integrating the overall evidence within and across diverse areas of research; and formulating the conclusions themselves. These aspects of the committee's review required thoughtful

OCR for page 124
Veterans and Agent Orange: Update 1998 consideration of alternative approaches at several points. They could not be accomplished by adherence to a narrowly prescribed formula. Rather, the approach described here evolved throughout the process of review and was determined in important respects by the nature of the evidence, exposures, and health outcomes at issue. Both the quantitative and the qualitative aspects of the process that could be made explicit were important to the overall review. Ultimately, the conclusions about association expressed in this report are based on the committee's collective judgment. The committee has endeavored to express its judgments as clearly and precisely as the data allowed. Integration of New Evidence As stated above, this second biennial update concentrates on evaluating the evidence published following the completion of work on Update 1996 and VAO. For each disease, the new evidence is evaluated, and conclusions are based on the totality of accumulated evidence, not just on recently published studies. For one health outcome—urinary bladder cancer—the committee found new evidence that was sufficient to change the conclusion reached in previous reports. Although there is no evidence that exposure to herbicides is related to bladder cancer, relative risks in largest groups of exposed individuals under study tended to be greater than 1. This new information led the committee to change the classification of bladder cancer from limited/suggestive evidence of no association to inadequate or insufficient evidence to determine whether an association exists. For all other health outcomes, evidence appearing since the publication of Update 1996 reinforced, or was not considered strong enough to change, the previous conclusions. SUMMARY OF THE EVIDENCE Categories of Association The categories of association used by the committee are the same as those used in the previous reports. Consistent with the charge to the Secretary of Veterans Affairs in P.L. 102-4, the distinctions between the categories are based on "statistical association," not on causality. Thus, standard criteria used in epidemiology for assessing causality (Hill, 1971) do not strictly apply. The distinctions between the categories reflect the committee's judgment that a statistical association would be found in a large, well-designed epidemiologic study of the outcome in question in which exposure to herbicides or dioxin was sufficiently well characterized and appropriately measured. The categories of association are as follows: Sufficient Evidence of an Association Evidence is sufficient to conclude

OCR for page 124
Veterans and Agent Orange: Update 1998 that there is a positive association. That is, a positive association has been observed between herbicides and the outcome in studies in which chance, bias, and confounding could be ruled out with reasonable confidence. For example, if several small studies that are free from bias and confounding show an association that is consistent in magnitude and direction, this may constitute sufficient evidence for an association. Limited/Suggestive Evidence of an Association Evidence is suggestive of an association between herbicides and the outcome, but it is limited because chance, bias, and confounding could not be ruled out with confidence. For example, if at least one high-quality study shows a positive association but the results of other studies are inconsistent, this may constitute limited/suggestive evidence of an association. Inadequate/Insufficient Evidence to Determine Whether an Association Exists The available studies are of insufficient quality, consistency, or statistical power to permit a conclusion regarding the presence or absence of an association. For example, if studies fail to control for confounding, contain inadequate exposure assessment, or have inadequate sample size, this may constitute inadequate/ insufficient evidence to determine whether an association exists. Limited/Suggestive Evidence of No Association There are several adequate studies, covering the full range of exposure levels that humans are known to encounter, that are mutually consistent in not showing a positive association between exposure to herbicides and the outcome at any level of exposure. A conclusion of "no association" is inevitably limited to the conditions, level of exposure, and length of observation covered by the available studies. In addition, the possibility of a very small elevation in risk at the levels of exposure studied can never be excluded. REFERENCES Begg CB, Berlin JA. 1989. Publication bias and dissemination of clinical research. Journal of the National Cancer Institute 81:107-115. Berlin JA, Begg CB, Louis TA. 1989. An assessment of publication bias using a sample of published clinical trials. Journal of the American Statistical Association 84:381-392. Dickersin K, Min Y-I, Meinert CL. 1992. Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards. Journal of the American Medical Association 267:374-378. Dickersin K. 1990. The existence of publication bias and risk factors for its occurrence. Journal of the American Medical Association 263:1385-1389. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. 1991. Publication bias in clinical research. Lancet 337:867-872. Fingerhut MA, Halperin WE, Marlow DA, Piacitelli LA, Honchar PA, Sweeney MH, Greife AL, Dill PA, Steenland K, Suruda AJ. 1991. Cancer mortality in workers exposed to 2,3,7,8-tetrachlorodibenzo-p-dioxin. New England Journal of Medicine 324:212-218. Hill, AB. 1971. Principles of Medical Statistics, 9th ed. New York: Oxford University Press. Huff J, Lucier G, Tritscher A. 1994. Carcinogenicity of TCDD: experimental, mechanistic, and epidemiologic evidence. Annual Review of Pharmacology and Toxicology 34:343-372.

OCR for page 124
Veterans and Agent Orange: Update 1998 Huff J. 1993. Chemicals and cancer in humans: first evidence in experimental animals. Environmental Health Perspectives 100:201-210. Institute of Medicine (IOM). 1994. Veterans and Agent Orange: Health Effects of Herbicides Used in Vietnam. Washington, DC: National Academy Press. Institute of Medicine. 1996. Veterans and Agent Orange: Update 1996. Washington, DC: National Academy Press. Institute of Medicine. 1997. Characterizing Exposure of Veterans to Agent Orange and Other Herbicides Used in Vietnam: Scientific Considerations Regarding a Request for Proposals for Research. Washington, DC: National Academy Press. Stern JM, Simes RJ. 1997. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. British Medical Journal 315:640-645.