2
Review of Methods

As noted in Chapter 1, the committee was asked to review and comment on specific aspects of the draft IRIS assessment of formaldehyde. This chapter provides general comments on the methods and structure of the document. The committee’s rationale for providing general comments is that the specific elements of the charge are inseparable from the approach used by the U.S. Environmental Protection Agency (EPA) for the development of the assessment and presentation of its findings.1 In responding to questions posed in its charge and developing its report, the committee noted some recurring methodologic problems that cut across components of the charge.

The general problems that the committee identified are not unique to the draft IRIS assessment of formaldehyde. Committees of the Board on Environmental Studies and Toxicology (BEST) of the National Research Council (NRC) have reviewed a number of IRIS assessments in the last decade, including three (NRC 2005, 2006, 2010) in the last 5 years. Some of the general problems identified by the present committee have been commented on by the other BEST committees. For example, the 2006 NRC report on dioxin and related compounds commented on the need for formal, evidence-based approaches for noncancer effects, the need for transparency and clarity in the selection of data sets for analysis, and the need for greater attention to uncertainty and variability (NRC 2006). The 2010 NRC review of the draft IRIS assessment of tetrachloroethylene found similar problems and provided a chapter, “Moving Beyond the Current State of Practice,” that addressed methodologic issues and the failure to establish clear and transparent methods for carrying out and presenting the assessment (NRC 2010). That report also provided a broad set of recommendations on characterization of uncertainty.

1

The committee distinguishes between the process used to generate the draft IRIS assessment and the overall IRIS process that includes not only generation of the assessment but the many layers of review. In this report, the committee is focused on the approach used to generate the draft assessment.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 24
2 Review of Methods As noted in Chapter 1, the committee was asked to review and comment on specific aspects of the draft IRIS assessment of formaldehyde. This chapter provides general comments on the methods and structure of the document. The committee’s rationale for providing general comments is that the specific ele- ments of the charge are inseparable from the approach used by the U.S. Envi- ronmental Protection Agency (EPA) for the development of the assessment and presentation of its findings.1 In responding to questions posed in its charge and developing its report, the committee noted some recurring methodologic prob- lems that cut across components of the charge. The general problems that the committee identified are not unique to the draft IRIS assessment of formaldehyde. Committees of the Board on Environ- mental Studies and Toxicology (BEST) of the National Research Council (NRC) have reviewed a number of IRIS assessments in the last decade, including three (NRC 2005, 2006, 2010) in the last 5 years. Some of the general problems iden- tified by the present committee have been commented on by the other BEST committees. For example, the 2006 NRC report on dioxin and related com- pounds commented on the need for formal, evidence-based approaches for non- cancer effects, the need for transparency and clarity in the selection of data sets for analysis, and the need for greater attention to uncertainty and variability (NRC 2006). The 2010 NRC review of the draft IRIS assessment of tetrachloro- ethylene found similar problems and provided a chapter, “Moving Beyond the Current State of Practice,” that addressed methodologic issues and the failure to establish clear and transparent methods for carrying out and presenting the as- sessment (NRC 2010). That report also provided a broad set of recommenda- tions on characterization of uncertainty. 1 The committee distinguishes between the process used to generate the draft IRIS assessment and the overall IRIS process that includes not only generation of the assess- ment but the many layers of review. In this report, the committee is focused on the ap- proach used to generate the draft assessment. 24

OCR for page 24
Review of Methods 25 The present chapter addresses the general assessment methods and covers identification of the studies considered, their evaluation, and the weight-of- evidence assessment. These issues are also addressed within the context of the specific health outcomes evaluated (see Chapters 4-5). REVIEW OF THE METHODOLOGY OF THE DRAFT IRIS ASSESSMENT OF FORMALDEHYDE IRIS has the overall purpose of evaluating human health effects that may arise from exposure to environmental contaminants (EPA 2010a). An IRIS as- sessment addresses noncancer and cancer effects as appropriate and provides descriptive and quantitative information:  “Noncancer effects: Oral reference doses and inhalation reference concentrations (RfDs and RfCs, respectively) for effects known or assumed to be produced through a nonlinear (possibly threshold) mode of action. In most instances, RfDs and RfCs are developed for the noncarcinogenic effects of substances” (EPA 2010a).  “Cancer effects: Descriptors that characterize the weight of evidence for human carcinogenicity, oral slope factors, and oral and inhalation unit risks for carcinogenic effects. Where a nonlinear mode of action is established, RfD and RfC values may be used” (EPA 2010a). A sequence of activities is involved in conducting IRIS assessments and in calculating RfCs and unit risk estimates. Figure 2-1 is a generic schema that describes the steps used to generate the draft IRIS assessment and the actions needed at each step. The figure is the committee's representation of that process, as gleaned from the assessment. Although the draft IRIS assessment does not explicitly acknowledge these steps, they are implicit in the approach and are ordered as shown. Evaluate Select Studies for Calculate Evaluate Identify Weight of Derivation of RfC and UR Evidence Evidence Evidence RfCs and URs Apply IRIS Apply Weight-of- Assess Heterogeneity Use Systematic Apply Uniform Approach Evidence Criteria Select Key Studies Review Process Approach to Study Evaluation FIGURE 2-1 Elements of the IRIS process. Abbreviations: IRIS, Integrated Risk Infor- mation System; RfC, reference concentration; and UR, unit risk.

OCR for page 24
26 Review of EPA’s Draft IRIS Assessment of Formaldehyde In fact, the general approach receives little description in the draft assess- ment. The methods for conducting the assessment are described in a two-page introduction to a document whose main chapters and appendixes occupy about 1,000 pages. The committee notes that the introductory chapter of the formalde- hyde assessment is almost identical with that of other IRIS assessments (see, for example, the IRIS summary for chloroprene, EPA 2010b). The two pages con- stitute the sole description of the methods used by the authors and cite numerous EPA guidance documents, some dating to 1986 (see Appendix B for some of the most relevant portions of those documents). Some of the guidance documents are cited at appropriate points in the draft assessment, but their specific roles in the preparation of the draft are not clear. In general, the committee found that the draft assessment was not prepared in a transparent, consistent fashion with clear linkages to an underlying frame- work as it moves from review of the relevant evidence to calculation of the RfCs and unit risk estimates and characterization of their uncertainty and variability. The committee did not find sufficient documentation of methods and criteria for identifying the epidemiologic and experimental evidence to be reviewed, for evaluating individual studies, for assessing weight of evidence, for selecting individual studies for derivation of toxicity and risk estimates, or for characteriz- ing uncertainty and variability. Summary sections that synthesize the evidence are variable and too often brief or not present, and strength of evidence is not characterized with standardized descriptors. The committee emphasizes that its criticism regarding the lack of docu- mentation is not a recommendation for adding lengthy summaries of the indi- vidual guidance documents to the introductory chapter. It is suggesting that clear concise descriptions of key criteria used to include studies in the analysis, to exclude studies, or to advance studies for calculation of RfCs and unit risk esti- mates are needed. Nuances concerning specific health outcomes could be ad- dressed in the introductory sections on those outcomes. The following sections provide comments on the general steps of the process. Again, specific aspects are addressed in Chapters 4-5 for each health outcome. Literature Identification The ability to identify and filter studies is crucial for any literature review that is synthesizing the potential effects of a suspected hazard. The evaluation of all relevant studies in an IRIS review process is analogous to the collection of relevant studies for a meta-analysis. A general approach to literature review is provided in Chapter 1 of the draft IRIS assessment of formaldehyde. EPA used a Chemical Abstracts Service Registry Number (CASRN) and “at least one com- mon name” (EPA 2010c, p. 1-2) to search for relevant publications. The specific databases searched are not listed. PubMed searches are critical for identifying the literature on risks to health; the committee notes that PubMed does not spe- cifically capture the CASRN.

OCR for page 24
Review of Methods 27 The state of the art of literature searches now involves providing an exten- sive description of the databases searched and the search terms used. Chapter 1 of the draft IRIS assessment does not provide a list of the search terms used, such as terms that were used for the various health outcomes relevant to formal- dehyde. The draft assessment also does not describe the results of searches, so the numbers of articles identified and excluded are unavailable to readers. Study Evaluation The draft IRIS assessment evaluates many individual studies in a variety of disciplines. A description of the methods for evaluating individual studies is not provided, and it appears to this committee that studies were not reviewed with a common template for assessing their strengths and weaknesses. The committee notes that the template for evaluation would vary appropriately with the type of research study being considered. Such a strategy is not uniformly evident, and the evidence considered is not presented consistently in informative tables. In some cases, there is a tendency to describe the studies ultimately se- lected for the derivation of the RfC in favorable terms. For example, a cross- sectional study by Krzyzanowski et al. (1990)—a study selected for the calcula- tion of an RfC for respiratory effects—is referred to as “well-designed and exe- cuted” (EPA 2010c, p. 4-41) without emphasis on the inherent weaknesses of its cross-sectional design. The committee found one study selected for advance- ment for calculation of an RfC (Ritchie and Lehnen 1987) to be potentially sub- ject to severe bias and would not have recommended it for advancement. Spe- cifically, selection of the study population was based on a visit to a physician and referral for formaldehyde-concentration measurement, and the concentra- tion-response gradient was considered by the committee to be implausibly steep (see Chapter 4 for further discussion). Synthesis of Evidence and Evaluation of Causation In evaluating the evidence of causation, the draft IRIS assessment cites various EPA guidelines that apply weight-of-evidence approaches in assessing the strength of evidence. Those guidelines have been developed over a period of nearly 2 decades, and consequently consistency of methods is lacking from out- come to outcome (Appendix B). The implementation of the guidelines appears to be subjective and not standardized. The committee found variable detail in how the weight-of-evidence criteria had been applied. Uniformly developed discussions applying the weight-of-evidence criteria cannot be identified at ap- propriate points in the text. In some sections, the discussion of biologic evidence is particularly weak (for example, in the case of asthma pathogenesis) and often not reflective of the current state of knowledge.

OCR for page 24
28 Review of EPA’s Draft IRIS Assessment of Formaldehyde SUMMARY In summary, when the review of studies used in the draft IRIS assessment of formaldehyde is compared with the current standard for evidence-based re- views and causal inference, limitations in each step used to generate the draft IRIS assessment are evident. For example, the methods are not clearly de- scribed, the review approaches are not transparent, and there is no indication that evidence-grading strategies were uniformly applied. In addition, the selection approach to identifying studies for RfC calculation appears ad hoc. The commit- tee emphasizes that it is not recommending that EPA add an extensive discus- sion of its guidelines to the draft IRIS assessment. It is recommending that key factors used to exclude, include, or advance studies be discussed. REFERENCES EPA (U.S. Environmental Protection Agency). 2010a. Frequent Questions: What is IRIS? Integrated Risk Information System, U.S. Environmental Protection Agency, Washington DC [online]. Available: http://www.epa.gov/IRIS/help_ques.htm# whatiris [accessed Dec. 28, 2010]. EPA (U.S. Environmental Protection Agency). 2010b. Toxicological Review of Chloro- prene (CAS No. 126-99-8). In Support of Summary Information on the Integrated Risk Information System (IRIS). EPA/635/R-09/01F. U.S. Environmental Protec- tion Agency, Washington, DC. September 2010 [online]. Available: http:// www.epa.gov/iris/toxreviews/1021tr.pdf [accessed Jan. 6, 2010]. EPA (U.S. Environmental Protection Agency). 2010c. Toxicological Review of Formal- dehyde (CAS No. 50-00-0) – Inhalation Assessment: In Support of Summary Information on the Integrated Risk Information System (IRIS). External Re- view Draft. EPA/635/R-10/002A. U.S. Environmental Protection Agency, Washington, DC [online]. Available: http://cfpub.epa.gov/ncea/iris_drafts/reco rdisplay.cfm?deid=223614 [accessed Nov. 22, 2010]. Krzyzanowski, M., J.J. Quackenboss, and M.D. Lebowitz. 1990. Chronic respiratory effects of indoor formaldehyde exposure. Environ. Res. 52(2):117-125. NRC (National Research Council). 2005. Health Implications of Perchlorate Ingestion. Washington, DC: National Academies Press. NRC (National Research Council). 2006. Health Risks from Dioxin and Related Com- pounds: Evaluation of the EPA Reassessment. Washington, DC: National Academies Press. NRC (National Research Council). 2010. Review of the Environmental Protection Agency's Draft IRIS Assessment of Tetrachloroethylene. Washington, DC: Na- tional Academies Press. Ritchie, I.M., and R.G. Lehnen. 1987. Formaldehyde-related health complaints of resi- dents living in mobile and conventional homes. Am. J. Public Health. 77(3):323-328.