3

Crosscutting Issues with the TSCA Approach

INTRODUCTION

Chapter 2 examined the systematic review approach taken by the U.S. Environmental Protection Agency (EPA) in its Toxic Substances Control Act (TSCA) risk evaluations, based on Application of Systematic Review in TSCA Risk Evaluations (herein the 2018 guidance document), and further elaborations of its processes as documented in the specific evaluations examined by the committee—the Draft Risk Evaluation for Trichloroethylene (TCE) and the Final Risk Evaluation for 1-Bromopropane (n-Propyl Bromide) (EPA 2020a,c). This chapter builds on Chapter 2, addressing crosscutting and more general issues related to the use of systematic review in TSCA risk evaluations. The committee first offers overall findings and then turns to specific topics.

Additionally, in framing the committee’s overall evaluation in this chapter, the report notes that EPA and specifically its Office of Pollution Prevention and Toxics (OPPT) have faced the herculean challenges of developing their risk assessment processes while meeting the schedule for evaluations specified in the 2016 amended TSCA. As noted in Chapter 1, the agency also needs to operate with the definition of weight of evidence (WOE) provided in the Procedures for Chemical Risk Evaluation Under the Amended Toxic Substances Control Act, referred to as the “Risk Evaluation Rule” (40 CFR Part 702, 82 FR 33726). The definition in the Risk Evaluation Rule deviates from the more conventional meaning of the term WOE and application of approaches for WOE. Suggestions are made with regard to better aligning terminology in TSCA assessments with usual practice.

OVERALL FINDINGS

Restating the Statement of Task, this committee was asked to “review EPA’s guidance document on Application of Systematic Review in TSCA [Toxic Substances Control Act] Risk Evaluations and associated materials to determine whether the process is comprehensive, workable, objective, and transparent.” In summary, the committee finds that the process outlined in the 2018 guidance document, and as elaborated and applied in the example evaluations, does not meet the criteria of “comprehensive, workable, objective, and transparent.” The committee’s evaluation was made difficult by the incomplete and hard-to-follow documentation of many details of the process—adequacy of documentation is requisite for achieving transparency, objectivity, and replicability.

The committee found that the systematic reviews within the draft risk evaluations considered did not meet the standards of systematic review methodology. The committee applied the critical appraisal tool for systematic review “assessment of multiple systematic reviews” (AMSTAR-2) to the hazard assessment in the draft TCE risk evaluation and found the appraisal process to be unnecessarily complicated due to insufficient and unclear documentation. Despite this barrier to applying the AMSTAR-2 instrument, the committee found that the TCE hazard assessment did not perform positively on the vast majority of AMSTAR-2 questions. Hence, the committee concluded that the hazard assessment within the TSCA TCE risk evaluation was of critically low quality, meaning that the review had “more than one critical flaw and should not be relied on to provide an accurate and comprehensive summary of the available studies” (Shea et al. 2017, p. 6). Consequently, the committee suggests that the OPPT team comprehensively reevaluate its approach so as to achieve the state of the practice for systematic review.

In the committee’s judgment, the specific and general problems in TSCA risk evaluations are partially due to the decision to develop a largely de novo approach, rather than starting with the foundation offered by approaches that were extant in 2016. OPPT was challenged by the statutory schedule for completing assessments. Nonetheless, looking forward, the committee strongly recommends that OPPT reconsider its overall strategy. Further guidance for moving forward is offered below.

THE USE OF SYSTEMATIC REVIEW FOR THE RISK EVALUATIONS

From the point of view of the broad range of stakeholders involved with TSCA risk evaluations, the evaluations need to be developed with methods that are rigorous, reproducible, valid, and transparent. Systematic review is a method that meets these requirements, if carried out correctly. As discussed in Chapter 1, the statute itself, however, did not specifically require systematic review but discussed that systematic review would be used.

Systematic review methods are already established for the evidence streams (e.g., human, animal or ecological receptors, or mechanistic) contributing to hazard assessment for human health and ecological receptors. However, comparable systematic review approaches were not available for the other evidence streams included in TSCA assessments (see Figure 1-1). As a result, OPPT staff embarked on extending methods of systematic review to chemical properties, fate and transport, and exposures of the population generally and in occupational settings—components that are broadly related to the human and ecological exposure assessments. Consequently, the resulting processes for these components of the evaluations are still evolving according to OPPT and, up to now, have required substantial effort for their development and implementation on the part of the OPPT TSCA team. The approaches for these evidence streams are not yet fixed and lack rigorous evaluation by OPPT staff or external committees. This committee understands the enormity of the task of carrying out TSCA evaluations but notes that such innovations, in order to be broadly accepted, need rigorous evaluation and testing by multiple stakeholders.

Under the amended TSCA, EPA is required to evaluate the “weight of the scientific evidence” with the definition beginning with “means a systematic review method” (40 CFR Part 702, 82 FR 33726). The committee notes that “systematic review method” is left undefined. Without a clear definition of “a systematic review method,” the committee inferred from the presentations by OPPT staff and the 2018 guidance document that the TSCA program interpreted the rule to mean “systematic review” as conventionally defined. With this interpretation of the rule in mind and considering the full set of requirements for what constitutes systematic review, the committee has found that OPPT has not been performing “systematic review” for the various evidence streams in a way that meets current standards.

The committee finds that any evidence-based methods applied to each evidence stream should meet the required characteristics of the process for “weight of the scientific evidence” because per the amended TSCA this process “uses a pre-established protocol to comprehensively, objectively, transparently, and consistently identify and evaluate each stream of evidence, including strengths, limitations, and relevance of each study.” In the committee’s reading, the requirement is for a protocol for addressing each stream of evidence. Systematic review, as defined by this committee and others, is not specifically described.

While the state of the practice for gathering and reviewing evidence is not as thoroughly specified for the evidence streams other than those contributing to hazard assessment, the committee did not find that approaches were being used that are consistent with best practices for exposure assessment and for fate and transport. Experience to date shows that developing de novo a systematic review approach that can be applied across all evidence streams has proved challenging, and perhaps impracticable, and strains resources. Review strategies are needed that meet the criteria of being “comprehensive, workable, objective, and transparent.” While each study in all evidence streams may need to be identified,

examined, and synthesized for the elements of hazard assessment, such review methodology does not necessarily directly extend to running a fate and transport model, for example.

Additionally, under some circumstances there may be reasonable alternatives to carrying out a de novo systematic review; for example, the relevant literature may be non-existent or too limited in scope or there may be a recent systematic review that meets quality standards. In some cases, it may be possible to use an alternative approach to systematic review as long as it meets the transparency, consistency, reproducibility, and comprehensiveness requirements of evidence-based methodologies. When utilizing an alternative evidence-based methodology in lieu of systematic review, however, the rationale for the deviation from the systematic review should be explicitly stated in the risk evaluation scoping document.

DEVELOPMENT OF NEW METHODS OR UTILIZATION OF EXISTING METHODS

Beginning with its initial assessments, OPPT developed review approaches that were particular to its mandate under TSCA. As OPPT began to develop its approach following the 2016 amended TSCA, methods were extant for carrying out comprehensive, transparent assessments of the hazards posed by environmental agents (e.g., those elaborated by the Office of Health Assessment and Translation [OHAT], EPA’s Integrated Risk Information System [IRIS] Program, and the Navigation Guide). Additionally, instruments were available for assessing risk of bias and certainty of evidence, and versions thereof were being tailored for environmental agents. Nonetheless, OPPT did not adopt these general methodologies nor use available tools that are directly applicable to the TSCA charge and instead embarked on the arduous task of developing new methods and instruments.

The committee suggests that OPPT step back from the approach that it has taken and consider what components of the OHAT, IRIS, or Navigation Guide methods could be incorporated directly and specifically into hazard assessment (NASEM 2018; OHAT 2019; Woodruff and Sutton 2014). These methods have a trajectory of use and community acceptance and reflect the state of the practice. The committee also suggests that OPPT evaluate the ways that these existing methods could be modified for other evidence streams. In addition, OPPT should use existing guidance within the agency such as the Guidelines for Human Exposure Assessment and the Guidelines for Ecological Risk Assessment.

Similarly, as described in Chapter 2, there are already established tools for assessing risk of bias and certainty that have been used extensively and that could be used in TSCA risk evaluations. The committee recommends that OPPT give full consideration to existing approaches related to all evidence streams before continuing on the track of developing new instruments that may not be needed.

WEIGHT OF EVIDENCE

Throughout this report, there are comments on the consequences of the definition of WOE in the Risk Evaluation Rule and the conflation of “weight of the scientific evidence” with “a systematic review method” as necessitated by the Risk Evaluation Rule. This deviation from convention comes with consequences for describing the application of WOE principles in evidence integration.

OPPT is cognizant of the discrepancy between the definition in the Risk Evaluation Rule and usual practice; for example, during a presentation on February 28, 2020, OPPT staff described one step with regard to risk evaluation/risk characterization: “Describe the weight of the scientific evidence for the identified hazard and exposure” (Susanna Blair, presentation to the committee, February 28, 2020).

The committee understands that the definition of WOE within the Risk Evaluation Rule is not easily changed but suggests that OPPT adopt a specific term to describe the WOE evaluation during the evidence integration step, other than the term “weight of evidence,” to avoid the semantic clash between the definition in the Risk Evaluation Rule and its application during the evaluation process. Alternatives might include “strength of evidence” or “certainty of evidence” as utilized in the Grading of Recommendations Assessment, Development and Evaluation process (Whaley et al. 2020).

Regardless of terminology, a narrative description should be provided that describes the basis for the determination of the strength of evidence during the evidence integration step for all applicable data streams. The committee proposes the use of standard descriptors for the strength of evidence as with the Integrated Science Assessments for the National Ambient Air Quality Standards (EPA 2019c).

ENHANCING CLARITY OF DOCUMENTATION OF THE ASSESSMENT METHODS

The committee carefully examined the 2018 guidance document, considered several TSCA evaluations, and was briefed by OPPT staff on multiple occasions, including with presentations and poster sessions. Nonetheless, the committee’s work to complete the Statement of Task was limited by a lack of complete, coherent, and readily accessible material on how the assessments were conducted, particularly as the approach evolved across the assessments from the first 10 to the 20 now in progress. The committee anticipates that the many stakeholders who use these documents are similarly struggling. In fact, with this lack of clarity, the transparency requirement of the WOE definition in the rule is not adequately met, complicating a determination as to whether the requirements for the WOE assessment (i.e., a process that is comprehensive, objective, transparent, and consistent) are achieved. Moreover, one of the advantages of systematic review is that it is reproducible and updateable. The committee finds that TSCA evaluations would be extremely difficult if not impossible to reproduce or update with new information. Standardization of documentation of every step in the process is a critical step toward this goal.

Consequently, the committee suggests the development of enhanced documentation that would facilitate both development and use of TSCA evaluations. Systematic review methodology itself provides structure and recommendations on planning, conducting, and reporting (Moher et al. 2009).

The committee recommends that a handbook of TSCA systematic review and evidence integration methodology be put together that details the steps in the process. Throughout this report, the committee points to problems of documentation. For example, there needs to be a record of decisions made for each publication identified during the systematic review or other evidence-gathering processes. Such a handbook would likely need internal and external review and require substantial time for its development. However, in the committee’s view and drawing on its experience in writing this report, the 2018 guidance document is not adequate as a stand-alone document to describe how systematic reviews are carried out in TSCA risk evaluations. The committee believes that the effort of developing and publicly vetting a handbook will pay off in the long run by making the process more straightforward and transparent as well as easier to follow.

Recognizing that preparation of a comprehensive new document describing procedures will need substantial time for planning and development, OPPT staff should assemble an “evergreen” compilation of how reviews are carried out, ideally documenting evolution of practice from the 2018 document to the present. The inclusion of a comprehensive glossary of key terms, including all the terms used throughout the review process, will be important. Such a document should capture changes made to the review process subsequent to the 2018 guidance document and describe what changes have been made, the rationale for the changes, and the risk assessments to which they apply.

SUMMARY

The committee was in strong consensus that the processes used by OPPT do not meet the evaluation criteria specified in the Statement of Task (i.e., comprehensive, workable, objective, and transparent). OPPT faced substantial challenges in integrating review methods on the schedule required by the Lautenberg Act. Those challenges have not yet been successfully met. Chapter 2 includes a number of specific recommendations as to how to improve the methods for assessments, both in general and with reference to particular elements of the evaluation process.

Chapter 3 covers broad issues in relation to the Statement of Task. The general recommendations from this chapter are summarized as follows:

• The OPPT approach to systematic review does not adequately meet the state of the practice. The committee suggests that OPPT comprehensively reevaluate its approach to systematic review methods, addressing the comments and recommendations of Chapter 2.
• With regard to hazard assessment for human and ecological receptors, the committee comments that OPPT should step back from the approach that it has taken and consider components of the OHAT, IRIS, and Navigation Guide methods that could be incorporated directly and specifically into hazard assessment.
• The committee finds that OPPT’s use of systematic review for the evidence streams for which it has not been previously adapted to be particularly unsuccessful. Given these novel applications of systematic review, the committee suggests that OPPT elaborate plans for continuing the refinement of methods, ideally, in collaboration with internal and external stakeholders. The committee also suggests that OPPT evaluate the ways that existing OHAT, IRIS, and Navigation Guide methods could be modified for the other evidence streams. In addition, OPPT should use existing guidance within the agency such as the Guidelines for Human Exposure Assessment, the Guidelines for Ecological Risk Assessment, and the operating procedures for the use of the ECOTOXicology knowledgebase, as following existing guidelines would improve transparency of the assessments.
• The committee recommends that a handbook for TSCA review and evidence integration methodology be put together that details the steps in the process. Throughout this report, the committee points to problems of documentation. The committee believes that the effort of developing and publicly vetting a handbook will pay off in the long run by making the process more straightforward, transparent, and easier to follow.

There is an ongoing cross-sector effort on developing and validating new tools and approaches for exposure, ecotoxicology, environmental health, and other new areas of application of systematic review. The committee strongly recommends that OPPT staff engage in these efforts. The approaches used for TSCA evaluation would benefit from the substantial external expertise available as well as additional transparency and acceptance by the different stakeholders and society in general as these tools are developed. The refinements recommended by this committee would help boost the ability of actions taken under the Frank R. Lautenberg Chemical Safety for the 21st Century Act to advance the mission of EPA: “to protect human health and the environment.”