2
Model Use in the Environmental Regulatory Decision Process

Regulatory model use at EPA can be contentious. Decisions based on model results might have important public health or environmental consequences and impose substantial costs. Like other aspects of regulation, models are used and evaluated within an environment of legislative requirements, regulatory review, extensive comment by interest groups and other federal agencies, and legal challenge. Within this environment, the development, maintenance, and use of models diverge in important ways from research modeling in academia or nonregulatory modeling in the public and private sectors.

In spite of the challenges, the use of computational models within the regulatory decision process at EPA is a continually growing practice. This growth is in response to greater demands for quantitative assessment of regulatory activities, including analysis of how well environmental regulatory activities fulfill their objectives and at what cost. Models are essential for estimating a variety of relevant characteristics—including pollutant emissions, ambient conditions, and dose—when direct observation would be inaccessible, infeasible, or unethical. Finally, models allow regulators to move away from technology-based regulations that do not use quantitative analysis for assessing their benefits. This chapter describes the diversity of model use at EPA and the current integration of models into its regulatory policies. It highlights how EPA regulatory model use is influenced by legislative mandates and executive orders as well as oversight from the courts and outside participants.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 40
Models in Environmental Regulatory Decision Making 2 Model Use in the Environmental Regulatory Decision Process Regulatory model use at EPA can be contentious. Decisions based on model results might have important public health or environmental consequences and impose substantial costs. Like other aspects of regulation, models are used and evaluated within an environment of legislative requirements, regulatory review, extensive comment by interest groups and other federal agencies, and legal challenge. Within this environment, the development, maintenance, and use of models diverge in important ways from research modeling in academia or nonregulatory modeling in the public and private sectors. In spite of the challenges, the use of computational models within the regulatory decision process at EPA is a continually growing practice. This growth is in response to greater demands for quantitative assessment of regulatory activities, including analysis of how well environmental regulatory activities fulfill their objectives and at what cost. Models are essential for estimating a variety of relevant characteristics—including pollutant emissions, ambient conditions, and dose—when direct observation would be inaccessible, infeasible, or unethical. Finally, models allow regulators to move away from technology-based regulations that do not use quantitative analysis for assessing their benefits. This chapter describes the diversity of model use at EPA and the current integration of models into its regulatory policies. It highlights how EPA regulatory model use is influenced by legislative mandates and executive orders as well as oversight from the courts and outside participants.

OCR for page 40
Models in Environmental Regulatory Decision Making REGULATING WITHOUT COMPUTATIONAL MODELS Although models are essential tools if regulators are to be able to predict the risks or the effects of their regulations on the natural and human environment, models are neither necessary nor sufficient to produce the regulations themselves. In the 1970s, when the legislative framework underlying most of today’s environmental policy was first established, few sophisticated computational environmental models—models designed to predict the environmental consequences of human activity—existed. Moreover, the monitoring networks capable of quantitative description of the state of the environment were rudimentary, and the technology for measurement of pollutant discharges of various kinds and their environmental effects were much less developed than today’s technology. It was in this setting that most modern environmental regulatory statutes first appeared, including the Clean Air Act (CAA) of 1967, the Federal Water Pollution Control Act of 1972 and renamed the Clean Water Act (CWA) in 1977, and the Safe Drinking Water Act of 1976. Regulatory designs at the time necessarily minimized the use of computational models in the regulatory process. The models that did exist played little role in that process because the new environmental statutes emphasized the use of technology-based pollution discharge regulation. Technology-based regulation requires polluters to adopt a particular technology (or, in some cases, achieve a level of performance associated with a particular technology) without regard to the potential or actual environmental improvements that would result. Even before the implementation of the federal environmental statutes, technology-based regulation partly relied on there being some level of pollution abatement practiced by at least some plants in most industries. EPA was to find those plants and set a performance standard for all plants that was based in some way on what most plants were doing. Usually the congressional mandate involved the use of the words “best technology,” and it was left to EPA to interpret and give operational meaning to the various designations of “best.” For example, industrial water pollutant dischargers had to meet “best practicable treatment” (BPT) technology standards by 1977 and “best available treatment economically achievable” (BATEA or, more often, BAT) standards by 1983. In industries such as food processing and laundries that generated wastewater that resembled domestic waste (in constituents if not in strength), the usual interpretation of BPT was a performance standard that approximated what good secondary (biological) treatment could do. For other

OCR for page 40
Models in Environmental Regulatory Decision Making industries, BPT was often defined as the “average of the best” plants in the industry identified as having wastewater treatment in place. For BAT, the standard was the “best of the best,” at least until the CWA Amendments of 1977, which redirected BAT toward the control of toxic pollutants in wastewater. It should be noted that technology-based standards are not the only policy instrument that makes it possible to regulate without having to predict the environmental effects of environmental regulations. Indeed, the need to predict the consequences of regulations depends not on the policy instrument but on the policy goal. If the goal is to achieve a level of emissions reductions rather than environmental quality, there is no need to inquire into environmental effects, regardless of policy instrument. Other environmental policies proposed in the early 1970s shared that property, including several proposals using economic incentives.1 Like technology-based standards, none of these proposals had an environmental objective beyond the notion that a reduction in effluent discharges would be an improvement and that policies could be fine-tuned later, when scientists had collected more data and achieved a better understanding of environmental processes. Although the CAA and CWA of the 1970s (as well as other environmental statutes) made extensive use of technology-based standards, it would be misleading to leave the impression that their regulatory arsenals were not limited to such standards. Both statutes also had explicit environmental goals, measurement criteria for determining when the goals were met, and timetables for meeting them. For example, the National Ambient Air Quality Standards (NAAQS) in the 1970 CAA focus on reducing air pollutant concentrations to levels that are protective of human health and public welfare. This legislation required states to develop state implementation plans (SIPs), which are subject to EPA approval. Such approval was contingent on whether the plans, when implemented, would reduce emissions enough to allow the ambient standards to be met. EPA would come to base these SIP approval decisions on emission-inventory models linked to air quality models. In a similar manner, the CWA specified further regulatory action in “water-quality-limited” waters, where the imposition of the technology-based 1 In 1970, a tax on sulfur emissions as a partial alternative to some of the air quality regulation then under consideration in Congress was proposed by President Nixon. In November 1971, an effluent-charge amendment to clean water legislation then under consideration was offered and debated in the Senate (Kelman 1982; Kneese and Schultze 1975).

OCR for page 40
Models in Environmental Regulatory Decision Making standards was considered insufficient to achieve water quality standards. Eventually, that section of the CWA gave rise to the “total maximum daily load” program. Technology-based regulation proved to be a crude approach to pollution abatement policy. Moreover, it did not ultimately relieve Congress and EPA of the need for models to assess whether abatement policies were sufficient to achieve ambient goals. However, at a time when few models were available for linking pollution abatement to environmental improvement, technology-based standards provided a basis for regulating pollutant discharges that did not require knowledge of what the effects of such regulation would be. Today technology-based regulations are still in use, primarily in circumstances in which data and models do not yet permit an adequate assessment of the effects of regulation on environmental or health end points and in which other approaches have failed to generate regulations (these two situations overlap substantially). For example, Title III of the 1990 CAA Amendments changed the primary focus of hazardous air pollutant (HAP) regulation from a risk-based approach to a two-step process, where the primary focus has been on a technology-based approach to mandate promulgation of emissions standards for sources based to some extent on maximum achievable control technologies (MACT), followed by a residual risk assessment. In the preceding regime, regulators made little progress in producing regulations, largely because the inadequacies of data and models linking emissions of HAPs to adverse health effects. The current approach directs EPA to develop a MACT standard for each industrial source category, defined in part by high emissions of listed pollutants. Since 1993, EPA has promulgated over 100 MACT regulations (for the list, see EPA 2006b). After a MACT has been applied, EPA is to perform a residual risk assessment to evaluate the adequacy of the MACT, which might require additional controls if significant risks still exist. REGULATORY MODEL CLASSIFICATIONS There are many ways to classify the regulatory models used by EPA, each with its own perspectives and particular advantages and disadvantages. Two broad categorizations are used here: (1) a functional perspective that categorizes models based on their representation of scientific and other processes that translate human activities and natural systems interactions into environmental impacts and (2) a regulatory perspective that categorizes models based on how they are used in environ-

OCR for page 40
Models in Environmental Regulatory Decision Making mental regulation. In short, we see these as attempting to represent how an environmental scientist, engineer, or economist might see model use and how a regulator or stakeholder might see model use. In presenting a science perspective and a regulatory perspective, the committee acknowledges that the user community for environmental regulatory models is diverse, and a single perspective on model classification is not possible. More perspectives provide insights into model use, insights that are not possible from a single perspective. Looking at the functions of models as representing different environmental and human processes helps to emphasize the role of individual models and the need to integrate across multiple models for many regulatory activities. Looking at models from the perspective of their role in a complex regulatory setting helps to make clear the role of legislation and regulation in determining modeling objectives and the separate modeling responsibilities for EPA, state, and local governments. Given the wide range of model applications and large number of models used in environmental regulation, the committee does not attempt to present an inventory of models used by EPA. The most exhaustive inventory with descriptions of individual models is EPA’s Council on Regulatory Environmental Models (CREM) (EPA 2006c), although many other web sites are devoted to describing various programs’ modeling initiatives (see Table 2-1). CREM’s knowledge-base documents more than 100 models used by various offices at the agency. It is the single best, although incomplete, inventory of models at EPA. The information available on each model includes user information on obtaining and running the model and model documentation, including conceptual basis, scientific details, and results of evaluation studies. A full review of the knowledge base has recently been completed by EPA Science Advisory Board and is beyond the committee’s charge (EPA 2006d). However, we note that additions to the CREM knowledge base have ceased since 2004, with the exception of several climate change models that were added in 2006. For the knowledge base to reach its full capability, it needs to be updated continually and to include all types of models used at EPA, including those in the health risk assessment field. Regulatory Models from a Functional Perspective In this section, we discuss models categorized according to how they fit into a description of the processes that translate human activities

OCR for page 40
Models in Environmental Regulatory Decision Making and natural systems interactions into environmental impacts. Figure 2-1 shows an illustration of the pathways from activities to emissions to impacts. In the figure, individual components simulate the relationships between human activities and emissions, emissions and concentrations, concentrations and exposures, and exposures and impacts. It also indicates the feedback of impacts on human activities and natural processes. Appendix C provides examples of specific models from the model categories. The figure provides an approximate categorization of how computational models used in environmental analysis have historically been grouped, in particular, in economic, environmental, and human health models. This perspective allows for the identification of particular types of models and the linkages among these models. Each box is highly aggregated and could be expanded into a diagram of sub-boxes. An example of how this aggregate representation might be represented in more detail will be discussed with respect to human health risk assessment in a later section. The categories of models that are integral to environmental regulation include activity models, natural and anthropogenic emissions models, fate and transport models, exposure models, dose models, human health models, environmental and ecosystem impact models, and economic impact models. Although the categories of models shown in TABLE 2-1 Examples of EPA’s Web Sites Containing Model Descriptions for Individual Programs National Exposure Research Laboratory Models Web Site http://www.epa.gov/nerl/topics/models.html Atmospheric Sciences Modeling Division Web Site http://www.epa.gov/asmdnerl/index.html Office of Water’s Water Quality Modeling Web Site http://www.epa.gov/waterscience/wqm Center for Subsurface Modeling Support Web Site http://www.epa.gov/ada/csmos.html National Center for Environmental Assessment’s Risk Assessment Web Site http://cfpub.epa.gov/ncea/cfm/nceariskassess.cfm?ActType=RiskAssess National Center for Computational Toxicology Web Site http://www.epa.gov/ncct

OCR for page 40
Models in Environmental Regulatory Decision Making FIGURE 2-1 Basic modeling elements relating human activities and natural systems to environmental impacts. Figure 2-1 are not specific to environmental media, the models that fit into each category tend to be further subdivided by media. For example, the generic category of environmental fate and transport models can be subdivided further into various types of subsurface containment transport models, surface-water quality models, and air quality models (Schnoor 1996; Ramasawami et al. 2005). Scope of Regulatory Model Applications Table 2-2 contains short descriptions of some of EPA’s regulatory activities that rely on modeling. These environmental regulatory modeling activities typically occur as a subset of the full system summarized in Figure 2-1. The underlying statutory requirements, the regulations implementing the statutory requirements, and the importance of the activity dictate the nature of the modeling analysis. For example, assessing the toxicity of new pesticides and other chemicals in the environment may focus on just the fate and transport or toxicity portion of the system. Assessing the risks from leaking underground petroleum storage tanks,

OCR for page 40
Models in Environmental Regulatory Decision Making TABLE 2-2 Examples of Major EPA Documents That Incorporate a Substantial Amount of Computational Modeling Activities Air Quality Criteria Documents and Staff Papers for Establishing NAAQS Summarize and assess exposures and health impacts for the criteria air pollutants (ozone, particulate matter, carbon monoxide, lead, nitrogen dioxide, and sulfur dioxide). Criteria documents include results from exposure and health modeling studies, focusing on describing exposure-response relationships. For example, the particulate matter criteria document placed emphasis on epidemiological models of morbidity and mortality (EPA 2004c). The Staff Paper takes this scientific foundation a step further by identifying the crucial health information and using exposure modeling to characterize risks that serve as the basis for the staff recommendation of the standards to the EPA Administrator. For example, models of the number of children exercising outdoors during those parts of the day when ozone is elevated had a major influence on decisions about the 8-hour ozone national ambient air quality standard (EPA 1996). State Implementation Plan (SIP) Amendments A detailed description of the scientific methods and emissions reduction programs a state will use to carry out its responsibilities under the CAA for complying with NAAQS. A SIP typically relies on results from activity, emissions, and air quality modeling. Model-generated emissions inventories serve as input to regional air quality models and are used to test alternative emission-reduction schemes to see whether they will result in air quality standards being met (e.g., ADEC 2001; TCEQ 2004). Regional scale modeling has become an integral part of developing state implementation plans for new 8-hour ozone and fine particulate matter standards. States, local governments, and their consultants do this analysis. Regulatory Impact Assessments for Air Quality Rules RIAs for air quality regulations document the costs and benefits of major emission-control regulations. Recent RIAs have included emissions, air quality, exposure, and health and economic impacts modeling results (e.g., EPA 2004b). See Box 2-3 for a further discussion of the RIA. Water Regulations Total Maximum Daily Load (TMDL) Determinations For each impaired water body, a TMDL documents a state-designated water quality standard need to meet a designated use for that water body and the amount by which pollutant loads need to be reduced to meet the standard. TMDLs utilize water quality and/or nutrient loading models. States and their consultants do the majority of this modeling, with EPA occasionally doing the modeling for particularly contentious TMDLs (EPA 2002b; George 2004; Shoemaker 2004; Wool 2004). Leaking Underground Storage Tank Program Assesses the potential risks associated with leaking underground gasoline storage tanks. At an initial screening level, it may assess only one-dimensional transport of a conservative contaminant using an analytical model (Weaver 2004).

OCR for page 40
Models in Environmental Regulatory Decision Making Development of Maximum Contaminant Level for Drinking Water Assess drinking water standards for public water supply systems. Such assessments can include exposure, epidemiology, and dose-response modeling. (EPA 2002c; NRC 2001b, 2005b). Pesticides and Toxic Substances Programs Pre-manufacturing Notice Decisions Assess risks associated with new manufactured chemicals entering the market. Most chemicals are screened initially as to their environmental and human health risks using structure-activity relationship models. Pesticide Reassessments Requires that all existing pesticides undergo a reassessment based on cumulative (from multiple pesticides) and aggregate (exposure from multiple pathways) health risk. This includes the use of pesticide exposure models. Solid and Hazardous Wastes Regulations Superfund Site Decision Documents Includes the remedial investigation, proposed plan, and record of decision documents that detail the characteristics and cleanup of Superfund sites. For many hazardous waste sites, a primary modeling task is utilizing groundwater modeling to assess the movement of toxic substances through the substrate (Burden 2004). The remedial investigation for a mining megasite might include water quality, environmental chemistry, human health risk, and ecological risk assessment modeling (NRC 2005a). Human Health Risk Assessment Benchmark Dose (BMD) Technical Guidance Document EPA relies on both laboratory animal and epidemiologic studies in assessing the noncancer effects of chronic exposure to pollutants (that is, the reference dose [RfD] and the inhalation reference concentration, [RfC]). These data are modeled to estimate the human dose-response. EPA recommends the use of BMD modeling, which essentially fits the experimental data to use as much as the available data as possible (EPA 2000). Guidelines for Carcinogen Risk Assessment The cancer guidelines set forth a revised set of recommended principles and procedures to guide EPA scientists and others in assessing the cancer risks resulting from exposure to chemicals or other agents in the environment. One of the principal advancements was to describe approaches that consider mode-of-action data, if available, in the quantitative assessment. The guidelines are also used to inform agency decision makers and the public about risk assessment procedures (EPA 2005a). Ecological Risk Assessment Guidelines for Ecological Risk Assessment The ecological risk assessment guidelines provide general principles and give examples to show how ecological risk assessment can be applied to a wide range of systems, stressors, and biological, spatial, and temporal scales. They describe the strengths and limitations of alternative approaches and emphasize processes and approaches for analyzing data rather than specifying data collection techniques, methods, or models (EPA 1998).

OCR for page 40
Models in Environmental Regulatory Decision Making especially during initial assessments, focuses solely on the fate-transport component. The SIP process, which involves extensive emissions and air quality modeling, stops at simulating atmospheric concentrations of air pollutants. Ideally, regulations would be informed by understanding the whole of the paradigm, shown in Figure 2-1, from human activities through adverse outcomes. However, only the most important regulatory assessments, such as some of those done for federal rules that have major economic impacts, include a simulation of processes from activity to health impacts. These are the rules that generate most of the benefits and costs of environmental regulation, and the modeling effort can be enormous. A recent example of such an analysis is the regulatory impact assessment (RIA) for the control of air pollutant emissions from nonroad diesel engines (EPA 2004b). Even the extensive modeling that accompanied this rule cannot quantitatively consider all aspects of the problem. For example, in discussing behavioral responses to increasing costs for nonroad diesel engines, stakeholders suggested that equipment users may substitute different equipment (gasoline engines) or even labor (the use of a laborer and shovel instead of a backhoe) for more expensive diesel engines (EPA 2004b). Such behavioral aspects were only discussed qualitatively in the report. Incorporating behavior into environmental regulatory models is discussed more generally in Box 2-1. Linkages among the different processes are not seamless. Each category often is represented by a separate model and regulatory analyses often require that inputs and outputs of one model interface with other models in separate categories. Sometimes temporal or spatial scales do not line up and results from one model may not have natural counterparts in the models with which it interfaces. An example is from the air quality analysis in which emissions from vehicles and other sources that are estimated at the regional level must be allocated spatially and estimates of aggregated hydrocarbon emissions must be disaggregated by species for input into the air quality model. More fundamentally, the linking of these different categories means the linking of separate disciplines. To properly link different modeling categories requires the building of interdisciplinary bridges, which is an ongoing effort at EPA. Although there are software tools and integrated models that allow multiple processes to be combined into a single modeling framework as discussed in a subsequent section, such a model still faces the difficulty of needing to rely on the expertise from multiple disciplines. The level of effort dedicated to environmental regulatory applications varies greatly. This variation is a critical consideration when

OCR for page 40
Models in Environmental Regulatory Decision Making BOX 2-1 Incorporating Human Behavior into Environmental Models For regulatory purposes it is important to not only model natural systems but also human activities and their interactions with natural systems. These interactions, which can always be found at either end of the causal chain, shown in Figure 2-1, and often in the middle as well, require models from the social sciences, usually economics. A key modeling consideration is the extent to which such models incorporate human behavior. The earliest models used for environmental regulatory purposes had little if any behavioral content. The effects of both regulations and environmental changes were estimated without considering the full range of responses available to economic agents—individuals, households, and firms. One of the first models to demonstrate that possible behavioral responses could affect the costs or effectiveness of regulations was developed by Gruenspecht (1982), who pointed out that the common regulatory practice of requiring more stringent and more costly abatement for new sources of pollutants than for existing sources could retard the turnover of existing equipment. Behavioral responses are sensitive to the details of regulatory design, and numerous models appeared in the economics literature describing the unintended consequences of such real-world policies as CAFE (Kwoka 1983) and vehicle inspection and maintenance (Hubbard 1998). Behavioral responses also affect other outcomes of interest to EPA, including regulatory enforcement (Harrington 1989), pollution abatement subsidies (Freeman 1978; Rubin 1985). Behavioral responses to adverse environmental consequences, such as private defensive expenditures, have also been analyzed. For many years, EPA made frequent use of behavioral models for policy analysis and regulatory impact analysis. In cases involving economic incentives, behavioral models are essential because the behavioral response is what drives the policy outcome. For example, analysis of proposed emissions cap-and-trade policies to control airborne sulfur dioxide emissions from the electric power industry requires the agency to predict the behavior of utilities in the permit market. For this task, EPA uses the integrated planning model, a proprietary dynamic linear programming model that determines the least-cost loading of generating capacity to meet electricity demand. The optimization simulates the expected outcome in the permit market. Not all of EPA’s regulatory models that could incorporate behavioral responses to regulation do. For example, the MOBILE model, which projects average regional or national motor vehicle-emission rates under a variety of regulatory design parameters, does not consider the effects that regulatory alternatives might have on fleet composition or vehicle use through their effects on vehicle or fuel prices. MOBILE’s failure to anticipate behavioral responses to regulation has been most noticeable in the motor vehicle emissions inspection and maintenance program (I/M) component, which has underestimated the ability of motorists to avoid I/M tests altogether and overestimated the ability of those tests to identify high-emitting vehicles as well as the effectiveness of vehicle repair (e.g., NRC 2001a; Holmes and Cicerone 2002, and references therein).

OCR for page 40
Models in Environmental Regulatory Decision Making flicts of interest and the need for a panel to balance biases. OMB’s guidance has greater emphasis on the need to make key elements of the review available to the public. The EPA Science Inventory keeps a list of the different science activities and their required levels of peer review. Its activities are broad and described at http://cfpub.epa.gov/si/si_pr_agenda.cfm. The guidance on regulatory models calls for reviews with the goals of “judging the scientific credibility of the model including applicability, uncertainty, and utility (including the potential for misuse) of results, and not for directly advising the agency on specific regulatory decisions stemming in part from consideration of the model output” (EPA 1994c). Box 2-4 lists elements of peer review described by EPA for use with regulatory models. This guidance also offers a framework for reviewing model development, model application, and environmental regulatory decision making. It explains that policy decisions resulting from the science and other factors are required by law to be made by EPA decision makers. The policy decisions are often subject to public comment. BOX 2-4 Elements of External Peer Review for Environmental Regulatory Models Model Purpose/Objectives What is the regulatory context in which the model will be used and what broad scientific question is the model intended to answer? What is the model’s application niche? What are the model's strengths and weaknesses? Major Defining and Limiting Considerations Which processes are characterized by the model? What are the important temporal and spatial scales? What is the level of aggregation? Theoretical Basis for the Model—formulating the basis for problem solution What algorithms are used within the model and how were they derived? What is the method of solution? What are the shortcomings of the modeling approach? Parameter Estimation What methods and data were used for parameter estimation? What methods were used to estimate parameters for which there were no data? What are the boundary conditions and are they appropriate? Data Quality/Quantity Questions related to model design include: What data were utilized in the design of the model?

OCR for page 40
Models in Environmental Regulatory Decision Making How can the adequacy of the data be defined taking into account the regulatory objectives of the model? Questions related to model application include: To what extent are these data available and what are the key data gaps? Do additional data need to be collected and for what purpose? Key Assumptions What are the key assumptions? What is the basis for each key assumption and what is the range of possible alternatives? How sensitive is the model toward modifying key assumptions? Model Performance Measures What criteria have been used to assess model performance? Did the data bases used in the performance evaluation provide an adequate test of the model? How does the model perform relative to other models in this application niche? Model Documentation and Users Guide Does the documentation cover model applicability and limitations, data input, and interpretation of results? Retrospective Does the model satisfy its intended scientific and regulatory objectives? How robust are the model predictions? How well does the model output quantify the overall uncertainty? Source: EPA 1994c. EPA has several forums to conduct peer reviews: the EPA Science Advisory Board (SAB), the EPA Clean Air Science Advisory Committee (CASAC), the EPA Science Advisory Panel (SAP), or ad hoc committees. They are described in more detail in Box 2-5. The first three organizations are convened under the Federal Advisory Committee Act and are subject to requirements of that act, including that all meetings and deliberations must be public. Major ad hoc committees also hold open meetings. Typically, the charges to SAB, CASAC, and SAP are broad. Ad hoc committees are often used for more in-depth reviews. All types of peer review are of substantial value, but the adequacy of peer review of a model must be judged in context with the need for evaluation of each major step from model conception to application. Major reviews, such as those performed by SAB, besides providing valuable input to agency scientists and managers, can become a part of the administrative record and can be used in court challenges. Examples of model peer reviews are the SAB reviews of the 3MRA model (EPA 2004e), the SAB review of the EPA Region 5 critical ecosystem assessment model (EPA

OCR for page 40
Models in Environmental Regulatory Decision Making BOX 2-5 The Different Types of Science Advisory Panels at EPA The CASAC was established under the CAA to review EPA’s NAAQS and report to the EPA administrator. It is administratively housed in SAB. This group reviews the “criteria documents” of the criteria air pollutants to evaluate whether the information contained is adequate to support a decision. They also review the staff paper that has the EPA staff’s recommendations for the standard. Both documents rely on models. SAB traces its history to 1978. Its charge is to provide independent science and technical advice, consultation, and recommendations to the EPA administrator on the technical bases for agency positions and regulations. Most of its activities involve reviewing technical documents, including numerous model reviews (e.g., EPA 2004e, 2005b). SAB also produced the Resolution on the Use of Mathematical Models by EPA for Regulatory Assessment and Decision-Making (EPA 1989). The federal Insecticide, Fungicide, and Rodenticide Act established SAP in 1975. The Food Quality Protection Act mandated a science review board of scientists who would be available to SAP on an ad hoc basis. SAP provides scientific advice, information, and recommendations to the EPA administrator on pesticides and pesticide-related issues as to the impact of regulatory actions on health and the environment. Several SAP panels have focused on models to predict exposures to pesticides or on pesticide health assessments that were partly based on models. SAP panels summarize their discussions and issue recommendations in the minutes of the meetings (e.g., EPA 2005c). Ad hoc committees are often used by EPA when the document being reviewed does not have the impact that invokes the need for SAB, CASAC, or SAP. As related to models, they might involve highly technical reviews before the SAB-level stage or might be for risk assessments that include some degree of reliance upon models. 2005b), and the SAP preliminary evaluation of physiologically based pharmacokinetic and pharmacodynamic modeling for the N-methyl carbamate pesticides (EPA 2005c). Public Review Public review of a regulatory model concerns review and comments by stakeholders during the public comment periods of external peer review activities or during the “notice and comment” period that accompanies rule-making activities. Herein, “stakeholder” is defined as a person or nonfederal entity and external to the agency not involved in the above-described peer review. They include members of the general public. Thus, many individuals and entities are stakeholders and have different interests, capabilities, and capacities to perform this role. For

OCR for page 40
Models in Environmental Regulatory Decision Making example, consider the different capabilities to generate comments on models and model results between a member of the general public with limited abilities to perform computational analysis and a corporation or other organization with a substantial scientific staff. These differences need to be understood and accommodated when fulfilling the intent and actual requirements for public review. When EPA requests a peer review by CASAC, SAB, or SAP, the document is made public, and the public is able to comment at the public meetings of these organizations as per the Federal Advisory Committee Act. Furthermore, EPA is required by statute to solicit comments from affected parties and the public at large on all final proposals for agency action (5 U.S.C. § 553). A mandatory “notice and comment” process is intended to ensure that the agency informs the public of its activities and takes their concerns and input into account. According to statute, EPA must also make all relevant documents in the record supporting its decision available to the public for viewing during the comment process. Interagency Review EPA’s regulations are developed and implemented as part of a larger federal fabric. For example, some of EPA’s regulations affect other agencies directly (for example, Department of Defense Superfund sites) and indirectly (for example, economic consequences to policies of other agencies). A example of an EPA model that plays a critical role in another agency’s activities is the motor vehicle emissions factor model, (MOBILE), which plays an important role in the Department of Transportation (DOT) transportation planning activities (Ho 2004). This has inspired DOT to evaluate aspects of MOBILE directly (Tang et al. 2003a,b). Thus, there is a variety of both formal and informal processes for interagency review of regulatory models and analysis based on these models. The majority of interagency reviews involve mandatory oversight by OMB, although other agencies may also engage in more informal review and comment. Under various executive directives, OMB review is generally cursory unless the regulatory program, which the model informs, is deemed to be “significant” with respect to its economic implications (Graham 2004). OMB oversees these process requirements and will work with the agencies to ensure that their regulatory analyses are satisfactory. OMB review of other agencies’ rule-makings is generally established through executive order and, while these presidential

OCR for page 40
Models in Environmental Regulatory Decision Making directives are mandatory, agency violations cannot be enforced through the courts. Completing the Review Cycle Several of the processes of external reviews are still not transparent in regard to the disposition of the comments. In some instances, the effect of comments on the regulatory process is not clear. It is understood that not all comments are appropriate or useful, even though all need to be carefully considered. Thus, the issue is transparency—those commenting, from prominent scientists on the SAB to members of the general public, need to understand how their comments were considered. EPA’s Peer Review Handbook (EPA 2006a) discusses this issue and calls for a written record of response to comments. EPA has an exemplary process in terms of transparency for the NAAQS where a public docket contains both the original comment and the agency’s responses. Legal Challenges to EPA’s Models Laws and executive orders not only provide a mechanism for increased external inputs to EPA’s models but also provide opportunities for adversarial challenge. There are two formal opportunities for interested parties to challenge EPA’s models. The first and most established is the ability of interested parties to challenge agency action in court. If the model supports a regulation and has been subject to notice and comment, the courts give EPA considerable deference. Thus, challenges to EPA models are successful only when the regulation (and/or underlying model) is in conflict with EPA’s statutory mandate, has been determined to be inconsistent with Administrative Procedure Act requirements, or is “arbitrary and capricious” (5 U.S.C. § 706). As one court summarized in reviewing a model: “This Court must not undertake an independent review of EPA’s scientific judgments; our inquiry focuses only on whether the agency has met the statutory requirement for ‘sufficient evidence.’” (National Oilseed Processors Ass’n v. Browner, 924 F. Supp. 1193, 1209 [D.D.C. 1996], affirmed in part and reversed in part on other grounds, Troy v. Browner, 120 F.3d 227 [D.C. Cir. 1997]). If the model has not been subject to notice and comment but creates obligations for private parties—for example, at the permitting or enforcement stage—those af-

OCR for page 40
Models in Environmental Regulatory Decision Making fected by the model can typically challenge either the model or its application in court. In some of the cases, the agency may receive much less deference from the courts compared to the situation where the model has been subject to notice and comment (for example, see United States v. Plaza Health Laboratories, Inc., 3 F.3d 643 [2d Cir. 1993]) (applying the rule of lenity, rather than deferring to EPA, in interpreting “point source” in a criminal CWA prosecution). Generally, a complete model history documenting the justification for various decisions related to model design and development may help the agency defend a model against formal challenges. EPA’s models have sometimes been challenged, and in some cases, challengers have been successful in forcing the model or its application back to EPA to correct what the courts view as fundamental flaws. Some of this judicial activity may be a result of EPA’s past, ad hoc approach to developing and using models; a more rigorous and formalized approach might ward off some of these challenges by instituting more rigorous modeling practices in the agency. For example, when EPA declines to explain its decision or revise a supporting model even after receiving comments refuting one of the model’s critical assumptions, the courts have invalidated and remanded the model back to EPA. Challengers have also been successful when they establish that EPA’s model is not applicable to a particular subset of industries, activities, or locations. If EPA applies a generic air dispersion model to a large power plant located in a meteorologically unusual setting, such as the shores of Lake Erie, EPA might have to test the location to establish that the model provides some reliability in that setting, or it must be prepared to explain why its model should be accepted as is (for example, State of Ohio v. EPA, 784 F.2d 224 [6th Cir. 1986] and 798 F.2d 880 [6th Cir. 1986]).8 Finally, if challengers disagree with embedded policy judgments, such as the risk adversity of assumptions built into a risk assessment, courts will sometimes invalidate a model and not defer to the agency (Gulf South Insulation v. Consumer Product Safety Commission, 701 F.2d 1137 [5th Cir. 1983]). However, this line of cases is more complex and unpredictable (Pierce 8 Remanding EPA’s air dispersion model because EPA had not adequately demonstrated that its CRSTER model took into account the “specific meteorological and geographic problems” of the modeled large power plants situated on the shores of Lake Erie. It was therefore arbitrary and capricious for EPA to allow a 400% increase in emissions “without evaluation, validation, or empirical testing of the model at the site.”

OCR for page 40
Models in Environmental Regulatory Decision Making 1988).9 Because these legal challenges are time-consuming and costly, they are typically mounted only when an affected or interested party stands to gain something important—whether it is gaining less stringent regulatory requirements or positive publicity for members—from a challenge. A second, more recent opportunity for external challenge to model use in the regulatory process is through the Information Quality Act (Treasury and General Government Appropriations Act for Fiscal Year 2001, Pub. L. No. 106-554, § 515, 114 Stat. 2763 [2000]), which is implemented through OMB’s Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies (OMB 2001). Some of challenges under the Information Quality Act result from EPA’s occasional ad hoc approach to developing and using models. This statutory provision allows any interested person to file “requests for correction” on “information” that is “unreliable” or lacks other qualities, such as objectivity or integrity. To date, courts have refused to review these challenges, but the challenges can be appealed inside the agency and the agency must respond to complaints that the information, including information used in models or the models themselves, is unreliable. However, there are continued efforts to make challenges under the Information Quality Act reviewable by the judiciary (Shapiro et al. 2006). Challenges filed under the Information Quality Act to date generally target technical decisions within EPA that have important economic consequences (EPA 2006i). In at least one instance—the Competitive Enterprise Institute’s (CEI’s) challenge to the climate change models used in the National Assessment on Climate Change—the challenge has been directed specifically at agency models (EPA 2003c). In the case, CEI argued that the models were not reliable and had not been adequately peer reviewed. The agencies denied the petitions and CEI’s internal appeals. CEI then appealed its case to the D.C. District Court where CEI ultimately withdrew its case. Information Quality Act challenges brought by affected parties sometimes seek correction of flaws or technical misstatements in agency documents, but in other instances, as in the Competitive Enterprise Institute’s challenge, they have requested that the agencies cease dissemination of the information. If an agency 9 Arguing that judges on the D.C. Circuit may be substituting their own interpretations of ambiguous statutes for agencies and randomly reversing agency policy making in rule-makings.

OCR for page 40
Models in Environmental Regulatory Decision Making denies a petition on appeal, as has been the case for most IQA challenges filed to date, the challenge fails. THE CHALLENGES OF MODELING IN A REGULATORY ENVIRONMENT This chapter has described the types of models used in EPA’s regulatory activities, how models fit into the regulatory process, and legal and other constraints governing their use. Modeling is a difficult enterprise even when it is not being conducted in an adversarial regulatory environment. Further, the range of model applications is vast, and many agencies and stakeholders are involved in producing analysis. When the demands of regulatory accountability, transparency, public accessibility, and technical rigor are added to the challenges typically encountered in modeling, the task becomes much more complex. Although improvements to EPA regulatory modeling efforts are possible, EPA clearly has made important advances in the science of environmental modeling and has been a global leader in using models in the environmental regulatory decision process. However, future regulatory modeling activities will be challenged by new scientific understandings, expanding sources of environmental and human observations, and new issues. To meet the challenges, continued improvement in model practices will be required. In this chapter, the committee offers recommendations related to continuing improvements to the accessibility of regulatory modeling. Later in this report, we offer recommendations related to model evaluation; principles for model development, selection, and application; and model management. Model Goals Models are used in regulatory settings when EPA determines that a model will be useful in reaching or enforcing a regulatory decision. Given the diversity of regulatory aims and targets, however, a wide variety of models and modeling goals can exist. At one extreme, the agency can use a model that provides the best technical analysis of the concentrations of ambient air pollutants and resulting health and environmental impacts most likely to result from combined industrial and nonindustrial emissions controls. This precision is desired because of the enormous compliance costs associated with emissions controls and the enormous

OCR for page 40
Models in Environmental Regulatory Decision Making health costs if air pollutants are not correctly estimated and exceed allowable levels. At the other extreme, EPA might want to use a model that provides only a crude and inexpensive prediction for a system. The regulatory environment also creates the opportunity for many different types of legal constraints on modeling that are foreign in non-regulatory settings. Congress may instruct, for example, that a regulation err on the side of over-predicting public health harms. Other constraints might result from legislative mandates that EPA develop and use models in situations where resources, including both time and financial support, are scarce. Time and resource limitations can also lead EPA to use existing models outside their “application niche,” a set of conditions for which the model is designed to be useful. There is some evidence, for example, that EPA and other agencies have sometimes used a model in a setting where the model no longer provides useful guidance. For example, EPA’s generic test to predict the toxicity of wastes in landfill settings (the Toxicity Characteristic Leaching Potential Test) generally adopts worst-case assumptions. Yet, in some disposal settings, the worst-case assumptions have been challenged successfully as inapplicable for specific types of disposal operations, such as for the disposal of a particular type of waste (potliner waste) in a monofill (for example, see Columbia Falls Aluminum Co. v. U.S. Environmental Protection Agency, 139 F.3d 914 [D.C. Cir. 1998]; Edison Electric Institute v. U.S. Environmental Protection Agency, 2 F.3d 438 [D.C. Cir. 1993]; and Association of Battery Recyclers, Inc. v. U.S. Environmental Protection Agency, 208 F.3d 1047 [D.C. Cir. 2000]). Technical Reliability The sometimes contentious environment for regulatory models also creates important impediments for ensuring the technical reliability of EPA’s models. Formal evaluation processes required by administrative law may deter meaningful model reevaluation and adjustment over time. Once a regulatory action has survived the multilayered review and challenge processes, it may remain in place for some time. Indeed, rule-making requirements can be read to require that the agency undergo notice and comment and the risk of judicial review every time it revises a model that supports a rule-making, since it must ensure that there has been “meaningful public comment” on all aspects of its final rule (for

OCR for page 40
Models in Environmental Regulatory Decision Making example, see Small Refiner Lead Phasedown Task Force v. EPA, 705 F.2d 506, 540-41[D.C. Cir. 1983]). This inertia is not ideal for any regulatory decision, but it is particularly unfortunate for models. The cumbersome regulatory procedures and the finality of the rules that survive them are directly at odds with the dynamic nature of modeling and the goal of improving models in response to experience. Although some stakeholders may prefer a constant model because of the stability it provides, this model might not reflect the most updated science. Transparency and Accountability In the regulatory environment, EPA has the responsibility to ensure that a model’s development and use is transparent. Because modeling is often a very technical exercise, EPA faces a challenge in making all of the underlying decisions intertwined within a model intellectually accessible to a nontechnical audience. A model that attempts to determine the fate of a chemical in soil, for example, may involve choices between competing assumptions, such as the percolation rate of a chemical at a particular location. Selection of the most appropriate assumption in some cases may depend not only on technical judgment but also on the policy goals of the modeling effort. A recent EPA report documents how science mingles with policy in health risk assessment (EPA 2004a). If the model is supposed to err on the side of protecting health and the environment, the model may need to err on the side of quicker percolation rates when several choices are plausible. Making these choices explicit and accessible is a challenge because policy judgments can be numerous and varied in their importance. Nevertheless, administrative processes expect EPA to make many of these types of judgments and technical decisions transparent so that affected stakeholders and the general public can comment on the model and its regulatory implications. Because models are uncertain and are used to make policy, stakeholders necessarily play a vital role in EPA’s development, use, and evaluation of models. Differing interpretations of risk, risk preferences, and a range of other values and understandings mean that a broad array of participants will have much to add to the modeling exercise. As a result, these various constituencies and individuals must be able to participate in the model evaluation process through various activities, including producing their own supporting or conflicting model results, and challenging the legitimacy or accuracy of a model in public comments or judicial actions.

OCR for page 40
Models in Environmental Regulatory Decision Making Clearly, EPA faces many difficult challenges in making its models, particularly its complex models, accessible to the diverse interests. Nevertheless, EPA has taken a major step in the right direction through the CREM database of models. This information further enhances the transparency and understandability of models to a wide array of interested participants. Despite these efforts, however, stakeholders with limited resources or technical expertise still face substantial barriers to being able to evaluate EPA’s models, comment on important model assumptions, or use the models in their own work. Recommendations EPA should place a high priority on ensuring that stakeholders and others have access to models for regulatory decision making. To ensure that its models database contains all actively used models, EPA should continue its support for the intra-agency efforts of CREM. A more formal process may be needed to ensure that CREM’s models database is complete and updated with information that is at least equivalent to information provided for models currently contained in the database. Yet, even with a high-quality models database, EPA should continue to develop initiatives to ensure that its regulatory models are as accessible as possible to the broader public and stakeholder community. The level of effort should be commensurate with the impact of the model use. It is most important to highlight the critical model assumptions, particularly the conceptual basis for a model and the sources of significant uncertainty. Meaningful stakeholder involvement should be solicited at the model development and model application stages of regulatory activity, when appropriate. EPA could improve model accessibility through a variety of activities, such as requiring an additional interface for each model to help to identify the assumptions and sources of parameters and other uncertainties and providing additional user and stakeholder training. However, even if full information on a model is available, technical expertise will still be required to judge independently its quality and suitability for regulatory application. Each of these recommendations requires staff time and resources, which may be considerable. Thus, EPA’s efforts to enhance opportunities for public participation in any particular case must be balanced against other agency priorities.