National Academies Press: OpenBook

Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress (2004)

Chapter: 2 Committee's Approach to Evaluation of Research Progress

« Previous: 1 Introduction
Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×

2
Committee’s Approach to Evaluation Of Research Progress

INTRODUCTION

The Committee on Research Priorities for Airborne Particulate Matter was created in November 1997 with the charge of developing an agenda for research on airborne particulate matter (PM), monitoring the progress of the agenda, and evaluating the gains in scientific knowledge that resulted. The underlying purpose was to reduce uncertainties in the scientific basis for establishing the National Ambient Air Quality Standards (NAAQS) for PM. In establishing its research agenda, the committee used an integrated risk assessment framework that includes sources of air pollution and subsequent effects on health—a conceptual sequence that is integral to strategies for air pollution control (see Figure 1-1 in Chapter 1) (NRC 1998). This framework provides a basis for synthesizing available information and identifying uncertainties potentially amenable to resolution through future research. Using this strategic framework, the committee identified 10 key areas of uncertainties and corresponding topics of research need and described a research portfolio to obtain evidence directed at these uncertainties. In selecting the topics, the committee used three criteria: (1) scientific value, (2) decisionmaking value, and (3) feasibility and timing. The research agenda was set out in the committee’s first report (see Chapter 1). In subsequent reports published in 1999 and 2001 (NRC 1999, 2001), the committee added three criteria relevant to evaluating the planning, management, and implementation of the agenda: (1) interaction, (2) integration, and (3) accessibility.

While the committee has been operative, the U.S. Environmental Protection Agency (EPA) has been proceeding with its mandated process to review the NAAQS for PM (Table 2-1). The timetable for EPA’s review

Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×

reflects requirements of the Clean Air Act. Research complying with the committee’s portfolio, which was designed to inform both current and future decisions on standards and implementation, could not be fully sequenced with this near-term review. The divergent timetables of EPA’s immediate process and of the scientific research initiated after the committee’s first report reflect the exigencies of implementing a major research initiative and the challenges of scheduling the pace of scientific research, which inevitably has unanticipated hurdles. Typically, at least several years are needed to move from funding to publication of even the smallest study. Large toxicological and epidemiological studies of air pollution often provide published findings 5-10 years after funding is started.

The research agenda proposed by the committee is intended to have an impact on the PM NAAQS (including its four elements: the indicator, the averaging time, the statistical form, and the level), on the subsequent implementation of that NAAQS, and on future NAAQS reviews. This chapter describes the committee’s approach to evaluating research progress. Although the committee evaluates the research findings on PM on their contribution to the advancement of scientific knowledge without specific concern for EPA’s application, the committee is also mindful of the impact of the research agenda and the resulting research findings on the PM NAAQS, its implementation, and future reviews.

BACKGROUND

A key step in the use of scientific evidence for policy formulation is gauging research progress resulting in reduction of uncertainty. Committees of the National Research Council (NRC) and the Institute of Medicine (IOM) are frequently asked to evaluate evidence and reach conclusions. Examples of such evaluations include the toxicity of environmental agents, the value of specific health-care practices, and approaches to disease prevention. The NRC does not give its committees a template or a rigid process for these evaluations, although some committees have had mandated approaches. Committees of EPA’s Science Advisory Board often carry out similar evaluations, as do a myriad of committees of other agencies. Often the work of these committees involves hazard identification, reaching a discrete decision as to the potential of a particular agent or exposure circumstance to produce an increase in certain adverse health effects. In some cases, the evaluation may extend to a quantitative evaluation of the exposure-dose-response relationship for the agent.

Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×

TABLE 2-1 EPA’s Review and Implementation Timetable for Particulate Matter Standards

Past Actions

 

1971

EPA issues total suspended particles (TSP) NAAQS

1979-1987

Criteria and standards are reviewed

1987

1994-1997

EPA issues PM10 NAAQS

Criteria and standards are reviewed

1997

EPA issues PM2.5 and revised PM10 NAAQS

1999

EPA designates areas unclassifiable, regarding attainment of NAAQS for PM2.5

1998-2000

PM2.5 monitors are placed nationwide

1998-2003

PM2.5 monitoring data collected nationwide

Planned Actions

 

2004a

EPA will complete 5-year scientific review of PM2.5standards, leading to possible revision

2002-2005

EPA will designate nonattainment areas for PM2.5

2005-2008

States will submit implementation plans for meetingPM2.5 standard.

2012-2017

States will have up to 10 years and two 1-year extensions to meet PM2.5 standards

aYear may be revised by EPA.

The approaches used to evaluate an association between exposure to an agent and the occurrence of particular diseases have been variable and are still evolving. Some of the earliest attention to developing formal criteria for making such evaluations was given to evidence on cancer.

  • 1964 Report of the Surgeon General on Smoking and Health: One of the earliest evaluations of this kind was the 1964 Report of the Surgeon General on Smoking and Health (USDHEW 1964). Although this evaluation and the associated report considered multiple diseases, the primary focus—and impact—related to its conclusions on the association between smoking and lung cancer. The criteria used to evaluate the association are shown in Box 2-1.

  • International Agency for Research on Cancer (IARC): IARC initiated a program in 1971 to evaluate the carcinogenic risk of chemicals to humans (IARC 1972). The IARC program utilizes international working

Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×

BOX 2-1
Criteria Used to Evaluate the Association Between Smoking and Lung Cancer

  • Consistency of the association

  • Strength of the association

  • Specificity of the association

  • Temporal relationship of the association

  • Coherence of the association

Source: USDHW 1964.

groups of experts to conduct critical reviews and evaluate the evidence on carcinogenicity of a wide range of human exposures. The IARC panels assess the strength of the evidence that certain exposures could alter the incidence of cancer in humans. The greatest weight is given to epidemiological data but without extrapolation beyond the range of the data available. Quantitative assessments are not made. The evaluations also include consideration of the results of studies in laboratory animals. The overarching view of the IARC is that “in the absence of adequate data on humans, it is biologically plausible and prudent to regard agents and mixtures for which there is sufficient evidence of carcinogenicity in experimental animals as if they presented a carcinogenic risk to humans” (IARC 2001). Supporting data on the mechanisms by which agents may cause cancer are also considered. IARC’s categorization according to the strength of evidence is presented in Box 2-2.

  • Guidelines for Carcinogen Risk Assessment (EPA 1986): Soon after it was created, EPA was confronted with the need to develop guidelines for evaluating the carcinogenic risk of various chemicals and other agents. The guideline evolved with practice. The hazard identification facet, which addresses whether the agent has carcinogenic properties, was closely patterned after the evaluation procedure used by IARC. However, EPA, unlike IARC, could not stop with the hazard identification phase. The needs of a number of statutes required the agency to establish the carcinogenic potency (the dose-response or, most typically, the exposure-response function), which could be combined with exposure assessment to yield a

Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×

BOX 2-2
IARC’s Scheme for Categorizing an Agent, Mixture, or Exposure Circumstance

Group 1: The agent (mixture) is carcinogenic to humans. The exposure circumstance entails exposures that are carcinogenic to humans.

This category is used when there is sufficient evidence of carcinogenicity in humans. An exception is when evidence of carcinogenicity is less than sufficient in humans but sufficient in experimental animals and when evidence is strong that the agent (mixture) acts through a relevant mechanism of carcinogenicity in exposed humans.

Group 2

This category includes agents, mixtures, and exposure circumstances for which, at one extreme, the degree of evidence of carcinogenicity in humans is almost sufficient, as well as those for which, at the other extreme, there are no human data but there is evidence of carcinogenicity in experimental animals. Agents, mixtures, and exposure circumstances are assigned to either group 2A (probably carcinogenic to humans) or group 2B (possibly carcinogenic to humans) on the basis of epidemiological and experimental evidence of carcinogenicity and other relevant data.

2A: The agent (mixture) is probably carcinogenic to humans. The exposure circumstance entails exposures that are probably carcinogenic to humans.

This category is used when there is limited evidence of carcinogenicity in humans and sufficient evidence of carcinogenicity in experimental animals. In some cases, an agent (mixture) may be classified in this category when there is inadequate evidence of carcinogenicity in humans, sufficient evidence of carcinogenicity in experimental animals, and strong evidence that the carcinogenesis is mediated by a mechanism that also operates in humans. An exception is when an agent, mixture, or exposure circumstance may be classified in this category solely on the basis of limited evidence of carcinogenicity in humans.

2B: The agent (mixture) is possibly carcinogenic to humans. The exposure circumstance entails exposures that are possibly carcinogenic to humans.

This category is used for agents, mixtures, and exposure circumstances for which there is limited evidence of carcinogenicity in humans and less than sufficient evidence of carcinogenicity in experimental animals. It may also be used when there is inadequate evidence of

Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×

carcinogenicity in humans, but there is sufficient evidence of carcinogenicity in experimental animals. In some instances, an agent, mixture, or exposure circumstance may be placed in this group when there is inadequate evidence of carcinogenicity in humans but limited evidence of carcinogenicity in experimental animals together with supporting evidence from other relevant data.

Group 3: The agent (mixture or exposure circumstance) is not classifiable as to its carcinogenicity in humans.

This category is used most commonly for agents, mixtures, and exposure circumstances for which the evidence of carcinogenicity is inadequate in humans and inadequate or limited in experimental animals. An exception is when agents (mixtures) that have inadequate evidence of carcinogenicity in humans but sufficient evidence in experimental animals are placed in this category because there is strong evidence that the mechanism of carcinogenicity in experimental animals does not operate in humans. Agents, mixtures, and exposure circumstances that do not fall into any other group are also placed in this category.

Group 4: The agent (mixture) is probably not carcinogenic to humans.

This category is used for agents or mixtures for which evidence suggests a lack of carcinogenicity in humans and experimental animals. In some instances, agents or mixtures may be classified in this group when evidence of carcinogenicity is inadequate in humans, but evidence that is consistently and strongly supported by a broad range of other relevant data suggests a lack of carcinogenicity in experimental animals.

Source: IARC 2001.

quantitative assessment of risk. The controversy surrounding the entire carcinogenic risk assessment process led the agency to request the assistance of the NRC, the results of which are described below.

Other committees of the NRC and IOM have also identified systems for evidence evaluation and synthesis to inform policy processes. Several key and relevant frameworks are reviewed below as background.

  • Risk Assessment in the Federal Government: Managing the Process (NRC 1983): The goal of this report was to develop a process in

Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×

which scientific evidence could be organized, judged, and synthesized in a sound and reproducible manner; thereby, ensuring the integrity of the evidence used to inform risk management policy. That committee examined preexisting inference guidelines to identify relevant lessons applicable to risk assessment, but the focus of the report was on systematic evaluation of scientific evidence for policy determinations. Types of evidence considered by that committee included epidemiological, animal bioassay, and experimental studies along with comparisons of molecular structure. Like IARC (1972), the NRC committee viewed epidemiological studies as the most convincing evidence of risk, but animal bioassays were recognized as more commonly available and useful, as long as their results could be extrapolated to humans. All types of studies were evaluated for their strengths and weaknesses and for their contributions to risk assessment processes. Specific questions that should be used to evaluate the literature were listed for each of the four steps of the proposed risk assessment paradigm (NRC 1983, pp. 29-33). That committee concluded that risk assessment guidelines should enable the evaluation of data quality, be flexible but include default statements, allow for the incorporation of new evidence as it emerges, and result in a judgment of the overall strength of the evidence.

  • Science and Judgment in Risk Assessment (NRC 1994): Throughout the 1980s, risk assessment procedures, especially those used for carcinogenic risks, generated substantial debate. In response to this controversy, the Clean Air Act Amendments of 1990 directed the administrator of EPA to engage the NRC in a review of the methods that EPA used to estimate toxicological risk. The EPA administrator and the NRC proceeded as directed with the formation of a committee whose evaluation and findings are documented in its report. The committee largely endorsed the recommendations of the 1983 NRC report discussed above (NRC 1994). The report identified and addressed six themes that the committee viewed as cutting across the various steps of risk assessment and arising from criticisms of each step. These themes are default options (the generic approaches taken in the absence of specific scientific knowledge), data needs, validation, uncertainty, variability, and aggregation. Although all the themes are relevant to this report, it is appropriate to make special note of two themes:

  • Uncertainty: Has EPA taken sufficient account of the need to consider, describe, and make decisions in light of the inevitable uncertainty in risk assessment?

Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
  • Variability: Has EPA sufficiently considered the extensive variation among individuals in their exposures to toxic substances and in their susceptibility to cancer and other health effects?

In this report, considerable attention is given to both uncertainty and variability.

  • Veterans and Agent Orange: Health Effects of Herbicides Used in Vietnam (IOM 1994): The IOM Veterans and Agent Orange Committee was charged with evaluating the strength of the scientific evidence for associations between herbicides and a list of adverse health outcomes but not with judging the causal nature of associations found. Although the ultimate use of its work was for processing compensatory claims, the committee was not responsible for linking the results of its evaluation with policy decisions. Extensive consideration was given in its report to the rationale and structure needed to complete the task. The committee defined three key questions, identified the types of evidence needed to address them, considered both quantitative and qualitative analytical methods, developed criteria to assess the overall strength of the evidence, and defined four categories to summarize its integrated interpretation of the evidence for each health outcome.

Even though the committee built on Hill’s (1971) criteria for causality and IARC’s (1977) categories of association, it began judging the evidence from a point of neutrality: integration of the evidence moved the committee’s judgment either in a specific direction (for or against an association’s existence) or kept it neutral. Furthermore, quantitative assessments were not sufficient alone to judge the evidence, but results consistent with the qualitative analysis could strengthen the rationale for the final categorization. Key elements of the committee’s process are summarized in Box 2-3.

  • Multiple Immunizations and Immune Dysfunction (IOM 2002): The Immunization Safety Review Committee systematically reviews the scientific evidence for a series of selected vaccine safety concerns, assesses the societal importance of the issues, and provides recommendations for further action (IOM 2002). In a report on multiple immunizations and immune dysfunction, the committee articulated its study process and its framework to assess the scientific evidence of vaccines and adverse health outcomes—for example, the hypothesis that multiple immunizations can harm the developing immune system. The committee adopted the framework for causality developed by previous IOM committees (1991, 1994) (Table 2-2). However, public confusion about the term “biological

Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×

BOX 2-3
Categories of Causal Conclusions

  • No evidence

  • Evidence is inadequate to accept or reject a causal relationship

  • Evidence favors rejection of a causal relationship

  • Evidence favors acceptance of a causal relationship

  • Evidence establishes a causal relationship

Sources: IOM 1991, 1994.

plausibility” indicated that further clarification of its evaluative framework was needed. The committee recognized that Hill’s original intent in regard to the biological plausibility criterion in judging epidemiological causality

TABLE 2-2 Evaluation of the Strength of the Evidence on Agent Orange (Are herbicides statistically associated with the health outcome?)

Types of evidence

 

  • Animal models

  • Observational studies

  • Exposure studies

Criteria for strength of the evidence

 

  • Quantitative

  • Resolution

  • Uncertainty and confidence

  • Qualitative

  • Comprehensiveness

  • Neutrality

Categories of association

 

  • Sufficient evidence of an association

  • Limited, suggestive evidence of an association

  • Inadequate, insufficient evidence to determine whether an association exists

  • Limited, suggestive evidence of no association

 

Source: IOM 1994.

Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×

was to have a basis to judge the consistency of existing biological knowledge with epidemiological findings in a single study (Hill 1971). However, the goal of the committee’s comprehensive review of the literature on biological mechanisms was to determine whether a mechanism had been hypothesized only or whether there was support for a mechanism in experimental, animal, or human research. Consequently, the committee separated its evaluation of the literature on biological mechanisms from its judgment of the evidence for causality in population-based studies. The committee then combined the determination about biological mechanisms with its assessment of clinical and epidemiological data to judge whether scientific evidence supported causal relationships between multiple immunizations and immune dysfunctions. Figure 2-1 shows the committee’s approach, which proceeds from a synthesis of the evidence to conclusions based on that synthesis to recommendations for public health responses. The committee was responsible for (1) assessing both the scientific evidence and the societal importance of the issue to develop its conclusions and (2) presenting policy, research, and communication recommendations based on its conclusions. According to its charge, the committee was permitted to recommend that immunization policy be reviewed or analyzed but could not recommend changes to immunization policies.

The present committee found applicable concepts and approaches in the work of these and other NRC committees. In proposing its integrated risk assessment framework, this committee drew on the formalization of risk assessment that emerged with the 1983 NRC report Risk Assessment in the Federal Government: Managing the Process and was rearticulated in Science and Judgment in Risk Assessment (NRC 1994) and the need to identify points of uncertainty requiring default assumptions in carrying out a quantitative risk assessment emphasized in the 1994 NRC report. The committee also found the dual and parallel synthesis approach for animal and human data of the Immunization Safety Review Committee (IOM 2002) to be a valuable model. However, the present committee faced the task of evaluating progress on its research agenda and not with evidence review for the purpose of reaching a causal conclusion. In addition, present policies are largely based on evidence related to the quantitative risks of air pollution rather than on the determination that a hazard exists—the emphasis of most reports reviewed above.

EPA has a process for evaluating evidence on pollutants for which NAAQS are set (Figure 2-2). The agency faces the challenge of periodic evidence evaluation as the criteria document and staff paper are prepared for the criteria pollutants for which NAAQS are set. The NAAQS process includes three key steps. The first step is preparation of a criteria document,

Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×

FIGURE 2-1 Charge to the Immunization Safety Review Committee. Source: IOM 2002.

Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×

FIGURE 2-2 Schematic of EPA’s evaluation of evidence for setting standards for criteria pollutants. A separate process (not shown here) is used to design and implement control strategies. Abbreviation: CASAC, Clean Air Scientific Advisory Committee.

which catalogs and summarizes all the available peer-reviewed and published information on the criteria pollutant, such as PM, under consideration. The criteria document is prepared by EPA’s Office of Research and Development with major input from scientists outside the agency. No formal methodology is in place for this summarization, nor for moving from the criteria document to the staff paper, the second step.

The second step is preparation of a staff position paper, which summarizes the literature that specifically informs decisions on the four elements of a NAAQS. This document, prepared by EPA’s Office of Air Quality Planning and Standards, concludes with recommendations on the range of options for the four elements of the NAAQS. The third step is EPA’s promulgation (final decision) of the proposed NAAQS, including each of the four elements.

EPA’s Clean Air Scientific Advisory Committee (CASAC) reviews the criteria document and the staff paper, and when CASAC considers that

Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×

the documents provide a scientifically adequate review of the literature, a closure letter is sent to the EPA administrator. CASAC has the authority to review the proposed NAAQS, but has not always done so. Following establishment of the NAAQS, states are required to develop implementation plans for control strategies to achieve the NAAQS.

In its consideration of the above and other approaches, the committee did not find a specific model that could be directly transferred to its review and particularly to its evaluation of the degree of change in uncertainty. It did find useful elements in many of the processes elaborated by others. Additionally, any synthesis process adopted by the committee would need to be tailored to reflect the specifics of characterizing the association of PM with risk to human health. Among the challenging aspects of the task are the multiplicity of sources of PM, the complexity of airborne PM, the strong possibility that multiple mechanisms are relevant to disease causation, and the lack of specificity for the health outcomes of concern.

The committee noted that the evidence evaluation systems that it had reviewed did not cover a number of large scientific issues relevant to the review and implementation of a NAAQS. The evidence evaluation systems reviewed focused on classifying causality of associations between exposure to an agent and an increase in the probability of adverse health outcome in a population, thus, identifying a hazard. In both the setting of the NAAQS for PM and in the implementation of control strategies to achieve the NAAQS, there is a need to move beyond the issue of whether PM or some other pollutant is hazardous and consider the potency of PM or the other pollutant. Knowledge of exposure of a population and the potency of the pollutant are required to estimate the population risk.

In the setting of the NAAQS, EPA is required to set the standard at a level that provides an adequate margin of safety for protecting the public, including sensitive populations, from adverse health effects. Although EPA cannot consider costs or benefits in setting the NAAQS, it used risk assessments that quantify changes in health outcomes at various pollutant concentrations to inform decisions on the NAAQS. In the implementation phase, cost-benefit information can be used to select control options. Thus, information on the potency and other aspects of exposure-dose-response relationships is another consideration in evaluating research progress on PM. Some examples include characterization of the toxicity-determining characteristics of particles and quantifying, with sufficient certainty, exposure-dose-response relationships with outcome measures. The discrete characterization of associations as causal or not causal does not address these evidence synthesis examples.

Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×

THE COMMITTEE’S EVALUATIVE APPROACH

The committee accepts the premise that the ultimate goal of research on air pollution, as for any other environmental hazard, is to improve public health and secondarily to protect public welfare (Figure 2-3). Figure 2-3 sets out a sequence that begins with a research agenda that responds to major information uncertainty and ends with demonstrated benefit to public health. For air pollution, this sequence may well span years or decades, given the timing of research and the schedule for regulation and implementation. Although this framework is simplistic, the committee found it to be useful for identifying evaluation indicators on differing time scales. For this report, the committee focuses on the more proximal indicators of progress on the research agenda and the consequences of this work: funding, research initiated, publications, and extent of incorporation of findings of new research motivated by the committee’s agenda into the criteria document and staff paper.

Even within the life span of the committee, substantial research has been funded, initiated, and published. EPA’s fourth external review draft of the criteria document for PM, for example, includes 289 references in its exposure chapter, 294 references in its toxicology chapter, and 431 references in its epidemiology chapter (EPA 2003a). This research has been carried out in diverse venues, including EPA’s intramural and extramural research activities as well as academic and other research institutions funded by the National Institutes of Health and other organizations, such as the Electric Power Research Institute (EPRI) and the Health Effects Institute (HEI), with interest in the health effects of air pollution. The committee’s research portfolio was directed primarily toward EPA, but it also considered PM-related research conducted, funded, or planned by other agencies and organizations in the United States and abroad. Therefore, in evaluating available research, the committee considered relevant U.S. research supported by EPA as well as that supported by other sources and key research outside the United States initiated in the proper timeframe. For each topic, the committee surveyed

  • Expenditures by EPA and by other agencies to the extent possible. For EPA, expenditures are stratified as intramural or extramural.

  • Research Projects initiated by EPA, other agencies, and the research community in general.

  • Publications in the peer-reviewed literature during the time that the committee’s reports were expected to have had impact.

Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×

The published findings represent another point for evaluation, as they reduce the points of uncertainty that motivated their conduct. The committee is aware of publications on several of the 10 topics that reflect studies implemented as a consequence of the committee’s reports. For these studies, the committee subjectively gauged the extent to which uncertainty had been reduced. The committee also assessed the studies related to each topic quantitatively and qualitatively: (1) Have relevant studies been published? (2) How many? (3) Are the findings consistent? (4) Is sufficient information available to explain any substantial discrepancies in the findings? and (5) To what extent has uncertainty been reduced? The committee used these five criteria as a yardstick within each topic to gauge reduction of uncertainty, in addition to its overall judgment of the degree of reduction in uncertainty (see Chapter 3).

To facilitate this aspect of its evaluation, the committee carried out workshops during the year before this report was prepared, bringing together expert panels in the general areas corresponding to its research topics: exposure assessment (topics 1 and 2), sources and models (topics 3 and 4), and epidemiology and toxicology (topics 5, 6, 7, 8, 9, and 10). In these workshops, researchers and others who were not committee members were asked to provide an evaluation of progress on specific areas of research. In addition, a workshop was held to consider methodologies for evaluation of research advances and reduction of uncertainty. Discussion at that workshop served as a basis for this chapter.

For this report, the committee also assessed the extent to which research initiated at its recommendation has been considered in the criteria document and staff paper. Given the limited time since the committee’s first report (approximately 6 years), this criterion would be expected to have limited sensitivity except perhaps for topic 1 (exposure assessment) and topic 10 (methods). Other more-distal criteria shown in Figure 2-3 cannot be expected to be informative at present but should be monitored and included in any future reports on research progress related to PM.

Research implementation, management, and interactions were also considered by the committee with respect to enhancing research progress by using the three criteria (accessibility, interaction, and integration). These criteria will be applied in general, across the range of research (see Chapters 4, 5, and 6).

Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×

FIGURE 2-3 Schematic representation of process for reducing disease risk through research.

Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×

CONSIDERATIONS IN INTERPRETING RESEARCH PROGRESS ON PARTICULATE MATTER

The committee notes that reduction of uncertainty, the overall goal of its research agenda, is never complete. EPA by the mandate of the Clean Air Act with regard to the NAAQS is inevitably making decisions in a background of scientific uncertainty. As it seeks to satisfy the requirements of the Clean Air Act to set standards to protect public health with an “adequate margin of safety,” its decisionmaking inherently involves PM concentrations at which effects do not occur with known probability or certainty. Additionally, the health effects of PM and their mitigation are a topic of evident complexity. Often, resolution of one uncertainty only highlights other uncertainties that still cloud decisionmaking. Furthermore, as the methods of scientific inquiry deepen, uncertainties that have seemingly been resolved may again become a focus of research as new methods facilitate resolution at a more profound level of inquiry.

Although the committee addressed a broad range of research topics in evaluating scientific process, it did not choose to judge one of its initial recommendations, the establishment of centers for research on PM. The EPA Science Advisory Board (SAB) completed an evaluation of the centers in 2002 (EPASAB 2002).

Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
Page 47
Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
Page 48
Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
Page 49
Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
Page 50
Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
Page 51
Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
Page 52
Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
Page 53
Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
Page 54
Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
Page 55
Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
Page 56
Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
Page 57
Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
Page 58
Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
Page 59
Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
Page 60
Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
Page 61
Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
Page 62
Suggested Citation:"2 Committee's Approach to Evaluation of Research Progress." National Research Council. 2004. Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress. Washington, DC: The National Academies Press. doi: 10.17226/10957.
×
Page 63
Next: 3 Synthesis of Research Progress on Particulate Matter »
Research Priorities for Airborne Particulate Matter: IV. Continuing Research Progress Get This Book
×
Buy Paperback | $57.00 Buy Ebook | $45.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In 1997, the U.S. Environmental Protection Agency (EPA) established regulatory standards to address health risks posed by inhaling tiny particles from smoke, vehicle exhaust, and other sources. At the same time, Congress and the EPA began a multimillion dollar research effort to better understand the sources of these airborne particles, the levels of exposure to people, and the ways that these particles cause disease.

To provide independent guidance to the EPA, Congress asked the National Research Council to study the relevant issues. The result was a series of four reports on the particulate-matter research program. The first two books offered a conceptual framework for a national research program, identified the 10 most critical research needs, and described the recommended timing and estimated costs of such research. The third volume began the task of assessing initial progress made in implementing the research program. This, the fourth and final volume, gauged research progress made over a 5-year period on each of the 10 research topics. The National Research Council concludes that particulate matter research has led to a better understanding of the health effects caused by tiny airborne particles. However, the EPA, in concert with other agencies, should continue research to reduce further uncertainties and inform long-term decisions.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!