National Academies Press: OpenBook

Interim Report of the Committee on Research and Peer Review in EPA (1995)

Chapter: 'THE COMMITTEE'S INITIAL ASSESSMENT'

« Previous: THE TASKS OF THIS COMMITTEE
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

competitive research grants, cooperative and interagency agreements,research contracts, and fellowships.

  1. Laboratory Site-Review Procedures. The committee will assess EPA's procedures for conducting site reviews of ORD laboratories andresearch programs.

  2. Research Staff Career-Development and Performance-Evaluation Procedures. The committee will assess the career development and evaluationprocess for scientific and technical staff in EPA's Office of Researchand Development.

THE COMMITTEE'S INITIAL ASSESSMENT

ORD's 1993 Report to Congress

In a report accompanying EPA's appropriations for fiscal year 1993, the congressional conferencecommittee requested EPA to review the balance between its basic andapplied research, stating

The Committee believes that, for a number of reasons, EPA has failedto sufficiently address the issue of basic research. Due to the largenumber of regulatory and statutory mandates, the Agency has focussedon short-term applied research. Basic research can be used to identifyand assess environmental problems which pose the greatest risk tohuman health and the environment.

In its responding report to Congress, Fundamental and Applied Research at the Environmental ProtectionAgency (EPA, 1993), EPA stated that it did not conduct or fund true “basic” research but did carry out “fundamental” research. Although both types of research are intended toadvance scientific knowledge, the report explained, basic researchenvisions no applications, whereas all EPA research must be relevantto the agency's statutory responsibilities.

The 1993 report described four major categories of activity in ORD:

Fundamental research, defined as “theoretical or experimental investigations to advance scientificknowledge where such knowledge is relevant to understanding pollution,the environment or human health, but immediate practical applicationis not a direct objective.” Examples of EPA's fundamental research are investigationsof mechanisms of chemical toxicity, chemical persistence or bioaccumulation,and ecosystem functioning.

Application-directed research, defined as “research directed to the solution of defined problems, based on existingscientific understanding; the solutions may involve the creationof new processes, procedures, methodologies, or situations that

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

will serve a practical or utilitarian purpose.” Examples of EPA's application-directed research are construction of pollutant transportmodels, creation of new monitoring methods, and formulation of ecologicalrisk-assessment frameworks.

Development, defined as “the work required to bring a new process, technique, methodology,or piece of equipment to the production or application stage.” Examplesare field calibration and testing of models, field validation ofmonitoring methods, and pilot-scale optimization of control technologies.

Technical assistance, defined as “the application of specialized technical knowledge or services toassist others in accomplishing their mission.” Examples are riskassessments, quality assurance, site monitoring, peer review, andexpert testimony.

Acknowledging that such categories are somewhat imprecise and subjective,EPA estimated that approximately 29% of ORD's total resources wasdevoted to fundamental research. The other 71% was devoted to application-directedresearch (43%), development (19%), and technical assistance (9%).EPA provided an updated report to Congress in 1994 (EPA, 1994a).

EPA's 1993 report also summarized ORD's “issue-based planning process.” That process was organized around a selected set of 38 environmentalproblems and scientific questions, grouped into 12 broad researchthemes (Table 1). The report described the distribution of resources among the 12themes (Figure 1). The report also stressed several strategic goals for the R&D program, indicating that ORD will seek to place greater emphasison the greatest health risks, important ecological problems, innovativeapproaches to risk reduction, and collaboration with outside scientists.Finally, the report described how ORD's resources have changed eachyear since 1980, noting that its extramural budget has more thandoubled since its low point in 1983, but the total personnel levelin ORD has remained fairly constant since the mid-1980s (Figure 2).

The research program described in Fundamental and Applied Research at the Environmental ProtectionAgency (EPA, 1993) reflects many factors, including the program's historical origins, traditional areas of strength of ORD's laboratories, EPA's regulatory support requirements, directives from Congress, missionambiguities, and resistance to change. But EPA's research programno longer exists as described in the 1993 report. It has been supersededby major changes being made by ORD. Consequently, the committee hasmade its initial assessment of EPA's research program in the contextof those changes, as presented on the following pages.

ORD's “New Beginning”

As the present National Research Council committee began its work,EPA was already making major changes in its research program. Theprocess leading to those changes

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

Table 1. ORD Research Themes and Issues. Source: EPA, 1993.

Protecting Ecological Systems

Ecosystems

  • Coastal and marine

  • Large lakes and rivers

  • Wetlands

  • Terrestrial systems

Assessments

  • Contaminated sediments

  • Aquatic ecocriteria

  • Nonpoint sources

  • Ecorisk assessment methods

Emerging Ecological Issues

  • Habitat/Biodiversity

  • Environmental releases of biotechnology products

Environmental Monitoring and Assessment

  • Environmental Monitoring and Assessment Program

Global Change

  • Global warming

  • Stratospheric ozone depletion

Air Pollution

  • Acid deposition/aerosols

  • Air toxics

  • Criteria air pollutants

  • Pollutants from motor vehicles

  • Indoor air pollution

Drinking Water Contamination

  • Drinking water pollutants and disinfection

  • Groundwater

Waste Management

  • Municipal solid waste

  • Hazardous waste

  • Sludge and wastewater

Environmental Clean-up

  • Surface clean-up

  • Bioremediation

Human Health Risk Assessment

  • Human exposure

  • Health effects

  • Risk assessment methods

Innovative Technology and Outreach

  • Pollution prevention

  • Innovative technologies

  • Environmental education

  • International and national technology transfer

Exploratory Research and Special Environmental Problems

  • Environmental review of new chemicals

  • Lead and other heavy metals

  • Anticipatory research on emerging environmental problems

  • Exploratory grants & centers

Infrastructure

  • Infrastructure

Cross Program

  • Cross program

began in 1993 with a decision by the administrator, at the requestof Congress and in connection with the vice president's NationalPerformance Review, to evaluate all of EPA's laboratories in comparisonwith the agency's scientific and technical needs. The evaluationbegan with a study by the MITRE Corporation, working with a teamof EPA officials, EPA's Science Advisory Board, and the NationalAcademy of Public Administration. The resulting report, A Comprehensive Study of EPA Scientific and Technical Laboratoriesand Their Facilities and Capabilities (MITRE, 1994), provided extensivedocumentation on the laboratories and their functions, as well asan analysis of

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

Figure 1. ORD extramural dollars and staff allocations by researchtheme. Source: EPA, 1993.

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

Figure 2. ORD extramural and staff resources, 1980-1991. Source:EPA, 1993.

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

five principal options for reorganizing them. Among its findings,the MITRE report expressed concern about the lack of clear, agreed-uponmission statements for EPA, ORD, or the laboratories. It also commentedupon the excessive use of contract personnel at EPA laboratories,various facility and equipment problems, and the need for improvementin quality-assurance and research-planning practices. Regarding laboratoryreorganization, the MITRE report favored a functional reorganizationpreviously recommended in Environmental Research and Development Strengthening the FederalInfrastructure (Carnegie Commission on Science, Technology, andGovernment, 1992).

EPA's Science Advisory Board (SAB) and the National Academy of PublicAdministration (NAPA) reviewed the MITRE report (NAPA, 1994; EPASAB,1994). The SAB report generally concurred with MITRE in endorsinga variation of the Carnegie Commission model for laboratory organization,but it argued that management improvements were needed in ORD beforereorganization should occur. The SAB report expressed strong concernabout the current demands for EPA laboratory scientists to devotemuch of their time as contract managers. It also expressed concernabout the possibility that decreased ORD involvement in short-term,applied research and technical assistance to the regulatory and regionaloffices might lead to the expansion of separate research programsin the regulatory offices, which might be destructive to the overallquality and effectiveness of ORD's research role in the agency. TheNAPA review also generally supported the MITRE recommendations butemphasized that ORD's mission and goals be clarified before any laboratoryreorganization take place.

In response to the MITRE study and the SAB and NAPA reviews, an agencywide,senior-level steering committee prepared final recommendations tothe administrator. Their report, Research, Development, and Technical Services at EPA: A New Beginning (EPA, 1994), recommended that the following changes in ORD's research program be made over the next 3 years:

Strategic planning: Initiate a new strategic-research planning process in concert withother offices of EPA.

The Role(s) of ORD: Increase long-term research from about 30% to at least 50% of ORD's total program budget.

ORD's laboratories: Functionally consolidate ORD's 12 laboratories, three field stations, and four assessment centersinto three national laboratories and a national assessment center,thereby decentralizing research management responsibility to thelaboratories. Reduce Washington, D.C, headquarters staff of ORD byhalf through attrition and reassignment. Replace some contract personnelat EPA laboratories with federal personnel. Establish new scientificcareer-track opportunities and performance evaluation proceduresfor EPA laboratory scientists.

Research grants: Increase annual funding for extramural, investigator-initiated,peer-reviewed, competitive research grants and centers from the presentlevel of

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

$20 million to $100 million. Create a competitive, investigator-initiatedresearch grants program for in-house scientists at ORD's laboratories.

Peer Review: Strengthen and expand peer-review practices for proposals, publications,risk assessments, and laboratory programs.

Fellowships: Create a program of 300 graduate student fellowships.

The administrator accepted these recommendations, and as discussedbelow, the changes under way amount to the most dramatic changesin the history of ORD.

Strategic Planning

It is difficult, if not impossible, to conduct or even to assessa research program like EPA's without first having a clear understandingof the agency's mission and scientific goals, as well as the mission,specific goals, and strategic plans of its research program. Unfortunately,throughout the history of EPA, the agency has never confronted theseneeds in a consistent and thorough manner.

EPA was created in 1970 by executive order, not by legislation. Noformal overall mission has ever been enacted for EPA, and no formal,agreed-upon scientific or research mission has been enacted for ORD.So-called mission statements for EPA and ORD have appeared from timeto time in various EPA documents, but those statements have reflectedthe changing visions of successive EPA officials, not statutory mandates.The programs and capabilities of EPA, including ORD, have derivedlargely from an eclectic collection of statutes that address separateproblem areas but lack a unified approach to environmental problems.This is somewhat ironic, because one of the principal arguments presentedin 1970 to justify the creation of EPA from its disparate predecessorprograms was the need for a unified environmental program for thenation (CEQ, 1970).

The lack of a clear, agreed-upon mission statement and coherent researchplanning process has been one of the most frequently expressed concernsby previous independent review groups evaluating EPA's research program.Essentially every one of the studies cited in this report, from theNational Research Council's 1974 report to EPA's 1994 “New Beginning” report, expressed concern about the ambiguityin ORD's research mission and urged EPA to develop specific, agreed-uponmission and vision statements for ORD and its organizational components.Yet, as recently as December 1994, when EPA's newly appointed assistantadministrator for research and development briefed the present committeeat its first meeting, he informed the committee that the missionof ORD was “still under discussion” as part of ORD's current restructuring process (R. Huggett, pers. comm.,1994). This was a quarter of a century after EPA and ORD were created.

The most recently promulgated version of EPA's mission (EPA, 1994c) is

Improving and preserving the environment in the U.S. and around theglobe

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

Working with partners to protect human health, ecosystems, and thebeauty of the environment, using the best available science

Promoting innovative solutions to environmental problems

Protecting and sustaining the productivity of natural resources

The current provisional mission statement for ORD, still under discussionwithin the agency, is

Conducting research in support of effective policies and regulationsfor environmental protection

Integrating technical and scientific knowledge to provide scientificallysound advice for EPA's programs

Providing national leadership for environmental science and research

In this committee's opinion, the above mission statements for EPA and ORD, as wellas the preliminary mission statements that ORD has developed forits new national laboratories and national centers (Figure 3) are steps in the right direction but need substantially more detailand explanation. Most sorely needed are a coherent scientific andtechnical strategy statement for EPA, a strategic plan for ORD, anda strategic plan for each ORD laboratory and center that is consistentwith the agency and ORD plans. Each strategic plan should consistat a minimum of a vision statement, a mission statement, and a planfor achieving them. These documents need not be lengthy or complex,but they should be robust and specific enough to enable the agencyand this committee to evaluate the intended role of ORD and its organizationalcomponents in providing scientific and technical knowledge to supportnational environmental programs, policies, and decisions, as wellas to identify unnecessary geographical and functional duplicationand significant gaps in ORD activities. It is crucially importantthat the strategic-planning process drive the development of ORD's budget, instead of being driven by it.

In 1993, Administrator Browner asked EPA's Science Advisory Board to undertake an environmental futures studyto examine scientific approaches for predicting and analyzing over-the-horizonenvironmental problems of the next 5 to 30 years and to advise EPAon how it can systematically prepare to anticipate them. The SABreport is expected soon. In response to the report Research, Development, and Technical Services at EPA: A New Beginning (EPA, 1994), ORD has undertaken a fundamental re-examination ofits research planning and is engaged in a major effort to implementa new strategic research-planning process in concert with other officesof EPA. In September 1994, ORD formed a research coordination council,involving EPA regulatory and regional offices, to develop recommendationsfor research priorities and strategic planning. A new high-levelposition,

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

Figure 3. ORD's new national laboratories and centers.

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

deputy assistant administrator for science, has been created withinORD to manage the planning process. A base review of the ORD researchprogram is also being conducted by staff teams from ORD and the regulatoryoffices with a view toward recommending strategic changes in theprogram. Each ORD laboratory will be expected to develop its ownplans explicitly related to the overall strategic plans. ORD expectsthat the new planning process will produce 5-year strategic plansthat will in turn provide a framework for the agency's budget process.

The present committee believes that these efforts will be of crucialimportance to ORD and should receive very high priority. The committeeeagerly awaits the first results of ORD's new strategic-planningprocess and will assess them and their implications in its finalreport.

The Role(s) of ORD

Historically, ORD has struggled to balance two primary roles: a researchrole mainly involving long-term fundamental and anticipatory researchthat is largely, but by no means entirely, investigator-initiatedand a technical assistance role of directed support and short-termapplied projects for EPA's regulatory and regional offices. Someagency officials have maintained that ORD's dual roles have beenmutually reinforcing and generally advantageous, but the difficultiesof serving multiple regulatory-office clients while trying to sustaina fundamental and anticipatory research program have posed challengesto ORD throughout its history. ORD has continually been torn betweencompeting demands: on the one hand to lead and on the other handto serve the rest of EPA.

EPA acknowledged this difficulty in the preface to Fundamental and Applied Research at the Environmental ProtectionAgency (EPA, 1993) stating

Indeed, the difficulty for EPA as well as for other regulatory agencieshas been meeting the needs of many research clients. EPA's researchprogram must strike a balance between providing data and technicalsupport for ‘front-line' regulators solving environmental problems today and building a science knowledge base necessary to manage our ecologicalresources wisely in the coming decades; understand how pollutantsaffect our health; and prevent or reduce environmental risks in thefuture.

In the past, ORD has been predominantly involved in short-term activities,reacting to the needs of EPA's regulatory and regional offices. Manyscientists within and outside the agency believe that ORD shouldbecome more autonomous and more of a proactive leader for the restof the agency, giving greater emphasis to long-term, over-the-horizon,fundamental and anticipatory research that develops the knowledgeto lead EPA into new strategies and levels of understanding, regardlessof whether such research provides near-

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

term help for current regulatory programs. Regulatory officials,on the other hand, often argue that EPA's funds, including ORD's, are provided to supportthe agency's existing statutory mandates and regulatory programs,so ORD should provide the regulatory offices with more technicalassistance and short-term, quick-payoff applied work.

This committee believes that the long-term, anticipatory researchrole is of crucial importance. EPA and ORD were created in 1970 withan understanding of environmental issues that is now largely outdatedand rapidly being subsumed in new concepts such as sustainable developmentand industrial ecology (EPASAB, 1988; NAE, 1994; NAPA, 1994; OSTP,1994). These concepts envision the integration of environmental scienceand technology throughout the entire economy. They are not simply(or in many cases even primarily) concerned with remediation of existingimpacts and compliance with so-called “end-of-pipe” regulations. If ORD is toparticipate effectively in developing and implementing new conceptsand policy directions, its scope of activities must be appropriatelyredirected and expanded. ORD must address not only the individualpollution-related problems that have traditionally concerned EPA,but also the research to address complex topics such as sustainabledevelopment and the societal efforts needed to maintain biologicaldiversity. Research should lead the activities of EPA, and not justfollow past efforts or currently perceived needs. EPA's researchshould address future problems as well as the remediation of pastproblems.

Some research problems are sufficiently broad or complex that theycan be addressed effectively only by a long-term, focused effort.In addition, a sustained program of anticipatory research would generallybe expected to reduce the need for reactive projects. An effectiveover-the-horizon research program seeks to anticipate and addressfuture scientific needs in support of environmental protection, therebyreducing future needs for inefficient and expensive crash efforts.

In the opinion of this committee, it is difficult for ORD to leadthe agency by means of research when it is also required to serveclient regulators who have variable understanding or appreciationof science but a strong say in ORD's budget and priorities. A researchorganization tends to be poorly suited for so-called firefightingactivities. Such activities tend to have a narrow focus and to pre-emptlong-term research programs. They tend to consume the resources ofa research organization disruptively and disproportionately. Thereis also some inevitable risk to ORD's scientific credibility whenit provides technical support for regulatory strategies that havebeen predetermined, or are perceived to be. In addition, regulatorystrategies, which are typically prescriptive and specific, tend tofreeze concepts and methods in time. The scientific components ofregulations tend to get caught in the same freeze. The closer thatORD is tied to the regulatory programs, the greater is the risk thatORD will to some extent be working on outdated problems or with outdatedapproaches—waging the last war instead of preparing for the nextone.

On the other hand, using ORD for short-term scientific assistanceto regulatory and regional offices has some undeniably importantbenefits. Perhaps the greatest dividends are the resulting improvementsin the scientific aspects of regulations, the maintenance of

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

an experienced scientific core group within the agency, and the knowledgeof agency issues that the research scientists obtain through suchexperience. An experienced scientific core group can be of greatimportance whenever emergency requirements for scientific expertisearise. ORD's technical assistance provides the regulatory officeswith competent scientific help, and it enables the ORD research scientiststo keep abreast of regulatory and policy developments elsewhere inEPA.

Until recently, roughly 30% of ORD's total resources have been devoted to long-term fundamental andanticipatory research. In the current plan, ORD is increasing thatproportion to at least 50%. This would mean a corresponding reductionin the short-term applied projects and technical assistance thatORD provides for the agency's regulatory offices.

The committee endorses ORD's current efforts to increase the priority and proportion of long-termresearch in ORD, despite the consequent reduction of short-term technicalprojects supporting regulatory and regional programs. EPA's researchprogram should give greater emphasis to evaluating potential environmentalconcerns and over-the-horizon possibilities, as well as new approachesto managing current problems, not just remediating past problems.Research should lead the directions of an agency like EPA. At thisearly stage in its study, the committee has not had sufficient timeto offer judgment on the optimal proportions for ORD, but it doesconclude that the past predominance of short-term, reactive activitiesmay have been excessive, and ORD's contemplated increase appearsto be a move in the right direction. The committee further recommendsthat EPA's budget-development process be reconsidered with a viewtoward ensuring that ORD has sufficient autonomy to conduct the expandedlong-term research program despite the consequent reduction in short-termpayoff to the agency's regulatory offices. There is also a criticalneed for the research budgeting process to be made more understandableto persons both within and outside the agency, so it can become moreuseful for managing, tracking, and explaining the ORD program.

However, as ORD moves to increase its long-term research and itsgrants program, safeguards are needed to ensure that the importantscientific needs of EPA's regulatory programs and regional officesare not compromised. A great burden will be placed upon the new agencywidestrategic-planning process to ensure that this does not occur. Thenature of ORD's technical support role should be reviewed with respectto the goals of such interactions, the appropriate degree of interactionwith program office staff, the timing of such interactions, and theadministrative channels for such interactions. The committee alsoendorses the idea of examining several major, completed rulemakingprocesses to determine the types, degree, and timing of science interactionsbetween regulatory offices and ORD staff, and the effect of suchinteractions (or lack of them) in shaping agency decisions. Fromthere, and taking into account the varying needs and constraintsof regulatory processes, more concrete and effective guidance forfuture interactions could emerge.

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

Despite the positive changes under way in ORD, the present committeehas continuing concerns about agencywide scientific quality assuranceand ORD's role, or lack of role, in it. No EPA official below thelevel of the administrator has the overall responsibility for EPA's scientific and technical performance or credibility, and no scientisthas ever been appointed as EPA administrator. Although improvingscience in EPA has been a priority of some administrators, includingthe current one, that cannot always be assumed. EPA's assistant administratorfor research and development has been responsible for the agency's research program and for providing technical expertise and supportto the agency's regulatory and regional offices, but those officeshave the responsibility for regulatory decisionmaking and policy,and they are not required to follow scientific advice from ORD.

In Safeguarding the Future: Credible Science, Credible Decisions (EPA, 1992), the authors noted that

The science advice function within EPA—that is the process of ensuring that policy decisions are informedby a clear understanding of the relevant science—should be enhancedand coherently organized within EPA. The Administrator should appointa distinguished scientist or engineer to function as EPA's ScienceAdvisor.

The 1992 Credible Science report presented strong justification for such an appointment, andit defined the roles that such an individual would perform in theagency. The report also assessed arguments for and against threeorganizational alternatives: appointing a new staff member in theadministrator's immediate office; giving the additional responsibilitiesto the assistant administrator for research and development; or givingthem to a subordinate official in ORD. In response to the report,EPA appointed a science adviser on the immediate staff of the administrator,but lacking substantial managerial authority, that individual hasgenerally lacked the means to perform effectively the agencywiderole envisioned by the 1992 panel.

The present committee strongly concurs with the assessment and recommendationsof the 1992 report concerning a science adviser and believes thatsuch a role is vital to EPA. In essence, the senior science advisershould serve as the chief scientific and technical officer of theagency with overall responsibility for coordinating and overseeingEPA's scientific and technical activities. Based upon the 1992 reportand subsequent related developments, this committee recommends thatEPA adopt an alternative recommendation from the 1992 Credible Sciencereport; EPA should broaden and strengthen the role of the assistantadministrator for research and development to include the agencywideaspects of the science adviser position envisaged by the CredibleScience panel. The committee believes that this individual must bea qualified scientist or engineer. [Note: As part of the changesbeing made in ORD, the administrator's current science adviser isbeing appointed

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

director of a new national center for extramural research and qualityassurance, reporting to the assistant administrator for researchand development.]

The committee specifically recommends that the assistant administratorfor research and development be designated as EPA's chief scientificofficer. In addition to the programs of ORD, this individual wouldbe responsible for

  • Defining the most important scientific issues facing EPA, and developingand overseeing a rational agencywide approach to acquiring the necessaryscientific information, either through intramural efforts or throughextramural programs and cooperative efforts involving academia, othergovernment agencies, and the private sector in this country and abroad.

  • Coordinating and overseeing scientific quality-assurance and peer-reviewactivities throughout the agency, including activities in supportof the regulatory and regional offices. This function would includeensuring that the qualifications and training of EPA's technicalpersonnel meet the requirements of their assignments.

ORD's Laboratories

In 1992, the Carnegie Commission report Environmental Research and Development - Strengthening the FederalInfrastructure (1992) recommended that ORD's laboratories, field stations, and assessment centers be consolidated and restructured along scientific disciplinary lines.Subsequent reports from the MITRE Corporation (1994), EPA's ScienceAdvisory Board (1994), and the National Academy of Public Administration(1994) generally concurred with the Cargegie report's recommendaton.The EPA-wide “New Beginning” steering committee (EPA, 1994) recommended a modifiedversion of the Carnegie proposal involving a functional consolidationand reorganization of ORD's laboratories and headquarters offices.This reorganization is under way. The new organization of ORD issummarized in Figure 3.

The committee believes that the main immediate impact of this reorganizationwill be the decentralization of laboratory management and administrationfrom headquarters to the laboratories. This will streamline operationsby giving more decisionmaking responsibility to laboratory managersand by eliminating a layer of headquarters staff. Although the missionsof individual laboratories will be adjusted, no ORD laboratory willbe abolished in the reorganization. Each laboratory director willsimply report to the director of a newly designated national laboratoryinstead of to an ORD suboffice director at headquarters. The directorsof the national laboratories will report directly to the assistantadministrator for research and development. The headquarters subofficesto which the laboratories previously reported are being abolished,and ORD accordingly plans to reduce its headquarters staff by about50% through reorganization and attrition.

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

Another important aspect of the reorganization is its focus on therole of ORD's laboratories in reducing uncertainty in identifying,understanding, and managing environmental risks. ORD has adoptedthe National Research Council's (1983) “Red Book” paradigm for risk assessment and riskmanagement as the organizing principle for the missions of the newnational laboratories. Focusing the laboratories on core areas ofresearch instead of trying to cover the full scope of EPA's scientificneeds was recommended by EPA's Science Advisory Board (1988). Emphasison reducing the uncertainties in risk assessments was recommendedby the Credible Science report (EPA, 1992). Although the missions and scientific requirementsof EPA and ORD are certainly broader in scope than the risk-assessmentparadigm, ORD has chosen to focus its in-house laboratories on coreareas of research related to the aspects of risk assessment thathistorically have been of greatest interest to EPA and its laboratories.The committee concurs with this decision but notes that ORD consequentlymust give greater emphasis to other areas of research in its extramuralprograms and partnerships with other organizations, as discussedbelow. The committee also notes that economic analysis generallyhas not been adequately addressed under the current risk managementparadigm; the committee intends to discuss that in its final report.

Many ORD laboratory scientists have been required to perform dualroles as hands-on researchers and contract managers. This raisesquestions about the desired or optimum role of EPA laboratory scientists.Clearly a fundamental requirement is that ORD must have in-housescientists who are immersed in and familiar with the state of theart in scientific areas that are critical to EPA. It is also importantthat ORD have qualified staff to manage extramural scientific projectsand communicate knowledge to regulatory and enforcement decisionmakers.In the scientific areas that EPA judges to be of primary importance,it is highly desirable that EPA have its own firsthand experts whogain in-depth knowledge through their own cutting-edge research.

Partnerships and Outreach

By itself, ORD can never meet all of EPA's needs for scientific knowledge. The agency's needs are too broad. ORD has successfully been a scientific leaderin some areas, such as aquatic toxicity and bioconcentration, butit is not possible for ORD to be a leader in all areas of scientificknowledge required by EPA. EPA should realistically recognize thelimits of its research capabilities compared with the broad scopeof its needs for scientific knowledge and information. As recommendedin Future Risk: Research Strategies for the 1990s (EPASAB, 1988), EPA should focus on a few core research areas thatit can handle well and rely on partnerships and outreach for otherscientific needs. The agency should foster greater awareness andutilization of research findings obtained elsewhere. EPA should stimulateand coordinate research that is responsive to its needs at nationaland international institutions. The agency should be a leader insome areas of

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

research, but it must be at least well informed in all relevant areas.To accomplish this, EPA will need a cadre of scientists responsiblefor maintaining a thorough awareness in specific research areas thatare important to the mission of the agency. In this context, thecommittee recommends that ORD develop a stronger and more structuredapproach to the retrieval, synthesis, and use of the results of researchconducted by scientists not affiliated with or supported by EPA.This includes research conducted or sponsored by other federal andstate agencies, universities, and industries, both in this countryand abroad.

In testimony before the House Science Committee on January 6, 1995,Administrator Browner emphasized that EPA alone cannot adequatelyidentify emerging issues and generate the knowledge needed to solvelocal, regional, and global problems. To be effective, AdministratorBrowner stated, EPA should work collaboratively with other agenciesand expand its scientific partnerships with the outside academicand scientific community. As examples of EPA's efforts to expandsuch partnerships, she cited EPA's participation in the Committeeon Environmental Research of the National Science and TechnologyCouncil and the U.S.-Canadian efforts to develop a North Americanresearch plan for tropospheric ozone, as recommended by the NationalResearch Council (1991).

The present committee strongly supports those views and believesthat additional mechanisms for achieving such collaboration shouldbe pursued. As noted in the 1992 report of the Carnegie Commission,U.S. environmental research programs should collaborate more closelywith those of industry and other nations. The Carnegie report suggesteda number of mechanisms for doing this.

To further such interactions, the committee also recommends thatORD consider increasing sabbatical assignments for ORD researchersto gain experience in other scientific organizations, as well asbringing more scientists from universities, other government agencies,and private organizations to ORD laboratories and centers for visitingappointments.

The Research Grants Program

One of the most important ways by which ORD forms partnerships andenlists more scientists and research organizations to collaboratein EPA's scientific enterprise is through its research grants program.Recommended by the National Research Council in 1977 and initiatedby EPA in 1980, this is a competitive, peer-reviewed, extramuralinvestigator-initiated research grants program. Its budget has fluctuatedbetween $5 million and $25 million per year. Fiscal year 1994 fundingfor the program was $21 million.

As recommended by the “New Beginning” report (EPA, 1994), ORD plans to increase the extramural grantsprogram to $100 million per year within 3 years. EPA estimates thatthis would enable funding of about 400 new grants per year. The grantswould typically last 3 years and average about $100,000 per grantper year.

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

The committee welcomes EPA's decision to increase the magnitude of research support to scientistsoutside the agency, including universities and independent researchorganizations. However, it is important that this redirection ofsupport not critically injure other key research programs. The committee's final report will seek to address the issues of balance and theiroverall effects. To help justify the increases in the grants program,the committee recommends that ORD endeavor to do a much better jobof disseminating the results of its grant-funded research, criticallyassess the relevance and usefulness of such research to the restof EPA, and demonstrate the beneficial impact of such research onagency decisions and actions.

In the first year of the expansion, fiscal year 1995, ORD is increasingthe grants program to $44 million. Approximately half of this amountis being awarded through solicitations conducted by ORD's Officeof Exploratory Research. As requested by Congress in EPA's fiscalyear 1995 appropriations report, the other half is being awardedthrough a process conducted jointly by ORD and the National ScienceFoundation (NSF). Working in partnership, ORD and NSF have developedjoint program announcements, peer-review procedures, award decisionprocedures, and funding for several topic areas of mutual interest.

This committee believes that ORD is deriving some beneficial insightsand experience from the collaboration with NSF this year. The experiencegained this year should help ORD in the future. However, ORD facessome impediments that NSF does not. The administration of ORD's grantsprogram has been hampered by a very small workforce. Furthermore,the NSF grants process emphasizes quality but not program relevance;ORD's grants process must address both.

In a letter to the chairman of this committee dated January 6, 1995,Assistant Administrator Huggett discussed the need for ORD's grantsprogram to find ways to improve the reliability of multiyear grantfunding and to become more streamlined administratively. The committeestrongly concurs with both of these goals and believes they willbe very important factors in ORD's efforts to form new partnershipsand leverage the academic community more effectively. Expansion ofthe research grants and centers program will also require that aconsiderably greater portion of ORD's workforce be assigned to thisprogram.

The funds needed by ORD to increase the grants program nearly fivefold,from $21 million to $100 million per year, are expected to come fromreductions in other ORD programs. Concerns have been expressed aboutthe potential effects on those other programs. The committee recommendsthat ORD look to its contracting budgets for a substantial portionof the required funds. In the past, ORD has depended too heavilyon contractors; this is partly due to personnel limitations and federalprocurement practices. But this pattern is also partly due to habitsformed at EPA. ORD's extramural research needs have often been announcedin legalistic ways, as in the Commerce Business Daily or the Federal Register, which are not readily available or understandable to some academicscientists who might have expertise that is most needed by ORD. Thecontract solicitations

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

are typically cast in business and legal language and stipulationsthat strongly favor corporations with strong business and legal staffs.This combination of factors has discouraged some of the nation'sbest scientists and engineers from responding to and working withor for EPA. Some of these scientists could be enlisted through theexpanded grants program.

The committee does not recommend substantial decreases in ORD's budgets for cooperative agreements, interagency agreements, ormultidisciplinary institutional grants, because those mechanismsare very important to ORD's research partnerships and outreach efforts.

To increase awareness among scientists inside and outside of EPAabout research funded by ORD's intramural and extramural researchmechanisms, the committee recommends that ORD establish a currentresearch information system like that used in the U.S. Departmentof Agriculture and the land grant universities. This system wouldprovide annually updated one-page descriptions of objectives, currentprogress, and publications resulting from such projects in all ORDlaboratories and centers and in all organizations supported by ORD.

Peer Review

One of the recommendations of Safeguarding the Future: Credible Science, Credible Decisions (EPA, 1992) was that the application of peer review and qualityassurance be expanded and strengthened throughout EPA. In response,Administrator Reilly established an agencywide peer-review policyin January 1993. In June 1994, Administrator Browner expanded thepolicy and instituted an agencywide peer-review program to implementit. Under the coordination of EPA's Science Policy Council, ORD andother offices of EPA have developed standard operating proceduresfor peer reviews. ORD is also participating in an interagency examinationof peer-review practices under the auspices of the Committee on EnvironmentalResearch of the National Science and Technology Council.

In its October 1994 Interim Report to Congress, Development of Peer Review Systems at EPA, ORD reported that peer review will continue to be conducted andwill be strengthened on research program plans, research proposals,interim stages of ongoing long-term research, and completed researchproducts. The committee concurs with this plan and recommends thatan appropriate set of peer-review procedures be developed and appliedessentially to all intramural and extramural research projects andprograms. This includes the research conducted by EPA scientistsand engineers at ORD laboratories and centers, as well as the extramuralresearch conducted, by others (or cooperatively with others) throughindividual investigator grants, multidisciplinary grants, researchcenters, other cooperative agreements, interagency agreements, fellowshipsand training grants, onsite research support contracts, and otherresearch contracts. Notwithstanding the deadlines and other factorsnecessarily involved in determining when peer review is

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

feasible and appropriate, and what levels and types of peer reviewshould be conducted, there should be strong presumptions favoringpeer review, the involvement of external experts in such reviews,and the nomination of such experts by independent referees insteadof project managers.

The committee endorses and commends ORD's current efforts to strengthen peer-review policies and proceduresto improve scientific quality in all of the research it supports.The committee makes this endorsement because it believes that moreeffective use of peer review, as well as greater emphasis on publicationof EPA research results in the open and readily accessible, peer-reviewedscientific literature, will increase the quality of EPA science andthe credibility of EPA decisions. At the same time, it should berecognized that peer review is one of many management tools, nota decision mechanism by itself. A program manager needs to know theviews of qualified peers, but such peers often cannot be expectedto integrate factors outside the document presented for review, suchas the relevance, need, and priority of a new research activity orthe role of research findings in a context that necessarily includesstatutory requirements, economics, and many other things. Furthermore,although peer review is widely used and highly regarded, it is poorlyunderstood by many, and it has come under serious study only in recentyears. Peer review also is an expensive and personnel-intensive process.It requires the services of many different types of persons—skilledprogram officers and advisers, imaginative investigators, competentpeer reviewers, and efficient grants administration specialists.These individuals must work together in a constructive, trusting,and harmonious way to achieve the extent of streamlining that isnecessary to make competitive, peer-reviewed systems of intramuraland extramural grants effective and efficient. It is also very importantthat the limited supply of qualified peer reviewers be utilized efficiently.

The committee recommends that ORD continue to have its research programplans reviewed by a group of experts that includes individuals fromoutside as well as inside EPA. The committee concurs with ORD's plansto give special attention to peer reviews of the new strategic plansand new areas of research. The committee further recommends thatresearch proposals, whether unsolicited or received in response toEPA solicitations, should continue to be evaluated by mail reviews,expert panel reviews, or both. For interim reviews of long-term researchprojects extending for more than 3 years, the committee recommendsthat ORD continue to conduct research progress reviews at suitableintervals. The committee will comment on the procedures for suchreviews, including laboratory site reviews, in its final report.

For completed research work products, the committee encourages ORDto continue and to expand its longstanding practice of urging theresearch scientists it supports to publish their research in peer-reviewedjournals that meet international standards of scientific quality.To the extent possible, intramural and extramural research supportedby EPA should be published in peer-reviewed journals that are opento scientific and public scrutiny. When such publication is not possible,for example, when the volume of the research results that are importantto the agency is so large that the pertinent results cannot

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

be accommodated in a peer-reviewed journal, panels of experts shouldmake an evaluation of quality that is essentially equivalent to thatof the peer-reviewed scientific literature. These evaluations shouldinclude scientists and engineers from outside ORD and also outsideEPA. The committee also wishes to emphasize that ORD should striveto make its research products, including such “gray” literature, morewidely known and accessible.

Most intramural research in EPA research laboratories and centershas been funded as laboratory-administered research projects. Inthe future, ORD scientists and engineers also will be eligible toapply for additional research funds provided through competitivelyawarded intramural research project grants. The committee recommendsthat periodic peer reviews should be made of all intramural researchprojects, including those funded through laboratory-administeredresearch funds and those funded through competitively awarded additionalresearch funds. These periodic reviews should be made by panels thatinclude some scientists from outside the EPA laboratory or centerin which the research is conducted and should include reviews ofongoing research as well as evaluation of completed research products.

The Fellowships Program

ORD has initiated a national program of 100 graduate student fellowshipsin fiscal year 1995 and plans to increase the number to 300 by fiscalyear 1997. The fellowships are competitive, and applications arebeing peer reviewed.

The committee commends EPA for the decision to launch this program,which is intended to help ensure an adequate number of qualifiedand diverse personnel to carry on future environmental research.The federal government, which has sponsored much of the researchin this area and is also responsible for translating research resultsinto policies, risk assessments, and risk-management actions, shouldalso assume responsibility for providing support for education andtraining programs in environmental science and technology. The federalinvestment in education programs should, in general, keep pace withoverall research and development needs, if critical shortages ofenvironmental personnel are to be avoided. Thus, the EPA studentfellowship program should be more focused on those specialities forwhich there is evidence of strong current and future demands. Forexample, in certain areas of specialization in toxicology, the disparitybetween supply and demand is acute. Thus, neurotoxicologists andgenetic toxicologists who work on environmental problems are virtuallyunobtainable. Similarly, opportunities for specialized educationin environmental engineering, ecology, epidemioloy, and biostatisticsare presently inadequate. The availability of social scientists forenvironmental research programs is also very limited. EPA's fellowshipsprogram should emphasize disciplines such as these.

The committee recognizes that estimation of research manpower supply,needs, and demands is particularly difficult in disciplines relatedto environmental protection and

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×

environmental health. Estimates can be made, however. Trends in jobplacements, the number of people completing educational requirementsin individual disciplines, and the public and private funding forenvironmental programs should be among the factors in determiningthe scope, emphasis and priority specialization areas for educationalsupport through EPA's graduate student fellowship program.

Conclusion

The committee concurs with the general scope and direction of thechanges being made in ORD, as discussed above. Several of these changeswere recommended by previous independent expert groups, and thiscommittee agrees with them. The committee believes that the changeswill improve EPA's research program in many ways. The currently plannedincrease in long-term research will help ORD perform a stronger anticipatoryrole and improve the quality of its program by bringing a betterbalance and quality of research, as well as by attracting more talentedscientists. The functional laboratory reorganization and reductionin headquarters personnel will decentralize laboratory managementso that the laboratories will report to research directors insteadof administrators. By basing the new organizational design on therisk-assessment/risk-management paradigm, ORD's new laboratory organizationwill promote the important goal of reducing the uncertainties inrisk assessment and risk management. The expanded peer-reviewed researchgrants program will bring many additional scientists and researchinstitutions into the picture, provide the opportunity for high-qualityresearch in areas where EPA is not a leader but needs information,and help to promote outreach and collaboration between EPA and otherorganizations. The graduate fellowship program will help to educatethe environmental scientists and engineers of the future. And thegrants and fellowships programs will build connections with academicscientists.

These changes will entail some tradeoffs. Some EPA regulatory programsand regional offices will probably see a reduction in immediate payoffas ORD shifts to more long-term research at the expense of short-termapplied projects and technical assistance. Some EPA laboratory scientistsand contractors will probably see reductions in budgets for in-houseand contract work as ORD redirects some of those resources to increasethe extramural grants program. Some ORD headquarters activities willdecrease as a result of the decentralization of resources and managementauthority to the laboratories.

Despite such tradeoffs, the committee recommends that ORD stay thecourse with the changes under way, as discussed above. The principalchallenges to ORD will be to create a successful strategic planningprocess and to maximize both the quality and value of returns fromits intramural and extramural research and educational investments.In its final report, the committee will review ORD's program in thecontext of these changes and challenges.

Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
This page in the original is blank.
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 13
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 14
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 15
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 16
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 17
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 18
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 19
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 20
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 21
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 22
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 23
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 24
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 25
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 26
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 27
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 28
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 29
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 30
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 31
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 32
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 33
Suggested Citation:"'THE COMMITTEE'S INITIAL ASSESSMENT'." National Research Council. 1995. Interim Report of the Committee on Research and Peer Review in EPA. Washington, DC: The National Academies Press. doi: 10.17226/9202.
×
Page 34
Next: REFERENCES »
Interim Report of the Committee on Research and Peer Review in EPA Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!