Environmental Management Technology-Development Program at the Department of Energy: 1995 Review (1996)

In its deliberations during 1995 and its review of the subcommittee reports attached in Appendix A, CEMT identified five significant points that DOE should address if it is to manage the development of environmental management technologies effectively.

In an environment of severely constrained federal budgets, setting clear priorities for technology-development investments has become increasingly important. In a previous report by this committee evaluating the development of environmental-management technologies (NRC, 1995), DOE was encouraged to focus technology development on clearly identified problems and to consider systematic use of comparative risk and risk/benefit assessment in evaluating environmental-management alternatives that form the basis for technology-development needs. During the DOE briefings and site visits, CEMT has not seen evidence that this systematic evaluation of alternatives or that any such approach to prioritizing DOE-EM's technology investments based on need, risk, and other important factors has been implemented.

A systematic evaluation of environmental-management technologies must be an underlying framework for prioritizing technology needs. A framework that has worked well for complex technological systems is that of a structured set of problem/solution scenarios responding to a needs assessment. These scenarios should be evaluated against explicit quantitative criteria, based on a robust set of criteria performance measures such as risks, costs, schedule, and benefits. Embodied in each scenario are many factors, including different technology requirements, safety

impacts, cost implications, and environmental impacts (see Figure 1). The basic approach for prioritizing technology development needs in this manner is outlined below:

• develop and perform an assessment of needs for a given problem or issue;
• develop a comprehensive set of scenarios for different solutions to each problem or issue, covering the major steps from characterization to disposal;
• screen the scenarios to a manageable few, utilizing appropriate screening criteria based on the chosen performance measures;
• quantify the performance measures of an appropriate set of screened scenarios;
• rank the scenarios based on the values assigned to the performance measures; and
• derive the technology-development needs from this ranking of scenarios.

Figure 1:

Relationship of Technology Development to Waste Remediation

This process would ensure that the technology needs identified are related to known problems because the scenarios are based on actual problems that must be solved. The process would also assess the relative value of developing technologies that perform as assumed in the scenarios.

Once the technology needs have been identified, evaluated, and prioritized, additional factors on performance measures may be considered in setting priorities among the available technology-development investment options. Additional performance measures could include the likelihood of success and the need for a balanced portfolio of technology-development investments (i.e., a mix of near-term, low-risk investments and longer-term, high-risk, high-payoff investments).

Recommendation: DOE should develop and rank remediation scenarios leading to a prioritized list of technology needs as noted above. The scenarios should be structured to define only the major steps from characterization through disposal (or storage) and to highlight only the major choices to be made. It is important that the scenarios be presented clearly; too much detail can obscure the basic issues.

The need to establish objectives for remediation and disposal also was emphasized in the previous NRC report (NRC, 1995), which stated the following (p. 3):

Further, the report stated (p. 4):

Based on DOE presentations, discussions with DOE staff, and review of DOE documents, the committee concludes that these considerations do not appear to have been part of the technology-development program to date. Until such goals are established, it will be impossible to develop an effective focused research and development program with definitive end points.

The committee is aware of the difficulty in the present regulatory and public climate of establishing fixed, precise objectives. Nevertheless, the lack of such objectives, even if tentative, is adversely affecting attempts to organize and conduct focused and timely technology development. The end points and technologies must be factored into the system early and with reasonable certainty if the program is to be carried out in a cost-effective and timely manner and with the greatest safety to the public. This process requires, at the very least, that working hypotheses be established by DOE with or without the help of other interested parties as to waste streams and waste-disposal conditions. The eventual environmental impact of waste-management practices results from the combined characteristics of waste-treatment end products, the source term, and the disposal environment. The impact also depends on the behavior of future societies, particularly with respect to various ways in which they inadvertently might intrude into the waste-disposal facility. In fact, most performance assessments show that human intrusion is the dominant pathway of exposure. The types of human intrusion scenarios that are assumed are a matter of policy.

Recommendation: A greater focus should be put on defining the required characteristics of the waste streams and the nature of the ultimate disposal sites, even if tentative and only as working targets, is necessary to manage effectively the EM technology-development program (see Appendix A for more details).

A previous NRC report (NRC, 1995) states that, "The environmental remediation of the DOE weapons complex is of such a magnitude that in many ways it is an experiment." It is not simply a task of applying known technology with well-understood effectiveness to well-defined cleanup problems. In many cases, the DOE complex presents unique waste management and remediation problems for which there is no prior experience, and in others, it faces more common problems (e.g., ground water contaminated with dense nonaqueous phase liquid [DNAPL]) for which there are no current satisfactory remediation options. This view that the DOE environmental remediation program is the largest nondefense technology-

development program in the country, was supported by Assistant Secretary for DOE-EM Thomas Grumbly in CEMT's opening session.

Recognizing the experimental nature of the DOE cleanup problem, the previous NRC report (NRC, 1995, p. 3) pointed out that ". . .the initial phases of technology development should be considered an exploration of the means to meet environmental remediation requirements. Remediation efforts now underway are also part of the iterative process of technology development and application, which should be recognized by the focus area groups and the steering committee in their integrated team work efforts, and by feedback mechanisms in the system so that future decisions can benefit."

For effective feedback of lessons learned, progress and results must be documented. Performance measures are needed for the technical-development program. In a study of best R&D management practices in private industry³, more than 50 percent of the respondents put "Learn from post-project audits" in the top third of R&D practices with the greatest potential gain relative to the effort involved.

Systematic evaluation of performance measures is needed in both the technology R&D and technology-application phases, as discussed below.

1. The R&D phase. There should be clear criteria for success or failure of technology R&D efforts, and those efforts should be evaluated while they are being conducted so that unpromising research can be terminated. Little evidence has been seen within EM-50 of any systematic approach to such assessment and evaluation. For example, DOE plans four large-scale decontamination and decommissioning (D&D) demonstrations during the next two years, but it is not clear to the committee how these demonstrations will be evaluated and documented to capitalize on their successes and failures for use in future D&D activities.
2. The application phase. A technology that appears to work well in the demonstration phase may experience difficulties in large-scale application. For that reason, monitoring of the results of such applications is an important part of an assessment and evaluation program. For example, CEMT's Landfills Subcommittee recommends a focused program of long-term performance assessment of barrier walls and caps, because these are key components of the present DOE environmental-restoration strategy. If privatization is aggressively pursued, it will be essential that documentation of the evaluation and assessment process and the corresponding performance measures be built into the contractual expectations for successful bidders.

It is important to recognize that these evaluations of effectiveness are a necessary part of the learning process and should not be viewed as being intended to fix blame for "failures." The aim is to learn from these demonstrations, whether or

not they are judged to be successes. A program of technology development necessarily includes failures as well as successes along the way, and the failures provide valuable information.

Explicit evaluation procedures with quantitative criteria are needed as a basis for determining which projects should be continued and which should be terminated (more detail is provided in the section on prioritization). Termination of R&D projects has been identified by the Industrial Research Institute as one of 12 high-potential activities for improving R&D return.⁴

In this connection, clear identification and evaluation of outcomes would provide a better basis for organizational performance measures than a simple count of completed activities. For example, EM-50 might be scored by considering the number of technological challenges overcome, rather than simply the number of demonstrations conducted, as is now the case in some areas. Similarly, EM-40 might be scored by considering the number of remediation problems positively affected (in risk and/or cost reduction) by implementation of better-than-baseline technology. Use of such performance-based measures would contribute to orienting the DOE organizations toward problem solving.

DOE's ability to apply the results of technology-development efforts will depend to a significant extent on acceptance by outside parties, most importantly, the responsible regulatory authorities and the affected communities and states. For this reason, the committee stated in its previous report (NRC, 1995) that it would be desirable to establish a process "whereby new technology-development efforts and their anticipated results can be taken into account in renegotiating tripartite agreements." The committee believes that the likelihood of acceptance by key outside parties can be enhanced if they are involved in appropriate ways in the process of selecting technologies for development and, of particular importance, the process of assessing and evaluating the results of development and demonstration efforts.

Recommendation: DOE should develop clear protocols and performance measures for assessing and documenting the effectiveness of technology-development projects and field applications. Explicit and measurable criteria for success and procedures for evaluation must be included, including external peer review. Specific evaluation procedures should be included as an integral part of projects, and the results of these evaluations should be documented. These procedures should be applied to current and future projects to identify "lessons learned" that could be applied in future decisions. DOE also should develop a process to disseminate the results of such evaluations throughout the complex to ensure that the lessons are indeed "learned" by decision makers.

Review of the DOE-EM guidance document on merit review and peer evaluation issued in late 1994 (USDOE, 1994a) provides little evidence that a credible external peer-review system is in place to help guide environmental technology development in the EM program. The external peer-review system needed for this guidance must be separate from, and in addition to, the National Research Council's ongoing role of providing overall programmatic review, because the NRC is not a formal reviewer requiring follow-up by DOE-EM.

Peer review is accepted and widely used as a management mechanism to enhance the quality of R&D programs. To be credible and effective, the peer-review process must assure the technical qualifications of the reviewers and avoid real or perceived conflicts of interest that might affect the outcome. Agencies such as the National Science Foundation (NSF), the Environmental Protection Agency (EPA), the National Institutes of Health (NIH), and the National Aeronautics and Space Administration (NASA) have well-established peer-review procedures (e.g., the Science Advisory Board of the EPA) that have been thoroughly tested and could serve as models for DOE's technology-development program.

High priority should be given to the development and implementation of a rigorous external peer-review process to enhance the scientific and technical quality of the EM technology-development program in the near term. To be most effective, procedures should be standardized as far as possible across all functional elements of EM and especially for all focus areas. The technology review process should be tied to and coordinated with DOE's current system of stages or "gates" that determine the level of maturation for technology development (see Chapter 3, Landfills Focus Area). It is appropriate that internal review should include site representation and technology users to ensure that technology-development efforts are timely and responsive to DOE problem-solving needs. However, external peer review is the appropriate mechanism to ensure the technical soundness and merit of specific projects. Selection of external peer reviewers should be based only on technical qualifications.

Additional mechanisms, such as citizen groups and regulators, should be used to obtain the input of stakeholders. Federal agencies have frequently found it useful to seek the advice of independent scientific and technical groups such as the American Physical Society (APS), the Federation of American Societies for Experimental Biology (FASEB), and the American Institute of Biological Science (AIBS) to identify qualified peer reviewers. Such external technical involvement helps an agency establish and maintain the credibility and competence of its scientific and technological programs.

External peer review can also provide a mechanism for consistent guidance. Each focus area should select qualified experts to serve as peer reviewers on a continuing basis for an extended period. Such an approach would provide long-term guidance by allowing the experts to track decisions made in the DOE remedial technology-development program. Although some focus areas have technical review

panels with membership outside of DOE, these panels will not serve the same role as that of external peer reviews.

Peer review can be conducted in many forms: project-focused rapid response, specific technology area review, and comprehensive research needs assessment. By making use of this full range of review modes, DOE-EM can avail itself of constructive criticism without slowing progress toward its technology-development goals.

A helpful associated activity is expert elicitation, which uses expert judgment in a controlled and structured manner to enhance the knowledge base for making a decision. Expert elicitation is being used to provide important input to the performance assessments of the potential repositories and facilities for the disposal and storage of nuclear waste. Agencies such as DOE and the U.S. Nuclear Regulatory Commission (USNRC) are developing detailed guidance documents based on their experiences with expert elicitation. Because of the interrelationship between peer review and expert elicitation, the committee intends to review these guidance documents in future considerations of the peer-review process.

Peer review is being applied to some current programs, such as those of the Landfills Focus Area. Examples of such activities are the review of technology projects presented at the TRU (transuranic waste), TRU Mixed, and Mixed Low-Level Waste Treatment Technologies Technical Peer Review (November 13–15, 1995, in Dallas, Tex.), and the Non-Destructive Assay/Non-Destructive Evaluation (NDA/NDE) Review (January 25–26, 1996, in Pittsburgh, Pa.) Another documented effort is the recently completed review of a microbial filter project, prior to the commitment of funds for development costs in later stages of development. These examples illustrate that peer review can be used to help guide environmental technology development throughout the EM program.

The committee emphasizes that DOE needs an effective external peer-review process and that it should be implemented in a uniform, consistent manner.

Recommendation: DOE-EM needs to develop and uniformly apply a standardized peer-review process designed to assess the following parameters in all five focus areas: the appropriateness of the identified technology needs, the appropriateness of projects to meet specific technology needs, and the soundness of the technical approach being used or proposed on separate projects.

The DOE-EM program needs to improve its awareness and understanding of the availability and development status of technologies applicable to its multiple waste-management problems, not only in its own laboratories and contract organizations, but throughout industry, universities, and worldwide. It should be stated that DOE-EM is not alone in this deficiency; U.S. industry lacks this focus as

well. DOE-EM needs this knowledge of available technologies for the three main purposes that follow:

• to compare its technology needs to the status of technology, leading to the identification of technology gaps that require development efforts;
• to ascertain the extent to which proven technologies that exist in other agencies, the private sector, or overseas might be applicable to a specific problem, thus providing the possibility that further technology development is not required; and
• to assess the potential for commercialization and collaborative development of the technologies being considered for development.

Review of the DOE program and documents have shown that DOE's efforts to keep abreast of the status of technologies related to its needs have been inadequate. Technologies that already exist have been redeveloped, more so in areas related to wastes containing hazardous chemicals (e.g., DNAPL issues), than in areas related to wastes containing radionuclides. In addition, developers routinely are urged to pursue commercialization when many technologies are only applicable to unique DOE wastes or situations. There is also a risk that necessary technology development may be affected adversely, using the rationale that privatization could meet the needs, despite the fact that environmental cleanup conditions are unique to the DOE system and there is no incentive for the private sector to develop technologies.

Recommendation: DOE-EM should undertake an explicit effort to inventory the status of technologies relevant to its interests and disseminate the results throughout the program. This inventory should consider both the wastes to be addressed and the conditions (e.g., requirements for remote operation). It should cover both domestic and international venues and all sectors (government, commercial, academic). Such an inventory will allow DOE-EM to identify those research areas where solutions or partial solutions are already available from external sources, as well as those development areas where major DOE-EM technology-development efforts are required.