5
Overall Findings and Recommendations

The committee found that DOE’s RD&D programs in fossil energy and energy efficiency have yielded significant economic, environmental, and national security benefits; important technological options for potential application in a different (but possible) economic, political, and/or environmental setting; and important additions to the stock of engineering and scientific knowledge in a number of fields. It lauds the DOE’s recent efforts to focus on outcomes of its R&D activities, particularly through its R&D portfolio analysis (DOE, 1999), and believes that the benefits matrix will serve to usefully refine these ongoing activities.

However, the committee also found that DOE has not employed a consistent methodology for estimating and evaluating the benefits of its RD&D programs in these (and, presumably, other) areas. The evaluation frameworks employed by DOE policy makers vary considerably among programs and often rely on inconsistent or unrealistic economic assumptions. Importantly, evaluations tend to focus on the economic benefits from the deployment of technologies, at the expense of the broader array of benefits (realized and otherwise) flowing from these investments of public funds.

Finally, the committee found that the benefits flowing from DOE’s RD&D programs were influenced by the structure and management of the programs. The committee believes that the structure of many of DOE’s RD&D programs in both fossil energy and energy efficiency is now much more conducive to program success, to industry cost sharing throughout all stages of the programs (i.e., inception to market readiness), and to systematic development of technologies from conceptualization, bench- and pilot-scale testing, through demonstration and, in some cases, introduction into the commercial market than it was when the programs were launched, in the heat of the energy crises of the 1970s. Not surprisingly, however, the committee believes that further improvements in program management and structure are possible within current and likely future budgets.

The committee does not recommend an appropriate level of spending by DOE on fossil energy and energy efficiency RD&D programs, nor does it attempt to compare in a more detailed way the returns from the investments of public funds in these two programs. The diverse array of programs and the wide range of sectors served makes any such comparison very difficult, and the complex nature of the benefits flowing from these RD&D programs makes it even more difficult.

As noted elsewhere in this report, it is impossible to assign a dollar figure to all of these benefits, nor are there feasible or defensible metrics for valuing the full array of outputs of these programs. Ultimately, the determination of “appropriate” levels of spending on RD&D in these two programs requires setting priorities in response to a long-range national energy policy, combined with the exercise of political and economic insights, scientific and technological judgments, and expert advice from external and (as nearly as possible) objective evaluations. The committee’s suggested framework for evaluating the programs can inform the application of these judgments. But in public RD&D programs, no less than in those of industry, the allocation of RD&D funds among diverse programs requires the application of priorities based on stated policy, as well as the exercise of judgment, rather than the mechanical application of cost-benefit or other financial techniques.

The committee’s findings and recommendations come in three areas:

  • The benefits (as outlined above) of DOE RD&D programs in fossil energy and energy efficiency;

  • The assessment of DOE’s techniques for evaluating its RD&D programs in fossil energy and energy efficiency and recommendations for improving the techniques; and

  • The assessment of DOE’s R&D portfolio in fossil energy and energy efficiency and recommendations for improvements in its structure and management.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 62
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 5 Overall Findings and Recommendations The committee found that DOE’s RD&D programs in fossil energy and energy efficiency have yielded significant economic, environmental, and national security benefits; important technological options for potential application in a different (but possible) economic, political, and/or environmental setting; and important additions to the stock of engineering and scientific knowledge in a number of fields. It lauds the DOE’s recent efforts to focus on outcomes of its R&D activities, particularly through its R&D portfolio analysis (DOE, 1999), and believes that the benefits matrix will serve to usefully refine these ongoing activities. However, the committee also found that DOE has not employed a consistent methodology for estimating and evaluating the benefits of its RD&D programs in these (and, presumably, other) areas. The evaluation frameworks employed by DOE policy makers vary considerably among programs and often rely on inconsistent or unrealistic economic assumptions. Importantly, evaluations tend to focus on the economic benefits from the deployment of technologies, at the expense of the broader array of benefits (realized and otherwise) flowing from these investments of public funds. Finally, the committee found that the benefits flowing from DOE’s RD&D programs were influenced by the structure and management of the programs. The committee believes that the structure of many of DOE’s RD&D programs in both fossil energy and energy efficiency is now much more conducive to program success, to industry cost sharing throughout all stages of the programs (i.e., inception to market readiness), and to systematic development of technologies from conceptualization, bench- and pilot-scale testing, through demonstration and, in some cases, introduction into the commercial market than it was when the programs were launched, in the heat of the energy crises of the 1970s. Not surprisingly, however, the committee believes that further improvements in program management and structure are possible within current and likely future budgets. The committee does not recommend an appropriate level of spending by DOE on fossil energy and energy efficiency RD&D programs, nor does it attempt to compare in a more detailed way the returns from the investments of public funds in these two programs. The diverse array of programs and the wide range of sectors served makes any such comparison very difficult, and the complex nature of the benefits flowing from these RD&D programs makes it even more difficult. As noted elsewhere in this report, it is impossible to assign a dollar figure to all of these benefits, nor are there feasible or defensible metrics for valuing the full array of outputs of these programs. Ultimately, the determination of “appropriate” levels of spending on RD&D in these two programs requires setting priorities in response to a long-range national energy policy, combined with the exercise of political and economic insights, scientific and technological judgments, and expert advice from external and (as nearly as possible) objective evaluations. The committee’s suggested framework for evaluating the programs can inform the application of these judgments. But in public RD&D programs, no less than in those of industry, the allocation of RD&D funds among diverse programs requires the application of priorities based on stated policy, as well as the exercise of judgment, rather than the mechanical application of cost-benefit or other financial techniques. The committee’s findings and recommendations come in three areas: The benefits (as outlined above) of DOE RD&D programs in fossil energy and energy efficiency; The assessment of DOE’s techniques for evaluating its RD&D programs in fossil energy and energy efficiency and recommendations for improving the techniques; and The assessment of DOE’s R&D portfolio in fossil energy and energy efficiency and recommendations for improvements in its structure and management.

OCR for page 62
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 BENEFITS OF DOE’S RD&D IN FOSSIL ENERGY AND ENERGY EFFICIENCY Finding 1. DOE investments in RD&D programs in both the fossil energy and energy efficiency programs during the past 22 years have contributed to the well-being of U.S. citizens by producing economic benefits, options for the future, and knowledge benefits. It is the committee’s judgment that the benefits of these programs substantially exceed the programs’ costs and contribute to improvements in the economy, the environment, and national security, as described below. The energy efficiency programs evaluated by the committee accounted for roughly 20 percent of the expenditures on all energy efficiency programs in the past 22 years. The committee believes that the programs it reviewed constitute a representative sample of all energy efficiency programs and that the conclusions from this analysis are applicable throughout the energy efficiency portfolio. The committee’s evaluation of fossil energy programs examined programs accounting for more than 70 percent of fossil energy expenditures during this 22-year period. There is a marked difference in the character of the fossil energy programs launched from 1978 to 1985 and that of the programs launched from 1986 to 2000. The fossil energy programs of the 1978 to 1986 period, which was dominated by an atmosphere of crisis following the 1973 oil embargo, emphasized a high-risk strategy for circumventing commercial-scale demonstrations by going directly from bench-scale to large-scale demonstrations to make synthetic fuels from coal and shale oil and to produce oil using enhanced oil recovery techniques. In the second period, however, the fossil energy R&D program was systematic and involved a more diverse portfolio and greater emphasis on increasing the efficiency of electric power generation using natural gas, on reducing the environmental impact when burning coal, and on advanced oil and gas exploration and production. The committee found that a relatively small number of programs in energy efficiency and fossil energy accounted for the majority of the economic and environmental benefits. This characteristic of RD&D programs, in which a few “home runs” are responsible for the majority of returns on investments, is shared by industrial R&D programs and underscores the importance of maintaining a diversified portfolio of investments. The areas in which these benefits were greatest relative to program expenditures include residential and commercial construction, an industry that historically was not particularly innovative, and technologies to reduce environmentally harmful pollution. These are precisely the areas in which one would anticipate that public R&D programs are most likely to prove most effective (see below for additional discussion). By contrast, DOE efforts to push the technology to commercial application in large, accelerated RD&D programs such as coal liquefaction have been extremely risky and prone to cost overruns and generally have yielded relatively small realized economic, environmental, or security benefits relative to their high costs. Again, however, this tendency is not unique to DOE RD&D programs but has been demonstrated in numerous other federal and civilian technology RD&D programs. Finding 1a. Economic benefits. Although the committee was not always able to separate the DOE contribution from that of others, the net realized economic benefits in the energy efficiency and fossil energy programs were judged by the committee to be in excess of the DOE investment. In the energy efficiency programs reviewed by the committee, most of the realized economic benefits to date are attributable to three relatively modest projects in the building sector carried on the late 1970s and 1980s and continuing into the 1990s. The committee estimated that the total realized economic benefits associated with the energy efficiency programs that it reviewed were in the $30 billion range (valued in 1999 dollars), substantially exceeding the roughly $7 billion (1999 dollars) investment for energy efficiency RD&D over the 22-year life of the programs. The committee estimated that the realized economic benefits associated with the fossil energy programs that it reviewed amounted to nearly $11 billion (1999 dollars) over the same 22-year period, some of which is attributed to costs avoided by demonstrating that more stringent environmental regulation is unnecessary for waste management and by addressing airborne toxic emissions. As was noted earlier, the estimated economic benefits of programs instituted during the 1986 to 2000 period, $7.4 billion, exceeded the estimated $4.5 billion cost of that period’s fossil energy programs. The realized economic benefits associated with fossil energy programs from 1978 to 1986, $3.4 billion in 1999 dollars, were, however, less than the costs of the period’s fossil energy programs, $6.0 billion in 1999 dollars. In addition to realized benefits, a number of technologies were developed that provide options for the future if economic or environmental concerns justify their use. For example, the Office of Fossil Energy’s advanced turbine systems (ATS) and integrated coal gasification combined-cycle (IGCC) systems are technologically ready options awaiting changes in the energy marketplace. The energy efficiency RD&D also produced option benefits, with the Partnership for a New Generation of Vehicles (PNGV) and the forest products Industries of the Future program being important examples. The committee made no attempt to evaluate very recent technologies resulting from current R&D programs that extend beyond the period covered by our assessment. For example, carbon sequestration, Vision 21, and the 21st Century Truck Initiative were not assessed, because any benefits that accrue are expected to occur outside the time frame for the committee’s evaluation. This retrospective nature of the evaluation means that no conclusions about the eco-

OCR for page 62
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 nomic, option, or knowledge benefits of the current energy efficiency or fossil energy program portfolios can be drawn from this study. Finding 1b. Environmental benefits. Substantial reductions in pollution evidently resulted from the technologies developed in these programs. Although it is difficult to assign a monetary value to environmental benefits, the committee estimates that the fossil energy and energy efficiency RD&D programs yielded environmental benefits valued at anywhere from $64 billion to $90 billion. For example, in the energy efficiency program, dramatic improvements in refrigerator efficiency reduced the demand for electricity and the pollution resulting from its generation. In the fossil energy program, the atmospheric fluidized-bed combustion program, the flue gas desulfurization program, and the NOx reduction program provided significant environmental benefits, assuming their current deployment and using existing regulation as a baseline. The committee’s attempt to assign a monetary figure to these environmental benefits may well understate them, since it did not try to quantify the environmental value of reduced waste, habitat preservation, and smaller footprints for advanced drilling and other new technologies. Finding 1c. National security benefits. National security has been enhanced by a number of the programs. A number of fossil energy programs (enhanced oil recovery and seismic technology) increased oil production and reserves in the United States and thereby reduced U.S. dependence on imported oil. Several other technologies (one is indirect liquefaction) could produce security benefits if deployed in an environment of significantly higher oil prices. Energy efficiency programs also have provided national security benefits by decreasing oil use somewhat (although not—significantly—in the transportation sector) and, to a lesser degree, by improving the reliability of the electric and natural gas infrastructure. Although automotive fuel economy regulations have provided significant national security benefits by reducing dependence on petroleum in transportation, DOE’s research programs have proven disappointing in this regard. But this will change because of PNGV, which has made significant progress toward its goals, including an 80-mpg full-size automobile. This is a very significant energy security option benefit. The committee assigned dollar values to the oil reduction achieved by the energy efficiency efforts, but not to the electricity reduction. Finding 1d. Knowledge benefits. All the technologies funded by the DOE add to our stock of knowledge in varying degrees. Some of these technologies are still in the R&D stage and it is too early to know if they will be a commercial success; many current programs fall into this category. However, significant knowledge benefits have already been realized from older programs such as advanced turbine systems, which contributed to improved methods for fabricating advanced materials and ceramics, enhanced knowledge of design and cooling techniques for turbine components, and improved understanding of the effects of sulfur on protective coatings within these systems. Fossil energy programs such as enhanced oil recovery, Western gas sands, and seismic technology also have yielded significant knowledge benefits in the areas of reservoir characterization, seismic imaging, and algorithm development. In addition to this analysis of the individual classes of benefits embodied in the conceptual framework, the committee reached the following summary conclusions: The largest (by an order of magnitude) apparent benefits were realized as avoided energy costs in the buildings sector in energy efficiency and environmental cost avoidance from the NOx reductions achieved in fossil energy. This result is not surprising in a balanced research portfolio, which will have its share of failures and moderate successes. On the other hand, it is not possible to predict, a priori, which projects in the portfolio will hit the jackpot. This skewed distribution of realized benefits (the NOx benefit is an environmental benefit, not an economic benefit) underscores the importance of systematically accounting for the less quantifiable benefits by entering them in the benefits matrix. The large realized benefits accrued in areas where public funding would be expected to have considerable leverage. The buildings sector is fragmented, and the prevailing incentive structure is not conducive to technological innovation. The NOx reduction achieved in FE is considered to be an environmental rather than an economic benefit because private markets cannot easily capture it. The importance of standards for driving technology innovation in buildings and in transportation cannot be overemphasized. Often DOE energy efficiency research has been used to provide a proper basis for standards. Important but smaller realized benefits were achieved in the Office of Fossil Energy’s oil and gas program and in the industry programs of EERE’s energy efficiency research. In these cases, the industries involved did have significant private incentives to capture the benefits of energy R&D. Nevertheless, the committee concluded that DOE participation took advantage of the private sector activity to realize additional public benefits. In these cases, however, clearly specifying the DOE role is critically important to ensuring that public funding is likely to produce appropriate benefits. Forced government introduction of new technologies not yet economic has not been a successful strategy. Many of these programs, such as fuels from coal or technologies that would greatly reduce the use of oil in the transportation

OCR for page 62
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 sector, originated in the 1970s and 1980s and were designed to produce large energy security benefits. Recent programs in both energy efficiency and fossil energy have recognized the importance of industry collaboration and of responding to likely economic or policy conditions to create credible benefits. DOE’S APPROACH TO EVALUATING ITS RD&D PROGRAMS Finding 2. Managers of both the energy efficiency and the fossil energy RD&D programs did not utilize a consistent methodology or framework for estimating and evaluating the benefits of the numerous projects within their programs. Evaluations of individual projects or programs within energy efficiency or fossil energy often relied on unrealistic economic frameworks and assumptions or on frameworks and assumptions that were inconsistent across programs or across successive evaluations of any single program or project. These internal evaluations also assigned considerable weight to realized economic benefits to the near-exclusion of other types of benefits. In the judgment of this committee, this narrow focus on realized economic benefits is an inappropriate basis for the evaluation of public RD&D programs in the energy field. Neither DOE nor the outside agencies that evaluated its R&D programs developed a consistent framework for assessing the benefits and costs of these programs, some of which must be appropriately apportioned to the private sector. As a result, decision makers do not have good information on which to base decisions about the effectiveness of R&D expenditures. In the committee’s view, this situation leads to an overemphasis on evaluations of realized benefits, especially economic ones. Although important, these benefits are not the only ones that DOE programs aim to produce. One reason to focus on realized economic benefits is that these should be proportional to environmental and security benefits. In the energy efficiency case studies, there were no environmental or security benefits unless the technology had strongly penetrated the market. In addition to a tendency to assign too much weight to realized economic benefits, especially avoided costs and unshared costs, the inconsistent approach adopted by DOE policy makers to evaluate their programs often was associated with an overstatement of economic benefits. In some cases, such as in low-e windows, DOE failed to consider the costs and benefits of the next best technology, thereby attributing too large a benefit to the technology to which it had contributed. In other cases, as in fossil energy’s ATS program, DOE made unjustified economic claims for the impact that these programs have had on the commercial products now in the marketplace. In all of the examples cited here, the result of DOE’s unrealistic assumptions was a significant overstatement of the benefits of its RD&D programs. The committee believes that a consistent, well-articulated set of assumptions and categories for evaluating costs and benefits would encourage the use of clearer, more realistic assumptions, in many cases reducing a tendency to overstate the benefits attributable to DOE RD&D programs. The piecemeal and inconsistent evaluation methodologies currently employed by DOE also make more difficult the development of a portfolio approach for assessing the overall structure, budget allocations, and appropriateness of program objectives for both fossil energy and energy efficiency. The committee believes that the adoption of the comprehensive, consistent methodology employed in this report would aid policy makers in evaluating their R&D portfolios. The benefits matrix adopted for this study is a robust framework for evaluating program outcomes. Its application imposes a rigor on the evaluation process that clarifies the benefits achieved and the relationship among them. The utility of this framework as an evaluation tool depends fundamentally on the application of specific guidelines for characterizing the benefits produced by program outcomes (e.g., realized vs. option vs. knowledge) and for assigning a value to them. The guidelines developed for this project proved to be a reasonable starting point, but the committee’s experience showed that many issues of characterization and valuation remain to be resolved. The framework is valuable for another reason as well. The rows represent the social objectives of DOE: a reduction in the cost of energy services (economic benefits), environmental benefits, and security benefits. Thus the magnitude of these benefits is a measure of how well DOE is doing in meeting its social good objectives. This is the sort of information needed to evaluate the DOE portfolio. Although the analytic framework needs many improvements, the committee’s application of it and of related guidelines helped identify a number of cases in which benefits were overestimated in the data submitted to it by DOE. In other cases, the committee was able to rationalize DOE claims of very large benefits that might on the surface appear somewhat implausible. Recommendation. DOE should adopt an analytic framework similar to that used by this committee as a uniform methodology for assessing the costs and benefits of its R&D programs. DOE should also use an analytic framework of this sort in reporting to Congress on its programs and goals under the terms of the Government Performance and Results Act. Recommendation. To implement this recommended analytic approach, DOE should consider taking the following steps: Adopt and improve guidelines for benefits characterization and valuation. Convene a workshop of DOE analysts, decision makers, and committee members to discuss

OCR for page 62
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 the problems encountered in the application of the committee’s guidelines (Appendix D) and to consider how to begin the improvement process. Adopt consistent assumptions to be used across programs. Adopt procedures to enhance the transparency of the process, clarifying the choices made in characterizing benefits and the methodology used in valuing them. Provide for external peer review of the application of the analytic framework to help ensure that it is applied consistently for all programs. This independent peer review team should include individuals from industry and other sectors that are not connected to the programs being evaluated. Seek to include the views of all stakeholders in public reviews of its R&D programs. Finding 3. One of the most difficult problems in the evaluation of RD&D programs is determining DOE’s share of the benefit of a program in which industry (and even other government agencies) also made significant contributions. It is essential to spell out specifically the concrete results achieved by DOE’s participation in such programs relative to the efforts of other investors. The discussion of individual fossil energy and energy efficiency programs in preceding chapters shows that DOE programs are effective in very diverse ways, and better data on the nature of program results will aid policy makers in assessing the appropriateness of program structures. Recommendation. Application of the framework requires data that often are difficult to obtain within DOE. DOE should work to overcome these problems by (among other things) consistently recording historical budget and cost-sharing data for all RD&D projects. Industry incurs significant costs to commercialize technology developed in DOE programs, and—especially in the assessment of economic benefits—these costs should be documented where possible. Industry’s investment in many of the technologies evaluated is likely very high. However, for this report, cost sharing was assumed to be industry’s share of the costs of RD&D involving DOE. From the point of view of public benefits, which entity deserves the credit is much less important than that public benefits should exceed public investment costs. Public costs may be quite modest compared to benefits if they catalyze private investments in innovation. PORTFOLIO MANAGEMENT Finding 4. The committee’s review of the fossil energy and energy efficiency programs underscores the significant changes in energy policy during the nearly three decades of the programs’ existence. There have been changes in technological possibilities; expectations about energy supply, prices, and security; DOE programmatic goals; the national and international political environment; and the feasibility and performance of various technological approaches. But this combination of change and uncertainty means that diversification across technological approaches, R&D performers, and likely future states of the world is essential within DOE’s R&D portfolio. A balanced R&D portfolio is particularly important, since individual R&D projects may well fail to achieve their goals. Rather than viewing the failure of individual R&D projects as symptoms of overall program failure, DOE and congressional policy makers should recognize that project failures generate considerable knowledge and that a well-designed R&D program will inevitably include such failures. An R&D program with no failures in individual research projects is pursuing an overly conservative portfolio. Another lesson learned is that care must be taken to assure that goals and objectives are not set so far out as to be utterly unattainable. This does not mean that “stretch” goals are avoided. But it does mean that unrealistic goals should not be promoted to such an extent that interim or compromised successes are ignored, and the overall program is labeled a failure. Recommendation. DOE’s R&D portfolio in energy efficiency and fossil energy should focus first on DOE (national) public good goals, and it should have (1) a mix of exploratory, applied, development, and demonstration research and related activities, (2) different time horizons for the deployment of any resulting technologies, (3) an array of different technologies for any programmatic goals, and (4) a mix of economic, environmental, and security objectives. In addition, it is important to effectively integrate the results of exploratory research projects with applied RD&D activities within individual programs. The committee recommends regular, external review of the DOE energy R&D portfolio, dropping projects that do not have a likelihood of successfully meeting goals set for the program. Finding 5. A significant number of DOE’s programs have focused on environmental issues as part of the national strategy. This is an important role for DOE and could be facilitated by more formal interaction with EPA and the private sector. At present, there is no formal mechanism for communication or interaction between the parties. Recommendation. DOE should work to establish improved communication with EPA and the private sector, with the goal of accelerating deployment of environmentally clean technologies. Finding 6. The case studies illustrate a number of instances in which spending on programs continued past a point justified by program performance as evidenced by the program’s inability to meet technological milestones. This failure to apply rigorous scrutiny to technological progress affected the overall evaluation of benefits across both fossil energy

OCR for page 62
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 and energy efficiency R&D portfolios. Indeed, these programs adversely affected the overall benefit-cost balance for fossil energy and energy efficiency R&D portfolios. In the case of the MHD program, for example, Congress continued to appropriate funds for nearly a dozen years after DOE stopped asking for them. Nearly 45 percent of the $1.02 billion (1999 dollars) in total DOE expenditures on this program was appropriated by Congress between 1982 and 1993, despite the fact that DOE requested funds for the program only once during that time. Since the program yielded no direct benefits and only limited knowledge benefits, its continuation by congressional action for a lengthy period diverted public funds that might have been better spent on other programs. The MHD case is one in which Congress ignored DOE recommendations that program funding be terminated. But a more public review and set of expert recommendations for such termination could have made it more politically costly or difficult for Congress to continue funding. In hindsight, it is apparent that the continued investment of public funds in a program past the point at which it is capable of attaining its original goals drives up costs, especially when the project is continued into early-stage or commercial development. Recommendation. DOE should develop clear performance targets and milestones, including the establishment of intermediate performance targets and milestones, at the inception of demonstration and development programs (in cooperation with industry collaborators, where appropriate) and employ these targets and milestones as go/no-go criteria within individual projects and programs. These performance targets and milestones should be incorporated into DOE funding requests to congressional appropriators. Consideration should be given to the type of research performed when evaluating these targets, as preset milestones may not be applicable in programs focused on exploratory research. Key milestones that can be used in conjunction with established goals for measuring progress and detecting problems should be established for all program and project activity. Use of milestones for monitoring progress is a well-known “best practice” in managing any R&D portfolio, and the DOE energy efficiency portfolio could benefit from more widespread use in managing program and project decisions. As noted previously, the evaluation framework recommended by the committee involves the retrospective analysis of program outcomes, and recently initiated programs may not yet have achieved the ultimate outcomes projected for them. It is therefore important to develop interim milestones and metrics that enable policy makers and program managers to assess intermediate progress toward the ultimate project or program objectives and to make any needed adjustments in program structure or budget in a timely fashion. The knowledge gained during the research may justify reconsideration of these targets. Finding 7. The committee’s review of DOE RD&D programs suggests that programs seeking to support the development of technologies for rapid deployment are more likely to be successful when the technological goals of these programs are consistent with the economic incentives of users to adopt such technologies. Not all DOE RD&D programs have sought such near-term technology deployment, nor does the committee believe that all DOE programs should pursue such near-term goals. Nonetheless, for the programs in which these goals are central, the case studies illustrate a number of instances in which the adoption of the results of DOE RD&D programs, and the associated realization of economic benefits, was aided by regulatory, tax, or other policies that significantly improved the attractiveness of these technologies to prospective users. Conversely, the case studies include a number of instances in which the attainment by DOE RD&D programs of their technical goals (and the production of option or knowledge benefits) did not produce substantial economic benefits because incentives for users to adopt these technologies were lacking. In addition to calling attention to the importance of consistency between the goals of DOE RD&D programs and various public policies and coordination in the development of such goals and associated policies, this point underscores the importance of close collaboration between DOE and industrial users of such technologies in establishing program goals and technological performance targets. Recommendation. Where its RD&D programs seek to develop technologies for near-term deployment, DOE should consider combining support for RD&D with the development of appropriate market incentives for the adoption of these technologies based on an understanding of market conditions and consumer needs. These incentives span the gamut from product standards to tax incentives. Conversely, it is unrealistic to expect immediate deployment of technologies developed with public funds that are suited to a very different environment of energy-related costs and prices. But such technologies may provide significant option and knowledge benefits, and they represent appropriate targets for DOE RD&D programs. Finding 8. The committee’s case studies highlight the importance of flexibility in the RD&D program structure, especially the need for continual reevaluation of program goals against change in the regulatory or policy environment, in projected energy prices and availability, and in the performance or availability of alternative technologies, among other factors. One approach to such ongoing evaluation relies on regular peer review by panels of technical experts selected from nonparticipating firms, academia, and other

OCR for page 62
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 sources of expertise. The work of review panels appears to have benefited the PNGV program, among others. Such panels also may serve as useful mechanisms to assess program performance against targets and milestones and may reduce the likelihood of programs continuing beyond the point where their benefit-cost ratios decline sharply. Recommendation. DOE should expand its reliance on independent, regular, external reviews of RD&D in energy efficiency and fossil energy program goals and structure, enlisting the participation of technical experts who are not otherwise involved as contractors or R&D performers in these programs. Finding 9. The committee found that cost sharing between DOE and industrial collaborators frequently improved the performance of RD&D programs and enhanced the level of economic and other benefits associated with such programs. The appropriate level of cost sharing depends on the specific circumstances of individual programs, including the characteristics of the technology and structure of the industry where the technology may be deployed. Recommendation. DOE should maintain its current policies encouraging industry cost sharing in RD&D programs. In general, industry’s share of program costs should increase as a project moves from early-stage or exploratory R&D through development to demonstration. Policy makers should ensure that an emphasis on collaboration with industry in the formulation of R&D priorities and R&D performance does not result in an overemphasis on near-term technical objectives within the DOE R&D portfolio or in the neglect of public good objectives. Finding 10. The committee’s case studies suggest that the appropriate role for DOE in RD&D programs varies, depending on whether a given program is focused on exploratory research, development, or demonstration, as well as the structure of the industry (including the amount of industry-funded R&D or the presence of well-established industrial R&D consortia) within which a given technology will be deployed. Some industries with which DOE has worked in technology development, such as home building, are populated by numerous small firms that perform little or no internal R&D. In this situation, the DOE role in RD&D extended from technology development through demonstration of the feasibility and cost-effectiveness of new technologies. Deployment of the technology was accelerated by the development of energy-efficiency standards that were widely adopted by local building-code authorities. A somewhat similar situation prevailed in the DOE drilling and enhanced oil recovery programs. The U.S. domestic oil-exploration industry is populated by a large number of small firms with little or no internal R&D, and in this situation DOE acted in part as a supplier of generic R&D that produced useful tools and concepts for industrywide use. Many of the programs in OIT also have involved work to demonstrate technological concepts in industries with relatively small privately funded R&D budgets. The PNGV program, on the other hand, presents a sharp contrast. The U.S. automotive industry is much more highly concentrated and populated by many firms with substantial internal R&D budgets. In this situation, DOE’s role has been one of working with industry to define an agenda for precompetitive R&D that contributes to DOE goals as well as industry needs and that would not raise antitrust issues. DOE also provided financial support for the more long-term elements of the agreed-upon R&D agenda. But much of the R&D performed within the PNGV program is undertaken by the participating firms, in contrast to the situation in the energy efficiency buildings programs. DOE plays an important third-party role between the regulator (DOT), EPA, and the industry, which establishes the credibility of new, expensive knowledge from non-EPA studies that inform the regulatory process. Still another structure for R&D serving the public interest is DOE’s activities in environmental characterization and control. Here, DOE technology demonstration and characterization have contributed to the development of lower-cost methods to meet emissions targets, while also providing federal regulatory agencies with technical information to formulate more realistic and cost-effective regulations. These programs differ greatly in their budgets, in the mix of public and private funding for the RD&D activities they perform, in the “division of labor” between public and private sector actors in the performance of that RD&D, and in the mix of near- and long-term RD&D activities they support. In addition, the operation of these programs has involved a varied mix of policies supporting the adoption of new technologies. The more successful DOE programs have been structured to respond to the unique technological and economic circumstances of each industrial sector that they seek to serve, and they have thereby served the public interest more effectively. Part of the challenge surrounding the program requires that DOE define areas in which its funding or performance of R&D is likely to prove most effective. Recommendation. DOE should strive to build flexibility into the structure of its RD&D programs. DOE RD&D programs have contributed to technological progress and knowledge in a variety of ways that are influenced by the structure and characteristics of the relevant industrial sectors. DOE should structure its RD&D programs to be flexible and regularly evaluate program goals and structure. The committee found that DOE RD&D programs in fossil energy and energy efficiency have developed greater

OCR for page 62
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 flexibility and sensitivity to the needs of the relevant industrial sectors over the past 15 years. The committee applauds this trend and urges that DOE policy makers continue to explore creative and adaptive solutions to the requirements of collaborative RD&D in very diverse industrial sectors. REFERENCE Department of Energy. 1999. Energy Resources R&D Portfolio Analysis. Volume 1: Summary Report. Panel Report to the Research & Development Council. August. Washington, D.C.: U.S. DOE.

OCR for page 62
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 This page in the original is blank.