Executive Summary

From the time of the first Organization of Arab Petroleum Exporting Countries oil embargo nearly 30 years ago, the United States has looked to new technology for solutions to its energy problems. Indeed, the first government reports to recommend an energy research and development (R&D) agenda appeared within weeks of that 1973 event. In 1975, President Ford created the Energy Research and Development Administration (ERDA), consolidating under one umbrella existing R&D energy programs from several agencies. In late 1977, ERDA became part of the new Department of Energy (DOE). And today, energy R&D remains a major element of DOE’s mission.

From 1978 through 1999, the federal government expended $91.5 billion (2000 dollars) on energy R&D, mostly through DOE programs. This direct federal investment constituted about a third of the nation’s total energy R&D expenditure, the balance having been spent by the private sector. Of course, government policies—from cost sharing to environmental regulation to tax incentives—influenced the priorities of a significant fraction of the private investment. On balance, the government has been the largest single source and stimulus of energy R&D funding for more than 20 years.

In legislation appropriating funds for DOE’s fiscal year (FY) 2000 energy R&D budget, the House Interior Appropriations Subcommittee directed an evaluation of the benefits that have accrued to the nation from the R&D conducted since 1978 in DOE’s energy efficiency and fossil energy programs. In response to the congressional charge, the National Research Council formed the Committee on Benefits of DOE R&D on Energy Efficiency and Fossil Energy (the committee).

From its inception, DOE’s energy R&D program has been the subject of many outside evaluations. The present evaluation asks whether the benefits of the program have justified the considerable expenditure of public funds since DOE’s formation in 1977, and, unlike earlier evaluations, it takes a comprehensive look at the actual outcomes of DOE’s research over two decades.

BACKGROUND

A Historical Perspective

From 1978, debate about how best to spend the public’s money has surrounded DOE’s research program. Perhaps the most important change in the debate has been the evolving understanding of the larger goals of energy policy and hence of R&D objectives. Reducing dependence on energy imports (especially oil) persisted as a central tenet of energy policy into the 1980s. During that period, government R&D policy stressed development of alternative liquid fuels. By the early 1980s, more faith was placed in market forces to resolve energy supply and demand imbalances and in the development of technologies to enlarge the former and constrain the latter. In consequence, federal research goals shifted and began to stress long-term, precompetitive R&D. After 1992, technology priorities moved in the direction of renewable energy sources and energy efficiency. And the role of federal funding, having swung between support of expensive demonstration projects and limited funding of basic research, settled into a preference for cost sharing in the form of public-private partnerships.

This brief recounting of the shifting forces that shaped energy R&D over the last 25 years conveys a sense of the twists and turns of both program goals and management philosophy that DOE’s research managers have had to follow since 1978. Without an appreciation of these shifts, evaluating the successes and failures of DOE’s research program would be a very frustrating and puzzling enterprise.

Energy Efficiency and Fossil Energy Research at DOE

The two program areas—energy efficiency and fossil energy—that lie within the scope of this study have expended about $22.3 billion in federal funds since 1978, or about 26 percent of the total DOE expenditure on energy R&D of approximately $85 billion (2000 dollars). Their funding histories reflect the changes in goals and philosophies that have characterized energy research at DOE.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 Executive Summary From the time of the first Organization of Arab Petroleum Exporting Countries oil embargo nearly 30 years ago, the United States has looked to new technology for solutions to its energy problems. Indeed, the first government reports to recommend an energy research and development (R&D) agenda appeared within weeks of that 1973 event. In 1975, President Ford created the Energy Research and Development Administration (ERDA), consolidating under one umbrella existing R&D energy programs from several agencies. In late 1977, ERDA became part of the new Department of Energy (DOE). And today, energy R&D remains a major element of DOE’s mission. From 1978 through 1999, the federal government expended $91.5 billion (2000 dollars) on energy R&D, mostly through DOE programs. This direct federal investment constituted about a third of the nation’s total energy R&D expenditure, the balance having been spent by the private sector. Of course, government policies—from cost sharing to environmental regulation to tax incentives—influenced the priorities of a significant fraction of the private investment. On balance, the government has been the largest single source and stimulus of energy R&D funding for more than 20 years. In legislation appropriating funds for DOE’s fiscal year (FY) 2000 energy R&D budget, the House Interior Appropriations Subcommittee directed an evaluation of the benefits that have accrued to the nation from the R&D conducted since 1978 in DOE’s energy efficiency and fossil energy programs. In response to the congressional charge, the National Research Council formed the Committee on Benefits of DOE R&D on Energy Efficiency and Fossil Energy (the committee). From its inception, DOE’s energy R&D program has been the subject of many outside evaluations. The present evaluation asks whether the benefits of the program have justified the considerable expenditure of public funds since DOE’s formation in 1977, and, unlike earlier evaluations, it takes a comprehensive look at the actual outcomes of DOE’s research over two decades. BACKGROUND A Historical Perspective From 1978, debate about how best to spend the public’s money has surrounded DOE’s research program. Perhaps the most important change in the debate has been the evolving understanding of the larger goals of energy policy and hence of R&D objectives. Reducing dependence on energy imports (especially oil) persisted as a central tenet of energy policy into the 1980s. During that period, government R&D policy stressed development of alternative liquid fuels. By the early 1980s, more faith was placed in market forces to resolve energy supply and demand imbalances and in the development of technologies to enlarge the former and constrain the latter. In consequence, federal research goals shifted and began to stress long-term, precompetitive R&D. After 1992, technology priorities moved in the direction of renewable energy sources and energy efficiency. And the role of federal funding, having swung between support of expensive demonstration projects and limited funding of basic research, settled into a preference for cost sharing in the form of public-private partnerships. This brief recounting of the shifting forces that shaped energy R&D over the last 25 years conveys a sense of the twists and turns of both program goals and management philosophy that DOE’s research managers have had to follow since 1978. Without an appreciation of these shifts, evaluating the successes and failures of DOE’s research program would be a very frustrating and puzzling enterprise. Energy Efficiency and Fossil Energy Research at DOE The two program areas—energy efficiency and fossil energy—that lie within the scope of this study have expended about $22.3 billion in federal funds since 1978, or about 26 percent of the total DOE expenditure on energy R&D of approximately $85 billion (2000 dollars). Their funding histories reflect the changes in goals and philosophies that have characterized energy research at DOE.

OCR for page 1
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 Energy Efficiency Programs Energy-efficient technologies can reduce the life-cycle costs of energy-consuming goods and services paid by consumers and industry, reduce pollutant emissions, reduce the risk of oil supply interruptions, and help to stabilize the electricity system and make it more reliable. DOE’s energy efficiency research, development, and demonstration (RD&D) programs have helped to improve the energy efficiency of buildings technology and industrial and transportation technologies. The transportation sector has always received the largest share of the budget (42 percent in 2000 and, cumulatively, 43 percent between 1978 and 2000). In the early years of the program (for example, in FY 1978), buildings received 40 percent of the funds and industry, 18 percent. In FY 2000, there was less of a difference, with buildings receiving 25 percent of the funds and industry, 32 percent. Over the entire program, industry and buildings each received about 28 percent of the funds. Fossil Energy Programs Research in the Office of Fossil Energy has historically focused on two programs: the Office of Coal and Power Systems and the Office of Natural Gas and Petroleum Technology. Very large budgets from 1978 through 1981 were provided in response to the energy crises of the 1970s and early 1980s. During that period, over 73 percent of the money was provided for technologies to produce liquid and gas fuel options from U.S. energy resources—coal and oil shale. Over the 1978 to 2000 study period, 58 percent of the expenditures were for RD&D in coal utilization and conversion. Of this, approximately one-half was spent on direct liquefaction and gasification for building and operating large, commercial-scale demonstration plants between 1978 and 1981. In 1978, the coal conversion and utilization portion of the budget represented 68 percent of the total fossil energy expenditures, but since then, as funding for direct liquefaction and gasification declined, it has represented a considerably lower percentage. In 2000, it represented only 30 percent of the overall fossil energy budget for the technology programs analyzed. The share of Office of Fossil Energy funds devoted to environmental characterization and control was 4 percent of the total over the study period, partly because the Environmental Protection Agency (EPA) maintained a large program in this area prior to 1985. The share of funds for the electricity production programs averaged 24 percent over the study period, and the share of funds for the oil and gas programs averaged 14 percent, one-third of which was for shale oil R&D in the early period. EVALUATION FRAMEWORK AND CASE STUDIES In theory, evaluating the benefits and costs of DOE’s research program should be relatively straightforward. It would require adding up the total benefits and costs of research conducted since 1978, determining what proportion of each is attributable to DOE funding, and calculating the difference between the DOE contributions and the cost of achieving them. In practice, methodological challenges abound. Of these, the most fundamental is how to define and systematically capture the diverse benefits that result from publicly funded research within a dynamic environment of marketplace activity, technological advancement, and societal change. See Chapter 2 and Appendix D for further details on the framework for doing this. Evaluation Framework Justification for public sector research rests on the observation that public benefits exist that the private sector cannot capture. In such cases, the private costs of developing and marketing a technology may exceed the benefits that the private sector can capture. The committee developed a comprehensive framework based on this general philosophy that would define the range of benefits and costs, both quantitative and qualitative, that should be considered in evaluating the programs. Depending on the outcomes of the R&D undertaken, the principal benefit of a program, for example, may be the knowledge gained and not necessarily realized economic benefits. The matrix shown in Figure ES-1 and discussed below provides an accounting framework for the consistent, comprehensive assessment of the benefits and costs of the fossil energy and energy efficiency R&D programs. The matrix can be completed for each discrete program, project, or initiative that has a definable technological objective and outcome. The framework is intended to summarize all net benefits to the United States, to focus attention on the main types of benefits associated with the DOE mission, and to differentiate benefits based on the degree of certainty that they will one day be realized. It has been designed to capture two dimensions of publicly funded R&D: (1) DOE research is expected to produce public benefits that the private economy cannot reap and (2) some benefits may be realized even when a technology does not enter the market-place immediately or to a significant degree. The classes of benefits (corresponding to the rows of the matrix) are intended to capture types of public benefits appropriate to the objectives of DOE R&D programs. Based on these stated objectives, the committee adopted the three generic classes of benefits (and related costs) for the energy R&D programs—economic, environmental, and security benefits: Economic net benefits are based on changes in the total market value of goods and services that can be produced in the U.S. economy under normal conditions, where “normal” refers to conditions absent energy disruptions or other energy shocks and the changes are made possible by technological advances stemming from R&D.

OCR for page 1
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000   Realized Benefits and Costs Options Benefits and Costs Knowledge Benefits and Costs Economic benefits and costs       Environmental benefits and costs       Security benefits and costs       FIGURE ES-1 Matrix for assessing benefits and costs. Environmental net benefits are based on changes in the quality of the environment that have occurred or may occur as a result of a new technology RD&D program. Security net benefits are based on changes in the probability or severity of abnormal energy-related events that would adversely impact the overall economy, public health and safety, or the environment. The three columns in the matrix are the first step toward a more explicit definition of the benefits to be included. They reflect different degrees of uncertainty about whether a given benefit will be obtained. Two fundamental sources of uncertainty are particularly important—technological uncertainties and uncertainties about economic and policy conditions (Figure ES-2). Rather than attempting to fully characterize the uncertainty of benefits, the committee used these two distinctions—the state of technology development and the favorability of economic and policy conditions—to define the columns of the matrix (Figure ES-1). The first column, “realized benefits and costs,” is reserved for benefits that are almost certain—that is, those for which the technology is developed and for which the economic and policy conditions are favorable for commercialization of the technology. The second column, which includes less certain benefits, is called “options benefits and costs.” These consist of benefits that might be derived from technologies that are fully developed but for which economic and policy conditions are not likely to be, but might become, favorable for commercialization. All other benefits, to the extent they exist, are called “knowledge benefits and costs.” The framework recognizes that the technologies being evaluated may be in different stages of the RD&D cycle, and by its nature, it represents a snapshot in time, with a focus on outcomes of the work performed. To arrive at entries for the cells of the matrix, a logical and consistent set of rules for measuring the results of the individual initiatives is also necessary. These rules define more exactly the meanings of the rows and columns, and they provide a calculus for measuring the values to be entered in each of the cells. Case Studies To assess the benefits of the energy efficiency and fossil energy programs within this evaluation framework, the com- Economic/Policy Conditions\Technology Development Technology Developed Technology Development in Progress Technology Development Failed Will be favorable for commercialization Realized benefits Knowledge benefits Knowledge benefits Might become favorable for commercialization Options benefits Knowledge benefits Knowledge benefits Will not become favorable for commercialization Knowledge benefits Knowledge benefits Knowledge benefits FIGURE ES-2 Derivation of columns for the benefits matrix.

OCR for page 1
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 mittee prepared a series of case studies on technologies and programs selected by the committee for examination. It should be noted that there were large differences in project scale, size, complexity, and time horizon between the energy efficiency and fossil energy programs. In particular, the fossil energy program tends to be characterized by relatively large, long-term projects. As a result, the committee was able to select a manageable number of case studies—22—that covered almost all of the research expenditures in the DOE fossil energy program since 1978. In contrast, the energy efficiency program, especially in the buildings and industry programs, is composed of a large number of relatively small projects. The committee determined that it was not possible to analyze enough cases to capture a large fraction of DOE’s research expenditures in these areas. Therefore, the committee selected 17 case studies that, in its expert opinion, were sufficiently representative to permit the testing of the analytical framework and to draw reliable conclusions about the success or failure of the overall program. The criteria for selecting this representative group are explained in Chapter 3. Perhaps the most difficult analytic problem is assigning to DOE a proportion of the overall benefit of an R&D program that properly reflects DOE’s contribution to it. In most of the case studies, DOE, industry, and sometimes other federal and nonfederal governmental research organizations contributed to the outcome of the research program. The committee found no reliable way to quantify the DOE contribution in most cases, and doing so remains a methodological challenge for the future. For the purposes of this study, it simply attempted to specify in its case study analyses the specific role that DOE played—the outcome that would not have happened had DOE not acted. Based on this assessment, the committee used conservative judgment to characterize the DOE contribution for purposes of developing findings and recommendations. No conclusions about the benefits of unevaluated current energy efficiency or fossil energy programs can be drawn from this study. In Tables ES-1 and ES-2, each of the 39 case studies the committee examined is slotted into the benefits matrix. If a technology has more than one kind of benefit, the primary benefit is indicated by boldface type. Energy Efficiency Although the issues, problems, and solutions for energy efficiency may be different for each of the three end-use sectors (buildings, industry, and transportation), lessons learned from one sector are often applicable to all the sectors. To study the energy efficiency program comprehensively, the committee selected case studies to illustrate the main components of the program, important examples of RD&D activities, and the range of benefits and costs that the program has yielded (see Selection of the Case Studies in Chapter 3). The 17 case studies represent $1.6 billion, or about 20 percent, of the total $7.3 billion energy efficiency TABLE ES-1 Energy Efficiency Technology Case Studies Slotted in the Matrix Cells That Are Most Relevant Today Type of Benefit Realized Benefits Options Benefits Knowledge Benefits Economic benefits (net life-cycle energy cost reductions) Low-e glass Electronic ballasts Advanced refrigerators Advanced turbine systems Oxygen-fueled glass furnace Lost foam casting DOE-2 (applied to design) Forest products Forest products Compact fluorescents DOE-2 (applied to standards) Compact fluorescents Black liquor gasification Forest products Oxy-glass technology (applied to other areas) Lost foam Free-piston Stirling heat pump (failure) Environmental benefits Indoor air quality, infiltration, and ventilation Electronic ballasts Advanced refrigerators Low-e glass Oxy-glass PNGV DOE-2 Indoor air quality (IAQI&V) Forest products Catalytic converters for diesels PEM fuel cell for transportation and distributed generation Black liquor gasification Advanced batteries for electric vehicles Indoor air quality (sick buildings) Stirling engine for automobiles (failure) Security benefits Advanced turbine systems PNGV DOE-2 (peak load analysis) Advanced batteries for electric vehicles PEM fuel cells for transportation and distributed generation NOTE: PEM, proton exchange membrane; PNGV, Partnership for a New Generation of Vehicles. The table does not indicate possible future position as a result of completing R&D. No significance should be attached to the ordering of the entries in the cells. When more than one type of benefit is relevant for a technology, the primary benefit is shown in bold.

OCR for page 1
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 TABLE ES-2 Fossil Energy Technology Case Studies Slotted in the Matrix Cells That Are Most Relevant Today Type of Benefit Realized Benefits Options Benefits Knowledge Benefits Economic benefits Drilling/completion/stimulation Atmospheric fluidized-bed combustion Western gas sands Eastern gas shales Improved enhanced oil recovery Field demonstration programs Seismic technology Coal-bed methane Waste management and utilization Improved indirect liquefaction Improved direct liquefaction Drilling/completion/stimulation Atmospheric fluidized-bed combustion Advanced turbine system Fuel cells Western gas sands Eastern gas shales Improved enhanced oil recovery Shale oil Flue gas desulfurization IGCC Coal preparation Mercury and air toxics Improved indirect liquefaction Drilling/completion/stimulation Improved direct liquefaction Pressurized fluidized-bed combustion Advanced turbine system Fuel cells Gas to liquids Magnetohydrodynamics Western gas sands Eastern gas shales Improved enhanced oil recovery Field demonstration Seismic technology Flue gas desulfurization Coal-bed methane Downstream fundamentals IGCC Coal preparation Waste management Mercury and air toxics Environmental benefits Drilling/completion/stimulation Atmospheric fluidized-bed combustion Western gas sands Eastern gas shales Improved enhanced oil recovery Field demonstration programs Seismic technologies NOx control Coal-bed methane Improved indirect liquefaction Drilling/completion/stimulation Pressurized fluidized-bed combustion Advanced turbine systems Fuel cells Eastern gas shales Field demonstration programs Shale oil Flue gas desulfurization NOx control IGCC Improved indirect liquefaction Drilling/completion/stimulation Fluidized-bed combustion Advanced turbine systems Improved enhanced oil recovery Shale oil Field demonstration Seismic technology Flue gas desulfurization IGCC NOx control Waste management Mercury and air toxics Security benefits Drilling/completion/stimulation Improved enhanced oil recovery Field demonstration programs Seismic technologies Improved indirect liquefaction Drilling/completion/stimulation Improved direct liquefaction Field demonstration programs Shale oil Drilling/completion/stimulation Fuel cells NOTE: When more than one type of benefit is relevant for a technology, the primary benefit is shown in boldface type. NOx, oxides of nitrogen; IGCC, integrated gasification combined cycle. R&D expenditures over the 22-year period. Included are both successes and failed or terminated projects. As noted above, the selection process did not involve a statistical sampling of all the projects; instead, it attempted to choose a representative sample of energy efficiency projects. Fossil Energy The committee compiled case studies for 22 of the fossil energy RD&D programs funded between 1978 and 2000. These case studies account for nearly $11 billion (73 percent) of the $15 billion appropriated to the Office of Fossil Energy for RD&D during the period. CONCLUSIONS AND RECOMMENDATIONS The committee found that DOE’s RD&D programs in fossil energy and energy efficiency have yielded significant benefits (economic, environmental, and national security-related), important technological options for potential application in a different (but possible) economic, political, and/or environmental setting, and important additions to the stock of engineering and scientific knowledge in a number of fields. The committee also found that DOE has not employed a consistent methodology for estimating and evaluating the benefits from its RD&D programs in these (and, presum-

OCR for page 1
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 ably, in other) areas. Importantly, DOE’s evaluations tend to focus on economic benefits from the deployment of technologies, rather than taking into account the broader array of benefits (realized and otherwise) flowing from these investments of public funds. Finally, the committee found that how DOE’s research programs were organized and managed made a real difference to the benefits that were produced by the research. Benefit-Cost Assessment The committee found that DOE investments in RD&D programs in both the fossil energy and energy efficiency programs during the past 22 years produced economic benefits, options for the future, and knowledge benefits. Although the committee was not always able to separate the DOE contribution from that of others, the net realized economic benefits in the energy efficiency and fossil energy programs were judged by the committee to be in excess of the DOE investment. In the programs reviewed by the committee in the energy efficiency area, most of the realized economic benefits to date are attributable to three relatively modest projects in the building sector carried out in the late 1970s and 1980s and continuing into the 1990s. The committee estimated that the total net realized economic benefits associated with the energy efficiency programs that it reviewed were approximately $30 billion (valued in 1999 dollars), substantially exceeding the roughly $7 billion (1999 dollars) in total energy efficiency RD&D investment over the 22-year life of the programs. The committee estimated that the realized economic benefits associated with the fossil energy programs that it reviewed amounted to nearly $11 billion (1999 dollars) over the same 22-year period, some of which it attributed to costs avoided by demonstrating that more stringent environmental regulation is unnecessary for waste management and for addressing airborne toxic emissions. The realized economic benefits of fossil energy programs instituted from 1986 to 2000, $7.4 billion, exceeded the estimated $4.5 billion cost of the programs during that period. However, the realized economic benefits associated with the fossil energy programs from 1978 to 1986, estimated as $3.4 billion in 1999 dollars, were less than the costs of this period’s fossil energy programs ($6.0 billion in 1999 dollars). In addition to realized benefits, a number of technologies have been developed that provide options for the future if economic or environmental concerns justify their use. For example, the advanced turbine system (ATS) and the integrated gasification combined-cycle (IGCC) system are technologically ready options awaiting changes in the energy marketplace. The energy efficiency programs in RD&D also produced option benefits, with Partnership for a New Generation of Vehicles (PNGV) and forest products (Industries of the Future) being important examples. Substantial reductions in pollution evidently resulted from technologies developed in these programs. Although it is difficult to assign a monetary value to environmental benefits, the committee estimates that both RD&D programs yielded environmental benefits valued conservatively at $60 billion to $90 billion. National security has been enhanced by a number of the programs. For example, a number of fossil energy programs (enhanced oil production and seismic technologies) increased oil production and reserve additions in the United States and thereby reduced U.S. dependence on imported oil. Although fuel economy regulation has provided significant national security benefits by reducing the country’s dependence on petroleum in transportation, DOE’s research programs have proven disappointing in this regard. The options benefit of PNGV, although not yet realized, is in the oil security area. All the technologies funded by the DOE add to our stock of knowledge in varying degrees. In addition to its analysis of the individual classes of benefits embodied in the conceptual framework, the committee reached the following summary conclusions: By an order of magnitude, the largest apparent benefits were realized as (1) avoided energy costs in the buildings sector in energy efficiency and (2) avoided environmental costs from the NOx reductions achieved by a single program in fossil energy. This result is not surprising given the balanced research portfolio, which also includes its share of failures and modest successes. These large realized benefits accrued in areas where public funding would be expected to have considerable leverage. For one thing, the buildings sector is fragmented, and the prevailing incentive structure is not conducive to technological innovation. For another, the NOx reduction achieved in fossil energy is an environmental benefit that private markets cannot easily capture. The importance of standards pulling technological innovation in buildings and transportation cannot be exaggerated. Often, DOE energy efficiency research has been used to provide a proper basis for standards. Important but smaller realized benefits were achieved in fossil energy’s oil and gas program and energy efficiency’s industry programs. Here, the committee concluded that DOE participation indeed took advantage of the private sector activity to realize additional public benefits. In these cases, however, a clearly defined DOE role is crucial to ensuring that public funding is likely to produce appropriate benefits. Forced government introduction of new technologies has not been a successful strategy. Recent programs in both energy efficiency and fossil energy have recognized the importance of industry collaboration and of responding to likely economic or policy conditions to create credible benefits.

OCR for page 1
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 Program Evaluation The committee found that managers of both the energy efficiency and the fossil energy RD&D programs did not utilize a consistent methodology or framework for estimating and evaluating the benefits of the numerous projects within their programs. In addition to a tendency to assign too much weight to realized economic benefits, especially avoided costs and unshared costs, the inconsistent approach adopted by DOE policy makers to evaluation of their programs often was associated with an overstatement of economic benefits. The benefits matrix adopted for this study is a robust framework for evaluating program outcomes. Its application imposes a rigor on the evaluation process that clarifies the benefits achieved and the relationship among them. Recommendation. DOE should adopt an analytic framework similar to that used by this committee as a uniform methodology for assessing the costs and benefits of its R&D programs. DOE should also use an analytic framework of this sort in reporting to Congress on its programs and goals under the terms of the Government Performance and Results Act. Recommendation. To implement this recommended analytic approach, DOE should consider taking the following steps: Adopt and improve guidelines for benefits characterization and valuation. Convene a workshop of DOE analysts, decision makers, and committee members to discuss the problems encountered in the application of the committee’s guidelines and to consider how to begin the improvement process. Adopt consistent assumptions to be used across programs. Adopt procedures to enhance the transparency of the process. Provide for external peer review of the application of the analytic framework to help ensure that it is applied consistently for all programs. Seek to include the views of all stakeholders in public reviews of its R&D programs. DOE programs may be effective in very diverse ways, and better data on the nature of program results will aid policy makers in assessing the appropriateness of program structures. It is essential to report specifically the concrete results achieved by DOE’s participation in such programs relative to the efforts of other investors. Application of this framework requires data that often are difficult to obtain within DOE. Public costs may be quite modest compared with the benefits if they catalyze private investments in innovation. Recommendation. DOE should consistently record historical budget and cost-sharing data for all RD&D projects. Industry incurs significant costs to commercialize technology developed in DOE programs, and—especially in the assessment of economic benefits—these costs should be documented where possible. Portfolio Management The committee’s review of the fossil energy and energy efficiency programs underscores the significant changes in energy policy during the nearly three decades of the programs’ existence. There have been changes in technological possibilities; expectations about energy supply, prices, and security; DOE programmatic goals; the national and international political environment; and the feasibility and accomplishments of various technological approaches and R&D performers. A balanced R&D portfolio is particularly important since individual R&D projects may well fail to achieve their goals. Rather than viewing the failure of individual R&D projects as symptoms of overall program failure, DOE and congressional policy makers should recognize that project failures generate considerable knowledge and that a well-designed R&D program will inevitably include such failures. An R&D program with no failures in individual research projects is pursuing an overly conservative portfolio. Recommendation. DOE’s R&D portfolio in energy efficiency and fossil energy should focus first on DOE (national) public good goals, and it should have (1) a mix of exploratory, applied, development, and demonstration research and related activities, (2) different time horizons for the deployment of any resulting technologies, (3) an array of different technologies for any programmatic goals, and (4) a mix of economic, environmental, and security objectives. In addition, it is important to effectively integrate the results of exploratory research projects with applied RD&D activities within individual programs. Recommendation. DOE should develop clear performance targets and milestones, including the establishment of intermediate performance targets and milestones, at the inception of demonstration and development programs (in cooperation with industry collaborators, where appropriate) and employ these targets and milestones as go/no-go criteria within individual projects and programs. The committee’s review of DOE RD&D programs suggests that programs seeking to support the development of technologies for rapid deployment are more likely to be successful when the technological goals of these programs are consistent with the economic incentives of users to adopt such technologies. For the programs in which these goals are central, the case studies illustrate a number of instances in

OCR for page 1
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 which the adoption of the results of DOE RD&D programs and the associated realization of economic benefits were aided by regulatory, tax, or other policies that significantly improved the attractiveness of these technologies to prospective users. Conversely, the case studies include a number of instances in which the attainment by DOE RD&D programs of their technical goals (and the production of option or knowledge benefits) did not produce substantial economic benefits, because incentives for users to adopt these technologies were lacking. Such technologies may provide significant option and knowledge benefits, and they represent appropriate targets for DOE RD&D programs. Recommendation. Where its RD&D programs seek to develop technologies for near-term deployment, DOE should consider combining support for RD&D with the development of appropriate market incentives for the adoption of these technologies based on an understanding of market conditions and consumer needs. The committee’s case studies highlight the importance of flexibility in the RD&D program structure, especially the need for periodic reevaluation of program goals against change in the regulatory or policy environment, the projected energy prices and availability, and the performance or availability of alternative technologies, among other factors. Recommendation. DOE should expand its reliance on independent, regular, external reviews of RD&D in energy efficiency and fossil energy program goals and structure, enlisting the participation of technical experts who are not otherwise involved as contractors or R&D performers in these programs. The committee found that cost sharing between DOE and industrial collaborators frequently improved the performance of RD&D programs and enhanced the level of economic and other benefits associated with such programs. Recommendation. DOE should maintain its current policies encouraging industry cost sharing in RD&D programs. In general, industry’s share of program costs should increase as a project moves from early-stage or exploratory R&D through development to demonstration. Policy makers should ensure that an emphasis on collaboration with industry in the formulation of R&D priorities and R&D performance does not result in an overemphasis on near-term technical objectives within the DOE R&D portfolio or in neglect of public good objectives. The committee’s case studies suggest that an appropriate role for DOE in RD&D programs varies, depending on whether a given program is focused on exploratory research, development, or demonstration, as well as the structure of the industry (including the amount of industry-funded R&D or the presence of well-established industrial R&D consortia) within which a given technology will be deployed. The committee found that DOE RD&D programs in fossil energy and energy efficiency have developed greater flexibility and sensitivity to the needs of the relevant industrial sectors over the past 15 years. The committee applauds this trend and urges that DOE policy makers continue to explore creative and adaptive solutions to the requirements of collaborative RD&D in very diverse industrial sectors. Recommendation. DOE should strive to build flexibility into the structure of its RD&D programs.