Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) Summary OVERVIEW In recent years, federal oversight of public expenditures has sought to integrate performance and budgeting. Notably, the Government Performance and Results Act (GPRA) was passed in 1993 “in response to questions about the value and effectiveness of federal programs” (GAO, 1997, p. 11). GPRA and other mandates have led agencies to develop indicators of program performance and program outcomes. The development of indicators has been watched with keen interest by Congress, which asked the National Research Council (NRC) for a series of reports using quantitative indicators to evaluate the effectiveness of applied energy R&D. The first such report1 took a retrospective view of the first 23 years of R&D programs sponsored by the Department of Energy (DOE) on fossil energy and energy efficiency.2 That report found that DOE-sponsored research had netted large commercial successes—such as advanced refrigerator compressors, electronic lighting ballasts, and emission control technology for flue gas desulfurization (NRC, 2001). Other programs, however, were judged to have been costly failures in which large R&D expenditures did not result in a commercial energy technology (NRC, 2001). A follow-up NRC committee was assigned the task of adapting the retrospective methodology to the assessment of the future payoff of continuing programs (NRC, 2005a). The present report continues the NRC’s investigation of R&D outcome indicators and applies the benefits evaluation methodology to six DOE R&D activities. The report further defines indicators for environmental and security benefits and refines the evaluation process based on the experience with the case studies. Evaluating the outcome of R&D expenditures requires an analysis of program costs and benefits. Doing so is not a trivial matter. First, the analysis of costs and benefits must reflect the full range of public benefits that are envisioned, accounting for environmental and energy security impacts as well as economic impacts. Second, the analysis must consider how likely the research is to succeed and how valuable the research will be if successful. Finally, the analysis must consider what might happen if the government does not support the project: Would some non-DOE entity undertake it or an equivalent activity that would produce some or all of the benefits of government involvement? The process and methodology developed by the Phase One committee, summarized in Appendix F, are designed to address these challenges. Pursuant to GPRA, DOE submits its own analysis of program costs and benefits in an annual report to Congress accompanying the President’s Budget Request. Additional analyses are required of DOE and all federal agencies by the Office of Management and Budget (OMB), the executive agency that formulates and administers the federal budget. The President’s Management Agenda (PMA) (OMB, 2001) sets forth nine agency-specific reforms and five governmentwide goals (OMB, 2005); a set of R&D investment criteria was spelled out in 2003, implementing one of the agency-specific goals of the PMA (Marburger and Daniels, 2003). Also in 2003, OMB inaugurated the Program Assessment Rating Tool (PART) (OMB, 2003, pp. 47-53) to assess and improve program performance. The principal responsibility at DOE for developing applied energy technologies resides in three offices—the Office of Energy Efficiency and Renewable Energy (EERE), the Office of Fossil Energy (FE), and the Office of Nuclear Energy Science and Technology (NE).3 Of their combined budget authority of $2.4 billion in FY05, the three offices devoted approximately $1.3 billion to R&D. The R&D pro- 1 Requested by Congress in the conference report of the Consolidated Appropriations Act for fiscal year (FY) 2000 (House Report 106-479, p. 493. November 18, 1999. Washington, D.C.: U.S. Government Printing Office). 2 These programs include only those that were at the time under the jurisdiction of the U.S. House Appropriations Subcommittee on the Interior and Related Agencies. 3 The Office of Electricity Delivery and Energy Reliability, created in 2002, could be considered a fourth applied energy program.
OCR for page 2
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) grams are complemented by policy measures such as tax incentives to encourage early adoption of advanced technologies by consumers, efficiency standards for household appliances, and production tax credits for certain renewable energy sources (EIA AEO, 2006). The programs evaluated by the NRC were limited to R&D within FE and to the portion of EERE’s R&D devoted to energy conservation.4 Research within FE has traditionally been divided between the Office of Coal and Power Systems (CPS) and the Office of Oil and Natural Gas. CPS administers a suite of clean coal R&D, which has the goal of ensuring the generation of clean, reliable, and affordable electricity from coal. The Office of Oil and Natural Gas supports research and policy options to ensure clean, reliable, and affordable supplies of oil and natural gas for American consumers.5 The energy conservation portion of EERE’s R&D is related to technologies for the efficient end-use of fuels and electricity in vehicles, in industrial processes, and within building envelopes. Two general activities have been the focus of this Phase Two study: refining the methodology developed in Phase One and applying it to additional R&D projects. The committee improved the methodology for estimating environmental benefits (from, for example, reduced emissions) and estimating national security benefits (from, for example, reduced oil imports). In parallel, the committee selected six R&D activities to be the subject of case studies, which were carried out by separate expert panels appointed by the NRC. The activities selected for review included three within EERE—the Chemicals subprogram of the Industrial Technologies Program, the Distributed Energy Resources Program, and the activities related to light-duty hybrid electric vehicles within the Vehicles Technologies Program—and three within FE—the Integrated Gasification Combined Cycle subprogram, the Carbon Sequestration Program, and the Exploration and Production activities of the Natural Gas Technologies Program. This Phase Two study shows that the basic analytical structure, using decision trees, works well and can be implemented with the appropriate panels. HOW PROGRAM BENEFITS ARE EVALUATED Introduction The primary effects of DOE’s programs are seen to be these: (1) they reduce technical risk, (2) they reduce market risk, and (3) they accelerate the introduction of the technology into the marketplace. The methodology developed by the Phase One committee used expert panels to review the DOE R&D program and estimate the expected economic, environmental, and energy security benefits of the program in three different global economic scenarios, with the results summarized in a matrix such as that shown in Figure S-1 (see Appendix F for generalized definitions of economic, environmental, and energy security benefits). The expert panel evaluation process is facilitated by a decision analysis consultant, and the panels construct simple decision trees to describe the main technical and market uncertainties associated with the program and the impact of DOE support on the probability of various technical and market outcomes. The decision trees used by all the panels assessed changes in technical and market risks. The acceleration effect was considered separately by each panel. In some cases, acceleration increases the likelihood that a project will attain the program goals of completion by a critical date, which is then accounted for in the assessment of technical risk. In other cases, the panels accounted for acceleration in their benefit calculations, which assume that if the technology is ultimately developed in the absence of the government program the net benefits accrue only for a limited time. The calculations based on the decision trees allow the benefits of each R&D project to be estimated for combinations of outcomes—technical and market—in each of these global economic scenarios. These scenario- and outcome-specific results would typically be estimated using a simple spreadsheet model in conjunction with more sophisticated models such as the Energy Information Administration’s (EIA’s) National Energy Modeling System (NEMS).6 For example, NEMS might be used to estimate prices and demand for various energy sources in a particular global economic scenario. A spreadsheet model might be used to estimate the demand for the particular technology of interest given different levels of effectiveness and/or costs. The overall benefit of the DOE R&D program is given as the difference between the expected benefits with DOE support and the expected benefits without DOE support, where the expected benefits are given as a probability-weighted average of the benefits in particular technical and market outcomes. To ensure consistency across the expert panels, the process calls for the use of common scenarios and assumptions across evaluations and an oversight committee that provides guidance to the panels. This kind of panel-based probabilistic assessment of R&D programs is common in many industries, in particular, the pharmaceutical industry (Sharpe and Keelin, 1998). 4 The remainder of EERE’s R&D was devoted to energy supply from renewable resources. Beginning with FY06, funding for all FE and EERE R&D programs was consolidated into one appropriations account subject to the jurisdiction of the House Appropriations Subcommittee on Energy and Water Development and Related Agencies (CRS, 2005). 5 Available at <http://www.fossil.energy.gov/programs/oilgas/index.html>. 6 NEMS is a computer-based, energy-economy system for modeling U.S. energy markets that projects the production, imports, conversion, consumption, and prices of energy, subject to assumptions about macroeconomic and financial factors, world energy markets, resource availability and costs, behavioral and technological choice criteria, cost and performance characteristics of energy technologies, and demographics.
OCR for page 3
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) FIGURE S-1 Results matrix. Methodology Public programs often have social benefits that are not valued by markets. Assessing the value of such benefits is inherently difficult, involves ambiguity, and, even as an academic matter, a range of possible answers. For the DOE programs, two broad classes of benefits have this characteristic: the environmental benefits of energy technology and the security benefits of energy savings or energy alternatives. These program attributes are in general critical components of the benefits package—indeed, if a program can be justified simply on a market benefits basis, the rationale for government participation might be open to question. Environmental Benefits Although there are a host of land, water, and, perhaps, public health impacts to consider, the evaluation of benefits in monetary units related to reducing criteria air pollutants is both the primary environmental benefit identified in the present study and the class of benefits for which valuation methods are most advanced. Recommendation 1: Panels should apply valuations in monetary units to criteria air pollutant emissions in the results matrix but not to other types of pollutant emissions.
OCR for page 4
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) The valuations used should be the allowance price forecasts for the future period. Energy Security Benefits Electricity. While the complex relationship between electricity supply and security is becoming clearer, analysts are a long way from having methods for valuing reductions in security threats contributed by technologies such as distributed generation.7 Recommendation 2: Panels conducting prospective benefits assessments should describe reductions in threats to energy security related to electricity supply as physical quantities of oil and gas. Oil and Gas. Increases in U.S. oil and gas consumption and imports may impose incremental costs that are not fully reflected in the market price. For oil, several social cost components have been estimated in various studies, together comprising the so-called “oil premium.” In principle, similar estimates could be made for natural gas, but the committee is unaware of any such research. Recommendation 3: Panels should describe energy security benefits related to reduced oil and natural gas consumption quantitatively in the benefits matrix as physical quantities of oil and gas. The time pattern of the oil consumption impacts should be made explicit, along with an assessment of the probable state of the oil market during those future times. Conclusions The committee also reached the following conclusions in regard to the methodology: The committee endorses the decision tree framework for use in estimating the benefits of DOE’s applied research programs. However, panels must take care and follow the guidance of a decision analyst (the consultant) to understand how to assign probabilities and how to specify the government role clearly. The global scenarios developed by the committee for all panels proved to be a valuable tool for characterizing and quantifying the benefits of the DOE R&D programs. However, the Phase Two experience shows that panels sometimes needed to clarify aspects of a scenario to address issues that were important to the specific R&D program. The NEMS model is important for providing baseline energy prices and demands, but using it to estimate the prices and demands of all different program outcomes is unlikely to yield refinements that can affect the estimated benefits in a meaningful way. The success or failure of competing or complementary technologies can significantly affect the value of a DOE applied research project. Although Phase Two shed light on this issue and provided some methodological guidance for dealing with it, more work is required to describe a method for estimating the benefits of DOE’s overall portfolio made up of separate programs or subprojects. The panels were generally successful in implementing the committee’s methodology, but the Phase Two experience did highlight some process issues that should be considered in future studies. Process The experience with the six expert panels and case studies has led the committee to make several recommendations on the process. In particular, the committee found that the commitment and the technology background of the panel members determined the quality of the assessment of a program. Recommendation 4: Panel composition and level of expertise must be critically considered during the selection process. If a panel concludes that certain skills are not possessed by its members, it should consider adding members or using an outside expert to brief it. The leadership role of the panel chair cannot be over-emphasized. For a panel to succeed, the chair had to take the lead in interacting with DOE to ensure that the best possible information was available to the panel before its first meeting. Moreover, the panels where the chairs spent significant time ensuring that all panel members were fully familiar with the process and methodology produced the best assessments. Recommendation 5: The panel chair should spend a fair amount of time outside the actual panel meetings working with the DOE program managers, DOE management, and the independent consultant(s). The primary responsibility of the consultant(s) was to maintain consistency across the panels in applying the methodology and facilitating the analysis. This included structuring the decision trees, facilitating the assignment of probabilities to technical and market outcomes, and providing guidance for and assistance in modeling benefits. Phase 7 The Distributed Energy Resources Panel noted that security pertains to (1) safety from terrorist attacks, (2) insensitivity to energy disruptions caused by reductions in oil imports (or other imports), and (3) customer protection from the effects of disruptions to utility electric service. A unique benefit of distributed energy production technologies is to decentralize power production and locate it at or close to loads, hence providing benefits of types (1) and (3).
OCR for page 5
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) Two made use of the same consultant for all of the panels, and the arrangement worked very well. Recommendation 6: Depending on the number of programs being evaluated and the panels’ schedules, it might be necessary to have more than one consultant. One consultant should focus on the decision tree development and probability assessment and the other on the modeling of benefits. The timeliness and quality of information provided by DOE to the various panels played a critical role in facilitating the deliberations and conclusions of the panel and impacted the quality and utility of its evaluation. Completion of panel evaluations was contingent on the panel’s receiving synoptic information and benefits calculations inputs. Recommendation 7: Since the usefulness of the benefits estimates depends on the quality and timeliness of information available to the panels, DOE management should give its full support for providing the necessary information. DOE at all levels should buy into this process because it is useful for managing and assessing its programs. If this commitment is not clear, the committee should explore all avenues for gaining DOE support. Quality control continued to be important in ensuring the consistency, and therefore the utility, of the panel evaluations. Recommendation 8: An oversight committee should apply the quality control process to several elements of the study process, including ensuring appropriate panel membership and composition, orienting the panel chair and consultant, monitoring the panel’s progress, monitoring information received from DOE for adequacy and consistency, and reviewing and revising the process itself. RESULTS OF BENEFITS EVALUATIONS Approach The expert panels evaluated benefits using the process and methodology summarized above and described in detail in Chapters 3 and 4 of the Phase One report (NRC, 2005a): That process and the associated methodology are summarized here in Appendix F, which was provided to panelists before their first meeting. The improvements to the methodology described in the above section on methodology were suggested by the committee in parallel with the case studies and thus did not alter the approach taken by the expert panels. However, future case studies will make use of these suggestions. Results of Six Case Studies As noted above, the committee selected six case studies to test the proposed methodology and guide refinements and extensions, three within EERE and three within FE. The results of the benefits evaluation for EERE activities are summarized in Table S-1. The three EERE activities had combined annual funding of $115 million in FY05 out of a total of $768 million spent on R&D by EERE. The results of the benefits evaluation for FE activities are summarized in Table S-2. The three FE activities had combined annual funding of $105 million in FY05 out of a total of $561 million spent on R&D by FE. The completion costs are cumulative quantities, calculated assuming the program receives level (constant) funding at the FY05 amount, starting in FY06 through the year in which major goals are achieved. Tables S-1 and S-2 display program benefits as calculated in each of three standard scenarios that were used in all six case studies: (1) a reference scenario; (2) a high oil and gas prices scenario; and (3) a carbon-constrained scenario. The reference scenario is based on the EIA Reference Case from its Annual Energy Outlook 2005, with oil and gas prices ranging from $34 per barrel and $5.3 per thousand cubic feet (Mcf), respectively, in 2005 to $30 per barrel and $4.8 per Mcf, respectively, in 2025. In contrast, the High Oil and Gas Prices scenario has oil and gas prices ranging from $68 per barrel and $10.6 per thousand cubic feet (Mcf), respectively, in 2005, to $78 per barrel and $9.7 per Mcf, respectively, in 2025. The Carbon Constrained scenario assumes a carbon price of $100 per ton of carbon emissions, equivalent to $27 per ton of carbon dioxide. A fourth scenario is invoked as needed to calculate benefits in a future state of the world that would appeal to the unique performance characteristics of the technology under consideration. Economic benefits are measured in dollars, environmental benefits in terms of the physical quantities of avoided emissions, and security benefits as physical quantities of reduced resource consumption (some of which would be reflected in reduced imports). All benefits are computed net of savings that would have been realized in the absence of the DOE program—that is, as a result of R&D by U.S. industry or foreign entities. The results summarized in Tables S-1 and S-2 are described in more detail in Chapter 1 and Appendixes H-M. When reviewing these results it is important to bear in mind that these studies were conducted to test the proposed methodology and guide refinements and extensions. The committee believes that the estimated benefits are useful indications of program benefits, but the reader should be aware that some panels expressed concern about the reliability of their estimates because they lacked good information about some aspects of the program.
OCR for page 6
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) TABLE S-1 Benefits of Three EERE R&D Programs Program Program Completion Costs (Assuming Level Funding) Economic Benefits (Cumulative Net Savings)a Environmental Benefits (Cumulative Reduction in Emissions) Security Benefits (Cumulative Reduction in Resource Consumption) Industrial Technologies Program—Chemicals $75 M through 2015 Ref: $534 M High O&G: $950 M CC: $550 M All scenariosb 24,700 MT CO 15,000 MT SO2 22,600 MT NOx 280 MT PM 540 MT VOCs 2.87 MMTCE All scenarios Natural gas: 89 Bcf Petroleum: 1.3 million bbl Distributed Energy Resourcesc $205 M through 2015 Ref: $57 M High O&G: $46 M CC: $64 M Other: $83 Me Unknownd Ref: 10 TBtu of primary energy High O&G: 8 TBtu CC: 11 TBtu Other: 15 TBtue Light-Duty Vehicle Hybrid Technologiesf $567 M through 2012 Ref: $5.9 B to $7.2 Bg High O&G: $27.5 B to $28.2 B CC: $7.3 B to $8.5 Bg Ref: 28 MMTCE High O&G: 51 MMTCE CC: 32 MMTCE Ref: 219-224 M bbl gasoline High O&G: 398-405 M bbl gasoline CC: 248-252 M bbl gasoline NOTES: High O&G, High Oil and Gas Prices scenario; Ref, Reference Case from EIA’s Annual Energy Outlook 2005 (EIA, 2005b); CC, Carbon Constrained scenario; Other, fourth scenario added by panel; MMTCE, million metric tons carbon equivalent; TBtu, trillion British thermal units; Bcf, billion cubic feet; bbl, barrels; GHG, greenhouse gases; MT, metric tons; M, million; B, billion; CO, carbon monoxide; SO2, sulfur dioxide; NOx, oxides of nitrogen; VOCs, volatile organic compounds; PM, particulate matter. aFor ITP–Chemicals program, benefits are cumulative through 2030. For the DER program, benefits are cumulative through 2025. For LDV Hybrid Technologies program, benefits are cumulative through 2050. Economic benefits have been discounted at 3 percent annually. bThe chemical industries panel concluded that the scenarios would produce insignificantly small changes in the volumes of oil and gas saved; therefore, the physical quantities reported for environmental and security benefits are the same for all scenarios. Economic benefits differ because prices differ from one scenario to the next. cIncludes only the end-use system integration and interface activity of the DER program. dDER program can improve or worsen the environment depending on location, fuel used or displaced, and the technology deployed. The DER panel was not confident about assigning environmental benefits to the DER program as modeled and presented by DOE. eThe “other” scenario for distributed energy assumed constrained electricity transmission. fIncludes the following elements of the vehicles technology program: Hybrid and Electric Propulsion, excluding projects related to heavy vehicles; Advanced Combustion R&D, limited to the combustion and emission control R&D activity; and Materials Technology, excluding projects related to heavy vehicles and excluding the high-temperature materials laboratory activity. gThe two values correspond to two different market scenarios—high and low—for hybrid vehicle penetration. ADVICE TO USERS OF THE BENEFITS EVALUATION RESULTS The committee has developed some insights into the methodological strengths and weaknesses of the proposed process. These insights, recorded in the form of recommendations below, may assist decision makers with interpreting and applying the results of the analysis. Policy measures that have nothing to do with research can have an effect on when and whether the benefits of some programs will be realized. For example, the benefits of carbon capture and sequestration depend on the size and timing of a carbon tax (or equivalent policy intervention in the market). The scenarios are a valuable tool for characterizing and quantifying the benefits of the DOE R&D program. Recommendation 9: Decision makers should consider the impact of other policy measures—that is, policies not related to research—in all domains of action (federal, state, and international, say) when considering the results of prospective benefits evaluations. Having a common set of scenarios is useful in general, although additional scenarios may be called for in some cases. While defining the scenarios more completely would be helpful for interpreting the outcomes of the analysis, at the same time it is essential to preserve flexibility by keeping the scenarios as broad as possible. The panel evaluations permit calculation of a benefit-cost ratio. However, the benefit-cost ratio is not the correct metric for optimizing the allocation of additional resources among the programs in a portfolio. Recommendation 10: To allocate resources, DOE should know the marginal benefit of a budget increase on a program-by-program basis. To calculate the marginal benefit, the decision tree should be examined to identify the outcomes that
OCR for page 7
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) TABLE S-2 Benefits of Three FE R&D Programs Program Program Completion Costs (Assuming Level Funding) Economic Benefits (Cumulative Net Savings)a Environmental Benefits (Cumulative Reduction in Emissions) Security Benefits (Cumulative Reduction in Consumption or Importation of Natural Gas)b Natural Gas Exploration and Productionc $140 M through 2015 Ref: $220 M High O&G: $590 M CC: $300 M Not quantifiedd Ref: 1.2 Tcf High O&G: 0.6 Tcf CC: 1.2 Tcf Integrated Gasification Combined Cycle Technology $750 M through 2020 Ref: $6.4 B to $7.8 B High O&G: $7 B to $47 B Ref: −90 to 30 MMTCE High O&G: 34 to 36 MMTCE Ref: up to 4.5 Tcf natural gas High O&G: up to 3.6 Tcf natural gas Carbon Sequestration $875 M through 2020 CC: $3.5 B Other:e $3.9 B CC and other:e Net environmental benefits are likely zero.f CC and other:e Net security benefits are likely zero.g NOTES: High O&G, High Oil and Gas Prices scenario; Ref, Reference Case from Energy Information Administration’s Annual Energy Outlook 2005 (EIA, 2005b); Other, fourth scenario added by panel; CC, Carbon Constrained scenario, which assumes a $100 tax per ton of carbon emissions; MMTCE, million metric tons carbon equivalent; Tcf, trillion cubic feet; quad, quadrillion British thermal units; MT, metric tons; M, million; B, billion. aFor the Natural Gas Exploration and Production program, benefits are cumulative through 2025. For IGCC and Carbon Sequestration, benefits accrue over the 20-year book life of new plants built through 2025. Economic benefits have been discounted at 3 percent annually. bFor the natural gas technology program, security benefits are expressed as offsets of imported natural gas. For the IGCC program, benefits are reduced U.S. consumption. cIncludes the exploration and production activities only. dModest environmental benefits could result from a reduction in disturbed area and, if fuel switching from coal occurs, from lower carbon emissions. eThe “other” scenario devised for the carbon sequestration program assumes a $300 tax per ton of carbon emissions. fOther technologies that could be deployed in a CC regime would reduce emissions by the same amount as those of the DOE program, albeit at higher cost. gThe least-cost alternatives to the technology under development in the DOE program would be a combination of nuclear generation and renewable electricity that also would reduce natural gas consumption. The costs of other technologies that would reduce carbon emissions were taken from DOE forecasts. would be most sensitive to changes in budget levels. In the Phase One study, for example, the lighting program proved to be highly budget-dependent. When such sensitivities exist, the decision tree can be re-estimated for a different budget level, using the committee’s methodology. The marginal benefit associated with the change in budget level is the difference between the net benefits of the two calculations. The methodology estimates benefits for each of three scenarios that describe future states of the world, but it does not attempt to combine the three sets of benefits into a single set. As a result, the decision maker must weigh the alternative scenarios in arriving at judgments about the benefits of the overall research portfolio. Presumably, the portfolio would contain a balance of projects that would produce acceptable results across the range of scenarios outcome. The methodology estimates public benefits in three areas—economic, environmental, and energy security. While these three types of benefits reflect DOE’s strategic goals (DOE, 2005a), the committee recognizes that other kinds of benefits may be important in evaluating some projects. For example, markets demand that automobile manufacturers produce cars that not only meet the fuel economy objective that is important to DOE but also satisfy several other needs as well. An example of another need might be employment. Recommendation 11: If benefits in areas other than economic, environmental, or energy security are found to occur, they should be noted in the text accompanying the results matrix. However, the matrix should stay focused on the three main types of benefits to facilitate comparisons across programs.
Representative terms from entire chapter: