Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) (2007)

A methodology is presented for prospectively evaluating the benefits of applied energy research and development (R&D) programs. The types of benefits to be evaluated are based on three fundamental objectives that have guided energy policy since the energy crisis of 1973-1974: economic improvement, environmental protection, and energy security. Two principal sources of uncertainty in the benefits calculation are considered: (1) the risk associated with the technical success of an R&D program and (2) the risk associated with market acceptance of a technology. The methodology calls for the use of a decision tree as a framework for organizing the benefits calculation, which is then presented in a results matrix that uniformly summarizes important data and estimated benefits for all technology programs.

Prospective evaluation is complicated by uncertainty about how the future will unfold. A standard way to take uncertainties into account in cost-benefit analysis is to consider the “expected benefit,” which is the probability-weighted average of the benefits associated with all the possible outcomes of a program.

The benefits framework proposed for prospective evaluation incorporates these characteristics of the possible outcomes and attendant investment risk. This framework is summarized in matrix form in Figure F-1. The bottom three rows represent the same three kinds of benefits—economic, environmental, or security benefits—considered in the retrospective analysis, Energy Research at DOE: Was It Worth It? (NRC, 2001).

Economic net benefits are based on changes in the total market value of goods and services that can be produced in the U.S. economy under normal conditions, where “normal” refers to conditions absent energy disruptions or other energy shocks. Economic value can be increased either because a new technology reduces the cost of producing a given output or because the technology allows additional valuable outputs to be produced by the economy. Economic benefits are characterized by changes in the valuations based on market prices. This estimation must be computed on the basis of comparison with the next-best alternative, not some standard or average value.

Environmental net benefits are based on changes in the quality of the environment that have occurred, will occur, or may occur as a result of the technology. These changes could occur because the technology directly reduces the adverse impact on the environment of providing a given amount of energy service, for example by reducing the sulfur dioxide emissions per kilowatt-hour of electric energy generated by a fossil fuel fired power plant, or because the technology indirectly enables the achievement of enhanced environmental standards, for example by introducing the choice of a high-efficiency refrigerator. Environmental net benefits are typically not directly measurable by market prices. They can often be quantified in terms of reductions in net emissions or other physical impacts. In some cases market values can be assigned to the impacts based on emissions trading or other indicators.

Security net benefits are based on changes in the probability or severity of abnormal energy-related events that would adversely impact the overall economy, public health and safety, or the environment. Historically, these benefits arose in terms of “national security” issues, initially the assurance of energy resources required for a military operation or a war effort. Subsequently they focused on dependence on imported oil and vulnerability to interdiction of supply or cartel pricing as a political weapon. More recently, the economic disruptions of rapid international price fluctuations from any cause have come to the fore.

These three classes of benefits have been chosen to reflect the programmatic goals of the U.S. Department of Energy (DOE) offices for which the study was conducted. The three classes are not meant to be comprehensive. For example,

FIGURE F-1 Results matrix for evaluating benefits and costs prospectively.

they do not include the benefits of fundamental research sponsored by the Office of Science, health benefits, or other quality of life benefits that could be unintended but real consequences of some applied R&D programs.

The benefit of a new technology will often depend on developments quite unrelated to the technology itself. For example, the benefit of energy-efficient lighting will depend on the cost of electricity, which in turn depends on the costs of fuels like natural gas and coal used to generate electricity. Similarly, the economic benefits associated with carbon sequestration will depend on carbon emissions being constrained or taxed and the level of constraint. Thus, assumptions about the future of prices and environmental constraints, among other things, can have a significant effect on the prospective benefits of a technology.

The scenarios in Figure F-1 represent three possible future states of the world that are likely to affect the benefits associated with a wide variety of DOE applied R&D programs. It is recommended that the same three scenarios be used to evaluate all the programs. The use of a common set of scenarios across program evaluations will allow reviewers to consider many programs without having to learn definitions for multiple scenarios and will facilitate comparisons

across programs. To ensure consistency, it is important for the scenarios used in the review of different programs to be built on precisely the same assumptions (e.g., the same oil and gas price assumptions); it is not sufficient for them to be similar in some high-level or vague sense.

The three global scenarios are as follows:

1. Reference Case. This is the scenario developed by the Energy Information Administration and described in the Annual Energy Outlook (AEO) (EIA, 2004). The AEO provides detailed forecasts of U.S. energy supply, demand, and prices through 2025. This scenario represents the government’s official base-case forecast. The 2004 Reference Case assumed as follows:

   • World oil prices decrease from their current levels to about $24 in 2010 and then increase to about $27 per barrel in 2025.

   • Natural gas consumption increases significantly—i.e., 23 trillion cubic feet (Tcf) to 26 Tcf in 2010 and 31 Tcf in 2025—with prices decreasing from current levels to $3.49 per thousand cubic feet (Mcf) in 2010, then increasing to $4.47 per Mcf in 2025.¹

   • Primary energy consumption increases from 97.7 quadrillion British thermal units (quads) in 2002 to 136.5 quads in 2025.

   • GDP grows 3.0 percent per year to 2025.

   • Carbon dioxide emissions from energy consumption grow from 5,729 million metric tons in 2002 to 8,142 million metric tons in 2025.

2. High Oil and Gas Prices. This scenario assumes that oil prices will remain very high throughout the period and that constraints on natural gas supply lead to higher natural gas prices and higher electricity prices. For example, the oil price in 2012 in this scenario is $33.41 versus $23.98 in the Reference Case, and the natural gas price in 2012 is $4.53 per Mcf versus $3.92 per Mcf in the Reference Case.

3. Carbon Constrained. This scenario, developed by the DOE, assumes that U.S. emissions of carbon are constrained in response to environmental concerns. Specifically, this scenario assumes that the Global Climate Change Initiative goal of an 18 percent reduction in national greenhouse gas intensity (below the 2002 level) is achieved by 2012 (White House, 2002) and that annual emissions are held constant at that level thereafter. Relative to the Reference Case, this leads to increased demand for natural gas and increased prices (for example, $6.79 per Mcf in 2012 versus $5.54 per Mcf in 2012 in the 2004 Reference Case) as well as greater reliance on renewable electricity, unless carbon sequestration technologies are successful.

The three scenarios considered in the prospective benefits matrix (Figure F-1) are not intended to capture everything that could happen in the future. Indeed, there are an uncountable number of different possible futures, including other levels of oil and gas prices and carbon constraints. Therefore, reviewers are discouraged from specifying probabilities for these scenarios in order to calculate a single “expected benefit” that represents a probability-weighted average across the three scenarios. Rather, the three scenarios are intended as a representative set of scenarios that highlight particular policy issues and provide a form of sensitivity analysis for the benefits analysis. Displaying the results across the three scenarios rather than collapsing the scenarios into a single expected value allows reviewers to focus on scenarios that they view as more likely or as representing particular policy objectives or interests. For example, a policy maker contemplating constraints on carbon emissions might look carefully at benefits in the Carbon Constrained scenario to see what DOE is doing to help prepare the United States for this possibility.

Projects and programs may yield benefits in some but not all scenarios. In the study Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward (NRC, 2005a), the carbon sequestration program was viewed by the expert panel as providing benefits only in the Carbon Constrained scenario. The panel did not believe the sequestration technology would be deployed under the two other scenarios and so would yield zero benefit. But even if the benefit of a program is zero for a given scenario, it is important that all program evaluations consider benefits in all of the scenarios. While carbon constraints or high oil and gas prices might not be particularly relevant to the performance of lighting technologies, the economic benefit associated with efficiency enhancement might vary across scenarios as electricity prices change.² Even if the benefit does not change, the scenarios should be considered and expected benefits reported, because at some point reviewers may combine benefits across projects or programs for a particular scenario, and the absence of an estimate for benefit in a given scenario should not be confused with zero benefit.

While these three global scenarios should serve to illustrate the potential benefits of most DOE applied R&D programs, if a particular program is designed to provide benefits under some other set of circumstances, the DOE and/or review panels are invited to consider benefits in additional scenarios. In such a case, the alternative scenario should be described clearly and its benefits for this scenario should be calculated and reported in a manner consistent with the principles outlined elsewhere in this report. It should be considered in addition to the three global scenarios considered in the prospective benefits matrix—that is, it should not be

viewed as a substitute for or a modification of one or more of those scenarios.

The estimated benefit of a program is subject to multiple sources of uncertainty, both as a result of what happens in the program itself and what happens to the alternative technologies or in the policy environment. The basic principle of program assessment remains the same as in a retrospective analysis: Find the difference between social benefits with the government program and without it. However, the implications of the various uncertainties must be considered carefully. In this section, the formal mechanism for estimating benefits—the decision tree framework—is described. First, some of the key factors to consider in benefit assessment are discussed.

It is essential first to define the outcomes for which it is worth calculating the benefits. Three factors typically determine the alternative outcomes of an applied R&D program. One (or two) of the factors might not be important in a specific case, but their ubiquity implies that each assessment should look at all three so that analysis teams can determine which of them need to be explicitly incorporated into the benefit calculations. Each is discussed briefly.

An estimation of the expected net benefit of a government program involves an explicit or implicit comparison of the possible outcomes with the government program and the possible outcomes absent the program. For example, a government program might lead to a research team making a significant technology advance. The expected benefits of that advance could be estimated. However, to determine the net benefits of the government support, it is necessary to consider the extent to which this or other research teams are likely to have achieved the same technology advance absent the government support and to estimate the benefit and probability of the advance. The expected net benefit of the government program is the difference between the expected benefits with and without government support, not the expected benefit of the technology advance with government support.

Similarly, estimation of the expected net benefits of a government program requires either explicit or implicit consideration of how the market would evolve without the technology being developed by the DOE research program. Take, for example, the estimation of net benefits of a successful government program for new solid state lighting technology. One could assess the benefits of the new lighting technology advance assuming it is coupled with a government program designed to hasten its market adoption. One would have to compare these benefits with those of a program designed to hasten the adoption of the next-best lighting technology—say, the next generation of compact fluorescent lights. It would not be appropriate to estimate the expected net benefits based on a comparison with the existing generation of compact fluorescents or (and this would be even less appropriate) the existing generation of incandescent lighting technology.

In some cases, there might be considerable uncertainty about the benefits that accrue due to advances in alternative technologies; in some cases the benefits might change radically depending on potential, but uncertain, advances in the next-best technology. Commercial penetration of a moderately successful fuel cell car, for example, might be substantially reduced if the fuel efficiency of hybrid-electric vehicles improves dramatically. In cases such as this, it may be necessary to consider explicitly different levels of success for the next-best technologies and the probabilities that these levels will occur.

In many cases, estimating the expected net benefits of a government program requires either explicit or implicit consideration of enabling or complementary technologies. Whenever the market acceptance of a particular technology depends strongly on the existence of other technologies complementing the technology in question or enabling it, it is necessary to assess the probability that enabling or complementary technology will be successful. For example, a successful government program to advance the technology of hydrogen fueling stations for light-duty vehicles would have little benefit unless the various technologies for hydrogen vehicles—particularly fuel cells and onboard storage—were to advance enough that many people would choose to purchase hydrogen fuel cell vehicles; thus it is necessary to assess the probabilities that these complementary technologies will be successful in order to accurately estimate the net benefits of the government program to advance the technology of hydrogen fueling stations.³

In some cases, the benefits of a program are greatly enhanced by complementary technologies but the program

will provide at least some value even in their absence. If the distinction is important, then an assessment of the probability of success for the complementary technology and the expected benefits of the program considering both potential outcomes—the benefits of the program with and without the complementary technology—will be needed.

Estimating benefits requires the application of a decision tree process that includes consideration of the three key components of government energy R&D program evaluation and clarifies the relationships among them. Figure F-2 illustrates the possible relationships among these three key components using a decision tree, where the first node (the decision node) represents the government action—to pursue the program or not; the second node (first chance node) is the possible outcome of the program; and the third node (second chance node) represents the multiple factors that determine market acceptance, including developments in the next-best technology and the success of enabling and complementary technology programs.

FIGURE F-2 Decision tree.

The decision tree provides a framework for organizing the benefits calculation. The expert panel must specify probabilities for the various uncertainties in the tree. Because the probabilities need not be the same in each global scenario, the expected benefit could differ in each scenario. Then, as discussed in the preceding section, the value of the government program is the difference between the expected benefit of the with-government-support alternative and that of the without-government-support alternative.

Figure F-3 illustrates the decision tree applied to the advanced lighting program, with numerical values included. These numerical values are provided only to show the general structure of such a decision tree and to illustrate the calculations that would be used.

In this decision tree, the government has one basic decision, to invest in the R&D program or not to invest. In either case, three possible levels of lighting efficacy could be achieved by U.S. industry: 150 lumens per watt (lpw), 100 lpw, or no change from the current situation. If DOE invests, then the probability of the greatest advance, 150 lpw, would be increased to 10 percent (from 0 percent in the absence of DOE research); the probability of the medium advance, 100 lpw, would be increased to 50 percent (from 30 percent in the absence of DOE research); and the probability that there would be no advance would be decreased to 40 percent (from 70 percent in the absence of DOE research). Note that these probability assessments do not show that DOE investment will guarantee success—they show that DOE investment increases the probability of successful outcomes.

DOE investment also can change the probability that other

FIGURE F-3 Example of decision tree applied to advanced lighting program. The net expected benefit is the difference between the upper and lower half, or $813 million. The expected benefit of the upper half of the tree (for which DOE invest? = yes) is $3,136 million—the sum of the first nine probability-weighted benefits in the column on the far right. The expected benefit of the lower half of the tree (for which DOE invest? = no) is $2,323 million—the sum of the second set of nine probability-weighted benefits. See also discussion in the text.

nations will have R&D successes. These probabilities are shown in the first column. This decision tree assumes that there will be a 5 percent probability that other nations will achieve 150 lpw if DOE conducts a research project but no probability of that success in the absence of DOE research. It assumes that there will be a 50 percent probability of other nations achieving 100 lumens per watt, with or without DOE investment. With DOE research, there is a 45 percent probability that other nations will not have any incremental improvement; without DOE research, that probability would be 50 percent.

Given those probabilities of U.S. and foreign success, the probabilities of the combinations of outcomes are given in the second column. For example, the probability of U.S. industry achieving 150 lpw and other nations achieving 100 lpw is .05, calculated as the product of .10 and .50.

The third column provides the conditional benefit estimates, which vary according to the combinations of outcomes being considered. Here again, the numbers are presented only for the purpose of illustration. For example, the benefit to the United States, conditional on U.S. industry’s achieving 150 lpw and other nations’ achieving 100 lpw is $4,926 million.

The contributions to the expected value are given in the third column, as the product of the numbers in the first and second columns. The contribution to expected value of the combination—U.S. industry achieving 150 lpw and other nations achieving 100 lpw—is $246 million, the product of .05 and $4,926 million.

The expected value of all possible combinations of outcomes, given DOE R&D investment, is the sum of the top nine numbers in the column on the right. This sum is $3,136 million. The expected value of all possible combinations of outcomes if DOE makes no such R&D investment is the sum of the bottom nine numbers in the column on the right. This sum is $2,323 million.

Note that as a result of U.S. industry R&D and R&D in other nations, the United States would capture benefits even without DOE investment. But those benefits would be smaller than they would be if DOE did invest in R&D.

The difference between the benefits with DOE investment—$3,136 million—and the benefits absent DOE investment—$2,323 million—is the expected value of the gain resulting from DOE investment. This difference of $813 million is the overall benefit from the DOE investment. The overall benefit is not the benefit with DOE investment—$3,136 million—since a large share of those gains to the United States would occur even without DOE investment.

Evaluating the probabilities associated with the different branches on the tree—the likelihood of technical success for the program and the likelihood it will achieve different levels of market penetration—is a critical part of the work of the expert panels. The following subsections include guidelines for assessing probabilities at each of the nodes in the decision tree.

The difference between paths with and without the government program is a measure of the government role. As indicated earlier, evaluation seeks not just to measure change that accompanies a government program, project, or other activity, but also to determine whether the change is attributable to specific government intervention. Evaluation must rule out alternative, competing explanations for an observed or predicted change. For example, a government program may be launched to develop a new energy-conserving technology, and, indeed, it may succeed. But the same outcome, or a portion of it, may have resulted without the government program. The decision tree methodology suggests using a counterfactual approach in the attempt to isolate or demonstrate the effects of the program under evaluation. In a decision tree analysis, one branch describes the path if a specified government program with identified characteristics, including a defined level of funding, is adopted. An alternative branch of the tree describes the path without that government program. Other branches may describe the path of the government program at various funding levels. The probabilities and values for the paths along each branch are based on the assumption that the program is funded at the specified level. By comparing the resulting expected value benefits for alternative branches, the differential expected benefits attributable to the government program can be identified.

A government program may affect societal benefits in multiple ways, including the following:

1. The program might result in technology development and use that otherwise would not have occurred. If it is thought that nothing would have happened in the foreseeable future without the government program, this is captured in the benefits stream for the without-government-program branch. The only benefits associated with the without-government-program branch are those attributable to the next-best technology available when the government program would have been completed.

2. The program might accelerate technology development and use. If the government program is viewed as causing the development and use of a technology that otherwise would be developed in the same way but at a slower pace, this effect is captured in the timing of the estimated stream of benefits underlying the benefit calculations at the end of the with-government-program branch as compared with the timing of the benefits stream at the end of the without-government-program branch. For example, if the with-government-program branch is expected to result in the technology being developed and adopted into use, say, 3 years sooner

than the without-government-program branch, the stream of benefits will be shifted forward by 3 years and the government’s contribution will be credited with the difference in the expected present value of the two benefits streams.

1. The program might improve a technology and make it more attractive to users. If the government program provides significant advances over the next-best technology, this can be captured in several ways. The advance may increase the probability of market acceptance. It may improve market penetration. It may generate larger unit benefits in use. Each of these effects can be captured in one of the branches of the decision tree.

2. The program might increase the probability of technical or market success. If the government program reduces the risk associated with achieving technical or market success, this is captured in the comparative probabilities assigned to the with- and without-government-program branches of the tree in the technical and/or market risk decision nodes. If the government role is to reduce technical risk, a higher probability of technical success assigned to the with-government-program branch will result in greater expected benefits projection for that branch and a larger expected benefit will be attributed to the government program, calculated as the difference between the expected values of the with- and without-government-program branches.

3. The program might enlarge the scope of the technology to make it more “enabling.” Enabling technologies are characterized as generating larger-than-average spillover benefits; that is, benefits that accrue broadly in society rather than more narrowly to direct private-sector investors in the research. If the government program enlarges the scope of the technology in ways that create a more enabling technology platform, the effect is captured in the benefits stream of the with-government-program branch of the decision tree, which includes a larger societal benefit than the withoutgovernment-program branch.

4. The program might increase collaborative and multidisciplinary research. The government program may promote collaboration among researchers by helping to overcome existing barriers to collaboration. Networks of collaborative R&D activity are increasingly seen as playing an important role in innovation. If the government program increases collaboration and if increased collaboration increases the likelihood of technical or market success, this effect is captured by assigning different probabilities of success to the with- and without-government-program branches. If, on the other hand, the change in collaborative effort increases the scope of the research, increasing its enabling characteristics, this effect is captured in the with- and without-government-program branches as differences in the estimates of societal benefits.

5. The program might produce a combination of effects. If the government program is expected to have more than one of the effects listed above, these may affect both the comparative probabilities assigned to technical and market success and the comparative projected benefit streams estimated for the with- and without-government-program branches of the decision tree. Multiple effects can easily be accommodated in the context of the decision tree analysis. As in the other cases, it remains necessary to take the difference between the with- and without-government-program branches to find the expected benefits of the government program.

In retrospective evaluations, one knows if a specific research program was successful in terms of meeting its technical (and commercial) objectives. In prospective evaluations, it is not known in advance whether a program will be successful. To calculate benefits one must therefore consider the likelihood that the program will be successful. While technical success can be viewed in relation to achievement of a single goal or outcome, a given research program might be judged successful across a range of outcomes, each with its own probability of technical success. As is discussed above, if the possibility of multiple outcomes is ignored, research benefits may be underestimated. Moreover, the use of a single “representative” outcome will usually be inaccurate and will mask assumptions about the relationships between the program outcomes and the alternative technologies.

DOE often identifies single goals for programs, and these goals may be stretch goals in the sense of being at the high end of possible outcome value but having a relatively low probability of technical success. Attaining less optimistic goals with delayed or only partial benefits may still have considerable value as well as a higher probability of technical success. Consideration of stretch goals alone is therefore also likely to lead to an underestimate of benefits. The number and nature of the outcomes to include will vary by project.

Probabilities of technical success (at any outcome level) are likely to depend on the level (and, in some cases, the quality) of resources—financial, technical, and managerial—expected over the life of the program, so that probabilities change with funding/resource level and the calculated expected net benefits are conditional on that level.

“Market risk” refers to the probability of a given technology being moved out of the lab and deployed into use.⁴ This risk is expressed as a probability for each case under evaluation and entered into a third node for each relevant branch of the decision tree. It is critical to the estimation of expected

benefits, as an applied R&D program yields benefits only if the technology is used.

Probabilities are applied to market penetration estimates. The probability that a technology will be deployed is distinct from the rate at which it is deployed—that is, its market penetration. It is necessary to estimate the rate at which the technology will come into use.

Many factors influence whether, and to which extent, a technology will be deployed into use. These factors do not apply to all programs, and it is recommended that the chief factors that influenced the probability estimates be identified in either a row at the top of the results matrix or in an appendix. These factors include the following:

• Market demand. To what extent does the technology meet a current demand? Does it enable users to solve an important problem or exploit a promising opportunity?

• Competition. What are the competing technologies that the target technology must overcome in order to be accepted in the market? Note that there are always competing technologies. Are others working on the same technology? Are others working on different technologies that might meet the same need? In the United States? Outside the United States? Who is performing the work?

• Window of opportunity. How long will the need for the technology exist? Are there factors that might eliminate or reduce the need for this technology? For example, are advances in a competing technology likely to outstrip those in the target technology, wiping out its advantages? When might this occur?

• Potential hazards. Are there potential environmental or safety concerns that might limit the use of the technology? For example, are there any by-products that might create an environmental or biological hazard, incurring mitigation costs or exposure to liability?

• Ease and cost of implementation. To what extent does adoption of the technology require changing existing systems or ways of doing things? For example, it would be easy to introduce a new catalyst into an existing reactor. On the other hand, a hydrogen-powered automobile would require a new fuel distribution system if the hydrogen is centrally produced, making implementation more difficult. To what extent will adoption of the technology require large capital investment? To what extent will adoption cause disruptions and downtime in current operations? To what extent will adoption require worker retraining?

• Resistance by special interests. To what extent will those adversely affected by the new technology lobby to retard its adoption, and how successful are they likely to be?

• New regulations. To what extent are new regulations likely to promote or impede the adoption of the technology? For example, unexpectedly stringent environmental regulations on diesel emissions made obsolete the fuel and engine research programs designed to meet the more modest objectives that had been anticipated. Conversely, adoption of appliance efficiency standards can lead to development of new technologies, as in the case of refrigerators (NRC, 2001, pp. 97-98).

• Complementary and prerequisite technologies. Is adoption of the new technology dependent on the availability of other technologies? Are these technologies still in the pipelines of other R&D programs? Will they be available in time to support the technology under evaluation?

Market risk factors are often critical to evaluating the potential of an R&D program. Indeed, for investments in fairly specific technologies, the risks associated with market acceptance may overwhelm those associated with technical success. Alternatively, fundamental programs that yield results applicable to a range of technologies and market conditions may be less susceptible to the market risks discussed here: They may be applicable in a wide range of regulatory regimes and may contribute to multiple technologies, at least some of which are likely to be available and of interest in the relevant time frame. Of course, long-term fundamental programs may be very risky in that the likelihood that all of their goals are achieved is remote.

Long-term, fundamental R&D requires further development and often further research before reaching a commercial outcome. Applied R&D such as the DOE programs considered in this report aims to develop technologies with specific performance and cost criteria that will be commercial in a time frame consistent with the schedule of the applied R&D project. It is this latter type of R&D having more immediate commercial applicability that the methodology has been designed to evaluate. The types of benefits evaluated in the methodology—for example, economic benefits—are consistent with the goals of technology development programs and the expectations of those making the investments. A further difference between fundamental and applied R&D is that the former has knowledge as its primary goal while the latter has knowledge as a by-product. Knowledge benefits may include unanticipated and not closely related technological spin-offs that are made possible by the research programs. Because the methodology proposed here is for applied R&D it has economic, environmental, and security benefits as its primary goal and does not give credit for knowledge generated in the course of a technology’s development.

All panels will report their results consistently using the benefits results template discussed in this Appendix (see Figure F-4). The format requires that each panel present similar information about the program under review and its findings so that users of the results can make informed program and funding decisions. The benefits results template consists of three sections. The first section lists important program information provided by the Department of Energy (DOE). The second section presents the panel’s results matrix of

the expected benefits of the program using the committee’s methodology applied to DOE’s expectation of program outcomes as well as the alternative outcomes anticipated by the panel. The third section of the template provides the panel’s findings and conclusions and expert opinions regarding the application of the methodology to the program.

In addition, panels must prepare a brief report—about 10 pages—to include background, summary of the DOE program, technical and market risk assessment, results and discussion, technical and market success, benefits estimation, role of DOE funding, decision tree, benefits calculations, results matrix, summary, and conclusions (including recommendations).

The first section of the benefits results template will provide summary information characterizing the DOE program. This information will include an identification of the program goals; funding, including outside funding; and the program’s critical interdependencies with other DOE program efforts. This information is provided directly by DOE.

The results matrix in the template is for presenting the panel’s views of the program’s technical and market risks and the expected benefits of the R&D program. Market risks reflect the panel’s expert opinion of the program’s market acceptance assuming that the goals have been met. Assessing the market risk includes consideration of price, infrastructure development and support, ease of use, competition from other technologies or innovations, time, and end-user preferences. In the committee’s decision tree framework, there is no single point estimate of technical or market risk, owing to the many possible outcomes. The quantitative estimates of probability are recorded in the decision tree, as discussed in Appendix F, whereas the results matrix should be annotated with a discussion of the key factors that contribute to the technical and market risk.

The benefits estimated by the panel represent an expected value benefit. This is calculated by adjusting the benefits that DOE assumes will be accrued if the program goal is met for the technical and market risks identified by the panel. Figure F-4 provides expected benefits for three scenarios to reflect a bounding of benefits for three possible future states.

The panel’s opinion and quantification and discussion of the risks are provided in the third section of the template in Figure F-4. The information required to be included in this section is listed in the template itself. The panel must complete the matrix using sufficient clarity and transparency to allow readers and potential users of the matrix to make informed and reasoned decisions about future goals, funding levels, and expected benefits. The template should be used by all panels to ensure consistency in reporting, use, and interpretation.