Propose a conceptual framework that captures the key features of prospective benefits evaluation.
Appoint expert panels to apply the framework to three DOE programs. The committee decided that three studies would be needed to test the methodology and selected the advanced lighting, fuel cells, and carbon sequestration programs for such study. Each panel was chaired by a committee member, but the panel members were mainly persons with expertise in the technology and markets pertaining to the programs. It was the judgment of the committee that this flexible approach would stimulate critical thinking about the methodological needs and practical problems of obtaining the relevant information.
Receive a report from each panel and use it as a basis for the committee’s recommendations.
Evaluate the experience reported by the panels and modify the methodology accordingly.
Present conclusions in the Phase One report, which will also contain recommendations for specific steps to be taken in Phase Two of the ongoing effort.
It is important to emphasize that Phase One has been focused tightly on the basic problem of adapting the retrospective methodology to a prospective context. As a result, some important issues had to be deferred to Phase Two. For example, the issue of how to characterize security benefits has not been explored. In addition, the committee cautions against using the three expert panel case studies to assess the benefits of the three programs, because the panels’ findings on the programs were developed for the express purpose of helping the committee to refine its methodology.
The methodology of the retrospective study rested on two principal concepts. One was the benefits matrix. This matrix proved to be useful to decision makers for assessing R&D benefits because it focused attention on the public good benefits—economic, environmental, and security—that are the objectives of DOE’s applied energy R&D programs. These benefits were captured in the rows of the matrix. Additionally, the matrix identified a number of possible outcomes of the R&D programs, ranging from successful deployment of a technology in private markets to the generation of knowledge that was useful but did not result in a successful technology.
The other key concept in the retrospective study was the “cookbook” (Appendix D in that study), which contained detailed instructions for how to calculate the benefits in each cell of the matrix. These instructions were written for analysts, but they proved to be of value to decision makers as well. The cookbook provided a consistent set of assumptions, concepts, and rules that all analysts should use. This consistency, in turn, allowed decision makers to confidently compare the benefits reported for different technologies.
In contrast to retrospective evaluation, however, prospective evaluation is complicated by uncertainty about how the future will unfold. In the committee’s view, the chief uncertainties are of three kinds:
Uncertainty about the technological outcome of a program. Research is inherently an uncertain process, and any evaluation of a research program must consider the likelihood that the program’s goals will be met. However, if a program’s goals are not fully met on time and on budget, the program may still produce important technological advances that have benefits that should be reflected in the evaluation.
Uncertainty about the market acceptance of a technology. It is possible for a research program to meet all of its own technical goals yet produce a technology that is not accepted in the marketplace and therefore has no economic benefit, because another technology has met the same need sooner, better, or more cheaply.
Uncertainty about future states of the world. The benefit of a new technology will often depend on developments quite unrelated to the technology itself. For example, the benefits of carbon sequestration technology will be greatest if carbon emissions are regulated, giving electricity producers incentives to deploy the technology.
These three categories of uncertainty will apply to all programs, but the relative impact of a particular category will vary from program to program.
The benefits framework that the committee proposed to the expert panels for prospective evaluation included a results matrix that summarizes possible outcomes and attendant investment risk for various categories of benefits.
Building on the experience gained by the expert panels, the committee recommends a methodology for prospective benefits evaluation having six main elements:
Rigorous definition of benefits, to be used consistently for all programs. One important addition to the techniques used for the retrospective study is discounting. Another is the use of “expected benefits” as the measure of a program’s value. An expected benefit is the expected value of net benefits and should be calculated as the probability-weighted average of benefits for all important outcomes analyzed in each scenario.
Common scenarios, used for all technologies, that describe future states of the world for which benefits are being estimated. The committee uses the three scenarios that are currently used by DOE’s Office of Fossil Energy in its own benefits analysis (NETL, 2004): (1) the Reference Case,