National Academies Press: OpenBook

Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) (2007)

Chapter: 3 Methodology for Prospective Evaluation of Deparment of Energy Programs

« Previous: 2 Results of Applying the Methodology
Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

3
Methodology for Prospective Evaluation of Department of Energy Programs

The Phase One study of prospective benefits of DOE’s applied R&D programs (NRC, 2005a) recommended a specific methodology for benefits calculation based on experience with applying a conceptual methodological approach to DOE’s fuel cell, carbon sequestration, and advanced solid-state lighting programs. The present study—the Phase Two study—tests the recommended methodology (see Appendix F for a summary description of the methodology from Phase One) on six programs in a consistent manner. One of its objectives is to refine the methodology and to assess its broader applicability. In addition, the committee examines procedures for estimating the monetary value of environmental and security benefits, a topic that had been deferred from the Phase One study. The results of the committee’s analyses of these topics are summarized below, beginning with the valuation of environmental and security benefits.

VALUATION OF ENVIRONMENTAL AND SECURITY BENEFITS

Assessing the prospective benefits of DOE’s R&D programs involves challenges common to many public programs but not usually present in private business assessments. Economic benefits—the linchpin of private investments—may be difficult to calculate, and for complex programs critical data are frequently lacking and uncertainty prevails about how successful a technology might turn out to be.1 Nevertheless, there is general agreement among economists about the principles for evaluating economic benefits. Markets that measure the economic value of related technology and (in some cases) futures markets that provide an evaluation of risk can guide an evaluation of economic benefits.

In contrast, public programs often have social benefits that are not valued by markets. Assessing the value of such benefits is inherently difficult: It involves ambiguity and, even as an academic matter, a range of possible answers. For the DOE programs, two broad classes of benefits have this characteristic: the environmental consequences of energy technology and the security implications of energy savings or energy alternatives. These program attributes are in general critical components of the benefits package—indeed, if a program can be justified simply on an economic basis, there may be no rationale for government participation.

This section reviews current practice in evaluating environmental and security benefits and considers ways in which it can be adapted to assessing DOE benefits.

Valuing Air and Water Pollution Emissions

For goods and services purchased in a competitive market (a market in which both buyers and sellers are “price takers” that have no long-term influence on price), the market price represents the economy’s best valuation of an additional unit of that good or service. Determining a social value for a good or service is difficult in the absence of a competitive market. Economists have developed a number of approaches to help estimate the social values of goods and services not valued in the market. The advantages and disadvantages of the most relevant approaches for valuing environmental benefits are explored in the following sections. This discussion focuses on the benefit of reducing air pollution, which is both the primary environmental benefit identified in the Phase Two study and the type of benefit for which, as explained below,

1

Economic net benefits are based on changes in the total market value of goods and services that can be produced in the U.S. economy under normal conditions, where “normal” refers to conditions absent energy disruptions or other energy shocks. Economic value can be increased either because a new technology reduces the cost of producing a given output or because the technology allows additional valuable outputs to be produced by the economy. Economic benefits are characterized by changes in the valuations based on market prices. This estimation must be computed on the basis of comparison with the next-best alternative, not some standard or average value (NRC, 2005a, p. 16).

Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

value estimates are most advanced.2 The committee advocates applying valuations to emissions of air pollutants but not to other types of pollutant emissions.

Damage Function Models

The Phase Two methodology results in an estimate of the quantitative change in the level of air pollutants resulting from the technology being studied. Turning this change in emission levels into an economic value requires two additional steps. The first step in evaluating the benefits of pollution abatement is to estimate the impact of pollutants on things society values, e.g., health, visibility, outdoor recreation, quality and availability of raw materials, ecosystem services, and other measures of environmental quality. Damage functions must be estimated for the most important effects of environmental pollution. In particular, the impact on human health of air pollution includes days lost due to restricted activity, increases in the incidence of asthma and bronchitis, and even premature deaths. The second step is to place a monetary value on the damages (or reductions in damage) to health, visibility, outdoor recreation, materials deterioration, and the natural environment. Both steps present significant problems.

In published studies, the various physical effects are modeled to determine the quantitative effects of changes in ambient levels of pollution. For example, how does the incidence of asthma or bronchitis change as air pollution levels change? Despite decades of damage research, however, both the qualitative and quantitative effects of such pollution are uncertain. One reason for this uncertainty is that our understanding of how the damage is incurred improves over time. For example, the Environmental Protection Agency’s (EPA’s) benefit-cost analysis of the fine particulate matter (PM2.5) and ozone standards for 1970-1990 ascribed all premature mortality associated with air pollution to PM2.5, assuming that the contribution of other pollutants was negligible (EPA, 1997a). A further uncertainty surrounds the extent of life shortening associated with a premature death due to PM2.5. In addition, estimating damage functions is complicated by interactions with other factors and the changing nature of air pollution across areas and over time. The spatial and temporal variations greatly complicate the task of predicting environmental impacts in a generic manner suitable for application in a prospective planning study. Most of the existing damage functions for water pollution apply to specific sites; there is only a limited ability to produce reliable generic damage functions for waterborne pollutants. Hence, where generic regional or national damage functions have been developed, these have typically been for air rather than water pollution.3 In addition, water availability is a growing issue in many parts of the country. This can often be linked to energy resource recovery, production, and use or to the production of hazardous wastes as part of energy systems’ life cycles. Other impacts are more difficult to quantify and include those related to land use, ecosystem impacts, and aesthetics. This complexity and regional specificity hamper efforts to monetize environmental benefits related to water.

Willingness-to-Pay Studies

Once the type and the extent of physical damage due to increased air pollution are known, the next step is to place a monetary value on this damage. However, there is no direct market valuation of a reduced incidence of asthma attacks or of being able to see a landmark from 30 miles away rather than only 10 miles away, because there are no markets for asthma attacks or visibility.

To appraise things like this, economists employ techniques from the field known as nonmarket valuation. Nonmarket valuation seeks to measure in monetary terms the value that people place on items they care about, regardless of whether the item is a conventional marketed commodity (e.g., a loaf of bread, a new car) or something that the person cares about but that cannot be purchased in a market (e.g., a beautiful view at sunset, a pristine wilderness, a historic monument, an excellent public school system, or a healthy body). Conceptually, these nonmarket items are measured in monetary terms by considering the change in income that is equivalent to them, in terms of its impact on the individual’s well-being. Thus, while the items themselves are not monetary in nature and they cannot be obtained by the individual through the

2

The valuation of pollutants in environmental media other than air is not discussed here. Conceptually, such valuations would be similar to that of air pollutants. In practice, the damage functions apply to specific sites, and data are difficult to elicit.

3

There are two major contexts in which generic damage functions have been developed, oriented mainly to airborne emissions. One is “environmental costing” by public utility commissions (PUCs). The other is the EPA’s retrospective assessment of the effects of the Clean Air Act on the “public health, economy and environment of the United States,” mandated by Section 812 of the Clean Air Act Amendments of 1990. EPA’s assessment, The Benefits and Costs of the Clean Air Act, 1970 to 1990, was published in October 1997. In addition, prior to the adoption of deregulation in the late 1990s, the PUCs in 29 states had adopted or were considering adopting some form of environmental costing for the purpose of comparing electricity generation alternatives in the context of utility planning and regulation. In most cases, this took the form of an approved schedule (or spreadsheet model) of “adders” designed to keep score of the environmental costs associated with different methods of electricity generation under the specific conditions applicable to that state. These adders were not actually used in setting prices nor were they charged to electricity users, but they were used in identifying the least-cost source of electricity generation, based on a consideration of the total social cost, including environmental externalities, rather than just the private cost to the particular utility. The model used by New York State and the issues generally involved in environmental costing are described and discussed in papers presented to a symposium on environmental costing, edited by Shogren and Smulders and published in Resource and Energy Economics 18(4) (December 1996).

Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

expenditure of his own funds, their monetary value to the individual is represented by the amount of money that could be exchanged for them while leaving the individual equally well off before and after the exchange. In other words, the economic value of an item can be expressed as the amount of money that an individual would be willing to exchange for that item if an exchange were possible.

Accordingly, economists need to find a trade-off to measure economic value. Either they find a relevant trade-off that occurs naturally (revealed preference) or they create one through a survey or an experiment (stated preference). As an example of the former, economists might examine people’s decisions to spend money on various items to protect themselves against the risk of some particular illness: These people are presumably making a trade-off between spending the money and avoiding the illness. Another example of revealed preference is the statistical analysis of how wages vary among occupations and, in particular, of the extent to which higher wages are offered for more risky occupations. The estimated risk differential in wages is used as a measure of how willing people are to trade off more money for a greater risk of death. As an example of stated preference a government that is thinking of introducing a new program (one that will save people’s lives, say, or reduce air pollution) surveys people to learn how much they would be willing to pay for the program. Respondents might be told that, if the program is introduced, a household like theirs would have to pay x dollars per year in additional taxes and might then be asked if they would vote for or against it. The tax is varied across different respondents, and the responses are used to trace out a demand curve showing the percent of people who would vote for the program at each different dollar amount. The median of the curve (i.e., the dollar amount that 50 percent of the population would be willing to pay) can be used as a measure of the value placed on the item by the population surveyed.

Both approaches—stated and revealed preference—to the estimation of economic value can have problems in practice. The specific choice being used to exemplify the trade-off in a revealed preference study might not adequately discern what the researcher has in mind, or it might be difficult in practice to disentangle the trade-off of interest from other factors involved in the subjects’ decision making. Or, respondents might not find the trade-off credible, or they might think they will not actually have to pay the higher taxes.4 All of these are challenges that researchers have to deal with.

While uncertainties surround both the damage functions and the economic valuations used to monetize the consequences of energy technologies, the general experience has been that the uncertainties associated with the damage functions are even larger than those associated with the economic valuations.

Just as the physical damage caused by a particular discharge can vary greatly depending on the location and timing of the discharge, with the variability being much greater for waterborne than airborne emissions, so, too, can the economic value that people assign to that particular physical consequence vary greatly depending on the location and timing of the damage. In part, this is due to the spatial (and sometimes temporal) variation in the availability of substitutes for the good or service suffering damage: Other things being equal, the more substitutes that are available, the smaller the associated economic impact. In addition, people’s preferences may vary spatially or temporally as a function of differences in norms and expectations. There also tends to be a difference on the economic valuation side of impact assessment with respect to air pollution versus water pollution. Typically, the main impacts of air pollution are on human health, whether morbidity or mortality; impacts on amenities and recreation, materials, and ecosystems tend to play a smaller role.5 With water pollution, by contrast, the major consequences tend to be recreation and ecosystem impacts, with human health playing a lesser role. The economic values associated with recreation and ecosystem impacts tend to vary spatially and temporally and generally do not lend themselves well to generic assessment. By contrast, the economic values associated with human health impacts lend themselves better to generic assessment. Moreover, for its assessments of the human health consequences of pollution, EPA has developed over the years generalized or consensus economic valuation of a “statistical life,” and of morbidity impacts.6 Such estimates are routinely used by the EPA in a generic manner.7

Because of the relative paucity of generic damage functions and generic economic valuations associated with water pollution, as compared to air pollution, the committee recommends that the assessment of environmental impacts

4

Whether or not people respond truthfully to the hypothetical trade-off in a stated preference survey is a matter of controversy. It depends in part on the skill with which the survey has been constructed.

5

For example, in the retrospective assessment of the Clean Air Act, the EPA determined that human health impacts (specifically, the avoidance of premature mortality) accounted for more than 90 percent of the calculated economic benefit (EPA, 1997a and 1997b).

6

The value of a statistical life is the value associated with a change of one in the expected total number of annual deaths from some cause; it is a statistical probability—the particular person whose life is affected is not known and will never be individually identified. For example, a statistical life could be the one more random traffic death in addition to the 42,000 killed on the highways each year.

7

The EPA’s Retrospective Analysis of the Clean Air Act used 26 individual willingness-to-pay studies as the basis of its distribution on health effects from premature mortality (EPA, 1997b). Of these 26 studies, five were based on contingent valuation methods that sought the willingness-to-pay of individuals to avoid the risk of premature death. The remaining studies were economic analyses that estimated the additional wages paid to workers for accepting increases in risk of premature death. The EPA used the median estimate of these studies, $4.8 million per life, in its benefits calculations.

Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

associated with DOE’s applied R&D programs focus on air pollution consequences and human health impacts.

A Statistical Look at Existing Damage Valuation Studies

To illustrate the effects of the foregoing uncertainties, Table 3-1 provides a sample of the existing economic valuation literature on the damage associated with air pollution. It shows estimates of the damage cost for conventional pollutant and greenhouse gas emissions in constant 1992 dollars per ton ($1992/t) for these releases. Note that the existing body of literature presents a wide range of estimates of the damage resulting from an additional ton of pollution. Uncertainty is evident: The maximum estimates are 6 to 1,000 times greater than the minimum estimates. (The above paragraph is adapted from Matthews and Lave (2000).)

It is important to note that these estimates of the damage costs are not the same thing as the cost of reducing emissions of the pollutant in question—that is, the abatement cost. The marginal abatement cost of an emission reduction can be estimated with much greater confidence than damage costs, especially for pollutants for which a market in abatement allowances exists.

TABLE 3-1 Estimates of the Social Damage Costs of Air Emissions

Species

No. of Studies

Cost of Environmental Externality (1992 $/t of air emissions)

Minimum

Median

Mean

Maximum

Carbon monoxide (CO)

2

1

520

520

1,050

Nitrogen oxides (NOx)

9

220

1,060

2,800

9,500

Sulfur dioxide (SO2)

10

770

1,800

2,000

4,700

Particulate matter (PM10)

12

950

2,800

4,300

16,200

Volatile organic compounds (VOCs)

5

160

1,400

1,600

4,400

Global warming potential (in CO2 equivalents)

4

2

14

13

23

NOTE: CO2 equivalent is the amount of CO2 that would cause the same amount of radiative forcing as a given amount of another greenhouse gas. When used with concentrations, it refers to the instantaneous radiative forcing caused by the greenhouse gas or the equivalent amount of CO2. When used with emissions, it refers to the time-integrated radiative forcing over a specified time horizon caused by the change in concentration produced by the emissions.

SOURCE: Cifuentes and Lave (1993), CEC (1993), Desvousges et al. (1994), Fankhauser (1994), Koomey (1990), Ottinger (1992), OTA (1994), and Zuckerman et al. (1995).

For example, the 1990 Clean Air Act Amendments sets up a plan for buying and selling SO2 and NOx allowances. Each allowance allows the owner to put one ton of that pollutant into the air. The Act sharply reduced the allowable emissions of these two pollutants over time. Plants that are able to reduce more than the required amount are allowed to sell their excess allowances. Plants that are not able to meet the required reductions are forced to buy allowances.

The prices of these allowances reflect the marginal cost of abatement rather than the social value put on the pollution abated. Such prices depend on the stringency of abatement, the current technology for abatement, and the ability to shift the allowance to the future.

The price of an SO2 allowance has been as low as $70 per ton (/t) in 1996 and is currently around $850/t (EPA, 2006). EPA projects that allowance prices will rise to $1,200/t by 2020. The NOx abatement cost is somewhat more complex because it depends on location—areas like Los Angeles have stringent NOx curtailment, for example, and other areas have much less stringent abatement. Los Angeles NOx allowance prices have been as low as $200/t in 1998 and are currently selling at a high of around $7,700/t.

Ideally, public policy would constrain the emissions of pollutants until the marginal cost of abatement was equal to the marginal benefit (i.e., the decrease in total damage cost). If so, the allowance price, an estimate of the marginal cost of abatement, should approximate the social value of abating these pollutants. To test this ideal against the real world, Table 3-2 compares data on damage costs and abatement costs. It includes the median damage costs from Table 3-1 and an additional damage cost estimate from a study of European markets (Banzhaf et al., 2002). Although the damage costs differ considerably across studies, all are within the range that Matthews and Lave (2000) identified from previous studies. Of interest is the disparity between allowance prices and damage costs. The allowance price for both NOx and CO2 seems, respectively, to be near or above the maximum

TABLE 3-2 Estimates of Pollution Abatement Costs

Species

Median Social Damage Cost, Taken from Table 3-1

Price of an Allowance in 2006

Estimated Abatement Cost from Banzhaf et al.

Carbon monoxide (CO)

520

 

 

Nitrogen oxides (NOx)

1,060

7,700

1,160

Sulfur dioxide (SO2)

1,800

850

3,850

Particulate matter (PM10)

2,800

 

2,060

Volatile organic compounds (VOCs)

1,400

 

 

Global warming potential (in CO2 equivalents)

14

30

8

Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

social cost estimates. In contrast, the allowance price for SO2 is closer to the minimum abatement cost.


Finding: Although there are a host of land, water, and perhaps public health impacts to consider, the benefits, in monetary units, of reducing criteria air pollutants is both the primary environmental benefit considered in the present study and the type of benefit for which valuation methods are most advanced.


Recommendation 1: Panels should apply valuations in monetary units to criteria air pollutant emissions in the results matrix, but not to other types of pollutant emissions. The valuations used should be the allowance price forecasts for the future period.

Valuing Energy Security Benefits
Security Benefits Related to Electricity Supply

Security considerations affect the electricity supply in two ways. One security concern relates to the type of fuel used to produce the electricity. The United States relied on imported petroleum for much of its electricity generation in 1973 at the time of the boycott by the Organization of Petroleum Exporting Countries (OPEC), but less than 2 percent of electricity is generated from petroleum at present. Now, almost 20 percent of electricity is generated with natural gas. Increasing the importation of natural gas in the form of liquefied natural gas (LNG) from nations in the Middle East can lead to a concomitant increase in the risk of terrorists or nations trying to influence our policies. The next section includes a discussion of the valuation issues related to oil and gas dependence.

The second security concern is that the electricity generation system at present provides a tempting target for terrorists. The resulting blackout could affect tens of millions of people and lead to billions of dollars of costs. For example, on August 14, 2003, 50 million people, from Cleveland to Toronto to New York City, lost their electrical service (U.S.-Canada Power Systems Outage Task Force, 2004). The immediate effect was gridlock in the cities, since the traffic signals stopped working, and danger and inconvenience as elevators and trains (including subways) stopped operating. The disruptions are estimated to have cost $4 billion to $10 billion (U.S.-Canada Power Systems Outage Task Force, 2004) and would have been much more expensive except that many of the customers who would have suffered the highest cost from the power failure had backup generators.8

One way to reduce blackouts is by having distributed generation, particularly where the generators are small and located at the customer’s site. Distributed generators can be internal combustion engines (diesel- or natural-gas-fueled), microturbines run on natural gas, or renewable energy systems using biomass, wind, or solar energy. Zerriffi et al. (2002) estimate that having distributed generation (DG) in the system could improve reliability 10-fold, at little additional cost. In some applications, distributed generators can also have economic advantages. They can provide power for peak demand, saving considerable costs. Alternatively, they can be used for combined heat and power (CHP). If rejected heat from the power cycle is utilized as process heat, the energy utilization factor of the CHP system could be close to 100 percent, compared with 30 to 50 percent for a central station generator. In some cases, however, these generators would replace large central station generators powered principally by coal or nuclear fuel. If so, increased reliability and one type of security would be gained by use of imported fuels for DG machines, but another type of security would be decreased.

Thus, there is a complicated relationship between security and the electricity supply system. Large blackouts are relatively frequent and costly.9 The system is an easy target for terrorists. If terrorists could mount a compound attack, they could paralyze a city by taking out the electricity supply and then attack high-value targets knowing that police, fire, medical services, and security services would find it hard to respond. Adding DG to the current electricity system could increase reliability and security, removing the electricity system as a terrorist target, and could also increase the efficiency with which fuels are used. However, an increase in oil- or natural-gas-powered DG would require more imports of oil and natural gas, increasing the other kind of energy security threat.


Finding: While the complex relationship between the electricity supply and security is becoming clearer, analysts are a long way from having methods for valuing reductions in security threats contributed by technologies such as distributed generation.


Recommendation 2: Panels conducting prospective benefits assessments should describe reductions in threats to energy security related to electricity supply as physical quantities of oil and gas.

Security Benefits Related to Oil and Gas Consumption

In addition to environmental externalities, increases in

8

Rose et al. (2005) investigated the extent to which the initial losses can be made up later, such as by rescheduling production. They estimate that close to 80 percent of the initial losses could be recovered by rescheduling production, shopping another time, or other substitute actions. Thus, the true economic cost of a disruption is likely to be different from the cost estimated without considering these mitigating factors.

9

Analysis of North American Electric Reliability (NERC) Distribution Analysis Working Group (DAWG) database indicates an average of three blackouts per year that were greater than 1,000 MW and one or two blackouts per year that affected 1 million or more people. See Hines et al. (2006) for an analysis of large blackouts in the United States.

Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

U.S. oil and gas consumption and imports may impose incremental costs on the United States that are not reflected fully in the market price. Therefore, individual households and businesses may choose to consume more oil or gas than is optimal from the U.S. perspective, and reductions in consumption could increase national welfare. Such incremental costs might include (1) increases in the price of imported oil; (2) macroeconomic disruption externalities associated with price volatility; (3) adverse consequences for U.S. foreign relations; and (4) increased military expenditures to secure energy supplies. These costs have been estimated for oil consumption and are reported below. In principle, similar estimates can be made for natural gas, but the committee is unaware of such research having been done.

Collectively, these cost components are sometimes called the “oil premium.” A comprehensive review by Leiby et al. (1997) suggested a preferred range for the oil premium at $0-$5/bbl; however, under broader assumptions their range extends to $10/bbl. Another recent NRC committee assumed an oil premium of $5/bbl in its examples, with a range of $1-$10/bbl (NRC, 2002a). This somewhat broader range mainly reflects the updating of Leiby et al. (1997) for higher baseline oil prices. Parry and Darmstadter (2003) put their best assessment of the oil premium at about $5 per barrel and cite a range in the literature of $0-$14/bbl. However, those studies focused on the first two cost components listed above. And, they were completed when there was significantly more excess capacity and lower prices in the oil markets than is the case now. But these two components of the oil premium can be expected to depend heavily on world oil market conditions and to vary sharply over time as these conditions change.

Each of the factors has been discussed qualitatively by the recent report of an independent task force sponsored by the Council on Foreign Relations, National Security Consequences of United States Oil Dependency (2006).The following paragraphs discuss each of these components of the oil premium in turn.10


Impacts of U.S. Demand on World Petroleum Prices. Reductions in U.S. petroleum consumption may have the effect of lowering world prices, with U.S. consumers paying less to oil producers, and increases in U.S. consumption may increase world prices. From the point of view of U.S. economic welfare, any reduction in payments to foreign oil producers can be considered a terms-of-trade benefit.11 Previous studies have valued this benefit at $1-$5/bbl (Leiby et al., 1997; Parry and Darmstadter, 2003), with the range depending primarily on assumptions about the import supply elasticity and the price of oil. Such estimates depend crucially on the elasticity of supply and demand in the world oil market. When there is very little extra capacity, as can be expected over the foreseeable future, shifts in the worldwide supply and demand for oil can have significant impacts on the world oil price. Because world oil markets clear, supply changes and demand changes must be equal, with prices adjusting to preserve the equality.

Assume that in the short run, the elasticity of demand12 for oil on the world market is between −00.1 and −00.2. Thus, when supply cannot adjust—that is, the elasticity of supply is zero—a 1 percent increase in world oil consumption (0.86 million barrels per day (mmbpd)) can be expected to increase world oil price by between 5 percent and 10 percent ($3.50-$7.00/bbl, at current prices.) A decrease in world oil consumption can be expected to decrease world oil price correspondingly. Assume that the United States is importing 11 mmbpd, or 4.015 billion barrels per year, and paying $70/bbl, the annual cost would be $281 billion per year. If the United States increased its imports to 11.86 mmbpd (annual imports of 4.329 billion barrels) and the oil price increased to $73.50/bbl, the annual cost would increase to $318 billion, up by $37 billion. The total annual cost to the United States would increase by $118 per additional barrel of oil imported during the year, an amount far higher than the market price of $70. The difference between the additional cost to the United States and the market price—$48/bbl—is a term-of-trade cost to the United States and thus, from its perspective, a financial externality. If the oil price were to increase by $7/bbl, the annual cost would increase to $333 billion, up by $52 billion. The total annual cost to the United States would increase by $167 per additional barrel of oil imported during the year; the financial externality would be $97/bbl.

When there is much unused oil production capacity and the OPEC nations are operating so as to keep oil price at a target level, increases in oil demand will have virtually no effect on world oil price; equivalently, the elasticity of supply of oil is very high. In that case this financial externality is near zero. To illustrate, assume that in a time of significant unused oil production capacity, the elasticity of oil supply is 10—that is, an increase in oil consumption of 1 percent (0.86 mmbpd) increases oil price by 0.1 percent. Then such an increase in oil consumption would increase price to $70.07/bbl and import costs would increase to $303 billion, a total increase of $22 billion. The total annual cost to the United States would increase by $71 per additional barrel of

10

Greene and Ahmad (2005) treat the economic costs “as arising from the use of monopoly power in the world oil market,” and distinguish their approach from those treating the oil premium as an externality, such as the approach of Parry and Darmstadter (2003).

11

Terms-of-trade issues exist in markets other than the oil market. However, for most nonagricultural goods, nations have agreed through the World Trade Organization (WTO) and previous trade agreements to operate competitively. But in oil markets, the OPEC has exercised its ability to keep prices higher than competitive levels and such agreements do not operate.

12

The elasticity of demand measures the ratio of percentage change in demand to percentage change in price, all else equal. Thus for a 10 percent increase in price, an elasticity of demand of −00.1 implies that deman would decrease by 1 percent and an elasticity of demand of −00.2 implies that demand would decrease by 2 percent.

Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

oil imported during the year; the financial externality would be only $1/bbl.

Thus, for those times during which the elasticity of supply of oil is zero, the financial externality could be between $48/bbl and $97/bbl, while for those times during which the elasticity of supply of oil is as high as 10, the financial externality would be as low as $1/bbl.


Oil Supply Disruptions.13 Wages and prices for many goods are “sticky” and do not adjust seamlessly to changes in input prices or macroeconomic variables. In particular, spikes in oil prices increase input costs and reduce the marginal productivity of labor. In a labor market with sticky wages, such price increases could lead to increased unemployment and reductions in gross domestic product (GDP). Oil price volatility also adversely affects owners of vehicles and other fixed capital that operates on oil, who are limited in their ability to adjust oil consumption when prices rise or fall. Volatility imposes an extra cost on these consumers. These costs are reduced if the economy is less dependent on oil. Estimates of the value of these disruption externalities range from $0 to $8 per barrel (Leiby et al., 1997; Parry and Darmstadter, 2003). As a point estimate, the National Highway Traffic Safety Administration (NHTSA) (2006) uses $2/bbl. See Jones and Leiby (1996) and Jones et al. (2004) for reviews of the impact of oil price volatility on the macroeconomy.

These estimates can be expected to be considerably higher when there is very little unused capacity in world oil markets. When there is a large amount of unused capacity, disruptions in oil supply from one part of the world can be met be compensating increases in supply from other regions—that is, the short-run elasticity of supply can be expected to be very high. In those circumstances disruptions in supply from one region lead to relatively small increases in world oil price and few macroeconomic dislocations. However, when there is very little unused capacity, disruptions in oil supply cannot be met by compensatory increases in supply from other regions; the short-run elasticity of supply can be expected to be very small. In those circumstances disruptions in supply lead to large increases in world oil price and very costly macroeconomic dislocations.


Increased Oil Imports and U.S. Foreign Relations. In the current tight world oil market situation, increases in oil consumption and thus of oil imports limit foreign policy options (for example, in our relationships with Saudi Arabia) and may be putting the United States in a position of competing with China and other oil-importing countries. Additional imports increase the revenues flowing to oil-exporting nations whose political agendas conflict with those of the United States (for example, Venezuela) and give those nations increased ability to use their oil revenues to press their strategic advantage over the United States. Additional U.S. imports tighten the world oil market; a tight world oil market discourages other importing nations from taking strong stands against the actions of major oil exporters (for example, Iran’s nuclear program).

Although this factor has been discussed qualitatively in the Council on Foreign Relations’ independent task force report National Security Consequences of United States Oil Dependency (2006), the report did not provide quantitative estimates of these externalities. The committee is not aware of any study that attempts to quantify the impact of increased oil imports on U.S. foreign relations.


Military Expenditures Related to Securing Energy Supplies. These costs of U.S. oil consumption are not reflected in the price of oil. However, it is not clear that these costs would vary significantly with changes in U.S. oil consumption. For example, military activities in world regions that are vital suppliers of oil undoubtedly serve a wide range of security and foreign policy objectives so that most analysts estimating the benefits of marginal changes in oil consumption and imports have not included these costs in their estimates, either because they believe that the military expenditures would not change significantly with changes in oil imports or because they have no method of rationally quantifying the externality.


Summary. Reasonable estimates of some of the externalities related to oil consumption exist and could be used to assign an economic value to reducing oil consumption. However, the estimates will be highly dependent on time and on expected future oil market conditions. Moreover, the committee is at this time not aware of any good quantification of some of the externalities, so they must be discussed qualitatively unless and until reasonable quantifications are developed.


Finding: Increases in U.S. oil and gas consumption and imports may impose incremental costs that are not fully reflected in the market price. Some of the cost components of this oil premium have been estimated in various studies. In principle, similar estimates could be made for natural gas, but the committee is unaware of such research having been done.


Recommendation 3: Panels should describe energy security benefits related to reduced oil and natural gas consumption quantitatively in the benefits matrix as physical quantities of oil and gas. The time pattern of the oil consumption impacts

13

Macroeconomic externalities cannot be measured by simply looking at the change in prices, costs, or quantities of energy. The committee includes them in a separate category from the economic benefits in part to reflect the difficulty of measurement and the substantial uncertainty that surrounds their occurence, severity, and assessment. Of course the macroeconomic externalities—like the other security and environmental benefits—have real implications for the economic health of the country.

Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

should be made explicit, along with an assessment of the probable state of the oil market during those future times.

METHODOLOGICAL CONSIDERATIONS IN MEASURING BENEFITS

The Phase One prospective benefits study discussed and demonstrated principles for calculating the economic benefits of DOE programs and proposed a decision tree methodology for benefits estimation based on these principles. (At the start of their work, the six panels all received the methodology as it stood at the conclusion of Phase One. See Appendix F.) As noted at the outset of this chapter, one objective of this Phase Two study was to refine the methodology and assess its utility.

As detailed in Appendix F, the general structure of prospective benefit assessment methodology involves (1) characterizing relevant states of the world according to prices, regulations, and other constraints that influence the benefits of the technology (i.e., the scenarios); (2) considering plausible technological outcomes of the program and assigning probabilities to each outcome; and (3) evaluating the extent to which the technology is deployed under a given scenario as well as probable competition from alternative technologies that may have also been developed over the project period. Each plausible technological outcome, together with market and policy conditions, yields economic savings and environmental and security benefits. Estimating the net benefit of DOE’s programs requires two additional steps. The first is to work out the difference due to the DOE program—that is, to compare the outcome with the DOE program with the outcome had there been no such program The second step is to estimate the likelihood of each potential technological and market outcome and then weight the benefit of each outcome by this likelihood. This process is summarized in the decision tree analysis associated with each of the panel studies.

In the following sections the committee draws conclusions about the methodology based on the experience of applying it to six DOE applied research programs. Where appropriate, it recommends refinements to the methodology and offers guidance for its implementation in the future. The committee’s conclusions and recommendations are based on its review of the experiences of the six panels, which are summarized in the panel reports. The conclusions and recommendations of the committee fall in five areas:

  • Decision tree analysis. The committee endorses the decision tree framework for use in estimating the benefits of DOE’s applied research programs. However, panels must take care, and they will require guidance from a decision analyst (the consultant) to understand how to assign probabilities and isolate the government impact.

  • Global scenarios. The scenarios developed by the committee for all panels proved to be a valuable tool for characterizing and quantifying the benefits of the DOE R&D programs. However, the Phase Two experience shows that panels sometimes needed to have the scenarios clarified for them to be able to address issues that were important to the specific R&D program.

  • The National Energy Modeling System (NEMS). This model is important for providing baseline energy prices and demands, but using it to estimate the prices of and demands of all different program outcomes is unlikely to yield refinements to the estimated benefits.

  • Modeling alternative technologies. The success or failure of competing or complementary technologies can significantly affect the value of a DOE applied research project. Although Phase Two clarified this issue and provided some methodological guidance for dealing with it, more work is required to describe a method for estimating the benefits in DOE’s overall portfolio.

  • Implementation issues. The panels were generally successful in implementing the committee’s methodology, but the Phase Two experience turned up three process issues that should be considered in future studies.

Decision Tree Analysis

The primary aims of DOE’s programs are these: (1) to reduce technical risk, (2) to reduce market risk, and (3) to accelerate the introduction of the technology into the marketplace. To understand how a government program can yield net benefits and to estimate benefits under different scenarios, the committee involved in Phase One of this project developed a decision tree methodology that integrates the assessments of experts and yields estimates of market risks, technical risks, and project outcomes under the three global scenarios.

Constructing a decision tree that captures a minimal yet representative set of outcomes for a DOE research program is the most important part of the assessment exercise, as well as a very challenging task. It requires knowledge of markets, the technology status, and program alternatives. The benefits estimated using the recommended methodology thus depend crucially on a panel’s ability to (1) construct a decision tree that captures the key technical and market risks of an applied research program, (2) make reasonable estimates of those risks, (3) assess the timing of technology development,14 and (4) assess the differential success of the government research program relative to that of non-DOE efforts in achieving the stated goals of the program. The panels all succeeded in developing an appropriate decision tree for their DOE

14

The 5-year rule had been recommended by the NRC Committee on Benefits of DOE R&D on Energy Efficiency and Fossil Energy R&D (see NRC, 2001) for use in the absence of better information. Because the experience with the panels convinced the committee that the 5-year rule is often inadequate for prospective evaluation, it recommends a more elaborate methodology in this report.

Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

programs. For this reason, the committee continues to recommend the use of the decision tree framework. However, the construction of the tree is a difficult process, so the committee also continues to recommend, as detailed in Chapter 4, the use of a trained decision analysis consultant to facilitate this process.

The panels were also able to specify the probabilities required to complete the tree, but again the task was challenging. The committee is particularly concerned about the experts’ ability to estimate very low probabilities. For example, experts may not be able to distinguish between .1 and .01 when assigning probabilities to technical and market success, even though a tenfold difference can greatly influence the calculated values. Alternatively, it is possible that a very small probability—say .1 or .05—overestimate the still lower probability an expert believes in but finds difficult to justify or defend. These issues also point to the need for a skilled facilitator who can help panelists appreciate the impact of these differences and/or perhaps decompose the overall assessment into a series of smaller and easier assessments. For example, rather than directly assessing a very small probability of success, the assessment can perhaps be decomposed into assessments of success for several technical hurdles required for success; none of these hurdles may on its own have a very low probability, yet the overall probability of success (given by the product of these probabilities) may be quite low. In cases with very low probabilities of very large benefits, it is important for the discussion of the risks in the matrix to make this clear.

Another challenge that arose in some panels with respect to decision trees was isolating the effect of governmentfunded research, particularly when the government program is small in relation to private sector activity aimed at the same goal. The decision tree methodology proposed in Phase One and Phase Two by the committee calls for estimating this benefit as the difference between benefit of the R&D with government support and the benefit without public support. In this case, both benefits estimates are very large and uncertain numbers, and the difference is relatively small and difficult to estimate with much confidence. Consideration should therefore be given to modifying the methodology or to using alternative approaches. In addition, the committee encourages panels to describe explicitly the role of the government effort and its expected benefits when there is already a relatively large amount of non-DOE research.

Global Scenarios and Their Implementation

As did its predecessor, the Phase Two committee defined three global scenarios for each panel to use: a Reference Case scenario, a High Oil and Gas Prices scenario, and a Carbon Constrained scenario:

  • Reference Case. The Reference Case is consistent with the Energy Information Administration’s (EIA’s) reference case in its Annual Energy Outlook 2005 (EIA, 2005b). World oil prices increase from $24.10/bbl in 2003 to about $30/bbl in 2025. Natural gas consumption increases significantly—i.e., from 22 trillion cubic feet (Tcf) in 2003 to 31 Tcf in 2025—with wellhead prices decreasing from $4.98 in 2003 to $3.64 per thousand cubic feet (Mcf) in 2010, then increasing to $4.79 per Mcf in 2025.15 There is assumed to be an increase in primary energy consumption from 98.22 quadrillion British thermal units (quads) in 2003 to 133.8 quads in 2025. GDP is expected to grow 3.1 percent per year through 2025. U.S. carbon dioxide emissions from energy consumption are assumed to grow, from 5,788 million metric tons in 2003 to 8,062 million metric tons in 2025.

  • High Oil and Gas Prices. The High Oil and Gas Prices scenario assumes that oil prices will remain very high throughout the period and that constraints on natural gas supply lead to higher natural gas prices and higher electricity prices. The committee doubled the prices forecast in the EIA (2005b) High Price A scenario to arrive at its own set of prices it felt to be more likely given the recent upsurge in prices. For example, the oil price in 2010 in this scenario is $67.98 (including the committee’s doubling) versus $25 in the Reference Case, and the natural gas price in 2010 is $7.34 per Mcf versus $3.64 per Mcf in the Reference Case. In the period after 2025, the real price remains constant.

  • Carbon Constrained. The Carbon Constrained scenario is consistent with that developed by DOE and assumes that U.S. emissions of carbon are constrained in response to environmental concerns. Specifically, this scenario assumes that the Global Climate Change Initiative goal of an 18 percent reduction in national greenhouse gas intensity (below the 2002 level) is achieved by 2012 (White House, 2002). This effort is implemented as a tax of $100 per ton of carbon (/t C) emissions beginning in 2012, increasing at 3 percent in real terms thereafter, and assumes that annual emissions are held constant at that level thereafter.

Each panel was encouraged to define additional scenarios if it thought that such scenarios would more accurately portray the program’s benefits. Only one panel needed an entirely new scenario. The Panel on DOE’s Distributed Energy Resources Program found that DG technologies, and CHP in particular, are of much greater value to host facilities located in large urban areas of the country. As defined by the panel, the scenario assumes that sufficient power is not available for some extended time (weeks or months) in one or more high-demand load pockets in the affected regions of the country. The critical power shortage can be ameliorated by taking several actions, including reducing voltage, imposing selected rolling blackouts; terminating supply to selected high demand customers with interruptible electricity service contracts; increasing real time electricity prices; implementing special energy efficiency programs; and installing CHP

15

All prices are stated in 2003 dollars and thus do not reflect inflation.

Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

systems in load-constrained areas. This energy-constrained scenario recognizes the value of bringing new electricity to an urban area and increases the expected net benefits of CHP by over 40 percent.

Two other panels found that one or more of the global scenarios required some clarification to address issues encountered by the DOE program in question:

  • The Panel on DOE’s Carbon Sequestration Program found that a good deal of private sector R&D was taking place and thought that even more R&D would take place if it were widely known that carbon taxes or constraints were imminent. To address the timing of new carbon taxes or constraints, the panel assumed that the carbon-constrained scenario would include an announcement of the carbon tax 5 years before the tax is levied. The announcement would expedite private sector R&D and reduce the impact of DOE funding. This assumption about the timing of the announcement of constraints had a significant effect on the estimated benefit.

  • The Panel on DOE’s Integrated Gasification Combined Cycle (IGCC) Technology Program found the scenario approach to be valuable but adjusted the scenarios to account for the dependence of outcomes on the success of other R&D programs and the dissemination of competing technologies. Thus, the panel estimated one benefit level if IGCC displaces pulverized coal and a higher level if it displaces natural gas combined cycle (NGCC) power plants. This difference is particularly noticeable for the High Oil and Gas Prices scenario.

  • Finally, the Panel on DOE’s Light-Duty Hybrid Vehicle Technology Program found the scenarios useful but found it difficult to translate DOE’s program goals into expected benefits in the three global scenarios because different elements of the program reacted differently to different scenarios. For example, with the R&D on improved battery performance, the Reference Case led to moderate benefits for the program; the High Oil and Gas Prices led to higher probabilities of achieving the goal and higher benefits; and the Carbon Constrained scenario led to an intermediate result. The benefits and probabilities for the vehicle light-weighting R&D varied by scenario in a different way, and the advanced combustion engine R&D was not sensitive to the choice of scenario.

The National Energy Modeling System

NEMS is the basic tool used by DOE to estimate energy prices and consumption. The committee’s recommended methodology proposes to use NEMS to develop prices and quantities for the global scenarios, but also suggested that in most cases a simpler spreadsheet analysis would be sufficient to account for the changes in outcomes caused by introducing a new technology in a given scenario. This approach was taken by the panels in the Phase Two study.

NEMS is a critical tool for deriving price and quantity projections of programs that could have the potential for economy-wide consequences. Its value lies in identifying and characterizing equilibrium and feedback effects. For example, carbon constraints will dramatically increase the cost of electricity generated from coal in the absence of sequestration technology. This will have a number of economic consequences, each of which is relevant to evaluating the economic benefits of the DOE sequestration program. Absent the technology, demand will rise for natural gas (as will its price) as an alternative to coal for generating electricity. The cost increase is large enough to affect both demand and supply: Given the increased costs, electricity demand itself will diminish without sequestration technology. Estimates depend on assumptions and projections about economic growth, industrial use, base load versus peak load deployment opportunities, and other technological developments and cannot be easily calculated without NEMS.

NEMS also yields costs of alternative technologies, thereby providing additional guidelines for the choice of scenarios. Gasification, like sequestration, has the potential to bring about major shifts in the electricity generation mix, electricity prices, and fuel input prices. The cost projections for alternative technologies indicate the range of IGCC cost and performance characteristics over which IGCC will satisfy a large share of baseload power needs, only a moderate share, or no share at all (if it is unable to compete with other technologies).

Although NEMS is invaluable for these purposes, a key consideration in using it is whether small changes in parameters—for example, a 10 percent increase or decrease in the price of the technology—are likely to yield estimates significantly different from, or, possibly, even more accurate than estimates yielded by modifying or interpolating from a few NEMS runs. The model itself, as well as cost estimates for the technologies and the assigned probabilities, is subject to uncertainty. Furthermore, considerable time and resources are needed to use the NEMS model.

The work of the panels in Phase Two suggests that the committee’s recommended approach of using simple spreadsheet models calibrated to NEMS results is feasible and workable. In particular, each panel was able to estimate the change in benefits associated with reasonably small variations in cost by interpolating from several NEMS runs rather than requiring separate estimates for each of the alternatives considered in the analysis. In some panels, NEMS was not needed at all. For example, the chemical plant energy efficiency projects considered by the Panel on DOE’s Chemical Industrial Technologies Program involved niche markets, so that market prices for energy inputs and overall demand for other goods are unlikely to be significantly affected by the outcome of the program.

Although the spreadsheet approach to calculating benefits was successfully applied in each case, some committee members felt that the panels did not devote enough time and attention to the benefits calculation model. The benefit mod-

Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

eling process, like decision tree modeling and probability assessment, is challenging. It requires considerable expertise and judgment and should not be considered a cleanup task to be taken care of by the consultant and one or two panelists. Rather, the approach and logic underlying these calculations should be presented to all the panel members so they can understand and debate its key assumptions.

Modeling Alternative Technologies

The success or failure of alternative technologies can significantly affect the value of a DOE applied research program. These other technologies may be competing programs, each of which is directed to the same goal, or complementary programs that must also be successful if the program being analyzed is to produce a benefit. Technologies that have a major impact on project benefits should be included in the analysis, but this is methodologically challenging. The Phase Two experience helped to define the problem but did not fully resolve it.

When dealing with competing technologies, the benefits analysis must consider the entire set of competing projects, so benefits will not be double counted. In these cases, investing in competing technologies increases the probability of success but not the benefits associated with success.

A similar situation exists, but with different net program benefits, when two distinct competing technologies are invested in by different parties (e.g., IGCC at DOE and pulverized coal technology by private companies). While the likelihood of a technology’s success and its associated benefit increases in this situations, the net benefit of the government program decreases with the likelihood that the private technology will succeed. When the alternative technology programs are conducted outside DOE, the committee’s methodology can estimate the benefits satisfactorily by including outcomes for these competing technologies in the decision tree. For example, the success of pulverized coal technology with carbon capture affects the likely market penetration of IGCC technology. That relationship can be reflected in the decision tree directly by assigning probabilities to the relevant outcomes.

Benefits can be more difficult to assess with complementary programs, although the principles remain the same. For example, the main components of the DOE R&D electricity programs in fossil energy and energy efficiency are complementary, most obviously, coal gasification and sequestration. Perhaps less obvious are the complementarities of electricity programs generally with the fuel cell and DG programs. These last two technologies use natural gas and reduce the share of peak power plants in the generation mix. Overall, according to the NEMS results, the program increases both demand for IGCC and its attractiveness relative to NGCC.

While more work is required to define the portfolio analysis in the presence of complementarities, the committee notes that it will be important to consider the connections among DOE programs when grouping programs for a benefits assessment. For example, a review of the full DER effort may be more desirable than a review of the CHP element alone. Aggregating along these lines can reduce the complications of complementarities by subsuming them in a more general decision tree analysis.

The Phase Two studies provide relatively little experience in modeling dependence between DOE projects and technologies. The Panel on DOE’s Chemical Industrial Technologies Program examined a portfolio of small research projects but treated them as mutually independent; that is, all could succeed or fail without significantly affecting the benefits of any of the others. The panel showed that in this limited case, it is possible to assign overall success probabilities to each project and to create a cumulative probability distribution of expected benefits using a Monte Carlo simulation. However, further work would be required to analyze jointly a portfolio of projects where the success of one could affect the benefits of others.

In a full implementation of this program assessment process, the committee recommends that in selecting activities for review the interdependencies of program elements be considered. For example, the IGCC, gas turbine, carbon sequestration, and stationary fuel cell activities are all programs within the Office of Fossil Energy that address a carbon-constrained scenario. In addition to these activities is the FutureGen coal demonstration project. If the review of these components is carefully coordinated, decision makers would gain a more integrated view of the benefits of the overall program.

Implementation Issues

The panels were generally successful in implementing the committee’s methodology, but the Phase Two experience uncovered three issues that should be considered in future studies:

  1. The natural gas and hybrid vehicles panels were not able to calculate benefits for certain scenarios. To a large extent this was due to the inadequate support provided by DOE program staff. To a lesser extent, the strong technical competency of the panel members led them to focus more on technical risks than market risks. The committee recommends that the next assessment engage an additional consultant with benefits modeling expertise to ensure that the benefits are calculated in a consistent manner. The committee determined that it would be helpful as well to have these calculations performed between the first and second panel meetings and made available to the panelists, on a preliminary basis, at the start of the second panel meeting.

  2. If the DOE programs account for only a small share of the overall international and/or industrial effort in the area, it is very important for the panel to be knowledgeable about that overall effort so that it is able to reliably assess

Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

the potential impact of the DOE program. Both the Natural Gas Exploration and Production Program and the Chemical Industrial Technologies Program are in this category, and it is these panels that had difficulty assessing the benefits of the two DOE programs.

  1. Applying the prospective benefits methodology to DOE’s light-duty vehicle hybrid technology R&D program required the panel to specify key items that were not always apparent from the documents and information provided by DOE. In particular, some of the program goals were not described explicitly and completely. For example, setting a cost target of $28/kW for a battery by the year 2010 does not describe the objective adequately for assessment purposes. Does the cost target mean a customer could actually buy a battery at that cost? Does it mean that the technology exists that in principle would allow a commercial firm to make such a product? Does it mean the 500,000th production unit or the first? All these conditions must be specified for the assessment method to succeed, and both reviewers and proponents must state their goals quite explicitly.

Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 40
Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 41
Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 42
Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 43
Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 44
Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 45
Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 46
Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 47
Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 48
Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 49
Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 50
Suggested Citation:"3 Methodology for Prospective Evaluation of Deparment of Energy Programs." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 51
Next: 4 Expert Panel Process »
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) Get This Book
×
Buy Paperback | $69.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Since its inception in 1977 from an amalgam of federal authorities, the U.S. Department of Energy (DOE) has administered numerous programs aimed at developing applied energy technologies. In recent years, federal oversight of public expenditures has emphasized the integration of performance and budgeting. Notably, the Government Performance and Results Act (GPRA) was passed in 1993 in response to questions about the value and effectiveness of federal programs. GPRA and other mandates have led agencies to develop indicators of program performance and program outcomes. The development of indicators has been watched with keen interest by Congress, which has requested of the National Research Council (NRC) a series of reports using quantitative indicators to evaluate the effectiveness of applied energy research and development (R&D).

The first such report took a retrospective view of the first 3 years of DOE R&D programs on fossil energy and energy efficiency. The report found that DOE-sponsored research had netted large commercial successes, such as advanced refrigerator compressors, electronic lighting ballasts, and emission control technology for flue gas desulfurization. However, some programs were judged to be costly failures in which large R&D expenditures did not result in a commercial energy technology. A follow-up NRC committee was assigned the task of adapting the methodology to the assessment of the future payoff of continuing programs.

Evaluating the outcome of R&D expenditures requires an analysis of program costs and benefits. Doing so is not a trivial matter. First, the analysis of costs and benefits must reflect the full range of public benefits that are envisioned, accounting for environmental and energy security impacts as well as economic effects. Second, the analysis must consider how likely the research is to succeed and how valuable the research will be if successful. Finally, the analysis must consider what might happen if the government did not support the project: Would some non-DOE entity undertake it or an equivalent activity that would produce some or all of the benefits of government involvement?

This second report continues to investigate the development and use of R&D outcome indicators and applies the benefits evaluation methodology to six DOE R&D activities. It provides further definition for the development of indicators for environmental and security benefits and refines the evaluation process based on its experience with the six DOE R&D case studies.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!