2
Framework for the Study

OVERVIEW

In theory, evaluating the benefits and costs of DOE’s research program should be relatively straightforward. It would require adding up the total benefits and costs of research conducted since 1978, determining what proportion of each benefit is attributable to DOE funding, and calculating a balance between the DOE contributions and the cost of achieving them. In practice, of course, methodological challenges abound. Of these, the most fundamental is how to define and systematically capture the diversity of benefits that result from publicly funded research within a dynamic environment of marketplace activity, technological advancement, and societal change. In this chapter, the framework the committee developed for doing so is discussed, as well as comments on some of the implications of applying it.

THE SETTING

Basic economic principles suggest that the private sector undertakes research and commercializes technologies when private firms can capture economic benefits in excess of the costs of achieving them. Justification for public sector research rests on the observation that the private sector cannot capture some of the benefits. Environmental benefits not recognized in market prices provide a familiar example of this principle, but there are others, including the difficulty of capturing proprietary benefits from basic research.

As background for its study of DOE-sponsored R&D, the committee decided to examine the role played by industry and government in developing the technologies that successfully came to market and therefore presumably produced significant private benefits. The committee, with the help of outside experts, compiled a list of the most important advances in fossil energy and energy efficiency technology over the past two decades. Based on the experience of the committee and other experts, judgments were then made about the significance of both industry and DOE funding and DOE’s achievement of each of these technologies (Table 2–1).

The technologies listed in Table 2–1 probably all benefited from what may be called “critical facilitating tech-

TABLE 2–1 The Most Important Fossil Energy and Energy Efficiency Technological Innovations Since 1978

Technology Now in the Marketplace

Level of DOE Influence

Fossil energy

Efficient gas turbine in stationary systems

A/M

3-D seismic imaging

A/M

Deep water drilling and production

A/M

Improved oil and gas reservoir characterization and modeling

A/M

Improved oil and gas drilling: horizontal, deviated, and extended

A/M

Diamond drill bits

D

Coal-bed methane

I

Flue gas cleanup

I

Atmospheric fluid-bed combustion

I

Fracture technology for tight gas

I

Oil refinery optimization

A/M

Longwall coal mining

A/M

Coal cleaning

A/M

Energy efficiency

More efficient electric motors

A/M

Higher mileage automobiles

A/M

More efficient electronic ballasts

D

More efficient household refrigerators

D

More effective insulation

I

Synthetic lubricants

A/M

More efficient gas furnaces

A/M

More energy-efficient windows

I

More efficient industrial processes

A/M

More efficient buildings

I

NOTE: Influence levels: A/M, absent or minimal; I, influential; D, dominant.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 13
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 2 Framework for the Study OVERVIEW In theory, evaluating the benefits and costs of DOE’s research program should be relatively straightforward. It would require adding up the total benefits and costs of research conducted since 1978, determining what proportion of each benefit is attributable to DOE funding, and calculating a balance between the DOE contributions and the cost of achieving them. In practice, of course, methodological challenges abound. Of these, the most fundamental is how to define and systematically capture the diversity of benefits that result from publicly funded research within a dynamic environment of marketplace activity, technological advancement, and societal change. In this chapter, the framework the committee developed for doing so is discussed, as well as comments on some of the implications of applying it. THE SETTING Basic economic principles suggest that the private sector undertakes research and commercializes technologies when private firms can capture economic benefits in excess of the costs of achieving them. Justification for public sector research rests on the observation that the private sector cannot capture some of the benefits. Environmental benefits not recognized in market prices provide a familiar example of this principle, but there are others, including the difficulty of capturing proprietary benefits from basic research. As background for its study of DOE-sponsored R&D, the committee decided to examine the role played by industry and government in developing the technologies that successfully came to market and therefore presumably produced significant private benefits. The committee, with the help of outside experts, compiled a list of the most important advances in fossil energy and energy efficiency technology over the past two decades. Based on the experience of the committee and other experts, judgments were then made about the significance of both industry and DOE funding and DOE’s achievement of each of these technologies (Table 2–1). The technologies listed in Table 2–1 probably all benefited from what may be called “critical facilitating tech- TABLE 2–1 The Most Important Fossil Energy and Energy Efficiency Technological Innovations Since 1978 Technology Now in the Marketplace Level of DOE Influence Fossil energy Efficient gas turbine in stationary systems A/M 3-D seismic imaging A/M Deep water drilling and production A/M Improved oil and gas reservoir characterization and modeling A/M Improved oil and gas drilling: horizontal, deviated, and extended A/M Diamond drill bits D Coal-bed methane I Flue gas cleanup I Atmospheric fluid-bed combustion I Fracture technology for tight gas I Oil refinery optimization A/M Longwall coal mining A/M Coal cleaning A/M Energy efficiency More efficient electric motors A/M Higher mileage automobiles A/M More efficient electronic ballasts D More efficient household refrigerators D More effective insulation I Synthetic lubricants A/M More efficient gas furnaces A/M More energy-efficient windows I More efficient industrial processes A/M More efficient buildings I NOTE: Influence levels: A/M, absent or minimal; I, influential; D, dominant.

OCR for page 13
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 nologies,” most of which DOE had some part in developing. These technologies include the following: Improved materials and catalysts; Improved instrumentation, sensors, and controls; Improved computer hardware; Improved software; Improved process and combustion modeling; and High-bandwidth communications. The committee did not attempt to evaluate the role of DOE in these critical facilitating technologies. This analysis, admittedly subjective, nevertheless suggests that the private sector did in fact develop and deploy many important technologies without DOE participation. On the other hand, DOE did make an influential or dominant contribution in 9 of the 22 technologies reviewed. The rough conclusion to be drawn from these observations is that the DOE funding of energy R&D is not necessarily associated with the most obviously attractive advances. Rather, as basic economic principles suggest, DOE research should also, and even mostly, be associated with public policy objectives. THE FRAMEWORK Based on this general philosophy, the committee developed a comprehensive framework to define the range of benefits and costs, both quantitative and qualitative, that should be considered in evaluating the programs. The framework is intended to summarize all net benefits to the United States, to focus attention on the major types of benefits associated with the DOE mission, and to differentiate benefits based on the degree of certainty that the benefits will one day be realized. It has been designed to capture two dimensions of publicly funded R&D: (1) DOE research is expected to produce public benefits that the private economy cannot reap and (2) some benefits may be realized even when a technology does not enter the marketplace immediately or to a significant degree. The matrix shown in Figure 2–1 and discussed below provides an accounting framework for the consistent, comprehensive assessment of the benefits and costs of the fossil energy and energy efficiency R&D programs. The matrix can be completed for each discrete program, project, or initiative that has a definable technological objective and outcome. The framework recognizes that the technologies being evaluated may be in different stages of the RD&D cycle; as well, by its nature, the framework represents a snapshot in time, with a focus on outcomes of the work performed. Class of Benefits (Rows of the Matrix) The classes of benefits, which correspond to the rows of the matrix, are intended to capture types of public benefits appropriate to DOE R&D programs. DOE’s current stated mission spells out these benefits in general terms, as follows (DOE, 2000): “To foster a secure and reliable energy system that is environmentally and economically sustainable, to be a responsible steward of the Nation’s nuclear weapons, to clean up our own facilities, and to support continued United States leadership in science and technology.” The Strategic Plan expands on the energy aspect of the mission as follows: “The Department is working to assure clean, affordable, and dependable supplies of energy for the Nation, now and in the future. That means increasing the diversity of energy and fuel choices and sources, bringing renewable energy sources into the market, strengthening domestic production of oil and gas, supporting commercial nuclear energy research, and increasing energy efficiency” (DOE, 2000). The fossil energy and energy efficiency programs each have a mission statement, and the individual R&D initiatives or projects may have more explicit and focused objectives. The approach of each program to benefit analysis, as   Realized Benefits and Costs Options Benefits and Costs Knowledge Benefits and Costs Economic benefits and costs       Environmental benefits and costs       Security benefits and costs       FIGURE 2–1 Matrix for assessing benefits and costs.

OCR for page 13
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 presented to the committee in briefings and background documents, reflects the general themes of the DOE mission statement and is encompassed within it. Based on these stated objectives, the committee adopted the three generic classes of benefits (and related costs) for the energy R&D programs: “economic,” “environmental,” and “security” benefits. The entry in each cell of the matrix is a measure of the economic, environmental, or security net benefit further characterized according to the column classification schemes, discussed below. Economic costs, or undesirable consequences, are quantified as negative components of net benefits, and economic benefits, or desirable consequences, as positive components. Ideally, the entries in the cells would be quantitative measures of each category of net benefits; in some cases, however, only qualitative descriptors are possible. Economic net benefits are based on changes in the total market value of goods and services that can be produced in the U.S. economy under normal conditions, where “normal” refers to conditions absent energy disruptions or other energy shocks. The benefit must be measured net of all public and private costs. Economic value is increased either because a new technology reduces the cost of producing a given output or because it allows additional valuable outputs to be produced by the economy. Economic benefits are characterized by changes in the valuations based on market prices. These benefits must be estimated on the basis of comparison with the next best alternative, not some standard or average value. The “next best alternative” is defined as a technology (or combination of technologies) that is available and commercially proven that would accomplish essentially the same objective as a technology being evaluated and would be the technology of choice for a buyer in the market. This avoids the common problem of comparing a new technology with technology currently in general use rather than with technology that is already available and that could replace the existing technology. In many instances, there may be no alternative better than the one in general use. Environmental net benefits are based on changes in the quality of the environment that have occurred, will occur, or may occur as a result of the technology. A technology could directly reduce the adverse impact on the environment of providing a given amount of energy service by, for example, reducing sulfur dioxide emissions per kilowatt-hour of electric energy generated by a fossil fuel-fired power plant, or by indirectly enabling the achievement of enhanced environmental standards (by, for example, introducing the choice of a high-efficiency refrigerator). Environmental net benefits are typically not directly measurable by market prices but by some measure of the valuation society is willing to place on changes in the quality of the environment. They can often be quantified in terms of reductions in net emissions or other physical impacts. In some cases, market values can be assigned to the impacts based upon emissions trading or other indicators. Security net benefits are based on changes in the probability or severity of abnormal energy-related events that would adversely impact the overall economy, public health and safety, or the environment. Historically, these benefits arose in terms of national security issues, i.e., they were benefits that assured energy resources required for a military operation or a war effort. Subsequently, they focused on dependence upon imported oil and the vulnerability to interdiction of supply or cartel pricing as a political weapon. More recently, the economic disruptions of rapid international price fluctuations from any cause have been emphasized. Currently, the economic and health and safety consequences of unreliable energy supply have become a more general security issue. The reliability of electric power grids was the initial concern, but natural gas transportation and storage and petroleum refining and product supply systems are now receiving attention. Security net benefits can be seen as special classes of economic net benefits or environmental net benefits. They are “special” because they accrue from preventing events that have a relatively low likelihood or a low frequency of occurrence. Range of Benefits (Columns of the Matrix) The columns in the matrix are the first step toward a more explicit definition of the benefits to be included. They recognize a range of benefits from R&D that are logical measures of the value of the programs. The categories are “realized,” “options,” and “knowledge.” The three columns reflect degrees of uncertainty about whether the particular benefits have been or will be obtained. Two fundamental sources of uncertainty are particularly important: technological uncertainties and uncertainties about economic and policy conditions. The technology development programs can be classified according to whether the technology has been developed, is still in progress, or has terminated in failure. All else being equal, a technology still under development is less likely to result in benefits than a technology that has already been successfully developed, since technological success is not assured in the former case. However, even if a technology is never successfully developed, the knowledge gained in the program could lead to another beneficial technology. Similarly, if a technology is fully developed and economic and policy conditions are favorable for its commercialization, there can be reasonable confidence that future benefits will accrue. However, it may be that economic and policy conditions are not expected to be favorable but might become favorable under plausible circumstances. In this case, the benefits may occur, but their probability is lower. Finally, while it may be virtually certain that the economic and policy conditions will never become favorable and that the technology itself will never be adopted, the knowledge associated with the technology development may be appli-

OCR for page 13
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 cable in other ways, possibly—but not probably—resulting in benefits. Rather than attempting to fully characterize the uncertainty of benefits, the committee has used two distinctions—state of technology development and the favorability of economic and policy conditions—to place a benefit in one of the three columns. The first column in Figure 2–1, called “realized benefits,” is reserved for benefits that are almost certain: those for which the technology has been developed and economic and policy conditions favor its commercialization. The second column, which includes less certain benefits, is called “options benefits.” These are benefits that may be derived from technologies that are fully developed but for which economic and policy conditions might but are not likely to favor commercialization. All other benefits, to the extent they exist, are called “knowledge benefits.” The category is thus a very broad one. It includes knowledge generated by programs still in progress, programs terminated as failures, and programs that were technological successes but will not be adopted because economic and policy conditions will never be favorable. Figure 2–2 summarizes the committee’s notions of the range-of-benefit columns. Realized net benefits can be characterized as economic, environmental, or security benefits. They accrue from technologies for which the R&D has been completed and that have been or are ready to be commercialized on an economic basis under current economic, regulatory, and tax conditions. Options net benefits can also be characterized as economic, environmental, or security benefits; they are based on technologies for which the R&D has been completed and for which the costs and technical capabilities are reasonably certain but that have not been commercialized. These technologies are not commercially viable under current economic conditions, but some plausible future circumstance, such as changed price structures, limitations on alternative technologies or resources, or evolving health or environmental standards could make them a valuable option. Knowledge benefits—also classifiable as economic, environmental, or security—comprise useful or potentially useful scientific knowledge and technology that have resulted from the R&D initiatives and that are not reflected in the realized or options benefits. Measures of Value (Entries in the Matrix Cells) To arrive at entries for the cells of the matrix, a logical and consistent set of rules for measuring the results of the individual initiatives is also necessary. These rules define more exactly the meanings of the rows and columns and provide a calculus for measuring the values to be entered in each of the cells. A complete discussion of the rules to be applied in using the matrix was prepared to guide the committee’s own efforts and to request information from DOE. It is presented as Appendix D of this report. Some of the more important rules are abstracted here to assist the reader in understanding the results of the evaluation. Economic Benefits The estimate of economic benefits resulting from an R&D initiative is intended to measure the net economic gain captured by the economy. The impact of a new technology is measured by comparing it with the next best alternative that was available when the technology was introduced or that would have been available absent the DOE efforts. Benefits are intended to be net of all economic costs of achieving the benefits, not just the cost to the direct participants in the R&D initiative. Benefits and costs are to be calculated on the basis of the life cycle of investments. Dollar amounts are all expressed in constant 1999 dollars. The committee did not discount benefits, costs, or governmental expenditures but added together benefits from different years, adjusted only for inflation. Neither macroeconomic stimulation of the national economy or the creation of jobs is to be considered a benefit Economic/Policy Conditions\Technology Development Technology Developed Technology Development in Progress Technology Development Failed Will be favorable for commercialization Realized benefits Knowledge benefits Knowledge benefits Might become favorable for commercialization Options benefits Knowledge benefits Knowledge benefits Will not become favorable for commercialization Knowledge benefits Knowledge benefits Knowledge benefits FIGURE 2–2 Derivation of columns for the benefits matrix.

OCR for page 13
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 of an R&D initiative. In today’s national economic circumstances, such impacts are more likely to be transfers rather than net increases at the national level. In any case, the investment of similar amounts of funds elsewhere in the economy would also have impacts. To attribute net macroeconomic benefits to a particular R&D initiative, therefore, would be highly speculative and should not be done. Unintended improvements in economic activities that are unrelated to the objectives of the R&D initiative usually should not be counted as benefits in evaluating the success of the R&D. Such serendipitous results may offset the costs to the public of the initiative, but they are a random consequence of investment. Ancillary benefits might have resulted from investing the funds elsewhere. Judgment must be applied in specific cases to determine if the results are relevant to the objectives of the initiative. Environmental Benefits Environmental benefits result when the introduction of a new technology RD&D program makes possible an improvement (or reduced degradation) in measures of environmental quality. Most often, the benefit is a net reduction in toxins or other harmful emissions compared with the situation that would have prevailed in the absence of the technology. Such benefits might be achieved by improving emission controls or increasing the efficiency of emission-producing processes. In some cases, an environmental benefit may be a net reduction in the use of environmental resources for the provision of energy services, including a reduction of adverse impacts on land use, air and water quality, or aesthetics. Savings in the costs of achieving a given standard of emission control or a required level of remediation would be considered to be an economic benefit. Environmental benefits result only if there is a net improvement in environmental quality from what would have been the case absent the DOE program. Security Benefits The prevention or mitigation of macroeconomic losses resulting from energy disruptions can be considered as a security benefit. Transient and unpredicted impacts on the national economy of sudden and/or unpredicted service interruptions or price shocks can severely impair productivity at the national level, leading to real costs that can be estimated. Reductions in the probability or severity of such events are appropriate measures of the security benefit of R&D initiatives. It may be possible to calculate a reasonable realized security benefit—for example, in the case of a technology that has demonstrably reduced the frequency of electric service interruptions. More often, however, security benefits based on changing the probability of international energy disruptions will be difficult to quantify and will instead be described qualitatively. Realized Economic Benefits In computing realized economic benefits, the net lifecycle effects of a completed technology are considered. However, the decreases in damages associated with reduced releases of materials as a result of the new installations may last for much longer times. Benefits are included for the entire time of this decreased damage. Realized economic benefits should include the results of the life-cycle operation of all capital stock utilizing the technology that has been installed through the year 2000 and that is projected to be installed through 2005 (the 2005 rule). A new technology may well be adopted for new installations beyond a 5-year horizon, but for technologies that provide significant economic benefits that can be captured by private sector investments, it is reasonable to assume that at some point a comparable improvement would have been introduced in the absence of the DOE R&D initiative. Adopting a 5-year limit (the 5-year rule) on future installations but allowing the full useful life of the installations to be considered provides a reasonable but conservative estimate of the contribution of the technology without introducing speculative projections of its longer-range impact. The committee’s calculations also assume that the DOE R&D or demonstration program advanced the introduction of new technology into the market by 5 years. Options Benefits Options benefits are credited to those technologies for which the R&D has been completed and the technological and economic attributes are reasonably well known. These technologies can be considered to be “on the shelf” and available for commercialization if future circumstances warrant. They may be uneconomic under current pricing conditions but become viable if the costs of alternatives rise. They may also become viable if the alternatives are curtailed by increasingly stringent environmental, health, or safety regulations or by unexpected constraints on fuels or other resources. Judgment must be used in specific cases. Not all unsuccessful R&D initiatives can be viewed as potentially viable in situations that have credible possibilities of occurring. Knowledge Benefits Knowledge benefits are defined as scientific knowledge and useful technological concepts resulting from the R&D that have not yet been incorporated into commercialized results of the program but hold promise for future use or are useful in unintended applications. These are products of the research that have value over and above the benefits that have been accounted for in the other two columns of the matrix. Knowledge benefits may include unanticipated and not closely related technological spin-offs that are made possible by the research programs. This is probably the broadest and most heterogeneous category of benefits.

OCR for page 13
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 CONDUCT OF THE STUDY The committee began its work in June 2000. As envisioned by the statement of task, the committee first developed an analytic framework for assessing benefits. The committee reviewed a number of reports (see Appendix C) prepared by others over the years evaluating DOE’s R&D program. Unlike most of these reports, the charge for this project focuses attention on assessing the actual outcomes of DOE’s energy R&D programs. The committee therefore elected to take a case-study, data-intensive approach to this project, recognizing that time and resource constraints would prevent it from resolving every analytic issue and closing all the gaps in data that ideally would be needed to implement the analytic framework. Because of these constraints, the committee identified a representative sample of programs and projects as a basis for arriving at overall findings and recommendations. As outlined in the discussion of the task statement in Chapter 1, this selection was designed both to identify lessons learned from the range of programs conducted by DOE and to evaluate the utility of the analytic framework in a diversity of circumstances. The committee then asked the Office of Fossil Energy and the Office of Energy Efficiency and Renewable Energy at DOE to provide information required by the framework, and to do so following the detailed procedures specified in Appendix D. Both the framework and the procedures are essential parts of the methodology developed by the committee. Both offices supplied a great deal of statistical and analytic information in response to the committee’s request. Much of the data provided had to be developed specifically for this study. Because the programs changed over time, the task of documenting programs as far back as 1978 was at times extremely challenging. Each of the 39 case studies was assigned to a committee member for analysis. With the help of an independent consultant, committee members assessed the DOE submissions for quality and conformance to the analytic methods prescribed by the committee. Considerable iteration and correction took place in this process to ensure that the committee’s procedures were followed. As the study proceeded, the framework was refined. The cooperation of DOE staff in this process was exemplary, and it is gratefully acknowledged. The committee met as a whole and in subgroups to ensure that the analytic process was being applied consistently across all of the case studies. In addition, considerable attention was paid to the use of common assumptions, designed to promote comparability of results across case studies as well as conservatism in the valuing of benefits. One such assumption is embodied in the 5-year rule, which assumes the technology would have entered the market 5 years later without government involvement. For example, if a technology entered the market with DOE involvement in 1992, the 5-year rule assumes the technology would have gotten to market in 1997 without a government program. Another assumption is the 2005 rule, by which the committee assessed benefits for all the technologies evaluated by the committee as being installed in the market by 2005 and assessed those benefits over their useful economic life. The year 2005 was used because the committee was reasonably sure of economic and other conditions up to that time and did not want to project out further because of uncertainties. As part of its deliberations, the committee invited members of government, industry, and public interest groups to comment on the goals, performance, and effectiveness of the relevant DOE research and development programs over the period of interest. Appendix B lists the formal comments received during the course of the project. In analyzing the case studies, the committee also directly contacted other representatives of industries that participated with DOE in the case study programs to secure their views on the value of the research and DOE’s role in it. In these ways, the committee attempted to be conservative in the judgments it drew from the available data. While much more can and should be done to refine the methodology launched with this study, the committee believes the methodology has come far enough to allow stating with confidence the findings and recommendations included in this report. ASSESSMENT OF THE METHODOLOGY The committee considers that the analytic methodology described in this chapter is useful as an internally consistent and comprehensive framework for the objective comparison of the benefits and costs of energy R&D programs across programs and technologies. Its opinion is based on the actual application of the methodology in the 39 case studies of diverse technologies. In the course of this experience, however, a number of lessons bearing on the methodology’s implications and future utility were identified. To provide perspective on the more detailed analyses that follow, as well as to suggest directions for improvement, several of the lessons learned are discussed here: Specifying categories of benefits by means of systemic analysis is a useful discipline. In particular, benefit evaluation must take care to give adequate weight to benefits other than realized economic benefits (the upper left corner cell of the matrix). Quantifying realized economic benefits is usually easier than quantifying the kinds of benefits that fit in the eight other cells, and the temptation is great to focus on these easily quantified benefits. But, as the committee has noted, environmental and security benefits, while harder to value in dollar terms, are equally important objectives of public funding. Similarly, creating options in the face of future oil price changes and acquiring knowledge that can be

OCR for page 13
Energy Research at DOE was it Worth it?: Energy Efficiency and Fossil Energy Research 1978 to 2000 used by many private sector actors are important public benefits. Many of the case studies in the committee’s sample experienced the changes in policy and other changes that occurred in DOE R&D programs outlined in Chapter 1. This fact needs to be taken into account in judging the outcomes observed when the committee applied the framework. For example, the programs to develop a technology for making liquid fuels from coal were not notably successful. However, had oil prices continued to rise, as expected at the time the program was designed, the outcome might have been more favorable. In some cases, these effects are so striking that the committee notes them explicitly. In all cases, the reader should consider the context of the program before arriving at a final judgment about its benefits. More refined analysis of knowledge benefits would improve the methodology. The committee’s focus on outcomes results in many benefits falling into the knowledge category. In some cases, this is because recently begun research projects have not yet had time to achieve their expected results. In other cases, research that is abandoned before producing a realized or optional technology also produces mainly knowledge benefits. Distinguishing between these two kinds of knowledge benefits may provide useful information that the present version of the methodology does not provide. The committee’s use of the 5-year rule should not be be interpreted to mean that the only effect of federal R&D is to accelerate the introduction of a technology into the marketplace by 5 years. The committee recognizes that there may be many effects of federal R&D, including the acceleration of a technology into the marketplace by more than 5 years, or other effects such as an increase in the ultimate market penetration of a technology. The committee used the 5-year rule because it needed a uniform, conservative standard for the analysis of these particular case studies. As noted earlier, quantification of the benefits suffers from inherently difficult methodological problems. The time and resource constraints of this study made it difficult even to apply fully the valuation methods that do exist. Where it has used quantified benefits to support its findings and recommendations, the committee considers it has been conservative in establishing upper and lower bounds for its benefit estimates. In general, the committee believes it is more likely than not that a more thorough analysis would increase the values of the benefits that the committee has assigned to DOE’s programs. Perhaps the most difficult analytic problem is assigning to DOE a proportion of the overall benefit of an R&D program that properly reflects DOE’s contribution to it. In most of the case studies, DOE, industry, and—sometimes—other federal and nonfederal governmental research organizations contributed to the outcome of the research program. In some cases, as in the development of seismic technology, for example, industry made virtually all of the contribution, but DOE nevertheless made an important one. The committee has found no reliable way to quantify the DOE contribution in most cases, and doing so remains a methodological challenge for the future. For the purpose of this study, the committee has simply attempted to identify in its case study analyses the specific role that DOE played, by looking at the outcome that would not have happened had DOE not acted. The committee considers that it has used conservative judgment in characterizing the DOE contribution for the purpose of developing findings and recommendations. REFERENCE Department of Energy (DOE). 2000. Strategic Plan. Strength Through Science: Powering the 21st Century. Washington, D.C.: U.S. Department of Energy. Available online at <http://www.energy.gov/index/indexs.html>.