Measuring the Impacts of Federal Investments in Research A Workshop Summary (2011) / Chapter Skim
Currently Skimming:
Pages 119-152
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 119... ...
APPENDIX C THE PROMISES AND LIMITATIONS OF PERFORMANCE MEASURES Irwin Feller Senior Visiting Scientist, American Association for the Advancement of Science and Professor Emeritus, Economics, Pennsylvania State University I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of Science, whatever the matter may be. -Baron William Thomson Kelvin.
|
From page 120... ...
120 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH program evaluators and those weaned in the days of program-planningbudgeting. Addressing this analytical diffuseness, this paper assesses the promises and limitations of performance measures as means of measuring economic and other returns of the Federal government's investments in basic and applied research.
|
From page 121... ...
APPENDIX C 121 and technology activities…", and "… develop science of science policy" tools that can improve management of their research and development portfolios and better assess the impact of their science and technology investments."To these formal requirements may be added recent and likely increasing demands by congressional authorization and appropriations committees that agencies produce quantitative evidence that their activities have produced results, or impacts. Theory here stands for a more complex, bifurcated situation, creating what Manski has termed dueling certitudes: internally consistent lines of policy analysis that lead to sharply contradictory predictions.
|
From page 122... ...
122 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH This emphasis on context produces a kaleidoscopic assessment, such that promises and limitations change shape and hues as the decision and organizational contexts shift. An emphasis on context also highlights the analytical and policy risks of assessing the promises and limitations of performance impact measures in terms of stylized characteristics.
|
From page 123... ...
APPENDIX C 123 Analytical Framework and Scope The paper's analytical framework and empirical findings derive mainly from economics, although its coverage of performance measures is broader than economic statistics and its treatment of impact assessment is based mainly on precepts of evaluation design. The choice of framework accords with the workshop's objective, which is suffused with connotations of efficiency in resource allocation, or more colloquially, seeking the highest possible returns on the public's (taxpayer's)
|
From page 124... ...
124 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH of improved or new products produced; cost reductions from new and improved products/processes) ; pre-ultimate outputs (e.g., savings, cost reductions, and income generated by improved health, productivity, safety, and mobility of the workshop at sectoral and national levels)
|
From page 125... ...
APPENDIX C 125 program-planning-budgeting initiatives of the 1960s and continuing on through its several variants, are a search for decision algorithms that will lead to the improvement in government budgeting and operations and a search for criteria for setting priorities for science (Shils, 1969)
|
From page 127... ...
APPENDIX C 127 strengthening the basis of long term public support of these investments. • The search for measures that accurately depict what an agency/program has accomplished may serve as a focusing device, guiding attention to the shortcomings of existing data sets and thus to investments in obtaining improved data.
|
From page 128... ...
128 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH innovation policy contains several examples of Will Roger's observation that, "It isn't what we don't know that gives us trouble, it's what we know that ain't so." • Presented as a method of assessing returns to Federal investments in research, performance measurement provides policymakers and performers with an expanded, more flexible and adaptable set of measures than implied by rate of return or equivalent benefit-cost calculations. Criticism of what is seen as undue reliance on these latter approaches is longstanding; they are based in part on technical matters, especially in the monetization of non-market outputs, but also on the distance between the form that an agency's research output may take and the form needed for this output to have market or other societal impacts.
|
From page 129... ...
APPENDIX C 129 economy in not adequately covering the growth of the service sector or the internationalization of sources and performers of R and D, and did not adequately connect R and D expenditures with downstream "impact" measures, such as innovations. The result has been a major revision of these surveys, undertaken by NSF's Science Resources Statistics Division.
|
From page 130... ...
130 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH between ex ante and ex post decision making settings. Fundamental differences exist in the theoretical, analytical and empirical knowledge bases for using performance measures to determine whether past investments have produced the research expected of them and using such measures to decide upon the magnitude and direction of new funds.
|
From page 131... ...
APPENDIX C 131 based agnosticism of many scholars of the extent to which findings based on past studies can be used to forecast the specific magnitude and characteristics of future Federal investments in research. As noted by Crespi and Guena, for example, "After more than 50 years scholarly work on the importance of academic research, there is still little systematic evidence on how such investments can lead to increase levels of scientific output, improved patenting and innovative output, better economic performance and, ultimately, to increase national wealth" (Crespi and Geuana, 2008, p.
|
From page 132... ...
132 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH the nature of its production function. We can never predict the output which will be generated by a given volume of inputs.
|
From page 133... ...
APPENDIX C 133 as EMI in the United Kingdom and GE in the U.S., leading to the now ubiquitous presence of MRI (Kelves, 1997; Roessner, et.
|
From page 134... ...
134 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH objectives set for Federal research programs–productivity increase, for example, is not synonymous with job creation. • Leaving aside issues associated with data availability and quality, the casual linkages between program/agency objectives and the choice of measures to be used can be fuzzy.
|
From page 135... ...
APPENDIX C 135 relationship between US Nobel Prize awards and aggregate U.S. performance objectives in economic growth and health highlights these risks (U.S.
|
From page 136... ...
136 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH to 1 return for medical treatment and a 30-to-1 return for research and dissemination costs related to behavioral change. More generally, to the extent that single measures are used, they become the de facto criterion by which performance is judged, and in a system of performance-based budgeting the basis on which decisions about which future funding is made.
|
From page 137... ...
APPENDIX C 137 seemingly independent measures, so that what seems to be the richness of the approach in effect reduces to variants of a single measure. Perhaps most importantly, the presence of multiple objectives for a program increases the likelihood of trade-offs among facets of performance, so that an increase in one agency/program objective can be achieved only at the expense of a decrease in another objective.
|
From page 138... ...
138 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH research to a positive indicator of fulfillment of a university's "third mission" objectives in promotion and tenure packets. It may also change as a result of legislative or court decisions and/or firm strategies in the uses made of the measured activity- reported shifts, for example, in increased patent activities by firms and the trend towards using them as a source of revenue as well as a means of protecting intellectual property (Cohen, 2005)
|
From page 139... ...
APPENDIX C 139 pop again anew, not infrequently with new terms being used to describe recurring questions. The big three science policy questions, in the U.S.
|
From page 140... ...
140 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH fail to supply the (Pareto-) optimal quantity of certain types of R and D5 and (3)
|
From page 141... ...
APPENDIX C 141 issues involved in linking R and D with technical change and/or productivity change, that the business sector funds two-thirds of US R and D, and that total Federal R and D expenditures are the sum of multiple House and Senate appropriations' bills, how does one move from the 85% share of growth in per capita income attributed to technological progress contained in Rising Above the Gathering Storm to findings such as Boskin and Lau's estimate that 58% of the economic growth of the United States between 1950-1998 was attributable to technical progress to determining the optimal R and D/GDP ratio? How would the optimal level of Federal expenditures on R and D change if new findings suggested that existing estimates overstated the contribution of technical progress by 20/30 percentage points, or conversely, understated this contribution by a like amount?
|
From page 142... ...
142 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH all these proposals could be funded, what means should be used to select from among them? What measures of performance/output/outcomes should be used to assess past performance in determining-out year investments or near-term R and D priorities.
|
From page 143... ...
APPENDIX C 143 One would like to do better than this. Here, if anywhere, is where performance measurement may have a role.
|
From page 144... ...
144 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH each field according to its deeds and each according to its needs. Moreover, the interconnectedness argument applies to historical determinants and levels of support; it is of limited guidance in informing budget decision -- show much more or less, given existing levels of support?
|
From page 145... ...
APPENDIX C 145 dependent on the actions of agents outside the control of Federal agencies, to single or artificially aggregated measures; the substitution of bureaucratically and/or ideologically driven specification and utilization of selective measures for the independent judgment of experts; and the distortion of incentives for science managers and scientists that reduces the overall performance of public investments. To all these limitations must be added that to date there is little publically verifiable evidence outside the workings of OMB-agency negotiations that implementation of a system of performance measurement has appreciably improved decision making with respect to the magnitude or allocation of Federal research funds.
|
From page 146... ...
146 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH correct to state say that none of the proximate intermediate output measures, patents or publications for example, are good predictors of the ultimate impacts that one is seeking–increased per capita income, improved health–some such measures are essential to informed decision making. Adding impetus to this line of reasoning is that the environment in which U.S.
|
From page 147... ...
APPENDIX C 147 Given the above recitation of promises and limitations, the optimal course of action seems to be what Feuer and Maranto have termed science advice as procedural rationality (2010)
|
From page 148... ...
148 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH Boroush, M
|
From page 149... ...
APPENDIX C 149 Feller, I
|
From page 150... ...
150 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH Heisey, P., King, K., Rubenstein, K., Bucks, D., and Welsh, R.2010. Assessing the Benefits of Public Research Within an Economic Framework: The Case of USDA's Agricultural Research Service.
|
From page 151... ...
APPENDIX C 151 Massachusetts Institute of Technology (2011) The Third Revolution: The Convergence of the Life Sciences, Physical Sciences, and Engineering.
|
From page 152... ...
152 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH Rosenberg, N
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.