Measuring the Impacts of
Federal Investments in
Research

A WORKSHOP SUMMARY

Steve Olson and Stephen Merrill, Rapporteurs

Committee on Measuring Economic and Other Returns
on
Federal Research Investments

Board on Science, Technology, and Economic Policy
Committee on Science, Engineering, and Public Policy

Policy and Global Affairs

THE NATIONAL ACADEMIES

THE NATIONAL ACADEMIES PRESS
Washington, D.C.
www.nap.edu



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
Steve Olson and Stephen Merrill, Rapporteurs Committee on Measuring Economic and Other Returns on Federal Research Investments Board on Science, Technology, and Economic Policy Committee on Science, Engineering, and Public Policy Policy and Global Affairs

OCR for page R1
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This study was supported by Contract/Grant No. SMA-1019816 between the National Academy of Sciences and the National Science Foundation; Contract/Grant No. N01-OD-4-2139, TO #231, between the National Academy of Sciences and the National Institutes of Health; Contract/Grant No. G104P00159 between the National Academy of Sciences and the U.S. Geological Survey; Contract/Grant No. 59-9000-0-0093 between the National Academy of Sciences and the U.S. Department of Agriculture; Contract/Grant No. EP-11-H-001414 between the National Academy of Sciences and the Environmental Protection Agency; Contract/Grant No. DE-SC000614 between the National Academy of Sciences and the Department of Energy; Contract/Grant No. NNH10CC488,TO #5, between the National Academy of Sciences and NASA. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project International Standard Book Number -13:978-0-309-21748-4 International Standard Book Number -10:0-309-21748-2 Additional copies of this report are available from the National Academies Press, 500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624- 6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu. Cover: The cover design incorporates a feature of the 1924 National Academy of Sciences building in Washington. Sculpted by Lee Lawrie, the bronze cheneau, running the length of the roof, features alternating figures of owls and lynxes, symbolizing wisdom and alert observation, respectively. Copyright 2011 by the National Academy of Sciences. All rights reserved. Printed in the United States of America

OCR for page R1
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council. www.national-academies.org

OCR for page R1

OCR for page R1
COMMITTEE ON MEASURING ECONOMIC AND OTHER RETURNS ON FEDERAL RESEARCH INVESTMENTS NEAL LANE (Co-Chair), Malcolm Gillis University Professor, Rice University BRONWYN HALL (Co-Chair), Professor of Economics, University of California at Berkeley and University of Maastricht ALAN GARBER, Henry J. Kaiser, Jr. Professor and Professor of Medicine; Director, Center for Health Policy, Stanford University PAULA STEPHAN, Professor of Economics, Georgia State University PRABHU PINGALI, Deputy Director, Agricultural Development, Global Development Program, The Bill and Melinda Gates Foundation WALTER POWELL, Professor of Education, Stanford University and External Professor, The Santa Fe Institute DAVID GOLDSTON, Director, Government Affairs, Natural Resources Defense Council ALEXANDER FRIEDMAN, Chief Investment Officer, UBS Wealth Management JOHN STASKO, Professor and Associate Chair, School of Interactive Computing, Georgia Institute of Technology ALFRED SPECTOR, Vice President of Research and Special Initiatives, Google, Inc. ERIC WARD, President, The Two Blades Foundation NEELA PATEL, Director of External Research, Global Pharmaceutical R and D, Abbott Laboratories MICHAEL TURNER, Bruce V. and Diana M. Rauner Distinguished Service Professor, Kavli Institute for Cosmological Physics, The University of Chicago Staff STEPHEN A. MERRILL, Project Director GURUPRASAD MADHAVAN, Program Officer and Project Co- director KEVIN FINNERAN, Director, Committee on Science, Engineering, and Public Policy v

OCR for page R1
STEVE OLSON, Consultant Writer DANIEL MULLINS, Program Associate CYNTHIA GETNER, Financial Associate vi

OCR for page R1
BOARD ON SCIENCE, TECHNOLOGY, AND ECONOMIC POLICY National Research Council PAUL JOSKOW (Chair), President, Alfred P. Sloan Foundation LEWIS COLEMAN, President, DreamWorks Animation JOHN DONOVAN, Chief Technology Officer, AT and T ALAN GARBER, Henry J. Kaiser, Jr. Professor and Professor of Medicine; Director, Center for Health Policy, Stanford University RALPH GOMORY, President Emeritus, Alfred P. Sloan Foundation MARY GOOD, Donaghey University Professor and Dean Emeritus, Donaghey College of Information Science and Systems Engineering, University of Arkansas at Little Rock RICHARD LESTER, Professor and Department Head, Department of Nuclear Science and Engineering, Massachusetts Institute of Technology AMORY HOUGHTON, Jr., Former Member of Congress DAVID MORGENTHALER, Founding Partner, Morgenthaler Ventures WILLIAM MEEHAN, Lecturer in Strategic Management and Raccoon Partners Lecturer in Management, Stanford Graduate School of Business; and Director Emeritus, McKinsey and Company JOSEPH NEWHOUSE, John D. MacArthur Professor of Health Policy and Management and Director, Division of Health Policy Research and Education, Harvard University EDWARD PENHOET, Director, Alta Partners ARATI PRABHAKAR, General Partner, U.S. Venture Partners WILLIAM RADUCHEL, Strategic Advisor and Independent Director KATHYRN SHAW, Earnest C. Arbuckle Professor of Economics, Graduate School of Business, Stanford University LAURA D'ANDREA TYSON, S.K. and Angela Chan Professor of Global Management, Haas School of Business, University of California, Berkeley HAL VARIAN, Chief Economist, Google, Inc. ALAN WM. WOLFF, Of Counsel, Dewey and LeBoeuf LLP vii

OCR for page R1
Ex-Officio Members RALPH CICERONE, President, National Academy of Sciences CHARLES VEST, President, National Academy of Engineering HARVEY FINEBERG, President, Institute of Medicine Staff STEPHEN A. MERRILL, Executive Director CHARLES WESSNER, Program Director SUJAI SHIVAKUMAR, Senior Program Officer DAVID DIERKSHEIDE, Program Officer MCALISTER CLABAUGH, Program Officer PAUL BEATON, Program Officer CYNTHIA GETNER, Financial Associate DANIEL MULLINS, Program Associate DAVID DAWSON, Program Associate viii

OCR for page R1
COMMITTEE ON SCIENCE, ENGINEERING, AND PUBLIC POLICY National Academy of Sciences National Academy of Engineering Institute of Medicine GEORGE WHITESIDES (Chair), Woodford L. and Ann A. Flowers University Professor, Harvard University LINDA ABRIOLA, Dean of Engineering, Tufts University CLAUDE CANIZARES, Vice President for Research, Associate Provost and Bruno Rossi Professor of Experimental Physics, Massachusetts Institute of Technology MOSES CHAN, Evan Pugh Professor of Physics, Pennsylvania State University RALPH CICERONE (Ex-Officio), President, National Academy of Sciences PAUL CITRON, Retired Vice President, Technology Policy and Academic Relations, Medtronic, Inc. RUTH DAVID, President and Chief Executive Officer, ANSER (Analytic Services), Inc. HARVEY FINEBERG (Ex-Officio), President, Institute of Medicine JUDITH KIMBLE, Investigator, Howard Hughes Medical Institute; Professor of Biochemistry and Molecular Biology and Medical Genetics, University of Wisconsin DAN MOTE, Jr. (Ex-Officio), President and Glenn Martin Institute Professor of Engineering, University of Maryland PERCY PIERRE, Vice President and Professor Emeritus, Michigan State University ALBERT REECE, Vice President for Medical Affairs, Bowers Distinguished Professor and Dean, School of Medicine, University of Maryland, Baltimore SUSAN SCRIMSHAW, President, The Sage Colleges WILLIAM SPENCER, Chairman Emeritus, SEMATECH MICHAEL TURNER, Bruce V. and Diana M. Rauner Distinguished Service Professor, Kavli Institute for Cosmological Physics, The University of Chicago ix

OCR for page R1
CHARLES VEST (Ex-Officio), President, National Academy of Engineering NANCY WEXLER, Higgins Professor of Neuropsychology, Columbia University Staff KEVIN FINNERAN, Director THOMAS ARRISON, Senior Program Officer GURUPRASAD MADHAVAN, Program Officer PETER HUNSBERGER, Financial Associate MARION RAMSEY, Administrative Associate NEERAJ GORKHALY, Research Associate x

OCR for page R1
ACKNOWLEDGMENT OF REVIEWERS This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Academies’ Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for quality and objectivity. The review comments and draft manuscript remain confidential to protect the integrity of the process. We wish to thank the following individuals for their review of this report: George Bo-Linn, Gordon and Betty Moore Foundation; Susan Cozzens, Georgia Institute of Technology; Kenneth Gertz, University of Maryland; Diana Hicks, Georgia Institute of Technology; and Peter Hussey, RAND Corporation. Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the content of the report, nor did they see the final draft before its release. Responsibility for the final content of this report rests entirely with the rapporteurs and the institution. xi

OCR for page R1

OCR for page R1
CONTENTS 1 INTRODUCTION AND OVERVIEW ............................................... 1 2 THE USES AND MISUSES OF PERFORMANCE MEASURES..... 7 The Promise and the Limits of Measuring the Impact of Federally Supported Research ........................................................................ 7 Innovation as an Ecosystem............................................................... 11 Overcoming the Challenges of Research Measures .......................... 14 Discussion .......................................................................................... 16 3 IMPACTS ON THE U.S. ECONOMY AND QUALITY OF LIFE.. 19 Federal Research and Productivity .................................................... 19 Indirect Economic Benefits of Research ........................................... 21 Beyond Citations and Patent Reference Counts ................................ 22 Discussion .......................................................................................... 23 4 IMPACTS ON BIOMEDICAL AND HEALTH RESEARCH ......... 25 Reviewing the Literature on Health Impacts ..................................... 25 The Volatility of Federal R and D Support ....................................... 29 Medical Device Innovation................................................................ 30 Making Decisions in the Pharmaceutical Industry ............................ 31 Research and Outcomes Case Study: Pediatric HIV ......................... 33 Discussion .......................................................................................... 35 5 MIXED MARKET AND NON-MARKET IMPACTS OF RESEARCH ...................................................................................... 37 Measuring Progress toward Goals in Agricultural Productivity ........ 37 Investment Decisions at DuPont........................................................ 39 Challenges in Quantifying Research Value in Agriculture ............... 40 Measuring Success in Conservation .................................................. 42 National Security Benefits ................................................................. 44 Public Problem Solving ..................................................................... 46 Discussion .......................................................................................... 47 6 IMPACTS OF RESEARCH ON THE LABOR MARKET AND CAREER DEVELOPMENT ............................................................. 49 R and D Spending and the R and D Workforce ................................. 49 Surveys of Graduate Students and Postdoctoral Fellows .................. 51 xiii

OCR for page R1
The Complex Network of Skills and Investments ............................. 55 Discussion .......................................................................................... 59 7 INTERNATIONAL PERSPECTIVES ON MEASURING RESEARCH IMPACTS .................................................................... 61 Medical Research Council Evaluation System .................................. 61 Measuring Impacts of Research Funding In the European Union ..... 63 Measuring Impacts of Science, Technology, and Innovation Investments in Brazil .................................................................... 66 Discussion .......................................................................................... 67 8 EMERGING METRICS AND MODELS ......................................... 69 Assessing Research at NSF ............................................................... 69 The STAR Metrics Project ................................................................ 72 Reconstructing Networks of Discovery ............................................. 75 Creating Knowledge from Data ......................................................... 76 Measuring the Impact of Star Scientists ............................................ 78 Visual Analytics ................................................................................ 79 Considerations in Building Comprehensive Databases ..................... 81 Discussion .......................................................................................... 82 9 PITFALLS, PROGRESS, AND OPPORTUNITIES ......................... 85 Pitfalls on the Road to Understanding ............................................... 85 Progress in Understanding the Issues ................................................ 86 Opportunities Posed by Greater Understanding ................................ 87 Concluding Remarks ......................................................................... 88 REFERENCES ....................................................................................... 89 A WORKSHOP AGENDA .................................................................. 91 B BIOGRAPHICAL INFORMATION ................................................ 99 C THE PROMISES AND LIMITATIONS OF PERFORMANCE MEASURES, Irwin Feller ............................................................... 119 D THE IMPACT OF PUBLICLY FUNDED BIOMEDICAL AND HEALTH RESEARCH: A REVIEW, Bhaven Sampat ................... 153 xiv