Measuring the Impacts of
Federal Investments in
Research
A WORKSHOP SUMMARY
Steve Olson and Stephen Merrill, Rapporteurs
Committee on Measuring Economic and Other Returns
on
Federal Research Investments
Board on Science, Technology, and Economic Policy
Committee on Science, Engineering, and Public Policy
Policy and Global Affairs
THE NATIONAL ACADEMIES
THE NATIONAL ACADEMIES PRESS
Washington, D.C.
www.nap.edu
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study was supported by Contract/Grant No. SMA-1019816 between the National Academy of Sciences and the National Science Foundation; Contract/Grant No. N01-OD-4-2139, TO #231, between the National Academy of Sciences and the National Institutes of Health; Contract/Grant No. G104P00159 between the National Academy of Sciences and the U.S. Geological Survey; Contract/Grant No. 59-9000-0-0093 between the National Academy of Sciences and the U.S. Department of Agriculture; Contract/Grant No. EP-11-H-001414 between the National Academy of Sciences and the Environmental Protection Agency; Contract/Grant No. DE-SC000614 between the National Academy of Sciences and the Department of Energy; Contract/Grant No. NNH10CC488,TO #5, between the National Academy of Sciences and NASA. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project
International Standard Book Number -13:978-0-309-21748-4
International Standard Book Number -10:0-309-21748-2
Additional copies of this report are available from the National Academies Press, 500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu.
Cover: The cover design incorporates a feature of the 1924 National Academy of Sciences building in Washington. Sculpted by Lee Lawrie, the bronze cheneau, running the length of the roof, features alternating figures of owls and lynxes, symbolizing wisdom and alert observation, respectively.
Copyright 2011 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council.
This page intentionally left blank.
COMMITTEE ON MEASURING ECONOMIC AND OTHER RETURNS ON FEDERAL RESEARCH INVESTMENTS
NEAL LANE (Co-Chair), Malcolm Gillis University Professor, Rice University
BRONWYN HALL (Co-Chair), Professor of Economics, University of California at Berkeley and University of Maastricht
ALAN GARBER, Henry J. Kaiser, Jr. Professor and Professor of Medicine; Director, Center for Health Policy, Stanford University
PAULA STEPHAN, Professor of Economics, Georgia State University
PRABHU PINGALI, Deputy Director, Agricultural Development, Global Development Program, The Bill and Melinda Gates Foundation
WALTER POWELL, Professor of Education, Stanford University and External Professor, The Santa Fe Institute
DAVID GOLDSTON, Director, Government Affairs, Natural Resources Defense Council
ALEXANDER FRIEDMAN, Chief Investment Officer, UBS Wealth Management
JOHN STASKO, Professor and Associate Chair, School of Interactive Computing, Georgia Institute of Technology
ALFRED SPECTOR, Vice President of Research and Special Initiatives, Google, Inc.
ERIC WARD, President, The Two Blades Foundation
NEELA PATEL, Director of External Research, Global Pharmaceutical R and D, Abbott Laboratories
MICHAEL TURNER, Bruce V. and Diana M. Rauner Distinguished Service Professor, Kavli Institute for Cosmological Physics, The University of Chicago
Staff
STEPHEN A. MERRILL, Project Director
GURUPRASAD MADHAVAN, Program Officer and Project Co-director
KEVIN FINNERAN, Director, Committee on Science, Engineering, and Public Policy
STEVE OLSON, Consultant Writer
DANIEL MULLINS, Program Associate
CYNTHIA GETNER, Financial Associate
BOARD ON SCIENCE, TECHNOLOGY, AND ECONOMIC POLICY
National Research Council
PAUL JOSKOW (Chair), President, Alfred P. Sloan Foundation
LEWIS COLEMAN, President, DreamWorks Animation
JOHN DONOVAN, Chief Technology Officer, AT and T
ALAN GARBER, Henry J. Kaiser, Jr. Professor and Professor of Medicine; Director, Center for Health Policy, Stanford University
RALPH GOMORY, President Emeritus, Alfred P. Sloan Foundation
MARY GOOD, Donaghey University Professor and Dean Emeritus, Donaghey College of Information Science and Systems Engineering, University of Arkansas at Little Rock
RICHARD LESTER, Professor and Department Head, Department of Nuclear Science and Engineering, Massachusetts Institute of Technology
AMORY HOUGHTON, Jr., Former Member of Congress
DAVID MORGENTHALER, Founding Partner, Morgenthaler Ventures
WILLIAM MEEHAN, Lecturer in Strategic Management and Raccoon Partners Lecturer in Management, Stanford Graduate School of Business; and Director Emeritus, McKinsey and Company
JOSEPH NEWHOUSE, John D. MacArthur Professor of Health Policy and Management and Director, Division of Health Policy Research and Education, Harvard University
EDWARD PENHOET, Director, Alta Partners
ARATI PRABHAKAR, General Partner, U.S. Venture Partners
WILLIAM RADUCHEL, Strategic Advisor and Independent Director
KATHYRN SHAW, Earnest C. Arbuckle Professor of Economics, Graduate School of Business, Stanford University
LAURA D'ANDREA TYSON, S.K. and Angela Chan Professor of Global Management, Haas School of Business, University of California, Berkeley
HAL VARIAN, Chief Economist, Google, Inc.
ALAN WM. WOLFF, Of Counsel, Dewey and LeBoeuf LLP
Ex-Officio Members
RALPH CICERONE, President, National Academy of Sciences
CHARLES VEST, President, National Academy of Engineering
HARVEY FINEBERG, President, Institute of Medicine
Staff
STEPHEN A. MERRILL, Executive Director
CHARLES WESSNER, Program Director
SUJAI SHIVAKUMAR, Senior Program Officer
DAVID DIERKSHEIDE, Program Officer
MCALISTER CLABAUGH, Program Officer
PAUL BEATON, Program Officer
CYNTHIA GETNER, Financial Associate
DANIEL MULLINS, Program Associate
DAVID DAWSON, Program Associate
COMMITTEE ON SCIENCE, ENGINEERING, AND PUBLIC POLICY
National Academy of Sciences
National Academy of Engineering
Institute of Medicine
GEORGE WHITESIDES (Chair), Woodford L. and Ann A. Flowers University Professor, Harvard University
LINDA ABRIOLA, Dean of Engineering, Tufts University
CLAUDE CANIZARES, Vice President for Research, Associate Provost and Bruno Rossi Professor of Experimental Physics, Massachusetts Institute of Technology
MOSES CHAN, Evan Pugh Professor of Physics, Pennsylvania State University
RALPH CICERONE (Ex-Officio), President, National Academy of Sciences
PAUL CITRON, Retired Vice President, Technology Policy and Academic Relations, Medtronic, Inc.
RUTH DAVID, President and Chief Executive Officer, ANSER (Analytic Services), Inc.
HARVEY FINEBERG (Ex-Officio), President, Institute of Medicine
JUDITH KIMBLE, Investigator, Howard Hughes Medical Institute; Professor of Biochemistry and Molecular Biology and Medical Genetics, University of Wisconsin
DAN MOTE, Jr. (Ex-Officio), President and Glenn Martin Institute Professor of Engineering, University of Maryland
PERCY PIERRE, Vice President and Professor Emeritus, Michigan State University
ALBERT REECE, Vice President for Medical Affairs, Bowers Distinguished Professor and Dean, School of Medicine, University of Maryland, Baltimore
SUSAN SCRIMSHAW, President, The Sage Colleges
WILLIAM SPENCER, Chairman Emeritus, SEMATECH
MICHAEL TURNER, Bruce V. and Diana M. Rauner Distinguished Service Professor, Kavli Institute for Cosmological Physics, The University of Chicago
CHARLES VEST (Ex-Officio), President, National Academy of Engineering
NANCY WEXLER, Higgins Professor of Neuropsychology, Columbia University
Staff
KEVIN FINNERAN, Director
THOMAS ARRISON, Senior Program Officer
GURUPRASAD MADHAVAN, Program Officer
PETER HUNSBERGER, Financial Associate
MARION RAMSEY, Administrative Associate
NEERAJ GORKHALY, Research Associate
ACKNOWLEDGMENT OF REVIEWERS
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Academies’ Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for quality and objectivity. The review comments and draft manuscript remain confidential to protect the integrity of the process.
We wish to thank the following individuals for their review of this report: George Bo-Linn, Gordon and Betty Moore Foundation; Susan Cozzens, Georgia Institute of Technology; Kenneth Gertz, University of Maryland; Diana Hicks, Georgia Institute of Technology; and Peter Hussey, RAND Corporation.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the content of the report, nor did they see the final draft before its release. Responsibility for the final content of this report rests entirely with the rapporteurs and the institution.
This page intentionally left blank.
CONTENTS
2 THE USES AND MISUSES OF PERFORMANCE MEASURES
The Promise and the Limits of Measuring the Impact of Federally Supported Research
Overcoming the Challenges of Research Measures
3 IMPACTS ON THE U.S. ECONOMY AND QUALITY OF LIFE
Federal Research and Productivity
Indirect Economic Benefits of Research
Beyond Citations and Patent Reference Counts
4 IMPACTS ON BIOMEDICAL AND HEALTH RESEARCH
Reviewing the Literature on Health Impacts
The Volatility of Federal R and D Support
Making Decisions in the Pharmaceutical Industry
Research and Outcomes Case Study: Pediatric HIV
5 MIXED MARKET AND NON-MARKET IMPACTS OF RESEARCH
Measuring Progress toward Goals in Agricultural Productivity
Investment Decisions at DuPont
Challenges in Quantifying Research Value in Agriculture
Measuring Success in Conservation
6 IMPACTS OF RESEARCH ON THE LABOR MARKET AND CAREER DEVELOPMENT
The Complex Network of Skills and Investments
7 INTERNATIONAL PERSPECTIVES ON MEASURING RESEARCH IMPACTS
Medical Research Council Evaluation System
Measuring Impacts of Research Funding In the European Union
Measuring Impacts of Science, Technology, and Innovation Investments in Brazil
Reconstructing Networks of Discovery
Measuring the Impact of Star Scientists
Considerations in Building Comprehensive Databases
9 PITFALLS, PROGRESS, AND OPPORTUNITIES
Pitfalls on the Road to Understanding
Progress in Understanding the Issues
Opportunities Posed by Greater Understanding
C THE PROMISES AND LIMITATIONS OF PERFORMANCE MEASURES, Irwin Feller
D THE IMPACT OF PUBLICLY FUNDED BIOMEDICAL AND HEALTH RESEARCH: A REVIEW, Bhaven Sampat