National Academies Press: OpenBook

Measuring the Impacts of Federal Investments in Research: A Workshop Summary (2011)

Chapter: 9 PITFALLS, PROGRESS, AND OPPORTUNITIES

« Previous: 8 EMERGING METRICS AND MODELS
Suggested Citation:"9 PITFALLS, PROGRESS, AND OPPORTUNITIES." National Research Council. 2011. Measuring the Impacts of Federal Investments in Research: A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13208.
×

9

PITFALLS, PROGRESS, AND OPPORTUNITIES

During the final session of the meeting, members of a panel shared their opinions of the major messages and unanswered questions that emerged from the two days of presentations and discussions. Theirs were individual observations rather than an expression of a consensus on the part of the panel or the workshop participants as a whole.

PITFALLS ON THE ROAD TO UNDERSTANDING

  • The selection of specific measures inevitably focuses attention and effort on what is being measured. Their value can decay as more of what is being measured is produced. Further, the selection of metrics can reduce the valuable diversity of the research system and its potential for serendipity.
  • Agricultural research has been so successful that Americans now pay less for food per capita than in almost any other country and any other time. This success may have had the perverse effect of undermining funding for basic agricultural science, since the need for productivity gains seems less pressing.
  • Research funding volatility has major consequences for the decisions made by research performers. For example, the doubling of the NIH budget drove a large expansion of biomedical research facilities at research universities in the expectation that increases would continue. The suspension of real growth at NIH halted the growth of indirect cost recovery to pay for those buildings, with adverse effects for other parts of the university. Funding patterns also send messages to students about desirable fields of research – messages that may be at odds with long-term employment prospects in those fields. Volatility is problematic in firms as well as in federal research.
Suggested Citation:"9 PITFALLS, PROGRESS, AND OPPORTUNITIES." National Research Council. 2011. Measuring the Impacts of Federal Investments in Research: A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13208.
×
  • An assessment of even a narrow field requires taking an average from disparate processes and systems, which can cause such assessments to be overly broad. For example, the number of patents granted within a particular field may be important, but individual researchers should not be judged by how many patents each one has generated.
  • If all past research had been required to justify its value in terms of practical benefits, advances that have led to massive practical benefits would not have occurred.
  • The knowledge generated by fundamental research has an intrinsic value regardless of its application. Without it, applied work would stagnate.
  • Policymakers and the public in general agree on the value of research. Could research that fails to identify many of the benefits of science undermine that consensus and therefore be harmful?

PROGRESS IN UNDERSTANDING THE ISSUES

  • The ever-growing power of the Web and the information sharing it enables will facilitate the analysis of research outputs. Natural language processing, machine learning technologies, and crowd sourcing will increasingly glean many reasonably accurate metrics from publications, patents, social networks, blogs, and so forth, and this capability will increase over time. Furthermore, this approach will be less costly and provide more information than government-mandated reporting. However, government agencies will need to create new tools to use these data to help fulfill their missions.
  • The benefits of research results, both in terms of new knowledge and trained students, are vastly different from discipline to discipline and even from subdiscipline to subdiscipline. Thus, the determination of impacts requires very detailed analysis that is highly sector specific. For example, the evaluation of physics is different than the evaluation of computer science, and the evaluation of theoretical computer science is different than the evaluation of research in parallel computation.
Suggested Citation:"9 PITFALLS, PROGRESS, AND OPPORTUNITIES." National Research Council. 2011. Measuring the Impacts of Federal Investments in Research: A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13208.
×
  • Relatively small expenditures on increasing the dissemination of research results could greatly enhance the beneficial impacts of research without entailing major new funding initiatives.
  • Some questions may not be answerable, and identifying those questions may usefully focus attention on the questions that can be answered.
  • The major discoveries that result from basic research are outliers that generally are very hard to predict. They emerge rarely, but they are the most important. How can these be accommodated in assessments of the value of research?
  • As science becomes more interdisciplinary, more collaborative, more international, more digital, more open, more expensive, more diverse, and more fast-paced, measuring impacts will face new and difficult challenges.

OPPORTUNITIES POSED BY GREATER UNDERSTANDING

  • The science of science policy has an opportunity to examine the broader issues of economic growth and societal change if it interprets its agenda broadly. As an example of an important albeit difficult question, are additional funds most usefully spent on health-related R and D or on insurance? Some analysts have cited the drop in deaths from cardiovascular disease starting in 1965 as an outcome of biomedical research, but that was also the year when Medicare was instituted.
  • The plural of anecdotes may not be data, but anecdotes can be more powerful than data in swaying policymakers, even if they are not necessarily representative.
  • A heightened emphasis on accountability within government will increase the need to produce metrics of research impacts. The research community needs to understand why this is important, especially because they can contribute ideas that would benefit data collection and analysis.
  • Research funders and performers have many opportunities to work with the private sector in measuring the impacts of research, since the private sector spends considerable time and money working on this issue.
Suggested Citation:"9 PITFALLS, PROGRESS, AND OPPORTUNITIES." National Research Council. 2011. Measuring the Impacts of Federal Investments in Research: A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13208.
×
  • The optimal amount of research for the United States as a percentage of GDP still has not been determined. Is it possible to overspend on R and D? To what extent should education be emphasized in that spending?

CONCLUDING REMARKS

Finally, several speakers on the panel emphasized that scholars studying these issues should be humble, sensitive, and do no harm, which is a message Irwin Feller delivered at the beginning of the workshop. The returns on federal investments in research are extremely complex and occur within the context of a complex economy and society. Analysts should avoid claiming more for the utility of their work than is warranted.

Suggested Citation:"9 PITFALLS, PROGRESS, AND OPPORTUNITIES." National Research Council. 2011. Measuring the Impacts of Federal Investments in Research: A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13208.
×
Page 85
Suggested Citation:"9 PITFALLS, PROGRESS, AND OPPORTUNITIES." National Research Council. 2011. Measuring the Impacts of Federal Investments in Research: A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13208.
×
Page 86
Suggested Citation:"9 PITFALLS, PROGRESS, AND OPPORTUNITIES." National Research Council. 2011. Measuring the Impacts of Federal Investments in Research: A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13208.
×
Page 87
Suggested Citation:"9 PITFALLS, PROGRESS, AND OPPORTUNITIES." National Research Council. 2011. Measuring the Impacts of Federal Investments in Research: A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13208.
×
Page 88
Next: REFERENCES »
Measuring the Impacts of Federal Investments in Research: A Workshop Summary Get This Book
×
Buy Paperback | $54.00 Buy Ebook | $43.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The enactment of the America COMPETES Act in 2006 (and its reauthorization in 2010), the increase in research expenditures under the 2009 American Recovery and Reinvestment Act (ARRA), and President Obama's general emphasis on the contribution of science and technology to economic growth have all heightened interest in the role of scientific and engineering research in creating jobs, generating innovative technologies, spawning new industries, improving health, and producing other economic and societal benefits. Along with this interest has come a renewed emphasis on a question that has been asked for decades: Can the impacts and practical benefits of research to society be measured either quantitatively or qualitatively?

On April 18-19, 2011, the Board on Science, Technology, and Economic Policy (STEP) and the Committee on Science, Engineering and Public Policy (COSEPUP) of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine, held a workshop to examine this question. The workshop sought to assemble the range of work that has been done in measuring research outcomes and to provide a forum to discuss its method. The workshop was motivated by a 2009 letter from Congressman Rush Holt (D-New Jersey). He asked the National Academies to look into a variety of complex and interconnected issues, such as the short-term and long-term economic and non-economic impact of federal research funding, factors that determine whether federally funded research discoveries result in economic benefits, and quantification of the impacts of research on national security, the environment, health, education, public welfare, and decision making.

Measuring the Impacts of Federal Investments in Research provides the key observations and suggestions made by the speakers at the workshop and during the discussions that followed the formal presentations.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!