National Academies Press: OpenBook
« Previous: 4 Findings
Suggested Citation:"5 Discussion." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2000. Experiments in International Benchmarking of US Research Fields. Washington, DC: The National Academies Press. doi: 10.17226/9784.
×

5
DISCUSSION

COSEPUP identified several key strengths, weaknesses, and other factors that influence the success of international benchmarking in evaluating research.

5.1 Some Strengths of Benchmarking

International benchmarking produces information that is, in the opinion of COSEPUP, valuable and relevant to researchers, administrators, and policy-makers. Three examples are the heavy reliance of mathematics leadership on foreign talent, the 20- to 30-year age difference between materials equipment in the US and that in several other countries, and the influence of managed care on clinical research. Although most information did not contradict prevailing views, it is unlikely that the results of each report could have been achieved as efficiently by any other technique, given the paucity of the data and information required for a traditional quantitative approach.

International benchmarking is rapid and inexpensive compared with procedures that rely entirely on the assembly of quantitative information. The use of qualitative judgments also has merit. In the words of one panel chair, the panels were able to get "80% of the value in 20% of the time" for a far lower cost.

5.2 Some Weaknesses of Benchmarking

In retrospect, the experiments revealed several methodologic weaknesses that can be addressed in future benchmarking activities. For example, non-US members should be included in the oversight group that selects the panels. The same features that make a virtual-congress

Suggested Citation:"5 Discussion." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2000. Experiments in International Benchmarking of US Research Fields. Washington, DC: The National Academies Press. doi: 10.17226/9784.
×

approach effective also expose such weaknesses as the potential for a bias that depends on the citizenship of the panelists who gather data for analysis. This increase the importance of including substantial proportions of non-US participants in all panels.

Multidisciplinary fields like materials science and engineering and immunology pose special challenges. For example, the immunology panel had to extract data from collaborative and international research; had to compare large enterprises with multiple smaller ones; and had to extract information on the specific field of immunology from related research fields in large, aggregate databases.

5.3 Other Observations about Benchmarking

The method by which the most important fields and subfields are identified is critical. For example, immunology is not considered a "discipline" in the traditional sense and does not have departmental status in most universities. The selection of subfields is a somewhat subjective process that might differ between one benchmarking exercise. Rather than being a drawback, however, such differences will reflect the continual shifting of the borders of modern fields. A field should be considered by the array of related domains between which investigators can move without leaving the realm of their expertise.

It is likely that benchmarking could be effective on a 3- or 5-year cycle because large fields of research change relatively slowly. Annual benchmarking probably would not be sensitive enough to reveal changes.

Our series of experiments has revealed that no benchmarking technique is sufficient by itself and that the utility of particular techniques varies by field. Therefore, each panel should use a variety of comparable qualitative and quantitative methods to afford cross-verification of results. The methods should be kept as independent as possible.

Because the accuracy of benchmarking depends heavily on panel members' personal knowledge of fields, panel members were more closely involved with the writing of the report than is frequently the case with committee-written reports.

Use of indicators that provide information on degree of uncertainty and reliability might enhance the presentation of the panel assessments of leadership status.

The extensive use of benchmarking would be enhanced by reliable, up-to-date information. The US field-specific data that are collected do not provide sufficient or timely information; non-US data are even more problematic.

A finding that the United States is the world leader in a research field might lead some to conclude that additional resources for that field are not warranted. This might or might not be the case. For example,

Suggested Citation:"5 Discussion." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2000. Experiments in International Benchmarking of US Research Fields. Washington, DC: The National Academies Press. doi: 10.17226/9784.
×

the mathematics report indicates that the United States is the world leader in mathematics. If the mathematics community requests additional resources, some policymakers might question the request on the grounds that the United States is already the world leader in that field. That concern has been expressed in connection with the life sciences. However, as the mathematics panel indicated in its report, the United States could drop from being ''the world leader" in mathematics research unless additional investments were made in some key subfields and unless more US students chose to enter the field. Thus, an assessment that the United States is the leader in a field does not necessarily imply that no additional resources are needed for the field.

Suggested Citation:"5 Discussion." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2000. Experiments in International Benchmarking of US Research Fields. Washington, DC: The National Academies Press. doi: 10.17226/9784.
×
Page 24
Suggested Citation:"5 Discussion." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2000. Experiments in International Benchmarking of US Research Fields. Washington, DC: The National Academies Press. doi: 10.17226/9784.
×
Page 25
Suggested Citation:"5 Discussion." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2000. Experiments in International Benchmarking of US Research Fields. Washington, DC: The National Academies Press. doi: 10.17226/9784.
×
Page 26
Next: 6 Conclusion »
Experiments in International Benchmarking of US Research Fields Get This Book
×
Buy Paperback | $87.00 Buy Ebook | $69.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

How can the federal government gauge the overall health of scientific research—as a whole and in its parts—and determine whether national funding adequately supports national research objectives? It is feasible to monitor US performance with field-by-field peer assessments. This might be done through the establishment of independent panels consisting of researchers who work in a field, individuals who work in closely related fields, and research "users" who follow the field closely. Some of these individuals should be outstanding foreign scientists in the field being examined. This technique of comparative international assessments is also known as international benchmarking.

Experiments in International Benchmarking of U.S. Research Fields evaluates the feasibility and utility of the benchmarking technique. In order to do this, the report internationally benchmarks three fields: mathematics, immunology, and materials science and engineering, then summarizes the results of these experiments.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!