EXECUTIVE SUMMARY
In its report Science, Technology, and the Federal Government: National Goals for a New Era,1 the Committee on Science, Engineering, and Public Policy (COSEPUP) of The National Academies recommended that the status of research fields might be evaluated by the use of international benchmarking. COSEPUP was referring to a technique that is designed to compare one country's or region's leadership status in research with world standards of excellence.
To assess the feasibility and utility of international benchmarking, COSEPUP proposed a set of experiments in three fields: mathematics, immunology, and materials science and engineering. The results of the experiments suggest that research leadership status by field can be assessed in a timely fashion at reasonable cost. International benchmarking has potential utility for policy-makers and federal agencies in determining the leadership status of the United States in particular fields, subfields, and possibly sub-subfields. It might also help federal agencies to comply with the Government Performance and Results Act by evaluating the quality of their own performance.
Committee members note especially the following points:
-
Panel leaders began the process as skeptics but came to believe that benchmarking can be reasonably objective and timely.
-
Two separate benchmarking experiments in mathematics—one at the National Academies, the other at the National Science Founda-
-
tion—despite dissimilar panels and mandates produced similar results, lending credence to the technique.
During the experiments, committee members were able to identify several particular strengths of international benchmarking:
-
Panels were able to identify institutional and human-resource factors crucial to maintaining leadership status in a field that is unlikely to have been identified by other methods.
-
Benchmarking allows a panel to determine the best measures for a particular field while providing corroboration through the use of different methods, as opposed to the "one-size-fits-all" approach of some common evaluation methods.
-
Benchmarking can produce a timely but broadly accurate "snapshot" of a field.
The experiments were sufficiently thorough to provide guidelines for future experiments, including the following:
-
Because of the panels' use of expert judgment, the choice of panelists is a key to the credibility of the results. A tendency toward national biases can be mitigated by ensuring diverse geographic membership of panels; the same is true of the groups that select the panel members. In particular, it is critical to include non-US participants in the selection of panelists and as panel members because they provide perspective and objectivity.
-
Because major fields of research change slowly, benchmarking can probably detect important changes in quality, relevance, and leadership in fields when conducted at intervals of 3-5 years. It is unlikely that changes can be detected by annual benchmarking.
-
The choice of research fields to be evaluated is both challenging and critical. A "field" might best be considered the array of related domains between which investigators can move without leaving their primary area of expertise.
-
Benchmarking produces information that administrators, policy-makers, and funding agencies find useful as they make decisions as to what activities a federal research program should undertake and respond to demands for accountability, such as the Government Performance and Results Act.
-
If federal agencies use benchmarking, the wide variation in agency missions dictates that each agency tailor the technique to its own needs.
-
Use of indicators that provide information on degree of uncertainty and reliability of benchmarking results might enhance the presentation of panel assessments of leadership status.
The present experiments should be regarded as an encouraging first step toward the design of an efficient and reliable evaluation tool. An accumulation of benchmarking exercises will lead to more-effective methods, better understanding of the factors that promote research excellence, and better decision-making by funders of science and technology.
