Calls for assessing the state of scientific fields are not new. Indeed, we are witnessing a period of increased introspection, especially with respect to evaluating the effectiveness of federal investments in research and development. Significant recent examples of this emphasis include the 1991 report of the Office of Technology Assessment, Federally Funded Research: Decisions for a Decade, and the Government Performance and Results Act of 1993 (PL 103-62). The quandaries are many. Who should do the assessing, and what criteria are to be used? What should be measured, and how do these measurements connect to desired outcomes? How are scientists to be held accountable for what they do?

The scientific community, principally through the National Research Council, has issued major studies of the status and directions of scientific fields and subfields in the past. Priorities in terms of the most promising research opportunities have been identified and have led in many cases to enhanced federal investments in these areas and reductions in areas judged less promising. Left undone, however, is a response to a central issue of scientific policy, namely, the setting of priorities across fields. As yet, the scientific community has not found a satisfactory method for accomplishing this task, although cross-priorities are of course being set by the federal political and budgetary process.

The Commission on Physical Sciences, Mathematics, and Applications confronted this issue of determining cross-priorities and judged that what seemed potentially promising was to assess the capacity of a field to retain its strength, adapt to new opportunities, connect with other fields, and the like—a set of criteria labeled as the “health” of a field or subfield. The hope was that if common criteria for assessment could be discovered and validated, then a comparison among different fields would guide the scientific community as well as the patrons of that community in setting priorities.

As our study has demonstrated, however, “health” assessments are harder to make than we initially thought. For example, it is easy to posit that what is bigger is better, but when the bigger must become smaller, the affected community often will marshal measures to show that the health of the field is deteriorating when the contrary may actually be the case. Past experience teaches us, for example, that a

small fraction of the workers in any given field produce the majority of output of value. Increasing the number of workers in a field may thus actually reduce the average output per worker. It is seductive to imagine that a set of metrics exist that, when tracked over time, will enable predicting with confidence the future prospects of a scientific field or subfield. Whereas some indicators may be useful for judging the normal pace of progress in a field as it follows its established paradigm, no evidence could be found that suggests that such statistical measures have long-range predictive power. How can we know whether some supposedly dull field in stasis or even in decline is on the verge of a remarkable rebirth that will result from an unexpected experimental discovery or from some new and vital idea coming from the work of just one individual?

We must not let a cycle be created in which the need for accountability leads to the use of measurement standards, which leads to polishing existing paradigms, which leads to further demands for accountability, and so on, with the effect that this cycle drives out investment in possible paradigm shifts. Put another way, we must avoid “looking for the lost key where the light is brightest”: we must avoid allowing what can be measured to become what matters, rather than seeking to measure what matters—which frequently are attributes that cannot be quantified.

A means must be found to judge when an incremental investment in a field is likely to yield only a minimal return in polishing the edges of an existing mode of research and when it will lead to ground-breaking advances. Thus, it must be granted that discipline assessments require both quantitative as well as qualitative measures, which the Commission believed when it set out to test its proposal for assessing the health of a field, and has now confirmed through this experiment.

In initiating this study, the Commission proposed a set of candidate indicators of “health” and established three discipline panels to review them in the areas of mathematics, astronomy, and atomic, molecular, and optical science. This report summarizes the principal lessons learned from this experiment.

I believe the study substantiates what many already knew intuitively. A large chasm exists between asking for performance measures or other indicators of the status of a discipline, and getting ones that are reliable and usable. Moreover, often unanticipated difficulties arise concerning the availability of data, the stability over time of ostensibly the same databases, efforts to compare what seem to be the same data, albeit collected by different groups or agencies, and efforts to interpret these data.

In some ways this experiment may be regarded as reporting negative results. It also makes no recommendations. In other ways, however, the Commission hopes that this report will be regarded as positive in that it indicates that a premium should be placed on wise judgment by informed individuals. The Commission believes that the findings should be reported and discussed as both a warning and a guidepost for future efforts to judge the effectiveness of public investments in science.

Richard N. Zare

Chair, Commission on Physical

Sciences, Mathematics, and Applications

The Commission on Physical Sciences, Mathematics, and Applications is grateful to the many individuals who participated in this pilot project, including the members of the three ad hoc panels that assessed the Commission's proposed indicators of the health of a discipline and related data sources; the participants in the Commission's August 1993 workshop, who helped integrate the results of the panels' work; and members of the National Research Council (NRC) staff who worked on various aspects of the study.

The Mathematics Assessment Panel was chaired by Shmuel Winograd of the IBM T.J. Watson Research Center. The panel members were Michael Artin, Massachusetts Institute of Technology; Ramesh Gangolli, University of Washington; Ramanathan Gnanadesikan, Rutgers University; Deborah Tepper Haimo, University of Missouri at St. Louis; Richard Herman, University of Maryland; George Nemhauser, Georgia Institute of Technology; Bruce Spencer, Northwestern University; and Mary Wheeler, Rice University.

Arthur Code, of the University of Wisconsin, chaired the Astronomy Assessment Panel. Peter Boyce, American Astronomical Society; Robert Brown, Space Telescope Science Institute; Claude Canizares, Massachusetts Institute of Technology; Frank Drake, University of California at Santa Cruz; Andrea Dupree, Harvard-Smithsonian Center for Astrophysics; Richard Lingenfelter, University of California at San Diego; Richard McCray, Joint Institute Laboratory for Astrophysics; Robert Rosner, University of Chicago; and Harry Shipman, University of Delaware, were members of this panel.

The Atomic, Molecular, and Optical (AMO) Science Assessment Panel was chaired by F. Barrymore Dunning of Rice University. Panel members were Lloyd Armstrong, Jr., University of Southern California; Alexander Dalgarno, Harvard-Smithsonian Center for Astrophysics; Elsa Garmire, University of Southern California; Daniel Larson, University of Virginia; Paul Foo-Hung Liao, Bellcore; Thomas McIlrath, University of Maryland; Ronald Phaneuf, University of Nevada; and Roberta Saxon, SRI International.

The Commission also would like to extend its thanks to the participants of the Workshop on the Assessment of the Health of the Physical and Mathematical Sci-

ences, which was held in Washington, D.C., in August 1993: Richard Nicholson (Chair), American Association for the Advancement of Science; John Andelin, Office of Technology Assessment (retired); Darrell Beschen, Department of Energy; Jennifer Bond, National Science Foundation; Joseph Clark, Department of Commerce; Arthur Code, University of Wisconsin; Alan Fechter, NRC's Office of Scientific and Engineering Personnel; Barry Gold, House Committee on Science, Space, and Technology; Marcel LaFollette, George Washington University; Daniel Larson, University of Virginia; James McCullough, National Science Foundation; Jerome Sacks, National Institute of Statistical Sciences; and Shmuel Winograd, IBM T.J. Watson Research Center.

Finally, the Commission is grateful to the staff of the National Research Council: Paul F. Uhlir, associate executive director of the Commission, who served as study director and workshop organizer; Hans Oser, Board on Mathematical Sciences, and Robert Riemer, Ronald Taylor, and Anne Simmons, Board on Physics and Astronomy, who provided staff support for the three panels; Julie Esanu and Jennifer Young, for the program assistance they provided to the panels and for the preparation of the panels' reports and final manuscript; Alan Fechter, Pamela Ebert Flattau, Susan Mitchell, Prudence Brown, Daniel Pasquini, George Orvis, and George Boyce, Office of Scientific and Engineering Personnel, who provided advice and support during the course of the study; and Susan Maurizi, who edited the final report.