Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
An Agenda for Improved Evaluation of Supercomputer Performance A Report Prepared by the Committee on Supercomputer Performance and Development Energy Engineering Board Commission on Engineering and Technical Systems National Research Council NATIONAL ACADEMY PRESS Washington, D.C. 1986
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This report has been reviewed by a group other than the authors according to procedures approved by a Report Review Committee consisting of members of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The National Research Council was established by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and of advising the federal government. The Council operates in accordance with general policies determined by the Academy under the authority of its congressional charter of 1863, which establishes the Academy as a private, nonprofit, self-governing membership corporation. The Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in the conduct-of their services to the government, the public, and the scientific and engineering communities. It is administered jointly by both Academies and the Institute of Medicine. The National Academy of Engineering and the Institute of Medicine were established in 1964 and 1970, respectively, under the charter of the National Academy of Sciences. This is a report of work supported by Grant No. DE-FG05-85ER25008, from the C.S. Department of Energy, and Grant No. N00014-85-G-0233, from the Office of Naval Research, U.S. Department of the Navy, to the National Academy of Sciences. Copies available from: Energy Engineering Board Commission on Engineering and Technical Systems National Research Council 2101 Constitution Avenue, N.W. Washington, D.C. 20418 Printed in the United States of America Second Printing
COMMITTEE ON SUPERCOMPUTER PERFORMANCE AND DEVELOPMENT E. F. INFANTE (Chairman), University of Minnesota - CLIFFORD N. ARNOLD, ETA Systems, Incorporated, St. Paul, Minnesota ROBERT R. BORCHERS, Lawrence Livermore National Laboratory, Livermore, California JAMES C. BROWNE, Department of Physics and Computer Science, University of Texas at Austin BILL L. BUZBEE, Los Alamos National Laboratory, Los Alamos, New Mexico ROBERT H. EWALD, Cray Research, Incorporated, Minneapolis, Minnesota SIDNEY FERNBACH, Alamo, California DUNCAN H. LAWRIE, Department of Computer Science, University of Illinois, Urbana, Illinois JOANNE L. MARTIN, International Business Machines Corporation, Yorktown Heights, New York VICTOR L. PETERSON, National Aeronautics and Space Administration, Moffett Field, California PAUL SCHNECK, Supercomputing Research Center, Lanham, Maryland JACK WORLTON, Los Alamos, New Mexico Liaison with Energy Engineering Board THELMA ESTRIN, School of Engineering and Applied Science, University of California at Los Angeles Staff DENNIS F. MILLER, Executive Director, Energy Engineering Board JOHN M. RICHARDSON, Study Director, Committee on Supercompute~ Performance and Development ROBERT COHEN, Senior Staff Officer, Committee on Supercomputer Performance and Development HELEN D. JOHNSON, Staff Associate, Energy Engineering Board CARLITA M. PERRY, Administrative Secretary, Committee on Supercomputer Performance and Development . . .
ENERGY ENGINEERING BOARD *HERBERT H. WOODSON, (Chairman), The University of Texas at Austin WILLIAM R. GOULD, (Chairman), Southern California Edison Company, Rosemead, California ALLAN J. BARD, The University of Texas at Austin *ROBERT J. BUDNITZ, Future Resources Associates, Inc., Berkeley, California THELMA ESTRIN, University of California at Los Angeles CHARLES F. GAY, Arco Solar, Chatsworth, California S. WILLIAM GOUSE, Mitre Corporation, McLean, Virginia NICHOLAS J. GRANT, Massachusetts Institute of Technology, Cambridge *BRUCE H. HANNON, University of Illinois at Urbana-Champaign, Urbana GARY H. HEICHEL, University of Minnesota, St. Paul JOSEPH M. HENDRIE, Brookhaven National Laboratory, Upton, New York WILLIAM W. HOGAN, Harvard University, Cambridge, Massachusetts BAINE P. KERR, Pennzoil Company, Houston, Texas HENRY R. LINDEN, Gas Research Institute, Chicago, Illinois EDWARD A. MASON, Amoco Research Center, Naperville, Illinois ALAN D. PASTERNAK, Energy Consultant, Sacramento, California THOMAS H. PIGFORD, University of California, Berkeley ADEL F. SAROFIM, Massachusetts Institute of Technology, MAXINE L. SAVITZ. The Garrett Corcoration. Los Anaeles. Cambridge , ~ California WESTON M. STACEY, JR., Georgia Institute of Technology, Atlanta RICHARD STEIN, The Stein Partnership, New York, New York THOMAS E. STELSON, Georgia Institute of Technology, Atlanta LEON STOCK, University of Chicago, Illinois GEORGE S. TOLLEY, University of Chicago, Illinois DAVID C. WHITE, Massachusetts Institute of Technology, Cambridge RICHARD WILSON, Harvard University, Cambridge, Massachusetts Technical Advisory Panel HAROLD M. AGNEW, GA Technologies, Inc., Solana Beach, California FLOYD L. CULLER, JR., Electric Power Research Institute, Palo Alto, California CHAUNCEY STARR, Electric Power Research Institute, Palo Alto, California ALBERT R.C. WESTWOOD, Martin Marietta Labs, Martin Marietta Corporation, Baltimore, Maryland Staff Dennis F. Miller, Executive Director, Energy Engineering Board Helen D. Johnson, Staff Associate Cheryl A. Winter, Staff Assistant *Term expired June 30, 1986
PREFACE This report presents the results of a brief study to assess current methods of evaluating supercomputer performance, identify opportunities for improvement, and recommend the outlines of a research agenda to realize these opportunities. A substantial scientific community and body of literature are devoted to computer performance measurement and evaluation. Impressive successes have been achieved in theory, prediction, and experimentation for systems that have sequential architecture and applications that involve data handling and manipulation. Less successful and mature are the methods and measurements for evaluation of system performance when scientific problems are run on supercomputers of diverse architectural characteristics. Performance evaluation is especially important for supercomputers, not merely because they are a costly resource but, more fundamentally, because they must often work at the limits of their performance to produce the results wanted. The complexity of these systems, together with the increasing diversity of computing architectures becoming available, makes evaluation of supercomputer performance more problematical, yet more important, than for the more common midrange computer systems. Moreover, what is learned about the performance of operational systems can be expected to guide the design of future systems. The U.S. Department of Energy and the Office of Naval Research, in response to these needs, arranged with the National Research Council to assess the improvements attainable in performance evaluation of large-scale scientific computers. On the basis of its own experience and its contacts with members of research and development community, the committee established for the purpose concluded that current performance measures and methods for supercomputer evaluation do indeed need significant improvement. Moreover, the committee judged that no well developed scientific foundation exists for supercomputer performance evaluation. In consequence, this report paints in broad strokes the outlines of this topic, its problems, and its possibilities. Filling in the details vii
of the panorama requires the successful accomplishment of research outlined in this report. In addition, it requires the help of a formal mechanism of the research community to track, assess, and disseminate research results with a view to bringing about commonly accepted methodologies of performance evaluation. The report is directed to persons in the federal agencies and elsewhere who fund research, and to the industrial and academic researchers who perform it. On behalf of the committee members I wish to voice our gratitude interest and support of Donald M. Austin, of the U.S. Department and Charles J. Holland, of the Office of Naval Researcn. . , , research results of for the Energy, Carl Died, of Cray Research; Troy L. Wilson, of International Business Machines Corporation; and the members of the Parallel Processing Group of the National Bureau of Standards generously shared their knowledge with members of the committee In the course of his review of the report manuscript, Peter J. Denning, of the Research Institute for Advanced Computer Science, provided the committee with valuable contributions to the topics presented here. The task was initiated by Dennis F. Miller, Executive Director of the Energy Engineering Board. The contributions to the committee's work by John M. Richardson and of the board's staff, are most gratefully acknowledged. Carlita M. Perry handled the manifold administrative tasks with grace, tact, and competence. Finally, I wish to thank the members of the committee for their extensive contributions of time and effort, and for their selfless sharing of knowledge and experience. I acknowledge all these contributions with sincere gratitude. Robert Cohen, of the board's . E. F. Infante viii Chairman committee on Supercomputer Performance and Development
CONTENTS SUMMARY 1 INTRODUCTION. The Problem / 9 Approach to the Study , / 11 PRINCIPLES FOR THE EVALUATION OF SUPERCOMPUTERS. The Context for Evaluation / 13 Criteria for Evaluation / 14 Matching Computational and Application Characteristics The Cycle of Performance Evaluation ~ 17 / 17 13 THE CURRENT STATE OF THE ART IN EVALUATION OF SUPERCOMPUTERS.. 21 Lessons from the Past / 21 Performance Metrics / 23 Test Programs for Performance Measurement / 24 Analytical Approach to Performance Evaluation Summary / 28 References / 29 / 27 4 IMPROVEMENTS ATTAINABLE IN PERFORMANCE EVALUATION 31 Models and Classification Schemes / 32 Requirements for Produc ing Measurements Five Stages in Performance Evaluation System Performance Issues / 41 Conclusion / 42 References / 43 1X / 37 / 38
5 AN AGENDA FOR RESEARCH. Opportunities / 5.1 Research Program Specifications Conduct of Research Program / 5.7 APPENDIX A STATEMENT OF TASK. APPENDIX B ANNOTATED SHORT BIBLIOGRAPHY OF ALGORITHM- OR. APPLICATION-SPECIALIZED COMPUTER SYSTEMS x / 5.2 53
