Panel on Statistical Methods in Software Engineering

Committee on Applied and Theoretical Statistics

Board on Mathematical Sciences

Commission on Physical Sciences, Mathematics, and Applications

National Research Council

National Academy Press
Washington, D.C. 1996

The National Academies of Sciences, Engineering, and Medicine
500 Fifth St. N.W. | Washington, D.C. 20001

Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
Statistical Software Engineering STATISTICAL SOFTWARE ENGINEERING Panel on Statistical Methods in Software Engineering Committee on Applied and Theoretical Statistics Board on Mathematical Sciences Commission on Physical Sciences, Mathematics, and Applications National Research Council National Academy Press Washington, D.C. 1996

OCR for page R1
Statistical Software Engineering NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce Alberts is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievement of engineers. Dr. Harold Liebowitz is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce Alberts and Dr. Harold Liebowitz are chairman and vice-chairman, respectively, of the National Research Council. This project was supported by the Advanced Research Projects Agency, Army Research Office, National Science Foundation, and Department of the Navy's Office of the Chief of Naval Research. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors. Furthermore, the content of the report does not necessarily reflect the position or the policy of the U.S. government, and no official endorsement should be inferred. Copyright 1996 by the National Academy of Sciences. All rights reserved. Library of Congress Catalog Card Number 95-71101 International Standard Book Number 0-309-05344-7 Additional copies of this report are available from: National Academy Press, Box 285 2101 Constitution Avenue, N.W. Washington, D.C. 20055 800-624-6242 202-334-3313 (in the Washington metropolitan area) B-676 Printed in the United States of America

OCR for page R1
Statistical Software Engineering PANEL ON STATISTICAL METHODS IN SOFTWARE ENGINEERING DARYL PREGIBON, AT&T Bell Laboratories, Chair HERMAN CHERNOFF, Harvard University BILL CURTIS, Carnegie Mellon University SIDDHARTHA R. DALAL, Bellcore GLORIA J. DAVIS, NASA-Ames Research Center RICHARD A. DEMILLO, Bellcore STEPHEN G. EICK, AT&T Bell Laboratories BEV LITTLEWOOD, City University, London, England CHITOOR V. RAMAMOORTHY, University of California, Berkeley Staff JOHN R. TUCKER, Director

OCR for page R1
Statistical Software Engineering COMMITTEE ON APPLIED AND THEORETICAL STATISTICS JON R. KETTENRING, Bellcore, Chair RICHARD A. BERK, University of California, Los Angeles LAWRENCE D. BROWN, University of Pennsylvania NICHOLAS P. JEWELL, University of California, Berkeley JAMES D. KUELBS, University of Wisconsin JOHN LEHOCZKY, Carnegie Mellon University DARYL PREGIBON, AT&T Bell Laboratories FRITZ SCHEUREN, George Washington University J. LAURIE SNELL, Dartmouth College ELIZABETH THOMPSON, University of Washington Staff JACK ALEXANDER, Program Officer

OCR for page R1
Statistical Software Engineering BOARD ON MATHEMATICAL SCIENCES AVNER FRIEDMAN, University of Minnesota, Chair LOUIS AUSLANDER, City University of New York HYMAN BASS, Columbia University MARY ELLEN BOCK, Purdue University PETER E. CASTRO, Eastman Kodak Company FAN R.K. CHUNG, University of Pennsylvania R. DUNCAN LUCE, University of California, Irvine SUSAN MONTGOMERY, University of Southern California GEORGE NEMHAUSER, Georgia Institute of Technology ANIL NERODE, Cornell University IMGRAM OLKIN, Stanford University RONALD F. PEIERLS, Brookhaven National Laboratory DONALD ST. P. RICHARDS, University of Virginia MARY F. WHEELER, Rice University WILLIAM P. ZIEMER, Indiana University Ex Officio Member JON R. KETTENRING, Bellcore Chair, Committee on Applied and Theoretical Statistics Staff JOHN R. TUCKER, Director JACK ALEXANDER, Program Officer RUTH E. O'BRIEN, Staff Associate BARBARA W. WRIGHT, Administrative Assistant

OCR for page R1
Statistical Software Engineering COMMISSION ON PHYSICAL SCIENCES, MATHEMATICS, AND APPLICATIONS ROBERT J. HERMANN, United Technologies Corporation, Chair STEPHEN L. ADLER, Institute for Advanced Study PETER M. BANKS, Environmental Research Institute of Michigan SYLVIA T. CEYER, Massachusetts Institute of Technology L. LOUIS HEGEDUS, W.R. Grace and Company JOHN E. HOPCROFT, Cornell University RHONDA J. HUGHES, Bryn Mawr College SHIRLEY A. JACKSON, U.S. Nuclear Regulatory Commission KENNETH I. KELLERMANN, National Radio Astronomy Observatory KEN KENNEDY, Rice University THOMAS A. PRINCE, California Institute of Technology JEROME SACKS, National Institute of Statistical Sciences L.E. SCRIVEN, University of Minnesota LEON T. SILVER, California Institute of Technology CHARLES P. SLICHTER, University of Illinois at Urbana-Champaign ALVIN W. TRIVELPIECE, Oak Ridge National Laboratory SHMUEL WINOGRAD, IBM T.J. Watson Research Center CHARLES A. ZRAKET, Mitre Corporation (retired) NORMAN METZGER, Executive Director

OCR for page R1
Statistical Software Engineering Preface The development and the production of high-quality, reliable, complex computer software have become critical issues in the enormous worldwide computer technology market. The capability to efficiently engineer computer software development and production processes is central to the future economic strength, competitiveness, and national security of the United States. However, problems related to software quality, reliability, and safety persist, a prominent example being the failure on several occasions of major local and national telecommunications networks. It is now acknowledged that the costs of producing and maintaining software greatly exceed the costs of developing, producing, and maintaining hardware. Thus the development and application of cost-saving tools, along with techniques for ensuring quality and reliability in software engineering, are primary goals in today's software industry. The enormity of this software production and maintenance activity is such that any tools contributing to serious cost savings will yield a tremendous payoff in absolute terms. At a meeting of the Committee on Applied and Theoretical Statistics (CATS) of the National Research Council (NRC), participants identified software engineering as an area presenting numerous opportunities for fruitful contributions from statistics and offering excellent potential for beneficial interactions between statisticians and software engineers that might promote improved software engineering practice and cost savings. To delineate these opportunities and focus attention on contexts promising useful interactions, CATS convened a study panel to gather information and produce a report that would (1) exhibit improved methods for assessing software productivity, quality, reliability, associated risk, and safety and for managing software development processes, (2) outline a program of research in the statistical sciences and their applications to software engineering with the aim of motivating and attracting new researchers from the mathematical sciences, statistics, and software engineering fields t o tackle these important and pressing problem areas, and (3) emphasize the relevance of using rigorous statistical and probabilistic techniques in software engineering contexts and suggest opportunities for further research in this direction. To help identify important issues and obtain a broad range of perspectives on them, the panel organized an information-gathering forum on October 11-12, 1993, at which 12 invited speakers addressed how statistical methods impinge on the software development process, software metrics, software dependability and testing, and software visualization. The forum also included consideration of nonstandard methods and select case studies (see the forum program in the appendix). The panel hopes that its report, which is based on the panel's expertise as well as information presented at the forum, will contribute to positive advances in software engineering and, as a subsidiary benefit, be a stimulus for other closely related disciplines, e.g., applied mathematics, operations research, computer science, and systems and industrial engineering. The panel is, in fact, very enthusiastic about the opportunities facing the statistical community and hopes to convey this enthusiasm in this report. The panel gratefully acknowledges the assistance and information provided by a number of individuals, including the 12 forum speakers—T.W. Keller, D. Card, V.R. Basili, J.C. Munson, J.C. Knight, R. Lipton, T. Yamaura, S. Zweben, M.S. Phadke, E.E. Sumne r, Jr., W. Hill, and J. Stasko—four anonymous reviewers, the NRC staff of the Board on Mathematical Sciences who supported the various facets of this project, and Susan Maurizi for her work in editing the manuscript.

OCR for page R1
Statistical Software Engineering This page in the original is blank.

OCR for page R1
Statistical Software Engineering Contents     EXECUTIVE SUMMARY   1 1   INTRODUCTION   5 2   CASE STUDY: NASA SPACE SHUTTLE FLIGHT CONTROL SOFTWARE   9     Overview of Requirements   9     The Operational Life Cycle   10     A Statistical Approach to Managing the Software Production Process   10     Fault Detection   11     Safety Certification   12 3   A SOFTWARE PRODUCTION MODEL   13     Problem Formulation and Specification of Requirements   14     Design   14     Implementation   16     Testing   18 4   CRITIQUE OF SOME CURRENT APPLICATIONS OF STATISTICS IN SOFTWARE ENGINEERING   27     Cost Estimation   27     Statistical Inadequacies in Estimating   29     Process Volatility   30     Maturity and Data Granularity   30     Reliability of Model Inputs   31     Managing to Estimates   32     Assessment and Reliability   32     Reliability Growth Modeling   32     Influence of the Development Process on Software Dependability   36     Influence of the Operational Environment on Software Dependability   37     Safety-Critical Software and the Problem of Assuring Ultrahigh Dependability   38     Design Diversity, Fault Tolerance, and General Issues of Dependence   38     Judgment and Decision-making Framework   39     Structural Modeling Issues   40     Experimentation, Data Collection, and General Statistical Techniques   40     Software Measurement and Metrics   41 5   STATISTICAL CHALLENGES   43     Software Engineering Experimental Issues   43     Combining Information   46     Visualization in Software Engineering   48

OCR for page R1
Statistical Software Engineering     Configuration Management Data   49     Function Call Graphs   50     Test Code Coverage   50     Code Metrics   50     Challenges for Visualization   52     Opportunities for Visualization   52     Orthogonal Defect Classification   59 6   SUMMARY AND CONCLUSIONS   61     Institutional Model for Research   62     Model for Data Collection and Analysis   62     Issues in Education   64     REFERENCES   67     APPENDIX: FORUM PROGRAM   72