Appendix D

Detailed Comparisons of Ada and Other Third-Generation Programming Languages

This appendix presents details of studies, summarized in Chapter 2 in the sections titled "Analyses of Language Features" and "Comparisons of Empirical Data," that have analyzed the technical features of programming languages and have used empirical project data to compare languages' relative strength with respect to desired characteristics. Given the unscientific nature of most of the data, no strong conclusions are warranted. But in a general sense, most experience with Ada supports the conclusion that it benefits DOD warfighting applications; few data were found to refute such a conclusion, although the absence of such data may reflect a lack of any organized effort to promote the use of languages other than Ada in warfighting systems.

EVALUATIONS OF LANGUAGE FEATURES

A 1985 Federal Aviation Administration (FAA) study compared Ada with four other languages by conducting expert evaluations based on 48 technical language features arranged in six categories, and combining the evaluation results with the results of performance benchmark test results (IBM, 1985). A follow-on study was conducted by the Software Engineering Institute (SEI) in 1991 using the same expert evaluation methodology as the original study to compare Ada 83 with C++, but without the benchmark tests (Weiderman, 1991). The studies, the results of which are presented in Table D.1, compared the languages against the maximum possible scores for each of the six categories, which can be summarized as follows:

  • Capability—facets of the implementation language relevant to programming or software engineering;
  • Efficiency—factors relevant to optimization of generated code and run-time utilization of resources;


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 92
--> Appendix D Detailed Comparisons of Ada and Other Third-Generation Programming Languages This appendix presents details of studies, summarized in Chapter 2 in the sections titled "Analyses of Language Features" and "Comparisons of Empirical Data," that have analyzed the technical features of programming languages and have used empirical project data to compare languages' relative strength with respect to desired characteristics. Given the unscientific nature of most of the data, no strong conclusions are warranted. But in a general sense, most experience with Ada supports the conclusion that it benefits DOD warfighting applications; few data were found to refute such a conclusion, although the absence of such data may reflect a lack of any organized effort to promote the use of languages other than Ada in warfighting systems. EVALUATIONS OF LANGUAGE FEATURES A 1985 Federal Aviation Administration (FAA) study compared Ada with four other languages by conducting expert evaluations based on 48 technical language features arranged in six categories, and combining the evaluation results with the results of performance benchmark test results (IBM, 1985). A follow-on study was conducted by the Software Engineering Institute (SEI) in 1991 using the same expert evaluation methodology as the original study to compare Ada 83 with C++, but without the benchmark tests (Weiderman, 1991). The studies, the results of which are presented in Table D.1, compared the languages against the maximum possible scores for each of the six categories, which can be summarized as follows: Capability—facets of the implementation language relevant to programming or software engineering; Efficiency—factors relevant to optimization of generated code and run-time utilization of resources;

OCR for page 92
--> Table D.1 Language Functionality Attributes     FAA/IBM Study       SEI Study Language Attributes Maximum Score Ada 83 C Pascal Jovial Fortran Ada 83 C++ Capability 16.7 16.1 9.6 10.4 7.6 3.9 15.3 11.3 Efficiency 16.4 8.0 11.8 10.8 11.0 11.1 10.7 10.9 Availability/ Reliability 22.6 21.5 11.6 14.5 15.6 10.3 19.1 12.6 Maintainability/ Extensibility 17.4 14.0 10.2 12.2 6.8 8.3 13.6 11.4 Life-cycle cost 11.3 8.2 7.4 7.8 4.9 5.2 8.4 8.0 Risk 15.6 8.8 8.9 7.6 9.6 8.2 11.7 9.8 Total 100.0 76.6 59.5 63.3 55.5 47.0 78.8 64.0   SOURCES: Federal Aviation Administration (FAA)/IBM data from IBM (1985); Software Engineering Institute (SEI) data from Weiderman (1991). Availability/Reliability—factors influencing day-to-day safety of operational systems; Maintainability/Extensibility—factors influencing long-term viability of operational systems; Life-cycle cost—elements of cost associated with or affected by the implementation language; and Risk—areas of uncertainty or concern associated with or affected by the implementation language. Ada scored the highest overall in both studies. It scored the highest in each individual category, except for efficiency, where it was ranked last, and risk, where it was ranked first in the SEI study and third in the FAA study. PROJECT EVALUATION DATA In comparisons of Ada with other programming languages in actual development projects, empirical data on characteristics such as expressibility, maintainability, defect rates, reusability, and cost factors are of particular interest. Since Ada has been used in relatively large programs, a larger proportion of data has been collected for Ada than for contemporaneous commercial programming languages like C and C++. Much of the data are subjective in nature, and care must be taken in drawing conclusions from such studies. For instance, the data in Table D.2 are confounded by differences in the expressive power of a source line of code in different programming languages. One way of normalizing uses tables of source lines of code per function point (Jones, 1995). However, as shown in Table D.2, these ratios have wide variability. Lubashevsky (1996) reports variations in source lines of code per function point exceeding factors of 2 for C and 6 for C++. Finally, there are differences in expressiveness for the same language across different application domains.

OCR for page 92
--> Table D.2 Source Lines of Code per Function Point Language Low Mean High Ada 83 60 71 80 C 60 128 170 C++ 30 53 125   SOURCE: Data from Jones (1995). Open-Source Data The Verdix Ada Development System (VADS) data (Zeigler, 1995) covers a period of approximately 10 years of development and enhancement of VADS compilers, debuggers, builder tools, and run-time support systems using both C and Ada. Table D.3 summarizes the comparative C and Ada experience data over this period. Zeigler analyzed potential confounding effects of relative C and Ada software complexity, personnel capability, and learning curve effects and found that these factors did not cause any significant bias. The study thus provides a strong case that Ada was significantly better than C, in terms of both life-cycle cost and freedom from defects, for this large (over 1 million lines of both Ada and C) project in the compiler and tools area. However, the study can only be suggestive rather than definitive about the applicability of this result to other domains and other teams (the Verdix teams were both composed of high-capability, low-turnover personnel). Since 1976, NASA's Software Engineering Laboratory (SEL) has been carefully collecting software data on non-embedded, non-real-time satellite flight dynamics software. Figure D.1 (McGarry et al., 1994) summarizes SEL's history of development error rates over this period. The data show a significant general decrease in error rates over the period, due to such techniques as code reading, systematic testing, and Cleanroom techniques. The data from 1988 to 1994 also show significantly lower error rates for Ada than for Fortran, measured per 1,000 lines of delivered code, or KDLOC (an average of 2.1 defects/KDLOC for Ada vs. 4.8 defects/KDLOC for Fortran). SEL's analyses for Ada initially concluded that owing to reuse, Ada projects experienced significant cost and schedule reductions compared to Fortran projects. Subsequently, however, the Ada object-oriented reuse approach was applied to Fortran projects, resulting in comparable gains. Proprietary Data This section summarizes proprietary data from Reifer (1996) and Jones (1994) that compare Ada to other programming languages with respect to cost, delivered defects, reliability, and productivity. Because the source data and detailed analysis are unavailable, these results are difficult to assess. Cost Data on cost, reliability, and maintainability have been compiled in a proprietary database of outcomes of 190 software projects 3 years old or less (Reifer, 1996). These projects were written in Ada 83, Ada 95, C, C++, and other third-generation programming languages (3GLs, such as Cobol or Fortran) and were compared to a norm for each domain, drawn from a database of over 1,500 projects 7 years old or less.

OCR for page 92
--> Table D.3 Verdix Ada Development System Development Data Characteristic C Ada Script Files Other Files Totals Total lines 1,925,523 1,883,751 117,964 604,078 4,531,316 SLOCa 1,508,695 1,272,771 117,964 604,078 3,503,508 Files 6,057 9,385 2,815 3,653 21,910 Updates 47,775 34,516 12,963 12,189 107,443 New features 26,483 23,031 5,594 6,145 61,253 Fixes/feature .52 .25 .82 .17 .41 Fixes/KSLOCb 9.21 4.59 39.02 1.75 7.25 Cost/SLOC $10.52 $6.62 $15.38 $3.72 $8.10 Defects/KSLOC .676 .096 n/a n/a .355 a SLOC: Source lines of code. b KSLOC: 1,000 SLOC. SOURCE: Zeigler (1995). Reifer's project cost data, shown in Table D.4, indicate that, in general, Ada, C, and C++ programs were less expensive per delivered source line of code (SLOC) than other 3GL programs or the domain norm for military applications, although C and C++ programs were less expensive than Ada programs in all military domains except airborne and spaceborne weapons systems. In commercial domains, the development cost of C and C++ code was again less than or equivalent to the cost of Ada, and Ada was roughly equivalent in cost to other 3GLs and the domain norms. However, the wide variations in expressive power of a line of code in Ada, C, and C++ (see Table D.2) tend to overshadow the cost/SLOC differences shown in Table D.4. Another study used telecommunications systems to evaluate object-oriented programming languages (Jones, 1994). The study used as a baseline a private branch exchange switch software application of 1,500 function points in size, compared across several telecommunications companies. Based on these data, Table D.5 shows the cost of developing the project in different languages, in dollars per function point. The results indicate that, in this domain, coding in C++ was less expensive than coding in Ada, while using C was more expensive. Delivered Defects Reifer's (1996) data, shown in Table D.6, indicate that in the 190 software projects represented, Ada code had fewer delivered defects than C and C++ code, which, in turn, had fewer defects than other 3GLs. In command and control and telecommunications applications, the difference between Ada and C/C++ was smaller than the difference between C/C++ and other 3GLs and the norm. In commercial products, information systems, and weapons systems, Ada code clearly had fewer defects than code in C, C++, and other 3GLs. Based on Jones's (1994) data, Table D.7 indicates that, for telecommunications applications, the reliability in delivered defects per function point was roughly equivalent for C++ and Ada, and higher for C code.

OCR for page 92
--> FIGURE D.1 NASA-SEL development error rates. KDLOC: 1,000 delivered lines of code. (Reprinted by permission from McGarry et al. (1994). Copyright 1994 by Carnegie Mellon University.)

OCR for page 92
--> Table D.4 Cost Data by Language (dollars per delivered source line of code) Application Domain Ada 83 Ada 95 C C++ 3GL Norm Command & Control Commercial 50 n/a 40 35 50 45 Military 75 n/a 75 70 100 80 Commercial Products 35 30 25 30 40 40 Information Systems Commercial n/a n/a 25 25 30 30 Military 30 35 25 25 40 35 Telecommunications Commercial 55 n/a 40 45 50 50 Military 60 n/a 50 50 90 75 Weapons Systems Airborne & Spaceborne 150 n/a 175 n/a 250 200 Ground-based 80 n/a 65 50 100 75   SOURCE: Reprinted from Reifer (1996). Table D.5 Telecommunications Project Costs by Language (dollars per function point)   Assembly C CHILL Pascal Ada 83 Ada 9Xa C++ Smalltalk Costb 5,547 2,966 2,260 1,993 1,760 1,533 1,180 1,007 a Ada 9X figure based on simulation. b At a monthly salary rate of $10,000. SOURCE: Data from Jones (1994). The fairly consistent factor-of-10,000 relationship between the data on cost per function point shown in Table D.5 and the data on defects per function point shown in Table D.7 suggests that Jones's cost and defect data are highly correlated to some third factor. This third factor appears to be the mean SLOC/function point ratio from Jones (1995), shown in Table D.2. Comparing these ratios across C, Ada, and C++ indicates that they are approximately, although not exactly, equal, as shown in Table D.8. Reliability Data from Reifer (1996) in Table D.9 indicate that the incidence of failure for Ada programs was lower than that with C, C++, and other 3GLs. The outcome of this reliability comparison is similar to the defect comparison; Ada was superior to C and C++, which were better than other 3GLs.

OCR for page 92
--> Table D.6 Delivered Defect Data by Language (post-delivery errors per KSLOCa) Application Domain Ada 83 Ada 95 C C++ 3GL Norm Command & Control Commercial 1.5 n/a 1.9 1.5 2.7 2.5 Military 0.7 n/a 1.0 1.2 2.3 2.0 Commercial Products 2.8 3.5 5.0 3.0 4.5 4.0 Information Systems Commercial 4.0 n/a 7.0 5.1 7.0 7.0 Military 3.0 n/a 6.0 4.0 6.0 6.0 Telecommunications Commercial 1.6 n/a 2.0 1.7 3.3 3.1 Military 1.0 n/a 1.5 1.2 2.7 2.5 Weapons Systems Airborne & Spaceborne 0.3 n/a 0.8 0.6 1.0 1.0 Ground-based 0.5 n/a 0.8 0.7 1.0 1.0 a KSLOC: 1,000 source lines of code. SOURCE: Reprinted from Reifer (1996). Table D.7 Defects for Telecommunications Applications (delivered defects per function point)   Assembly C CHILL Pascal Ada 83 Ada 9Xa C++ Smalltalk Defects 0.52 0.29 0.23 0.20 0.17 0.15 0.14 0.13 a Ada 9X figure based on simulation. SOURCE: Data from Jones (1994). Table D.8 Correlation in Function Point (FP) Measures   SLOCa/FP (Jones, 1995) Dollars/FP (Jones, 1994) Defects/FP (Jones, 1994) C/Ada 1.80 1.69 1.71 C++/Ada 0.75 0.67 0.82 a SLOC: Source lines of code.

OCR for page 92
--> Table D.9 Reliability Data by Language (weeks until next major repair incidence) Application Domain Ada 83 Ada 95 C C++ 3GL Norm Command & Control Commercial 3.0 n/a 2.5 n/a 2.0 2.5 Military 4.0 n/a 3.0 n/a 2.0 2.5 Commercial Products 1.0 n/a 0.4 1.0 0.4 0.5 Information Systems Commercial 1.0 n/a 0.5 0.6 0.4 0.5 Military 0.8 n/a 0.5 n/a 0.4 0.5 Telecommunications Commercial 3.0 n/a 1.0 2.0 1.5 1.8 Military 4.0 n/a 2.0 3.0 2.0 2.0 Weapons Systems Airborne & Spaceborne 8.0 n/a 3.0 n/a 2.5 2.5 Ground-based 6.0 n/a 3.0 n/a 2.0 2.5   SOURCE: Reprinted from Reifer (1996). Productivity As described in Chapter 2 in the section titled "Software Engineering Process and Architecture," one of the key relationships in software development productivity is the relationship of development effort to program size that is, the value of the "process exponent." A MITRE study (Giallombardo, 1992) found substantial improvements in productivity for large embedded military software developments provided by Ada compared with other 3GLs. The study found that effort, as measured by staff-months required per 1,000 equivalent delivered source instructions (KEDSI; which counts terminal semicolons rather than source lines of code), increased linearly for the large Ada projects (process exponent close to 1). For large non-Ada projects, effort increased more than linearly; the increase in effort was also greater than that predicted by the COCOMO model, which has a process exponent of 1.2 for embedded or very large systems (Boehm and Royce, 1988). The differences were clearest for system sizes greater than 100 KEDSI (which the MITRE study used as a definition of "large"). In a study of telecommunications applications, Jones (1994) compared the productivity of several 3GLs, using function points per staff-month as the metric. The results, presented in Table D.10, indicate that, in this domain, C++ developments had a higher productivity rate than those using Ada, while developments using C were less productive than those using Ada. Table D.10 Productivity for Telecommunications Projects by Language (function points per staff-month)   Assembly C CHILL Pascal Ada 83 Ada 9Xa C++ Smalltalk Productivity 1.80 3.37 4.42 5.01 5.68 6.52 8.47 9.99 a Ada 9X figure based on simulation. SOURCE: Data from Jones (1994).

OCR for page 92
--> CONCLUSION In summary, based on the results of currently available empirical data and feature analysis comparisons, a conclusion that Ada is superior, with respect to availability/reliability and a lower incidence of defects, appears warranted. The evidence is not strong enough to assert Ada's superiority in the cost area. However, there is some positive evidence in the cost area, and when it is combined with the anecdotal conclusions favoring Ada (described in "Anecdotal Experience from Projects" in Chapter 2) and the lack of solid evidence indicating less expensive custom software development in other languages, a case can be made that using Ada provides cost savings in building custom software, particularly for real-time, high-assurance warfighting applications.