Appendix D
Detailed Comparisons of Ada and Other Third-Generation Programming Languages
This appendix presents details of studies, summarized in Chapter 2 in the sections titled "Analyses of Language Features" and "Comparisons of Empirical Data," that have analyzed the technical features of programming languages and have used empirical project data to compare languages' relative strength with respect to desired characteristics. Given the unscientific nature of most of the data, no strong conclusions are warranted. But in a general sense, most experience with Ada supports the conclusion that it benefits DOD warfighting applications; few data were found to refute such a conclusion, although the absence of such data may reflect a lack of any organized effort to promote the use of languages other than Ada in warfighting systems.
EVALUATIONS OF LANGUAGE FEATURES
A 1985 Federal Aviation Administration (FAA) study compared Ada with four other languages by conducting expert evaluations based on 48 technical language features arranged in six categories, and combining the evaluation results with the results of performance benchmark test results (IBM, 1985). A follow-on study was conducted by the Software Engineering Institute (SEI) in 1991 using the same expert evaluation methodology as the original study to compare Ada 83 with C++, but without the benchmark tests (Weiderman, 1991). The studies, the results of which are presented in Table D.1, compared the languages against the maximum possible scores for each of the six categories, which can be summarized as follows:
- Capability—facets of the implementation language relevant to programming or software engineering;
- Efficiency—factors relevant to optimization of generated code and run-time utilization of resources;
Table D.1 Language Functionality Attributes
|
|
FAA/IBM Study |
|
|
|
SEI Study |
||
Language Attributes |
Maximum Score |
Ada 83 |
C |
Pascal |
Jovial |
Fortran |
Ada 83 |
C++ |
Capability |
16.7 |
16.1 |
9.6 |
10.4 |
7.6 |
3.9 |
15.3 |
11.3 |
Efficiency |
16.4 |
8.0 |
11.8 |
10.8 |
11.0 |
11.1 |
10.7 |
10.9 |
Availability/ Reliability |
22.6 |
21.5 |
11.6 |
14.5 |
15.6 |
10.3 |
19.1 |
12.6 |
Maintainability/ Extensibility |
17.4 |
14.0 |
10.2 |
12.2 |
6.8 |
8.3 |
13.6 |
11.4 |
Life-cycle cost |
11.3 |
8.2 |
7.4 |
7.8 |
4.9 |
5.2 |
8.4 |
8.0 |
Risk |
15.6 |
8.8 |
8.9 |
7.6 |
9.6 |
8.2 |
11.7 |
9.8 |
Total |
100.0 |
76.6 |
59.5 |
63.3 |
55.5 |
47.0 |
78.8 |
64.0 |
SOURCES: Federal Aviation Administration (FAA)/IBM data from IBM (1985); Software Engineering Institute (SEI) data from Weiderman (1991). |
- Availability/Reliability—factors influencing day-to-day safety of operational systems;
- Maintainability/Extensibility—factors influencing long-term viability of operational systems;
- Life-cycle cost—elements of cost associated with or affected by the implementation language; and
- Risk—areas of uncertainty or concern associated with or affected by the implementation language.
Ada scored the highest overall in both studies. It scored the highest in each individual category, except for efficiency, where it was ranked last, and risk, where it was ranked first in the SEI study and third in the FAA study.
PROJECT EVALUATION DATA
In comparisons of Ada with other programming languages in actual development projects, empirical data on characteristics such as expressibility, maintainability, defect rates, reusability, and cost factors are of particular interest. Since Ada has been used in relatively large programs, a larger proportion of data has been collected for Ada than for contemporaneous commercial programming languages like C and C++.
Much of the data are subjective in nature, and care must be taken in drawing conclusions from such studies. For instance, the data in Table D.2 are confounded by differences in the expressive power of a source line of code in different programming languages. One way of normalizing uses tables of source lines of code per function point (Jones, 1995). However, as shown in Table D.2, these ratios have wide variability. Lubashevsky (1996) reports variations in source lines of code per function point exceeding factors of 2 for C and 6 for C++. Finally, there are differences in expressiveness for the same language across different application domains.
Table D.2 Source Lines of Code per Function Point
Language |
Low |
Mean |
High |
Ada 83 |
60 |
71 |
80 |
C |
60 |
128 |
170 |
C++ |
30 |
53 |
125 |
SOURCE: Data from Jones (1995). |
Open-Source Data
The Verdix Ada Development System (VADS) data (Zeigler, 1995) covers a period of approximately 10 years of development and enhancement of VADS compilers, debuggers, builder tools, and run-time support systems using both C and Ada. Table D.3 summarizes the comparative C and Ada experience data over this period.
Zeigler analyzed potential confounding effects of relative C and Ada software complexity, personnel capability, and learning curve effects and found that these factors did not cause any significant bias. The study thus provides a strong case that Ada was significantly better than C, in terms of both life-cycle cost and freedom from defects, for this large (over 1 million lines of both Ada and C) project in the compiler and tools area. However, the study can only be suggestive rather than definitive about the applicability of this result to other domains and other teams (the Verdix teams were both composed of high-capability, low-turnover personnel).
Since 1976, NASA's Software Engineering Laboratory (SEL) has been carefully collecting software data on non-embedded, non-real-time satellite flight dynamics software. Figure D.1 (McGarry et al., 1994) summarizes SEL's history of development error rates over this period. The data show a significant general decrease in error rates over the period, due to such techniques as code reading, systematic testing, and Cleanroom techniques. The data from 1988 to 1994 also show significantly lower error rates for Ada than for Fortran, measured per 1,000 lines of delivered code, or KDLOC (an average of 2.1 defects/KDLOC for Ada vs. 4.8 defects/KDLOC for Fortran). SEL's analyses for Ada initially concluded that owing to reuse, Ada projects experienced significant cost and schedule reductions compared to Fortran projects. Subsequently, however, the Ada object-oriented reuse approach was applied to Fortran projects, resulting in comparable gains.
Proprietary Data
This section summarizes proprietary data from Reifer (1996) and Jones (1994) that compare Ada to other programming languages with respect to cost, delivered defects, reliability, and productivity. Because the source data and detailed analysis are unavailable, these results are difficult to assess.
Cost
Data on cost, reliability, and maintainability have been compiled in a proprietary database of outcomes of 190 software projects 3 years old or less (Reifer, 1996). These projects were written in Ada 83, Ada 95, C, C++, and other third-generation programming languages (3GLs, such as Cobol or Fortran) and were compared to a norm for each domain, drawn from a database of over 1,500 projects 7 years old or less.
Table D.3 Verdix Ada Development System Development Data
Characteristic |
C |
Ada |
Script Files |
Other Files |
Totals |
Total lines |
1,925,523 |
1,883,751 |
117,964 |
604,078 |
4,531,316 |
SLOCa |
1,508,695 |
1,272,771 |
117,964 |
604,078 |
3,503,508 |
Files |
6,057 |
9,385 |
2,815 |
3,653 |
21,910 |
Updates |
47,775 |
34,516 |
12,963 |
12,189 |
107,443 |
New features |
26,483 |
23,031 |
5,594 |
6,145 |
61,253 |
Fixes/feature |
.52 |
.25 |
.82 |
.17 |
.41 |
Fixes/KSLOCb |
9.21 |
4.59 |
39.02 |
1.75 |
7.25 |
Cost/SLOC |
$10.52 |
$6.62 |
$15.38 |
$3.72 |
$8.10 |
Defects/KSLOC |
.676 |
.096 |
n/a |
n/a |
.355 |
a SLOC: Source lines of code. b KSLOC: 1,000 SLOC. SOURCE: Zeigler (1995). |
Reifer's project cost data, shown in Table D.4, indicate that, in general, Ada, C, and C++ programs were less expensive per delivered source line of code (SLOC) than other 3GL programs or the domain norm for military applications, although C and C++ programs were less expensive than Ada programs in all military domains except airborne and spaceborne weapons systems. In commercial domains, the development cost of C and C++ code was again less than or equivalent to the cost of Ada, and Ada was roughly equivalent in cost to other 3GLs and the domain norms. However, the wide variations in expressive power of a line of code in Ada, C, and C++ (see Table D.2) tend to overshadow the cost/SLOC differences shown in Table D.4.
Another study used telecommunications systems to evaluate object-oriented programming languages (Jones, 1994). The study used as a baseline a private branch exchange switch software application of 1,500 function points in size, compared across several telecommunications companies. Based on these data, Table D.5 shows the cost of developing the project in different languages, in dollars per function point. The results indicate that, in this domain, coding in C++ was less expensive than coding in Ada, while using C was more expensive.
Delivered Defects
Reifer's (1996) data, shown in Table D.6, indicate that in the 190 software projects represented, Ada code had fewer delivered defects than C and C++ code, which, in turn, had fewer defects than other 3GLs. In command and control and telecommunications applications, the difference between Ada and C/C++ was smaller than the difference between C/C++ and other 3GLs and the norm. In commercial products, information systems, and weapons systems, Ada code clearly had fewer defects than code in C, C++, and other 3GLs.
Based on Jones's (1994) data, Table D.7 indicates that, for telecommunications applications, the reliability in delivered defects per function point was roughly equivalent for C++ and Ada, and higher for C code.
Table D.4 Cost Data by Language (dollars per delivered source line of code)
Application Domain |
Ada 83 |
Ada 95 |
C |
C++ |
3GL |
Norm |
Command & Control |
||||||
Commercial |
50 |
n/a |
40 |
35 |
50 |
45 |
Military |
75 |
n/a |
75 |
70 |
100 |
80 |
Commercial Products |
35 |
30 |
25 |
30 |
40 |
40 |
Information Systems |
||||||
Commercial |
n/a |
n/a |
25 |
25 |
30 |
30 |
Military |
30 |
35 |
25 |
25 |
40 |
35 |
Telecommunications |
||||||
Commercial |
55 |
n/a |
40 |
45 |
50 |
50 |
Military |
60 |
n/a |
50 |
50 |
90 |
75 |
Weapons Systems |
||||||
Airborne & Spaceborne |
150 |
n/a |
175 |
n/a |
250 |
200 |
Ground-based |
80 |
n/a |
65 |
50 |
100 |
75 |
SOURCE: Reprinted from Reifer (1996). |
Table D.5 Telecommunications Project Costs by Language (dollars per function point)
|
Assembly |
C |
CHILL |
Pascal |
Ada 83 Ada 9Xa |
C++ |
Smalltalk |
|
Costb |
5,547 |
2,966 |
2,260 |
1,993 |
1,760 |
1,533 |
1,180 |
1,007 |
a Ada 9X figure based on simulation. b At a monthly salary rate of $10,000. SOURCE: Data from Jones (1994). |
The fairly consistent factor-of-10,000 relationship between the data on cost per function point shown in Table D.5 and the data on defects per function point shown in Table D.7 suggests that Jones's cost and defect data are highly correlated to some third factor. This third factor appears to be the mean SLOC/function point ratio from Jones (1995), shown in Table D.2. Comparing these ratios across C, Ada, and C++ indicates that they are approximately, although not exactly, equal, as shown in Table D.8.
Reliability
Data from Reifer (1996) in Table D.9 indicate that the incidence of failure for Ada programs was lower than that with C, C++, and other 3GLs. The outcome of this reliability comparison is similar to the defect comparison; Ada was superior to C and C++, which were better than other 3GLs.
Table D.6 Delivered Defect Data by Language (post-delivery errors per KSLOCa)
Table D.7 Defects for Telecommunications Applications (delivered defects per function point)
|
Assembly |
C |
CHILL |
Pascal |
Ada 83 |
Ada 9Xa |
C++ |
Smalltalk |
Defects |
0.52 |
0.29 |
0.23 |
0.20 |
0.17 |
0.15 |
0.14 |
0.13 |
a Ada 9X figure based on simulation. SOURCE: Data from Jones (1994). |
Table D.8 Correlation in Function Point (FP) Measures
|
SLOCa/FP (Jones, 1995) |
Dollars/FP (Jones, 1994) |
Defects/FP (Jones, 1994) |
C/Ada |
1.80 |
1.69 |
1.71 |
C++/Ada |
0.75 |
0.67 |
0.82 |
a SLOC: Source lines of code. |
Table D.9 Reliability Data by Language (weeks until next major repair incidence)
Application Domain |
Ada 83 |
Ada 95 |
C |
C++ |
3GL |
Norm |
Command & Control |
||||||
Commercial |
3.0 |
n/a |
2.5 |
n/a |
2.0 |
2.5 |
Military |
4.0 |
n/a |
3.0 |
n/a |
2.0 |
2.5 |
Commercial Products |
1.0 |
n/a |
0.4 |
1.0 |
0.4 |
0.5 |
Information Systems |
||||||
Commercial |
1.0 |
n/a |
0.5 |
0.6 |
0.4 |
0.5 |
Military |
0.8 |
n/a |
0.5 |
n/a |
0.4 |
0.5 |
Telecommunications |
||||||
Commercial |
3.0 |
n/a |
1.0 |
2.0 |
1.5 |
1.8 |
Military |
4.0 |
n/a |
2.0 |
3.0 |
2.0 |
2.0 |
Weapons Systems |
||||||
Airborne & Spaceborne |
8.0 |
n/a |
3.0 |
n/a |
2.5 |
2.5 |
Ground-based |
6.0 |
n/a |
3.0 |
n/a |
2.0 |
2.5 |
SOURCE: Reprinted from Reifer (1996). |
Productivity
As described in Chapter 2 in the section titled "Software Engineering Process and Architecture," one of the key relationships in software development productivity is the relationship of development effort to program size that is, the value of the "process exponent." A MITRE study (Giallombardo, 1992) found substantial improvements in productivity for large embedded military software developments provided by Ada compared with other 3GLs. The study found that effort, as measured by staff-months required per 1,000 equivalent delivered source instructions (KEDSI; which counts terminal semicolons rather than source lines of code), increased linearly for the large Ada projects (process exponent close to 1). For large non-Ada projects, effort increased more than linearly; the increase in effort was also greater than that predicted by the COCOMO model, which has a process exponent of 1.2 for embedded or very large systems (Boehm and Royce, 1988). The differences were clearest for system sizes greater than 100 KEDSI (which the MITRE study used as a definition of "large").
In a study of telecommunications applications, Jones (1994) compared the productivity of several 3GLs, using function points per staff-month as the metric. The results, presented in Table D.10, indicate that, in this domain, C++ developments had a higher productivity rate than those using Ada, while developments using C were less productive than those using Ada.
Table D.10 Productivity for Telecommunications Projects by Language (function points per staff-month)
|
Assembly |
C |
CHILL |
Pascal |
Ada 83 |
Ada 9Xa |
C++ |
Smalltalk |
Productivity |
1.80 |
3.37 |
4.42 |
5.01 |
5.68 |
6.52 |
8.47 |
9.99 |
a Ada 9X figure based on simulation. SOURCE: Data from Jones (1994). |
CONCLUSION
In summary, based on the results of currently available empirical data and feature analysis comparisons, a conclusion that Ada is superior, with respect to availability/reliability and a lower incidence of defects, appears warranted. The evidence is not strong enough to assert Ada's superiority in the cost area. However, there is some positive evidence in the cost area, and when it is combined with the anecdotal conclusions favoring Ada (described in "Anecdotal Experience from Projects" in Chapter 2) and the lack of solid evidence indicating less expensive custom software development in other languages, a case can be made that using Ada provides cost savings in building custom software, particularly for real-time, high-assurance warfighting applications.