3
DOD Software Policy: Analysis and Recommendations

The kinds of software programs that DOD needs for weapons systems and those that commercial industry creates for popular use often do not share the same basic requirements. DOD's requirements for high-assurance which include reliability, availability, survivability, safety, and security are considerably more stringent, given the national security stakes involved. DOD's requirements for performance-critical real-time embedded operation continually stress the capabilities of available hardware and software technology. Its requirements for complex integrated systems mean that weapon systems software must perform at a level higher than typical commercial programs found at the same stage of development, and all of these requirements must be satisfied simultaneously. In this regard, DOD (and its primary contractors) continues to be at the leading edge of the development of large-scale software engineering technology. It is likely to remain in this position for many years to come, although an increasing number of commercial software systems are comparable to those of DOD in size and required robustness.

This chapter establishes a set of objectives and criteria for evaluating the cost-effectiveness of alternative programming languages for support of DOD missions. It then presents a business-case analysis to evaluate Ada against other programming languages (generally focusing on C and C++) with respect to these criteria. Drawing on the technical and empirical evaluations reviewed in Chapter 2, this analysis is performed for (1) DOD warfighting software with no direct commercial counterparts, and (2) commercially dominated applications that are commonly used in all organizations. The analysis concludes that Ada can provide DOD with a considerable advantage in DOD-dominated warfighting applications, but not in commercially dominated applications. It also concludes that ensuring the Ada advantage for warfighting applications will require DOD investment to sustain a robust Ada infrastructure, but that the benefits justify the investment.

The final sections of this chapter summarize the policy changes recommended by the committee on the basis of this analysis, evaluate this policy with respect to possible alternative DOD policies on Ada, and present an economic evaluation of the recommended investment strategy.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 34
--> 3 DOD Software Policy: Analysis and Recommendations The kinds of software programs that DOD needs for weapons systems and those that commercial industry creates for popular use often do not share the same basic requirements. DOD's requirements for high-assurance which include reliability, availability, survivability, safety, and security are considerably more stringent, given the national security stakes involved. DOD's requirements for performance-critical real-time embedded operation continually stress the capabilities of available hardware and software technology. Its requirements for complex integrated systems mean that weapon systems software must perform at a level higher than typical commercial programs found at the same stage of development, and all of these requirements must be satisfied simultaneously. In this regard, DOD (and its primary contractors) continues to be at the leading edge of the development of large-scale software engineering technology. It is likely to remain in this position for many years to come, although an increasing number of commercial software systems are comparable to those of DOD in size and required robustness. This chapter establishes a set of objectives and criteria for evaluating the cost-effectiveness of alternative programming languages for support of DOD missions. It then presents a business-case analysis to evaluate Ada against other programming languages (generally focusing on C and C++) with respect to these criteria. Drawing on the technical and empirical evaluations reviewed in Chapter 2, this analysis is performed for (1) DOD warfighting software with no direct commercial counterparts, and (2) commercially dominated applications that are commonly used in all organizations. The analysis concludes that Ada can provide DOD with a considerable advantage in DOD-dominated warfighting applications, but not in commercially dominated applications. It also concludes that ensuring the Ada advantage for warfighting applications will require DOD investment to sustain a robust Ada infrastructure, but that the benefits justify the investment. The final sections of this chapter summarize the policy changes recommended by the committee on the basis of this analysis, evaluate this policy with respect to possible alternative DOD policies on Ada, and present an economic evaluation of the recommended investment strategy.

OCR for page 34
--> POLICY OBJECTIVES AND CRITERIA RELEVANT TO MEETING THEM Relating Criteria to Objectives DOD increasingly is emphasizing the concept of "information dominance" as a key to military superiority (Powell, 1992; DOD, 1996d). Secretary of Defense William Perry has stated that "…our warfighting strategy sustains and builds ... on the application of information technology to gain great military leverage to continue to give us [an] unfair competitive advantage" (Perry, 1996a).1 Since software development is a fundamental aspect of information technology, an appropriate objective for DOD software policy is to ensure that DOD software enables systems that are superior to any that an adversary might develop, thus affording DOD a competitive advantage. Achieving superiority in software and information technology for military purposes requires attention to situation-dependent mixes of functionality and other attributes. In an electronic warfare countermeasure/counter-countermeasure competition, rapid and reliable modification of software is a critical capability. To thwart information system penetration, subversion, and threatened denial of service, computer security is paramount. Performance is critical for processing of wideband sensor data. Safety is critical to controlling the course of weapons. Interoperability is critical for joint operations. And for all DOD systems, DOD's budget limitations make cost critical. It would be convenient if these criteria could be combined into a single "DOD return on investment" index. Alternative policies could then be evaluated definitively with respect to this composite criterion. However, decades of policy analysis have led to the conclusion that such combined analyses are feasible only in the context of particular military scenarios (Quade, 1964; Quade and Boucher, 1968). This creates two equally difficult problems-aggregating the results from multiple scenarios, and evaluating the relevance of the scenarios in a rapidly changing world. Given this situation, a useful approach is to identify the critical criteria most relevant to DOD's achieving information dominance, to evaluate Ada and alternative programming languages with respect to these criteria, and to base conclusions and recommendations on a judgment-based weighting of the criteria. Given the objective of having superior software, such evaluation must consider the effect of DOD programming language choices both on DOD itself and on its adversaries. Critical Criteria in DOD's Selection of a Programming Language As pointed out in "Economics of Software Engineering" in Chapter 2, software cost-modeling has shown that the criteria of software functionality, cost, and speed of development can be related to other criteria that are more closely coupled to choice of programming language. These determinants are software size, process, development environment, and personnel. Additional criteria for assessing software quality that are crucial in supporting DOD's critical missions and ability to respond rapidly to changes include high-assurance and real-time performance, a set that covers reliability, security, safety, survivability, and real-time performance; and ease of change, a capability that enables rapid-response to changes in threats, technology, or mission requirements. Besides the individual production factors involving software tools, technology, and personnel, an even stronger determinant of international competitive advantage in information dominance is the existence of a socio-technical infrastructure, which couples the production factors with knowledge resources, marketing channels, strategic partnerships, user groups, closely linked customer-supplier chains, and trends in domestic demand, thus stimulating innovation and enabling rapid development of new software products. Porter's (1990) analysis of the significance of a strong national socio-technical infrastructure in determining the competitive standing of an industry or a service sector supports consideration of the socio-technical infrastructure's role in the warfighting sector also.

OCR for page 34
--> These seven criteria-software size, process, environment, personnel, high-assurance and real-time performance, ease of change, and socio-technical infrastructure are used in the business-case analysis that follows. Warfighting and Commercially Dominated Applications The current socio-technical infrastructure is probably the main factor considered by many commercial firms in choosing C or C++ as their primary programming language, even when they may have performed a technical evaluation that indicated Ada was superior.2 For example, the fact that much of the financial community's software expertise, tooling, components, and middleware is oriented to C and C++ would generally be enough to lead some firms to choose C or C++.3 Conversely, this argument can be turned in favor of DOD's maintaining an Ada-based competitive advantage in software for warfighting, a domain in which DOD of course plays the dominant role. The term "warfighting" is used in the business-case analysis that follows to determine whether the use of Ada would provide a clear advantage for DOD, when compared to the alternative of using (typically non-Ada) commercial software solutions. This determination should be made at the subsystem level, in the context of decisions to build or buy software for the system. For example, an integrated combat system such as AEGIS can have "warfighting" weapon control subsystems for which a custom Ada solution is superior, and "non-warfighting" data management and graphical user interface subsystems for which non-Ada commercial solutions are superior. Warfighting Applications There are two primary criteria for determining whether a subsystem belongs in the "warfighting" category: Relatively little commercial software and expertise is available for implementing the desired functions. For example, even though intelligence analysis is involved in warfighting, many of its functions (database update, query and visualization, report generation) can be readily satisfied via non-warfighting commercial software. The application requires software quality attribute levels higher than those supportable by commercial software. For many warfighting functions, these involve real-time performance, reliability, and survivability, particularly in high-stress, crisis-mode situations in which DOD information processing functions may be under attack. The application domains for warfighting software include, but are not necessarily limited to, the following areas. Also mentioned are related support services and capabilities that are nevertheless outside the very specialized domain of warfighting. Weapon control includes weapon sensor processing; guidance, navigation, and control; and combat-oriented weapon delivery platform control. Included also are special weapon delivery platform operator devices such as heads-up displays. Weapon control does not include administrative functions and "hotel services" for large weapon delivery platforms such as aircraft carriers, or support subsystems performing mainstream data management, networking, and graphical user interface functions.

OCR for page 34
--> Electronic warfare depends on software involved in rapid-response electronic detection, identification, discrimination, tracking, platform-based communication, and associated countermeasure and counter-countermeasure applications. It does not include software for support subsystems performing mainstream data management, networking, and graphical user interface functions. Wideband real-time surveillance includes processing of hard or soft real-time images and data from infrared, radar, or other sensors. It does not include off-line query and analysis of surveillance archives, or support subsystems performing mainstream data management, networking, and graphical user interface functions. Battle management and battlefield communication includes hard or soft real-time weapons allocation, targeting, control, coordination, damage assessment, and associated battlefield communications requiring such special capabilities as spread spectrum, anti-jamming, and frequency hopping. It does not include off-line monitoring, update, query, and analysis of battle asset status, or off-battlefield communications. Thus, the range of "warfighting command, control, and communications (C3I) applications" is narrower than previous categorizations such as "C3I" or "mission-critical." In addition to the non-warfighting, more generic applications mentioned above, warfighting also does not include testing, simulation, training, off-line analysis, maintenance, and diagnostics. Software for such applications might well be implemented in Ada, but other languages may be better choices in some situations. Commercially Dominated Applications Commercially dominated applications include office and management support, routine operations support, asset status monitoring, logistics, medicine, and non-battlefield communications processing. ADA BUSINESS-CASE ANALYSIS Table 3.1 summarizes the results of the committee's business-case analysis for DOD use of Ada versus other third-generation programming languages (3GLs) for both warfighting and commercially dominated applications. Evaluation of Ada with respect to software size, process, development environment, personnel, high-assurance and real-time performance, ease of change, and socio-technical infrastructure is presented below. Criteria for Evaluation of Ada Software Size The critical portions of warfighting applications are largely custom software, or components reused from previous defense applications. Reducing the size of these applications involves capitalizing on the existing applications software base. The largest fraction of this software base (approximately one-third) is in Ada, giving an appreciable (but not overwhelming) advantage to Ada. Furthermore, dropping Ada would leave DOD with a large body (approximately 50 million source lines) of Ada code to reengineer.

OCR for page 34
--> Table 3.1 Ada Business-Case Analysis Summary Criterion Warfighting Applications Commercially Dominated Applications Size   Critical portions largely custom, or DOD-managed reuse Some Ada advantage due to existing DOD Ada investments Dropping Ada would leave large existing DOD Ada software base unsupported   Predominantly commercial off-the-shelf (COTS)-based solutions; much less programming than with custom solutions COTS largely non-Ada, with many volatile interfaces; a considerable disadvantage for Ada Process   Some Ada advantage: early verification of architecture interface obviates need for rework   Ada advantage for custom software is less applicable for COTS; some Ada disadvantage in COTS-based rapid development processes Environment   DOD investment required for Ada parity in general tools DOD investment creates some Ada advantage in high-assurance, real-time tools   Non-Ada tools and techniques much stronger than Ada counterparts Personnel   DOD has dominant position in Ada/applications skill base, which requires some investment to sustain Expensive for adversaries to create comparable Ada/applications skill base Non-Ada/applications skill base achievable, but with initial cost, lower competitive advantage   Non-Ada/applications skill base much stronger than Ada counterpart High- assurance and real-time performance   Ada superior; attributes are success-critical   Ada superiority diluted by COTS; attributes are less success-critical Ease of change   Ada somewhat superior, more so for high-assurance changes   Ada somewhat superior for custom software but at a disadvantage for COTS-based applications (see "Size" above) Socio-technical infrastructure   Existing DOD Ada-based infrastructure stronger than alternatives; requires some investment to sustain Expensive for adversaries to match and sustain comparable infrastructure   Existing C/C++ -based infrastructure much stronger than Ada-based infrastructure In commercially dominated applications, software size can be significantly reduced by the use of commercial off-the-shelf (COTS) software and fourth-generation programming languages (4GLs). Given that most of these applications are developed in C and C++, and given current and likely future COTS volatility, it will generally be harder for DOD to use Ada, as compared to C and C++, to develop and sustain non-COTS portions of the software.

OCR for page 34
--> Process As discussed in "Influence of Software Environments, Tools, and Languages on the Software Engineering Process" in Chapter 2, Ada enables the definition and verification of architectural interfaces early in the development process. This capability reduces a major source of rework, giving Ada another appreciable, but not overwhelming, advantage in the largely custom warfighting sector. The more COTS-intensive, commercially dominated applications would not benefit much from Ada-based early interface checking (absent a significant investment in sustaining Ada interface specifications for each evolving COTS product). Further, COTS-based rapid development processes based on early availability of C and C++ interfaces would leave Ada at some disadvantage. Environment In warfighting applications, investment in very high-assurance, real-time Ada software tools has been beneficial to DOD. Without continued investment, however, Ada tools will not be robust in either warfighting or commercial applications. The predominance of C and C++ tools in such areas as mainstream databases and graphical user interfaces places Ada at a significant disadvantage. For all applications, the large commercial marketplace will continue to stimulate development of a large selection of general software tools (e.g., smart editors, "debuggers," static and dynamic analyzers, and tools for configuration management, quality assurance, and testing), more frequently supporting C and C++ than Ada. Ada parity for general tools would require additional DOD investment. Personnel For development of software for warfighting, DOD has in place an established group of experts who understand the existing base of Ada warfighting software and can expeditiously extend it. Some DOD investment in Ada-based education and training is necessary to sustain the current skill base and to add new personnel, but this investment is minor compared to the investment an adversary would need to make to compete across the board with DOD in warfighting software (either by building an Ada skill base or developing a comparable warfighting applications software base in another language). Competing in warfighting niches (e.g., information warfare and security) would still be feasible for adversaries, independent of the choice of programming language. Although DOD could retrain its warfighting software programmers to use C and C++, thus avoiding the need for education in more specialized programming languages, such an approach would require a considerable time delay as well as investment for retraining the existing Ada applications skill base, and it would eliminate the competitive advantage currently enabled by the Ada applications skill base. Because commercially dominated applications software is written mostly in C, C++, and other non-Ada languages such as Cobol, Java, and 4GLs, DOD use of Ada in this sector would forego the advantages of relying on the base of commercial skills. High Assurance and Real-Time Performance As described in Appendix B and summarized in the Chapter 2 section titled "Technical Evaluation of Ada 95 and Other Third-Generation Programming Languages," Ada provides a significant technical advantage for achieving high-assurance and real-time performance, two attributes that are critical to DOD's competitive advantage for many warfighting applications. In commercially dominated applications, levels of assurance that can be provided by COTS represent the standard. Complementing COTS-based capabilities with those provided by Ada would not

OCR for page 34
--> improve much on COTS levels of assurance and real-time performance, which in general are sufficient for all but a small fraction of commercially dominated applications. Ease of Change As described in Appendix B and summarized in the Chapter 2 section "Technical Evaluation of Ada 95 and Other Third-Generation Programming Languages," Ada's encapsulation, type checking, "generics," and other capabilities facilitate changes in software and lead to lower maintenance costs for custom software. With Ada, high levels of assurance can be sustained for modified software, a distinct advantage for warfighting applications. As discussed for the criterion of software size, requiring the use of Ada for COTS-dominated commercial applications would be a disadvantage because of the need to adapt to frequent COTS changes. Socio-Technical Infrastructure DOD has developed a strong base of aerospace contractors and subcontractors who have in common a familiarity with both Ada and warfighting applications, enabling them to quickly combine in various ways to address new needs for and opportunities in warfighting software. No DOD adversary has a comparable range of capabilities or is likely to make the major investment required to generate and sustain such capabilities. Thus, although a moderate DOD investment is required to sustain Ada-based warfighting software, the competitive advantage gained in warfighting applications makes it more than worthwhile. On the other hand, for commercially dominated applications, the pervasiveness of C and C++ in the socio-technical infrastructure places Ada at a major disadvantage. Conclusions In weighing the factors discussed above and listed in Table 3.1, the most compelling points for Ada's use for warfighting applications are the strong competitive advantages accruing to DOD in the areas of personnel, high-assurance and real-time performance, and socio-technical infrastructure. Evaluation of Ada with respect to software size, process, development environment, and ease of change does not produce a sufficient rationale for requiring Ada's use. With respect to these criteria, Ada offers some moderate advantages, but, in some cases (e.g., for general software tools), DOD investment is required just to achieve parity between Ada and other available programming languages. Based on the points above, there is sufficient justification to continue the requirement to use Ada in warfighting applications if DOD also commits to the associated necessary investment in sustaining the infrastructure for Ada. The committee believes that an appropriate level of investment is roughly $15 million per year (detailed in Chapter 5); the economic analysis presented at the end of this chapter indicates that this level of investment is justified. Considering that programming languages are the materials out of which software is built, this investment strategy is a familiar one for DOD. It is analogous to investing in and encouraging the use of high-performance physical materials that provide DOD with competitive advantages for its weapon systems (see Box 3.1). However, this observation is not meant to imply that Ada technology should be put on the Militarily Critical Technologies List and be subject to embargo; rather, DOD can gain an advantage by sustaining its Ada capability for warfighting systems. Consideration of the factors listed in Table 3.1 for commercially dominated applications leads to the conclusion that a DOD requirement for Ada-based software in commercially dominated applications

OCR for page 34
--> BOX 3.1 Ada 95 as a DOD Materials Investment Suppose that DOD has been developing a new generation of materials for high-performance weapon systems. These materials have four main characteristics: They are based on proven cost-effective development and performance with a previous generation of materials. Their use requires special manufacturing and maintenance expertise. DOD and its supplier community have the largest source of such expertise, including tooling and components. They provide the potential for DOD to create and maintain systems that will significantly outperform adersaries' weapon systems. They require a continuing investment in materials research and development and in manufacturing technology of less than 0.1 percent of the cost of the weapon systems in which they will be deployed. When such materials are physical materials that contribute to low-observable, high-strength, or high-temperature performance, DOD routinely makes such investments. The same rationale can be applied to investment in weapon systems software, in which programming languages operate as the counterpart of physical materials. Each programming language contributes to varying extents to a warfighting application's malleability, ability to withstand stress, and suitability for various operational environments. For embedded weapon systems, the rationale for investment outlined above applies well to Ada 95, which has characteristics analogous to those described for new materials above. Several studies (see Chapter 2) have indicated that Ada 83 (the previous programming "material") has enabled more cost-effective development and maintenance of embedded weapon systems software than have other languages. The DOD software community has by far the largest source of people capable of programming in Ada 83, particularly for embedded weapon systems, and DOD's strongest embedded weapon systems software tools and components are based on Ada 83. Ada 95 incorporates a decade of experience in ways that improve on Ada 83's capability to address real-time, distributed, and object-oriented systems. Ada 83's capabilities and Ada 95's potential are products of a continuing DOD investment of more than $10 million annually for the past decade. A comparable investment is necessary to provide DOD with the continuing improvements in Ada 95 tools, components, infrastructure, and education required to keep DOD well ahead of any other nation's ability to produce or modify embedded weapon systems software. If DOD chooses to implement its future embedded weapon systems software in other programming languages such as C or C++, it can still produce good systems. However, such systems would provide lower-levels of assurance than those produced with Ada 95. Also, choosing another programming language would require a large DOD investment in moving its weapon system software components, manufacturing expertise, and maintenance expertise from Ada to C or C++. A further result of such a transition would be to make DOD's weapon system software components and expertise easier for adversaries to assimilate.

OCR for page 34
--> If any adversary chose to build its embedded weapon systems software using C or C++ and commercial off-the-shelf (COTS) components, it would have some advantages in tooling and a large labor pool to draw from. However, it would have several formidable problems in competing with DOD's weapon systems software. Chief among these problems are the amount of new C and C++ software (a large fraction of 50 million lines) that would need to be developed to replace the current inventory of Ada code; the need to retrain the labor force in weapon systems applications; the mismatch between COTS software (e.g., "windows") and real-time, high-assurance requirements; and the need to match DOD's expertise in such areas as reuse (via Ada's package specifications and generics), sensor-based control algorithms and data structures in Ada, and Ada's real-time scheduling capability. is not justified. For custom 3GL software in this sector, Ada is a strong candidate and should be considered in programming language decisions. However, it should not be necessary to provide extra justification for the use of languages other than Ada. FINDINGS AND RECOMMENDATIONS Based on its assessment of today's environment for software development and its evaluation of DOD's current programming language policy (Chapter 1), its examination of trends in software engineering and comparison of various programming languages (Chapter 2), and the results of its business-case analysis to evaluate Ada in two software application domains (the first two sections of this chapter), the committee developed the following set of findings and recommendations for DOD. The recommendations address the use of Ada in warfighting software, the application in which the committee finds Ada to have demonstrated benefit; the proper scope and implementation of DOD software policy; investment in Ada; and collection of data as a basis for assessing the effectiveness of software and software policy. Ada Competitive Advantage Finding. Ada gives DOD a competitive advantage in warfighting software applications, including weapon control, electronic warfare, performance-critical surveillance, and battle management. Recommendation. Continue vigorous promotion of Ada in warfighting application areas. Rationale. Available project data and analyses of programming language features indicate that, compared with other programming languages, Ada provides DOD with higher-quality warfighting software at a lower life-cycle cost. DOD can increase its advantage by strengthening its Ada-based production factors (involving software tools, technology, and personnel) for warfighting software (see Chapters 2 and 3). Applicability of Policy to DOD Domains Finding. DOD's current requirement for use of Ada is overly broad in its application to all DOD-maintained software.

OCR for page 34
--> Recommendation. Focus the Ada requirement on warfighting applications, particularly in critical, real-time applications, in which Ada has demonstrated success. For commercially dominated applications, such as office and management support, routine operations support, asset monitoring, logistics, and medicine, the option of using Ada should be analyzed but should not be assumed to be preferable. Rationale. For warfighting software, supporting Ada-based production factors (involving software tools, technology, and personnel) gives DOD a competitive advantage. In this domain, eliminating the use of Ada would both compromise this advantage and diminish the capabilities for maintaining DOD's existing 50 million lines of Ada. In commercially dominated areas, pushing applications toward Ada would create a disadvantage for DOD (see Chapters 2 and 3). Scope of Policy Finding. DOD's current requirement for use of Ada overemphasizes programming language considerations. Recommendation. Broaden the current policy to integrate the choice of programming language with other key software engineering concerns, such as software requirements, architecture, process, and quality factors. Rationale. The current policy isolates the Ada requirement and the waiver process from other software engineering decisions, causing programs to make premature or non-optimal decisions (see Chapter 1). DOD has already taken steps to broaden the policy focus in its draft revision of its programming language policy, DOD Directive 3405.1; this report recommends modifications to that draft policy (Appendix A). Policy Implementation Finding. DOD's current Ada requirement and the related waiver process have been weakly implemented. Many programs have simply ignored the waiver process. Other programs make programming language decisions at the system level, but often a mix of Ada and non-Ada subsystems is more appropriate (see Chapter 1). Recommendation. Integrate the Ada decision process with an overall Software Engineering Plan Review (SEPR) process. Passing such a review should be a requirement for entering the system acquisition Milestone I and II reviews covered by DOD Instruction 5000.2. It should also be required for systems not covered in 5000.2, and recommended for DOD-directed software development and maintenance of all kinds. Rationale. The SEPR concept is based on the highly successful commercial architecture review board practice. The SEPR process involves peer reviewing not only the software and system development plans, but also the software and system architecture (building plan) and its ability to satisfy mission requirements, operational concepts, conformance with architectural frameworks, and budget and schedule constraints; the process also involves reviewing other key decisions such as choice of programming language (see Chapter 4). Investment in Ada Finding. For Ada to remain the strongest programming language for warfighting software, DOD must provide technology and infrastructure support.

OCR for page 34
--> Recommendation. Invest in a significant level of support for Ada, or drop the Ada requirement. The strategy developed by the committee recommends an investment level of approximately $15 million per year. Rationale. With investment, DOD can create a significant Ada-based complex of production factors (involving software tools, technology, and personnel) for warfighting application domains. Without such support, Ada will become a second-tier, niche language such as Jovial or CMS-2 (see Chapter 5). Software Metrics Data Finding. DOD's incomplete and incommensurable base of software metrics data weakens its ability to make effective software policy, management, and technical decisions. Recommendation. Establish a sustained commitment to collect and analyze consistent software metrics data. Rationale. The five sets of findings and recommendations above are based on a mix of incomplete and incommensurable data, anecdotal evidence, and expert judgment. For this study, the patterns of consistency in these sources of evidence provide reasonable support for the results but not as much as could be provided by quantitative analysis based on solid data. A few organizations within DOD have benefited significantly from efforts to provide a sound basis for software metrics; a DOD-wide data collection effort would magnify the net benefits (see Chapter 2). ASSESSMENT OF POLICY ALTERNATIVES Table 3.2 summarizes current DOD policy, the committee's recommended alternatives to the policy currently in force, and other programming language policy alternatives suggested to or considered by the committee. The subsections that follow in the text summarize the committee's rationale for preferring these recommended actions over the alternatives. Conditions for Requiring Ada There are five conditions to be examined in determining whether a software development is subject to the Ada requirement. The following subsections describe the approach to defining these conditions. The committee recommends that all of these conditions be met in order for Ada to be required.

OCR for page 34
--> Table 3.2 Policy Alternatives Recommended by the Committee and Alternatives Considered   Committee Policy Item Current Policy Recommendation Alternatives Considered CONDITIONS FOR REQUIRING ADA Application domain All DOD software systems; all sectors Application subsystem is in the warfighting sector • None       • Other subsets Maintenance DOD-directed maintenance Same as current policy • All DOD software       • Only DOD-performed maintenance Level of applicability Entire system Subsystem is critical or is larger than 10 KLOC • Other size, criticality criteria Existing solutions Consider COTS, NDIs 4GLs No better COTS, NDI, or 4GL solution exists • Require COTS, NDIs, 4GLs Other languages No life-cycle cost -effectiveness justification for use of another language Same as current policy • Require another language (C, C++, Java)       • Require object-oriented solutions ADA REQUIREMENT System coverage Entire system; 100% Ada code 95% or more of subsystem in Ada • Other percentages LANGUAGE CHOICE PROCESS Exceptions Waiver-based; independent of other software engineering reviews Part of Software Engineering Plan Review process • Tied to different software engineering reviews Approval SAEs or ASD (C3I) Approval delegated through SAEs to appropriate level; periodically reviewed by ASD (C3I) • ASD (C3I) only       • Delegated to project manager's superior; periodically reviewed by SAEs, ASD (C3I) INVESTMENT IN ADA INFRASTRUCTURE Level $10 M in FY94; $0 M in FY98 $15 M/year • $2 M/year (barely sustaining)       • $30 M/year (major initiative) NOTE: ASD (C3I), Assistant Secretary of Defense (Command, Control, Communications, and Intelligence); COTS, commercial off-the-shelf; 4GL, fourth-generation programming language; KLOC, 1,000 lines of code; NDIs, non-developmental items; SAE, Service Acquisition Executive.

OCR for page 34
--> Application Subsystem Is in the Warfighting Sector Current DOD policy requires that Ada be used for all applications (subject to a waiver process and the criteria in Table 3.2). The committee recommends requiring Ada only for subsystems in DOD-dominated warfighting applications, as defined above in this chapter. For other DOD applications, Ada should be considered as a candidate but should not be required. The focus on subsystems makes it possible to separate the weapon control aspects of a system, which should be written in Ada, from other aspects of the system, which may be written in Ada, but need not be. An aircraft carrier is a good example of such a system. As discussed above, in non-warfighting applications, the rapid pace of non-Ada commercial software solutions shifts the business-case away from Ada. For warfighting applications, a focused Ada strategy can generate a differential that enables DOD to develop and adapt software more readily than its adversaries. The primary alternative to be considered is to not impose any requirement. Advocates of this approach (AIA, 1996) argue that developers of warfighting software can choose a programming language by analyzing trade-offs, in the same way that they make decisions about other non-developmental items (NDIs), such as commercial hardware and COTS software. The committee recognizes that many DOD and contractor organizations can make good technical choices when given the freedom to do so. However, the uneven distribution of software expertise across the DOD acquisition system could lead to some poor results, therefore leaving DOD worse off under an open policy than it would be under a requirement to use Ada in specific critical applications. The primary problem with leaving programming language selection to developers is that DOD acquisitions still are done largely one system at a time, rather than according to a product-line approach. Currently the focus in source selection is the cost of developing individual systems, and less attention is paid to important factors, such as assurance and adaptability, that are difficult to quantify. Such an approach can frequently lead to situations in which bidders can propose cheap but fragile solutions (e.g., using immature commercial-grade non-Ada process control software). DOD's program management staff often does not have sufficient software expertise (see Chapter 1) to be able to disqualify such inadequate solutions. The net result for DOD warfighting software then becomes a mix of strong and weak Ada and non-Ada solutions, and a socio-technical infrastructure characterized by increasing problems with proliferation of programming languages. Requiring Ada in warfighting applications is not a panacea, but it should lead to better-performing systems for DOD. Other possible policy alternatives could require that Ada be used for other subsets of DOD software, such as all software maintained by DOD personnel. In this particular case, Ada's use would be required for a great deal of financial and other commercially dominated software, which the committee's business-case analysis indicates would not be a cost-effective approach for DOD. The committee was unable to identify other subsets in which mandating use of Ada would match the cost-effectiveness of requiring its use in warfighting applications.4 Maintenance Is Directed by DOD DOD's current requirement for use of Ada excludes software that is embedded in a warfighting system component (e.g., an altimeter) purchased as a supplier-maintained commercial item. In such cases, DOD can frequently capitalize on dual-use (commercial and military) products as it does on COTS software. The committee recommends maintaining this aspect of the policy. The alternative of requiring Ada for all software in DOD warfighting systems is less cost-effective because it cuts off the opportunity to use many of these dual-use components. Another alternative, requiring use of Ada only for DOD-performed rather than DOD-directed maintenance (i.e., maintenance performed by DOD as well as by its contractors) would invite proliferation of programming languages in DOD's contractor-maintained

OCR for page 34
--> warfighting software inventory, with the same consequent relative weakening of DOD's position discussed in the previous subsection. Subsystem Is Critical or Larger Than 10, 000 Lines of Code The committee concluded that DOD's current policy of requiring waivers for use of non-Ada software for small, non-critical subsystems can cause bureaucratic delays that add little value. DOD should consider alternative size and criticality criteria. In suggesting alternatives, the committee focused on the objective of keeping the policy simple. The metric of 10,000 lines of code is based solely on engineering judgment: the committee's consensus was that 3,000 lines was too low and 30,000 lines was too high. "Critical" should be interpreted as key to mission success. No Better COTS, NDI, or 4GL Solution Exists As discussed in Chapter 1, the current DOD policy establishing Ada's precedence over COTS, NDI, and 4GL solutions has led to a number of bureaucratic delays or incorrect choices of an Ada solution over a more cost-effective COTS solution. In addition to its recommended alternative, another alternative considered by the committee was to require the use of COTS, NDI, and 4GL solutions. As discussed in Chapter 2, these solutions can have many drawbacks for DOD and other systems, and requiring their use would push many DOD systems into adopting less cost-effective solutions. No Life-cycle Cost-effectiveness Consideration Justifies Use of Another Language This criterion for required use of Ada is identical to DOD's current policy. As recommended below, embedding of the Ada waiver process within the SEPR process (Chapter 4) should make justifications for use of non-Ada software both sounder and less onerous. Alternatives suggested to the committee were to require the use of other programming languages (e.g., C, C++, or Java) or to require the use of object-oriented approaches. The comparisons of language features in Chapter 2 and the business-case analysis presented above in this chapter do not support a recommendation for requiring other languages. With respect to object-oriented techniques, the committee's assessment was that these are not well enough defined or sufficiently mature to support a requirement for use. Ada Requirement If all of the above criteria hold for a given software development, the committee's recommendation is that the requirement to use Ada should hold. The following subsection analyzes the appropriate coverage of the requirement. 95 Percent or More of the Subsystem 's Warfighting Software Is to Be Written in Ada The current DOD policy calls for the entire system to be developed in Ada. The committee recommends only that at least 95 percent of the applicable subsystem use Ada. Again, the intent of this recommendation is to avoid bureaucratic delays in cases involving small amounts of non-Ada "glue code" or performance enhancements, and 95 percent is again based on engineering judgment. The committee believes that 90 percent is too low and 98 percent too high. As with the criterion for subsystem size above, the scope of this requirement could be subject to some manipulation by program

OCR for page 34
--> desire to evade the requirements. However, the committee concluded that coupling the Ada waiver process with the SEPR process detailed in Chapter 4 would eliminate most of the gamesmanship in a cost-effective manner. Language Choice Process The following subsections recommend alternative approaches to implementation and approval of the language choice process. Replace the Waiver Approval Process with Other DOD Software Reviews The current DOD Ada waiver approval process is disconnected from other reviews of a project, causing extra work and out-of-context decisions (see Chapter 1). As discussed in Chapter 4, embedding the waiver process within the SEPR process should make it sounder and less onerous. Requests for waivers could also be considered at other review and approval milestones. The committee's recommendation associates the SEPR reviews with the Major Automated Information Systems Review Council (MAISRC) reviews at Milestones I and II, but recommends that the SEPR reviews for major weapon systems be completed as prerequisites to the counterpart Defense Acquisition Board (DAB) Milestone I and II reviews (see Figure 3.1). The SEPR reviews could be embedded within the DAB reviews, but experience to date indicates that software issues have had little visibility in many previous DAB reviews. The SEPR reviews and programming language decisions could also be handled in conjunction with source selection, but source selection may occur either too early or too late to allow effective consideration of waiver requests, depending on the relative timing of source selection and DAB/MAISRC milestone reviews. Basing a SEPR review on a 60-day proposal effort could also make it too superficial and vulnerable to manipulation. On the other hand, requiring an SEPR review for source selection would likely stimulate better proposals. Reconsider the Level at Which Waivers Can Be Approved The current DOD waiver policy requires Service Acquisition Executive (SAE) approval for service level waivers and Assistant Secretary of Defense (C3I) approval for waivers in joint programs. As discussed in Chapter 1, this requirement for approval at extremely high levels has induced considerable avoidance of the waiver process (sometimes leading to inappropriate use of Ada, sometimes simply to disregarding of the requirement). The committee recommends that SAE-level approval via the SEPR process be required only for major system acquisition, and that authority for granting waivers be delegated, as part of the SEPR process, to the equivalent of a product-line manager (a Program Executive Officer for families of warfighting systems; a base commander for base-specific systems). This approach would make the reviews and waiver process more mission-relevant. Periodic reviews by SAEs and the Assistant Secretary of Defense (C3I) to assess the effectiveness of the SEPR and Ada waiver process would ensure their long-range effectiveness without involving top executives regularly in low-level system decisions. Waiver approval at levels even higher than the current level (e.g., that of the Assistant Secretary of Defense (C3I) for every project) could also be considered but would exacerbate the current problems of critical-path delays and avoidance of the waiver process. Delegation of authority for approval to even lower-levels (e.g., the project manager's direct superior) could also be considered, but in many cases managers at such levels do not have sufficient visibility or responsibility across product-line objectives to make appropriate decisions affecting long-range product-line effectiveness.

OCR for page 34
--> FIGURE 3.1 Integration Software Plan Review (SEPR) process with Defense Acquisition Board (DAB) and Major Automated Information Systems Review Council (MAISRC) milestones for weapon and information systems. Investment in Ada Infrastructure The current DOD strategy is to reduce investment in Ada from a $10 million annual budget in Fiscal Year 1994 to essentially zero in Fiscal Year 1998. Other programming languages that DOD uses but provides little or no support for include Jovial and CMS-2; at this level of investment, bindings, runtime systems, language features, and tools are kept current only by projects using the language. This level is inadequate to keep a programming language up to date and serves as a major disincentive for projects to continue to use the language. Thus, DOD's planned zeroing out of investment in Ada is incompatible with any DOD requirement to use Ada. Chapter 5 identifies a set of investments that are necessary to provide DOD with a competitive advantage in warfighting software, and indicates that its magnitude is roughly $15 million per year. The economic analysis in the following section indicates that the resulting cost savings justify this level of investment. An alternative policy would be to attempt to sustain Ada at a considerably lower-level, on the order of $2 million annually. This level would be enough to enable DOD to keep its current 50 million lines of Ada from becoming hopelessly obsolete, but not enough to keep new programs from deciding that Ada would require too much extra investment to justify its use. Another alternative would be a major initiative level of investment, at roughly $30 million per year. This was the level recommended by the 1991 DOD software technology strategy for a 3-year period (DOD, 1991). It included such additions as a demonstration/validation phase for Ada 95, a major suite of tools for re-engineering legacy software into Ada, and investment in beyond-Ada 95 programming language research. Since 1991, the primary window of opportunity for an Ada 95 demonstration/validation phase has passed; programming language research is now being pursued in general by the National Science Foundation and in particular by the Defense Advanced Research Projects Agency's Dynamic Object-Oriented Language initiative. Given this situation, a strategy of investment at the margin in such areas as re-engineering tools and real-time,

OCR for page 34
--> high-assurance tools at the $15 million annual level appears more cost-effective than a major initiative, and can be justified given the large installed base of Ada warfighting software. ECONOMIC ANALYSIS OF INVESTMENT IN ADA INFRASTRUCTURE Given the fragility of the underlying data, a complex economic analysis attempting to establish the optimal investment in Ada infrastructure is not justified. Instead, a simple analysis is presented to estimate how the recommended $15 million level of investment will affect Ada tool capability, tool experience, and language capability (see Box 3.2). The simplifying assumptions, all made on the conservative side, are as follows: Only the annual maintenance of the 50 million lines of Ada code in existing weapon systems software is considered. This assumption is conservative because it does not consider developing and sustaining current and future Ada systems. Only 8 percent of the software will be changed per year. This assumption is conservative with respect to the average rate of change of 11 percent per year for embedded control software and 20 percent per year for human-machine interactive software in the COCOMO database (Boehm, 1981). The cost per changed line of code is only $40. This assumption is conservative with respect to general estimates that changing weapon systems software costs at least $75 per line of code. Only the effects of investment on Ada tool capability and Ada language and tool experience are considered. This assumption is conservative with respect to other cost effects due to process, reuse, and size improvements (separate vs. common development of infrastructure software). The presence versus the absence of an investment program for Ada will affect improvement in tool capability and language and tool experience by one cost model rating level (see Box 3.2 for details). This assumption is conservative with respect to the relative effects on other languages that are not supported by such investment, such as Jovial and CMS-2. The effect of increasing tool capability and language and tool experience one rating level is an 8 percent improvement. This assumption is conservative with respect to most estimates generated by software cost models, which indicate that the improvement per rating level for these variables averages 10 to 12 percent (see Box 3.2). Given these assumptions, an estimate of the annual cost of sustaining the 50 million lines of code (50 MLOC) of existing DOD weapon systems software in Ada is (50 MLOC) (0.08/year) ($40/LOC) = $160 million/year. With a one-level improvement in tool capability and language and tool experience, the estimated annual cost of sustaining existing DOD weapon systems software in Ada is ($160 million/year) (0.92) (0.92) = $135 million/year. Thus, a conservative estimate of the annual maintenance cost savings resulting from a one-level improvement in capability is $25 million per year, considerably more than the annual investment in Ada of $15 million per year that would be required to achieve the improvement in rating level.

OCR for page 34
--> BOX 3.2 Software Cost Models and the Effects of Investment As discussed in Chapter 2, most current models for estimating software costs have input parameters or cost drivers reflecting the product's size and the project's personnel, development environment, and process characteristics. The personnel and environment cost drivers generally take the form of multipliers affecting the estimated cost. For example, the COCOMO 2.0 TOOL model rates cost drivers for tools and corresponding cost multipliers as shown in the following table. Relative Estimated Cost for Various Levels of Tool Support Support Level TOOL Rating Level Cost Multiplier Very Low Edit, code "debugging" tools 1.20 Low Simple front-end, back-end CASE; little integration 1.10 Nominal Basic life-cycle tools; moderately integrated 1.00 High Strong, mature life-cycle tools; moderately integrated 0.88 Very High Strong, mature, proactive life-cycle tools; well integrated with processes reuse 0.75 The stronger the project's tool support, the lower its estimated cost. The estimated cost of a project with low tool support (simple front-end and back-end computer-automated software engineering (CASE) tools with little integration) is 10 percent higher than the estimated cost of a project with nominal tool support (basic life-cycle tools with moderate integration). The values in the table are based on a mix of several organizations' experience and calibration to project data. COCOMO 2.0 is currently calibrated with data from 65 projects, for which the calibrated cost estimates are within 25 percent of the actual cost 60 percent of the time. The corresponding rating scale and cost multipliers by the COCOMO 2.0 LTEX (language and tool experience) model are shown immediately below. Relative Estimated cost of Various Levels of Support for Programming Language and Tool Experience Support Level LTEX Ratinga Cost Multiplier Very Low 2 months or less 1.24 Low 6 months 1.11 Nominal 1 year 1.00 High 3 years 0.90 Very High 6 years 0.82   a LTEX: Language and tool experience

OCR for page 34
--> The use of calendar time as a basis for evaluating experience has both the advantage and disadvantage of being simple. The estimated cost of a project using personnel with low (median of 6 months) language and tool experience is 11 percent higher than the cost of using personnel with a nominal (median of 1 year) level of experience. Other models estimate similar costs. For example, the Jensen-SEER model (Jensen and Lucas, 1983) and the Softcost-R model1 have cost drivers comparable to the COCOMO TOOL and LTEX cost driver, with cost differentials in the range of 10 to 12 percent per rating level. Checkpoint has a "CASE tools" cost driver, with cost differentials varying from 10 to 15 percent per rating level, depending on interactions with other cost drivers covering staffing, methods, and language level (Jones, 1996d). A recent sample of 25 Ada projects from the Air Force Cost Analysis Agency indicates an average TOOL rating for the level of tool support of "nominal" and an average LTEX rating for the level of language and tool experience support of "high." As discussed in Chapter 5, the committee's recommended strategy for DOD's investment in Ada tools is to invest at the margin with tool suppliers to develop tools with clear benefits that otherwise would not be built. With this investment, the level of Ada tool support likely would increase by one TOOL rating level over the next 5 years. Without this investment, the level of Ada tool support would likely decrease with respect to the pace of surrounding technology. Thus, it is reasonable to assume that the net effect of the Ada investment would be an improvement of one-level on the TOOL rating scale, and a corresponding 8 percent cost reduction. Another effect of the committee's recommended Ada investment strategy, discussed in Chapter 5, would be to stimulate education and training in Ada, again via collaborative investments with suppliers at the margin. Without this investment, it will be much harder 5 years from now to find sources of qualified Ada personnel, and the likely net effect will be a one-level decrease in the average LTEX rating for Ada projects. With the Ada investment, the likely effect will be an increase in the average rating. Thus, a conservative assessment of the net effect of the committee's recommended Ada investment would be an increase of one-level on the LTEX rating scale, and a corresponding 8 percent cost reduction. 1   D.J. Reifer, personal communication to B. Boehm, August 1996. NOTES 1.   See also Perry (1996b). 2.   This process is illustrated by the experience of Xerox as described in Weiderman (1991). Xerox's Digital Systems Department evaluated Ada, C++, and two other languages for use in large, real-time, embedded systems software, and concluded that Ada was superior in terms of language features, implementation, and cost. In practice, however, the bulk of Xerox's embedded systems software continues to be developed in C and C++. 3.   For example, Gerald Pasternack of Citicorp testified to the committee about low awareness of and a low supply of Ada programmers in the New York financial market. 4.   The committee heard from several DOD representatives that measures to reform acquisition emphasize performance requirements over military specifications. This trend can be interpreted as supporting the option of having no requirement to use Ada.