Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Summary The National Research Council’s Committee for Advancing Software-Intensive Systems Produc - ibility was commissioned by the Office of the Secretary of Defense (OSD) to examine the nature of the national investment in software research and ways to revitalize the knowledge base needed to design, produce, and employ software-intensive systems for tomorrow’s defense needs. This report contemplates Department of Defense needs and priorities (Chapter 1) for software producibility—that is, the capacity to design, produce, assure, and evolve innovative software-intensive systems in a predict - able manner while effectively managing risk, cost, schedule, and complexity. It suggests feasible actions related to software process and measurement (Chapter 2), architecture (Chapter 3), and assurance (Chapter 4), and it suggests a research agenda (Chapter 5) that focuses on issues critical to Department of Defense (DoD) software capability. Box S.1 summarizes several of the key messages of the findings and recommendations by showing how they address eight “myths” regarding software producibility. The key findings and recommenda - tions of the committee are presented in this Summary, and additional findings and recommendations are offered in subsequent chapters. A complete set is presented in Box S.2. This final project report builds on two prior reports—the discussion of technical and organizational issues in Summary of a Workshop on Software Intensie Systems and Uncertainty at Scale1 and a subsequent letter report focused on the rationale for investment in software research. 2 1 National Research Council (NRC), 2007, Summary of a Workshop on Software Intensie Systems and Uncertainty at Scale, Wash- ington, DC: National Academies Press. Available online at http://www.nap.edu/catalog.php?record_id=11936. Last accessed August 10, 2010. 2 NRC, 2008, Preliminary Obserations on DoD Software Research Needs and Priorities: A Letter Report , Washington, DC: National Academies Press. Available online at http://www.nap.edu/catalog.php?record_id=12172. Last accessed August 10, 2010 

OCR for page 1
 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE Box S.1 Eight Myths About Defense Software Producibility 1. The DoD’s software producibility challenges are predominantly challenges of management and process but not of technology. • (See Findings 1-1, 1-3, 1-4, 2-5, 4-2, 5-2 and Recommendations 1-1, 4-2, 5-1.) 2. The DoD and its contractors can rely on industry to innovate at a rate fast enough to solve the DoD’s hard technical problems and to stay ahead of its adversaries. • (See Findings 1-3, 1-4 and Recommendation 1-1.) 3. Software technology is approaching a plateau, which diminishes the need to invest in technol- ogy innovation. • (See Findings 1-5, 5-2 and Recommendations 4-2, 5-1.) 4. The software research community is doing potentially relevant theoretical work, but it has not led to advances of compelling importance to the DoD. • (See Finding 5-1.) 5. We have not yet developed effective mechanisms to mitigate the risks, particularly those related to scale and adoptability, associated with the transition to practice of innovative software-development technologies. • (See Findings 3-2, 3-4, 3-5, 4-2, 4-3 and Recommendations 2-1, 3-4, 4-2, 4-3.) 6. We will never create perfectly reliable and secure software, so we should focus primarily on provenance—trusted sources—rather than attempting to achieve assurance through improvements in practices and tools for evaluating artifacts directly. • (See Findings 4-1, 4-2 and Recommendations 4-1, 4-3.) 7. There is sufficient software research already underway, sponsored primarily by NSF and other basic science agencies, to meet the DoD’s software needs. • (See Recommendations 1-1, 5-1.) 8. Earned value management approaches based on code accumulation are a sufficient basis for managing software development programs, including incremental iterative development. • (See Findings 2-3, 2-4 and Recommendations 2-1, 2-2.) 1. RECOGNIzE THE PIVOTAL ROLE OF DOD SOFTWARE INNOVATION The continued increase in the DoD’s dependency on software is well documented by the Defense Science Board (DSB) and in multiple National Academies reports.3,4,5,6 This increase amounts to an order of magnitude of lines of software code every decade, and it is a natural consequence of the distinctive advantages of software as an engineering medium. Software is uniquely unbounded and flexible, can 3 Defense Science Board (DSB), September 2007, Report of the Defense Science Board Task Force on Mission Impact of Foreign Influence on DoD Software, Washington, DC: Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics. Available online at http://stinet.dtic.mil/oai/oai?&verb=getRecord&metadataPrefix=html&identifier=ADA473661. Last accessed August 20, 2010. 4 DSB, November 2000, Report of the Defense Science Board Task Force on Defense Software , Washington, DC: Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics. Available online at http://oai.dtic.mil/oai/oai?verb=getRecord &metadataPrefix=html&identifier=ADA385923. Last accessed August 20, 2010. 5 National Research Council (NRC), 2010, Achieing Effectie Acquisition of Information Technology in the Department of Defense , Washington, DC: National Academies Press. Available online at http://www.nap.edu/catalog.php?record_id=12823. Last ac - cessed August 20, 2010. 6 NRC, 1999, Realizing the Potential of C4I: Fundamental Challenges ,Washington, DC: National Academy Press. Available online at http://www.nap.edu/catalog.php?record_id=6457. Last accessed August 20, 2010.

OCR for page 1
 SUMMARY be delivered and upgraded electronically and remotely, and has the potential to be rapidly adapted to changing threats, operating environments, and platform technologies. Because it is unconstrained by traditional physical engineering limitations, the principal limits on what we can accomplish with software derive from human intellectual capacity to conceptualize and understand systems, to build tools to develop and manage them, and to provide assurance regarding critical functional and quality attributes. The same reports also indicate that the DoD would benefit from strategic steps to improve its abil - ity to design, develop, and assure complex software. Software not only is expanding in use but also is shifting into a more strategic and fundamental role in diverse systems. A vital question is how the DoD can ensure that it will be able to meet its software needs now and into the future. Finding 1-1: Software has become essential to a vast range of military system capabilities and opera - tions, and its role is continuing to deepen and broaden, including interlinking diverse system ele - ments. This creates both benefits and risks. Compounding these issues are the growing size, complexity, and geography of the supply chain structure for major software systems. This is a consequence of two powerful forces—the advance of technology that has enabled greater software modularization, and the globalization of software devel - opment activity. Although the United States continues to retain innovation leadership in software areas important to the DoD, there are factors that could cause the loss of that leadership. Some observers have speculated that software and information technology generally are reaching a plateau of capability and performance. This is a false and dangerous speculation—the capability and the complexity of hardware7 and software systems are both rising at an accelerating rate. Finding 1-5: It is dangerous to conclude that we are reaching a plateau in capability and technology for software producibility. To avoid loss of leadership, the DoD will need to become more fully engaged in the innovative processes related to software producibility. A key question addressed by the committee is to what extent the DoD, without providing any explicit R&D stimulus, can rely on industry—specifically the domestic defense industrial base and supporting vendors—to produce software innovations in areas of defense significance at a rate fast enough to allow the DoD to fully meet software requirements and remain ahead of potential adversaries. Finding the answer to this question is made more urgent by the expected continued rapid evolution of software capability worldwide. A loss of leadership could threaten the ability of the DoD not only to manifest world-leading capability, but also to achieve adequate levels of assurance for the diversely sourced software it intends to deploy. It will thus be essential for the DoD to reengage directly in the innovation process if it is to retain this necessary leadership. (See also Recommendation 5-1.) Finding 1-4: The DoD’s needs will not be sufficiently met through a combination of demand-pull from the military and technology-push from the defense or commercial information technology sec - tors. The DoD cannot rely on industry alone to address the long-term software challenges particular to defense. Defense requirements for software are in many respects similar to requirements in other sectors. But there are important areas where the DoD must push the envelope beyond mainstream capability 7 Moore’s Law is an informal predictive model created by Gordon Moore in 1965 for the number of transistors on integrated circuit chips. For decades, there has been a close correlation of transistor count with both processor clock speeds and overall computing capacity. Recently, due to a combination of factors, clock speeds have leveled off or even diminished, while the growth in general-purpose computing capacity has been achieved through the provisioning of multiple processors (called “cores”). This has created an added challenge related to concurrency for software developers, as elaborated in Chapters 4 and 5.

OCR for page 1
4 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE in order to meet its mission needs. These areas of “leading demand” include, for example, software assurance in the presence of highly sophisticated adversaries, architectural innovation and complexity, criticality with respect to safety, overall complexity and scale, and the arm’s-length relationship that the DoD has with its development teams—where mission stakeholders are often required to engage with development teams only through a legal and contractual interface. Recommendation 1-1: The DoD, through its Director of Research and Engineering (DDR&E), should regularly undertake an identification of areas of technological need related to software producibil - ity where the DoD has “leading demand” and where accelerated progress is needed to support the defense mission. 2. ACCEPT UNCERTAINTY: ATTACk RISkS AND ExPLOIT OPPORTUNITIES The management of innovative software development is largely a process of managing risks. Experi - ence shows that, in the absence of advanced process models, there is a correlation between the degree of precedent and routinization, on the one hand, and the ability to deliver results with predictable cost, schedule, and success in acceptance evaluation, on the other. With regard to the precedented elements—whose users can benefit, in terms of design costs and risks, from the experience of existing and prior users—the DoD benefits by adjusting its practices to conform to government and industry conventions, enabling it to exploit a broader array of more mature market offerings. When applied to innovative systems, however, the familiar sequential (“waterfall”) processes can often lead to costly surprises and increased programmatic risk. That is, what appears to be a “safe” conservative decision to follow the most basic process is in fact a dangerous decision that can drastically increase programmatic risk and the possibility of total project failure. The largest producibility challenges for the DoD, therefore, arise from its need to develop innovative, unprec - edented software systems. Such efforts at development necessarily build on precedented elements, and the unprecedented aspects may create substantial programmatic risk unless managed effectively. Effective management means identifying and mitigating the engineering risks that derive primarily from the innovative elements—architecture, assurance, requirements, design, scale, performance, etc. A well-managed incremental and iterative process, supported by appropriate iterative evaluation and measurement approaches, can more reliably lead to successful outcomes—lowering programmatic risk, even when there are significant engineering risks. Modern software governance is about managing uncertainty. This means treating project scope, plans, and resources as variables (not frozen baselines) and explicitly managing the variances in these variables until they converge to acceptable levels. This requires honest and well-informed assessments of engineering risks to effectively trade off cost, schedule, overall programmatic risk, and functionality. When there is substantial software-manifest functionality as well as software-related risks, there should be a close coupling of design and process decisions relating to hardware, software, and human- systems integration, with prioritization based on identified criteria. 8 Finding 2-1: Modern practice for innovative software systems at all levels of scale is geared toward incremental identification and mitigation of engineering uncertainties, including requirements uncertainties. For defense software, the challenge is doing so at a larger scale and in ways that are closely linked with an overall systems engineering process. Following the practice of other organizations that manage large engineering projects, the DoD has 8 Fred Brooks, 2010, The Design of Design: Essays from a Computer Scientist , Boston: Addison-Wesley. See also NRC, Richard Pew and Anne Mavor, eds., 2007, Human-System Integration in the System Deelopment Process: A New Look, Washington, DC: National Academies Press. Available online at http://www.nap.edu/catalog.php?record_id=11893. Last accessed August 20, 2010.

OCR for page 1
 SUMMARY adopted earned value management (EVM), which is “a means of determining the financial health of a project by measuring whether the work completed to date is in line with the budget and schedule planning.” One of the reasons for using EVM is to get early warning of potential problems. EVM tracks plans, progress, cost, earned value (the planned cost of actual progress), and variances in cost and sched- ule. The underlying technique is seemingly straightforward, but the application of EVM to innovative and unprecedented software-intensive systems poses challenges. These derive from the choice of EVM assessment and measurement strategies. Significant improvements are needed in our ability to value the creation of software assets such as validated architecture and design commitments or evidence in support of quality assurances. Finding 2-3: Extensions to earned value management models to include evidence of feasibility and to accommodate practices such as time-certain development are necessary conditions to enable suc - cessful application of incremental development practices for innovative systems. Finding 2-4: Research related to process, measurement, architecture, and assurance can contribute to the improvement of measurement practice in support of both routine management of engineering risks and value assessment as part of earned value management. The committee focuses (in Chapter 2) on six areas for improvement in the management of innova - tive software projects: (1) improved measurement and associated technology, (2) architecture validation using models, simulation, prototyping, etc., (3) program manager training and perceived career risks, (4) accretion of an accessible experience base and other shared resources that can facilitate sound deci - sion making over the long term, (5) acceptable shifts of early-stage emphasis for innovative systems from detailed functional requirements to architecture, scope, and process definition, and (6) the need for flexibility and adaptation in long-lived projects. Recommendation 2-1: The DoD should take aggressive actions to identify and remove barriers to the broader adoption of incremental development methods, including iterative approaches, staged acquisition, evidence-based systems and software engineering, and related methods that involve explicit acknowledgment and mitigation of engineering risk. An additional difficulty is the lack of a common basis for judging cost estimates. There are well- used metrics for hardware, but a uniform set of standards for measurement in software development is lacking, although there are candidate models. Recommendation 2-2: The DoD should take steps to accumulate high-quality data regarding project management experience and technology choices that can be used to inform cost estimation models, particularly as they apply to innovative software development projects. It is widely acknowledged, including within the DoD, that the department does not have sufficient organic personnel with the software expertise to meet its needs for today’s more software-intensive programs. This includes the expertise to effectively purchase the larger and less precedented systems as well as the precedented systems for which sensitivity to issues such as the choice of ecosystem is key. The necessary expertise includes understanding of process, architecture, requirements, and assurance, as well as of the trajectories and adoption trends for both the major commercial ecosystems and any involved DoD-intrinsic software ecosystems. Because the DoD does not currently have the requisite expertise and talent it needs for effective software producibility and the rapid pace of software devel - opment demands ongoing interaction with the field, the DoD must engage experts outside of the DoD and its primes. The DoD should adapt processes to facilitate input from outside experts throughout the systems-engineering lifecycle for software-intensive systems.

OCR for page 1
 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE Finding 2-6: The DoD has a growing need for software expertise, and it is not able to meet this need through intrinsic resources. Nor is it able to fully outsource this requirement to DoD primes. The DoD needs to be a smart software customer. This need is particularly significant for large-scale innovative software-intensive projects for which there are cross-cutting software architectural requirements and validation challenges. 3. ASSERT DOD ARCHITECTURAL LEADERSHIP The increasing complexity and scale of innovative software systems demand that the DoD play an active role in the definition of systems and software architecture throughout the project lifecycle. Soft - ware architecture is conventionally defined as “the structure or structures of the system, which comprise software components, the externally visible properties of those components, and the relationship among them.”9 Architecture is significant because it represents the earliest and often most important design decisions: those that are the hardest to change and the most critical to get right. Architecture is the first design artifact that addresses quality attributes such as performance, modifiability, reliability, security, and safety. Although having a well-matched architecture is not a guarantee of success, software systems that are not based on well-formulated software architectures are, in the committee’s view, more likely to exhibit the kind of software horror stories too often experienced in DoD acquisitions with respect to project risk. Finding 3-5: In systems with innovative functional or quality requirements, benefit is derived from an early focus on the most essential architectural commitments and quality attributes, with deferred commitment to specifics of functional characteristics. This approach can reduce the overall uncer- tainty of the engineering process and yield better outcomes. Architectural decision making for any particular software development project is profoundly influ - enced by precedent—knowledge of related ecosystems, of systems and hardware infrastructure, of available frameworks and libraries, and of previous experience with similar systems and projects. Small changes to architectural requirements can open or close opportunities to exploit rich, existing ecosys - tems, greatly influencing both cost and risk.10,11 Finding 3-1: Industry leaders attend to software architecture as a first-order decision, and many follow a product-line strategy based on commitment to the most essential common software architectural elements and ecosystem structures. Architecture embodies planning for flexibility—architecture commitments effectively define and encapsulate areas where change or diversity is anticipated, or not. Software architecture commitments thus enable product-line strategies. Finding 3-2: The technology for definition and management of software architecture is sufficiently mature, with widespread adoption in industry. These approaches are ready for adoption by the DoD, assuming that a framework of incentives can be created in acquisition and development efforts. The DoD experience with long-term software acquisition programs has provided strong evidence for the value of software architecture,12 and there are examples of programs that have followed an archi- 9 Len Bass, Paul Clements, and Rick Kazman, 2003, Software Architecture in Practice, 2nd Ed., Boston: Addison-Wesley. 10 Dennis M. Buede, 2000, The Engineering Design of Systems: Models and Methods, New York: John Wiley & Sons, Inc., pp. 7-8, 25. 11 Barry Boehm, Ricardo Valerdi, and Eric Honour, 2008, “The ROI of Systems Engineering: Some Quantitative Results for Software-Intensive Systems,” Systems Engineering 11(3):221-234. 12 Walker E. Royce, 1998, Software Project Management: A Unified Framework , Reading, MA: Addison-Wesley.

OCR for page 1
 SUMMARY tecture-driven acquisition strategy. These illustrate the benefits of pervasive commitment to an architec - ture-driven approach13,14 — including reduced engineering risk, reduced development and maintenance costs, decreased time to field, increased system agility, and improved system quality. The opportunity exists for the DoD to assert leadership across its diverse software-intensive systems portfolio. It may be difficult to ascertain which kinds of architectural commitments are essential to an inno - vative project—at the outset of a project, a small number of well-crafted “seed” commitments may be sufficient to enable a direction to be set. Generally speaking, architecture in the early stages of an innovative project should be the minimum commitment that yields the maximum value with respect to quality attributes and capability to incrementally implement functional capabilities. Refinement and elaboration—further architectural commitment— is then undertaken as part of an incremental iterative process. A corollary of this approach is that architecture leadership is best undertaken by individuals engaged directly in the engineering process and is best separate from activities related to ecosystems certification and other standards-related policy setting. Recommendation 3-2: This committee reiterates the past Defense Science Board recommendations that the DoD follow an architecture-driven acquisition strategy, and, where appropriate, use the software architecture as the basis for a product-line approach and for larger-scale systems potentially involving multiple lead contractors. Recommendation 3-3: The DoD should enhance existing practices to afford better distinctions between precedented portions of systems and innovative portions of systems, wherein architectures are developed both to encapsulate the innovative elements and to afford maximum opportunity to build on experience and existing ecosystems for precedented elements. These overall architectures, and particularly the innovative elements, should be subject to early and continuous validation, especially in systems that have requirements for interoperation. 4. ADOPT A STRATEGIC APPROACH TO SOFTWARE ASSURANCE One of the great challenges for both defense and civilian systems is software quality assurance. Software assurance encompasses reliability, security, robustness, safety, and other quality-related attri - butes as well as functionality and performance. Diverse studies suggest that overall software assurance costs account for 30 to 50 percent of total project costs for most software projects. 15 Despite this cost, current approaches to software assurance, primarily testing and inspection, are generally regarded as 13 Mark Kasunic, 2004, Army Strategic Software Improement Program (ASSIP) Surey of Army Acquisition Managers, Technical Report, Carnegie Mellon University/Software Engineering Institute (SEI), CMU/SEI-2004-TR-003. Available online at http://www.sei. cmu.edu/library/abstracts/reports/04tr003.cfm. Last accessed August 20, 2010. 14 Peter H. Feiler and Dionisio de Niz, 2008, ASSIP Study of Real-Time Safety-Critical Embedded Software-Intensie System, Engi- neering Practices, Special Report, Carnegie Mellon University/SEI, CMU/SEI-2008-SR-001. Available online at http://www.dtic. mil/cgi-bin/GetTRDoc?Location=U2&doc=GetTRDoc.pdf&AD=ADA480129. Last accessed August 20, 2010. 15 In “Software Debugging, Testing, and Verification” ( IBM Systems Journal (41)1, 2002), Brent T. Hailpern and P. Santhanam say, “In a typical commercial development organization, the cost of providing this assurance via appropriate debugging, testing, and verification activities can easily range from 50 to 75 percent of the total development cost.” In Estimating Software Costs (McGraw- Hill, 1998), T. Capers Jones provides a table relating percentage of defects removed vs. percentage of development effort devoted to testing, with data points, including 90 vs. 39, 96 vs. 48, and 99.9 vs. 58. In Software Cost Estimation with COCOMO II (Prentice Hall, 2000), Barry W. Boehm, Chris Abts, A. Winsor Brown, Sunita Chulani, Bradford K. Clark, Ellis Horowitz, Ray Madachy, Donald Reifer, and Bert Steece indicate that the cost of test planning and running tests is typically 20 to 30 percent plus rework due to defects discovered. In Balancing Agility and Discipline: A Guide for the Perplexed (Addison-Wesley, 2004), Barry Boehm and Richard Turner provide an analysis of the COCOMO II Architecture and Risk Resolution scale factor, indicating that the increase in rework due to poor architecture and risk resolution is roughly 18 percent for typical 10-KSLOC (KSLOC stands for thousand software lines of code) projects and roughly 91 percent for typical 10,000-KSLOC projects. (COCOMO II, or constructive cost model II, is a software cost, effort, and schedule estimation model.) This analysis suggests that improvements are needed in upfront areas as well as in testing and supporting the importance of architecture research, especially for ultra-large systems.

OCR for page 1
 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE inadequate. Testing, for example, cannot yield assurance for many kinds of failures related to security or non-determinism. In defense programs, the assurance process, including particularly the use of preventive approaches, is heavily complicated by the arms-length relationship that exists between a contractor development team and government stakeholders. Additionally, although the DoD relies extensively on vendor software and undertakes considerable testing of that software, it also implicitly relies on relationships founded in trust (rather than verification) to assure many quality attributes. 16 Failures in software assurance can be of particularly high consequence for defense systems due to their growing roles in protecting human lives, in war fighting, and in safeguarding national assets. In many life and death situations, optimum performance may not be the proper overriding assurance cri - terion, but rather the “minimization of maximum regret.” This is exacerbated by the fact that a full-scale operational test of many capabilities is not feasible, but assurance must nonetheless be achieved. Software assurance is a human judgment of fitness for use. For defense systems, there is particu - lar emphasis on addressing hazards related to security, availability and responsiveness, safety, policy adherence, and diverse other attributes, but there are many other quality attributes encompassed by software assurance. In practice, assurance judgments are based on application of a broad range of tech - niques that include both preventive and evaluative methods and that are applied throughout a software engineering process. It is false to conclude that assurance can be achieved entirely through acceptance evaluation such as achieved through DoD’s operational and systems test processes. In particular, it is well understood by software engineers and managers that quality, including security, is not “tested in,” but rather is “built in.” But there are great challenges to succeeding both in building in quality (preven - tive methods) and in assuring that it is there (evaluative methods). From a process perspective, there is overlap between preventive and evaluative methods—when used at the earliest stages in the process, evaluative methods shorten feedback loops and guide development choices. Development practices and technologies can profoundly influence the ability to achieve successful and cost-effective evaluation outcomes. These development choices range from choices of architecture to choices of programming language, coding style, and associated tooling. One of the great benefits of modern tooling is that a much more comprehensive record of development can be used to facilitate evaluation. Software assurance is different from reliability analysis for physical systems. Unlike other engineer- ing materials, software does not wear out or suffer transient faults. But it can suffer transient errors, for example, because of concurrency. This is both an obvious and a subtle point. It is obvious in the sense that there is no analog of metal fatigue, rust and oxidation, or other kinds of physical deteriora - tion or environmentally induced change in physical properties. It is subtle because software is often the mechanism of choice for handling such faults in associated hardware. When software delivers bad results, including transient errors, they are due to permanently faulty software design, which must be addressed by changes in the software code. The goal of assurance methods is to ultimately connect the code that is executed with architectural, functional, and quality requirements. Although software code is all that is necessary for the software to operate, considerable additional information is needed to effectively support ongoing evolution of the software over its lifespan, including architecture models, designs, test cases, etc. This information sup - ports an incremental process, in which chains of evidence can be created with links among the artifacts being created (and adapted) as the development process proceeds. Validation of these traceability links comes from diverse techniques including testing, inspection, analysis, model checking, and simulation. An example of a link is a test case that connects code with a particular expectation regarding behavior at an internal software interface. Advancement in research and practice could lead to chains of evidence 16 DSB, September 2007, Report of the Defense Science Board Task Force on Mission Impact of Foreign Influence on DoD Software, Wash- ington, DC: Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics. Available online at http://stinet. dtic.mil/oai/oai?&verb=getRecord&metadataPrefix=html&identifier=ADA473661. Last accessed August 20, 2010.

OCR for page 1
 SUMMARY that could both support quality claims and protect trade secrets or proprietary technology in compo - nents. For example, traceability links can include modeling with respect to various attributes, as well as analyses that link models with each other and with code. These links, in aggregate, can create chains of evidence as noted above. Software assurance (and producibility generally) are influenced not only by the extent of this design- related information but also by the means by which it is represented. There are four dimensions of rep - resentation that are most significant—formality (precise structure and meaning), modeling (reasoning about diverse aspects of a system), consistency (among various artifacts), and usability (feasibility for use by working development teams). Finding 4-1: The feasibility of achieving high assurance for a particular system is strongly influenced by early engineering choices, particularly architectural and tooling choices. Finding 4-2: Assurance is facilitated by advances in diverse aspects of software engineering practice and technology, including modeling, analysis, tools and environments, traceability, programming languages, and process support. Advances focused on simultaneous creation of assurance-related evidence with ongoing development effort have high potential to improve the overall assurance of systems. Because modern systems of all kinds draw on diverse components from diverse sources, there will necessarily be differences in the levels of trust conferred on both components and suppliers. This means that, in the parlance of cybersecurity, there are potential attack surfaces from within the software appli - cation as well as from the outside and that we must support rigorous defense at the interfaces within the application. In other words, the new perimeter is within the application rather than around it or its platform. Recommendation 4-1: Effective incentives for preventive software assurance practices and produc - tion of evidence across the lifecycle should be instituted for prime contractors and throughout the supply chain. Recommendation 4-2: The DoD should expand its research focus on and its investment in both fundamental and incremental advances in assurance-related software engineering technologies and practices. Recommendation 4-3: The DoD should examine commercial best practices for more rapidly tran - sitioning assurance-related best practices into development projects, including contracted custom development, supply-chain practice, and in-house development practice. 5. REINVIGORATE DOD SOFTWARE ENGINEERING RESEARCH The committee identified seven technology areas where research progress would make a difference for DoD’s software capability. • Architecture modeling and architectural analysis. Goals: (1) Facilitation of early validation for archi- tecture decisions, including measures, modeling and evaluation, and compliance. (2) Facilitation of architecture-aware systems management, including models of congruence and a means to manage rich supply chains, ecosystems, and infrastructure. (3) Facilitation of component-based development, includ- ing architectural designs for particular domains. • Assurance: alidation, erification, and analysis of design and code. Goals: (1) Effective evaluation for critical quality attributes. (2) Assurance for components in large heterogeneous systems. (3) Enhanced

OCR for page 1
0 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE portfolio of preventive methods to achieve assurance, ranging from process improvement and architec - tural building blocks to programming languages and coding practice. • Process support and economic models for assurance and adaptability. Goals: (1) Enhanced process sup- port for assured software development. (2) Models for evidence production in software supply chains. (3) Application of economic principles to process decision making. • Requirements. Goals: (1) More expressive models and supporting tools for both functional and quality attributes. (2) Improved support for traceability and early validation. • Language, modeling, coding, and tools. Goals: (1) Enhanced expressiveness of programming lan- guages to address current and emerging challenges. (2) Enhanced ability to exploit modern concur- rency, including shared memory multicore and scalable distributed memory. (3) Enhanced developer productivity for new development and evolution. • Cyber-physical systems. Goals: (1) Accelerated development of new conventional architectures for control systems. (2) Improved architectures for a wide range of embedded applications. • Human-system integration. Goal: (1) Development of engineering practices for complex systems in which humans play critical roles. This area is elaborated in another NRC report. 17 The committee made its selection of these seven technical areas on the basis of four considerations: (1) capabilities identified to have significant potential value through the committee’s examination of the key DoD software producibility priorities: process, measurement, architecture, and assurance, as reported in Chapters 2, 3, and 4; (2) capabilities that can be feasibly developed through a well-managed research program, based on accepted research management criteria (such as the Heilmeier questions for research program managers who propose new program ideas—see Chapter 5); (3) capabilities not addressed sufficiently by other federal agencies; and (4) capabilities that might not develop at a suf - ficient pace without explicit added investment. The proposed research would be undertaken by a mix of academia, government labs, and industry. Academic research has historically had a particular role in advancing DoD technical capability, through both research and expertise, and this role persists for software producibility. Finding 5-1: Academic research and development continues to be the principal means for develop - ing the most highly skilled members of the software workforce, including those who will train the next generation of leaders, and for stimulating the entrepreneurial activity that leads to disruptive innovation in the information technology industry. Both academic and industry labs are creating the fundamental advances in knowledge that are needed to drive innovation leadership in new technologies and to advance software technologies that are broadly applicable across industry and the DoD supply chain. Directions and priorities for university-originated invention are greatly influenced by funding levels and agency priorities. For example, the Defense Advanced Research Projects Agency’s (DARPA’s) deliberately strong relationship with the information technology (IT) research community, which began in the 1960s and endured for nearly 40 years, profoundly influenced IT research priorities, the overall culture of computer science research, and the substantial economic and social outcomes that resulted. This relationship is documented in NRC reports that trace the origins of IT innovations, each of which has led to a multibillion-dollar market.18 17 See NRC, Richard Pew and Anne Mavor, eds., 2007, Human-System Integration in the System Deelopment Process: A New Look, Washington, DC: National Academies Press. Available online at http://www.nap.edu/openbook.php?record_id=11893. Last accessed August 20, 2010. 18 See NRC, 2003, Innoation in Information Technology, Washington, DC: National Academies Press. Available online at http:// www.nap.edu/catalog.php?record_id=10795. Last accessed August 20, 2010. See also the predecessor report, NRC, 1995, Eoling the High Performance Computing and Communications Initiatie, Washington, DC: National Academy Press. Available online at http://www.nap.edu/catalog.php?record_id=4948. Last accessed August 20, 2010.

OCR for page 1
 SUMMARY Data from the Networking and Information Technology Research Development (NITRD) program and other sources indicate that there has been a significant reduction in federally sponsored research related to software producibility. From 2004 to 2010, overall funding for the NITRD program more than doubled. During the same period, the combined dollar allocation to the two categories most relevant to software producibility was reduced by almost half.19 (See Box 1.5 for details.) Expressed as a percent- age of the total NITRD budget, the combined allocation for the software-related categories dropped from 24.6 percent to 6.5 percent. Furthermore, it is the committee’s impression that in recent years, as a consequence of these reductions, many of the researchers in these areas have moved into other fields or scaled down their research efforts. Recommendation 5-1: The DoD should take immediate action to reinvigorate its investment in soft - ware producibility research. This investment should be undertaken through a diverse set of research programs throughout the DoD and should include academia, industry labs, and collaborations. It is important that researchers understand the challenges associated with the way the DoD develops software.20 This includes not only the particular technical challenges, but also the influences of factors such as the arm’s-length relationship between the DoD and the contractors doing the development. DoD research agencies have instituted programs to help younger faculty get the needed domain exposure. These are important to continue and broaden if university programs are to be relevant. Finding 5-2: Technology has a significant role in enabling modern incremental and iterative software development practices at levels of scale ranging from small teams to large distributed development organizations. There are significant and particular difficulties in managing research in topics related to software producibility. But there are also major opportunities based on recent progress in the field, including technology developments, scientific practice, and the overall environment of production practice. Taking challenges and opportunities together, the influences include (1) the maturation of software engineering as a discipline, leading to improved research methods and lower risk in technology transition—facilitat - ing more satisfactory responses to the Heilmeier questions; (2) the complexity of diffusion pathways and the variability of timescales, where some results can readily transfer to DoD practice, while others, often the most significant and influential, take longer and have more indirect pathways; (3) an emerg - ing concept of novelty that is often more closely tied with readiness with respect to infrastructure and the various exponential curves than with specific technical novelty—the question is often, What are the ideas whose time has come? (4) improved methods to assess progress in the absence of crisp quantita - tive measures of performance (e.g., how to assess the benefits of strong typing in a quantitative way) or when the focus of research is on developing such measures; and (5) unpredictability in the span of time from the emergence of a new idea to the readiness to transition that idea with respect to practice, infrastructure, and other variables. An additional difficulty is the development of models of return on investment in research related to software producibility. This difficulty is present for all investment in basic science and exploratory devel- opment, but it can be particularly vexing for computing technology and software. This difficulty has been the subject of intense study by the National Research Council and other groups; several reports have been produced that offer a historical perspective, showing the emergence of multiple multibillion-dollar 19 These categories are Software Design and Productivity (SDP) and High Confidence Software and Systems (HCSS). The reported amounts for SDP and HCSS do not include 2010 NIH funding for accounting reasons that are explained in Chapter 1. Comparisons are in constant dollars. 20 A brief description of such challenges can be found on p. 23 in NRC, 2010, Achieing Effectie Acquisition of Information Technol- ogy in the Department of Defense, Washington, DC: National Academies Press. Available online at http://www.nap.edu/catalog. php?record_id=12823. Last accessed August 20, 2010.

OCR for page 1
 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE markets on the basis of initial investment in worthy projects by NITRD agencies. Under the auspices of professional societies, similar studies were published relating specifically to software engineering research.21 These reinforce the extent of the impact of well-managed investments. Recommendation 5-2: The DOD should take action to undertake DoD-sponsored research programs in the following areas identified as critical to the advancement of defense software producibility: (1) architecture modeling and architectural analysis; (2) assurance: validation, verification, analy - sis of design and code; (3) process support and economic models for assurance and adaptability; (4) requirements; (5) language, modeling, coding, and tools; (6) cyber-physical systems; and (7) human- systems integration. 21 Mary Shaw, 2002, “The Tyranny of Transistors: What Counts about Software?” Proceedings of the Fourth Workshop on Economics- Drien Software Engineering Research, IEEE Computer Society, pp. 49-51; Barry Boehm, 2006, “A View of 20th and 21st Century Software Engineering,” Proceedings of the th International Conference on Software Engineering, ACM, pp. 12-29; and Leon J. Osterweil, Carlo Ghezzi, Jeff Kramer, and Alexander L. Wolf, 2008, “Determining the Impact of Software Engineering Research on Practice,” IEEE Computer 41(3):39-49.

OCR for page 1
 SUMMARY Box S.2 Compilation of Report Findings and Recommendations Chapter 1 Finding 1-1: Software has become essential to a vast range of military system capabilities and operations, and its role is continuing to deepen and broaden, including interlinking diverse system elements. This creates both benefits and risks. Finding 1-2: The growth in the role of software in systems is due to a combination of technological advances and a maturing of the supply chain structure associated with software systems development at all levels of scale. Finding 1-3: The DoD relies fundamentally on mainstream commercial components, supply chains, and software ecosystems for both business systems and many mission systems. Nonetheless, the DoD has special needs in its mission systems driven by the growing role of software in systems. As a result, the DoD needs to address directly the challenge of building on and, where appropriate, contributing to the development of mainstream software that can contribute to its mission. Finding 1-4: The DoD’s needs will not be sufficiently met through a combination of demand-pull from the military and technology-push from the defense or commercial information technology sectors. The DoD cannot rely on industry alone to address the long-term software challenges particular to defense. Recommendation 1-1: The DoD, through its Director of Research and Engineering (DDR&E), should regularly undertake an identification of areas of technological need related to software producibility where the DoD has “leading demand” and where accelerated progress is needed to support the de- fense mission. Finding 1-5: It is dangerous to conclude that we are reaching a plateau in capability and technology for software producibility. To avoid loss of leadership, the DoD will need to become more fully engaged in the innovative processes related to software producibility. Chapter 2 Finding 2-1: Modern practice for innovative software systems at all levels of scale is geared toward incremental identification and mitigation of engineering uncertainties, including requirements uncer- tainties. For defense software, the challenge is doing so at a larger scale and in ways that are closely linked with an overall systems engineering process. Finding 2-2: The prescription in DoD Instruction 5000.02 for the use of evolutionary development needs to be supplemented by the development of related guidance on the use of such practices as time- certain development, requirements prioritization, evidence-based milestones, and risk management. Finding 2-3: Extensions to earned value management models to include evidence of feasibility and to accommodate practices such as time-certain development are necessary conditions to enable success- ful application of incremental development practices for innovative systems. Finding 2-4: Research related to process, measurement, architecture, and assurance can contribute to the improvement of measurement practice in support of both routine management of engineering risks and value assessment as part of earned value management. continued

OCR for page 1
4 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE Box S.2 Continued Recommendation 2-1: The DoD should take aggressive actions to identify and remove barriers to the broader adoption of incremental development methods, including iterative approaches, staged acqui- sition, evidence-based systems and software engineering, and related methods that involve explicit acknowledgment and mitigation of engineering risk. Recommendation 2-2: The DoD should take steps to accumulate high-quality data regarding project management experience and technology choices that can be used to inform cost estimation models, particularly as they apply to innovative software development projects. Finding 2-5: Architectural expertise is becoming dramatically more important for the DoD, its advisors, and its contractors. There will be significant and immediate benefits from advances in the state of technical support for architecture. Recommendation 2-3: Update procurement, contracting, and governance methods to include an ear- ly and explicit architecture phase that reduces the predominant uncertainties in software intensive systems. Recommendation 2-4: Define architectural leadership roles for major SIDRE projects and provide pro- gram managers with channels for architectural expertise. Recommendation 2-5: Develop the technical and management infrastructure necessary to simultane- ously support stabilized, high-assurance development of the current evolutionary increment while concurrently evolving the plans and specifications for stabilized development of the next high-assur- ance increment. Finding 2-6: The DoD has a growing need for software expertise, and it is not able to meet this need through intrinsic resources. Nor is it able to fully outsource this requirement to DoD primes. The DoD needs to be a smart software customer. This need is particularly significant for large-scale innovative software-intensive projects for which there are cross-cutting software architectural requirements and validation challenges. Chapter 3 Finding 3-1: Industry leaders attend to software architecture as a first-order decision, and many follow a product-line strategy based on commitment to the most essential common software architectural elements and ecosystem structures. Finding 3-2: The technology for definition and management of software architecture is sufficiently mature, with widespread adoption in industry. These approaches are ready for adoption by the DoD, assuming that a framework of incentives can be created in acquisition and development efforts. Finding 3-3: The DoD would benefit from explicit attention to software architecture and industry best practice, including (1) formalizing career paths and role descriptions for software architects, (2) iden- tifying ways that DoD-aligned software architects can provide objective advice (see Chapter 2), and (3) enhancing organizational structures to support effective architectural leadership. Finding 3-4: Several DoD programs are using software architecture-driven acquisition with successful results.

OCR for page 1
 SUMMARY Box S.2 Continued Recommendation 3-1: Initiate a targeted research program to provide software architects with better tools and techniques for DoD systems. Recommendation 3-2: This committee reiterates the past Defense Science Board recommendations that the DoD follow an architecture driven acquisition strategy, and, where appropriate, use the software architecture as the basis for a product-line approach and for larger-scale systems potentially involving multiple lead contractors. Recommendation 3-3: The DoD should enhance existing practices to afford better distinctions be - tween precedented portions of systems and innovative portions of systems, wherein architectures are developed both to encapsulate the innovative elements and to afford maximum opportunity to build on experience and existing ecosystems for precedented elements. These overall architectures, and particularly the innovative elements, should be subject to early and continuous validation, especially in systems that have requirements for interoperation. Finding 3-5: In systems with innovative functional or quality requirements, benefit is derived from an early focus on the most essential architectural commitments and quality attributes, with deferred com- mitment to specifics of functional characteristics. This approach can reduce the overall uncertainty of the engineering process and yield better outcomes. Recommendation 3-4: The DoD should learn from commercial experience and, in addition, sponsor diverse areas of technical research to help reduce the engineering risk in architecting systems that include unprecedented functional and quality attributes. Chapter 4 Finding 4-1: The feasibility of achieving high assurance for a particular system is strongly influenced by early engineering choices, particularly architectural and tooling choices. Finding 4-2: Assurance is facilitated by advances in diverse aspects of software engineering practice and technology, including modeling, analysis, tools and environments, traceability, programming languages, and process support. Advances focused on simultaneous creation of assurance-related evidence with ongoing development effort have high potential to improve the overall assurance of systems. Recommendation 4-1: Effective incentives for preventive software assurance practices and production of evidence across the lifecycle should be instituted for prime contractors and throughout the supply chain. Recommendation 4-2: The DoD should expand its research focus on and investment in both fundamental and incremental advances in assurance-related software engineering technologies and practices. Recommendation 4-3: The DoD should examine commercial best practices for more rapidly transition- ing assurance-related best practices into development projects, including contracted custom develop- ment, supply chain practice, and in-house development practice. continued

OCR for page 1
 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE Box S.2 Continued Chapter 5 Finding 5-1: Academic research and development continues to be the principal means for developing the most highly skilled members of the software workforce, including those who will train the next generation of leaders, and for stimulating the entrepreneurial activity that leads to disruptive innovation in the information technology industry. Both academic and industry labs are creating the fundamental advances in knowledge that are needed to drive innovation leadership in new technologies and to advance software technologies that are broadly applicable across industry and the DoD supply chain. Finding 5-2: Technology has a significant role in enabling modern incremental and iterative software development practices at levels of scale ranging from small teams to large distributed development organizations. Recommendation 5-1: The DoD should take immediate action to reinvigorate its investment in software producibility research. This investment should be undertaken through a diverse set of programs across the DoD and should include academia, industry labs, and collaborations. Recommendation 5-2: The DoD should take action to undertake DoD-sponsored research programs in the following areas identified as critical to the advancement of defense software producibility: (1) architecture modeling and architectural analysis; (2) assurance: validation, verification, analysis of design and code; (3) process support and economic models for assurance and adaptability; (4) require- ments; (5) language, modeling, coding, and tools; (6) cyber-physical systems; and (7) human-systems integration.