Summary of a Workshop on Software-Intensive Systems and Uncertainty at Scale (2007)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

3 Wrap-up Discussion and Emergent Themes S everal major themes emerged from the day’s discussions on the challenges of developing future-oriented, large-scale systems that can cope with uncertainty at scale. These themes are not findings or recommendations of the study committee—those will be presented in the committee’s final report later in the study. Indeed, observations offered in some sessions contradicted those from other sessions—which perhaps reflects the notion that there are different kinds of very large-scale systems and that different kinds of systems will likely warrant different sorts of approaches. Instead, this section presents a brief overview of the inter­ related themes that arose over the course of the workshop’s discussions: • Architectural challenges in large-scale systems, • The need for software engineering capability, and • Open questions and research opportunities. Architectural Challenges in Large-Scale Systems The issue of architecture and frameworks for large-scale software- intensive systems coping with uncertainty at scale was raised repeatedly. One approach to this problem that was put forward in discussions would Because neither the workshop nor this summary was intended to be (or constituted as) a stand-alone product, contradictory views also emerged from different presenters during the day—for example, the desirability of not producing software in-house versus the desirability of producing all software in-house. 40 

WRAP-UP DISCUSSION AND EMERGENT THEMES 41 be to develop executable models and architectures that can evolve over the system’s life cycle. As the system develops and evolves, pieces that were originally mock-ups could be replaced with actual subsystems—this would be a way to provide ongoing feedback on the architecture, includ- ing system performance, functionality, and so on. A broader question asks what kind of structure and constraints can be imposed at a high level such that the overall system can be decomposed (as needed) into autonomous pieces. What specific architectural rules and approaches will get one there? It was suggested that a framework would be needed, along with clear specifications at the interface between the framework and components. Closely tied to the question of architecture is the significant challenge of how to develop systems architectures and definitions for highly decen- tralized organizations. For developing the sorts of systems under consid- eration, some believed a loosely coupled organizational style would be needed as systems scale up and more players enter the picture. Although such a shift might place intense pressure on organizational culture and management, “controlled chaos” rather than a very top-down, structured, and controlled approach might need to become the order of the day. This shift, however, might reflect a change of perspective on the essential com- monalities that hold an overall system together—there could, for example, be a shift from an overall structural model as the unifying factor to par- ticular agreements on how components and subsystems are to interact with each other through protocols, APIs, metadata standards, and the like. It may also be the case that there is an underlying issue driving this ten- sion that is not about coupling or control, but rather about the particular nature of the architectural commonalities that hold a system together. For example, the trend towards dynamic architecture demonstrates that purely structural commonalities are giving way to semantic and other less apparent—but perhaps more essential—commonalities in large systems. This has implications for software engineering capability (see below), in part because the frameworks and architecture for these systems will not go away—they and the people involved with them will need to persist for the lifetime of the system. And, of course, the architecture for these systems will transcend particular individual suppliers. The Need for Software Engineering Capability Writing large, safety-critical, real-time systems requires a commitment to genuine engineering discipline, even if it means constraining the design space—limiting flexibility—in order to conform to precedented prac- tices that enable application of this discipline. In addition, management becomes a significant challenge when it comes to very large-scale systems 

42 SOFTWARE-INTENSIVE SYSTEMS AND UNCERTAINTY AT SCALE that will need to depend on very large networks and supply chains,. It was suggested that this may lead to a focus on community-style collabo- ration and integration over the long term. In any event, best practices will differ depending on context, including the type of organization as well as the type of application or system under development. Participants noted that process practices will merit attention as well as technological prac- tices. Process will not solve everything, of course, but process is vital to assist with, among other things, the decomposition of large systems into incremental subsets for better visibility and reduced risk. It was noted that the focus on reuse may shift to earlier stages of the process and particu- larly to requirements. In general, supporting technology will be needed to enable the types of architectures and collaborations that large-scale systems merit. In particular, finding ways to increase the extent of abstrac- tion and automation in all aspects of software design, implementation, testing, maintenance, and so on, may be a productive avenue, especially with regard to scale and geographic distribution. Tools and strategies for coping with design and architectures for very large, physically distributed teams of people and organizations will be increasingly important along with techniques and approaches that support high levels of trust. In addi- tion, participants noted that the ability to use analytic techniques (such as model checking, static analysis, dynamic analysis, and so on) will be important. These types of tools are being used much more now than 5 or 10 years ago, but they are typically still early-generation tools. Continued research and investment (see below) will help improve and extend them. Even so, tools will not solve the problems of assurance or of large-scale system development. Process, expertise, and skills matter a great deal. Open Questions and Research Opportunities Ultimately, many of the challenges related to architecture and capabil- ity reflect problems that the community does not yet know how to solve. Over the course of the workshop, participants articulated several open research questions that should be addressed to make progress in address- ing uncertainty at scale for software-intensive systems. At a high level, the architectural and organizational challenges presented by large-scale systems merit investigation: What are the key kinds of commonalities that manifest architectural commitment, beyond a top-down structural design? Industry issues were among the other topics that were spot- lighted in the wrap-up discussion. These included (1) the extent to which service-oriented architecture (SOA) and SOA vendors will or will not make progress over the next 18 to 24 months in addressing DoD’s need for producibility at scale—for example, in contrast to the well-established enterprise resource planning framework and application servers offered 

WRAP-UP DISCUSSION AND EMERGENT THEMES 43 by vendors—and (2) the extent to which defense contractors can allocate resources to address software challenges that fall outside current contract parameters. Contracting issues that were noted in the discussion included (1) how to establish collaborative mechanisms for contractors and the DoD to work together, particularly in iterative fashion, on software assur- ance and risk-reduction problems, as well as (2) contracting complexities related to the integration of large software systems that include COTS subsystems for the DoD. Another industry issue noted in the discus- sion was industry’s misgivings about the availability of computer sci- ence and computer engineering new hires at the bachelor’s and master’s degree level. Related issues were software curricula development and the appropriateness of accreditation for software engineering and computer engineering and the need for computer science and computer engineering faculty and students to have hands-on industry experience in building systems. *** Discussions at the workshop suggested that many of the key ideas needed to make progress in developing large-scale systems and coping with uncertainty at scale will not be found through the traditional incre- mental advances made in various segments of the industry. While there are lessons to be learned and gleaned from the varieties of experience and perspective presented, the types of systems that are envisioned and that serve as a driver for DoD’s interests in software pose significant manage- ment, intellectual, and research challenges.