National Academies Press: OpenBook

An Assessment of Space Shuttle Flight Software Development Processes (1993)

Chapter: Appendix D: Overview of ASET IV & V Methodology

« Previous: Appendix C: Interim Report of the Committee for Review of Oversight Mechanisms for Space Shuttle Flight Software Process: Independent Verification and Validation for Space Shuttle Flight Software
Suggested Citation:"Appendix D: Overview of ASET IV & V Methodology." National Research Council. 1993. An Assessment of Space Shuttle Flight Software Development Processes. Washington, DC: The National Academies Press. doi: 10.17226/2222.
×

APPENDIX D

Overview of
ASET IV&V Methodology

Briefing Document Given to the Committee
By Intermetrics, Inc.

Suggested Citation:"Appendix D: Overview of ASET IV & V Methodology." National Research Council. 1993. An Assessment of Space Shuttle Flight Software Development Processes. Washington, DC: The National Academies Press. doi: 10.17226/2222.
×
This page in the original is blank.
Suggested Citation:"Appendix D: Overview of ASET IV & V Methodology." National Research Council. 1993. An Assessment of Space Shuttle Flight Software Development Processes. Washington, DC: The National Academies Press. doi: 10.17226/2222.
×

APPENDIX D

OVERVIEW OF ASET IV&V METHODOLOGY1

INTRODUCTION

This paper presents a general description of the technical analysis process used by Intermetrics in performing independent verification and validation (IV&V) of Shuttle flight software under the NSTS Avionics System Engineering Task (ASET) contract. Attachments provide further details on key elements of this methodology.

BACKGROUND

The Intermetrics ASET IV&V effort has, as its principal objective, the identification of potential safety-of-flight issues from within the ongoing flow of Shuttle flight-software changes. Intermetrics is charged with applying a multi-disciplinary, systems perspective to find safety problems that might otherwise go unrecognized. This perspective complements the expertise of the various Shuttle engineering subgroups which concentrate on their particular subsystems or engineering disciplines.

The primary focus of ASET IV&V is on two Shuttle problem reporting and change instruments--Space Shuttle Orbiter Avionics Software Discrepancy Reports (DRs) and Shuttle Software Change Requests (CRs). While these instruments are directed at software, the IV&V analysis of them takes into account the software 's effects on, and interrelationships with, other elements of the avionics system with which the software interacts. This includes the on-board guidance, navigation, and control (GN&C) systems in general, as well as with crew and ground procedures. The principal value added by the ASET IV&V effort is independent technical findings deriving from in-depth understanding of the nature and ramifications of these problems and changes.

The principal technical interface of ASET IV&V is with the Shuttle Avionics Software Control Board (SASCB), which reviews and approves or disapproves all flight-software DRs and CRs. There are typically numerous DRs and CRs considered for each new software build, or Operational Increment (OI), for multiple shuttle flights, and a lesser number that apply to individual flights. The ASET IV&V provides written briefings to the SASCB in the form of Software IV&V Reports (SIRs), and the IV&V personnel routinely attend Board meetings to provide supporting information. These briefings describe the problem or proposed change from a systems standpoint, and present a risk assessment to aid the Board in making its approval decision.

1  

Briefing document given to the Committee by Intermetrics, Inc. A few format changes have been made. Attachments are not included in this Appendix.

Suggested Citation:"Appendix D: Overview of ASET IV & V Methodology." National Research Council. 1993. An Assessment of Space Shuttle Flight Software Development Processes. Washington, DC: The National Academies Press. doi: 10.17226/2222.
×

The ASET IV&V analysts also routinely interact with the general Shuttle flight software and engineering communities. This includes participating in technical reviews and special task force groups working software/avionics problems. In some cases these groups address issues raised by Intermetrics. When warranted, the ASET IV&V analysts will write DRs on safety issues they have found. For changes approved by the SASCB that carry significant risk, followup analyses are performed to evaluate the correctness of the implementation and the adequacy of testing. Updated SIRs are submitted to document these follow-up analyses.

STANDARDIZED METHODOLOGY

Central to the process summarized above is a standardized approach to safety analysis adopted by the ASET IV&V organization. This approach has been devised and refined over the four-year duration of the ASET contract. The framework for the standardized analysis is the Analysis Checklist, Attachment 1.2 The checklist, in turn, contains a key element--Risk Assessment--that is defined in attachment 2. Both are described in the context of a multi-level IV&V concept.

LEVELS OF IV&V ANALYSIS

The ASET IV&V process entails three levels of analysis that correspond to the scope parameters described earlier in this chapter--limited, focused, and comprehensive. These are cumulative in the order presented, that is, focused goes beyond limited, and comprehensive goes beyond focused. For those CRs and DRs that are within scope (as defined below), a risk assessment is performed to determine which level of effort will be applied to a given CR or DR.

Due to the volume of changes and the resource limitations of the ASET contract, it is not possible to perform a complete, comprehensive IV&V on every Shuttle flight-software CR and DR. And, for the same reason, certain categories of problems or changes are ruled out of scope, such as those dealing exclusively with Vehicle Utility (VU) software, System Management/Payload (SM/PL) software, and software development tools. For those CRs and DRs that are within scope, such as the ascent GN&C, entry GN&C, on-orbit GN&C, sequencing, data processing system, and main engine controller, established criteria are applied in selecting the level of analysis to be performed. The criteria and the nature of the analysis are defined below for each of the three levels.

LIMITED ANALYSIS

A Limited analysis consists of determining answers to five basic questions. Listed under the section heading that appear on the SIR, these are as follows:

2  

Attachments are not included in this Appendix.

Suggested Citation:"Appendix D: Overview of ASET IV & V Methodology." National Research Council. 1993. An Assessment of Space Shuttle Flight Software Development Processes. Washington, DC: The National Academies Press. doi: 10.17226/2222.
×
  1. Problem/Change Description

    What is the true nature of the problem being described by a DR or the change being proposed by a CR?

  2. System Impact Analysis

    What is the effect of the problem or the change on the overall Shuttle system?

  3. Requirements Analysis

    For a DR, what requirements/constraints are being violated? For a CR, are the prescribed requirements changes appropriate, correct, and complete?

  4. Risk Assessment

    For a CR, and for a DR resulting in changes, what are the implementation and safety risks associated with implementing the change versus not implementing it? For a DR for which no change is proposed, what is the risk of not finding the problem?

  5. Disposition Analysis

    Is the proposed disposition appropriate?

A Limited analysis is performed on every CR and DR that is within the ASET IV&V scope. From this it is determined if further analysis, in the form of a Focused or Comprehensive analysis, needs to be performed. Limited analysis is deemed sufficient if the CR or DR is low in risk, needs very little or no testing, and requires no code change. Examples of items that fall into this category are DRs that are closed with a program note or waiver. Such DRs may eventually require a Focused or Comprehensive analysis on a later OI when a software change is implemented.

A key portion of this first stage of analysis is risk assessment, as it both aids the SASCB in its approval decision and serves as a basis for determining what further analysis is required. Risk assessment consists of evaluating two types of risk--safety risk and implementation risk. Safety risk is the risk that the system will be less safe with a change than without. Implementation risk is the risk that the change will not be done correctly due to its complexity or other factors. Assessment categorizes both kinds of risk as to whether they are low, medium, or high.

FOCUSED ANALYSIS

A Focused analysis consists of Limited analysis plus determination of answers to the following additional questions:

Suggested Citation:"Appendix D: Overview of ASET IV & V Methodology." National Research Council. 1993. An Assessment of Space Shuttle Flight Software Development Processes. Washington, DC: The National Academies Press. doi: 10.17226/2222.
×
  1. Code Analysis

    Have the code changes been correctly implemented, and do they create any new problems or risks?

  2. Level 6/7 Test/Verification Analysis

    Has development testing, Levels 6 and 7 (the first two levels of official qualification test) demonstrated the correctness and safety of the changes?

  3. Documentation Assessment

    Have all affected documents been changed and are those changes correct and complete as prescribed?

  4. Safety Assessment

    What safety-of-flight issues were revealed by the analysis and what other ones (already known to the program) exist?

A Focused analysis is performed on all CRs of moderate or greater risk and on DRs that require code changes. Focused analysis is generally deemed sufficient for changes that are adequately tested during software development (Levels 6 and 7), that have easily understood requirements, and that do not significantly impact Shuttle hardware of operational procedures.

During the Focused analysis the earlier decision on level of analysis is reevaluated. It may be decided at this point to change the ultimate analysis from Focused to Comprehensive or vice versa.

COMPREHENSIVE ANALYSIS

A Comprehensive analysis consists of Focused analysis plus answering the following additional questions:

  1. Analysis of Other Systems Implementations

    Have other changes besides code (hardware, I-loads, crew procedures, etc.) been correctly implemented, and do they create any new problems or risks?

  2. Complete Test/Verification Analysis

    Have official tests (Levels 6, 7, 8 and SAIL) collectively demonstrated the correctness and safety of the changes?

Suggested Citation:"Appendix D: Overview of ASET IV & V Methodology." National Research Council. 1993. An Assessment of Space Shuttle Flight Software Development Processes. Washington, DC: The National Academies Press. doi: 10.17226/2222.
×

All high risk and selected medium risk changes receive a Comprehensive analysis. These generally include ones for which adequate analysis requires a look at system-level testing (Level 8 and SAIL), that have very complex requirements, or that have significant impact on other systems besides software or on operational procedures. Also included are any late-breaking changes to flight software introduced as patches after Final Load.

KEY FEATURES OF METHODOLOGY

The ASET IV&V methodology includes three major features to enhance efficiency and ensure the quality of the analysis product:

  1. written analysis guidelines

  2. computer-based analysis tools

  3. peer reviews

The analysis guidelines are published in an Intermetrics internal document, the General Analysis Guide, which includes, among other things:

  • a checklist of analysis tasks;

  • guidelines for doing risk assessment;

  • instructions for preparing SIRs; and

  • lists and descriptions of analysis resources.

This guide promotes uniformity and thoroughness in the work of multiple analysts.

The computer-based analysis tools were developed specifically for the ASET IV&V effort and operate on copies of the actual Shuttle flight software downloaded from NASA to local computer systems. Included are parameter tracing, flowcharting, structured display and printout generation, and other tools. Also, a relational data base is used to track the status of all CRs and DRs subject to analysis.

The mechanism of peer review is used for all analyses, regardless of level to ensure the quality of the analysis product. When a SIR has been drafted, a group is assembled consisting of the designated analyst and any supporting analysts that contributed to the SIR, plus an appropriate number of other analysts (peers) from the ASET IV&V group. The draft SIR is evaluated in a supportive atmosphere, using the analysis checklist as a framework. If significant rework is needed a follow-up peer review may also be held. Such peer reviews are conducted when the first stage, Limited analysis is completed prior to SASCB review, and again when the Focused or Comprehensive level analysis has been performed. These peer reviews have been found to contribute significantly both to the motivation of the analyst and to the quality and uniformity of the analysis product.

Suggested Citation:"Appendix D: Overview of ASET IV & V Methodology." National Research Council. 1993. An Assessment of Space Shuttle Flight Software Development Processes. Washington, DC: The National Academies Press. doi: 10.17226/2222.
×
This page in the original is blank.
Suggested Citation:"Appendix D: Overview of ASET IV & V Methodology." National Research Council. 1993. An Assessment of Space Shuttle Flight Software Development Processes. Washington, DC: The National Academies Press. doi: 10.17226/2222.
×
Page 131
Suggested Citation:"Appendix D: Overview of ASET IV & V Methodology." National Research Council. 1993. An Assessment of Space Shuttle Flight Software Development Processes. Washington, DC: The National Academies Press. doi: 10.17226/2222.
×
Page 132
Suggested Citation:"Appendix D: Overview of ASET IV & V Methodology." National Research Council. 1993. An Assessment of Space Shuttle Flight Software Development Processes. Washington, DC: The National Academies Press. doi: 10.17226/2222.
×
Page 133
Suggested Citation:"Appendix D: Overview of ASET IV & V Methodology." National Research Council. 1993. An Assessment of Space Shuttle Flight Software Development Processes. Washington, DC: The National Academies Press. doi: 10.17226/2222.
×
Page 134
Suggested Citation:"Appendix D: Overview of ASET IV & V Methodology." National Research Council. 1993. An Assessment of Space Shuttle Flight Software Development Processes. Washington, DC: The National Academies Press. doi: 10.17226/2222.
×
Page 135
Suggested Citation:"Appendix D: Overview of ASET IV & V Methodology." National Research Council. 1993. An Assessment of Space Shuttle Flight Software Development Processes. Washington, DC: The National Academies Press. doi: 10.17226/2222.
×
Page 136
Suggested Citation:"Appendix D: Overview of ASET IV & V Methodology." National Research Council. 1993. An Assessment of Space Shuttle Flight Software Development Processes. Washington, DC: The National Academies Press. doi: 10.17226/2222.
×
Page 137
Suggested Citation:"Appendix D: Overview of ASET IV & V Methodology." National Research Council. 1993. An Assessment of Space Shuttle Flight Software Development Processes. Washington, DC: The National Academies Press. doi: 10.17226/2222.
×
Page 138
Next: Appendix E: Flight Software Verification and Validation Requirements »
An Assessment of Space Shuttle Flight Software Development Processes Get This Book
×
Buy Paperback | $45.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Effective software is essential to the success and safety of the Space Shuttle, including its crew and its payloads. The on-board software continually monitors and controls critical systems throughout a Space Shuttle flight. At NASA's request, the committee convened to review the agency's flight software development processes and to recommend a number of ways those processes could be improved.

This book, the result of the committee's study, evaluates the safety, oversight, and management functions that are implemented currently in the Space Shuttle program to ensure that the software is of the highest quality possible. Numerous recommendations are made regarding safety and management procedures, and a rationale is offered for continuing the Independent Verification and Validation effort that was instituted after the Challenger Accident.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!