National Academies Press: OpenBook
« Previous: 7 Building and Fire Research Laboratory
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

Chapter 8

Computer Systems Laboratory

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

PANEL MEMBERS

O.R. Pardo, Bechtel/Parsons Brinckerhoff, Chair

Herbert D. Benington, Paramax Systems Corporation

Dorothy E. Denning, Georgetown University

Clarence G. Feldmann, Consultant, Winchester, Massachusetts

James George, Mesa Graphics

Robert E. Kahn, Corporation for National Research Initiatives

Sandra M. Lambert, Citicorp

Roger R.A. Morton, Eastman Kodak Company

Lawrence R. Rabiner, AT&T Bell Laboratories

Michael B. Spring, University of Pittsburgh

Raymond T. Yeh, International Software Systems, Inc.

Submitted for the panel by its Chair, O.R. Pardo, this assessment of the fiscal year 1994 activities of the Computer Systems Laboratory is based on site visits by individual panel members, a formal meeting of the panel on March 9-11, 1994, in the Gaithersburg, Maryland, facilities of the National Institute of Standards and Technology, and on the annual report of the Computer Systems Laboratory.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

LABORATORY OVERVIEW

Mission

The mission of the Computer Systems Laboratory (CSL) is to help government and U.S. industry increase their productive and reliable use of computer and related telecommunications systems, improve security management and technology of computer and related telecommunications systems, and work closely with the users and producers of computer and telecommunications systems to improve the competitive posture of the U.S. computer-related industry.

Strategy

CSL improves the productive and reliable use of computers by developing standards, profiles of standards, guidelines, conformance tests, technical methods, measurement techniques, and prototypes and by providing advice on planning, deployment, and use of new information technologies. CSL improves security management and technology by developing timely and innovative solutions to security problems and technical, management, physical, and administrative standards and guidelines for the cost-effective security and privacy of sensitive information. CSL contributes to U.S. industrial competitiveness by developing and transferring to government and industry generic methods and techniques that facilitate technology development and improvement, collaborating with industry and industrial users to advance standards development and common implementation approaches, performing research to support technical activities, and participating in NIST-wide and government-wide planning for new technology-related initiatives.

Resources

CSL's full-time-equivalent (FTE) staff ceiling for fiscal year 1994 is 259. CSL currently has 282 staff members, of whom 234 are full-time permanent staff. The remainder are part-time permanent and intermittent staff. (Part-time faculty and student staff are counted against the FTE ceiling.) In addition, there are 32 guest researchers. The FTE ceiling limits CSL's performance. Fiscal year 1994 funding is estimated at $17.5 million from NIST Scientific and Technical Research and Services (STRS) funds, $1.1 million from NIST Competence Building Program funds, $0.5 million from NIST Information Technology Services funds, and $18.2 million from other (federal) agency (OA) funds.

CSL has a wide range of computer and communications equipment, some of which is dated, much of which is powerful and current. CSL space at the NIST campus consists principally of the second floor and portions of the basement, first, and third floors of the Technology Building, which it shares with other NIST units. This space is not adequate to meet CSL's laboratory and office requirements.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Assessment of Strategy

CSL's strategy, as stated above, is consistent with its mission and programs. CSL's development of standards, profiles of standards, guidelines, and conformance tests is excellent and relevant to national need. CSL's use of workshops, consortia, and surveys is extensive and should continue.

CSL could improve its strategic and technical planning. The strategies of each division are not completely uniform or clearly stated; however, many of the individual programs have clear, specific strategies. Two divisions have resource-based plans with milestones. The predominant absence of consistent, clearly defined division-level missions, strategies, and plans raises panel concerns as to whether CSL is working on the highest-priority programs, whether the programs represent a complete set, and whether CSL programs have and are achieving long-range goals.

Given that new NIST resources will be distributed by priority, CSL and division missions and strategies should be better focused. CSL program priorities should be based on plans derived from CSL strategy.

CSL programs demonstrate an excellent understanding of the open systems environment (OSE) and open systems communications models, but it is not always clear that CSL's program planning and assessment are based on a clear understanding of the trends in the information technology industry. Also, CSL's planning and assessment are unduly influenced by OA funding sources. If CSL's budget shifts to greater STRS funding, CSL should develop models of its customer base and technology base to assure that its technical programs mirror the needs of its clientele.

Technical planning based on such models promotes top-down planning; however, given CSL's historical role and continuing activities with individual government agencies, bottom-up planning must also continue. Balance between top-down and bottom-up planning at CSL will be affected by the extent of CSL's activities on the National Information Infrastructure (NII). Policy and other NII-related activities may or may not mesh with CSL's current mission. Nevertheless, the convergence of technologies, notably the anticipated ubiquity of the Internet, the spread of wide area networks, and the advent of sophisticated computer services, suggests that major changes in information technology will happen irrespective of federal policy. CSL should be proactive in defining NII requirements in general and for federal customers in particular. NII requirements, in turn, should influence CSL's research agenda, particularly at the cutting edge, for example, the research agenda of the Advanced Systems Division. A top-down approach could serve both as a basis for assessing CSL's support of commercial research and as a guide for standards-setting activities. Top-down planning could also provide user perspective when resources are allocated for competing metrology needs.

A major challenge in fiscal year 1994 is the planned merger of the Computing and Applied Mathematics Laboratory (CAML) and CSL to create the Information Technology Laboratory (ITL). This merger provides a major opportunity to focus existing resources and programs on the NII. However, management must carefully approach the merger in order to maintain staff morale, and, most importantly, maintain and strongly support core capabilities in applied mathematics, statistical engineering, software and communications testing, spoken and written language corpora development, and standards development. The staff of the two laboratories should work together to define the challenges and opportunities of the merger, providing input on the organization and programs to the NIST director.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Assessment of Technical Programs

CSL's programs are extensive in technical scope and number, with many fine, well-received accomplishments since the panel's fiscal year 1993 assessment. Highlights deserving special mention include the following.

  • Program planning. In response to 1993 panel discussions, the Computer Security Division developed excellent plans for each of its programs, which included detailed activity descriptions, milestones, forecast completion dates, funding sources, staffing, and relationships to other projects.

  • Five-year plan. The Systems and Network Architecture Division's 5-year plan included current and planned projects. Most importantly, it included the criteria for their selection.

  • Virtual library. This activity supports both the NII and NIST's overall mission. A broad consortium (the National Library Association) was used to establish program requirements and the need for standards.

  • Speech corpora. The Advanced Systems Division's Speech Recognition Group continued its production of new speech corpora CD-ROMs and its cooperation with researchers outside NIST to share software and to promote benchmarking. The group's work ensures that future commercial speech recognition products can be developed within a framework that is universally understood.

  • Software standards. The Information Systems Engineering Division's Software Standards Program validates several computer languages and maintains a CSL Validated Products List that has a large influence on government acquisitions. The division's meticulous work is particularly important to the U.S. software industry.

The “virtual library” that will put the CSL products and information bases on-line, and CSL's effort to step back and examine the standards, guidelines, and testing development process for process improvements, are particularly noteworthy. The panel's biggest concern about CSL's technical programs is the cap on FTE staff . The best that CSL can do is to focus on efficiency to meet its mission.

Several CSL programs require attention:

  • Parallel processing. The Advanced Systems Division is doing good work in parallel processing; however, given the resources of the Parallel Processing Group, the panel expected more progress.

  • Software correctness. Although useful, the Software Corrections Program seems out of place in the Systems and Network Architecture Division. The program seems to come from the bottom-up, i.e., to be built around an individual 's interest and skills, rather than from the top-down, that is, as a response to a need identified by management.

  • High-integrity software systems assurance. The “program slicer” is a useful tool. Given that most errors occur in the specification and software design stage, however, the Systems and Software Technology Division should give priority to the early stages of the software engineering process rather than continuing this tool's enhancement.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

The above highlights are discussed in more detail under the assessments of the various divisions.

Recommendations

The following are the panel's recommendations for CSL as a whole.

  • CSL management must carefully approach the merger with CAML to form ITL in order to maintain staff morale and maintain and support CSL core capabilities. The staff of the two laboratories should work together to define the challenges and opportunities of the merger, providing input on the organization and programs to the NIST director.

  • CSL must work creatively to overcome, as best it can, its severe staffing limitations. Inclusion of part-time faculty and students in the FTE ceiling is counterproductive; guest staffing should be controlled by funding, space, and mutual benefit.

  • CSL needs larger, more flexible space. If available space is off-campus, the space should be close enough to NIST facilities that a collegial relationship can be readily maintained with other NIST laboratories. With a mission expanded by the merger of CSL and CAML, ITL will need increasingly high bandwidth connections with the rest of NIST to assure full working ability and space on the main campus for the expanded visitor program.

  • CSL should, with active division buy-in, develop a model for its customer base and technology base. CSL should revise its strategies against its technology base, and, using the guidelines in Setting Priorities and Measuring Results at the National Institute of Standards and Technology (NIST, Gaithersburg, Md., January 1994), assess its current programs and plan new ones. Using its model of the customer base, CSL should look for additional participants for current workshops and consortia or identify new workshops and consortia.

  • Each CSL division should upgrade and perfect its multiyear plans, with detailed activity descriptions, milestones, forecast completion dates, funding sources, staffing, and relations to other NIST projects. CSL divisions should plan jointly. Joint programs involving more than one division (or laboratory) should have a single plan, showing the activities of each division, group, or laboratory.

  • The panel recommends a skills survey and a summary of available versus needed skills, which could be used to guide hiring and transfer within CSL.

  • CSL should assess the value of its “best-in-class” examples of projects that support industrial competitiveness. These “best-in-class” projects (with authentication by external experts) could be used as project role models and justification for future activities.

  • CSL's formal and informal networking with industry appears to be increasing and should continue to be a priority.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
  • CSL needs to adopt clear and consistent metrics to monitor and assess its work. For example, CSL should categorize desktop computers used in the emerging NII and track the extent to which CSL standards, guidelines, and other products are used in those computers.

CSL Responses to Fiscal Year 1993 Recommendations

CSL responses to the fiscal year 1993 panel assessment were, in general, positive and proactive. CSL collaborated in standards setting, demonstrated open systems applications in the Electronic Commerce Program, and extended the scope of NIST's open systems activities to cover interoperability among distributed, heterogeneous systems and the use of the Open Systems Environment Profile as a framework for organizing its seemingly disparate activities, as requested in the panel's fiscal year 1993 assessment.

Given below are some of the panel's fiscal year 1993 recommendations (quoted from the fiscal year 1993 assessment), followed by CSL's responses and panel comments.

  • “The Computer Systems Laboratory should adopt the strategic planning framework as outlined in Visiting Committee on Advanced Technology Annual Report 1993 (Figure 2, p. 5; Visiting Committee on Advanced Technology, National Institute of Standards and Technology, Gaithersburg, Md., January 1994)” (p. 216). CSL will use the strategic planning framework in rewriting its strategic plan. Materials provided to the panel were organized around the framework's format. The panel commends the response to date and stresses the need for continued refinement of planning, reporting, and measuring.

  • “As a prelude to strategic planning, the Computer Systems Laboratory should categorize current and planned projects as either fundamental research or applied research to determine the balance between these two categories, whatever their source of funding” (p. 216). CSL considers its work to be almost entirely applied research. Funding sources for CSL work are driven by the NIST budget process and NIST's strategic planning. The panel believes that some of CSL's work has a broader, hence more basic, application and recommends that CSL reconsider this recommendation.

  • “The Computer Systems Laboratory should determine specific needs of U.S. industry, civilian agencies, and Department of Defense agencies for its services” (p. 217). CSL continues to identify and assess requirements through workshops, forums, management meetings, participation in standards-development activities, contacts with agencies and users, collaboration with industry through agreements and other activities, and many additional interactions. The National Performance Review (NPR), the NII, and the High Performance Computing and Communications (HPCC) initiative have all identified and documented requirements for computer, communications, and security technologies. CSL will initiate efforts to develop and record formal documentation of the requirements identified. The panel believes CSL should do more than its response indicates. CSL will rely on other government programs to provide requirements and assessments. CSL has programs to provide the necessary requirements; however, it is not obvious that CSL has a thorough approach to requirements gathering, program prioritization, and progress assessment.

  • “To minimize internal inefficiencies and maximize external value, CSL could emulate industry's increasingly customer-driven approach by inviting its government clients to participate

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

in annual assessments of CSL's performance” (p. 218). CSL participates in Department of Commerce efforts for total quality management (TQM), which addresses these concerns. These efforts include customer assessments of performance. The panel will continue to monitor the effectiveness of CSL in using clients in the assessment process.

  • “The Computer Systems Laboratory should arrange for increased collaboration among the major users and promulgators of computer-related standards in prioritizing, setting, and adopting standards” (p. 218). This issue is addressed by the administration in The National Information Infrastructure: Agenda for Action (Information Infrastructure Task Force, Washington, D.C., September 1993), which describes the role of the government in promoting the development of the NII. The Agenda for Action called for NIST to establish a panel and work with other appropriate agencies to review the government's involvement in establishing network requirements and standards with domestic and international partners. That review, with input from the private sector and other levels of government, will consider the role of the government in the standards process and will identify opportunities for accelerating the deployment of the NII.

In October 1993, CSL named an interagency panel, the Federal Internetworking Requirements Panel (FIRP), to review open systems network requirements and to recommend policies on the use of networking standards by the federal government. FIRP addressed issues related to the Internet Protocol Suite and Open Systems Interconnection (OSI) specifications, two widely used yet incompatible protocol suites, as well as proprietary networking protocols. CSL will review its report (Report of the Federal Internetworking Requirements Panel, NIST, May 1994) and solicit widespread review and comments before any policy changes are considered.

CSL will pursue broader discussion of these issues through collaboration with the Information Infrastructure Project of the Science, Technology and Public Policy Program at the John F. Kennedy School of Government, Harvard University. A workshop was held in June 1994 to explore the development of standards that are critical to the NII. The workshop drew on experts in information policy, intellectual property, economics of standards and networks, and the role of consortia standards. A book of proceedings and supplemental materials will be produced to help clarify the strategic environment for standards development, help map and evaluate options for the federal government, and lay a foundation for future policy recommendations.

CSL will review the final report of FIRP and the results of the workshop conducted by the Information Infrastructure project. A panel of federal government experts, working in cooperation with the Information Infrastructure Task Force, will be convened to consider the workshop findings and to make recommendations on issues such as establishing a process for developing a high-level architecture for the NII and a process for determining standards needed for the NII, establishing effective planning processes to accelerate the development and implementation of needed standards, establishing a process for policy coordination on standards for the federal government, improving leadership of and strengthening the standards processes, and coordinating efforts of the standards developers.

  • “The Computer Systems Laboratory's goal for standards setting should be interoperability among distributed, heterogeneous systems, rather than the current, more limited vision of open systems” (p. 218). This goal has been adopted. The panel will continue to monitor CSL progress.

  • “The Computer Systems Laboratory should increase its collaboration with other NIST activities, particularly with NIST's Manufacturing Technology Centers” (p. 218). CSL collaborates actively with other NIST laboratories in planning for the HPCC and is involved in the

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

Manufacturing Technology Centers. The panel would like a report on CSL's progress for its fiscal year 1995 assessment.

  • “The Computer Systems Laboratory should develop, in collaboration with its government constituencies, measures for its success and/or failure in setting standards—e.g., vendor and user adoption of such standards” (p. 218). CSL continues to assess the success of its standards-setting activities; drops activities that are not productive, timely, or responsive to user requirements; and is exploring a variety of options for standards setting, including the formal voluntary standards process, the work of consortia, and the use of publicly available specifications. Many of these issues are being addressed by FIRP. Further, CSL management contributed to and will be guided by Setting Priorities and Measuring Results at the National Institute of Standards and Technology. The panel notes the need for further attention to issues of metrics as embodied in its recommendations above.

  • “The Computer Systems Laboratory should consider alternatives to its current facilities, e.g., temporary office space on campus or leased commercial space nearby, if additional space cannot be made available within NIST facilities” (p. 220). NIST is addressing space requirements as part of its building remodernization efforts. Relocation of CSL to an off-campus site is a possibility. The panel believes that this move is critical and CSL should plan it carefully to ensure it does not erode its effectiveness.

  • “The Computer Systems Laboratory should seek to acquire cutting-edge equipment that mirrors the scale and heterogeneity of government and commercial computing” (p. 220). CSL has initiated actions to acquire $1.5 million in new equipment during fiscal year 1994. The panel will continue to monitor progress in this area.

DIVISIONAL ASSESSMENTS

Information Systems Engineering Division
Mission

The Information Systems Engineering Division develops standards for information systems and provides technical assistance to government and industry in data administration, data management technology, computer graphics, and the validation of related software standards.

Strategy

The Information Systems Engineering Division conducts research; develops federal standards and guidelines; provides advisory services to federal agencies; performs technology forecasts; maintains testbed facilities in information systems technology, including data models, database systems, data administration, database design, computer graphics systems, programming languages, and advanced applications development facilities; and develops tools and techniques for testing the functions and performance of information systems software, including programming languages, computer graphics, database software, and data dictionary software both within systems and in computer networking environments. The Information Systems Engineering

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

Division uses three strategic methods to meet its mission. First, it obtains direction and support from industry (primarily information technology developers, integrators, and users), academia, and government through workshops, meetings, and cooperative agreements. Second, it delivers products and services such as Federal Information Processing Standards, profiles, and guidelines; prototypes; demonstrations; workshops; participation in the development of U.S. national standards and international standards; certification that software products conform to standards; and direct assistance to industry and federal, state, and local agencies. Last, it measures progress in terms of products and services delivered and the degree of acceptance in the market.

Resources

The Information Systems Engineering Division has a staff of 45, which includes full-time positions, guests, and research assistants. Fiscal year 1994 funding is estimated to be $2.1 million from STRS, $0.2 million from NIST Competence Building Program funds, and $3.6 million from OA sources.

The division maintains a broad selection of hardware ranging from desktop systems to personal computers (PCs). Current staff and hardware seem adequate, and future hardware acquisition plans will improve the computing power and range of equipment.

Assessment of Strategy

The Information Systems Engineering Division has a good grasp of its customers and their requirements. Conformance testing consumes the majority of effort within this division. Although such testing provides a valuable service to other agencies and industry, it leaves little time to explore emerging technologies and ad hoc standards that will influence future acquisition policies.

Assessment of Technical Programs

Both the Information Systems Engineering and the Systems and Software Technology Divisions are acquiring equipment and developing expertise in the area of virtual reality. Although this is commendable and directly supports the NII, there is little evidence to suggest that the two divisions are establishing a unified, collaborative approach to this area.

The Graphics Software Group performs research, develops standards and guidelines, and provides technical advisory services for the Graphical Kernel System (GKS), Programmers Hierarchical Interactive Graphics System (PHIGS), and Computer Graphics Metafile (CGM) and develops or licenses conformance tests to determine whether graphics products correctly implement graphics standards.

The Database and Graphics Group provides design, specification, acquisition, installation, integration, training, and maintenance support for all laboratories, administrative computing, and networking for the division. In addition, this group is participating in a joint project with the

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

Manufacturing Engineering Laboratory (MEL) and CSL's Advanced Systems Division to support the NII System Integration for Manufacturing Applications Program.

The Data Administration Group performs valuable work in the next-generation International Organization for Standardization (ISO) Information Resource Dictionary System draft and supporting test suite. It is correctly maintaining close collaboration with outside organizations, in particular the Data Administration Management Association (DAMA), which provides an appropriate and important forum for the group to contact “real-world” data administration problems and to transfer information on standards and research activities to the DAMA membership. The Graphics Software Group provides an excellent service in the validation of GKS, PHIGS, and CGM. Although this service has a wide audience, it tends to serve users of high-performance computer equipment rather than of desktop workstations and PCs. There is some concern that the majority of graphics applications that are being developed today are neither GKS nor PHIGS compliant but rather are written using the graphics facilities provided by X-Windows, MS Windows, or OS/2. This group should take the lead in determining the relationships between existing graphics standards and the facilities provided by these window systems. With a large proportion of this group's time spent certifying software for various standards, there seems to be a need for a study of testing techniques and automatic testing methods to provide more efficient testing and more accurate compliance measurements.

The Software Standards Validation Group is responsible for a wide range of standards activities and conformance testing. The group is responsible for validating several computer languages and for maintaining the CSL Validated Products List, which has a large influence in government acquisition activities.

Two of the groups are involved in some aspects of software reuse: the Data Administration Group's support of an Internal Revenue Service reuse effort, and the Software Standards Group's involvement in domain modeling. The area of reuse is important enough at this point in time for NIST to take an active role in emerging reuse technologies.

The Database Languages Group is primarily responsible for the standardization and conformance testing of Standard Query Language in its many forms. The group has an excellent grasp of its customer base and their requirements and performs a valuable service for NIST and its customers.

Recommendations

The following are the panel's recommendations for the Information Systems Engineering Division.

  • The Information Systems Engineering Division needs to find ways to off-load conformance testing to outside sources.

  • The Graphic Software Group in the Information Systems Engineering Division should expand its scope to include PC workstation graphical user interfaces.

  • The Information Systems Engineering Division must clarify and better coordinate its joint program on virtual reality with the Systems and Software Technology Division.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
  • The division must plan more explicitly, including planning for new technology areas. It seems to emphasize support for its ongoing activities, rather than support for new research activities.

  • The division should take an active role in software reuse technologies.

  • The division should study automatic testing methods for software standards.

  • The division should use the Computer Security Division's strategic plan as a model for its own planning.

  • The division should revisit the panel's fiscal year 1993 recommendation on conformance tests (see below).

Information Systems Engineering Division Responses to Fiscal Year 1993 Recommendations

Given below are the panel's fiscal year 1993 recommendations for the Information Systems Engineering Division (quoted from the fiscal year 1993 assessment), with the division's responses.

  • “In the next iteration of its strategic planning, the Information Systems Engineering Division should methodically identify its markets and customers, prioritize services to be provided, appraise the division 's capacity to meet customer needs, schedule delivery dates for services, and plan for the dissemination and delivery of services. Services that have decreasing or marginal utility should be identified for possible elimination or reduction. Also, the division should explore the gains from planning for joint programs with CSL's Systems and Software Technology Division” (p. 221). The division responded that is conducting in-house TQM training to emphasize the importance of the customer. The division evaluated the effectiveness of its programs in database languages, computer graphics, and the Information Resource Dictionary System. Planning is being driven from the CSL level, particularly through the development of HPCC plans. The CSL division chiefs meet weekly, and the CSL group managers meet at least quarterly, to discuss joint projects and concerns. The CSL training program in TQM partially addresses the concern for meeting customer needs. Other actions addressing strategic planning include participation in a CSL team to develop plans for a high-level prototype interface to the NII. Division staff members are developing and executing the detailed plans for HPCC projects. Many aspects of tactical and strategic program planning have been delegated to the group managers and members. All division personnel are encouraged to work on joint projects with other groups, divisions, and laboratories.

    The panel finds this response sidesteps the recommendation for a specified strategy by pointing out CSL's general approach to planning. The panel concurs that planning should be driven from the CSL level and that CSL should assure that plans are consistent and complete in form, format, and content. However, the division should have a specific strategy.

  • “The Information Systems Engineering Division should (1) improve the timeliness of its delivery of conformance tests (conformance tests have more impact during the ramping-up phase

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

than at the end of the product cycle), (2) evaluate the utility of the rewritten Graphical Kernel Standard conformance test as a function of timeliness in the product cycle, (3) apply formal description languages to assist in the development of conformance tests, (4) host industry leaders in a product area in need of testing, to discuss industrial needs for conformance testing and arrange for a collaboration in developing the conformance testing methodology and tools, and (5) search for alternative sources for the development, validation, and performance of conformance tests” (p. 222). The division responded that it has a new project to develop an application interface development environment (AIDE), which will include improved technologies for conformance test development. It developed Cooperative Research and Development Agreements for persistent object bases and remote database access demonstration and testing and is working with the Advanced Technology Program (ATP) and the Manufacturing Extension Partnership to develop new relationships with the health care and manufacturing industries. The panel finds the Information Systems Engineering Division response does not speak to the recommendation.

  • “The Information Systems Engineering Division should influence the entire spectrum of computing in government, including desktop computing rather than only the computing of a few large machines purchased by government. The division should become more involved in developing standards and practices for desktop computers” (pp. 221-222). CSL 's new AIDE will address this concern. The panel will monitor the AIDE project to see how it addresses desktop computing.

  • “The Information Systems Engineering Division should reevaluate its apparent strategy of seeking support for ongoing rather than new activities” (p. 222). The division responded that it has reduced its dependence on OA funding and is seeking industry support, through ATP, for a new program in health care. The panel wishes to see this demonstrated in the division's detailed planning.

  • “The Information Systems Engineering Division should develop an IRDS2 testbed” (p. 222). The division responded that it will develop a testbed if it gets the necessary funding. The panel believes that division planning should address planning for all programs before determining what level of funding is required. Once a priority has been set in relation to other projects, the division can then determine the level of funding required to add a program or the reprioritization necessary to fund this project instead of others.

Systems and Software Technology Division
Mission

The System and Software Technology Division supports users and industry through promotion and evolution of open systems concepts and technology.

Strategy

An important aspect of the Systems and Software Technology Division 's strategy is its relationships with other CSL divisions. The stated focus is not on the division program as an end

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

but on CSL and NIST strategic goals and how well the division supports, promotes, and evolves institutional goals through division-specific and cooperative projects.

The principal strategic objective of the division programs since the mid-1980s has been the development and promotion of open systems environments. This objective enabled the division to respond quickly to key NII issues relating to the quality, reliability, and interoperability of applications that will reside in and run on the national “information superhighway.” This division is divided into three groups.

The Software Engineering Group's objective is to improve the quality and productivity of software technology throughout the software life cycle. OSE standards and specifications are critical to the successful implementation of a widely distributed, heterogeneous environment and the realization of portable and interoperable applications. The division promotes the evolution of profiles that integrate OSE standards and specifications into seamless application domain environments.

The Office Systems Engineering Group's objective is to promote identification, development, and evolution of technologies to improve communication of electronic information. Formal software engineering techniques, methods, and tools are essential to the development of high-integrity software systems, which must be relied on by large portions of society for supplying critical services. Division efforts focus on identifying and providing the tools and methods to change software engineering from an art form into a formal engineering discipline.

The objective of the Distributed Systems Engineering Group is to facilitate the definition of open systems in a heterogeneous distributed environment. The use of information technology for information transfer throughout all aspects of society will become increasingly ubiquitous. The division investigates emerging technologies to understand how information will be used and conveyed in the future in an open and instantaneous manner while maintaining the fidelity and appropriate security and confidentiality of that information.

The division's seven tactical objectives are to (1) identify and nurture critical emerging information technology; (2) promote convergence of underlying base standards and specifications; (3) maintain public acceptance as an unbiased, neutral expert; (4) build broad-based consensus among industry, academia, and government; (5) act as a catalyst in the development and acceptance of critical information technology; (6) maintain both national and international perspective and participation; and (7) seize the opportunity to contribute and make a difference whenever possible.

Resources

Fiscal year 1994 funding for the Systems and Software Technology Division is estimated at $1.7 million from STRS, $0.2 million from NIST Competence Building Program funds, and $3.6 million from OA sources.

The staffing for fiscal year 1994 includes 41 permanent staff, 3 research associates, 4 visiting faculty, 5 students, and 1 guest researcher. Computer resources appear to be adequate, given the size of staff. It is not clear, however, that staffing is sufficient for all activities planned.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Assessment of Strategy

The Systems and Software Technology Division's strategy is well aimed to take advantage of the development of an “electronic government” as a part of the NPR, part of an overall NII, which promotes a vision of a web of networks, databases, and consumer electronics. Government research programs are expected to promote technological innovation and development of applications for the NII. Given the limited resources, however, the strategy needs more focus.

As a result of the NPR and NII initiatives, funding for the division is expected to increase significantly. However, a conflicting NPR goal of reducing the federal work force by some 252,000 translates into a limited opportunity for this division to hire staff necessary to meet its objectives.

Assessment of Technical Programs

The Systems and Software Technology Division's individual programs are of good quality overall, but the set of programs lacks cohesion. The Office Systems Group is more focused than at the panel's fiscal year 1993 assessment, but the Software Engineering Group needs to sharpen its focus, especially with the revival of integrated software engineering environments laboratories.

The key program areas in the Software Engineering Group are integrated software engineering environments (ISEE), high-integrity software systems assurance (HISSA), and telecommunications security analysis center (TSAC). The group's stated objective to improve quality and productivity of software technology throughout the software life cycle is indeed a lofty goal and perhaps somewhat misleading. One would expect to see activities that examine software development methodologies, alternative software life-cycle models, and their supporting environment and tool sets. However, the current and planned activities for this group center on a specific approach to a software engineering environment, the Portable Common Tool Environment (PCTE), a fine-grained approach to determining software safety (the program slicer tool), and an activity addressing telecommunications security. The group's stated objectives are much too broad to be successfully addressed with available resources. Further focus of the stated objectives should be a group priority.

In its 1993 assessment, the panel criticized the emphasis on PCTE as the delivery mechanism for the ISEE. There was concern that the concentration on PCTE would lead to a “standardization” of PCTE, not on the definition and specification of an ISEE, of which PCTE would be a representative example. It is clear that activities since the fiscal year 1993 assessment have addressed this concern, particularly work attempting to reconcile PCTE and the common object request broker architecture. The revival of the ISEE laboratories is important but requires considerable resources. In fact, the revival could become the focus of this group, working with other government agencies such as the Advanced Research Projects Agency (ARPA) (Software Adapatable Reliable System, or STARS, program) and Rome Laboratory (the software environment project ProSLCSE) to give ISEE a jump start.

The group's HISSA project addresses software assurance for already-coded systems. The panel believes that there is far more “bang for the buck” in examining development processes and developing specifications for safety-critical software development processes than in addressing the

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

problem at the code level. Most problems in addressing safety-critical requirements occur at the specification and design stage of development.

The Office Systems Engineering Group is developing the Virtual Library project. This activity is exciting and appropriate for this group. The project gives the group a clear focus and responsibility in an area that has recently attained national prominence because of the NII, and the end-product will clearly support the NIST goal of disseminating information to the public. The panel is concerned, however, that the product-oriented approach of the project may draw attention away from the more important goal of establishing standards and guidelines for the entire area of electronic document exchange and hypermedia approaches to information organization and retrieval. In other words, the Virtual Library project must result in more than a product. It should establish a set of requirements and specifications to support universal electronic document exchange. Working with appropriate workshops and industry groups, the group should accelerate development of the necessary standards and guidelines to support this objective.

The Systems and Software Technology Division should give attention to process engineering; commercial sectors are very active in process engineering, however, there is little guidance in standards. Now is the time for NIST to get involved and take a leadership role in these standards.

A significant concern in industry, particularly among government contractors but also increasingly among commercial software development organizations, is the Software Engineering Institute (SEI) capability maturity model (CMM). Although there is some interaction between this division and the SEI in several areas, the panel recommends that the division become more active in understanding and promoting the CCM, particularly with respect to how it maps into the standards arena.

The division should also consider Ada as one of the programming languages of choice. The panel finds a surprising ambivalence to the promotion and use of Ada as the design and programming language of choice. There seems to be a growing body of evidence that Ada as a design tool and programming language has successfully addressed concerns in the production of mission-critical software systems.

Recommendations

The following are the panel's recommendations for the Systems and Software Technology Division.

  • The Systems and Software Technology Division should revisit its objectives and strategy statements and revise them to define more focused and achievable goals.

  • The division should examine development processes and develop specifications for safety-critical software development processes.

  • The division should give attention to standards for process engineering.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
  • The Virtual Library project should work with industry to establish a set of requirements and specifications to support universal electronic document exchange.

  • The Information Systems and Engineering Division should extend relationships with other laboratories, focusing on working with other government agencies, as in the current ARPA (STARS) and Rome Laboratory (ProSLCSE) collaborations, to give the Information Systems and Engineering Division a jump start.

  • The division should take a more active role in the SEI capability maturity model, developing a program with SEI and taking a leadership role in mapping the CMM to the Generic Life Cycle Model.

  • The division should also consider Ada as one of the programming languages of choice.

Systems and Software Technology Division Responses to Fiscal Year 1993 Recommendations

Given below are the panel's fiscal year 1993 recommendations for the Systems and Software Technology Division (quoted from the fiscal year 1993 assessment), with the division's responses and panel comments.

  • “The Systems and Software Technology Division should take a more active role in developing the U.S. and international standards definitions of the neutral (meta) data formats (e.g., CASE Data Interchange Format, the conceptual schema/modeling facility, and STEP/Express), all addressing the problem of transferring and storing (in IRDS2) data and software definitions” (p. 226). These issues are being addressed by the Information Systems Engineering Division, which is working in the general area of neutral data formats and specifically the CASE Data Interchange Format. It is collaborating with NIST's Manufacturing Engineering Laboratory and with the Building and Fire Research Laboratory in the development of the Standard for Exchange of Product Model Data (STEP). The “conceptual schema/modeling facility” is essentially a research topic at the present time; the division is not currently participating in that work.

  • “The Systems and Software Technology Division should play a more proactive role in the ISO 9000 and Software Engineering Institute's Capability Maturity Model assessment, rather than merely following the progress of these two important standards-setting activities” (p. 227). A senior staff member in the Software Engineering Group has the task of tracking and analyzing information technology issues related to ISO 9000. The Computer Systems Laboratory and SEI recently signed a memorandum of understanding (MOU) that provides for stronger collaboration between them. One of the planned areas of collaboration is ISO 9000 and the SEI's CMM. In addition to ISO 9000 and its potential impact on the U.S. information technology industry, the division is also examining the growing impact of the ISO/International Electrotechnical Commission (IEC) quality and safety standards on the industry and the ability of U.S. businesses to market products containing software in Europe. The panel recommends that CSL develop a program in conjunction with SEI based on the MOU.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
  • “The division should provide a significant, positive leadership role in mapping the Capability Maturity Model to the division's understanding of the Generic Life Cycle Model” (p. 227). The division conducts activities with industry and is developing additional efforts to coordinate efforts with industry. Since the fiscal year 1993 assessment, this has included formation of the NIST-Department of Defense-Object Management Group-sponsored North American PCTE initiative, to promote adoption of an open, public software engineering tool repository interface standard; a highly successful workshop on digital systems reliability and nuclear safety; the establishment of TASC to address the issues of security and reliability based on government and industry priorities; and a proposal to establish a Center for High Integrity Software Systems Assurance to address industry and federal needs in improving the development and assurance of high-integrity software systems. The panel does not believe that this response addresses the issue and reiterates its recommendation.

  • “The Systems and Software Technology Division should investigate the use of fellows from industry, contractors, and part-time retired NIST staff. . . .” (p. 227). The division has pursued a variety of staffing opportunities in the past, including guest researchers from industry and faculty appointments, and will continue to explore all possible staffing arrangements that are consistent with NIST's administrative direction in order to meet objectives.

  • “The division should monitor the adequacy of its equipment continuously, even though its current facilities are adequate” (p. 227). The division will continue to do such monitoring. The panel will continue to monitor this key resource area.

Computer Security Division
Mission

The mission of the Computer Security Division is to develop standards, provide technical assistance, and conduct research to support both government and industry in the cost-effective protection of information and information technology (computer systems, communication networks, and applications). In its briefing for the panel, the division narrowed its mission focus to emphasize providing management and operational guidance, technology, and standards and technology transfer that improve security for the evolving NII.

Strategy

The Computer Security Division's strategy statement emphasizes that cost-effective security measures should be based on a rational balancing of costs and risks and that further security measures must be an appropriate combination of policies, plans, procedures, and technology. The Computer Systems Division addresses these elements through a program that encompasses security management, technology, and protocols, together with product criteria, evaluation, and certification. The division's program management strategy includes a mission focus that centers on the evolving NII; the leveraging of limited resources through collaboration with industry and federal agency assistance, e.g., possibly tapping into the NIST ATP to leverage resources; and efforts to promote government-industry-international cooperation.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Resources

The Computer Security Division has 42 FTE staff members. One secretary and two group managers were being recruited at the time of the assessment. Funding for fiscal year 1994 is estimated at $3.7 million from STRS and $1.8 million from OA sources. The division has a variety of PCs, workstations, local-area networks (LANs), and software, most of which is current technology.

Assessment of Strategy

The working strategy of the Computer Security Division appears reasonable but is not clearly stated in the strategic planning documents and presentations made to the panel. The strategic planning documents and presentations made to the panel include generic information (e.g., cost-effective security measures balance costs and risks) and touch on items that should be part of the division's strategy (e.g., developing security protocols), including items (e.g., leveraging resources) that are well-accepted practices for achieving any strategy. Both the strategic planning documents and presentations are too general and do not clearly specify what the division is doing to achieve its stated mission. A good mission statement does not need to be changed frequently. However, strategies are subject to continual change because of federal policy shifts, funding changes, and technology advances.

Since the 1993 panel assessment, the division has developed a plan for each of its programs with titles, descriptions of activities, funding sources and amount, expected completion dates, milestones and deliverables with due dates, staffing, other participants, and relationships to other projects or activities. These program plans will prove to be an extremely useful management tool for allocating resources as well as for monitoring program progress and results. Program plans should be used to track program progress and deliverables. Program plans currently project expected completion dates. Such dates should be compared against actual completion dates.

Assessment of Technical Programs

The Computer Security Division has evaluated the relevance of its technical programs to the NII as well as to trends and changes in the information security environment. The division has increased its resources in support of cryptography, user identification and authentication, “firewalls,” public-key infrastructure, NII security incident response, goal security architecture, common criteria and evaluation, generally accepted system security principles, and information dissemination. The division has decreased its emphasis on virus prevention and detection, contingency planning, Government Open System Interconnection Profile (GOSIP) security profiles and protection, and risk management and completed the development of the Computer Security Handbook. Except for the decrease in virus prevention and detection activities, the panel agrees with the Computer Security Division's reallocation of resources. In the NII, opportunities for virus infection may significantly increase. Both government and private-sector users could benefit from research on solutions to the problem of malicious software risks.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

The panel notes an absence of work on distributed authentication (verification of identity and access authorization across networks) using Kerberos-like functions. The use of token-based profiles for distributed authentication is widely discussed in information security forums, and at least three products are commercially available to fill the need. The security community would benefit from the division 's guidance in distributed authentication.

Although the division is a logical candidate to be the administration 's key escrow agent for public-key computer security systems, the magnitude of the division's key escrow-related work has already resulted in a shift of resources away from areas (e.g., firewalls) that would have been beneficial to both government and commercial users, and it has delayed related deliverables.

The panel notes that, whereas “1 percent of [NIST's] $520 million budget (is devoted) to computer and communications security” (Computerworld, May 9, 1994), standard corporate programs allocate about 10 percent of their computer systems budget to computer and communications security.

Recommendations

The following are the panel's recommendations for the Computer Security Division.

  • The Computer Security Division should develop a crisper and more complete strategy statement to support the Computer Security Division 's mission, which should also be consistently stated in all strategic planning documents and presentations.

  • The Computer Security Division should use program plans to track progress and deliverables.

  • The division should dedicate increased resources to virus prevention and detection and distributed authentication.

  • The division should consider assigning and increasing resources to programs having broad-based applicability, such as distributed authentication and Internet firewalls.

Computer Security Division Responses to Fiscal Year 1993 Recommendations

Given below are the panel's fiscal year 1993 recommendations for the Computer Security Division (quoted from the fiscal year 1993 assessment), with the division 's responses and panel comments.

  • “The Computer Security Division should develop criteria for determining how much of a networked system must be trusted. Much of this work would tie into the security needs of the National Information Infrastructure initiative” (p. 230). The division responded that it is looking into this issue and will report to the panel. Since the fiscal year 1993 assessment, the division has worked on four items that it believed could lead to the criteria: firewalls, identification and authorization, public-key infrastructure for digital signature, and Defense Information Systems

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

Agency separation kernel technology. It is not clear to the panel how the division's work on elements of the above four items will be combined to develop the criteria.

  • “Since several research projects on detection of intrusion are well under way outside NIST, the division should rely on external research rather than start an independent project” (p. 230). The division is not doing independent research in this area but is investigating and applying available techniques. The purpose is to provide guidance and transfer technology, and support the security needs of the NII. The panel finds CSL's approach reasonable.

  • “The Computer Security Division should continue to consult, develop guidelines, and provide training programs, especially in computer security risk management and analysis, and should complete or upgrade its computer security handbook” (p. 230). Although not reviewed by the panel, the Computer Security Handbook is complete. A training program on computer security is being developed.

  • “The Computer Security Division should consider expanding its activities in ensuring the security of medical information” (p. 230). The Computer Security Division responded that medical information is a critical application area but that resources are too limited to pursue specific solutions for protecting medical information, and many of the generic techniques developed for protecting sensitive information of all kinds will be applicable to the protection of medical information as well. Health care is a focus area of the ATP, and CSL division managers have established a formal relationship with ATP. The panel will assess the Computer Security Division's progress at the fiscal year 1995 program reviews.

  • “The division should take a much more active role in planning for research on ensuring overall National Information Infrastructure security requirements” (p. 230). The division participates in a CSL team to develop plans for a high-level prototype interface to the NII. Division staff members are developing and executing the detailed plans for HPCC projects, and division personnel are encouraged to work on joint projects with other groups, division, and laboratories. The Computer Security Division adjusted its focus this year to high-impact areas, most of which support the NII. The panel will monitor the progress of the planned deliverables in these areas.

Systems and Network Architecture Division
Mission

The Systems and Network Architecture Division promotes agreements and technical solutions that facilitate electronic information transfer.

Strategy

The activities that are within the scope of work for the Systems and Network Architecture Division include activities that support and strengthen the network infrastructure (e.g., routing and forwarding protocols, management protocols, security protocols); result in useful applications or groups of applications (e.g., electronic mail, directory services, electronic commerce); and are

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

needed to support previously listed activities that are not pursued sufficiently elsewhere (e.g., protocol testing and validation, performance studies, software reliability).

The division's customers are the user community in the government and commercial sectors that rely on networking technology; vendors of networking technology also are some of its most intensive users. Activities are focused on enabling new and better economies of scale and lower prices and, through standardization, curtailing duplication and thus reducing development costs and increasing interoperability.

The division's strategic thrusts are to participate in setting voluntary standards, developing and proposing prototype standards, integrating prototypes of standardized components to demonstrate electronic commerce applications, and technology transfer.

Resources

The Systems and Network Architecture Division's current available resources include 30 computer scientists and 6 guest scientists, $230,000 of in-kind equipment from industry, and equipment consisting of a mix of Sun workstations and PCs connected to a 10-Mb Ethernet and a LAN linked to a fiber-distributed data interface (FDDI) ring and through it to the network service provider, SURANET. Fiscal year 1994 funding is estimated at $2.4 million from STRS and $3.1 million from OA sources.

Resources appear adequate, except for a lack of space to hold demonstrations and conferences. A planned hookup to high-speed networks is jeopardized by the impending move of the division to off-site quarters. The laboratory facility should be planned to avoid disruption of the pending network connection, which might necessitate not moving the division laboratory.

Assessment of Strategy

The mission statement does not suggest a focus for the Systems and Network Architecture Division strategy. Excellent work is being done in the development of plans and the selection of specific projects; however, the big picture is missing. As a result, it is not clear to the panel how the division can measure its effectiveness. For example, participating in standards committee work and gaining competence in specific technological areas central to networking and network architectures are positive and useful efforts but are not clearly linked to a short- and long-term vision of the division's objectives.

The division has developed a detailed 5-year strategic plan that goes into current and planned projects and the criteria for their selection. The NII and Electronic Commerce initiatives, however, should have a significant impact on these plans and specific projects in the near future. The division should document how its networking efforts are linked to the above initiatives and measure progress based on achieving specific objectives related to usage-oriented metrics. If each enabling technology were more clearly related to these more visible programs, support might be more forthcoming. In particular, the objective of making electronic commerce accessible to small business should be specified among goals and objectives.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Assessment of Technical Programs

For the Systems and Network Architecture Division, the panel supports relaxation of efforts on GOSIP adherence, development of Internet Protocol Suite protocols, and recognition of Internet Engineering Task Force standards as recommended in the FIRP report.

The division's research appears to be state of the art as evidenced by its contributions to various technical committees and organizations. The work on “ software correctness,” however, appears to be out of scope for the network focus of the division's mission. It seems that existing staff skills result in an interest in an important area, but not one that matches the networking focus of the division.

The relation of the division work on secure electronic mail to privacy-enhanced mail, the Massachusetts Institute of Technology effort, and other state-of-the-art research should be documented and clarified so that the Systems and Network Architecture Division's efforts and goals can be compared with other research efforts.

Recommendations

The following are the panel's recommendations for the Systems and Network Architecture Division.

  • The Systems and Network Architecture Division's planning document should be recast in terms of the NII and electronic commerce so that the role and responsibility of the division are clear and its projects can be clearly identified with these national initiatives. The division's role in support of the ATP should also be specified in its strategic plan.

  • Division projects on the NII and electronic commerce should focus on interoperability testing, perhaps using the Electronic Commerce Integration Facility or some similar user-perspective basis on which to test the adequacy and usability of standards and prototypes.

  • The investigation into secure electronic mail needs to sharpen its goals and objectives.

  • The division's mission statement needs to be analyzed, consolidated, and made more visible. The current mission statement is not useful in focusing the strategy.

  • Laboratory facilities must be planned so as not to disrupt the division 's pending network connection, even if this means keeping the division on the main campus while other division move off-campus.

Systems and Network Architecture Division Responses to Fiscal Year 1993 Recommendations

In general, the response to the fiscal year 1993 recommendations was excellent. For example, the short- and long-term planning was improved significantly. Given below are the

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

panel's fiscal year 1993 recommendations for the Systems and Network Architecture Division (quoted from the fiscal year 1993 assessment), with the division's responses and panel comments.

  • “A division-wide skills assessment is in order if the Systems and Network Architecture Division is to address a broader network mission ” (p. 233). The CSL assesses skills continuously on an informal basis in assigning staff members to projects and developing plans for initiatives. The panel had intended that a skills survey would be conducted and available versus needed skills summarized. This summary would then be used to guide hiring and transfer within the CSL to address efforts that cannot be currently pursued because of unavailability of skills.

  • “In the area of electronic commerce, the Systems and Network Architecture Division should focus on only a few, well-defined pilot projects, because of the large scope of electronic commerce” (p. 233). Under the fiscal year 1994 initiative, CSL will collaborate with industry and other agencies to develop or apply technologies that enable electronic commerce in general and particularly as it applies to manufacturing electronic and mechanical components and subsystems. Fiscal year 1994 objectives include developing an electronic commerce integration and demonstration facility at NIST and soliciting industry participation and demonstration of products and investigating basic interoperability and integration issues related to use of value-added networks and electronic mail.

  • “The division should develop a program that focuses on automated standards testing, explores the use of formal definitions, and supports the development of tools such as ESTELLE and commercial automated test generators” (p. 233). The division responded that it will be focusing on addressing integration and interoperability issues related to electronic commerce, using commercial off-the-shelf products where available and developing prototype components when no commercial products are available. Evidence of this type of automated testing development was not presented to the panel during its fiscal year 1994 assessment. Test generation was presented by CSL as a general need, but no plan or organized effort toward this end was presented.

  • “The Systems and Network Architecture Division's exemplary strategic plan should be revised to keep pace with NIST 's expanding mission and requirements related to its role in developing the National Research and Education Network” (p. 233). The division will continue to update the plan.

  • “The Systems and Network Architecture Division should move beyond viewing its mission as bounded by open (i.e., nonproprietary) protocols. The division should retain its current internal Open Systems Interconnections (OSI) focus but should also pioneer in the promotion of OSI for use in the United States. The division should provide broad support for its customers in defining, operating, and evolving heterogeneous network systems. This approach should include refinement of the open system network concepts and continued development of associated supporting concepts, such as distributed systems networking, specific frameworks for addressing user requirements, and standards to ensure the interoperability of heterogeneous network systems” (pp. 233-234). The draft report of FIRP addresses many of these issues and the issues raised in the 1993 recommendation immediately following. CSL will review the FIRP recommendations and conduct other strategic studies to identify future actions that will support heterogeneous network systems. The panel anxiously awaits the development of a broader mission for the division.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
  • “The Systems Network Architecture Division should address problems that users experience with proprietary (or partially proprietary) systems and networks. . . . The Systems and Network Architecture Division should help ensure that tomorrow's open networks provide access to these major systems from workstations that span networks using open and de facto standards”. . . . (p. 234). The role of proprietary and de facto standards in achieving open networks is being reviewed. The panel awaits the review conclusions.

  • “The division should form a task force with counterparts in the Manufacturing Engineering Laboratory in order to define their separate and joint responsibilities for computer systems and networks—especially where real-time processes are involved” (p. 234). CSL and MEL are working together in support of the HPCC. Systems assurance, reliability, and security are being addressed in various aspects of the HPCC programs.

Advanced Systems Divisions
Mission

The Advanced Systems Division develops commercial enablers for key HPCC technologies likely to lead to large new markets. These enablers are cooperatively agreed on reference materials, tests, and instruments that accelerate the commercialization of otherwise “unfocused” academic and industrial research by providing a user (application)-specific focus, “broker” two or more industry segments whose cooperation via interfaces is essential for promoting new markets, and provide the measurement and instrumentation methods necessary to “bridge” research prototypes into a commercial paradigm or demonstrate the commercial merits of two or more competing research paradigms.

Strategy

The Advanced Systems Division has divided its programs into (1) high-performance computer systems, (2) digital communications technology, and (3) human-machine interfaces. The primary approach to selecting projects has been bottom-up. Consultations with researchers, industrial representatives, and, in particular, ARPA principal investigators suggest important topics that either receive insufficient attention or involve issues of metrology in which the Advanced Systems Division has a comparative advantage. The Advanced Systems Division research is designed to fill the gap.

Resources

Total estimated fiscal year 1994 funding is estimated at $4.0 million from STRS, $0.7 million from NIST Competence Building Program funds, $0.5 million from NIST Information Technology Services funds, and $5.3 million from OA sources. In-kind equipment resources total $300,000. This funding supports roughly 90 FTE employees, both in-house and contractor.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Assessment of Strategy

The Advanced Systems Division must develop a comprehensive division strategy from which individual group and program objectives can be developed. The group's current mission, strategy, and objectives often lack focus. As a result, priorities and resource allocations are not always obvious.

The issue of whether any research organization generates its work from a top-down or bottom-up perspective is eternal. On the one hand, CSL and the Advanced Systems Division can never be more than pygmies among giants in terms of resources; therefore, the division must employ a niche strategy. On the other hand, CSL and the Advanced Systems Division (together with ARPA) have unique functions (as the primary technical representative for government requirements and perhaps the best honest broker for the nation's standards effort) that dictate a broader view.

The balance between the two functions has to be affected by all the activity coming under the rubric of the NII. These critical functions, in turn, should influence CSL's research agenda, particularly at the cutting edge of the technology, where the Advanced Systems Division is positioned. A top-down approach would serve both as a basis for assessing the adequacy of supporting commercial research and as a guide for the organizational location of the most urgent standards-setting activity. A top-down approach would also provide a context for evaluating user perspective when resources are allocated to competing areas of metrology.

The division's desire to better support U.S. industry is increasing. As one might expect, the objectives for the Advanced Systems Division groups are considerably broader than resources allow. The challenge for division and CSL management is to prioritize the objectives, document the opportunities selected (the “best” based on mission, strategy, and priorities), and proceed with the programs selected, using available resources.

High-speed networks research is a critical part of the division's work. The impending move to off-site quarters is threatening to disrupt plans for connection to the high-speed network. Even if the laboratory stays, moving the staff will also adversely affect the experimentation with the network. Careful planning is needed.

The Advanced Systems Division typically works with diverse applied research topics (e.g., various forms of man-machine interfaces). A beginning-to-end testing process based on user scenarios could help prevent oversights resulting from testing a technology without a user community perspective. Such a process should be more productive and less costly than the exhaustive test procedures currently employed.

Projects are dropped when they are deemed ready for maintenance (e.g., the small computer systems interface [SCSI] and FDDI). The division 's transition process should be better defined and documented.

Assessment of Technical Programs

The Advanced Systems Division's technical programs are conducted in a competent manner. Papers, patents, and peer interaction with other institutions such as commercial laboratories or ARPA point to the high level of technical expertise and relevance accorded Advanced Systems Division work.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

In the Parallel Processing Group, efforts are focused in the right direction, if one accepts Multiple Instruction, Multiple Data (MIMD) as the best opportunity for advancing in high-performance computing. Given the size and history of this group, the panel expected more progress since the fiscal year 1993 assessment. The division has initiated work with parallel computer architectures (e.g., MultiKron work with MIMD). With MIMD and Single Instruction, Multiple Data as well as other new architectures on the horizon, the Advanced Systems Division must extend its experimentation with these new machine architectures. The development of compiler languages for controlling the new machine architectures needs careful attention. Such languages must not be sequentially oriented but designed to take advantage of the new architecture. Object-oriented concepts provide an alternative means of programming that is not sequentially based.

Fiscal year 1993 accomplishments for the Data Storage Group were not obvious; however, the plans outlined for the panel's review seem appropriate.

Again, it was difficult to assess the level of achievement of the Distributed Systems Group; however, the group is doing much to help rationalize the speech recognition industry and considerable productivity was indicated in materials the division provided for the panel.

The Speech Recognition Group's continued production of new speech corpora CD-ROMs and their cooperation with researchers outside NIST, both to share software and promote benchmarking, ensure that commercial speech recognition products are being developed within a framework that is universally understood.

The Image Recognition Group's benchmarking activities are appropriate; however, if the different research communities are to be better linked, there is a need to continually assess the value of developing new classification systems.

Advanced Communications Group activities in the area of asynchronous transfer mode are encouraging. There is considerable value in this group's obtaining experience by actually using commercially available leading-edge technologies. The use of MultiKron technology from the Parallel Processing Group provides a good example of cross-communication between CSL groups.

The Advanced Systems Division should consider several interoperability areas for additional work, such as developing a methodology for modularizing and staging complex standards (e.g., OSI, Integrated Services Digital Network [ISDN], and Ada suffer from being large and complex standards that must be implemented all at once); revisiting ISDN standards that permit a message stream (e.g., at 16 bits per cycle) over the same line to be dynamically allocated to various message types (e.g., voice, video, text, graphics); and determining technological prerequisites for proto-standards for query languages that support logical inferences and context-based (as opposed to keyword-based or index-based) text searches.

Recommendations

The following are the panel's recommendations for the Advanced Systems Division.

  • The Advanced Systems Division needs a baseline architecture or comprehensive model on which to relate the various division projects. In this way, plans can clearly indicate the role and priority of each project. Also, a comprehensive model could help with staffing decisions.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
  • Even though the division's metrology work appears to be highly cost-effective, the panel believes more work over the next few years should be oriented toward interoperability issues and toward developing standards derived from them.

  • The division should carefully plan to minimize disruption by the planned move to off-campus space.

  • The division should consider development of computer languages for controlling new machine architectures.

  • The division should consider more beginning-to-end testing processes based on user scenarios to avoid oversights resulting from testing a technology without a user community perspective.

  • Projects are dropped when they are deemed ready for maintenance (e.g., SCSI, FDDI). The Advanced Systems Division's transition process should be better defined and documented.

  • The Advanced Systems Division should consider additional work in several interoperability areas as discussed above.

Advanced Systems Division Responses to Fiscal Year 1993 Recommendations

Given below are some of the panel's fiscal year 1993 recommendations for the Advanced Systems Division (quoted from the fiscal year 1993 assessment), with the division 's responses and panel comments.

  • “The Advanced Systems Division should collaborate with other divisions within the Computer Systems Laboratory in developing program priorities ” (p. 237). CSL responded that planning is being driven from the CSL level, particularly through the development of HPCC plans. The CSL division chiefs meet weekly, and the CSL group managers meet at least quarterly, to discuss joint projects and concerns. Although work is under way in virtual reality, the panel thinks it needs to be defined as a CSL program, with a clear leading division, and the complementary programs in other divisions held to milestones and deliverables.

  • “The Advanced Systems Division should collect and maintain standard databases for developing and evaluating speech recognition systems ” (p. 237). With new and increased funding in fiscal year 1994, the division will initiate additional research and development of spoken-language-understanding technology in a new applications domain by developing a prototype that provides information about library catalog holdings in response to spoken natural language queries. Data (e.g., corpora) collected in this effort will be shared with the research community. The Advanced Systems Division appears to be collecting considerable data on speech recognition systems.

  • “The Advanced Systems Division should sponsor open competition in the development of speech and handwriting recognition technology, and the division should make better use of its special strengths and expertise to advance speech recognition research in general rather than serve

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

only a small part of the research community” (p. 237). ARPA contractors and others participate in CSL's annual benchmark tests for speech recognition, spoken language understanding, and, since the fiscal year 1993 assessment, language identification technologies. State-of-the-art work that cites these results frequently appears in publications such as those resulting from the Institute of Electrical and Electronics Engineers International Conferences on Acoustics, Speech, and Signal Processing. Corpora that NIST processes and produces are available to the research community through the Linguistic Data Consortium and in some cases also through the National Technical Information Service.

  • A major conference focusing on information retrieval techniques, TREC-3, was planned for November 1994. At the time of this assessment, 55 groups (up from the 31 groups participating in TREC-2) were signed up to participate and to run the full benchmarking tests. Among these groups are 25 commercial firms, including IBM, Xerox, Bellcore, NEC, and TRW; most of the major information retrieval companies, including Verity, BRS, Mead Data Central, West Publishing, and Oracle; and several small businesses that have benefited immensely from TREC as a validation of effective retrieval, including Conquest Software and CLARIT Corp. Fifteen of the participants are U.S. universities, including all the major research groups in information retrieval. Fifteen are foreign universities.

    The division is running the competition on the Second Census Optical Character Recognition Systems Conference and may do some future work in this area with NIST funds. Data exist for a third conference, and the UNIPEN project (a unique pen-based benchmarking activity for optical character recognition) would like the division to run a competition to evaluate on-line handwriting recognition. The division is discussing a similar approach with the Federal Bureau of Investigation on face recognition system evaluation. The Advanced Systems Division appears to be well involved in speech recognition.

  • “The division should expand its efforts to inform others about the MultiKron chip through publications, reports, and other mechanisms, e.g., professional contacts. The division should explore additional uses of the chip” (p. 237). The introduction of the virtual memory extension and Sbus interface boards will provide many new opportunities for users to devise their own uses. With the supplied software, MultiKron can be installed and then used as desired. Software links are easily within the reach of the programming community, whereas new hardware wiring is not.

    MultiKron, version I, showed that a number of features could be improved. The need for a collection network is not attractive, and so the new ARPA MultiKron II effort focuses on not needing one. Second, the 16 register/clocks were so popular among system developers at Intel that more were needed. MultiKron II will have virtual registers that number in the thousands; this feature allows users to access these attractive items and yet still allows the system to do fast context switches. In May 1993, ARPA began another 3-year funding cycle for this work. There are fresh avenues to be pursued in refining the device.

    The Advanced Systems Division does not appear to be devoting resources to publicizing and finding new uses for the MultiKron chip. The panel believes that this function may be best done by its manufacturer(s).

  • “The division's classical optical disk metrology should be further promoted” (p. 237). Earlier work on the testing methodology to predict optical disk life expectancy values is the subject of NIST special publication (SP) 500-200, which has been widely distributed and has been the basis for subsequent standards work.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×

The current data integrity (care and handling) study for the National Archives and Records Administration (NARA) will be the subject of a report to be published in the Fall 1994. NARA will be disseminating this report to the archivist community, and it will be widely available to industry and government agencies, including the optical disk standards community. (SP 500-101 on the care and handling of computer magnetic storage media was very widely distributed for 10 years and is still the subject of inquiries.)

Optical disk error monitoring and reporting work is the subject of American National Standards Institute/Association for Information and Image Management (AIIM) standard MS59-199X, which was on the AIIM ballot at the time of this assessment. The division played a key role in developing the ISO/IEC version of this standard. This work is jointly sponsored by NARA and two other federal agencies. Several manufacturers have indicated an intention to produce compliant products.

  • “. . . The division reports annually to the panel on what it can do given its expertise, funding sources, and time limitations. The division should also specify what it is not working on and why. . . .” (p. 237). CSL responded to the panel with a listing of programs it has chosen not to work on. However, the Advanced Systems Division does not appear to be systematically comparing its work agenda against its total range of choices. The panel reiterates the importance of this activity as part of the division's planning process.

Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 197
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 198
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 199
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 200
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 201
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 202
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 203
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 204
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 205
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 206
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 207
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 208
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 209
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 210
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 211
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 212
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 213
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 214
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 215
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 216
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 217
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 218
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 219
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 220
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 221
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 222
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 223
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 224
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 225
Suggested Citation:"8 Computer Systems Laboratory." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994. Washington, DC: The National Academies Press. doi: 10.17226/9198.
×
Page 226
Next: 9 Computing and Applied Mathematics Laboratory »
An Assessment of the National Institute of Standards and Technology Programs Fiscal Year 1994 Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!